-
Notifications
You must be signed in to change notification settings - Fork 3
/
Copy pathreverse-engineering-xcode-issue-crash-symbol.html
334 lines (293 loc) · 21 KB
/
reverse-engineering-xcode-issue-crash-symbol.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
<!DOCTYPE html>
<html lang="en">
<head>
<script src="https://use.fontawesome.com/afd448ce82.js"></script>
<!-- Meta Tag -->
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<!-- SEO -->
<meta name="author" content="Bruno Rocha">
<meta name="keywords" content="Software, Engineering, Blog, Posts, iOS, Xcode, Swift, Articles, Tutorials, OBJ-C, Objective-C, Apple">
<meta name="description" content="Have you ever had issues with Xcode not symbolicating crashes? Me too, and I found the fix by reverse engineering the IDE.">
<meta name="title" content="Reverse engineering a 5 year old Xcode issue">
<meta name="url" content="https://swiftrocks.com/reverse-engineering-xcode-issue-crash-symbol">
<meta name="image" content="https://swiftrocks.com/images/thumbs/thumb.jpg?4">
<meta name="copyright" content="Bruno Rocha">
<meta name="robots" content="index,follow">
<meta property="og:title" content="Reverse engineering a 5 year old Xcode issue"/>
<meta property="og:image" content="https://swiftrocks.com/images/thumbs/thumb.jpg?4"/>
<meta property="og:description" content="Have you ever had issues with Xcode not symbolicating crashes? Me too, and I found the fix by reverse engineering the IDE."/>
<meta property="og:type" content="website"/>
<meta property="og:url" content="https://swiftrocks.com/reverse-engineering-xcode-issue-crash-symbol"/>
<meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="https://swiftrocks.com/images/thumbs/thumb.jpg?4"/>
<meta name="twitter:image:alt" content="Page Thumbnail"/>
<meta name="twitter:title" content="Reverse engineering a 5 year old Xcode issue"/>
<meta name="twitter:description" content="Have you ever had issues with Xcode not symbolicating crashes? Me too, and I found the fix by reverse engineering the IDE."/>
<meta name="twitter:site" content="@rockbruno_"/>
<!-- Favicon -->
<link rel="icon" type="image/png" href="images/favicon/iconsmall2.png" sizes="32x32" />
<link rel="apple-touch-icon" href="images/favicon/iconsmall2.png">
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Source+Sans+3:ital,wght@0,200..900;1,200..900&display=swap" rel="stylesheet">
<!-- Bootstrap CSS Plugins -->
<link rel="stylesheet" type="text/css" href="css/bootstrap.css">
<!-- Prism CSS Stylesheet -->
<link rel="stylesheet" type="text/css" href="css/prism4.css">
<!-- Main CSS Stylesheet -->
<link rel="stylesheet" type="text/css" href="css/style48.css">
<link rel="stylesheet" type="text/css" href="css/sponsor4.css">
<!-- HTML5 shiv and Respond.js support IE8 or Older for HTML5 elements and media queries -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
<![endif]-->
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "BlogPosting",
"mainEntityOfPage": {
"@type": "WebPage",
"@id": "https://swiftrocks.com/reverse-engineering-xcode-issue-crash-symbol"
},
"image": [
"https://swiftrocks.com/images/thumbs/thumb.jpg"
],
"datePublished": "2021-08-31T14:00:00+02:00",
"dateModified": "2021-08-31T14:00:00+02:00",
"author": {
"@type": "Person",
"name": "Bruno Rocha"
},
"publisher": {
"@type": "Organization",
"name": "SwiftRocks",
"logo": {
"@type": "ImageObject",
"url": "https://swiftrocks.com/images/thumbs/thumb.jpg"
}
},
"headline": "Reverse engineering a 5 year old Xcode issue",
"abstract": "Have you ever had issues with Xcode not symbolicating crashes? Me too, and I found the fix by reverse engineering the IDE."
}
</script>
</head>
<body>
<div id="main">
<!-- Blog Header -->
<!-- Blog Post (Right Sidebar) Start -->
<div class="container">
<div class="col-xs-12">
<div class="page-body">
<div class="row">
<div><a href="https://swiftrocks.com">
<img id="logo" class="logo" alt="SwiftRocks" src="images/bg/logo2light.png">
</a>
<div class="menu-large">
<div class="menu-arrow-right"></div>
<div class="menu-header menu-header-large">
<div class="menu-item">
<a href="blog">blog</a>
</div>
<div class="menu-item">
<a href="about">about</a>
</div>
<div class="menu-item">
<a href="talks">talks</a>
</div>
<div class="menu-item">
<a href="projects">projects</a>
</div>
<div class="menu-item">
<a href="software-engineering-book-recommendations">book recs</a>
</div>
<div class="menu-item">
<a href="games">game recs</a>
</div>
<div class="menu-arrow-right-2"></div>
</div>
</div>
<div class="menu-small">
<div class="menu-arrow-right"></div>
<div class="menu-header menu-header-small-1">
<div class="menu-item">
<a href="blog">blog</a>
</div>
<div class="menu-item">
<a href="about">about</a>
</div>
<div class="menu-item">
<a href="talks">talks</a>
</div>
<div class="menu-item">
<a href="projects">projects</a>
</div>
<div class="menu-arrow-right-2"></div>
</div>
<div class="menu-arrow-right"></div>
<div class="menu-header menu-header-small-2">
<div class="menu-item">
<a href="software-engineering-book-recommendations">book recs</a>
</div>
<div class="menu-item">
<a href="games">game recs</a>
</div>
<div class="menu-arrow-right-2"></div>
</div>
</div>
</div>
<div class="content-page" id="WRITEIT_DYNAMIC_CONTENT">
<!--WRITEIT_POST_NAME=Reverse engineering a 5 year old Xcode issue-->
<!--WRITEIT_POST_HTML_NAME=reverse-engineering-xcode-issue-crash-symbol-->
<!--Add here the additional properties that you want each page to possess.-->
<!--These properties can be used to change content in the template page or in the page itself as shown here.-->
<!--Properties must start with 'WRITEIT_POST'.-->
<!--Writeit provides and injects WRITEIT_POST_NAME and WRITEIT_POST_HTML_NAME by default.-->
<!--WRITEIT_POST_SHORT_DESCRIPTION=Have you ever had issues with Xcode not symbolicating crashes? Me too, and I found the fix by reverse engineering the IDE.-->
<!--DateFormat example: 2021-08-31T14:00:00+02:00-->
<!--WRITEIT_POST_SITEMAP_DATE_LAST_MOD=2021-08-31T14:00:00+02:00-->
<!--WRITEIT_POST_SITEMAP_DATE=2021-08-31T14:00:00+02:00-->
<title>Reverse engineering a 5 year old Xcode issue</title>
<div class="blog-post">
<div class="post-title-index">
<h1>Reverse engineering a 5 year old Xcode issue</h1>
</div>
<div class="post-info">
<div class="post-info-text">Published on 31 Aug 2021</div>
</div>
<p>Xcode has a feature called <b>Organizer</b> that shows you important information about builds you sent to the App Store, with the most relevant ones being crashes and energy reports (for CPU/memory usage). In my experience, this feature will work perfectly fine if you pushed the build from your <i>own</i> machine, but if that's not true for any reason (the most common being because you're using a CI pipeline to do) then you might have a frustrating experience. To be specific, if you didn't push the build from your machine, then all reports provided by Xcode will be unsymbolicated:</p>
<div class="post-image">
<img src="https://i.imgur.com/9UaPOZb.png" alt="Alt">
</div>
<p>This is perfectly expected given that the symbol information comes from the <b>dSYM</b> package that is generated when you archive your app, so it's also normal for CI pipelines to store the dSYMs of a build somewhere that can be retrieved later. The problem is that <b>even</b> if you have a copy of the original build archive, chances are that Xcode will not know what to do. If you right-click a stack trace and press <i>Symbolicate</i>, you'll either get a useless error or nothing will happen.</p>
<div class="sponsor-article-ad-auto hidden"></div>
<p>This bug is not deterministic (some people have it, some people don't), and it's not clear at first glance what factors make it work or not. What I do know is that a <b>lot</b> of people have experienced this -- a Google search will reveal many StackOverflow and Apple Developer threads complaining about this issue, <a href="https://developer.apple.com/forums/thread/51873">with the earliest being 5 years old.</a> The threads point to many CLI alternatives that can be used to symbolicate crashes manually, but no one seems to be able to pinpoint exactly <i>why</i> Xcode doesn't work in the first place. As you might also expect, Apple has never said or done anything about this.</p>
<p><i>"But wait a second Bruno, why do you care about this bug? Isn't everyone using Firebase for crash reporting?"</i></p>
<p>That is true, but the Organizer contains more than just plain crashes. I was particularly interested in reading Xcode's <b>energy reports</b> to learn more about cases where iOS decides to shut down apps due to high CPU usage, which are not reported to Firebase.</p>
<p>Additionally, even though there are many workarounds available to symbolicate crashes manually, <b>none of them work for the energy reports I wanted to look at.</b> The most common workaround shared by developers is to extract the <code>symbolicatecrash</code> file that exists inside Xcode and call it manually from the CLI, but this only works for regular crashes. To be specific, <code>symbolicatecrash</code> is in fact what Xcode uses to symbolicate energy reports, but because Xcode stores them in a different format, you're not able to pass them to <code>symbolicatecrash</code> without first somehow converting them, something of which I had no interest in doing as I had no idea what format this tool is looking for in the first place. To make it worse, because issues with CPU and memory usage tend to be rare in the iOS world, I couldn't find <i>any</i> workaround for this. I made an angry Twitter post, and sometime after that realized that if I want to see these logs then I should reverse engineer Xcode and fix this issue myself.</p>
<h2>Finding the source of the error</h2>
<p>The only piece of relevant information I had about this issue is the error that Xcode returns when attempting to symbolicate an energy report. Although the error is useless, it's customized enough to allow me to search for it inside the binary:</p>
<div class="post-image">
<img src="https://i.imgur.com/NJJw17V.png" alt="Alt">
</div>
<p>I then <code>grep</code>'d the entire Xcode archive for this string. It took a very long time for it to finish given that my Xcode is about 30 gigabytes in size, but it did find the reference inside the <code>IDEAnalyticsKit</code> framework:</p>
<pre><code>Binary file .//PlugIns/IDEAnalyticsKit.framework/Versions/A/IDEAnalyticsKit matches
Binary file .//PlugIns/IDEAnalyticsKit.framework/Versions/Current/IDEAnalyticsKit matches
Binary file .//PlugIns/IDEAnalyticsKit.framework/IDEAnalyticsKit matches</code></pre>
<p>Xcode is not one giant binary, but a collection of many smaller specialized frameworks. In fact, the main Xcode binary seems to do almost nothing but launch the other frameworks.</p>
<p>After opening the <code>IDEAnalyticsKit</code> binary in Hopper, I was easily able to locate the Obj-C method that hardcoded the error string:</p>
<div class="post-image">
<img src="https://i.imgur.com/iBbWFVR.png" alt="Alt">
</div>
<p>Thanks to Hopper's amazing feature of converting assembly to pseudocode, navigating the decompiled binary is a breeze. By selecting the first instruction of a method, I could navigate the stack trace all the way to the method that is called when the <i>Symbolicate</i> button is touched:</p>
<div class="post-image">
<img src="https://i.imgur.com/dKfpaEs.png" alt="Alt">
</div>
<p>In short, what this method does is simply assert that we're in the main thread and then start the symbolization process by sending a completion handler that throws the error we're ending up in. By navigating the assembly code with the help of Hopper's pseudocode feature, we should be able to find out exactly what's causing it to fail.</p>
<p>Following the symbolization trace eventually led me to <i>another</i> framework, <code>DVTAnalytics</code>, where a more specialized method lives:</p>
<div class="post-image">
<img src="https://i.imgur.com/NqY13Jc.png" alt="Alt">
</div>
<p>Despite being a long method, there's nothing special going on here -- Xcode is simply grabbing information about the crash and initializing more specialized objects as it travels from framework to framework. After setting up another completion handler, the journey continues inside <code>symbolicateWithCallback</code>.</p>
<p>When I tried navigating to that method, Hopper greeted me with this awesome pop-up:</p>
<div class="post-image">
<img src="https://i.imgur.com/YvwFOgN.png" alt="Alt">
</div>
<p>It turns out that Hopper is smart enough to notice that <code>symbolicateWithCallback</code> is a Obj-C protocol method that is implemented by three different objects in this framework, which represent the three different types of crashes available in the Organizer (crash, energy report and disk report)! After picking the energy variant, I eventually fell here:</p>
<div class="post-image">
<img src="https://i.imgur.com/w4z5zFq.png" alt="Alt">
</div>
<p>This method is saying that we're initializing a <code>DVTLocalLogSymbolicator</code> object and asking it to continue the process, but I couldn't find it anywhere in this framework. After another <code>grep</code>, I found out that this and all the core symbolization logic was placed in a separate <code>DVTFoundation</code> framework. After locating the class and its <code>symbolicateLogData</code> method, I was finally able to locate where the symbolization is done:</p>
<div class="post-image">
<img src="https://i.imgur.com/Fuw2J77.png" alt="Alt">
</div>
<p>This is essentially saying that Xcode is fetching and calling the <code>symbolicatecrash</code> utility, which is not a surprise given that this is exactly the workaround that we mentioned in the beginning. However, I could immediately tell what the problem was -- when this workaround is suggested, the correct way of invoking this utility is by sending the dSYM that contains the symbols you're trying to translate:</p>
<pre><code>symbolicatecrash --dsym ./symbols.dSYM --output ./symbolicated.txt</code></pre>
<p>However, Xcode doesn't do that -- it just sets the output and hopes for the best. What happens essentially is that this tool is capable of searching for dSYMs on its own, but for some reason sometimes it just fails to do so. I could easily reproduce the issue by trying to symbolicate a regular crash without passing the dsym argument:</p>
<pre><code>Did not find dsym for (uuid)</code></pre>
<p>My first assumption was that this tool was probably looking for specific names or in specific folders, and we could confirm that by opening <code>symbolicatecrash</code> in Hopper and reverse engineering that logic. But quickly I realized that there was something fishy going on, because what I found instead was... nothing?</p>
<div class="post-image">
<img src="https://i.imgur.com/RAdUFL8.png" alt="Alt">
</div>
<p>After being stumped for a while, my colleague Åke quickly noticed that although macOS says that this file is a binary, it's actually just a really big perl script!</p>
<pre><code>#!/usr/bin/perl -w
#
# This script parses a crashdump file and attempts to resolve addresses into function names.
#
# It finds symbol-rich binaries by:
# a) searching in Spotlight to find .dSYM files by UUID, then finding the executable from there.
# That finds the symbols for binaries that a developer has built with "DWARF with dSYM File".
# b) searching in various SDK directories.
#
# Copyright (c) 2008-2015 Apple Inc. All Rights Reserved.
#
#</code></pre>
<p>This made everything else easier, because we could now freely modify this script and test our changes without having to generate new binaries. The first thing we did was locate the part of the code that parsed the arguments, which was right in the beginning:</p>
<pre><code># read and parse command line
my $opt_help = 0;
my $opt_verbose = 0;
my $opt_output = "-";
my @opt_dsyms = ();
my $opt_spotlight = 1;</code></pre>
<p>If we know that <code>symbolicatecrash</code> works by sending the dSYM manually, could we simply override the <code>opt_dsyms</code> param and fix the issue? The answer is <b>yes!</b></p>
<div class="post-image">
<img src="https://i.imgur.com/uSdkNsU.png" alt="Alt">
</div>
<p>This was sufficient for the data gathering we wanted to do, but I was still interested in knowing why this script couldn't find the dSYMs on its own. After a deeper inspection, we can see that the script attempts to locate the dSYM in three ways:</p>
<ul>
<li>The --dsym flag</li>
<li>The /Volumes/Build/UUIDToSymbolMap folder</li>
<li>The <code>Symbols</code> folder inside of every iOS/macOS SDK you have installed</li>
</ul>
<p>If neither of those returned a valid result, <b>it attempts to search the symbols through Spotlight, your macOS's search engine.</b> More specifically, it runs the following command:</p>
<pre><code>mdfind \"com_apple_xcode_dsym_uuids == $canonical_uuid\"</code></pre>
<p>This means that the failure to symbolicate crashes isn't a problem with Xcode, but that for some reason Spotlight failed to add the dSYM to its search index. After some quick searches about Spotlight issues, I found three ways to force Spotlight to re-index a folder (thus allowing the symbolication to work properly):</p>
<ul>
<li>Running the <code>mdimport</code> CLI tool, pointing directly to where the dSYM is stored.</li>
<li>This <a href="https://support.apple.com/en-us/HT201716">support article</a> from Apple</li>
<li>Simply moving the dSYMs somewhere else (Some folders might be blocked by Spotlight. I was never able to get it to work if the dSYms were stored in the <i>Downloads</i> folder, for example)</li>
</ul>
<div class="sponsor-article-ad-auto hidden"></div>
<p>I had mixed success with <code>mdimport</code> (in fact, I remember seeing this as a buried comment in one of the StackOverflow posts, so someone <i>did</i> also figure this out in the past), but the other two tricks usually solves it for me. I think this feature would've been better developed if Xcode allowed you to provide the dSYM manually instead of completely relying on Spotlight, but investigating this issue was a very fun couple of hours for me.</p>
</div>
</div>
<div class="blog-post footer-main">
<div class="footer-logos">
<a href="https://swiftrocks.com/rss.xml"><i class="fa fa-rss"></i></a>
<a href="https://twitter.com/rockbruno_"><i class="fa fa-twitter"></i></a>
<a href="https://github.com/rockbruno"><i class="fa fa-github"></i></a>
</div>
<div class="footer-text">
© 2025 Bruno Rocha
</div>
<div class="footer-text">
<p><a href="https://swiftrocks.com">Home</a> / <a href="blog">See all posts</a></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<!-- Blog Post (Right Sidebar) End -->
</div>
</div>
</div>
<!-- All Javascript Plugins -->
<script type="text/javascript" src="js/jquery.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/js/bootstrap.bundle.min.js" integrity="sha384-MrcW6ZMFYlzcLA8Nl+NtUVF0sA7MsXsP1UyJoMp4YLEuNSfAP+JcXn/tWtIaxVXM" crossorigin="anonymous"></script>
<script type="text/javascript" src="js/prism4.js"></script>
<!-- Main Javascript File -->
<script type="text/javascript" src="js/scripts30.js"></script>
<!-- Google tag (gtag.js) -->
<script async src="https://www.googletagmanager.com/gtag/js?id=G-H8KZTWSQ1R"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'G-H8KZTWSQ1R');
</script>
</body>
</html>