atos and dwarfdump won't symbolicate my address

问题

I received a crash report via AirBrake.io that isn't symbolicated. Since the crash report is not in exactly the same format as an Apple crashlog I can't just drop it on XCode as usual, so I took the exact same build from my XCode archive tried to symbolicate it on the commandline. With the following result:

$ atos -o kidsapp.app/kidsapp 0x0002fc4c
0x0002fc4c (in kidsapp)

I'm absolutely sure I'm using the same build as the crash report is from. So I also tried with dwarfdump:

$ dwarfdump --lookup 0x0002fc4c --arch armv7 kidsapp.app.dSYM
----------------------------------------------------------------------
 File: kidsapp.app.dSYM/Contents/Resources/DWARF/kidsapp (armv7)
----------------------------------------------------------------------
Looking up address: 0x000000000002fc4c in .debug_info... not found.
Looking up address: 0x000000000002fc4c in .debug_frame... not found.

Also no result. Is there anything else besides using the wrong dSYM file that I could do wrong? I know it's the correct one since this is the version referred in the crash report in AirBrake and it's in my XCode archive.

Any ideas/tips are welcome!

回答1:

First of all check if the dSYM is really the correct one for that app:

dwarfdump --uuid kidsapp.app/kidsapp
dwarfdump --uuid kidsapp.app.dSYM

Both should return the same result.

Next check if the dSYM has any valid content

dwarfdump --all kidsapp.app.dSYM

This should give at least some info, other than not found.

I guess that the dSYM is corrupt. In general you might want to use a crash reporter that gives you a full crash report with all threads and last exception backtrace information. I recommend using something based on PLCrashReporter, e.g. QuincyKit (Open Source SDK + Server + symbolication on your mac) or HockeyApp (Open Source SDK + Paid service + server side symbolication) (Note: I am one of the developers both!)

回答2:

I used the following arithmetic to figure it out:

slide + stack address - load address = symbol address

and

stack address is the hex value I get from my stack dump crash report (not a .crash file, just the stack dump).

and

slide is the vmaddr of the LC_SEGMENT cmd when running otool -arch armv7 -l APP_BINARY_PATH. Mine usually ends up being 0x00001000.

and

load address is the complicated piece. It is actually the difference between the bottommost stack address of the main thread and the FIRST address of the portion of my binary that contains symbols when running dwarfdump --arch armv7 --all DSYM_BINARY_PATH. This is simply the symbolic address of the main function. So if your bottom most crash address is 0x8000 and your main function's symbolic address is 0x2000 then your load address is 0x6000.

Now with ALL these pieces I can calculate the symbol address and put that into atos or dwarfdump: dwarfdump --lookup SYM_ADDR --arch armv7 APP_BINARY_PATH.

Example of the dump (you can see that the load address was 0x00003af4):

----------------------------------------------------------------------

File: /Users/user/Desktop/MyApp.xcarchive/dSYMs/MyApp.app.dSYM/Contents/Resources/DWARF/MyApp (armv7)

----------------------------------------------------------------------

0x00000024: [0x00003af4 - 0x00003b4e) main

0x00000098: [0x00003b50 - 0x00003d8c) -[MyAppDelegate application: didFinishLaunchingWithOptions:]

... the rest of the dump

The hardest part was realizing that one of the 2 static libraries I'd included had their symbols stripped before being link to my app's binary! That left a HUGE gap of symbol addresses so I only ended up with two-thirds of the symbols I needed in my dSYM.

Be sure to have the following flags set to NO in your static libraries xcode project so that when you link against it, you can pull in the symbols to your app's binary (which can later be stripped): COPY_PHASE_STRIP, DEAD_CODE_STRIPPING, and STRIP_INSTALLED_PRODUCT.

Now you may ask, "what do I do if the stack dump does not include the main function since it isn't on the main thread so that I cannot get the main function's stack address?". To that I would reply, "I haven't a friggin' clue!". Just cross your fingers and hope you can get a stack trace that includes the symbol address or use a crash reporting system that mimics Apple's crash logs, like PLCrashReporter.

[EDIT May 26, 2013] -

It was brought to my attention that the load address is really the address of the mach-o binary. Though what I described above can often work - it's not actually correct. This can be obtained via the CRASH REPORT, however the point of this answer was to provide the symbols of a crash when you don't have a crash report. The best way I've come to figuring out the load address when wanting to symbolicate is by making sure I log the load address with the stack addresses.

I've personally created a system for logging crashes (not crash reports) and having them sent to an S3 bucket where I can retrieve them later for debugging. When I start my application I cache the slide, the load address and the main function address for use if my app crashes and I send up a the stack addresses.

NOTE: the dyld functions use #include <mach-o/dyld.h>

slide = the address returned by _dyld_get_image_vmaddr_slide(0)

load address = the address returned by _dyld_get_image_header(0)

main function address = the last address in [NSThread callStackReturnAddresses] when called on the main thread

At crash time I'm sure to log [NSThread callStackReturnAddresses] and [NSThread callStackSymbols] as well as the architecture which can be retrieve by having this method:

- (NSString*) arch
{
    NSString* arch =
#ifdef _ARM_ARCH_7
        @"armv7";
#elif defined (_ARM_ARCH_6)
        @"armv6";
#else
        nil;
#endif

    return arch;
}

I don't yet know how to differentiate between armv7 and armv7s though.

So this may help in the future. I plan on taking everything I've learned and turning this into a simple crash tool - better than the natos tool (probably natos v2).

I've updated natos to support supplying the load address manually: https://github.com/NSProgrammer/natos

回答3:

For whom that certain times doesn't have the value for Load Address like this:

Jan 14 11:02:39 Dennins-iPhone AppName[584] <Critical>: Stack Trace: (
    0   CoreFoundation                      0x2c3084b7 <redacted> + 150
    1   libobjc.A.dylib                     0x39abec8b objc_exception_throw + 38
    2   CoreFoundation                      0x2c21cc35 CFRunLoopRemoveTimer + 0
    3   AppName                             0x0005a7db AppName + 272347

I've created a simple bash to help me debug:

#! /bin/bash
read -p "[Path] [App Name] [Stack Address] [Relative Address] " path appName runtimeAddress relativeAddress
loadAddress=`echo "obase=16;ibase=10;$((runtimeAddress-relativeAddress))" | bc`
atos -o $path/Payload/$appName.app/$appName -l $loadAddress $runtimeAddress -arch armv7

It just reads the path for the app, the app name, the runtime address, and the value after "+" signal (the decimal value) and then find the value for load address to run atos command.

回答4:

So my case: I'm receiving the crash strings out of the NSException.callStackSymbols.

The crash stack trace looks this way:

2   AppName                               0x00000001006c75b4 AppName + 2356660\r3   AppName                               0x00000001004f5cfc AppName + 449788\r4   UIKit                               0x000000018c0a8968 \u003credacted\u003e + 108\r5   UIKit      0x000000018c0a9328 \u003credacted\u003e + 28\r6   UIKit                               0x000000018beea250 \u003credacted\u003e + 1320\r7   UIKit                               0x000000018beede98 \u003credacted\u003e + 188\r8   UIKit                               0x000000018bcb5820 \u003credacted\u003e + 116\r9   UIKit                               0x000000018bbdec88 \u003credacted\u003e + 760\r10  UIKit                               0x000000018bbde610 \u003credacted\u003e + 312\r11  UIKit                               0x000000018bbde31c \u003credacted\u003e + 296\r12  UIKit                               0x000000018bbde3bc \u003credacted\u003e + 456\r13  QuartzCore                          0x0000000185b93b7c \u003credacted\u003e + 284\r14  libdispatch.dylib                   0x00000001811a8a2c \u003credacted\u003e + 16\r15  libdispatch.dylib                   0x00000001811b5654 \u003credacted\u003e + 1012\r16  CoreFoundation                      0x0000000181851650 \u003credacted\u003e + 12\r17  CoreFoundation                      0x000000018184f1a8 \u003credacted\u003e + 2272\r18  CoreFoundation                      0x000000018176f488 CFRunLoopRunSpecific + 552\r19  GraphicsServices                    0x0000000183735020 GSEventRunModal + 100\r20  UIKit         0x000000018bc09cc0 UIApplicationMain + 236\r21  AppName                               0x000000010048f714 AppName + 30484\r22  libdyld.dylib 0x000000018120dfc0 \u003credacted\u003e + 4

Don't include bitcode, as Apple can recompile your code, and dsym files from your archive in Organizer won't match.

Run such bash script:

#!/bin/bash
appName=AppName
runtimeAddress=0x00000001006c75b4
relativeAddress=2356660
loadAddress=`echo "obase=16;ibase=10;$((runtimeAddress-relativeAddress))" | bc`
atos -o $appName.app/$appName -l $loadAddress $runtimeAddress -arch arm64

Assuming, that you've created a test.sh (touch test.sh). If it's not possible to run the sctipt (./test.sh) because of the permissions issue: call chmod +x test.sh Now ./test.sh should work. And generate the result.

As well it's possible to symbolicate the crash file using AppName.app.dSYM file:

#!/bin/bash
appName=AppName
runtimeAddress=0x00000001006c75b4
relativeAddress=2356660
loadAddress=`echo "obase=16;ibase=10;$((runtimeAddress-relativeAddress))" | bc`
atos -o $appName.app.dSYM/Contents/Resources/DWARF/$appName -l $loadAddress $runtimeAddress -arch arm64