But it turns out that life debugging wasn’t even necessary: you can use the crash information also offline to figure out what function was responsible for the crash!
Preparing the binary
These days I am using the platform.io environment with the Atom editor to do my ESP8266 development. In order to have enough information in the binary you have to add some flags to the platformio.ini. In the section for your target platform add:
build_flags = -Og -ggdb -DDEBUG_ESP_PORT=Serial
which will cause the compiler to create a binary containing the symbol information required for debugging. Save it and make sure that the whole workspace gets re-compiled.. Upload it, start the serial console and provoke the crash:
Fatal exception 0(IllegalInstructionCause): epc1=0x4020188c, epc2=0x00000000
The interesting part here is the epc1 address, which is the program counter.
Finding the culprit
The next step was to figure out where the debugger was installed for the platformio environment.
On a Mac you can (currently) find it here:
~/.platformio/packages/toolchain-xtensa/bin/xtensa-lx106-elf-gdb
Now you can start it by telling it where your binary is:
~/.platformio/packages/toolchain-xtensa/bin/xtensa-lx106-elf-gdb ./.pioenvs/nodemcuv2/firmware.elf
(assuming that you are in the root of your project folder). The debugger should great you with:
Reading symbols from .pioenvs/nodemcuv2/firmware.elf...done. (gdb)
Now you can enter the epc1 pointer from above and get a pretty clear picture where the problem occured:
(gdb) info line *0x4020188c Line 116 of ".pioenvs/nodemcuv2/FrameworkArduino/Tone.cpp" starts at address 0x4020188c and ends at 0x40201891
Tone.cpp was the culprit! I quickly headed over to the Arduino/ESP8266 github repo and searched for problems with Tone and found the following issue: https://github.com/esp8266/Arduino/issues/1761
Summary
Even (or even more) on embedded devices debugging is a very powerful tool. But it might take some time to set it up properly. While I still did not succeed in doing operation on the open heart (life debugging) it wasn’t even necessary in this case. Having a program counter pointing me to the location of the crash helped already a lot. After debugging, don’t forget to remove the flags in platformio or you will miss compiler optimizations…
Now a beer! Did you like this post? It often takes me several hours of my free time to write one. If I was your neighbour would you offer me a beer for the hard work I did for you? The beauty is: beers can be teletransported with a painless donation using Paypal. A beer in a Swiss bar costs about USD $4.80. Or use this affiliate link if you order something on Banggood and I'll get a small kickback. Thank you!
For live debugging, you need to include GDBStub library and open gdb session once crash happens. Gdb should then present you with a prompt where you can type backtrace command, for example.
Hi Ivan. Thanks for your post and your great work for the ESP8266 community! I tried exactly that: Included GDBStub, added #define DEBUG_ESP_PORT Serial (is this necessary), forwarded the serial port to an IP port, listened with nc on that port and when the crash happened tried to connect with gdb to the ESP. But gdb stops with "Remote debugging using :9980" without presenting the (gdb) prompt. With this crash the Watchdog always restarted the ESP, so I tried to add the ((int*)0) = 0; in the suggested method. But the ESP kept restarting. Couldn't I connect because of the restarting? Or do I have to turn off Serial logging to free the Serial channel for gdb connection?
Knowing EPC isn’t enough in general case – it may point into some standard library (f.i., NULL pointer is passed incorrectly in some function).
For the proper investigation one needs to know the stack, which is usually dumped to the serial during the crash.
There is a tool named Exception Stack Trace Decoder (https://github.com/me-no-dev/EspExceptionDecoder) for Arduino IDE. It helps to decode the stack trace, showing functions (in a human-readable format) for each address in it.