[Resolved] Fixing the crash on Particle 0.7.0 firmware

jvanier 2018-04-17 01:56:58 UTC #1

The crash is fixed in the Carloop library version 2.1.0

To upgrade your application running in the Particle Web IDE open the Carloop library version 2.1.0, click Include in project and select your application.

To upgrade your application built with the Particle CLI, run particle library add carloop

To get your Photon or Electron back online if it's in a loop of booting, connecting to the cloud then blinking red and crashing, hold the SETUP button, tap RESET and let go of SETUP. The device will connect to the Particle Cloud and start breathing magenta instead of cyan. This is called Safe mode. You will then be able to flash your application that includes the fixed version of the library.

Original thread

With the release of Particle firmware 0.7.0 for the Photon and Electron the Carloop library started crashing in the begin() method.

I'll document how I'm investigating the issue in this thread.

jvanier 2018-04-17 02:12:56 UTC #2

First, I'll download the latest Particle firmware and Carloop library.

git clone https://github.com/carloop/carloop-library.git
git clone https://github.com/particle-iot/firmware.git
cd firmware
git checkout -t origin/release/v0.7.0

To debug effectively I'll create a local build of the Particle firmware and the Carloop library I can run using the Programmer Shield.

Before building the firmware, I have to install the local build tools. I downloaded gcc 5.3 for ARM to compile the 0.7.0 firmware.

jvanier 2018-04-17 02:31:09 UTC #3

I copied the Carloop library files and the minimal example into a new folder of the Particle firmware at user/applications/carloop. There are other ways to build the firmware with a user application, but this one is straightforward.

ls user/applications/carloop/
carloop.cpp  carloop.h  carloop_minimal.cpp  TinyGPS++.cpp  TinyGPS++.h

Then I started following the steps for using a debugger with the Particle Photon

To build a debugger friendly firmware, I used this command:

cd main
make clean all PLATFORM=photon USE_SWD_JTAG=y MODULAR=n APP=carloop

After a while the build succeeds and I get these lines:

arm-none-eabi-size --format=berkeley ../build/target/main/platform-6/carloop.elf
   text	   data	    bss	    dec	    hex	filename
 485336	   2228	  41744	 529308	  8139c	../build/target/main/platform-6/carloop.elf
arm-none-eabi-objdump -h -S ../build/target/main/platform-6/carloop.elf > ../build/target/main/platform-6/carloop.lst

jvanier 2018-04-17 03:02:28 UTC #4

I flashed the debug build by putting the Photon in DFU mode (hold SETUP, tap RESET and hold SETUP until the LED blinks yellow then release SETUP) then running

cd main
make program-dfu PLATFORM=photon USE_SWD_JTAG=y MODULAR=n APP=carloop

After flashing completed I got the blinking red crash as expected. Time to fire up the debugger!

Note to myself for when I'm done: remember to put the Photon back to a non-debug firmware otherwise I will be very confused why I can't flash from the Web IDE or CLI next time I use my Carloop.

cd modules
make clean all program-dfu PLATFORM=photon

The I used the OpenOCD and gdb tools as described in the debugger guide to connect to the Photon.

openocd -f interface/ftdi/particle-ftdi.cfg -f target/stm32f2x.cfg -c "gdb_port 3333"
arm-none-eabi-gdb -ex "target remote localhost:3333" build/target/main/platform-6/carloop.elf

I was now able to restart the microcontroller stopping at the first instruction, set breakpoints in the code and run the code.

monitor reset halt
break setup
continue

jvanier 2018-04-17 03:30:44 UTC #5

Putting a breakpoint on setup then tracing the calls leads to this part of the Carloop setup:

github.com

carloop/carloop-library/blob/f31a605369f68a3323a71fa2d5fcf804a6fe292a/src/carloop.cpp#L97


template <typename Config>
void Carloop<Config>::enableGPS()
{
pinMode(Config::GPS_ENABLE_PIN, OUTPUT);
digitalWrite(Config::GPS_ENABLE_PIN, Config::GPS_ENABLE_ACTIVE);


Serial1.begin(Config::GPS_BAUD_RATE);


if (!gpsSerialThread.is_valid()) {
    // Start a thread that will run this->receiveSerialChars()
    gpsSerialThread = Thread("gps_serial", [this]() { receiveSerialChars(); });
}
}


template <typename Config>
void Carloop<Config>::disableGPS()
{
digitalWrite(Config::GPS_ENABLE_PIN, Config::GPS_ENABLE_INACTIVE);
}


template <typename Config>

If I put a breakpoint on the panic_ function which is called when the Photon crashes I see that the function executing when the crash happens is

github.com

particle-iot/firmware/blob/647fd310718831ebb631d2cb6c7a9a59054955ba/wiring/inc/spark_wiring_thread.h#L159


        this->func_param_ = rhs.func_param_;
        this->exited_ = rhs.exited_;
        rhs.handle = OS_THREAD_INVALID_HANDLE;
        rhs.wrapper = NULL;
    }
    return *this;
}


private:


static os_thread_return_t run(void* param) {
    Thread* th = (Thread*)param;
    if (th->func_) {
        (*(th->func_))(th->func_param_);
    } else if (th->wrapper) {
        (*(th->wrapper))();
    }
    th->exited_ = true;
    os_thread_exit(nullptr);
}
};

So the culprit is pretty clearly something wrong in the thread handling.

Since the crash is in the GPS handler, a workaround to use Carloop until the issue is fixed is to disable the GPS (i.e. enable the CAN bus only).

carloop.begin(CARLOOP_CAN);

alanm 2018-04-17 03:50:07 UTC #6

this is an amazing write up, @jvanier!

jvanier 2018-04-17 04:31:41 UTC #7

I was able to reproduce the issue with a minimal Particle program. The following code works with 0.6.3 but crashes with 0.7.0 on a Photon.

#include "application.h"

Thread t;

int count = 0;

void run() {
  while(true) {
    count++;
    delay(1);
  }
}

void setup() {
  Serial.begin();
  t = Thread("count", run);
}

void loop() {
  Serial.printlnf("count=%d", count);
  delay(500);
}

I reported this regression to the Particle firmware team.

In the mean time I was able to figure out a workaround. I'll release a new version of the Carloop library with the workaround.

jvanier 2018-04-17 04:46:14 UTC #8

Alright, the fix is out with the Carloop library version 2.1.0.

Instructions for upgrading are in the first post of this thread.

cyclin_al1 2018-04-19 04:27:48 UTC #9

@alanm & @jvanier,

Thanks for looking into this and supporting the community!!!

stonej 2018-04-23 12:34:41 UTC #10

Thanks so much for the fix and continued support!

itsmedavid 2018-04-25 03:51:25 UTC #11

Thanks @jvanier for the solution !