Unit Testing Firmware

JoergS5

@benjamin-forest I'm sorry, I had other project with higher priority, so I had to postpone. But I am still planning unit testing.

My current plan is to separate code which is about the pure core like the movement planning and timers and isolate it from the rest like heatbed, fans etc. and unit test only the core. I want to compare this code to Klipper and Marlin code to better understand what's going on.

One additional possibility to watch the code indirectly is to monitor the movement commands with an oszilloscope or counter and analyse what is transferred to the motor controllers.

I final and very expensive method to analyze what's going on is using an ETM debugger. But Duet doesn't expose the necesssary pins for JTAG and has no hardware for ETM debugging. So the firmware must be migrated to a board capable of doing it. But ETM debuggers are costly in the range of 2000$ (e.g. Segger J-Trace), so I am thinking about how to achieve it cheaper. This would allow detailed analysis of what's happening in the FreeRTOS code exactly. ETM allows to capture and analyze every processed command and memory content of a Cortex MCU.

dc42

Duet 3 provides dedicated SWD pins for debugging. Older Duet WiFi/Ethernet boards provide JTAG pins, which could be used for debugging if the firmware was modified not to use those pins for other functions. But I've never needed to use a debugger on Duet 2.

JoergS5

@dc42 Duet 3 this is good news!
For Duet2/3, JTAG is not sufficient for ETM. For ETM, in most cases there is an addition STM32F1 chip on the board to pack the information, as the needed bandwidth is very high. But with JTAG it's possible to capture a short time similar to ETM capturing and storing it into a limited Cortex memory space (a few seconds) = called ETB. This may be sufficient for most analyzes. ETM can capture all.

benjamin.forest

@JoergS5 great, let me know if you have some results, i'm quite excited by Klipper as well.

benjamin.forest

@dc42 Thanks. So how do you perform your tests ? only manually ? you don't have too much regressions ?

dc42

@benjamin-forest said in Unit Testing Firmware:

@dc42 Thanks. So how do you perform your tests ? only manually ? you don't have too much regressions ?

I put together a test schedule based on what I have changed. I test new and modified functionality. I also review the code changes carefully before committing. Finally, I do tests prints on Cartesian and Delta machines. I used to include SCARA too.

Occasionally, for critical stuff that is easy to get wrong and difficult to test thoroughly, I run a formal verification tool to prove that the code conforms to the specification. That's why header file ecv.h is #included and many functions have declared preconditions. A few (e.g. function TemperatureSensor::GetPT100Temperature in src/Sensors/Temperaturesensor.cpp) have additional annotations to facilitate formal verification. The fast integer square root algorithms in RRFLibraries/src/Isqrt.cpp were also verified using the same tool, but I did that offline so the formal annotations are not in the code.

I'd like to run formal verification on the entire source code, but RRF uses too many C++ features that the tool doesn't yet support.

In practice, most of the bugs that crop up are specification errors, not coding errors. Sometimes this is because there is no clear unambiguous specification for GCode commands. Other times it's because I hadn't forseen some particular combination of conditions.

infiniteloop

@dc42

So how do you perform your tests ? only manually ?

In your reply, didn’t you forget to mention your Agile Error Production Device whose distributed RT-Java AI, equivalent to the wisdom of 10k zen buddhists, intercepts any bug before you can even think of it? – TBH I still like to invent my errors manually

oliof

Releasing beta versions and RCs and collecting community feedback is also part of the testing regimen by the way (-:

benjamin.forest

great, thank you. I'll keep that in mind.

benjamin.forest

Hi,

Has anyone made any progress on isolating parts of RepRapFirmware ? I'm struggling to create a new kinematics, and the debugPrintf debugging method is not my favorite ...

On the other side, my quick tentative to extract a few cpp files from Movement folder lead me to Core.h and other deeply target-related files, so it seems quite intricated. Am I missing something ?

All the best,

JoergS5

@benjamin-forest unfortunately you're right, and it is even worse, because of CoreNG, which has references to the ASF libraries. Maybe you can get some ideas if you look into the LPC port of RRF what can be isolated and how.

dc42

There are other ways of debugging:

Later Duets (Duet 3 and Duet 3 Mini) have a SWD port, which a debugger can be attached to. That said, I rarely use it unless I am debugging a startup problem.
When debugging kinematics, enabling Move and DDA debug using M111 will produce quite a lot of useful debug output.

CoreN2G already replaces CoreNG for the Duet 3 Mini and the Duet 3 expansion boards, and will do so eventually for Duet WiFi/Ethernet and Maestro too.

benjamin.forest

@JoergS5 Thanks a lot. I've seen it was a popular fork, I'll look into it.
@dc42 Thanks as well. I'll try to get something out of Movement / DDA debug.