Unit Testing Firmware
-
I am big fan of unit testing, I have some experience with it with Java. My proposal is to start unit testing the Duet firmware also. But for C++ there are different testing frameworks. I found so far: Catch 2, Google Test and RapidCheck. Is it possible to agree on one of those frameworks?
I want to start testing the Scara kinematics, but want to test all the drive and movement planning methods to understand them (e.g. validating movement - speed - time dependencies). -
Unit testing firmware is more challenging than unit testing desktop software.
Let's start with the easier part: unit testing modules that don't access the hardware. To run unit tests for those modules on a PC, you either need to know that the implementation-defined parts of the C and C++ languages that are used by the firmware are implemented in exactly the same way by the compilers for your PC and for your embedded platform, or (preferably) have an emulator running in the PC that reads the embedded firmware binary and simulates executing it on the appropriate ARM processor - see for example https://www.thefreecountry.com/emulators/arm.shtml (disclaimer: I haven't used any of those).
To run unit tests on those firmware modules that access hardware, you also need to emulate the hardware peripherals concerned, and the stepper motors, thermistors etc. connected to the hardware.
RepRapFirmware provides a small number of features akin to unit tests, for example direct motor movement (G1 S2 commands) and various subfunctions of M122.
-
the implementation-defined parts of the C and C++ languages that are used by the firmware are implemented in exactly the same way by the compilers for your PC
Typically there are large chunks of logic that are not compiler or architecture specific that can easily unit tested. An example from the PanelDue src/Hardware/Serialio.cpp, the function CheckInput is a finite state machine that parses a json stream and calls event handling functions. Same the ConvertUnicode method before it, both can easily being decoupled from the hardware.
https://github.com/dc42/PanelDue/blob/master/src/Hardware/SerialIo.cpp#L287
Not criticizing the code, just commenting on the concept of design for testability, though I am sure you can teach me a thing or two about software reliability
https://www.eschertech.com/company/index.php -
Thank you for the valid points to be considered. I will start at some high level methods, which are not near the hardware. The hardware must be encapsulated into mock objects then.
If the unit tests are successful and useful, you may decide whether it's worth using at a broader level.
Maybe someone has a hint what I should use as framework. From reading the descriptions, I will try RapidCheck. It creates the test cases automatically.
-
Hi Joerg,
Had you made any progress in your project ? I'm interested as well in modifying software without breaking anything, and unit testing seems a good start, even if of course we are limited to quite high level testing, since mocking hardware or using an emulator seem too costly to me.
If not, what can be done to test properly modifications made to the firmware, if possible with a script so non regression tests can be automated ? I assume here that the test are run against a real Duet board, maybe using the simulation mode ?
All the best,
-
@benjamin-forest I'm sorry, I had other project with higher priority, so I had to postpone. But I am still planning unit testing.
My current plan is to separate code which is about the pure core like the movement planning and timers and isolate it from the rest like heatbed, fans etc. and unit test only the core. I want to compare this code to Klipper and Marlin code to better understand what's going on.
One additional possibility to watch the code indirectly is to monitor the movement commands with an oszilloscope or counter and analyse what is transferred to the motor controllers.
I final and very expensive method to analyze what's going on is using an ETM debugger. But Duet doesn't expose the necesssary pins for JTAG and has no hardware for ETM debugging. So the firmware must be migrated to a board capable of doing it. But ETM debuggers are costly in the range of 2000$ (e.g. Segger J-Trace), so I am thinking about how to achieve it cheaper. This would allow detailed analysis of what's happening in the FreeRTOS code exactly. ETM allows to capture and analyze every processed command and memory content of a Cortex MCU.
-
Duet 3 provides dedicated SWD pins for debugging. Older Duet WiFi/Ethernet boards provide JTAG pins, which could be used for debugging if the firmware was modified not to use those pins for other functions. But I've never needed to use a debugger on Duet 2.
-
@dc42 Duet 3 this is good news!
For Duet2/3, JTAG is not sufficient for ETM. For ETM, in most cases there is an addition STM32F1 chip on the board to pack the information, as the needed bandwidth is very high. But with JTAG it's possible to capture a short time similar to ETM capturing and storing it into a limited Cortex memory space (a few seconds) = called ETB. This may be sufficient for most analyzes. ETM can capture all. -
@JoergS5 great, let me know if you have some results, i'm quite excited by Klipper as well.
-
@dc42 Thanks. So how do you perform your tests ? only manually ? you don't have too much regressions ?
-
@benjamin-forest said in Unit Testing Firmware:
@dc42 Thanks. So how do you perform your tests ? only manually ? you don't have too much regressions ?
I put together a test schedule based on what I have changed. I test new and modified functionality. I also review the code changes carefully before committing. Finally, I do tests prints on Cartesian and Delta machines. I used to include SCARA too.
Occasionally, for critical stuff that is easy to get wrong and difficult to test thoroughly, I run a formal verification tool to prove that the code conforms to the specification. That's why header file ecv.h is #included and many functions have declared preconditions. A few (e.g. function TemperatureSensor::GetPT100Temperature in src/Sensors/Temperaturesensor.cpp) have additional annotations to facilitate formal verification. The fast integer square root algorithms in RRFLibraries/src/Isqrt.cpp were also verified using the same tool, but I did that offline so the formal annotations are not in the code.
I'd like to run formal verification on the entire source code, but RRF uses too many C++ features that the tool doesn't yet support.
In practice, most of the bugs that crop up are specification errors, not coding errors. Sometimes this is because there is no clear unambiguous specification for GCode commands. Other times it's because I hadn't forseen some particular combination of conditions.
-
So how do you perform your tests ? only manually ?
In your reply, didn’t you forget to mention your Agile Error Production Device whose distributed RT-Java AI, equivalent to the wisdom of 10k zen buddhists, intercepts any bug before you can even think of it?
– TBH I still like to invent my errors manually 
-
Releasing beta versions and RCs and collecting community feedback is also part of the testing regimen by the way (-:
-
great, thank you. I'll keep that in mind.
-
Hi,
Has anyone made any progress on isolating parts of RepRapFirmware ? I'm struggling to create a new kinematics, and the debugPrintf debugging method is not my favorite ...
On the other side, my quick tentative to extract a few cpp files from Movement folder lead me to Core.h and other deeply target-related files, so it seems quite intricated. Am I missing something ?
All the best,
-
@benjamin-forest unfortunately you're right, and it is even worse, because of CoreNG, which has references to the ASF libraries. Maybe you can get some ideas if you look into the LPC port of RRF what can be isolated and how.
-
There are other ways of debugging:
- Later Duets (Duet 3 and Duet 3 Mini) have a SWD port, which a debugger can be attached to. That said, I rarely use it unless I am debugging a startup problem.
- When debugging kinematics, enabling Move and DDA debug using M111 will produce quite a lot of useful debug output.
CoreN2G already replaces CoreNG for the Duet 3 Mini and the Duet 3 expansion boards, and will do so eventually for Duet WiFi/Ethernet and Maestro too.
-
