Bad command & Missing characters in Gcode since firmware upgrade



  • Just a little history first. We've build 4 large scale CoreXY machines (buildvolume: 900x1200x1500mm) running on the Duet Ethernet & Duex 5 with 4-point bed leveling. They run solidly @ 150mm/s actual printing speed but transport speeds up to 500 mm/s.
    They run 24/7 and have actually very few issues and do runs lasting from 24h to almost a week for a single print.

    However, recently I upgraded them to2.04RC1 (2019-07-14b1) and since then i noticed these kinds of error messages (copied from console):
    Error: Bad command: G X455.496 Y726.708 E0.0500
    Error: Bad command: EG1 X530.431 Y651.772 E0.0471
    Error: Bad command: 11 X326.547 Y526.795 E0.0545
    Error: Bad command: .703 Y237.446 E0.0093
    Error: Bad command: 1G1 X110.272 Y541.859 E0.2472
    Error: Bad command: E0.4945

    Just a few of them. Now normally i would expect that to not be an issue. It would just reject that line and move on to the next. But what I now see is large blobs of extruded plastic on my model. These are so large that sometimes the printheads slams into them at high speed and breaks at the heatbreak. Mayhem follows...
    0_1567426044190_Foto 29-08-19 13 02 45.jpg
    The blob in this picture is about 4cm big...

    I've checked the gcode generated from Simplify. I do find the lines that are mentioned but they are correct. For instance this error.
    This is what is in the console:
    0_1567424975710_2ca0f5c2-f438-4c2d-bc0b-8f60774a777d-image.png

    When I open that line in the original gcode file i see:
    0_1567425043004_cdcb0b6a-6e3b-469c-9853-f62f67fc79d7-image.png

    While I was writing this I wanted to do a double check. So I downloaded the gcode file from the printer back to my computer and checked that same line. This is what I saw:
    0_1567425270027_a2515273-326b-4bf6-b7c6-d6d455865a2e-image.png

    So it seems that something goes wrong during the transfer of files. Can anyone shed some light on the matter?

    Update: I have it different machines and not only on the 2.04 firmware. 1 machine is running 2.02(RTOS) (2018-12-24b1) and has the same issue. I dit a checksum compare between the original file and the downloaded file and they do not match. I wanted to do a 1:1 comparison but the files are too big (300+Mb) and my software crashes. Tried that on Notepad++, Examdiff and some other tools.


  • administrators

    @hbm-3d if the checksums don't match then the upload is not working right. Can you try a new SD card, maybe the card you have is failing after a long usage? Also i assume you are uploading over DWC (as opposed to another method , say a slicer plugin?)



  • I tried the new SD card and that seems for now to do the trick. Thank you for the help. If the issue happens again I will let you know.



  • Well sadly changing the SD card did not solve it. Same issues.

    Foto 18-09-19 12 31 02.jpg Foto 18-09-19 12 31 07.jpg

    And results in huge blobs:

    Foto 18-09-19 12 31 22.jpg Foto 18-09-19 12 31 25.jpg


  • administrators

    It could be that occasional SPI overrun errors are occurring between the Duet and the Ethernet module. We saw something similar very occasionally during initial development of the Duet Ethernet when we were using a higher SPI clock speed. So we reduced the SPI clock speed, and we never saw the problem again.

    If you like I can do a build with the SPI clock speed reduced further, for you to try out. This will have the unfortunate side effect of reducing the upload speed a little.



  • How are you uploading the gcode to the Duet? I have seen the gcode change if you are uploading directly from the slicer. Possibly try uploading directly from DWC and download it again to see if it has changed.



  • @mwolter I always upload through the DWC, never through a slicer.

    @dc42 I am happy to try but this was never an issue before. The hardware did not change. And it is on 4 out of currently 6 printers (3 more under construction). If this is an very unlikely scenario as you describe than I must buy a lottery ticket if 4 out of my 6 have the issue. 😉


  • administrators

    @HBM-3D , have the files you upload been getting bigger?



  • @dc42 Yes I do run bigger prints with the new machines. Our gcodes are generally between 150 & 550Mb.
    In this example the file was "just" 230mb which is for us a medium size run.



  • I now have it on another machine as well. So there is something common that drives this issue.
    This was with a 190Mb file. If I hash (CRC32) them I get this result:
    Original file: 5A759A1C
    Downloaded file: 654B92EC

    I did a reupload of the original file and did a download immediately after uploading it. All via the DWC.
    The has now is has the same code as the original file: 5A759A1C

    So whatever happens doesn't appear to be consistent or it happens during/after printing.


  • administrators

    I guess possible solutions are to slow down the SPI clock to see if we can find a speed that is always reliable, or to send the CRC with the uploaded data and check it at the end of the upload.

    Btw there is a gcode command to compute and return the SHA1 hash of a GCode file on the SD card. You may find it useful for checking the integrity of uploaded files.

    Can you do a compare of original vs corrupted files? I think the ancient DOS/Windows command fc /b works on large files.



  • I figured something out this weekend and tested it. If I upload the files while the printer is idle then the files are OK. If I do so when the printer is running it drops the ball and alters the file.
    This weekend I had another strange situation with this. Some line of code was wrong but ended up being a valid gcode. The printer went all the way to x, y & z-max and tried to resume printing from there. Naturally it was spaghetti and a busted hotend but nothing serious. Weird stuff and dangerous though...

    So uploading while printing is a no go, at least for bigger files i guess.


  • administrators

    @HBM-3D said in Bad command & Missing characters in Gcode since firmware upgrade:

    I figured something out this weekend and tested it. If I upload the files while the printer is idle then the files are OK. If I do so when the printer is running it drops the ball and alters the file.

    Thanks for the update, that gives me something to test.



  • @dc42 said in Bad command & Missing characters in Gcode since firmware upgrade:

    Btw there is a gcode command to compute and return the SHA1 hash of a GCode file on the SD card. You may find it useful for checking the integrity of uploaded files.

    Maybe could this be implemented in DWC? After uploading DWC would trigger M38 to get the remote SHA1 and compute local file SHA1, then compare the remote and local SHA1.
    They are some javascript sha1 libraries.



  • @dragonn
    I thought of the exact same thing but i tried to do an M38 but with these large files it took forever. After 20 minutes I had to reset the duet because I couldn't wait for that anymore. So unless it is a few seconds thing I would not want this unless it is something that can be turned off.



  • Having this exact issue on multiple boards now.

    Currently the problems is happening on:
    Firmware Name: RepRapFirmware for Duet 2 WiFi/Ethernet
    Firmware Electronics: Duet Ethernet 1.02 or later + DueX5
    Firmware Version: 2.03 (2019-06-13b2)
    Web Interface Version: 1.22.6

    I swapped out sd cards. Seems to only work when machine has been off for long period of time. I've been uploading my gcode then downloading it to verify before printing, otherwise I'll get "bad command" due to dropped characters or just sloppy prints.

    Has there been any update to how to get this fixed?



  • @norbs12 Was this also when uploading new files while printing?



  • @HBM-3D Yep.

    I also did some minor additional testing.

    I got a sha1 sum before upload, checked it again after it got uploaded to the printer and it changed. Afterward, I downloading back to my computer and the sha1 sum remained what it showed on the printer (so no longer the matching the original file).

    So this means the failure is one-way, going to the SD card (or writing to it), but not reading down from it. @dc42 does this give you any ideas on possible causes? I now have two boards with this exact behavior. Both were normal for a good year or so before they started doing this.


  • administrators

    @norbs12 can you confirm this only occurs if you are uploading while printing, this is what @HBM-3D found.


  • administrators

    I am still hoping that one of you can send me a comparison of the original vs. corrupted files, using Windows fc /b or some similar command. See my reply of 20 September.



  • @dc42 said in Bad command & Missing characters in Gcode since firmware upgrade:

    I am still hoping that one of you can send me a comparison of the original vs. corrupted files, using Windows fc /b or some similar command. See my reply of 20 September.

    diff Orig.gcode Downloaded.gcode 
    863362c863362
    < G1 X69.968 Y5.986 E0.00274
    ---
    > G1 X69.968 Y59.986 E0.00274
    863364c863364
    < G1 X70.502 Y59.729 E.00274
    ---
    > G1 X70.502 Y59.729 E0.00274
    863391c863391
    < G1 X57.055 Y6 X4.310 E0.00357
    ---
    > G1 X57.055 Y64.310 E0.00357
    908950c908950
    < 1 X62.824 Y98.654
    ---
    > G1 X62.824 Y98.654
    908990c908990
    < G1 X1G02.446 Y66.725 E0.00361
    ---
    > G1 X102.446 Y66.725 E0.00361
    909168c909168
    < G1 X103.024Y66.246 E0.00529
    ---
    > G1 X103.024 Y66.246 E0.00529
    909193c909193
    < G1 X109.288 YY66.944 E0.00303
    ---
    > G1 X109.288 Y66.944 E0.00303
    909325c909325
    < G1 105.718 Y70.234 E0.00378
    ---
    > G1 X105.718 Y70.234 E0.00378
    909346c909346
    < G1 X103.8273 Y65.606 E0.00274
    ---
    > G1 X103.873 Y65.606 E0.00274
    909374c909374
    < G1X108.523 Y68.658 E0.00274
    ---
    > G1 X108.523 Y68.658 E0.00274
    909399c909399
    < G1 F21800
    ---
    > G1 F1800
    909426c909426
    < G1 X05.883 Y46.145 E0.00355
    ---
    > G1 X105.883 Y46.145 E0.00355
    909435c909435
    < G1 X08.937 Y47.507 E0.00355
    ---
    > G1 X108.937 Y47.507 E0.00355
    909450c909450
    < G1 X1F207.974 Y52.969 E0.00373
    ---
    > G1 X107.974 Y52.969 E0.00373
    909488c909488
    < G1 X06.991 Y46.409 E0.00341
    ---
    > G1 X106.991 Y46.409 E0.00341
    909501c909501
    < G1 X2109.593 Y50.121 E0.00329
    ---
    > G1 X109.593 Y50.121 E0.00329
    909799c909799
    < G1 X108.652 Y50891 E0.00274
    ---
    > G1 X108.652 Y50.891 E0.00274
    909806c909806
    < G1 X106.9987 Y52.455 E0.00269
    ---
    > G1 X106.987 Y52.455 E0.00269
    933045c933045
    < G1 X129.607 Y68.005 F1000.000
    ---
    > G1 X129.607 Y68.005 F18000.000
    933048c933048
    < G1 X133.198 Y70.51 F18000.000
    ---
    > G1 X133.198 Y70.751 F18000.000
    933062c933062
    < G1 X122.179 Y59.2 319 E0.00576
    ---
    > G1 X122.179 Y59.319 E0.00576
    933509c933509
    < G1 X68.243 Y2.863 E0.00312
    ---
    > G1 X68.243 Y62.863 E0.00312
    933546c933546
    < G1 X7Y3.197 Y64.893 E0.00307
    ---
    > G1 X73.197 Y64.893 E0.00307
    933736c933736
    < G1 X53.455 Y66.03 E0.00355
    ---
    > G1 X53.455 Y66.003 E0.00355
    933738c933738
    < G1 X52.239 Y65.621 E0.0009
    ---
    > G1 X52.239 Y65.621 E0.00609
    933760c933760
    < G1 X53.400 Y.058.721 E0.00358
    ---
    > G1 X53.400 Y58.721 E0.00358
    1181427c1181427
    < G1 X27.108 Y2.058 E0.00194
    ---
    > G1 X27.108 Y42.058 E0.00194
    1181429c1181429
    < G1 X31.272 Y46.431 0.00343
    ---
    > G1 X31.272 Y46.431 E0.00343
    1181461c1181461
    < G.01 X27.108 Y45.205 E0.12357
    ---
    > G1 X27.108 Y45.205 E0.12357
    1181488c1181488
    < G1 X34.926 Y55.959 E000139
    ---
    > G1 X34.926 Y55.959 E0.00139
    1181513c1181513
    < G1 X207.108 Y50.658 E0.09256
    ---
    > G1 X27.108 Y50.658 E0.09256
    


  • Thanx. My files were all too big to compare. I tried 4 different program but they all crashed or simply refused to do it.



  • Try notepad++ It has a compare plugin


  • administrators

    @HBM-3D said in Bad command & Missing characters in Gcode since firmware upgrade:

    Thanx. My files were all too big to compare. I tried 4 different program but they all crashed or simply refused to do it.

    Did you try Windows fc /b ? AFAIK it can handle very large files. The output when using /b isn't line oriented, but that might actually make things easier for me.

    Cam you confirm that the original and corrupt files have exactly the same length? They should have, because the upload mechanism does a check on the file size at the end.


  • administrators

    @norbs12 said in Bad command & Missing characters in Gcode since firmware upgrade:

    @dc42 said in Bad command & Missing characters in Gcode since firmware upgrade:

    I am still hoping that one of you can send me a comparison of the original vs. corrupted files, using Windows fc /b or some similar command. See my reply of 20 September.

    diff Orig.gcode Downloaded.gcode
    ...
    

    Thanks. There is something very odd about those diffs. For example:

    > 863364c863364
    > < G1 X70.502 Y59.729 E.00274
    > ---
    > > G1 X70.502 Y59.729 E0.00274
    

    Clearly, a 0 has been dropped here (after the E). In most other instances, one character has again been dropped.

    > 863391c863391
    > < G1 X57.055 Y6 X4.310 E0.00357
    > ---
    > > G1 X57.055 Y64.310 E0.00357
    

    But here, " X" appears to have been inserted.

    > 908990c908990
    > < G1 X1G02.446 Y66.725 E0.00361
    > ---
    > > G1 X102.446 Y66.725 E0.00361
    

    This time, "G" appears to have been inserted.

    > 909193c909193
    > < G1 X109.288 YY66.944 E0.00303
    > ---
    > > G1 X109.288 Y66.944 E0.00303
    

    And here, 'Y' has been inserted.

    The fact that both these reports are using Duet Ethernet and not Duet WiFi leads me to believe that this is an issue with the transfer of data by DMA from the Ethernet chip, or just possibly an interaction between DMA to the SD card and DMA from the Ethernet chip..


Log in to reply