After Firmware update from 3.4.5 to 3.5.2 my printer crashes

Chrashem

@droftarts
Thx for your reply Ian!
I installed the Plugin and I flew over the release notes, but at the moment I didn't find something strange or suspicious.
But in general: The more I read about the firmware the more I realise how stupid it was just to click "install" without deeper investigations regarding the config....
Anyway: I guess I learned my lesson on the hard way and the good point is I can repair / rebuild the machine.

For Probing I use a "Superpinda" by "Pepperl & Fuchs" (inductive sensor)

Here is my config.g

; Configuration file for Duet 3 MB 6HC (firmware version 3.3)
; executed by the firmware on start-up
;
; generated by RepRapFirmware Configuration Tool v3.3.15 on Sat Feb 25 2023 23:42:30 GMT+0100 (CET)

; General preferences
G90                                             ; send absolute coordinates...
M83                                             ; ...but relative extruder moves
M550 P"Duet 3"                                  ; set printer name
M669 K1                                         ; select CoreXY mode

; Drives
M569 P0.0 S0									; physical drive 0.0 goes forwards
M569 P0.1 S0									; physical drive 0.1 goes forwards
M569 P0.3 S1  ;gedreht                                 ; physical drive 0.3 goes forwards
M569 P0.4 S1  ;gedreht                                  ; physical drive 0.4 goes forwards
M569 P0.2 S0                                    ; physical drive 0.1 goes forwards
M569 P0.5 S1                                    ; physical drive 0.5 goes forwards
M584 X0.4 Y0.3 Z0.0:0.1:0.2 E0.5                ; set drive mapping
M350 X16 Y16 Z16 E16 I1                         ; configure microstepping with interpolation
M92 X80.00 Y80.00 Z800.00 E686.5               ; set steps per mm ; LDO Orbiter 2.0  (Nach Kalibrierung am 30.10. Esteps von 690 auf E686,5 gesetzt
;M92 X80.00 Y80.00 Z800.00 E573.67               ; set steps per mm ; LGX Lite
;E573.49 Nach kallibrierung am 4.3.23 (Initalwert waren 562,00 mm)
M566 X400.00 Y400.00 Z6.00 E300.00       ; set maximum instantaneous speed changes (mm/min) ;LDO Orbiter 2.0
;M566 X400.00 Y400.00 Z6.00 E120.00       ; set maximum instantaneous speed changes (mm/min) ; LGX Lite
M203 X10800.00 Y10800.00 Z1000.00 E7200.00 ; set maximum speeds (mm/min) ; LDO Orbiter 2.0
;M203 X10800.00 Y10800.00 Z1000.00 E3600.00 ; set maximum speeds (mm/min) ; LGX Lite
M201 X12000.00 Y12000.00 Z150.00 E10000.00              ; set accelerations (mm/s^2) 
M906 X1600 Y1600 Z1600 I30                 ; set motor currents (mA) and motor idle factor in per cent ; LDO Orbiter 2.0
M906 E1200 I10								;set motor currents (mA) and motor idle factor in per cent ; LDO Orbiter 2.0

M84 S30                                         ; Set idle timeout

; Axis Limits
M208 X0 Y0 Z0 S1                                ; set axis minima
M208 X305 Y300 Z300 S0                          ; set axis maxima

; Endstops
M574 X1 S1 P"io8.in"                            ; configure switch-type (e.g. microswitch) endstop for high end on X via pin io8.in
M574 Y2 S1 P"io7.in"                            ; configure switch-type (e.g. microswitch) endstop for high end on Y via pin io7.in

; Z-Probe
M558 P5 C"io6.in" H5 F400 T5000  
;M558 P1 C"io6.in" H5 F120 T6000                 ; set Z probe type to unmodulated and the dive height + speeds
;M558 H30                                        ;*** Remove this line after delta calibration has been done and new delta parameters have been saved

G31 P500 X-30 Y-13 Z1.192						; set Z probe trigger value, offset and trigger height ; Trigger height bei 60° Betttemperatur
;27.02.23: Z Offset 2,2
;28.02.23: Z Offset 1,804
M671 X-4.5:150:304.5 Y-4.52:305:-4.52 S5.      ; Positionen der Z Spindeln und maximaler Ausgleichsweg pro Spindel

M557 X7:275 Y15:280 P10:10                        ; define mesh grid

; Heaters
M308 S2 P"temp2" Y"pt1000" A"Gehause"			; Anlegen eines Gehäusesensors
M308 S0 P"temp0" Y"thermistor" T100000 B3950 A"Bed"    ; configure sensor 0 as thermistor on pin temp0 ; alter Sensor nur B3950
M950 H0 C"out2" T0                              ; create bed heater output on out2 and map it to sensor 0
M307 H0 B1 S1.00                                ; enable bang-bang mode for the bed heater and set PWM limit
M140 H0                                         ; map heated bed to heater 0
M143 H0 S120                                    ; set temperature limit for heater 0 to 120C
M308 S1 P"temp1" Y"thermistor" T100000 B4680 C6.455513e-8 A"Nozzle"    ; configure sensor 1 as thermistor on pin temp1
M950 H1 C"out1" T1                              ; create nozzle heater output on out1 and map it to sensor 1
M307 H1 B0 S1.00                                ; disable bang-bang mode for heater  and set PWM limit
M143 H1 S280                                    ; set temperature limit for heater 1 to 280C

; Fans
M950 F0 C"!out4+out4.tach" Q250					 ; Erzeugen eines Fans 0 an pin out 4 (gedreht) und tachosignal und seiner Frequenz
M106 P0 S0 L0.0 X0.8 H-1 C"Layerluefter"                                  ; set fan 0 value. Thermostatic control is turned off
M950 F1 C"out7" Q500                            ; create fan 1 on pin out7 and set its frequency
M106 P1 S1 H1 T45                               ; set fan 1 value. Thermostatic control is turned on


;Filterluefter
M950 F2 C"!out5+out5.tach" Q25000					;Erzeugen eines Fans an Pin out 5 (gedreht) und Tachosignal mit Freuquenz
M106 P2 S0 L0.0 X1.0 H-1 C"Filterluefter"		;Festlegen dass der Lüfter mit 0 Aufstartet, Maximal mit halber Geschwindigkeit drehen kann und Filterluefter heißt 

;M950 F0 C"out4" Q500                            ; create fan 0 on pin out4 and set its frequency
;M106 P0 S0 H-1                                  ; set fan 0 value. Thermostatic control is turned off
;M950 F1 C"out7" Q500                            ; create fan 1 on pin out7 and set its frequency
;M106 P1 S1 H1 T45                               ; set fan 1 value. Thermostatic control is turned on

; Tools
M563 P0 D0 H1 F0                                ; define tool 0
G10 P0 X0 Y0 Z0                                 ; set tool 0 axis offsets
G10 P0 R0 S0                                    ; set initial tool 0 active and standby temperatures to 0C


;Beschleunigungsaufnehmer
M955 P0 I56 C"spi.cs3+spi.cs2"  					;Beschleunigungsaufnehmer an Anschluss für Temperaturerweiterungsboard mit Ausrichtung 54 oder 56?

; Input Shaping
M593 P"mzv" F30										;Aufsetzen des INput Shapers

;Skew Adjustment
M556 S100 X-0.137		; Durchgeführt am 17.04.2024 mit PETG_Matt_Blau - 004_2024_04_17_PETG_Matt_Blau_Califlower Calculator v14.xlsx

; Custom settings are not defined

; Miscellaneous
M501                                            ; load saved parameters from non-volatile memory

and here is my bed.g:

; bed.g
; called to perform automatic bed compensation via G32
;
; generated by RepRapFirmware Configuration Tool v3.3.15 on Sat Feb 25 2023 23:42:30 GMT+0100 (CET)
;M561 ; clear any bed transform
;G29  ; probe the bed and enable compensation

;Tool nutzt die Offset Koordinaten von der Z Probe als Target Werte
;Eingestellt und angegeben wird der Probepunkt! --> Überprüfen ob Werte auch aufs Bett Passen! 
;X_Probe_Zielpunkt = X_Absolut_Nozzle + X_Offset_Probe
;Y_Probe_Zielpunkt = Y_Absolut_Nozzle + Y_Offset_Probe

M561 ;clear any bed transform
G30 P0 X7 Y12 Z-99999 ; Probe near a leadscrew
G30 P1 X150 Y283 Z-99999 ; Probe near a leadscrew
G30 P2 X275 Y12 Z-99999 S3 ;probe near a leadscrew and calibrate 3 motors

Thx for your Help!

droftarts

@Chrashem I can't see anything obvious in either your config.g or bed.g that would cause the crash. Could you post your homeall.g too, and any other macros that are called within it? To you use a tool change between probing?

If possible, it would be good to try and replicate this bug, and work out where it has come from, though I don't want you to damage your machine again!

Ian

Chrashem

@droftarts
Hi Ian,
thank you for your answer. I’m sorry for my late reply, but I was three days on vacation. A bit below you’ll find my homeall.g .
I assume that the homeall is fine because the homeing itself worked and the crash happened during the "true bed levelling", but maybe I'm wrong. To be honest it happened very quickly.
I just have one Tool and I didn’t use a toolchange.
So I don’t think that this is the root of evil.
I think there is no other way than try it again. Is there a special development or kind of logging mode which I can activate? That we gain more information about what’s going on?
Otherwise i be afraid, that I will crash my machine again and we still have no clue about why it happens.

What do you think? Should I limit the maximum speeds and accelerations for the test? Maybe than the printer is doing slow movements and I will gain some time to react if something is going wrong.

homeall.g:

; homeall.g
; called to home all axes
;
; generated by RepRapFirmware Configuration Tool v3.3.15 on Sat Feb 25 2023 23:42:30 GMT+0100 (CET)
G91                   	; relative positioning
G1 H2 Z5 F10000			; lift Z relative to current position
G1 H1 X-625 F4000		; move quickly to X Endstop
G1 X3 F600       	 	; go back a few mm
G1 H1 X-625 F360 		; Move slowly to X endstops once more (second pass)

G1 H1 Y625 F4000		; Move quickly to Y Endstop
G1 Y-3 F600 			; Go back a few mm
G1 H1 Y625 F360			; Move slowly to y endstop once more (second pass)

M201 X10000 Y10000        ; Return to full acceleration

G90                     ; Absolute positioning
G1 X150 Y150 F8000     ; Go to the center of the bed (300x300)
G30                     ; Home Z by probing the bed

G91                     ; Relative positioning
;G1 Z5 F100              ; Lift Z relative to current position
G90                     ; Absolute positioning



;G1 H1 X305            ; home X axis
;G1 H1 Y305            ; home Y axis
;G1 X-5 Y-5 F6000      ; go back a few mm
;G1 H1 X305 F360       ; move slowly to X axis endstop once more (second pass)
;G1 H1 Y305            ; then move slowly to Y axis endstop
;G1 H1 Z-305 F360      ; move Z down stopping at the endstop
;G90                   ; absolute positioning
;G92 Z0                ; set Z position to axis minimum (you may want to adjust this)

; Uncomment the following lines to lift Z after probing
;G91                  ; relative positioning
;G1 Z5 F100           ; lift Z relative to current position
;G90                  ; absolute positioning

droftarts

@Chrashem I wonder if, somehow, after homing the machine is left in relative movement mode. That might explain the strange movement of bed.g. You have G90 at the end of homeall.g, but also a number of commented out commands. If you run homeall.g, then check in the object model what movement mode (relative or absolute) is in use. I’m on holiday for a week, so can’t check where this is reported in the OM, as I’m away from all my printers!

Ian

Chrashem

@droftarts

Hi Ian,
i can fully understand your mentions. I set up the homeall.g nearly 1.5 years ago during the commissioning and never clean up, because it still worked...
Sorry for the mess :-).
But you're completely right a G91 directly followed by a G90 makes no sense for the G91.

What would you recommend?
-Clean up the homeall.g?
-limiting the speed and accelerations for the test?

And I'm sorry, but what do you mean with the Object model? Where I can find it?

Thank you for your help!

kind regards
Dennis

Phaedrux

@Chrashem said in After Firmware update from 3.4.5 to 3.5.2 my printer crashes:

And I'm sorry, but what do you mean with the Object model? Where I can find it?

https://docs.duet3d.com/en/User_manual/RepRapFirmware/Object_Model

Chrashem

Hi @Phaedrux,
thx for this information! At the moment I didn't know how I should use the object model useful with my knowlage.
When the printer is going crazy again, I have to stop it! Unfortunately the only thing which helped last time was to turn of the power supply. I guess that after I a restart the object model isn't so helpful or am I wrong?
Is there a reporting or logging of the movement which I can check later?

How should I proceed? From the mechanical point of view, my printer ist completely rebuilt.
But I didn't do a change in the configuration so far.

My Plan for the next steps / for the commissioning after is:

to clean up the homeall.g
to turn down (limit) the maximum values for speed and acceleration in the config.g
to do the commissioning again (https://docs.duet3d.com/en/How_to_guides/Commissioning)
keep on praying that the printer won't try again some suicide

Anything I forgot? Or do you recommend some other steps?
Is there logging / or report mode which I can use / turn on?

kind regards
Dennis

Phaedrux

The object model is like a listing of current values and settings that can be referenced in conditional gcode commands for programming purposes. Not exactly applicable for your current issues.

You may wish to reduce the motor current during homing to reduce the risk of damage in a crash. You can use the M913 command in your homing files at the beginning and end to reduce the current to a percentage and then restore it to full.

https://docs.duet3d.com/User_manual/Reference/Gcodes#m913-set-motor-percentage-of-normal-current

It can also help to manually send the commands in your homing files to see where an issue begins.

Getting a video of it in action when the problem occurs can also be helpful for us to diagnose.

Rolling back your firmware version to see if the issue is resolved or not would help determine if it's actually from updating the firmware or something else.

Chrashem

Hi @Phaedrux,
thx for your help!

The last days I did the commissioning of the printer. I build up a camera and checked every .g file line by line. So far it seems okay.
Homeing, levelling the bed and creating a bed mesh were working properly without strange moves or a crash.
Just the point that I didn't reduce or change the commands ensures that a bad feeling remains.

And if you think you solved the problem, suddenly an other problem appears...
It seems that something in the firmware changed, so the g-code generated by my superslicer wasn't working anymore.

After generating the g-code and try to simulate it on the duet, I got directly some messages that the simulation got canceled in reason of an illegal character "_" was detected.

It seems , that a few lines of custom code variables in the superslicer config, which I never payed attention before, are now a problem.

Its stuff like:
In the slicer

; once we start printing infill, increase square corner velocity to 10
{if extrusion_role=="InternalInfill"}SET_VELOCITY_LIMIT SQUARE_CORNER_VELOCITY=10 {endif}

leads to the following in the g-code

SET_VELOCITY_LIMIT SQUARE_CORNER_VELOCITY=5

This kind of lines a generating the "illegal character "_"" error message.

For a short test,I deactivate the lines by adding a ";" in the slicer config for a short test and generated some new g-code.

In generel it seems that the code is working, but in my case I caused a special situation with a crash which can be reproduced.

If try to simulate an old g-code version with the illigal character, it will abort the simulation with the error message.
After this I tried to simulating the "fixed" version (without a restart of the duet before), the simulation is successfully done. If I start the print, the printer is just lifting the bed against the printer head without moving the head--> crash.

If I restart the duet and try again to print the fixed version of the g-code, the printer is doing his job without a crash....

droftarts

@Chrashem said in After Firmware update from 3.4.5 to 3.5.2 my printer crashes:

{if extrusion_role=="InternalInfill"}SET_VELOCITY_LIMIT SQUARE_CORNER_VELOCITY=10 {endif}

That's a command that's meant for Klipper firmware. I'm not sure why RepRapFirmware was ignoring it before, or if you have changed settings in SuperSlicer. I'm not too familiar with SuperSlicer, if that's the slicer you are using, but it's based on Prusaslicer, and in Prusaslicer you can set the Gcode version in Printers > General > Firmware > G-code flavor. Make sure this is set to RepRapFirmware, not Klipper.

Ian

Chrashem

@droftarts

that's interesting! I checked my g-code flavor in the slicer, but it's already set to RepRap. I barely remember, but in my opinion the slicer was set to klipper by default at the installation . When I configured it, I set it to "RepRap".

This kind of code which causes the illegal character issue is in the "custom-code" area of the slicer. To adjust it you have to do it manually.
I know that I was wondering about it several times, but I never touched it since 1.5 years because the g-code still worked.

Now after the firmware update it was the first time the g-code from the slicer isn't working.
After deactivating these lines, the code runs (after a restart of the mainboard, otherwise it leads directly to a crash).

Summarised these are the following lines which a had to deactivate:

SET_GCODE_OFFSET Z=0

START_PRINT EXTRUDER_TEMP={first_layer_temperature+extruder_temperature_offset} BED_TEMP=[first_layer_bed_temperature]

{if extrusion_role=="InternalInfill"}SET_VELOCITY_LIMIT SQUARE_CORNER_VELOCITY=10 {endif}
{if extrusion_role!="InternalInfill"}SET_VELOCITY_LIMIT SQUARE_CORNER_VELOCITY=5 {endif}

I assume all of this are klipper commands?

droftarts

@Chrashem said in After Firmware update from 3.4.5 to 3.5.2 my printer crashes:

I assume all of this are klipper commands?

Yes, they look like Klipper commands. Maybe at some point the profiles got updated, and if you have RatRig selected, it put those commands in. Being in the custom Gcode section, they would still be output despite 'RepRap' being selected as the Gcode flavour.

I don't know if the RRF Gcode interpreter has changed to make the underscore _ an illegal character recently (it is possible, though I don't think there has been any change to this), but I can't see anything in the release notes saying this. I think the error is being generated because it is seeing the T in 'SET_...' as a tool change command, and then the underscore is not a valid character for the 'T' tool command.

Ian

Chrashem

I cleared all the commands and I did my first test print. But I have still a little issue with the movement. Sometimes my movement is a bit jerky.
For example, if the printer should print a straight line in a continuous movement, it's printing a straight line, but the movement isn't continuous.
It's hard to explain. Its not like a layer shift, its more like a "stopping" for a fraction of a second" You can also her hear that the movement isn't continuous.

I figured out, that it's not depending in which direction the movement is going.
So unfortunately it's not just one stepper. It's happening in diagonal or straight moves.

Edit:
I think I have to take a closer look at my belt drive system. In my opinion it isn't working fluently.
When I my move the printhead by hand it seems that there a some points with a lot of friction.
Maybe my damage is bigger than I thought.

droftarts

@Chrashem said in After Firmware update from 3.4.5 to 3.5.2 my printer crashes:

Sometimes my movement is a bit jerky. For example, if the printer should print a straight line in a continuous movement, it's printing a straight line, but the movement isn't continuous. It's hard to explain. Its not like a layer shift, its more like a "stopping" for a fraction of a second" You can also her hear that the movement isn't continuous.

While it is possible this is due to damage to the drive system, if you're using mesh compensation, it may also be due to the Z axis having to move when the mesh compensation requires it. Having slow Z speeds can make this more of an issue. Currently, your speeds are:

M350 X16 Y16 Z16 E16 I1                    ; configure microstepping with interpolation
M92 X80.00 Y80.00 Z800.00 E686.5           ; set steps per mm LDO Orbiter 2.0  (Nach Kalibrierung am 30.10. Esteps von 690 auf E686,5 gesetzt)
M566 X400.00 Y400.00 Z6.00 E300.00         ; set maximum instantaneous speed changes (mm/min) ;LDO Orbiter 2.0
M203 X10800.00 Y10800.00 Z1000.00 E7200.00 ; set maximum speeds (mm/min) ; LDO Orbiter 2.0
M201 X12000.00 Y12000.00 Z150.00 E10000.00 ; set accelerations (mm/s^2)

If you can increase the speed of the Z axis, particularly the M566 instantaneous speed change, that may help.

Ian

Chrashem

@droftarts
this is also a possibility. But on the other hand I could observe this jerky behaviour during a plane xy movement which is included in my homeall macro. During this there should be no Z movement.

But raising the speed is in general a good hint. I will try this. But I will also check the mechanical side of the gantry system. If there is something wrong which is causing the issue, i guess its better to fix this

Phaedrux

@Chrashem said in After Firmware update from 3.4.5 to 3.5.2 my printer crashes:

When I my move the printhead by hand it seems that there a some points with a lot of friction.

I'd be looking for a bad bearing, uneven belt tension, or skew in the gantry.

Chrashem

@droftarts @Phaedrux
at the moment I learned a lot about my printer....
...unfortunately on the hard way...

From the movement side now it seems okay. In the mechanic I found a bit of friction caused by skew in the gantry. I could fix it by aligning the gantry. The movements by manually pushing the print head were fluent and smooth. But printing a flat surface showed again a bit of jerking.
So as @droftarts mentioned I raise up the z-speed changes. (Maybe I carried it to far, because I raised also the speed and the acceleration)

M566 X400.00 Y400.00 Z50.00 E300.00 
M203 X10800.00 Y10800.00 Z2000.00 E7200.00
M201 X10000.00 Y10000.00 Z300.00 E10000.00

The first test was fine! I Printed two filled layers and it seems that there is no XY-Jerking anymore and the surface quality was also looking good.

But unfortunately a bit later my printer crashed again...

My printer was homed and I started a simulation of a g-code.
This g-code seems to work because its the test part with the surface, I printed before... I got the message that the simulation was done successfully.
After this I tried to do a bed levelling by G32 and at this moment the printhead didn't move in a xy direction and the z axis was just lifting the bed without stopping against the printhead... Like the crash in my first post here...
Unfortunately the z-speed was very high, but fortunately the printhead was on a position where the bed could tilt, so there is no heavy damage. Just one piece from the z-axis is broken and I can replace it.

It feels like the root of evil is anywhere in the combination of the simulation and the G32....
I thought, that I got maybe a state where the printer isn't homed and no x and y movements are allowed? But in this case I thought that the z axis should stop when the probe is triggering the bed... (and the probe is working).

Is there a possibility to do a kind of a simple snapshot of the OM before and after the simulation?

For more testing I will try to reduce the travel speed during the probing and raise the diving height (just to get generating more time to react):

M558 P5 C"io6.in" H15 F300 T1000

Hopefully I can test more tomorrow, but first I have to replace the broken part of the z-Axis.

An other point which I noticed:
Is it possible, that the connection to DWC is more unstable? I recognised sometimes that the browser is generating a new connection to DWC.
With the old firmware I never got this problem. The position of the printer is the same, my laptop is the same and the router is also the same ( router and printer in the same room, distance just 3m away from each other...)

EDIT:
I could confirm it:
If I using a "G32" after a simulation (it doesn't matter if its a successful simulation or an failed one) my printer isn't moving the print head, but try to lifts the bed up till the crash.
I did several tests to get a better clue about it. After a simulation, DWC is reporting full movable axis. I could use the Buttons to move the axis and the printer is doing it fine. But if I click the bed levelling (G32) the printer is just lift the bed and would crash.
During this it doesn't matters which position my z-axis have. For example: If I have the probing height h set to 25 mm and the actual z position is 5 mm, normally the printer would gain the space to the probing height.
After the Simulation the printer is just lifting the bed...

If I do a homing of all axis after the simulation, and do then a G32 the printer is doing a fine levelling of the bed.
But if I don't do a full homing and do a single homing the behaviour depends on which axis I homed:

Simulation and homing x-axis: Lifting the bed -->crash
Simulation and homing y-axis: Lifting the bed -->crash
Simulation and homing x and y axis: Lifting the bed --> crash
Simulation and homing the z axis --> its just moving the y axis to the probing position (not the x axis) and tries to probe there. In my case the probe was not on the bed so I had to stop the printer manually to prevent a crash
Simulation and homing x y z axis --> G32 worked perfectly

Phaedrux

Can you enable some additional logging on the SBC to share here?

https://github.com/Duet3D/DuetSoftwareFramework/wiki/SBC-Setup-Guide#increasing-log-level

Chrashem

@Phaedrux
For sure!
I set up the logging. What movements or commands should I do with the logging on?

What do you want to see?

For the crash situation I have the problem, that I have to cut the power supply immediately I recognised, that the printer is out of control. I don't know if the logging will work under these conditions.

Chrashem

@Phaedrux @droftarts

At the moment I'm at home and I can run some tests for you with the logging. Can you tell me which movements do you want to see with the additional logging?

How can I share the log? Should I copy the text?