After Firmware update from 3.4.5 to 3.5.2 my printer crashes

Chrashem

@droftarts
Hi Ian,
thank you for your answer. I’m sorry for my late reply, but I was three days on vacation. A bit below you’ll find my homeall.g .
I assume that the homeall is fine because the homeing itself worked and the crash happened during the "true bed levelling", but maybe I'm wrong. To be honest it happened very quickly.
I just have one Tool and I didn’t use a toolchange.
So I don’t think that this is the root of evil.
I think there is no other way than try it again. Is there a special development or kind of logging mode which I can activate? That we gain more information about what’s going on?
Otherwise i be afraid, that I will crash my machine again and we still have no clue about why it happens.

What do you think? Should I limit the maximum speeds and accelerations for the test? Maybe than the printer is doing slow movements and I will gain some time to react if something is going wrong.

homeall.g:

; homeall.g
; called to home all axes
;
; generated by RepRapFirmware Configuration Tool v3.3.15 on Sat Feb 25 2023 23:42:30 GMT+0100 (CET)
G91                   	; relative positioning
G1 H2 Z5 F10000			; lift Z relative to current position
G1 H1 X-625 F4000		; move quickly to X Endstop
G1 X3 F600       	 	; go back a few mm
G1 H1 X-625 F360 		; Move slowly to X endstops once more (second pass)

G1 H1 Y625 F4000		; Move quickly to Y Endstop
G1 Y-3 F600 			; Go back a few mm
G1 H1 Y625 F360			; Move slowly to y endstop once more (second pass)

M201 X10000 Y10000        ; Return to full acceleration

G90                     ; Absolute positioning
G1 X150 Y150 F8000     ; Go to the center of the bed (300x300)
G30                     ; Home Z by probing the bed

G91                     ; Relative positioning
;G1 Z5 F100              ; Lift Z relative to current position
G90                     ; Absolute positioning



;G1 H1 X305            ; home X axis
;G1 H1 Y305            ; home Y axis
;G1 X-5 Y-5 F6000      ; go back a few mm
;G1 H1 X305 F360       ; move slowly to X axis endstop once more (second pass)
;G1 H1 Y305            ; then move slowly to Y axis endstop
;G1 H1 Z-305 F360      ; move Z down stopping at the endstop
;G90                   ; absolute positioning
;G92 Z0                ; set Z position to axis minimum (you may want to adjust this)

; Uncomment the following lines to lift Z after probing
;G91                  ; relative positioning
;G1 Z5 F100           ; lift Z relative to current position
;G90                  ; absolute positioning

droftarts

@Chrashem I wonder if, somehow, after homing the machine is left in relative movement mode. That might explain the strange movement of bed.g. You have G90 at the end of homeall.g, but also a number of commented out commands. If you run homeall.g, then check in the object model what movement mode (relative or absolute) is in use. I’m on holiday for a week, so can’t check where this is reported in the OM, as I’m away from all my printers!

Ian

Chrashem

@droftarts

Hi Ian,
i can fully understand your mentions. I set up the homeall.g nearly 1.5 years ago during the commissioning and never clean up, because it still worked...
Sorry for the mess :-).
But you're completely right a G91 directly followed by a G90 makes no sense for the G91.

What would you recommend?
-Clean up the homeall.g?
-limiting the speed and accelerations for the test?

And I'm sorry, but what do you mean with the Object model? Where I can find it?

Thank you for your help!

kind regards
Dennis

Phaedrux

@Chrashem said in After Firmware update from 3.4.5 to 3.5.2 my printer crashes:

And I'm sorry, but what do you mean with the Object model? Where I can find it?

https://docs.duet3d.com/en/User_manual/RepRapFirmware/Object_Model

Chrashem

Hi @Phaedrux,
thx for this information! At the moment I didn't know how I should use the object model useful with my knowlage.
When the printer is going crazy again, I have to stop it! Unfortunately the only thing which helped last time was to turn of the power supply. I guess that after I a restart the object model isn't so helpful or am I wrong?
Is there a reporting or logging of the movement which I can check later?

How should I proceed? From the mechanical point of view, my printer ist completely rebuilt.
But I didn't do a change in the configuration so far.

My Plan for the next steps / for the commissioning after is:

to clean up the homeall.g
to turn down (limit) the maximum values for speed and acceleration in the config.g
to do the commissioning again (https://docs.duet3d.com/en/How_to_guides/Commissioning)
keep on praying that the printer won't try again some suicide

Anything I forgot? Or do you recommend some other steps?
Is there logging / or report mode which I can use / turn on?

kind regards
Dennis

Phaedrux

The object model is like a listing of current values and settings that can be referenced in conditional gcode commands for programming purposes. Not exactly applicable for your current issues.

You may wish to reduce the motor current during homing to reduce the risk of damage in a crash. You can use the M913 command in your homing files at the beginning and end to reduce the current to a percentage and then restore it to full.

https://docs.duet3d.com/User_manual/Reference/Gcodes#m913-set-motor-percentage-of-normal-current

It can also help to manually send the commands in your homing files to see where an issue begins.

Getting a video of it in action when the problem occurs can also be helpful for us to diagnose.

Rolling back your firmware version to see if the issue is resolved or not would help determine if it's actually from updating the firmware or something else.

Chrashem

Hi @Phaedrux,
thx for your help!

The last days I did the commissioning of the printer. I build up a camera and checked every .g file line by line. So far it seems okay.
Homeing, levelling the bed and creating a bed mesh were working properly without strange moves or a crash.
Just the point that I didn't reduce or change the commands ensures that a bad feeling remains.

And if you think you solved the problem, suddenly an other problem appears...
It seems that something in the firmware changed, so the g-code generated by my superslicer wasn't working anymore.

After generating the g-code and try to simulate it on the duet, I got directly some messages that the simulation got canceled in reason of an illegal character "_" was detected.

It seems , that a few lines of custom code variables in the superslicer config, which I never payed attention before, are now a problem.

Its stuff like:
In the slicer

; once we start printing infill, increase square corner velocity to 10
{if extrusion_role=="InternalInfill"}SET_VELOCITY_LIMIT SQUARE_CORNER_VELOCITY=10 {endif}

leads to the following in the g-code

SET_VELOCITY_LIMIT SQUARE_CORNER_VELOCITY=5

This kind of lines a generating the "illegal character "_"" error message.

For a short test,I deactivate the lines by adding a ";" in the slicer config for a short test and generated some new g-code.

In generel it seems that the code is working, but in my case I caused a special situation with a crash which can be reproduced.

If try to simulate an old g-code version with the illigal character, it will abort the simulation with the error message.
After this I tried to simulating the "fixed" version (without a restart of the duet before), the simulation is successfully done. If I start the print, the printer is just lifting the bed against the printer head without moving the head--> crash.

If I restart the duet and try again to print the fixed version of the g-code, the printer is doing his job without a crash....

droftarts

@Chrashem said in After Firmware update from 3.4.5 to 3.5.2 my printer crashes:

{if extrusion_role=="InternalInfill"}SET_VELOCITY_LIMIT SQUARE_CORNER_VELOCITY=10 {endif}

That's a command that's meant for Klipper firmware. I'm not sure why RepRapFirmware was ignoring it before, or if you have changed settings in SuperSlicer. I'm not too familiar with SuperSlicer, if that's the slicer you are using, but it's based on Prusaslicer, and in Prusaslicer you can set the Gcode version in Printers > General > Firmware > G-code flavor. Make sure this is set to RepRapFirmware, not Klipper.

Ian

Chrashem

@droftarts

that's interesting! I checked my g-code flavor in the slicer, but it's already set to RepRap. I barely remember, but in my opinion the slicer was set to klipper by default at the installation . When I configured it, I set it to "RepRap".

This kind of code which causes the illegal character issue is in the "custom-code" area of the slicer. To adjust it you have to do it manually.
I know that I was wondering about it several times, but I never touched it since 1.5 years because the g-code still worked.

Now after the firmware update it was the first time the g-code from the slicer isn't working.
After deactivating these lines, the code runs (after a restart of the mainboard, otherwise it leads directly to a crash).

Summarised these are the following lines which a had to deactivate:

SET_GCODE_OFFSET Z=0

START_PRINT EXTRUDER_TEMP={first_layer_temperature+extruder_temperature_offset} BED_TEMP=[first_layer_bed_temperature]

{if extrusion_role=="InternalInfill"}SET_VELOCITY_LIMIT SQUARE_CORNER_VELOCITY=10 {endif}
{if extrusion_role!="InternalInfill"}SET_VELOCITY_LIMIT SQUARE_CORNER_VELOCITY=5 {endif}

I assume all of this are klipper commands?

droftarts

@Chrashem said in After Firmware update from 3.4.5 to 3.5.2 my printer crashes:

I assume all of this are klipper commands?

Yes, they look like Klipper commands. Maybe at some point the profiles got updated, and if you have RatRig selected, it put those commands in. Being in the custom Gcode section, they would still be output despite 'RepRap' being selected as the Gcode flavour.

I don't know if the RRF Gcode interpreter has changed to make the underscore _ an illegal character recently (it is possible, though I don't think there has been any change to this), but I can't see anything in the release notes saying this. I think the error is being generated because it is seeing the T in 'SET_...' as a tool change command, and then the underscore is not a valid character for the 'T' tool command.

Ian

Chrashem

I cleared all the commands and I did my first test print. But I have still a little issue with the movement. Sometimes my movement is a bit jerky.
For example, if the printer should print a straight line in a continuous movement, it's printing a straight line, but the movement isn't continuous.
It's hard to explain. Its not like a layer shift, its more like a "stopping" for a fraction of a second" You can also her hear that the movement isn't continuous.

I figured out, that it's not depending in which direction the movement is going.
So unfortunately it's not just one stepper. It's happening in diagonal or straight moves.

Edit:
I think I have to take a closer look at my belt drive system. In my opinion it isn't working fluently.
When I my move the printhead by hand it seems that there a some points with a lot of friction.
Maybe my damage is bigger than I thought.

droftarts

@Chrashem said in After Firmware update from 3.4.5 to 3.5.2 my printer crashes:

Sometimes my movement is a bit jerky. For example, if the printer should print a straight line in a continuous movement, it's printing a straight line, but the movement isn't continuous. It's hard to explain. Its not like a layer shift, its more like a "stopping" for a fraction of a second" You can also her hear that the movement isn't continuous.

While it is possible this is due to damage to the drive system, if you're using mesh compensation, it may also be due to the Z axis having to move when the mesh compensation requires it. Having slow Z speeds can make this more of an issue. Currently, your speeds are:

M350 X16 Y16 Z16 E16 I1                    ; configure microstepping with interpolation
M92 X80.00 Y80.00 Z800.00 E686.5           ; set steps per mm LDO Orbiter 2.0  (Nach Kalibrierung am 30.10. Esteps von 690 auf E686,5 gesetzt)
M566 X400.00 Y400.00 Z6.00 E300.00         ; set maximum instantaneous speed changes (mm/min) ;LDO Orbiter 2.0
M203 X10800.00 Y10800.00 Z1000.00 E7200.00 ; set maximum speeds (mm/min) ; LDO Orbiter 2.0
M201 X12000.00 Y12000.00 Z150.00 E10000.00 ; set accelerations (mm/s^2)

If you can increase the speed of the Z axis, particularly the M566 instantaneous speed change, that may help.

Ian

Chrashem

@droftarts
this is also a possibility. But on the other hand I could observe this jerky behaviour during a plane xy movement which is included in my homeall macro. During this there should be no Z movement.

But raising the speed is in general a good hint. I will try this. But I will also check the mechanical side of the gantry system. If there is something wrong which is causing the issue, i guess its better to fix this

Phaedrux

@Chrashem said in After Firmware update from 3.4.5 to 3.5.2 my printer crashes:

When I my move the printhead by hand it seems that there a some points with a lot of friction.

I'd be looking for a bad bearing, uneven belt tension, or skew in the gantry.

Chrashem

@droftarts @Phaedrux
at the moment I learned a lot about my printer....
...unfortunately on the hard way...

From the movement side now it seems okay. In the mechanic I found a bit of friction caused by skew in the gantry. I could fix it by aligning the gantry. The movements by manually pushing the print head were fluent and smooth. But printing a flat surface showed again a bit of jerking.
So as @droftarts mentioned I raise up the z-speed changes. (Maybe I carried it to far, because I raised also the speed and the acceleration)

M566 X400.00 Y400.00 Z50.00 E300.00 
M203 X10800.00 Y10800.00 Z2000.00 E7200.00
M201 X10000.00 Y10000.00 Z300.00 E10000.00

The first test was fine! I Printed two filled layers and it seems that there is no XY-Jerking anymore and the surface quality was also looking good.

But unfortunately a bit later my printer crashed again...

My printer was homed and I started a simulation of a g-code.
This g-code seems to work because its the test part with the surface, I printed before... I got the message that the simulation was done successfully.
After this I tried to do a bed levelling by G32 and at this moment the printhead didn't move in a xy direction and the z axis was just lifting the bed without stopping against the printhead... Like the crash in my first post here...
Unfortunately the z-speed was very high, but fortunately the printhead was on a position where the bed could tilt, so there is no heavy damage. Just one piece from the z-axis is broken and I can replace it.

It feels like the root of evil is anywhere in the combination of the simulation and the G32....
I thought, that I got maybe a state where the printer isn't homed and no x and y movements are allowed? But in this case I thought that the z axis should stop when the probe is triggering the bed... (and the probe is working).

Is there a possibility to do a kind of a simple snapshot of the OM before and after the simulation?

For more testing I will try to reduce the travel speed during the probing and raise the diving height (just to get generating more time to react):

M558 P5 C"io6.in" H15 F300 T1000

Hopefully I can test more tomorrow, but first I have to replace the broken part of the z-Axis.

An other point which I noticed:
Is it possible, that the connection to DWC is more unstable? I recognised sometimes that the browser is generating a new connection to DWC.
With the old firmware I never got this problem. The position of the printer is the same, my laptop is the same and the router is also the same ( router and printer in the same room, distance just 3m away from each other...)

EDIT:
I could confirm it:
If I using a "G32" after a simulation (it doesn't matter if its a successful simulation or an failed one) my printer isn't moving the print head, but try to lifts the bed up till the crash.
I did several tests to get a better clue about it. After a simulation, DWC is reporting full movable axis. I could use the Buttons to move the axis and the printer is doing it fine. But if I click the bed levelling (G32) the printer is just lift the bed and would crash.
During this it doesn't matters which position my z-axis have. For example: If I have the probing height h set to 25 mm and the actual z position is 5 mm, normally the printer would gain the space to the probing height.
After the Simulation the printer is just lifting the bed...

If I do a homing of all axis after the simulation, and do then a G32 the printer is doing a fine levelling of the bed.
But if I don't do a full homing and do a single homing the behaviour depends on which axis I homed:

Simulation and homing x-axis: Lifting the bed -->crash
Simulation and homing y-axis: Lifting the bed -->crash
Simulation and homing x and y axis: Lifting the bed --> crash
Simulation and homing the z axis --> its just moving the y axis to the probing position (not the x axis) and tries to probe there. In my case the probe was not on the bed so I had to stop the printer manually to prevent a crash
Simulation and homing x y z axis --> G32 worked perfectly

Phaedrux

Can you enable some additional logging on the SBC to share here?

https://github.com/Duet3D/DuetSoftwareFramework/wiki/SBC-Setup-Guide#increasing-log-level

Chrashem

@Phaedrux
For sure!
I set up the logging. What movements or commands should I do with the logging on?

What do you want to see?

For the crash situation I have the problem, that I have to cut the power supply immediately I recognised, that the printer is out of control. I don't know if the logging will work under these conditions.

Chrashem

@Phaedrux @droftarts

At the moment I'm at home and I can run some tests for you with the logging. Can you tell me which movements do you want to see with the additional logging?

How can I share the log? Should I copy the text?

Phaedrux

Can you reliably make the printer crash with certain movements? We just want to see if anything comes up in the logging.

Another thing to test would be to setup in standalone mode without the SBC and see if the issue remains.

Chrashem

@Phaedrux
Today I run the test and I have the logging of it.

2024_08_23_Logging_Error_G32.txt

Just an additional Information:
To be sure that I got no crash but don't have to cut the power supply, I raised the dive height and lowered the feed rate to generate some time to react.

Unfortunately the error is a bit different. Normally the bed crashed against the printhead, but with this configuration I didn't get a crash, but I got a strange feedback:

Here are the steps which I did in the test:

16:02 - Restart the Machine
16:02 - Home all Axis
16:05 - G32 to level the Bed --> Successful -->Leadscrew adjustments made: 0.152 0.166 0.166, points used 3, (mean, deviation) before (0.161, 0.006) after (-0.000, 0.000)
See here (answer is late because the movement took some time in reason of the low feed rate):

Aug 23 16:09:52 Duet3 DuetControlServer[7125]: [debug] HTTP: Finished code G30 P2 X275 Y12 Z-99999 S3 ;probe near a leadscrew and calibrate 3 motors => Leadscrew adjustments made: 0.152 0.166 0.166, points used 3, (mean, deviation) before (0.161, 0.006) after (-0.000, 0.000)

16:10 simulation of a job
16:11 G32 to level the Bed again --> Error happened

Aug 23 16:16:12 Duet3 DuetControlServer[7125]: [debug] HTTP: Finished code G30 P2 X275 Y12 Z-99999 S3 ;probe near a leadscrew and calibrate 3 motors => Error: Some computed corrections exceed configured limit of 5.00mm: -14.332 -14.352 -14.354