Z-axis / tramming issues with 3.6.0-alpha2+3

Exerqtor

Moving this over to a separate topic.

System info:
Duet Mini 5+ v1.02 ( 3.6.0-a2+), Duet 1LC v1.2a (3.5.2), standalone, 24v, CoreXY, 3-point bed leveling with individual Z steppers.

Copied from the topic above to catch up:

I've been doing roughly 24hours of printing on 3.6.0-a2+3 now without any noticable issues, but when I'm about to start print this morning (a print finished in the middle of the night, and the printer has just stood idle since that) it's acting up/crashing into the bed.

(The printer is a Voron Triden by the way.)

I start a print from the office and wait for the printer to start homing, hear it homing XY and when it goes of to home Z I hear steppers just turning and turning, so I go out and check on the printer. And it turns out that the rear-center Z stepper is standing still(with the carriage roughtly about where it should be), while both the front Z steppers are still going in a downwards motion about halfway along the Z-axis (I have "WobbleX" on the z-axis).

I sprint across the room and hit the E-stop and start troubleshooting, first checking out DWC and what it has to say:

26.7.2024, 09:58:32	Emergency Stop! Reset the controller to continue.
                        Error: Failed to home axes Z
26.7.2024, 09:58:32	Emergency stop, attemping to reconnect...
26.7.2024, 09:58:12	G28 Z
                        Warning: Driver 0.2 warning: phase A may be disconnected, phase B may be disconnected
26.7.2024, 09:55:59	Error: G10: Probe was not triggered during probing move
                        Cancelled printing file 0:/gcodes/Automower/Rim_automower_305_0.2mm_ABS_0.4n_2h14m.gcode, print time was 0h 16m
                        Error: in file macro line 114: G1: insufficient axes homed
26.7.2024, 09:53:33	Warning: Driver 0.2 warning: phase A may be disconnected, phase B may be disconnected
26.7.2024, 09:51:07	Warning: Driver 0.2 warning: phase A may be disconnected, phase B may be disconnected
26.7.2024, 09:49:20	Warning: Driver 0.2 warning: phase A may be disconnected, phase B may be disconnected
26.7.2024, 09:39:45	M32 "0:/gcodes/Automower/Rim_automower_305_0.2mm_ABS_0.4n_2h14m.gcode"
                        File 0:/gcodes/Automower/Rim_automower_305_0.2mm_ABS_0.4n_2h14m.gcode selected for printing

This all sounds somewhat odd since Driver 0.2 is the one for the rear-center stepper, which don't seem to be doing anything weird.

Here is a vizual representation of my drive mapping for reference:

; (While looking at the printer top down)
;             0.0-----0.1
;              |  0.2  |
;              |-------|
;              |0.3|0.4|
;               ---+---
;                Front

(0.0 & 0.1 is for the CoreXY motion system)

I continue troubleshooting:

I power down the printer and manually reset the two front axis & lock the bed. Then I flip the machine around to check the wiring which all looks good.
Put the machine back on it's feet, power up, remove the bed locks and home the printer: all seems fine. Ok? So I jog the printer through full Z motion min-max-min with Paneldue and all seems fine.
Not being able to find anything obviously borked I bring the bed up to heat again and attempt to run a tram routine. Tramming routine goes as followng:
1. Home all axis.
1. Coarse tramming pass with 10mm dive height.
1. Fine tramming with 2mm dive height until the deviation is bellow 0.005mm).
It runs the coarse pass, and ATTEMPTS to adjust the 3 z-axis:

2024-07-26 10:15:06 [warn] Leadscrew adjustments made: -2.315 2.178 1.401, points used 3, (mean, deviation) before (0.660, 1.775) after (-0.000, 0.000)

It proceeds to the fine pass @ 2mm dive height, but for some reason the z-axis steppers didn't do like they should on the course pass so it ends up scraping the nozzle allong the buildplate rather than hover 2mm above it (I've got Voron TAP, so it's not the biggest issue in the world with a slight "nozzle kiss" thankfully).
Again I have no other choice than to slap the E-stop to avoid anything getting destroyed.
I check the Z-stepper wiring once again since the steppers didn't move like expected, butI still can't find anything wrong.
Recheck I still got full range of motion on the Z axis, which I have: hence rulling out phase/connection issues to the Z-steppers (as far as I can understand).
Try once more, and the exact same happens, and that's where I'm at now.

Like I opened up with saying, I've done multible trams (restarted the printer a couple times and it trams the bed first print after each reboot), without issues during the roughly 24hours of print time I have on it with 3.6.0-a2+2.
And that this happens with the printer standing idle after a print has finished is somewhat weird (roughly 3.5 hours between the night print finished and i tried starting a new one this monring).

Status update:

I just had time to roll back too 3.5.2 to check if fixes the problem, which it did. So it looks like it's something going on somewhere in regard to Z axis controll in 3.6.0-a2+3 @dc42!

I haven't got anything else than eventlogging going on at the time so i don't know if it's much if any usefull info one can pick up from that. If you think it might i can supply the entire log from the last couple days.

dc42

@Exerqtor thanks for your report. Does this issue only occur when the printer has been standing idle for a while after the previous print, or does it also occur if you start another print immediately?

Exerqtor

@dc42 To be honest I haven't had time to test it any more since I had to get some prints going.

But the printer had been idle for a similar or longer amounts of time after the update without this happening earlier. Hence why I got so baffeled when it suddenly happened here to other day.

I can scrape through the event log and look at the relevant dates and times between the update and every consequtive print + how many and when trams it's successfully perfromed when I get the time.

I don't know how much value (if any) the eventlog history is for you @dc42. But I've sifted through it and removed all the fluff + added some comments on the different events to give some more reference to what I've been describing in the post.

>> Here is the eventlog <<

DonStauffer

@Exerqtor One of my Z steppers now adjusts in the wrong direction for G32. It's not that it has reversed direction in general, because other things work. But I can't level the bed. Every time it finds that point to be off, it moves it in the other direction and then the next time around it's off by twice as much. Kind of a problem.

Update: I've been trying to mess with it to get it level again and now my head has also hit the bed. This is a Railcore with 3 Z steppers, and #7 seem to be randomly reversing directions or not moving with the others. It's gotten to the point where I can't even do a home.

I can't really revert to 3.5 because the memory issues will keep me from working, and I can't use this version because I can't manage the bed because probing doesn't work right. So I'm completely down right now until a new version comes out.

Update 2: Now I'm getting Warning: Driver 7 warning: phase B may be disconnected

Exerqtor

@DonStauffer That's the exact same behaviour I was experiencing. Glad to hear it's not just me!

DonStauffer

@Exerqtor Doing an iterative G32:

Here's some data. I turned on the machine, then used my "single Z stepper" macro to adjust the best I could by eye. Just to be sure, I did an M999 next. Then I homed and did G32. My Z steppers are 5, 6, and 7. The console log:

G32

Leadscrew adjustments made: 0.771 -0.973 0.823, points used 3, (mean, deviation) before (0.064, 0.712) after (0.000, 0.000)

Leadscrew adjustments made: 0.010 -1.957 -0.017, points used 3, (mean, deviation) before (-0.811, 0.787) after (-0.000, 0.000)

Here you can see the adjustments to #5 and #7 are not bad, but obviously #6 got much worse, and fact it almost exactly doubled. So this stepper was adjusted in the wrong direction. But the stepper isn't reversed all the time. Homing works fine, and my "single z stepper" macro works fine. Also odd is that the first I noticed it, it was #7 that was doing this and #6 was fine. This without me changing anything (such as wiring or M569 in config.g). It's as if which stepper goes backwards just for leveling depends on the time of day.

When I started getting these messages, I emergency stopped. Once you start getting these with this problem, you just keep getting them:

Error: M280: Probe was not triggered during probing move
Error: Compensation or calibration cancelled due to probing errors

This is new behavior. To get the first error, previously something had go wrong that would happen to keep the probe from triggering. But it's triggering fine, because the carriage immediately goes back up with the dive height, just as with a normal probe. So my iterative leveling in bed.g is now just going around and around, giving these errors every time, and making no adjustments.

I notice you've gotten some of the "Warning: phase B may be disconnected" messages which usually mean a connector or wire problem. But checked all pins with a meter and they were all good. Not sure if this was because there was an intermittent plug issue that went away from pulling the plugs out to test, but it seems odd because I hadn't moved anything, and with you reporting the same thing...

gloomyandy

@DonStauffer Can I suggest that for now you disable as many (ideally all) of the fancy macros and test just the basics, that would hopefully allow you to switch between 3.6 and 3.5 so we can establish if the problems you are seeing are related to the new firmware or not (please remember this is an alpha release and may have problems).

o_lampe

@DonStauffer @Exerqtor
I was wondering if the motor that acts up during bed tramming is set to a different rotation direction? M586 Px S1 while the other two motors are M586 Px S0

DonStauffer

@o_lampe It varies. It's not just one motor.

DonStauffer

@gloomyandy I could, but hasn't Exerqtor already confirmed that? I think I saw a post where he said he reverted and the problem went away. This didn't happen on 3.5.2 ~2 or 3 weeks ago when I leveled the bed hundreds of times.

I understand about alpha. I was stuck because I couldn't print but I found a way for now that works OK. It's a pain though.

What I have done is set my bed.g for one iteration and used the S-1 parameter for the last G30, so it only prints the adjustments that are needed. Then I've been manually adjusting the various Z motors according to that. Then I re-home and G32 again. It took me a couple hours, but the bed is level and the printer is printing now. So it's clearly the G30 S3 that's wrongly adjusting the motors some of the time.

gloomyandy

@DonStauffer The more data points we can gather the faster we will be able to identify and hopefully fix the problems. From reading the reports it seems that @Exerqtor was able to tram his bed several times using 3.6 without any issues before then running into problems. I'm not sure how many times it worked after reverting to 3.5 but clearly for Exerqtor it is not a simple case of 3.5 works and 3.6 does not.

@all it might be useful to understand what direction the stepper that has problems is trying to move in (and perhaps what direction it moved in last), so for instance does the error only occur with a stepper that is trying to move up (or down) and did it last move in the same or a different direction.

@DonStauffer when you manually adjust the steppers to level the bed what commands are you actually using and do they work correctly?

Exerqtor

@gloomyandy Yeah it worked 2-3 times (at least) on 3.6 for me before acting up all off a sudden. On 3.5 it's running stable without issues (and it has been since release).

I'll smack together a simplified bed.g, verify it working stabily with 3.5 by doing 20-30 back too back G32's. Then upgrade too 3.6, enable debug logging and do the same there, and report back here with how it went.

gloomyandy

@Exerqtor If you can please make a note of the stepper that has a problem and the direction that it has been asked to move in.

edsped

Add another to the list of those with tramming issues. Just trashed my build sheet. I've never had an issue with the printer before so I'm pretty sure it's related to the 3.6 A... Reverted back to 3.5.2 and am re printing now. There's definitely a difference in tramming between the two...

Tramming with 3.6... This is the print that trashed the sheet...
Screenshot 2024-08-03 100058.jpg

Tramming with 3.5.2 After crash...
Screenshot 2024-08-03 101207.jpg

Damage...

Exerqtor

@gloomyandy said in Z-axis / tramming issues with 3.6.0-alpha2+3:

@Exerqtor If you can please make a note of the stepper that has a problem and the direction that it has been asked to move in.

I'm 99.9% sure the what happened in my case (when it was fucking up the adjustments in the coarse tramming pass) is that the "front right" axis (0.4 in my case) went up rather than down. But I might be able to verify this when I get time to run the macro I make for debug later.

@edsped said in Z-axis / tramming issues with 3.6.0-alpha2+3:

Add another to the list of those with tramming issues. Just trashed my build sheet. I've never had an issue with the printer before so I'm pretty sure it's related to the 3.6 A... Reverted back to 3.5.2 and am re printing now. There's definitely a difference in tramming between the two...

Ouch that sucks! Hope your hotend/nozzle didn't take any damage 😢

As a baseline reference this is my standard set off tramming macros (if I forgot to add a sub-macro let me know and I'll supply that as well.

bed.g
bed_probe_points

The homing macro's haven't been acting up so I didn't bother adding those.

I'll upload the simplified/debug `bed.g` once I get time to write it!

edsped

@Exerqtor

Nozzle, gammamaster, appears to be okay, No first layer or offset issues so I'm guessing the rapido is okay but I wouldn't be surprised if the heatbreak isn't a little tweaked. I've got thousands of toolchanges and the same with print starts and this is the first time, other than the initial commissioning of the printer that I had something like this happen. I'm definitely bummed but knew what I was getting into when I loaded an alpha firmware. Coarse tramming was definitely not as good on 3.6 as it is on 3.5.2 as can be seen in my screenshots.

edsped

My bed.g is pretty basic but am posting just i case it helps..
bed.g

DonStauffer

@gloomyandy

;-------------------------------------------------------------------------------
;	MoveZStepper Macro
;-------------------------------------------------------------------------------
;	Moves one Z stepper independently, for manual, coarse tramming. Creates
;	(or reuses) an A axis for the purpose, then hides it when done.
;
;	Parameters:
;		N	Stepper number (default: 7)
;		Z	Distance. which may be negative to move up (default: 1)
;		S	Speed (default: 150)
;-------------------------------------------------------------------------------
;	Don Stauffer					July, 2024
;-------------------------------------------------------------------------------
;if global.DebugLevel >= 1200
;	var This = "MoveZStepper"
;	set var.This = var.This^(exists(param.N) ? " N"^param.N : "")
;	set var.This = var.This^(exists(param.Z) ? " Z"^param.Z : "")
;	set var.This = var.This^(exists(param.S) ? " S"^param.S : "")
;	echo var.This
;-------------------------------------------------------------------------------
;	Parameters
;-------------------------------------------------------------------------------
var StepperNum = (exists(param.N) ? param.N : 7)
var Distance = (exists(param.Z) ? param.Z : 1)
var Speed = (exists(param.S) ? param.S : 150)
;-------------------------------------------------------------------------------
;	Echo back what will be done
;-------------------------------------------------------------------------------
echo "Moving Z stepper", var.StepperNum, "by", var.Distance
;-------------------------------------------------------------------------------
;	Create A axis if necessary; make it visible if it already exists
;-------------------------------------------------------------------------------
var DoesAAxisExist = false

while iterations < #move.axes
	if move.axes[iterations].letter != "A"
		continue
	set var.DoesAAxisExist = true
	break

if var.DoesAAxisExist
	M584 P4
M584 A{var.StepperNum}
;-------------------------------------------------------------------------------
;	Configuration adapted from config.g. Not sure if everything is needed here
;-------------------------------------------------------------------------------
M669 K1			    ;	CoreXY mode
M350 A16 I1		    ;	16x microstepping for axes & extruder, w/interpolation
M574 A1 S2			;	Set Endstop configuration (P"pin_name"); S2 = Z probe
M906 A1500 I60		;	Set motor currents (mA)
M203 A{var.Speed}	;	Maximum speeds (mm/min)

M566 A12			;	Jerk
M201 A90			;	Max acceleration

M208 A635			;	Set axis maxima & high homing switch positions
M208 A-0.5 S1		;	Set axis minima & low homing switch positions

M92 A3200			;	Steps/mm
;-------------------------------------------------------------------------------
;	Move the Z stepper as the "A axis"
;-------------------------------------------------------------------------------
G91
G1 A{var.Distance} H2
;-------------------------------------------------------------------------------
;	Reset everything back to the same as config.g
;-------------------------------------------------------------------------------
M584 X0 Y1 Z5:6:7	;	Map Z to drivers 5, 6, 7.

M669 K1			    ;	CoreXY mode
M350 Z16 I1		    ;	16x microstepping for axes & extruder, w/interpolation
M574 Z1 S2			;	Set Endstop configuration (P"pin_name"); S2 = Z probe
M906 Z1500 I60		;	Set motor currents (mA)M203 Z600
M203 Z600			;	Maximum speeds (mm/min)

M566 Z12			;	Jerk
M201 Z90			;	Max acceleration

M208 Z635			;   set axis maxima & high homing switch positions
M208 Z-0.5 S1		;   set axis minima & low homing switch positions

M92 Z3200			;   Steps/mm
;-------------------------------------------------------------------------------
;	Hide A-axis by setting number of visible axes back to 3
;-------------------------------------------------------------------------------
M584 P3
;-------------------------------------------------------------------------------
;if global.DebugLevel >= 1200
;	echo "Leaving MoveZStepper"
;-------------------------------------------------------------------------------

DonStauffer

@gloomyandy If you tell me the procedure I'll revert and then test it.

It's possible that the direction problem is that when a negative adjustment is needed, it does a positive adjustment. That does happen. I don't know if it happens the other way around, but I have seen this.

dc42

@DonStauffer @Exerqtor @edsped thanks for reporting this. I'm sorry for any damage caused.

I don't have a system with multiple Z motors that I can test on, but I think I know what is causing this. It's to do with getting the motor direction wrong when switching between normal Z moves and leadscrew adjustment moves. Please can you identify for me which of the following is happening:

When the first bed tramming move is executed, some of the motors move in the wrong direction.
Immediately after the first bed tramming move is executed, when executing a normal Z move (which could be part of a new G32 probing sequence), some of the the Z motors move in the wrong direction.
Both of the above.

If the main problem is #2 then a workaround for now might be to insert M400 after the G32 call, to ensure motion stops before another Z move is executed.