My quad carriage machine is failing on Tools. AGAIN.

Phaedrux

Does homeZ by itself work? G28 Z

MaracMB

@Phaedrux yes. But with occasional “glitching” as described.

Phaedrux

One thing I can think of to help narrow down the issue is to change over the motor mappings to have the Z motors on the Duet and move 2 of the carriages over to the Duex. If the problem homing moves to the motors on the Duex then we can be reasonably certain it's a comms issue between the duex and duet.

MaracMB

@Phaedrux this was already done. Having Z motors on duex instead of second and fourth X axis got me to here.

engikeneer

@MaracMB have you managed to isolate whether it is endstop related or motor related?

I'm wondering if you could keep the Z motors on the Duex, but swap the endstops to the Duet? Not sure if there's any firmware/hardware limitations to stop you? I know RRF3 gives you some more freedom...

Equally, have you tried swapping either the motor or endstops to different drivers/pins on the Duex?

Might give some indication of if it is comms or a hardware/board issue.

MaracMB

@engikeneer i suspect it’s endstop related because on a single color print the layers are perfect and motors in question move nicely.

Can’t use just the Duet board for the endstops as there are 7 endstops needed for movement.

I will try moving and remapping the endstops if it’s something board specific
and also swap the PSU, but doubt that will do any difference. Will see.
If that does not work, i’ll revert to v2 firmware and see if that does any good.

Optical enstops is what i like due to accuracy and lack of mechanical clicking. But maybe these Omrons aren’t best choice for duet as they are normally closed and need inverted logic... not sure why it works on everything else.

It is just that i got to 99%, started planing a kit, parts sourcing and suppliers, BOM list for DIY’ers, but having such a glitch is unacceptable and stopped my work.

Troubleshooting this little glitch is taking more time than getting all the way to here.

MaracMB

After almost giving up with this, i reverted to RRF 2.5.1

I suspect that something is wrong with having current versions of RRF3 in duet2 board. While i was unable to find a reason for homing glitches, i did experience slight temperature fluctuations of about +-0.2 C. Almost as there was a noise in the readings. I first suspected suboptimal PID tuning, but the fluctuations were there even at reboot, heaters in idle and at room temperature.

And that actually was the only thing that seemed wrong and gave it away. M122 said nothing.

Many have advised, not a single thing helped.
I have rewired everything 3 times (A LOT OF WIRING), fluctuations persisted on all thermistors.
Having rerouted the cables to boards, not a single thread crossed a ribbon cable between the Duet and Duex. Nothing changed.
I have disconnected everything and only kept VIN power and bed thermistor Still, fluctuation and glitches persisted.
This behaviour (homing twitches and temperature redings noise) was observed on all RRF3 versions.
Not a single variation of config.g did any good.

Then, i reverted to 2.5.1

Temperature readings are now stable, slowest i2c loop is well under 5ms. I have no doubt the occasional homing glitches experienced are also gone. will test this today, but i am very optimistic.

So i guess RRF3 is just not made (well..?) for duet2 boards.

With reverting to RRF2 I will loose conditionals, one axis with two endstops homing, flexibility with connecting stuff, new PID... but i can live with that if the machine at least works as expected and i can print with it.

It is sad that this took so long. it is not that i was the only person using RRF3 with duet2.

Phaedrux

@MaracMB said in My quad carriage machine is failing on Tools. AGAIN.:

I have no doubt the occasional homing glitches experienced are also gone. will test this today, but i am very optimistic.

Please let us know what you find.

dc42

This is a long thread and various things have been changed/reported. I think the problem you are having is that when using RRF3, the axes that use endstop switches connected to the DueX5 don't always home reliably. Is that correct?

I have it on my list to add code re-establish the endstop states on the DueX immediately prior to any homing move. I will try implement this in time for RRF 3.2beta5.

dc42

@MaracMB

Was I correct in my description of the problem in my previous post?
Did reverting to RRF 2.05.1 solve the problem?

dc42

@MaracMB said in My quad carriage machine is failing on Tools. AGAIN.:

@jay_s_uk when i hit homeall, and again until it fails (it's like a 3rd or fourth attempt) it grinds until i hit emergency stop.
Running m122 right after restart gives:

What we really need is a M122 report when it has failed, before you press emergency stop. Can you run it at lower current, so that the "grinding" doesn't do any damage and that you can get this report?

We intend to finalise RRF 3.2RC1 in the next few days, and I would really like to get to the bottom of the issue you are having so that we can implement any necessary fixes first.

MaracMB

@dc42 reverting to 2.5.1 really didn't do much sadly as in the end it didn't compute the endstops (altough registered as triggered) and me using v3 for so long i find v2 just too rigid to stick with it. I have moved ALL the wires away from IDE cable connection and now use the 3.2-beta4 (2020-11-26).

Here's how it stands now:
Machine homes more or less reliably, it does fail to home one axis ( U ) after doing quadruple printing. It always fails to home after print is finnished (G28 X U V W) . That makes it grind like there is no tommorrow on U.
The end code it fails to execute properly is:

G10 P5 S0 R0 		; turn off all heaters
M140 S0 		; turn off bed
T-1		; deselect tools
G28 X U V W		; home all heads
G1 Y140 F2500 		; move Y to front
M84 		; turn all motors off

But i can live with that for now as i can/will/should adjust the routine.

Interestingly, it does not fail to home at start with only "G28". So Yeeey!

I have a long print going on atm, but will send the M122 report tommorrow with a quick quadruple print where the issue happens.

Another issue that is present with RRF3, or at least with my setup, is pretty obvious temperature readings noise. It is most prominent for heaters at idle, but is for sure present also when heaters are on. No matter the PID tuning...

What to look at in the above picture is the jagged temperature curve of Heater 2, 3 and 4.
I didn't experience that with 2.5.1, but as said, i couldn't setup the endstops correctly and i just gave up with that. I need conditionals anyways.

If something is done about endstops in latest RRF3, that would be Fan-Tastic. I guess the temperature readings noise is minor. For now and me at least.

MaracMB

@dc42
OK. So it was PEBCAK
Grinding was caused because after print, U was sent too far before homing. That is settled. Probably.
So, do not stall the release because of me

But let's stay at endstops, because, another thing bothers me is now that machine works, filament runouts are needed.
I went with simple switches.

Config goes like this:

M591 D0 P2 C"duex.e4stop" S1
M591 D1 P2 C"duex.e5stop" S1
M591 D2 P2 C"duex.e6stop" S1
;M591 D3 P2 C"somewhere or duex.e5stop" S1

There are 2 issues i have now.

A. Obviously i need a 4th switch to be plugged into something. But i guess i could go in series and when either triggers, it stops. right?

yet if i go with individual switches, after i setup and connect 3rd switch, duet will shut down and wont turn on.

After immediate turn off and on, PanelDue says:
Error retrieveing WiFi status message: SPI timeout

Or at second attempt:
Error retrieving WiFi
xxxx something something i couldn't snap xxxx transfer is pending xxxx something about busy SPI xxxx

So.. i guess 4 individual runouts are a no go.

Here are M122 reports, if you bother looking at and helps you.
First one is with no runout sensors, just after power on

m122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.2-beta4 running on Duet WiFi 1.02 or later + DueX5
Board ID: 08DGM-9T6BU-FG3SN-6J9F6-3SN6L-KUWVG
Used output buffers: 3 of 24 (24 max)
=== RTOS ===
Static ram: 24108
Dynamic ram: 103812 of which 24 recycled
Never used RAM 2104, free system stack 190 words
Tasks: NETWORK(ready,181) HEAT(blocked,306) DUEX(blocked,35) MAIN(running,459) IDLE(ready,20)
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:02:18 ago, cause: software
Last software reset at 2020-12-10 10:45, reason: User, GCodes spinning, available RAM 2024, slot 2
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f000 BFAR 0xe000ed38 SP 0xffffffff Task MAIN
Error status: 0x04
MCU temperature: min 31.7, current 32.5, max 33.0
Supply voltage: min 23.9, current 24.0, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: position 0, standstill, SG min/max not available
Driver 1: position 0, standstill, SG min/max not available
Driver 2: position 0, standstill, SG min/max not available
Driver 3: position 0, standstill, SG min/max not available
Driver 4: position 0, standstill, SG min/max not available
Driver 5: position 0, standstill, SG min/max not available
Driver 6: position 0, standstill, SG min/max not available
Driver 7: position 0, standstill, SG min/max not available
Driver 8: position 0, standstill, SG min/max not available
Driver 9: position 0, standstill, SG min/max not available
Driver 10: position 0
Driver 11: position 0
Date/time: 2020-12-10 10:49:29
Cache data hit count 220746639
Slowest loop: 7.24ms; fastest: 0.20ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 1.0ms, write time 0.0ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 169, MinFreeDm: 169, MaxWait: 0ms
Bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed moves 0, StepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, StepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 16.04ms; fastest: 0.00ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- WiFi -
Network state is active
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
WiFi firmware version 1.23
WiFi MAC address 84:f3:eb:83:45:0d
WiFi Vcc 3.43, reset reason Power up
WiFi flash size 4194304, free heap 23488
WiFi IP address 164.8.114.139
WiFi signal strength -63dBm, mode none, reconnections 0, sleep mode modem
Clock register ffffffff
Socket states: 0 0 0 0 0 0 0 0				 
=== DueX ===
Read count 1, 0.43 reads/min

This, the second is with 2 runout sensors enabled because, as said, with 3 it wont work anymore:

m122 - restarted and idle
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.2-beta4 running on Duet WiFi 1.02 or later + DueX5
Board ID: 08DGM-9T6BU-FG3SN-6J9F6-3SN6L-KUWVG
Used output buffers: 3 of 24 (24 max)
=== RTOS ===
Static ram: 24108
Dynamic ram: 103892 of which 24 recycled
Never used RAM 2024, free system stack 188 words
Tasks: NETWORK(ready,181) HEAT(blocked,294) DUEX(blocked,35) MAIN(running,451) IDLE(ready,20)
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:03:16 ago, cause: software
Last software reset at 2020-12-10 10:40, reason: User, GCodes spinning, available RAM 1984, slot 1
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f000 BFAR 0xe000ed38 SP 0xffffffff Task MAIN
Error status: 0x04
MCU temperature: min 31.3, current 32.3, max 32.8
Supply voltage: min 23.8, current 24.0, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: position 0, standstill, SG min/max not available
Driver 1: position 0, standstill, SG min/max not available
Driver 2: position 0, standstill, SG min/max not available
Driver 3: position 0, standstill, SG min/max not available
Driver 4: position 0, standstill, SG min/max not available
Driver 5: position 0, standstill, SG min/max not available
Driver 6: position 0, standstill, SG min/max not available
Driver 7: position 0, standstill, SG min/max not available
Driver 8: position 0, standstill, SG min/max not available
Driver 9: position 0, standstill, SG min/max not available
Driver 10: position 0
Driver 11: position 0
Date/time: 2020-12-10 10:43:38
Cache data hit count 315341470
Slowest loop: 17.79ms; fastest: 0.21ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 1.9ms, write time 0.0ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 169, MinFreeDm: 169, MaxWait: 0ms
Bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed moves 0, StepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, StepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 16.10ms; fastest: 0.00ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 2 of 8
- WiFi -
Network state is active
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
WiFi firmware version 1.23
WiFi MAC address 84:f3:eb:83:45:0d
WiFi Vcc 3.43, reset reason Power up
WiFi flash size 4194304, free heap 23456
WiFi IP address 164.8.114.139
WiFi signal strength -62dBm, mode none, reconnections 0, sleep mode modem
Clock register ffffffff
Socket states: 0 0 0 0 0 0 0 0
=== Filament sensors ===
Extruder 0 sensor: ok
Extruder 1 sensor: ok
=== DueX ===
Read count 12, 3.66 reads/min

As seen, loop time and read count increase dramatically with only 2 runout sensors.

And here is M122 after a quick quadruple print that works now. This might tell you and guys something, i just post

m122 - after Quad print run
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.2-beta4 running on Duet WiFi 1.02 or later + DueX5
Board ID: 08DGM-9T6BU-FG3SN-6J9F6-3SN6L-KUWVG
Used output buffers: 3 of 24 (24 max)
=== RTOS ===
Static ram: 24108
Dynamic ram: 103812 of which 60 recycled
Never used RAM 2068, free system stack 114 words
Tasks: NETWORK(ready,181) HEAT(blocked,306) DUEX(blocked,35) MAIN(running,459) IDLE(ready,20)
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:26:33 ago, cause: software
Last software reset at 2020-12-10 10:45, reason: User, GCodes spinning, available RAM 2024, slot 2
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f000 BFAR 0xe000ed38 SP 0xffffffff Task MAIN
Error status: 0x04
MCU temperature: min 32.2, current 36.5, max 37.0
Supply voltage: min 23.7, current 23.9, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: position -13600, standstill, SG min/max 0/227
Driver 1: position 11200, standstill, SG min/max 0/1023
Driver 2: position 832, standstill, SG min/max 81/283
Driver 3: position 13440, standstill, SG min/max 0/175
Driver 4: position -12560, standstill, SG min/max 0/1023
Driver 5: position 12640, standstill, SG min/max 0/232
Driver 6: position 0, standstill, SG min/max 0/233
Driver 7: position 0, standstill, SG min/max 0/1023
Driver 8: position 0, standstill, SG min/max 0/1023
Driver 9: position 0, standstill, SG min/max 0/1023
Driver 10: position 0
Driver 11: position 0
Date/time: 2020-12-10 11:13:44
Cache data hit count 2541600989
Slowest loop: 65.48ms; fastest: 0.17ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 1.7ms, write time 27.5ms, max retries 0
=== Move ===
Hiccups: 1(0), FreeDm: 169, MinFreeDm: 70, MaxWait: 621792ms
Bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 1538, completed moves 1538, StepErrors 0, LaErrors 0, Underruns [0, 0, 7], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, StepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
Heater 0 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 200.56ms; fastest: 0.09ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- WiFi -
Network state is active
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 1
WiFi firmware version 1.23
WiFi MAC address 84:f3:eb:83:45:0d
WiFi Vcc 3.43, reset reason Power up
WiFi flash size 4194304, free heap 23488
WiFi IP address 164.8.114.139
WiFi signal strength -68dBm, mode none, reconnections 0, sleep mode modem
Clock register ffffffff
Socket states: 0 0 0 0 0 0 0 0
=== DueX ===
Read count 40, 1.65 reads/min

Latest config is here, but not sure it matters. I'll nag about having extra steppers on PT100 pins in a new thread and mark this one as solved.

dc42

The only thing I spotted from your M122 reports is the that amount of free memory is quite low. To tell whether running out of memory is the issue, I would need a M122 report after 3 runout sensors have been enabled and it has shut down and restarted. You could send the command to enable the 3rd sensor manually, so that when it reboots after shutting down, it won't shut down again.

RRF 3.2RC1 (due out in a few days) has more free memory.

MaracMB

@dc42 yeah i'd send you that but it just fails to start so, no can do. all sensors to one pin and it will be ok i guess. I'll need the others for 5th and maybe even 6th X axis anyways.

dc42

@MaracMB said in My quad carriage machine is failing on Tools. AGAIN.:

dc42 yeah i'd send you that but it just fails to start so, no can do

That's why I suggested sending the command to configure the 3rd sensor manually, e.g. from the DWC console.