Intermittent communication disruption between 6HC and 3HC

[deckingman]

@pkos said in Intermittent communication disruption between 6HC and 3HC:

Small update. I tried playing with the numbers a bit. G4 S25 is needed to make it work.

Is that number supposed to be this high?

I feel like this long of a delay appeared in 3.2. I'll see about downgrading to 3.1.1 and checking (especially since I am having issues with PanelDue updating temperatures - as in, if I set it to heat up and wait - it won't update - but that I'll take a look at later. ).

Nope. If you need to delay things by 25 seconds, something is very wrong indeed.

[pkos]

I switched back to 3.1.1 for the 6HC and 3.1.0 for the 3HC (since that's what was included in the single package).

The 3HC boots up instantly together with the 6HC every time - and I no longer need the G4.

It also solved a problem for me with DHT22 readings, which based on reports doesn't work on the 6HC, but does on Duet 2s.

There is a problem with the 3.1.0 though, where the 3HC will stop responding if it heats up, so I am stuck

[dc42]

@pkos said in Intermittent communication disruption between 6HC and 3HC:

Small update. I tried playing with the numbers a bit. G4 S25 is needed to make it work.

No. G4 S1 should be more than enough.

When you first apply power, does the 3HC acquire CAN sync (as shown by the LEDs blinking in sync) within a second or two?

[pkos]

@dc42 On 3.1.1 - yes.
On 3.2 - no - it takes about 25-30 seconds for the CAN to wake up and start flashing the LED on the 3HC. On 6HC the CAN led flashes almost immediately.

[dc42]

Which bootloader version do you have installed in the 3HC? M122 for the 3HC will report it. [Do not try to install a new bootloader.]

There is a problem with the 3.1.0 though, where the 3HC will stop responding if it heats up, so I am stuck

When it stops responding, does the LED on the 3HC continue to flash?

[pkos]

@dc42 said in Intermittent communication disruption between 6HC and 3HC:

Which bootloader version do you have installed in the 3HC? M122 for the 3HC will report it. [Do not try to install a new bootloader.]

On 3.2:
Bootloader ID - not available.

3.1.0 doesn't give that info. Just in case, below is the log I took just now on 3.1.0.

There is a problem with the 3.1.0 though, where the 3HC will stop responding if it heats up, so I am stuck

When it stops responding, does the LED on the 3HC continue to flash?
I'll have to double check that, but just in case this helps - here is a thread, where we spoke about this problem and where I reported a link to firmware that solved my problem back then.
https://forum.duet3d.com/post/167660

One more question, since we are talking

In the logs in the first post here, I see the 3HC reporting an AssertionFailed as reason for last software reset. Might that also point to a problem?

[dc42]

I think the slow statup is another instance of this issue https://forum.duet3d.com/topic/21224/can-connectivity-duet-3-mb6hc-to-exp3hc. Please try the beta firmware at https://www.dropbox.com/sh/wme9k0z86sytg33/AAAT6wrHp2eeJHK-dYoW1Um4a?dl=0.

The assertion failure appears to have been caused by a stack overflow. I will examine the stack trace.

[pkos]

@dc42 Thank you. I'll try it out and will report here.

[pkos]

@dc42

Here are the results of what I found after trying the 3.3 beta.

Quick summary first, then a bit more data (and diag logs at the bottom).

1. IMPROVEMENT: 3.3 beta solves the 3HC not initializing the CAN sync process on power up (I assume that's why the LED would only start flashing after about 25-30 seconds on 3.2).

2. REGRESSION: 3.3 beta does NOT solve sync issues on power up between 3HC and 6HC until a power cycle is performed (in 3.2, emergency stop would be enough to get sync).

3. NO CHANGE FROM 3.2: DHT22 still does not work, but this time I observed a something weird. In general - on 3.3 and 3.2 DWC still shows 2000C and 2000% (is fine on 3.1.1).

4. NO CHANGE FROM 3.2: Bootloader ID still shows not available

5. NO CHANGE FROM 3.2: Last software reset is still set to AssertionFault (although I did not expect any changes here).

Here's more info:

I started with uploading the firmware to 3HC.

The CAN LED now starts flashing immediately on power up and flashes quite fast (faster than on 3.1.0).

I then updated the 6HC to 3.3.

The 3HC does NOT get connected to the 6HC on first power on. 6HC flashes slowly, 3HC flashes quickly. They didn't sync after 30 minutes of waiting. I then hit the Emergency Stop button - in 3.2 - this would solve the lack of communication, but on 3.3 it did not. The boards don't sync up and each flashes differently.
However, after a quick power cycle - the boards immediately sync up on boot.
I am attaching logs that show this below.

After that, everything works except for the DHT22 connected to the main board. That still shows 2000C and 2000% (nothing changed with connections - and these work perfectly fine on 3.1.1).

The very weird thing I observed with the DHT22 on 3.3beta was that I left the printer on for about 10 minutes without touching it. When I cam back to it, I noticed that for the shortest moment, DHT22 readings were available, but the moment i started switching between tabs - readings went back to 2000C/%.

I rolled back to 3.1.1 and immediately the readings came back.

I'm attaching two screenshots of what this looked like (first shows the jump to 2k, then down to normal values, then back up to 2k, where it stayed until I rolled back to 3.1.1 visible on the second screen, where data is updated normally).

Diag logs from cold power up, lack of sync on emergency stop, but immediate sync on power cycle.

1/27/2021, 7:27:16 PM 	m122 b2
Diagnostics for board 2:
Duet EXP3HC firmware version 3.3beta (2021-01-26 20:03:04)
Bootloader ID: not available
Never used RAM 155180, free system stack 0 words
Move 160 HEAT 103 CanAsync 72 CanRecv 84 CanClock 74 TMC 64 MAIN 263 AIN 260
Last reset 00:00:12 ago, cause: software
Last software reset time unknown, reason: AssertionFailed, available RAM 173348, slot 0
Software reset code 0x0120 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0445f85f BFAR 0xe000ed38 SP 0x2002ff8c Task MAIN Freestk 4294967295 ok
Stack: 000002f2 0002a9b0 00020f41 00000000 0002256d 20002ec8 20003100 00000031 00000001 20002980 03800209 0001a72d 0001a709 00000000 00000000 ffffffff 0001de4b 200022d8 20002330 00029c48 ffffffed 00000000 00f00000 e000ef34 c0000000 200041dc 00020ea5
Driver 0: position 0, 410.0 steps/mm,  standstill, reads 28616, writes 16 timeouts 0, SG min/max 0/0, steps req 0 done 0
Driver 1: position 0, 80.0 steps/mm,  standstill, reads 28623, writes 11 timeouts 0, SG min/max 0/0, steps req 0 done 0
Driver 2: position 0, 80.0 steps/mm,  standstill, reads 28627, writes 11 timeouts 0, SG min/max 0/0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter 9, peak Rx sync delay 178, resyncs 0, no step interrupt scheduled
VIN: 24.1V, V12: 12.1V
MCU temperature: min 45.9C, current 45.9C, max 45.9C
Ticks since heat task active 45, ADC conversions started 12536, completed 12536, timed out 0
Last sensors broadcast 0x00000000 found 0 49 ticks ago, loop time 0
CAN messages queued 90, send timeouts 0, received 133, lost 0, free buffers 36, min 36, error reg 100026
dup 0, oos 0, bm 0, wbm 0
1/27/2021, 7:26:58 PM 	m122 b2
Error: M122: Response timeout: CAN addr 2, req type 6024, RID=11
1/27/2021, 7:26:53 PM 	Connection established
1/27/2021, 7:26:45 PM 	Connection interrupted, attempting to reconnect...
1/27/2021, 7:26:36 PM 	Emergency stop, attemping to reconnect...
1/27/2021, 7:24:16 PM 	m122 b2
Error: M122: Response timeout: CAN addr 2, req type 6024, RID=11
1/27/2021, 7:23:54 PM 	Connection established
1/27/2021, 7:23:46 PM 	Connection interrupted, attempting to reconnect...
1/27/2021, 7:23:37 PM 	Emergency stop, attemping to reconnect...
1/27/2021, 7:23:35 PM 	m122 b2
Error: M122: Response timeout: CAN addr 2, req type 6024, RID=12
1/27/2021, 7:23:26 PM 	m122 b2
Error: M122: Response timeout: CAN addr 2, req type 6024, RID=11
1/27/2021, 7:22:06 PM 	Connection established

Diag log from the 6HC:

m122 b0
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.3beta running on Duet 3 MB6HC v0.6 or 1.0 (standalone mode)
Board ID: 08DJM-956L2-G43S4-6JKDA-3SJ6T-1B6GH
Used output buffers: 3 of 40 (21 max)
=== RTOS ===
Static ram: 149784
Dynamic ram: 91668 of which 40 recycled
Never used RAM 109180, free system stack 182 words
Tasks: NETWORK(ready,270) ETHERNET(blocked,117) SENSORS(blocked,53) HEAT(blocked,299) CanReceiv(blocked,893) CanSender(blocked,365) CanClock(blocked,328) TMC(blocked,49) MAIN(running,922) IDLE(ready,20)
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 00:01:45 ago, cause: power up
Last software reset at 2021-01-27 19:26, reason: User, GCodes spinning, available RAM 109424, slot 1
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0044a000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
Aux0 errors 0,0,0
Aux1 errors 0,0,0
MCU temperature: min 36.4, current 41.0, max 41.1
Supply voltage: min 24.0, current 24.0, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.1, current 12.1, max 12.2, under voltage events: 0
Driver 0: position 0, standstill, reads 63168, writes 14 timeouts 0, SG min/max 0/0
Driver 1: position 0, standstill, reads 63168, writes 14 timeouts 0, SG min/max 0/0
Driver 2: position 0, standstill, reads 63168, writes 14 timeouts 0, SG min/max 0/0
Driver 3: position 0, standstill, reads 63169, writes 14 timeouts 0, SG min/max 0/0
Driver 4: position 0, standstill, reads 63169, writes 14 timeouts 0, SG min/max 0/0
Driver 5: position 0, standstill, reads 63169, writes 14 timeouts 0, SG min/max 0/0
Date/time: 2021-01-27 19:28:48
Slowest loop: 41.50ms; fastest: 0.07ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 3.3ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is ready with "m122 b0" in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== CAN ===
Messages queued 441, send timeouts 0, received 465, lost 0, longest wait 20ms for reply type 6024, peak Tx sync delay 6, free buffers 48 (min 47)
=== Network ===
Slowest loop: 2.70ms; fastest: 0.02ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions
HTTP sessions: 2 of 8
- Ethernet -
State: active
Error counts: 0 0 1 0 0
Socket states: 5 2 2 2 2 0 0 0

[dc42]

Thanks for the update. Your M122 trace for the EXP3HC indicated a stack overflow, so I've increased the stack size and put a new 3.3beta Duet3Firmware-EXP3HC.bin at https://www.dropbox.com/sh/wme9k0z86sytg33/AAAT6wrHp2eeJHK-dYoW1Um4a?dl=0. Please try it.

[pkos]

Will do and report later Thanks for the help

[pkos]

Small update for now, I'll write more details later. I need to get a couple prints out fast, so for now I'll switch to 3.1.1.

On the newest beta - connection is quick, no reboots necessary, seems like the initial bit is solved.

However, with that beta - I am unable to complete any print as at some point, the 3HC (which for me runs the extruder, hotend, hotend cooling and filament monitor) probably stops communicating with the 6HC and I get skipped steps, underextrusion and in general - failed prints.

Switching back to 3.1.1 makes it all run.

As soon as I get the next things printed, I'll start investigating more, including diag logs from immediately after a print fails.

[dc42]

@pkos said in Intermittent communication disruption between 6HC and 3HC:

Small update for now, I'll write more details later. I need to get a couple prints out fast, so for now I'll switch to 3.1.1.

On the newest beta - connection is quick, no reboots necessary, seems like the initial bit is solved.

However, with that beta - I am unable to complete any print as at some point, the 3HC (which for me runs the extruder, hotend, hotend cooling and filament monitor) probably stops communicating with the 6HC and I get skipped steps, underextrusion and in general - failed prints.

Switching back to 3.1.1 makes it all run.

As soon as I get the next things printed, I'll start investigating more, including diag logs from immediately after a print fails.

Thanks for the update. I am sorry we haven't managed to resolve this issue completely yet.

When you get a chance, I would appreciate it if you can do the following:

• Install the latest beta firmware from https://www.dropbox.com/sh/qr98k8fbkj5ue0k/AABPawUF99QVzDrheBQBDSxia?dl=0
• Run a print up to the point at which it starts failing (assuming it still fails)
• Pause the print
• Run M122 and M122 B# (where # is the expansion board address) and post the results
• Resume the print, and see whether doing the pause has fixed the issue, at least temporarily.

Thanks for your patience.

[pkos]

I'm almost done printing the last order, so I'll have more time to experiment tomorrow. I'll report afterwards.

Thanks for being awesome and helping out!

[pkos]

Quick update.

I'm 5 hours into the print, so far the print has not failed, no skipped steps or underextrusion, etc. The 3HC starts flashing quickly immediately, catches sync within 2-3 seconds.

There is one more test I need to do - if all goes well, I'll have it done tomorrow morning - that's to check how the 3HC behaves after a long power off (at least one hour). For now I've had the printer running pretty much 24/7 since Saturday.

I ran a quick M122 just in case you want to see what's going on right now (pasting below). The assertion failure is still there and DHT22 does not work.

Diagnostics for board 2:
Duet EXP3HC firmware version 3.3beta (2021-02-01 22:29:11)
Bootloader ID: not available
Never used RAM 154972, free system stack 0 words
Move 80 HEAT 78 CanAsync 72 CanRecv 82 CanClock 74 TMC 30 MAIN 263 AIN 260
Last reset 12:15:33 ago, cause: software
Last software reset time unknown, reason: AssertionFailed, available RAM 173348, slot 0
Software reset code 0x0120 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0445f85f BFAR 0xe000ed38 SP 0x2002ff8c Task MAIN Freestk 4294967295 ok
Stack: 000002f2 0002a9b0 00020f41 00000000 0002256d 20002ec8 20003100 00000031 00000001 20002980 03800209 0001a72d 0001a709 00000000 00000000 ffffffff 0001de4b 200022d8 20002330 00029c48 ffffffed 00000000 00f00000 e000ef34 c0000000 200041dc 00020ea5
Driver 0: position 19481636, 410.0 steps/mm,  ok, reads 39334, writes 0 timeouts 0, SG min/max 0/83, steps req 354599 done 354486
Driver 1: position 0, 80.0 steps/mm,  standstill, reads 39334, writes 0 timeouts 0, SG min/max not available, steps req 0 done 0
Driver 2: position 0, 80.0 steps/mm,  standstill, reads 39334, writes 0 timeouts 0, SG min/max not available, steps req 0 done 0
Moves scheduled 966506, completed 966505, in progress 1, hiccups 0, step errors 0, maxPrep 59, maxOverdue 3, maxInc 3, mcErrs 0, gcmErrs 0
Peak sync jitter 10, peak Rx sync delay 177, resyncs 0, next step interrupt due in 1138 ticks, enabled
VIN: 24.1V, V12: 12.1V
MCU temperature: min 45.7C, current 45.9C, max 46.1C
Ticks since heat task active 16, ADC conversions started 44133758, completed 44133757, timed out 0
Last sensors broadcast 0x00000000 found 0 22 ticks ago, loop time 0
CAN messages queued 1648, send timeouts 0, received 11585, lost 0, free buffers 36, min 36, error reg 0
dup 0, oos 0, bm 0, wbm 0

=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.3beta running on Duet 3 MB6HC v0.6 or 1.0 (standalone mode)
Board ID: 08DJM-956L2-G43S4-6JKDA-3SJ6T-1B6GH
Used output buffers: 1 of 40 (40 max)
=== RTOS ===
Static ram: 149800
Dynamic ram: 92100 of which 72 recycled
Never used RAM 108700, free system stack 118 words
Tasks: NETWORK(ready,228) ETHERNET(blocked,117) SENSORS(blocked,15) HEAT(blocked,280) CanReceiv(blocked,877) CanSender(blocked,337) CanClock(blocked,326) TMC(blocked,16) MAIN(running,616) IDLE(ready,20)
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 12:15:29 ago, cause: software
Last software reset at 2021-02-02 09:54, reason: User, GCodes spinning, available RAM 108908, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x04
Aux0 errors 331,331,331
Aux1 errors 0,0,0
MCU temperature: min 43.7, current 45.2, max 52.1
Supply voltage: min 23.9, current 24.0, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.0, current 12.1, max 12.2, under voltage events: 0
Driver 0: position 27530, ok, reads 12882, writes 38 timeouts 0, SG min/max 0/1023
Driver 1: position 2814, ok, reads 12882, writes 38 timeouts 0, SG min/max 0/1023
Driver 2: position 2725, ok, reads 12882, writes 38 timeouts 0, SG min/max 0/1023
Driver 3: position 0, ok, reads 12882, writes 38 timeouts 0, SG min/max 0/1023
Driver 4: position 0, ok, reads 12883, writes 38 timeouts 0, SG min/max 0/1023
Driver 5: position 0, ok, reads 12883, writes 38 timeouts 0, SG min/max 0/1023
Date/time: 2021-02-02 22:09:51
Slowest loop: 211.65ms; fastest: 0.04ms
=== Storage ===
Free file entries: 9
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 4.0ms, write time 152.3ms, max retries 0
=== Move ===
DMs created 125, maxWait 514613ms, bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves 458647, completed moves 458612, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 15], CDDA state 3
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
Heater 0 is on, I-accum = 0.0
Heater 1 is on, I-accum = 0.4
=== GCodes ===
Segments left: 1
Movement lock held by null
HTTP is ready with "m122 b0" in state(s) 0
Telnet is idle in state(s) 0
File is doing "G1 X190.481 Y213.861 E0.00113" in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== CAN ===
Messages queued 1143678, send timeouts 0, received 177174, lost 0, longest wait 21ms for reply type 6024, peak Tx sync delay 468, free buffers 48 (min 33)
=== Network ===
Slowest loop: 574.55ms; fastest: 0.02ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- Ethernet -
State: active
Error counts: 0 0 1 0 0
Socket states: 5 2 2 2 2 0 0 0

[pkos]

I'm sorry to be the bearer of bad news, but it's much worse now. I don't know if it's related in any way to the firmware shuffling of late, but now, I can't seem to be able to get a single print off successfully that takes longer than 8 hours.

Almost immediately after my post last night, I noticed that the printer stopped moving and paused on it's own. There were no messages in the console. The printer was just stopped above the print.

I hit resume and the printer seemed to try to move the carriage, but was unable to. It was just stuck in place and jittering around. The display on PanelDue showed significant changes in location, but there was only jittery motion.

I aborted the print and started another, fresh gcode again in hopes that it's just a fluke.

Alas, it was not. A night print that was supposed to take 9 hours froze sometime around the 7th hour (I can't tell for sure).

This time I grabbed diag logs from before I hit resume and after. I'm pasting them below along with my current config.g file.

To be sure - after I hit emergency stop on the panel, everything was moving perfectly fine, so it wasn't something mechanical that would prevent the printer from moving. I did notice that the quality of the print is much worse on the z axis - something I never had an issue with before.

I am now getting very worried that something more serious broke and will have the printer down longer.

Do the logs tell you anything?

Before resume:

m122 b0
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.3beta running on Duet 3 MB6HC v0.6 or 1.0 (standalone mode)
Board ID: 08DJM-956L2-G43S4-6JKDA-3SJ6T-1B6GH
Used output buffers: 1 of 40 (40 max)
=== RTOS ===
Static ram: 149800
Dynamic ram: 92100 of which 72 recycled
Never used RAM 108700, free system stack 118 words
Tasks: NETWORK(ready,228) ETHERNET(blocked,117) SENSORS(blocked,15) HEAT(blocked,274) CanReceiv(blocked,877) CanSender(blocked,337) CanClock(blocked,326) TMC(blocked,16) MAIN(running,616) IDLE(ready,20)
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 21:54:30 ago, cause: software
Last software reset at 2021-02-02 09:54, reason: User, GCodes spinning, available RAM 108908, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x04
Aux0 errors 0,0,0
Aux1 errors 0,0,0
MCU temperature: min 44.1, current 44.7, max 45.4
Supply voltage: min 21.6, current 24.0, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.0, current 12.1, max 12.2, under voltage events: 0
Driver 0: position 17407, standstill, reads 36452, writes 0 timeouts 0, SG min/max 0/1023
Driver 1: position 276, standstill, reads 36452, writes 0 timeouts 0, SG min/max 0/1023
Driver 2: position 8318, standstill, reads 36453, writes 0 timeouts 0, SG min/max 0/1023
Driver 3: position 0, standstill, reads 36453, writes 0 timeouts 0, SG min/max 0/1023
Driver 4: position 0, standstill, reads 36450, writes 2 timeouts 0, SG min/max 0/1023
Driver 5: position 0, standstill, reads 36450, writes 2 timeouts 0, SG min/max 0/1023
Date/time: 2021-02-03 07:48:52
Slowest loop: 49.84ms; fastest: 0.05ms
=== Storage ===
Free file entries: 9
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 3.4ms, write time 0.8ms, max retries 0
=== Move ===
DMs created 125, maxWait 0ms, bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves 935235, completed moves 935235, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 1], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
Heater 0 is on, I-accum = 0.0
Heater 1 is on, I-accum = 0.3
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is ready with "m122 b0" in state(s) 0
Telnet is idle in state(s) 0
File is doing "G1 X110.426 Y107.065 E0.07748 F1134.2" in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== CAN ===
Messages queued 494969, send timeouts 0, received 140596, lost 0, longest wait 21ms for reply type 6024, peak Tx sync delay 468, free buffers 48 (min 42)
=== Network ===
Slowest loop: 10.04ms; fastest: 0.02ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- Ethernet -
State: active
Error counts: 0 0 1 0 0
Socket states: 2 2 2 2 2 0 0 0



m122 b2
Diagnostics for board 2:
Duet EXP3HC firmware version 3.3beta (2021-02-01 22:29:11)
Bootloader ID: not available
Never used RAM 154972, free system stack 0 words
Move 80 HEAT 78 CanAsync 72 CanRecv 82 CanClock 74 TMC 30 MAIN 263 AIN 260
Last reset 21:54:31 ago, cause: software
Last software reset time unknown, reason: AssertionFailed, available RAM 173348, slot 0
Software reset code 0x0120 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0445f85f BFAR 0xe000ed38 SP 0x2002ff8c Task MAIN Freestk 4294967295 ok
Stack: 000002f2 0002a9b0 00020f41 00000000 0002256d 20002ec8 20003100 00000031 00000001 20002980 03800209 0001a72d 0001a709 00000000 00000000 ffffffff 0001de4b 200022d8 20002330 00029c48 ffffffed 00000000 00f00000 e000ef34 c0000000 200041dc 00020ea5
Driver 0: position 26155512, 410.0 steps/mm,  standstill, reads 45397, writes 0 timeouts 0, SG min/max not available, steps req 0 done 0
Driver 1: position 0, 80.0 steps/mm,  standstill, reads 45395, writes 0 timeouts 0, SG min/max not available, steps req 0 done 0
Driver 2: position 0, 80.0 steps/mm,  standstill, reads 45396, writes 0 timeouts 0, SG min/max not available, steps req 0 done 0
Moves scheduled 1320929, completed 1320929, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter 9, peak Rx sync delay 173, resyncs 0, no step interrupt scheduled
VIN: 24.1V, V12: 12.1V
MCU temperature: min 45.7C, current 45.9C, max 46.1C
Ticks since heat task active 184, ADC conversions started 78871426, completed 78871425, timed out 0
Last sensors broadcast 0x00000000 found 0 190 ticks ago, loop time 0
CAN messages queued 423, send timeouts 0, received 885, lost 0, free buffers 36, min 36, error reg 0
dup 0, oos 0, bm 0, wbm 0

After resume (and about 5 seconds of the carriage jittering):

m122 b0
Diagnostics for board 2:
Duet EXP3HC firmware version 3.3beta (2021-02-01 22:29:11)
Bootloader ID: not available
Never used RAM 154972, free system stack 0 words
Move 80 HEAT 78 CanAsync 72 CanRecv 82 CanClock 74 TMC 30 MAIN 263 AIN 260
Last reset 21:56:48 ago, cause: software
Last software reset time unknown, reason: AssertionFailed, available RAM 173348, slot 0
Software reset code 0x0120 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0445f85f BFAR 0xe000ed38 SP 0x2002ff8c Task MAIN Freestk 4294967295 ok
Stack: 000002f2 0002a9b0 00020f41 00000000 0002256d 20002ec8 20003100 00000031 00000001 20002980 03800209 0001a72d 0001a709 00000000 00000000 ffffffff 0001de4b 200022d8 20002330 00029c48 ffffffed 00000000 00f00000 e000ef34 c0000000 200041dc 00020ea5
Driver 0: position 26160116, 410.0 steps/mm,  standstill, reads 46672, writes 0 timeouts 0, SG min/max 0/192, steps req 22610 done 22610
Driver 1: position 0, 80.0 steps/mm,  standstill, reads 46672, writes 0 timeouts 0, SG min/max not available, steps req 0 done 0
Driver 2: position 0, 80.0 steps/mm,  standstill, reads 46671, writes 0 timeouts 0, SG min/max not available, steps req 0 done 0
Moves scheduled 1321401, completed 1321401, in progress 0, hiccups 0, step errors 0, maxPrep 59, maxOverdue 1, maxInc 1, mcErrs 0, gcmErrs 0
Peak sync jitter 10, peak Rx sync delay 173, resyncs 0, no step interrupt scheduled
VIN: 24.1V, V12: 12.1V
MCU temperature: min 45.7C, current 45.9C, max 46.1C
Ticks since heat task active 96, ADC conversions started 79008838, completed 79008837, timed out 0
Last sensors broadcast 0x00000000 found 0 102 ticks ago, loop time 0
CAN messages queued 588, send timeouts 0, received 1722, lost 0, free buffers 36, min 36, error reg 0
dup 0, oos 0, bm 0, wbm 0

=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.3beta running on Duet 3 MB6HC v0.6 or 1.0 (standalone mode)
Board ID: 08DJM-956L2-G43S4-6JKDA-3SJ6T-1B6GH
Used output buffers: 15 of 40 (40 max)
=== RTOS ===
Static ram: 149800
Dynamic ram: 92100 of which 72 recycled
Never used RAM 108700, free system stack 118 words
Tasks: NETWORK(ready,228) ETHERNET(blocked,117) SENSORS(blocked,15) HEAT(blocked,274) CanReceiv(blocked,877) CanSender(blocked,337) CanClock(blocked,326) TMC(blocked,16) MAIN(running,616) IDLE(ready,20)
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 21:56:48 ago, cause: software
Last software reset at 2021-02-02 09:54, reason: User, GCodes spinning, available RAM 108908, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x04
Aux0 errors 0,0,0
Aux1 errors 0,0,0
MCU temperature: min 44.5, current 44.8, max 45.0
Supply voltage: min 23.9, current 24.0, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.1, current 12.1, max 12.2, under voltage events: 0
Driver 0: position 80, standstill, reads 50864, writes 0 timeouts 0, SG min/max 0/990
Driver 1: position 80, standstill, reads 50864, writes 0 timeouts 0, SG min/max 0/1023
Driver 2: position 9190, standstill, reads 50863, writes 0 timeouts 0, SG min/max 0/1023
Driver 3: position 0, standstill, reads 50863, writes 0 timeouts 0, SG min/max 0/1023
Driver 4: position 0, standstill, reads 50863, writes 0 timeouts 0, SG min/max 19/1023
Driver 5: position 0, standstill, reads 50863, writes 0 timeouts 0, SG min/max 16/1023
Date/time: 2021-02-03 07:51:10
Slowest loop: 49.77ms; fastest: 0.04ms
=== Storage ===
Free file entries: 9
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 3.3ms, write time 0.8ms, max retries 0
=== Move ===
DMs created 125, maxWait 0ms, bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves 935861, completed moves 935861, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
Heater 1 is on, I-accum = 0.4
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is ready with "m122 b0" in state(s) 0
Telnet is idle in state(s) 0
File is doing "G1 X91.532 Y86.855 E0.01315" in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== CAN ===
Messages queued 1037, send timeouts 0, received 590, lost 0, longest wait 20ms for reply type 6024, peak Tx sync delay 468, free buffers 48 (min 46)
=== Network ===
Slowest loop: 3.99ms; fastest: 0.02ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions
HTTP sessions: 2 of 8
- Ethernet -
State: active
Error counts: 0 0 1 0 0
Socket states: 2 2 2 2 2 0 0 0


m122 b2
Diagnostics for board 2:
Duet EXP3HC firmware version 3.3beta (2021-02-01 22:29:11)
Bootloader ID: not available
Never used RAM 154972, free system stack 0 words
Move 80 HEAT 78 CanAsync 72 CanRecv 82 CanClock 74 TMC 30 MAIN 263 AIN 260
Last reset 21:56:48 ago, cause: software
Last software reset time unknown, reason: AssertionFailed, available RAM 173348, slot 0
Software reset code 0x0120 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0445f85f BFAR 0xe000ed38 SP 0x2002ff8c Task MAIN Freestk 4294967295 ok
Stack: 000002f2 0002a9b0 00020f41 00000000 0002256d 20002ec8 20003100 00000031 00000001 20002980 03800209 0001a72d 0001a709 00000000 00000000 ffffffff 0001de4b 200022d8 20002330 00029c48 ffffffed 00000000 00f00000 e000ef34 c0000000 200041dc 00020ea5
Driver 0: position 26160116, 410.0 steps/mm,  standstill, reads 46672, writes 0 timeouts 0, SG min/max 0/192, steps req 22610 done 22610
Driver 1: position 0, 80.0 steps/mm,  standstill, reads 46672, writes 0 timeouts 0, SG min/max not available, steps req 0 done 0
Driver 2: position 0, 80.0 steps/mm,  standstill, reads 46671, writes 0 timeouts 0, SG min/max not available, steps req 0 done 0
Moves scheduled 1321401, completed 1321401, in progress 0, hiccups 0, step errors 0, maxPrep 59, maxOverdue 1, maxInc 1, mcErrs 0, gcmErrs 0
Peak sync jitter 10, peak Rx sync delay 173, resyncs 0, no step interrupt scheduled
VIN: 24.1V, V12: 12.1V
MCU temperature: min 45.7C, current 45.9C, max 46.1C
Ticks since heat task active 96, ADC conversions started 79008838, completed 79008837, timed out 0
Last sensors broadcast 0x00000000 found 0 102 ticks ago, loop time 0
CAN messages queued 588, send timeouts 0, received 1722, lost 0, free buffers 36, min 36, error reg 0
dup 0, oos 0, bm 0, wbm 0

; Configuration file for Duet 3 (firmware version 3)
; executed by the firmware on start-up
;
; generated by RepRapFirmware Configuration Tool v3.1.3 on Sun Jul 12 2020 20:53:52 GMT+0200 (Central European Summer Time)

; General preferences
G90                                            	    ; send absolute coordinates...
M83                                                 ; ...but relative extruder moves
M550 P"Voron"                                   	; set printer name
M669 K1                                            	; select CoreXY mode

; Network
M552 P0.0.0.0 S1                                   	; enable network and acquire dynamic address via DHCP
M586 P0 S1                                          ; enable HTTP
M586 P1 S0                                         	; disable FTP
M586 P2 S0                                         	; disable Telnet

; Drives for XY
M569 P0.4 S1                                       	; A motor 
M569 P0.5 S1                                       	; B motor

; Drives for Z
M569 P0.0 S0                                       	; Front left 
M569 P0.1 S1                                       	; Back left
M569 P0.2 S0                                       	; Back right
M569 P0.3 S1                                       	; Front right

; Wait before addressing the 3HC
; G4 S25

; Drive for extruder 
M569 P2.0 S1                                       	; physical drive 1.0 goes forwards

; Drive mappings 
M584 X0.4 Y0.5 Z0.0:0.1:0.2:0.3 E2.0               	; set drive mapping

; General drive config like speeds, accel, jerk, etc
M350 X16 Y16 Z16 E16 I1			                   	; configure microstepping with interpolation
M92 X80.00 Y80.00 Z400.00 E410.00         			; set steps per mm
M566 X300.00 Y300.00 Z300.00 E120.00          		; set maximum instantaneous speed changes (mm/min)
M203 X18000.00 Y18000.00 Z2100 E1200		       	; set maximum speeds (mm/min)
M201 X3000.00 Y3000.00 Z350.00 E250.00    			; set accelerations (mm/s^2)
M906 X900 Y900 Z1000 E600 I30                		; set motor currents (mA) and motor idle factor in per cent
M84 S30                                             ; Set idle timeout

; Axis Limits
M208 X1 Y0 Z0 S1                                   	; set axis minima
M208 X301 Y305 Z300 S0                             	; set axis maxima

; Endstops
M574 X2 S1 P"!io1.in"                               ; X Hall effect endstop
M574 Y2 S1 P"!io2.in"                               ; Y Hall effect endstop
M574 Z1 S1 P"io3.in"                                ; Z microswitch endstop

; Z-Probe
M558 P5 C"io4.in" I1 H5 F120 T12000                 ; set Z probe type to switch and the dive height + speeds
M671 X354:354:-53:-53 Y0:370:370:0 S20        	 	; Define Z belts locations (Front_Left, Back_Left, Back_Right, Front_Right)

G31 P500 X0 Y25 Z0.85                              ; set Z probe trigger value, offset and trigger height
M557 X15:285 Y25:275 S40                           	; define mesh grid

; Hotbed
M308 S0 P"temp0" Y"thermistor" T100000 B4138       	; Thermistor
M950 H0 C"out0" T0                                 	; Heater
M307 H0 A157.8 C632 D1.4 V24.0 S0.6 B0			   	; Heater 0 model: gain 157.8, time constant 632.0, dead time 1.4, max PWM 0.60, calibration voltage 24.0, mode PID
M140 H0                                            	; Map heated bed to heater 0
M143 H0 S120                                       	; Set temperature limit for heater 0 to 120C

; Hotend
M308 S1 P"temp1" Y"thermistor" T100000 B4138       	; Thermistor
M950 H1 C"out1" T1                                 	; Heater

; Mosquitto
M307 H1 A593.7 C180.4 D7.6 V24.0 B0				   	; Heater 1 model: gain 593.7, time constant 180.4, dead time 7.6, max PWM 1.00, calibration voltage 24.0, mode PID

; Mosquitto HF
; M307 H1 A408.1 C161.3 D4.8 V24.0 B0				   	; Heater 1 model: gain 408.1, time constant 161.3, dead time 4.8, max PWM 1.00, calibration voltage 24.0, mode PID

; DHT22 sensor
M308 S10 P"0.spi.cs1" Y"dht22"       A"Chamber Temp"    ; Temperature (connected to cs0 port on the temp daughterboard slot
M308 S11 P"S10.1"     Y"dhthumidity" A"Chamber Hum[%]"  ; Humidity

; Main fans
M950 F0 C"2.out6" Q500                             	; Part cooling fan
M106 P0 S0 H-1 C"Part Fan"                        	; Default off; thermostatic control off 
M950 F1 C"out8" Q500                           	   	; Hotend cooling fan
;M106 P1 S1 H-1                                     	; Default off; thermostatic control off 
M106 P1 S1 H1 T45 C"Hotend Fan"                    	; Default on; thermistatic control on above 45C

; Aux fans (option to control daughter board fans based on main board temp sensor coming in RRF3.3, for now has to be manual)
M950 F2 C"2.out7" Q500                              ; Electronics compartment fan 1
M106 P2 S0 H-1 C"Electronics Fan 1"				; Default off; thermostatic control off 
M950 F3 C"2.out8" Q500                              ; Electronics compartment fan 2
M106 P3 S0 H-1 C"Electronics Fan 2"				; Default off; thermostatic control off 

M950 F4 C"!out4+out4.tach"							; Noctua exhaust filter fan
M106 P4 S0 H-1 C"Exhaust Fan"						; Default off 

; Tools
M563 P0 D0 H1 F0                                   	; define tool 0
G10 P0 X0 Y0 Z0                                    	; set tool 0 axis offsets
G10 P0 R0 S0                                       	; set initial tool 0 active and standby temperatures to 0C

; Push buttons and their config
M950 J5 C"io5.in" 									; Left button - light
M581 P5 T5 											; when button pushed, execute trigger5.g

M950 J6 C"io7.in"  									; Left button - heat up
M950 F5 C"io6.out"									; configure the relay as fan
M106 F5 S0 H-1 C"Light"
M581 P6 T6											; When pushed - trigger relay switch on io6.out

; Custom settings are not defined

; Miscellaneous
M911 S22 R23 P"M913 X0 Y0 G91 M83 G1 Z3 E-5 F1000" 	; set voltage thresholds and actions to run on power loss
M575 P1 S1 B57600                            		; enable support for PanelDue

[dc42]

@pkos, thanks for your reports. I'm sorry that you have lost two large prints.

As the head movement was not correct, the problem appears to be occurring on the main board, not on the EXP3HC.

Can you share one of those GCode files, so that I can run it here?

I am about to prepare a 3.2.1 build of the EXP3HC firmware. This should allow you to revert to RRF 3.2 without experiencing the slow startup issue on the EXP3HC.

[pkos]

Here we go then.

This is the file that I tried printing since last night where it failed both times.

https://we.tl/t-6WJmPSXoAq

Great news on the 3.2.1

I know I am probably getting annoying by now, but I'm one of those persistent people. The assertion failure is still there and I see that the bootloader ID is not there. Is this as it should be? Or maybe 3.2.1 will solve the problem?

EDIT: One more note - both prints failed at a different point in the gcode. One much earlier than the second one.

[dc42]

@pkos said in Intermittent communication disruption between 6HC and 3HC:

I know I am probably getting annoying by now, but I'm one of those persistent people. The assertion failure is still there and I see that the bootloader ID is not there. Is this as it should be?

The assertion failure report is the saved info from the crash when you were running older firmware. This is clear because the free memory and stack trace are exactly the same as before; whereas if the same crash happened again now, at the very least the free memory would be lower. What's happening is that "normal" software resets such as when you installed the new firmware did not clear out the data.

If you wish, you can replace the software reset data by sending M122 B2 P1004 to provoke a deliberate reset. The software reset data will then report a deliberate zero divide error.

The bootloader ID is not reported because older bootloaders do not report their ID. But please do not update the bootloader at this stage.

Thanks for providing the GCode file. I will run this on the bench to see if I can replicate the problem.

[pkos]

And now it's perfectly clear - thank you I am much more calm now.

For now, I'm back on 3.1.1 running prints - so far so good.
I'll wait for the 3.2.1 to see if that helps at all.

Then the last thing to solve for me will be the DHT22. 3.1.1 - works perfectly. Switch to 3.2 or 3.3 - nothing. Values jump to 2k.