Intermittent communication disruption between 6HC and 3HC

pkos

I'm sorry to be the bearer of bad news, but it's much worse now. I don't know if it's related in any way to the firmware shuffling of late, but now, I can't seem to be able to get a single print off successfully that takes longer than 8 hours.

Almost immediately after my post last night, I noticed that the printer stopped moving and paused on it's own. There were no messages in the console. The printer was just stopped above the print.

I hit resume and the printer seemed to try to move the carriage, but was unable to. It was just stuck in place and jittering around. The display on PanelDue showed significant changes in location, but there was only jittery motion.

I aborted the print and started another, fresh gcode again in hopes that it's just a fluke.

Alas, it was not. A night print that was supposed to take 9 hours froze sometime around the 7th hour (I can't tell for sure).

This time I grabbed diag logs from before I hit resume and after. I'm pasting them below along with my current config.g file.

To be sure - after I hit emergency stop on the panel, everything was moving perfectly fine, so it wasn't something mechanical that would prevent the printer from moving. I did notice that the quality of the print is much worse on the z axis - something I never had an issue with before.

I am now getting very worried that something more serious broke and will have the printer down longer.

Do the logs tell you anything?

Before resume:

m122 b0
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.3beta running on Duet 3 MB6HC v0.6 or 1.0 (standalone mode)
Board ID: 08DJM-956L2-G43S4-6JKDA-3SJ6T-1B6GH
Used output buffers: 1 of 40 (40 max)
=== RTOS ===
Static ram: 149800
Dynamic ram: 92100 of which 72 recycled
Never used RAM 108700, free system stack 118 words
Tasks: NETWORK(ready,228) ETHERNET(blocked,117) SENSORS(blocked,15) HEAT(blocked,274) CanReceiv(blocked,877) CanSender(blocked,337) CanClock(blocked,326) TMC(blocked,16) MAIN(running,616) IDLE(ready,20)
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 21:54:30 ago, cause: software
Last software reset at 2021-02-02 09:54, reason: User, GCodes spinning, available RAM 108908, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x04
Aux0 errors 0,0,0
Aux1 errors 0,0,0
MCU temperature: min 44.1, current 44.7, max 45.4
Supply voltage: min 21.6, current 24.0, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.0, current 12.1, max 12.2, under voltage events: 0
Driver 0: position 17407, standstill, reads 36452, writes 0 timeouts 0, SG min/max 0/1023
Driver 1: position 276, standstill, reads 36452, writes 0 timeouts 0, SG min/max 0/1023
Driver 2: position 8318, standstill, reads 36453, writes 0 timeouts 0, SG min/max 0/1023
Driver 3: position 0, standstill, reads 36453, writes 0 timeouts 0, SG min/max 0/1023
Driver 4: position 0, standstill, reads 36450, writes 2 timeouts 0, SG min/max 0/1023
Driver 5: position 0, standstill, reads 36450, writes 2 timeouts 0, SG min/max 0/1023
Date/time: 2021-02-03 07:48:52
Slowest loop: 49.84ms; fastest: 0.05ms
=== Storage ===
Free file entries: 9
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 3.4ms, write time 0.8ms, max retries 0
=== Move ===
DMs created 125, maxWait 0ms, bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves 935235, completed moves 935235, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 1], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
Heater 0 is on, I-accum = 0.0
Heater 1 is on, I-accum = 0.3
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is ready with "m122 b0" in state(s) 0
Telnet is idle in state(s) 0
File is doing "G1 X110.426 Y107.065 E0.07748 F1134.2" in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== CAN ===
Messages queued 494969, send timeouts 0, received 140596, lost 0, longest wait 21ms for reply type 6024, peak Tx sync delay 468, free buffers 48 (min 42)
=== Network ===
Slowest loop: 10.04ms; fastest: 0.02ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- Ethernet -
State: active
Error counts: 0 0 1 0 0
Socket states: 2 2 2 2 2 0 0 0



m122 b2
Diagnostics for board 2:
Duet EXP3HC firmware version 3.3beta (2021-02-01 22:29:11)
Bootloader ID: not available
Never used RAM 154972, free system stack 0 words
Move 80 HEAT 78 CanAsync 72 CanRecv 82 CanClock 74 TMC 30 MAIN 263 AIN 260
Last reset 21:54:31 ago, cause: software
Last software reset time unknown, reason: AssertionFailed, available RAM 173348, slot 0
Software reset code 0x0120 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0445f85f BFAR 0xe000ed38 SP 0x2002ff8c Task MAIN Freestk 4294967295 ok
Stack: 000002f2 0002a9b0 00020f41 00000000 0002256d 20002ec8 20003100 00000031 00000001 20002980 03800209 0001a72d 0001a709 00000000 00000000 ffffffff 0001de4b 200022d8 20002330 00029c48 ffffffed 00000000 00f00000 e000ef34 c0000000 200041dc 00020ea5
Driver 0: position 26155512, 410.0 steps/mm,  standstill, reads 45397, writes 0 timeouts 0, SG min/max not available, steps req 0 done 0
Driver 1: position 0, 80.0 steps/mm,  standstill, reads 45395, writes 0 timeouts 0, SG min/max not available, steps req 0 done 0
Driver 2: position 0, 80.0 steps/mm,  standstill, reads 45396, writes 0 timeouts 0, SG min/max not available, steps req 0 done 0
Moves scheduled 1320929, completed 1320929, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter 9, peak Rx sync delay 173, resyncs 0, no step interrupt scheduled
VIN: 24.1V, V12: 12.1V
MCU temperature: min 45.7C, current 45.9C, max 46.1C
Ticks since heat task active 184, ADC conversions started 78871426, completed 78871425, timed out 0
Last sensors broadcast 0x00000000 found 0 190 ticks ago, loop time 0
CAN messages queued 423, send timeouts 0, received 885, lost 0, free buffers 36, min 36, error reg 0
dup 0, oos 0, bm 0, wbm 0

After resume (and about 5 seconds of the carriage jittering):

m122 b0
Diagnostics for board 2:
Duet EXP3HC firmware version 3.3beta (2021-02-01 22:29:11)
Bootloader ID: not available
Never used RAM 154972, free system stack 0 words
Move 80 HEAT 78 CanAsync 72 CanRecv 82 CanClock 74 TMC 30 MAIN 263 AIN 260
Last reset 21:56:48 ago, cause: software
Last software reset time unknown, reason: AssertionFailed, available RAM 173348, slot 0
Software reset code 0x0120 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0445f85f BFAR 0xe000ed38 SP 0x2002ff8c Task MAIN Freestk 4294967295 ok
Stack: 000002f2 0002a9b0 00020f41 00000000 0002256d 20002ec8 20003100 00000031 00000001 20002980 03800209 0001a72d 0001a709 00000000 00000000 ffffffff 0001de4b 200022d8 20002330 00029c48 ffffffed 00000000 00f00000 e000ef34 c0000000 200041dc 00020ea5
Driver 0: position 26160116, 410.0 steps/mm,  standstill, reads 46672, writes 0 timeouts 0, SG min/max 0/192, steps req 22610 done 22610
Driver 1: position 0, 80.0 steps/mm,  standstill, reads 46672, writes 0 timeouts 0, SG min/max not available, steps req 0 done 0
Driver 2: position 0, 80.0 steps/mm,  standstill, reads 46671, writes 0 timeouts 0, SG min/max not available, steps req 0 done 0
Moves scheduled 1321401, completed 1321401, in progress 0, hiccups 0, step errors 0, maxPrep 59, maxOverdue 1, maxInc 1, mcErrs 0, gcmErrs 0
Peak sync jitter 10, peak Rx sync delay 173, resyncs 0, no step interrupt scheduled
VIN: 24.1V, V12: 12.1V
MCU temperature: min 45.7C, current 45.9C, max 46.1C
Ticks since heat task active 96, ADC conversions started 79008838, completed 79008837, timed out 0
Last sensors broadcast 0x00000000 found 0 102 ticks ago, loop time 0
CAN messages queued 588, send timeouts 0, received 1722, lost 0, free buffers 36, min 36, error reg 0
dup 0, oos 0, bm 0, wbm 0

=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.3beta running on Duet 3 MB6HC v0.6 or 1.0 (standalone mode)
Board ID: 08DJM-956L2-G43S4-6JKDA-3SJ6T-1B6GH
Used output buffers: 15 of 40 (40 max)
=== RTOS ===
Static ram: 149800
Dynamic ram: 92100 of which 72 recycled
Never used RAM 108700, free system stack 118 words
Tasks: NETWORK(ready,228) ETHERNET(blocked,117) SENSORS(blocked,15) HEAT(blocked,274) CanReceiv(blocked,877) CanSender(blocked,337) CanClock(blocked,326) TMC(blocked,16) MAIN(running,616) IDLE(ready,20)
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 21:56:48 ago, cause: software
Last software reset at 2021-02-02 09:54, reason: User, GCodes spinning, available RAM 108908, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x04
Aux0 errors 0,0,0
Aux1 errors 0,0,0
MCU temperature: min 44.5, current 44.8, max 45.0
Supply voltage: min 23.9, current 24.0, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.1, current 12.1, max 12.2, under voltage events: 0
Driver 0: position 80, standstill, reads 50864, writes 0 timeouts 0, SG min/max 0/990
Driver 1: position 80, standstill, reads 50864, writes 0 timeouts 0, SG min/max 0/1023
Driver 2: position 9190, standstill, reads 50863, writes 0 timeouts 0, SG min/max 0/1023
Driver 3: position 0, standstill, reads 50863, writes 0 timeouts 0, SG min/max 0/1023
Driver 4: position 0, standstill, reads 50863, writes 0 timeouts 0, SG min/max 19/1023
Driver 5: position 0, standstill, reads 50863, writes 0 timeouts 0, SG min/max 16/1023
Date/time: 2021-02-03 07:51:10
Slowest loop: 49.77ms; fastest: 0.04ms
=== Storage ===
Free file entries: 9
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 3.3ms, write time 0.8ms, max retries 0
=== Move ===
DMs created 125, maxWait 0ms, bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves 935861, completed moves 935861, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
Heater 1 is on, I-accum = 0.4
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is ready with "m122 b0" in state(s) 0
Telnet is idle in state(s) 0
File is doing "G1 X91.532 Y86.855 E0.01315" in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== CAN ===
Messages queued 1037, send timeouts 0, received 590, lost 0, longest wait 20ms for reply type 6024, peak Tx sync delay 468, free buffers 48 (min 46)
=== Network ===
Slowest loop: 3.99ms; fastest: 0.02ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions
HTTP sessions: 2 of 8
- Ethernet -
State: active
Error counts: 0 0 1 0 0
Socket states: 2 2 2 2 2 0 0 0


m122 b2
Diagnostics for board 2:
Duet EXP3HC firmware version 3.3beta (2021-02-01 22:29:11)
Bootloader ID: not available
Never used RAM 154972, free system stack 0 words
Move 80 HEAT 78 CanAsync 72 CanRecv 82 CanClock 74 TMC 30 MAIN 263 AIN 260
Last reset 21:56:48 ago, cause: software
Last software reset time unknown, reason: AssertionFailed, available RAM 173348, slot 0
Software reset code 0x0120 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0445f85f BFAR 0xe000ed38 SP 0x2002ff8c Task MAIN Freestk 4294967295 ok
Stack: 000002f2 0002a9b0 00020f41 00000000 0002256d 20002ec8 20003100 00000031 00000001 20002980 03800209 0001a72d 0001a709 00000000 00000000 ffffffff 0001de4b 200022d8 20002330 00029c48 ffffffed 00000000 00f00000 e000ef34 c0000000 200041dc 00020ea5
Driver 0: position 26160116, 410.0 steps/mm,  standstill, reads 46672, writes 0 timeouts 0, SG min/max 0/192, steps req 22610 done 22610
Driver 1: position 0, 80.0 steps/mm,  standstill, reads 46672, writes 0 timeouts 0, SG min/max not available, steps req 0 done 0
Driver 2: position 0, 80.0 steps/mm,  standstill, reads 46671, writes 0 timeouts 0, SG min/max not available, steps req 0 done 0
Moves scheduled 1321401, completed 1321401, in progress 0, hiccups 0, step errors 0, maxPrep 59, maxOverdue 1, maxInc 1, mcErrs 0, gcmErrs 0
Peak sync jitter 10, peak Rx sync delay 173, resyncs 0, no step interrupt scheduled
VIN: 24.1V, V12: 12.1V
MCU temperature: min 45.7C, current 45.9C, max 46.1C
Ticks since heat task active 96, ADC conversions started 79008838, completed 79008837, timed out 0
Last sensors broadcast 0x00000000 found 0 102 ticks ago, loop time 0
CAN messages queued 588, send timeouts 0, received 1722, lost 0, free buffers 36, min 36, error reg 0
dup 0, oos 0, bm 0, wbm 0

; Configuration file for Duet 3 (firmware version 3)
; executed by the firmware on start-up
;
; generated by RepRapFirmware Configuration Tool v3.1.3 on Sun Jul 12 2020 20:53:52 GMT+0200 (Central European Summer Time)

; General preferences
G90                                            	    ; send absolute coordinates...
M83                                                 ; ...but relative extruder moves
M550 P"Voron"                                   	; set printer name
M669 K1                                            	; select CoreXY mode

; Network
M552 P0.0.0.0 S1                                   	; enable network and acquire dynamic address via DHCP
M586 P0 S1                                          ; enable HTTP
M586 P1 S0                                         	; disable FTP
M586 P2 S0                                         	; disable Telnet

; Drives for XY
M569 P0.4 S1                                       	; A motor 
M569 P0.5 S1                                       	; B motor

; Drives for Z
M569 P0.0 S0                                       	; Front left 
M569 P0.1 S1                                       	; Back left
M569 P0.2 S0                                       	; Back right
M569 P0.3 S1                                       	; Front right

; Wait before addressing the 3HC
; G4 S25

; Drive for extruder 
M569 P2.0 S1                                       	; physical drive 1.0 goes forwards

; Drive mappings 
M584 X0.4 Y0.5 Z0.0:0.1:0.2:0.3 E2.0               	; set drive mapping

; General drive config like speeds, accel, jerk, etc
M350 X16 Y16 Z16 E16 I1			                   	; configure microstepping with interpolation
M92 X80.00 Y80.00 Z400.00 E410.00         			; set steps per mm
M566 X300.00 Y300.00 Z300.00 E120.00          		; set maximum instantaneous speed changes (mm/min)
M203 X18000.00 Y18000.00 Z2100 E1200		       	; set maximum speeds (mm/min)
M201 X3000.00 Y3000.00 Z350.00 E250.00    			; set accelerations (mm/s^2)
M906 X900 Y900 Z1000 E600 I30                		; set motor currents (mA) and motor idle factor in per cent
M84 S30                                             ; Set idle timeout

; Axis Limits
M208 X1 Y0 Z0 S1                                   	; set axis minima
M208 X301 Y305 Z300 S0                             	; set axis maxima

; Endstops
M574 X2 S1 P"!io1.in"                               ; X Hall effect endstop
M574 Y2 S1 P"!io2.in"                               ; Y Hall effect endstop
M574 Z1 S1 P"io3.in"                                ; Z microswitch endstop

; Z-Probe
M558 P5 C"io4.in" I1 H5 F120 T12000                 ; set Z probe type to switch and the dive height + speeds
M671 X354:354:-53:-53 Y0:370:370:0 S20        	 	; Define Z belts locations (Front_Left, Back_Left, Back_Right, Front_Right)

G31 P500 X0 Y25 Z0.85                              ; set Z probe trigger value, offset and trigger height
M557 X15:285 Y25:275 S40                           	; define mesh grid

; Hotbed
M308 S0 P"temp0" Y"thermistor" T100000 B4138       	; Thermistor
M950 H0 C"out0" T0                                 	; Heater
M307 H0 A157.8 C632 D1.4 V24.0 S0.6 B0			   	; Heater 0 model: gain 157.8, time constant 632.0, dead time 1.4, max PWM 0.60, calibration voltage 24.0, mode PID
M140 H0                                            	; Map heated bed to heater 0
M143 H0 S120                                       	; Set temperature limit for heater 0 to 120C

; Hotend
M308 S1 P"temp1" Y"thermistor" T100000 B4138       	; Thermistor
M950 H1 C"out1" T1                                 	; Heater

; Mosquitto
M307 H1 A593.7 C180.4 D7.6 V24.0 B0				   	; Heater 1 model: gain 593.7, time constant 180.4, dead time 7.6, max PWM 1.00, calibration voltage 24.0, mode PID

; Mosquitto HF
; M307 H1 A408.1 C161.3 D4.8 V24.0 B0				   	; Heater 1 model: gain 408.1, time constant 161.3, dead time 4.8, max PWM 1.00, calibration voltage 24.0, mode PID

; DHT22 sensor
M308 S10 P"0.spi.cs1" Y"dht22"       A"Chamber Temp"    ; Temperature (connected to cs0 port on the temp daughterboard slot
M308 S11 P"S10.1"     Y"dhthumidity" A"Chamber Hum[%]"  ; Humidity

; Main fans
M950 F0 C"2.out6" Q500                             	; Part cooling fan
M106 P0 S0 H-1 C"Part Fan"                        	; Default off; thermostatic control off 
M950 F1 C"out8" Q500                           	   	; Hotend cooling fan
;M106 P1 S1 H-1                                     	; Default off; thermostatic control off 
M106 P1 S1 H1 T45 C"Hotend Fan"                    	; Default on; thermistatic control on above 45C

; Aux fans (option to control daughter board fans based on main board temp sensor coming in RRF3.3, for now has to be manual)
M950 F2 C"2.out7" Q500                              ; Electronics compartment fan 1
M106 P2 S0 H-1 C"Electronics Fan 1"				; Default off; thermostatic control off 
M950 F3 C"2.out8" Q500                              ; Electronics compartment fan 2
M106 P3 S0 H-1 C"Electronics Fan 2"				; Default off; thermostatic control off 

M950 F4 C"!out4+out4.tach"							; Noctua exhaust filter fan
M106 P4 S0 H-1 C"Exhaust Fan"						; Default off 

; Tools
M563 P0 D0 H1 F0                                   	; define tool 0
G10 P0 X0 Y0 Z0                                    	; set tool 0 axis offsets
G10 P0 R0 S0                                       	; set initial tool 0 active and standby temperatures to 0C

; Push buttons and their config
M950 J5 C"io5.in" 									; Left button - light
M581 P5 T5 											; when button pushed, execute trigger5.g

M950 J6 C"io7.in"  									; Left button - heat up
M950 F5 C"io6.out"									; configure the relay as fan
M106 F5 S0 H-1 C"Light"
M581 P6 T6											; When pushed - trigger relay switch on io6.out

; Custom settings are not defined

; Miscellaneous
M911 S22 R23 P"M913 X0 Y0 G91 M83 G1 Z3 E-5 F1000" 	; set voltage thresholds and actions to run on power loss
M575 P1 S1 B57600                            		; enable support for PanelDue

dc42

@pkos, thanks for your reports. I'm sorry that you have lost two large prints.

As the head movement was not correct, the problem appears to be occurring on the main board, not on the EXP3HC.

Can you share one of those GCode files, so that I can run it here?

I am about to prepare a 3.2.1 build of the EXP3HC firmware. This should allow you to revert to RRF 3.2 without experiencing the slow startup issue on the EXP3HC.

pkos

Here we go then.

This is the file that I tried printing since last night where it failed both times.

https://we.tl/t-6WJmPSXoAq

Great news on the 3.2.1

I know I am probably getting annoying by now, but I'm one of those persistent people. The assertion failure is still there and I see that the bootloader ID is not there. Is this as it should be? Or maybe 3.2.1 will solve the problem?

EDIT: One more note - both prints failed at a different point in the gcode. One much earlier than the second one.

dc42

@pkos said in Intermittent communication disruption between 6HC and 3HC:

I know I am probably getting annoying by now, but I'm one of those persistent people. The assertion failure is still there and I see that the bootloader ID is not there. Is this as it should be?

The assertion failure report is the saved info from the crash when you were running older firmware. This is clear because the free memory and stack trace are exactly the same as before; whereas if the same crash happened again now, at the very least the free memory would be lower. What's happening is that "normal" software resets such as when you installed the new firmware did not clear out the data.

If you wish, you can replace the software reset data by sending M122 B2 P1004 to provoke a deliberate reset. The software reset data will then report a deliberate zero divide error.

The bootloader ID is not reported because older bootloaders do not report their ID. But please do not update the bootloader at this stage.

Thanks for providing the GCode file. I will run this on the bench to see if I can replicate the problem.

pkos

And now it's perfectly clear - thank you I am much more calm now.

For now, I'm back on 3.1.1 running prints - so far so good.
I'll wait for the 3.2.1 to see if that helps at all.

Then the last thing to solve for me will be the DHT22. 3.1.1 - works perfectly. Switch to 3.2 or 3.3 - nothing. Values jump to 2k.

dc42

@pkos said in Intermittent communication disruption between 6HC and 3HC:

Then the last thing to solve for me will be the DHT22. 3.1.1 - works perfectly. Switch to 3.2 or 3.3 - nothing. Values jump to 2k.

This is on my list to investigate.

dc42

The candiate EXP3HC 3.2.1 firmware is now avaiable at https://www.dropbox.com/sh/1lwimb98k6hzz3z/AAApVr_P6roUjnya4riDbGAba?dl=0. This includes the fix to the slow startup issue (LED not starting to blink immediately after power up).

Have you run any long prints using RRF 3.2 yet?

pkos

Not yet. For now I quickly printed the rest of functional parts in small batches, but I will give the new firmware a try.

dc42

@pkos said in Intermittent communication disruption between 6HC and 3HC:

Not yet. For now I quickly printed the rest of functional parts in small batches, but I will give the new firmware a try.

OK, in that case bear in mind that 3.2 might have the same issue that you saw with 3.3. You might want to try an air print first to save on filament.

pkos

Will do.

Quick question - do I need to redo PID for 3.2? I thought I saw that there is a new algorithm, but it's not necessary to redo PID.

I am seeing some very weird behavior on 3.2. Regardless of the temperature I set on the bed, it drops temps without the firmware screaming bloody murder and thermal runaway.

Attaching screenshot. Please note that the temp is 100C, while it's set at 105C.
Screenshot 2021-02-03 200220.png
Hotend temp dropping was my doing.

Just in case, log from the main board below.

m122 b0
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.2 running on Duet 3 MB6HC v0.6 or 1.0 (standalone mode)
Board ID: 08DJM-956L2-G43S4-6JKDA-3SJ6T-1B6GH
Used output buffers: 1 of 40 (21 max)
=== RTOS ===
Static ram: 149788
Dynamic ram: 94408 of which 76 recycled
Never used RAM 114560, free system stack 129 words
Tasks: NETWORK(ready,189) ETHERNET(blocked,109) SENSORS(blocked,19) HEAT(blocked,289) CanReceiv(blocked,848) CanSender(blocked,341) CanClock(blocked,352) TMC(blocked,19) MAIN(running,717) IDLE(ready,19)
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 00:29:42 ago, cause: power up
Last software reset at 2021-02-03 19:30, reason: User, GCodes spinning, available RAM 114560, slot 2
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
Aux0 errors 0,0,0
Aux1 errors 0,0,0
MCU temperature: min 37.9, current 44.5, max 46.5
Supply voltage: min 23.9, current 24.0, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.1, current 12.1, max 12.2, under voltage events: 0
Driver 0: position 23945, ok, reads 31637, writes 21 timeouts 0, SG min/max 0/1023
Driver 1: position 3358, ok, reads 31637, writes 21 timeouts 0, SG min/max 0/1023
Driver 2: position 8994, ok, reads 31637, writes 21 timeouts 0, SG min/max 0/1023
Driver 3: position 0, ok, reads 31638, writes 21 timeouts 0, SG min/max 0/1023
Driver 4: position 0, ok, reads 31638, writes 21 timeouts 0, SG min/max 0/1023
Driver 5: position 0, ok, reads 31638, writes 21 timeouts 0, SG min/max 0/1023
Date/time: 2021-02-03 20:01:50
Slowest loop: 44.47ms; fastest: 0.05ms
=== Storage ===
Free file entries: 9
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 3.4ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, maxWait 89884ms, bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves 70080, completed moves 70020, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state 3
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
Heater 0 is on, I-accum = 0.2
Heater 1 is on, I-accum = 0.3
=== GCodes ===
Segments left: 1
Movement lock held by null
HTTP is ready with "m122 b0" in state(s) 0
Telnet is idle in state(s) 0
File is doing "G1 X178.308 Y129.689 E0.02381" in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 79.63ms; fastest: 0.02ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- Ethernet -
State: active
Error counts: 0 0 1 0 0
Socket states: 2 5 2 2 2 0 0 0
=== CAN ===
Messages queued 73379, send timeouts 0, received 7629, lost 0, longest wait 5ms for reply type 6029, free buffers 48

Phaedrux

@pkos I would say it's worth trying the new PID tuning. The old values should be usable though.

pkos

I'll give another PID tune a try then just in case.
Still... weird that firmware didn't alert me at once that the temperature is not rising fast enough - especially since it was below the set temp and actually very slowly dropping - you don't see it on the screenshot above, but by the end of that print, temp was sub 100C.

dc42

You don't have to re-run PID tuning for 3.2 if you are happy with the PID performance under 3.1.1. You do need to re-tune if you want the benefit of hot end heater feedforward to compensate for changes in the speed of the print cooling fan.

pkos

Update from today.

I ran two tests today on 3.2 (6HC) and 3.2.1 (3HC).
One was a print and generic sync test, the other was a sync test after a long power down.

Quick sync, but print failed on 3.2 and 3.2.1
There was no problem with sync. That caught on pretty much immediately.
Print failed very quickly - after about 5 layers, I heard lost steps and the print was shifted. I stopped the print and turned the device off for the second part of the test.

Instant sync after a long power down.
I left the printer off for a good 6 hours. Now I came back and turned it on - it seemed almost as if both boards came up at the same time and synced up even before the rest of the system launched.

So I guess the sync problem can be considered solved.

A new symptom that I never had before is the behavior of the hotbed. Historically, this printer has been running just fine since May - often printing long prints, never had issues with temperature. I guess it's just coincidence that now I keep seeing so many issues

The problem is that after about an hour of the hotbed being heated up to ABS temps (about 105C) - I notice that the thermistor starts reporting dropping values, but the firmware doesn't complain about it. One screenshot was posted above. Today I had this happen a couple times. I now need to figure out what is going on there. I'm guessing a hardware issue, but I am curious why the firmware doesn't complain.

My hotbed is an 8mm alu plate with an AC Keenovo mat on it (it has the thermistor embedded in it). That mat is connected to the Duet 6HC via a Crydom SSR. There is a thermal fuse between the Crydom and the Keenovo mat (it triggers at 125C). The PSU is a proper Mean Well (bought from TME) not a noname PSU.

If you want me to shift to a new topic to track this, I'll be happy to, but for now - I'll ask here.

Is there a way to see what is being sent to the hotbed output from the 6HC? Forgive the comparison, but there is one nice feature in Klipper and Mainsail that would help debugging here. They show that data in their GUI (screenshot below). I'm wondering if there's a way to see that on a Duet with RRF - this would help debugging a lot since I could see what/if anything is sent to the board when the temps start dropping and how the numbers compare to the time, when temp is held steady at the preset temperature.

Screenshot 2021-02-04 205918.png

Is this kind of behavior normal? I set the temp of the hotbed to 105C, it gets there fine and stays there for a while, but - I see temps start dropping to 100C, but the firmware doesn't complain? Even if I set the temperature to 120C - temps do not rise, but keep falling. It's very slow (1C over 10 or so minutes), but still - it is falling.

I did run a new PID tune on the bed for the temps I usually print at (ABS, so ~105C). But that didn't eliminate the problem.

pkos

OK. So it seems the Keenovo mat is broken - or at least everything points to it being at fault regarding the temps.

What bugs me though is how come the firmware didn't object when instead of temp being on point, it started dropping...

dc42

@pkos said in Intermittent communication disruption between 6HC and 3HC:

What bugs me though is how come the firmware didn't object when instead of temp being on point, it started dropping...

The allowed temperature drop before a fault is registered is set by the M570 command. The default is 15C because when we tried setting it lower, too many users reported heater faults, mostly caused by print cooling fans blowing too much cold air at the heater block. I suggest you configure a lower maximum temperature drop, at least for the bed heater.

pkos

Understood. Thank you.

I have a new mat on the way (lucky to find someone wanting to get rid of theirs locally), but until I get it - I won't be able to do any more testing for the next couple of days.

dc42

@pkos said in Intermittent communication disruption between 6HC and 3HC:

Understood. Thank you.

I have a new mat on the way (lucky to find someone wanting to get rid of theirs locally), but until I get it - I won't be able to do any more testing for the next couple of days.

No problem.

I think the issue with missed steps you saw with firmware 3.2 was caused by a problem that I suspected to exist in firmware 3.2 on the MB6HC (caused by the code speed increase in 3.2 on MB6HC) which we fixed in 3.3. When we discovered this issue, I was surprised that we had no reports of missed steps with 3.2 on MB6HC. I guess you are the first user to use the high step rates on the MB6HC that are needed to trigger this problem.

I have back-ported the fix for this to release 3.2.1 which is currently undergoing testing.

pkos

Ah sweet that you found it too

Granted, I am printing at pretty fast speeds - infil at 150mm/s, move at 300 mm/s and this was visible usually around layer 3 or 4.

I can't wait to see how 3.2.1 will do then once I get my printer back online and I see you are looking at a solution for DHT22 (even if it means dropping DHT11)

You guys are amazing! Thanks for the support and patience. I am a big fan of the Duet ecosystem

dc42

The candidate 3.2.1 firmware already includes the fix for DHT sensors on MB6HC. It's already available on Dropbox but needs further testing before we release it.