Help diagnosing communication errors between Duet 3 6hc & 1XD

[dsela]

Hi,

I have a Duet 3 6HC main board connected to a 1XD expansion board that runs an external stepper driver.
Everything was working OK until yesterday I noticed that sometimes when I send a command to the external stepper, nothing happens.
I noticed that the LED STATUS on the 1XD doesn't blink - in normal operation it blinked in sync with the LED on the main board, as it's supposed to.

I tried a lot of things but my assumption at the moment is that the 1XD expansion board is damaged (maybe some moisture has gotten to it as the contacts on the CAN ports seemed to be a little green, but that's just a guess).

Without any obvious reason, it will start blinking again and then communication will be OK (tested through M122, when the status LED is blinking, M122 returns all good. When it's not blinking it returns CAN error not connected).

I tried fiddling around with the connections (power + CAN), sometimes it seems to do something, sometimes it doesn't. Tried replacing CAN cable, didn't help. Anyway even after I get communication it seems that it will just stop after a few minutes (status LED off) regardless of what I do.
I tried cleaning the entire PCB with contact cleaner, didn't change anything.

I've already ordered a new 1XD express from E3D but until it gets here I want to make sure that it is the problem. I'm worried it might be the main board or something else..Any ideas on how to better diagnose this problem?
The other functions for the main board (on-board steppers, limits switches, outputs to relays) all seem to work OK.

One other thing that I started noticing - when I send "M98 config.g" it seems that the board has a secondary power outage, most noticeable by the always-on fans going to 100% and than returning to the usual 30% set in config.g..I'm not sure if this was happening before the issue with the 1XD..is this normal behavior for sending M98?
The M98 returns all OK, other than the messages about CAN connection whenever the 1XD status LED is not blinking.

Any help would be appreciated..
Thanks

[chrishamm]

@dsela If it stops blinking, the board may lose power from time to time. Check if the power cable provides consistent power and make sure the fuse is properly fitted and not damaged.

[dsela]

@chrishamm I just checked the fuse and the power in for the 1XD, seems OK.
The LED 5V & VIN LED on the 1XD are on all the time.
I pushed the CAN jack in and out and then the STATUS LED started blinking in sync..after 30 seconds it just stopped. I tried pushing the jack in and out again and this time nothing happened..LED 5V & VIN LED stayed ON the whole time..very inconsistent behavior..

Thanks

[dc42]

@dsela please use M122 B# (where # is the CAN address of the board) to check the bootloader version on the 1XD board. If it is "unknown" or earlier than 2.4, update it to version 2.4.

[dsela]

Hi,

I updated the boot loader - seemed to help..now I seem to get communication with the 1XD how ever there still appear to be random outages..this is one:

M122 B40
Diagnostics for board 40:
Duet EXP1XD firmware version 3.4.4 (2022-10-14 11:45:39)
Bootloader ID: SAMC21 bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 5556, free system stack 43 words
Tasks: Move(notifyWait,0.0%,127) HEAT(notifyWait,0.1%,115) CanAsync(notifyWait,0.0%,64) CanRecv(notifyWait,0.0%,75) CanClock(notifyWait,0.0%,64) MAIN(running,96.5%,435) IDLE(ready,0.0%,40) AIN(delaying,3.4%,142), total 100.0%
Last reset 00:00:21 ago, cause: reset button
Last software reset time unknown, reason: HardFault, available RAM 14244, slot 1
Software reset code 0x0060 ICSR 0x00000003 SP 0x20002b48 Task MAIN Freestk 821 ok
Stack: 20003664 000000a5 20003984 2000363c 20003664 00016fc9 00016fe2 01000000 20000eb0 00000054 a5a5a5a5 a5a5a5a5 a5a5a5a5 20003608 00000000 a5a5a5a5 a5a5a5a5 00007b85 00000002 20003664 20003608 a5a5a5a5 a5a5a5a5 a5a5a5a5 a5a5a5a5 0001096b a5a5a5a5
Driver 0: pos 9600, 80.0 steps/mm, steps req 9600 done 4677
Moves scheduled 1, completed 0, in progress 1, hiccups 0, step errors 0, maxPrep 416, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter 1/4, peak Rx sync delay 510, resyncs 0/0, next step interrupt due in 1231 ticks, enabled
VIN voltage: min 26.5, current 26.5, max 26.5
MCU temperature: min 15.2C, current 15.2C, max 15.4C
Last sensors broadcast 0x00000000 found 0 178 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 195, send timeouts 0, received 204, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 299, adv 37074/37074

[dsela]

In addition to my last post, sometimes I still get no communication at all.
Sometimes its followed by:

M122 B40
Warning: Discarded std reply src=40 RID=22 exp=23 "Duet EXP1XD firmware version 3.4.4 (2022-10-14 11:45:39)
Boo"
Warning: Discarded std reply src=40 RID=22 exp=23 "tloader ID: SAMC21 bootloader version 2.4 (2021-12-10)
All a"
Warning: Discarded std reply src=40 RID=22 exp=23 "veraging filters OK"
Diagnostics for board 40:
Duet EXP1XD firmware version 3.4.4 (2022-10-14 11:45:39)
Bootloader ID: SAMC21 bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 5556, free system stack 88 words
Tasks: Move(notifyWait,0.0%,153) HEAT(notifyWait,0.1%,115) CanAsync(notifyWait,0.0%,64) CanRecv(notifyWait,0.0%,77) CanClock(notifyWait,0.0%,64) MAIN(running,96.5%,448) IDLE(ready,0.0%,40) AIN(delaying,3.4%,148), total 100.0%
Last reset 00:00:09 ago, cause: reset button
Last software reset time unknown, reason: HardFault, available RAM 14244, slot 1
Software reset code 0x0060 ICSR 0x00000003 SP 0x20002b48 Task MAIN Freestk 821 ok
Stack: 20003664 000000a5 20003984 2000363c 20003664 00016fc9 00016fe2 01000000 20000eb0 00000054 a5a5a5a5 a5a5a5a5 a5a5a5a5 20003608 00000000 a5a5a5a5 a5a5a5a5 00007b85 00000002 20003664 20003608 a5a5a5a5 a5a5a5a5 a5a5a5a5 a5a5a5a5 0001096b a5a5a5a5
Driver 0: pos 0, 80.0 steps/mm, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter 1/4, peak Rx sync delay 579, resyncs 0/0, no step interrupt scheduled
VIN voltage: min 26.5, current 26.5, max 26.5
MCU temperature: min 16.5C, current 16.9C, max 16.9C
Last sensors broadcast 0x00000000 found 0 10 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 99, send timeouts 0, received 93, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0

[jay_s_uk]

@dsela said in Help diagnosing communication errors between Duet 3 6hc & 1XD:

Last reset 00:00:09 ago, cause: reset button

is this you pressing the reset button to try and get it back online?

[dc42]

@dsela are the 1XD and the 6HC powered from the same PSU? If different PSUs, do they have a common ground?

[dsela]

@jay_s_uk said in Help diagnosing communication errors between Duet 3 6hc & 1XD:

@dsela said in Help diagnosing communication errors between Duet 3 6hc & 1XD:

Last reset 00:00:09 ago, cause: reset button

is this you pressing the reset button to try and get it back online?

No, I didn't touch anything (is there a reset button on the 1XD?)

@dc42 said in Help diagnosing communication errors between Duet 3 6hc & 1XD:

@dsela are the 1XD and the 6HC powered from the same PSU? If different PSUs, do they have a common ground?

Same PSU for both boards

[dc42]

@dsela please check that the CAN termination jumpers are fitted on the last expansion board on the CAN bus but not on any of the others.

Next time this issue occurs, please post a M122 report for the main board as well as for the 1XD.

[dsela]

@dc42
The CAN termination jumpers are fitted on the board, this is the only expansion board.
It looks like the main board is not losing power - these are the M122's I took from both at the same time (1 hour after power-up):

M122 B40
Diagnostics for board 40:
Duet EXP1XD firmware version 3.4.4 (2022-10-14 11:45:39)
Bootloader ID: SAMC21 bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 5556, free system stack 88 words
Tasks: Move(notifyWait,0.0%,153) HEAT(notifyWait,0.1%,115) CanAsync(notifyWait,0.0%,64) CanRecv(notifyWait,0.0%,77) CanClock(notifyWait,0.0%,64) MAIN(running,96.5%,443) IDLE(ready,0.0%,40) AIN(delaying,3.4%,142), total 100.0%
Last reset 00:01:43 ago, cause: reset button
Last software reset time unknown, reason: HardFault, available RAM 14244, slot 1
Software reset code 0x0060 ICSR 0x00000003 SP 0x20002b48 Task MAIN Freestk 821 ok
Stack: 20003664 000000a5 20003984 2000363c 20003664 00016fc9 00016fe2 01000000 20000eb0 00000054 a5a5a5a5 a5a5a5a5 a5a5a5a5 20003608 00000000 a5a5a5a5 a5a5a5a5 00007b85 00000002 20003664 20003608 a5a5a5a5 a5a5a5a5 a5a5a5a5 a5a5a5a5 0001096b a5a5a5a5
Driver 0: pos 0, 80.0 steps/mm, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter 1/4, peak Rx sync delay 522, resyncs 0/0, no step interrupt scheduled
VIN voltage: min 26.5, current 26.5, max 26.5
MCU temperature: min 19.0C, current 19.2C, max 19.3C
Last sensors broadcast 0x00000000 found 0 60 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 853, send timeouts 0, received 942, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0

M122 B0
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.4.4 (2022-10-20 16:19:01) running on Duet 3 MB6HC v1.01 (standalone mode)
Board ID: 08DJM-956BA-NA3TJ-6J9D8-3S86R-1U9QT
Used output buffers: 3 of 40 (26 max)
=== RTOS ===
Static ram: 152824
Dynamic ram: 98044 of which 0 recycled
Never used RAM 99804, free system stack 200 words
Tasks: NETWORK(ready,28.9%,210) ETHERNET(notifyWait,0.0%,436) HEAT(notifyWait,0.0%,322) Move(notifyWait,0.0%,351) CanReceiv(notifyWait,0.0%,773) CanSender(notifyWait,0.0%,335) CanClock(delaying,0.0%,334) TMC(notifyWait,8.3%,91) MAIN(running,62.8%,1083) IDLE(ready,0.0%,30), total 100.0%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 01:05:40 ago, cause: power up
Last software reset details not available
Error status: 0x00
Step timer max interval 131
MCU temperature: min 32.4, current 35.1, max 35.8
Supply voltage: min 25.9, current 25.9, max 26.0, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.0, current 12.1, max 12.1, under voltage events: 0
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min n/a, mspos 8, reads 55227, writes 0 timeouts 0
Driver 1: standstill, SG min n/a, mspos 8, reads 55227, writes 0 timeouts 0
Driver 2: standstill, SG min n/a, mspos 16, reads 55227, writes 0 timeouts 0
Driver 3: standstill, SG min n/a, mspos 16, reads 55227, writes 0 timeouts 0
Driver 4: standstill, SG min n/a, mspos 128, reads 55226, writes 0 timeouts 0
Driver 5: standstill, SG min n/a, mspos 8, reads 55226, writes 0 timeouts 0
Date/time: 2022-12-08 14:49:44
Slowest loop: 27.70ms; fastest: 0.06ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 2.2ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 0, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.0
Heater 1 is on, I-accum = 0.0
Heater 2 is on, I-accum = 0.0
Heater 3 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is ready with "M122 B0" in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty
=== CAN ===
Messages queued 33322, received 26434, lost 0, boc 0
Longest wait 5ms for reply type 6024, peak Tx sync delay 50802, free buffers 50 (min 49), ts 18499/16499/0
Tx timeouts 0,0,2000,0,0,1594 last cancelled message type 4514 dest 127
=== Network ===
Slowest loop: 2.86ms; fastest: 0.03ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 1 of 8
= Ethernet =
State: active
Error counts: 0 0 0 0 0 0
Socket states: 5 2 2 2 2 0 0 0
=== Multicast handler ===
Responder is inactive, messages received 0, responses 0

[dc42]

@dsela did you see any errors during that hour? Was the machine printing?

Please post your config.g file.

[dsela]

@dc42 I didn't see any errors, the only thing I can notice is the STATUS LED on the 1XD stops blinking, and then if I send M122 B40 it returns error.
The machine wasn't printing - the steppers including the external stepper were enabled.
I tried again with steppers enabled and external stepper driver disconnected from the main power - still either got errors when I tried M122 B40 or saw that "last reset cause - reset button" and not "power-up", meaning it rested on it's own after I turned everything on.

This is my config.g:

; Configuration file for Duet 3 MB 6HC 
; executed by the firmware on start-up
;
; generated by RepRapFirmware Configuration Tool v3.3.12 on Fri Aug 05 2022 12:05:46 GMT+0300 (Israel Daylight Time)
 
; General preferences
G90                                          ; send absolute coordinates...
M83                                          ; ...but relative extruder moves
M550 P"K6"                                   ; set printer name
G4 S2 ; wait for expansion boards to start
 
; Network
M552 P192.168.2.3 S1                             ; enable network and acquire dynamic address via DHCP
M586 P0 S1                                   ; enable HTTP
M586 P1 S0                                   ; disable FTP
M586 P2 S0                                   ; disable Telnet
 
; Drives
M569 P0.2 S0                                 ; physical drive 0.2 goes forwards
M569 P0.0 S0                                 ; physical drive 0.0 goes forwards
M569 P0.1 S1
M569 P0.3 S1                                 ; physical drive 0.3 goes forwards
M569 P40.0 S0 R0                            ; physical drive 0.4 goes forwards
M569 P40.0 T20:20:20:20
M569 P0.4 S0
M584 X0.2 Y0.0:0.1 Z0.3 U0.4 E40.0                  ; set drive mapping
M350 X8 Y16 Z8 U1 I1                                  ; configure microstepping with interpolation
M92 X320.00 Y132.84 Z320.00 U2500 E96             ; set steps per mm
M566 X30 Y100 Z30 U1 E5            ; set maximum instantaneous speed changes (mm/min)
M203 X450 Y6000 Z300 U10 E300     ; set maximum speeds (mm/min)
M201 X20.00 Y60.00 Z20.00 U1 E5        ; set accelerations (mm/s^2)
M906 X2500 Y4000 Z2500 U1500 I30              ; set motor currents (mA) and motor idle factor in per cent
M84 S30                                      ; Set idle timeout
 
; Axis Limits
M208 X0 Y0 Z0 U0 S1                             ; set axis minima
M208 X450 Y1000 Z180 U50 S0                        ; set axis maxima
 
; Endstops
M574 X1 S1 P"io3.in"                         ; configure switch-type (e.g. microswitch) endstop for low end on X via pin io0.in
M574 Y1 S1 P"io0.in"                         ; configure switch-type (e.g. microswitch) endstop for low end on Y via pin io2.in
M574 Z2 S1 P"io4.in"                         ; configure switch-type (e.g. microswitch) endstop for low end on Z via pin io3.in
 
';M574 U1 S1 P"!io5.in"
 
; Z-Probe
M950 S0 C"io7.out" ;
M558 P9 C"io7.in" H5 F120 T500 ;
G31 X33 Y-100 Z3 P25
M557 X50:150 Y300:400 S50
 
; Heaters
M308 S0 P"temp0" Y"thermistor" T100000 B4956 C1.587780e-7 A"Block" ;
M950 H0 C"out7" T0                           
M307 H0 B0 S1.00                             
M143 H0 S240                                 
M308 S1 P"temp1" Y"thermistor" T100000 B4725 C7.06e-8 A"PE-Upper";
M950 H1 C"out8" T1                           
M307 H1 B0 S1.00                             
M143 H1 S240                                 
M308 S2 P"temp2" Y"thermistor" T100000 B4725 C7.06e-8 A"PE-Lower";
M950 H2 C"out9" T2                           
M307 H2 B0 S1.00                             
M143 H2 S240
M308 S3 P"temp3" Y"thermistor" T100000 B4725 C7.06e-8 A"Nozzle";
M950 H3 C"out6" T3                           
M307 H3 B1 S1.00                             
M143 H3 S240
 
M570 H0 P3600 T15 
M570 H1 P3600 T15 
M570 H2 P3600 T15 
M570 H3 P3600 T15
 
 
; Fans
M950 F0 C"out4"                          ; create fan 0 on pin out4 and set its frequency
M106 P0 S0.3 H-1                               ; set fan 0 value. Thermostatic control is turned off
M950 F1 C"out5"                         ; create fan 1 on pin out5 and set its frequency
M106 P1 S0.3 H-1                            ; set fan 1 value. Thermostatic control is turned on
 
; Tools
M563 P0 D0 H0:1:2:3                          ; define tool 0
G10 P0 X0 Y0 Z0                              ; set tool 0 axis offsets
G10 P0 R0 S0                                 ; set initial tool 0 active and standby temperatures to 0C
 
 
T0
;M17
; Custom settings are not defined
 
 

[dsela]

*Tried again with steppers disabled

It doesn't let me edit for some reason

[dsela]

Came back to update that the issue has somehow resolved itself
The original 1XD board is working as normal..my best guess is that it had something to do with electrical noises..I have a few meters of cables going together in cable carriers, I took some out and tried a different path..maybe it did the trick.

Thanks for you support

[dc42]

@dsela thanks for updating us. Let us know if it starts happening again. If the Status light stops blinking again, please check whether the blue VIN and red 5V lights are still lit.