3HC Boot Issues

FlavaMuffin · 12 Apr 2021, 21:01

I am having an issue when trying to use the 3HC expansion board connected to the Duet 3 Main Board. I think it may be the same problem that "Max 0" had in his topic titled "Duet 3 EB 3HC intermittent CAN/boot? failure" that he posted earlier this month. When I startup the boards the mainboard does not recognize the expansion board, but the diagnostic light for starts blinking in time with the main board soon after startup. Even with the lights indicating a working connection a M112 command returns "Error: M122: Response timeout: CAN addr 1, req type 6024, RID=17". If I do a software restart of the mainboard using M999 it starts recognizing the expansion board M112 sends a valid response.

I presumed that the expansion board just needed longer to startup and added "G4 S10" to my main config in order to give it extra time, but the problem still persisted. I have yet to try increasing the Dwell time further as I figured this was a rather boot time for the expansion board and I noticed "Max 0" having similar issues. Should I try increasing the Dwell time further, or is this unexpected board behavior?

Since "Max 0" had a similar issue and my board has the same crystal as his, I am including a picture of the crystal, as well as the logs from the M112 commands. (Both of my boards appear to have the same crystal, but the picture is from the expansion board.)

Expansion Board Crystal:

M112 Output Pre M999 Command:

(M112)
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.2.2 running on Duet 3 MB6HC v1.01 or later (standalone mode)
Board ID: 08DJM-956BA-NA3TN-6J1F4-3S46S-KBAGT
Used output buffers: 1 of 40 (14 max)
=== RTOS ===
Static ram: 149788
Dynamic ram: 93868 of which 0 recycled
Never used RAM 115176, free system stack 192 words
Tasks: NETWORK(ready,201) ETHERNET(blocked,110) HEAT(blocked,294) CanReceiv(blocked,927) CanSender(blocked,371) CanClock(blocked,352) TMC(blocked,51) MAIN(running,1117) IDLE(ready,19)
Owned mutexes:
=== Platform ===
Last reset 00:01:41 ago, cause: power up
Last software reset at 2021-04-12 13:57, reason: User, GCodes spinning, available RAM 115140, slot 1
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0044a000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
Aux0 errors 0,0,0
Aux1 errors 0,0,0
MCU temperature: min 20.6, current 34.0, max 34.0
Supply voltage: min 24.0, current 24.0, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.1, current 12.2, max 12.3, under voltage events: 0
Driver 0: position 0, standstill, reads 49282, writes 14 timeouts 0, SG min/max 0/0
Driver 1: position 0, standstill, reads 49282, writes 14 timeouts 0, SG min/max 0/0
Driver 2: position 0, standstill, reads 49282, writes 14 timeouts 0, SG min/max 0/0
Driver 3: position 0, standstill, reads 49283, writes 14 timeouts 0, SG min/max 0/0
Driver 4: position 0, standstill, reads 49283, writes 14 timeouts 0, SG min/max 0/0
Driver 5: position 0, standstill, reads 49283, writes 14 timeouts 0, SG min/max 0/0
Date/time: 2021-04-12 14:12:48
Slowest loop: 3.97ms; fastest: 0.05ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 2.3ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 3.47ms; fastest: 0.02ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions
HTTP sessions: 1 of 8

Ethernet -
State: active
Error counts: 0 0 1 0 0
Socket states: 5 2 2 2 2 0 0 0
=== CAN ===
Messages queued 392, send timeouts 895, received 0, lost 0, longest wait 0ms for reply type 0, free buffers 48

(M112 B1)
Error: M122: Response timeout: CAN addr 1, req type 6024, RID=17

M112 Output Post M999 Command:

(M112)
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.2.2 running on Duet 3 MB6HC v1.01 or later (standalone mode)
Board ID: 08DJM-956BA-NA3TN-6J1F4-3S46S-KBAGT
Used output buffers: 3 of 40 (14 max)
=== RTOS ===
Static ram: 149788
Dynamic ram: 94420 of which 88 recycled
Never used RAM 114536, free system stack 182 words
Tasks: NETWORK(ready,165) ETHERNET(blocked,110) HEAT(blocked,294) CanReceiv(blocked,834) CanSender(blocked,371) CanClock(blocked,352) TMC(blocked,51) MAIN(running,1117) IDLE(ready,19)
Owned mutexes:
=== Platform ===
Last reset 00:05:26 ago, cause: software
Last software reset at 2021-04-12 14:15, reason: User, GCodes spinning, available RAM 115140, slot 2
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
Aux0 errors 0,0,0
Aux1 errors 0,0,0
MCU temperature: min 35.7, current 37.5, max 37.5
Supply voltage: min 23.9, current 24.0, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.1, current 12.2, max 12.3, under voltage events: 0
Driver 0: position 0, standstill, reads 11280, writes 14 timeouts 0, SG min/max 0/0
Driver 1: position 0, standstill, reads 11280, writes 14 timeouts 0, SG min/max 0/0
Driver 2: position 0, standstill, reads 11280, writes 14 timeouts 0, SG min/max 0/0
Driver 3: position 0, standstill, reads 11280, writes 14 timeouts 0, SG min/max 0/0
Driver 4: position 0, standstill, reads 11280, writes 14 timeouts 0, SG min/max 0/0
Driver 5: position 0, standstill, reads 11281, writes 14 timeouts 0, SG min/max 0/0
Date/time: 2021-04-12 14:20:27
Slowest loop: 4.51ms; fastest: 0.05ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 2.5ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 2.70ms; fastest: 0.02ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions
HTTP sessions: 1 of 8

Ethernet -
State: active
Error counts: 0 0 1 0 0
Socket states: 5 2 2 2 2 0 0 0
=== CAN ===
Messages queued 1324, send timeouts 0, received 3929, lost 0, longest wait 42ms for reply type 6031, free buffers 48

(M112 B1)
Diagnostics for board 1:
Duet EXP3HC firmware version 3.2.2 (2021-02-12)
Bootloader ID: not available
Never used RAM 154236, free system stack 178 words
HEAT 92 CanAsync 94 CanRecv 87 TMC 64 MAIN 216 AIN 259
Last reset 00:05:30 ago, cause: software
Last software reset data not available
Driver 0: position 0, 80.0 steps/mm, standstill, reads 23677, writes 11 timeouts 0, SG min/max 0/0
Driver 1: position 0, 80.0 steps/mm, standstill, reads 23679, writes 11 timeouts 0, SG min/max 0/0
Driver 2: position 0, 80.0 steps/mm, standstill, reads 23682, writes 11 timeouts 0, SG min/max 0/0
Moves scheduled 0, completed 0, in progress 0, hiccups 0
No step interrupt scheduled
VIN: 24.8V, V12: 12.3V
MCU temperature: min 41.9C, current 42.1C, max 42.1C
Ticks since heat task active 20, ADC conversions started 330262, completed 330261, timed out 0
Last sensors broadcast 0x00000018 found 2 23 ticks ago, loop time 0
CAN messages queued 3993, send timeouts 0, received 2992, lost 0, free buffers 36

FlavaMuffin · 13 Apr 2021, 05:08

I attempted adding a 30 second delay using G4 and still have had no luck.

pkos · 13 Apr 2021, 10:08

@flavamuffin Did you try 3.3 beta 2?

FlavaMuffin · 13 Apr 2021, 15:49

@pkos I have yet to try that that. I will later today and see if it remedies the problem.

dc42 · 13 Apr 2021, 18:05

@flavamuffin said in 3HC Boot Issues:

@pkos I have yet to try that that. I will later today and see if it remedies the problem.

If you install 3.3beta2 on the expansion board, you will also need to use 3.2beta2 on the main board, otherwise you will get only partial functionality.

A symptom of the slow startup issue is that after powering up the 3HC, the red LED stays off for a noticeable period of time, several seconds or more. Do you observe that?

FlavaMuffin · 13 Apr 2021, 22:20

@dc42 I have updated to 3.3beta2 on both boards and the problem still persists. I presume the red LED you are referring to is the diagnostic one by the RJ11 ports. That one remains off for roughly a minute after startup. Before updating from 3.2.2 to the 3.3beta2 it immediately started flashing in time with the main board once lit. Now it flashes rapidly for about 1 second after it initially lights up before flashing in time with the main board. If I am wrong in my presumption about which LED you are referring to, and you do mean the one on the corner of the board next to the other LEDs; that one turns on immediately.

dc42 · 14 Apr 2021, 06:22

@flavamuffin thanks, I did mean the LED between the CAN ports. Which country are you in?

FlavaMuffin · 14 Apr 2021, 16:42

@dc42 I am in the USA.

dc42 · 16 Apr 2021, 07:09

@flavamuffin, I am still trying to track down what causes this slow startup issue on a few boards, but unfortunately I still do not have a board that exhibits this issue. So I am having to make guesses about what is causing it.

Please can you try the Duet3Firware_EXP3HC.bin file at https://www.dropbox.com/sh/dupd9u1xppr1om5/AADYg4guuj2n8LJokVqtK-C7a?dl=0. This should identify itself (using M115 B1) as "Duet EXP3HC firmware version 3.3beta2+2 (2021-04-16 07:53:14)".

If that firmware still shows the slow startup issue, please try the firmware at https://www.dropbox.com/sh/fgpjmta9fut21gb/AADgv9v7MRqywjFj9vOXLvOra?dl=0. This should identify itself as "Duet EXP3HC firmware version 3.3beta2+2-12MHz (2021-04-16 08:04:22)".

If that also exhibits the slow startup issue, please contact your supplier to have your board replaced under warranty.

By "slow startup", I mean that after applying power the red LED between the CAN connectors takes longer than about a second to start flashing.

Do not attempt to update the bootloader on your EXP3HC.

FlavaMuffin · 16 Apr 2021, 17:14

@dc42 Firmware version 3.3beta2+2 did not seem to have any effect on the problem. Version 3.3beta2+2-12MHz seems to have worsened the issue. It causes the board to take several minutes to initialize. I am going to start an inquiry with my supplier about replacing the board.

dc42 · 16 Apr 2021, 17:34

@flavamuffin, thanks for trying those. I authorize a warranty replacement for your board.

FlavaMuffin · 16 Apr 2021, 17:42

@dc42 Thanks for all your help! I hope you end up finding the cause for the issue.