CAN connectivity: Duet 3 MB6HC to EXP3HC

dc42

@kroybal said in CAN connectivity: Duet 3 MB6HC to EXP3HC:

I have recently swapped out two Toolboard 1LCs, and my tool distribution board for an EXP3HC board. So far it has solved issues with my x-axes losing steps and resulting in monumental layer shifts.

Have you see this thread https://forum.duet3d.com/topic/20713/duet3-6ch-3ch-expansion-board-missing-steps/151?_=1611221812299 ? In short, we managed to reproduce this type of issue, identified the causes, and fixed them in the unofficial 3.3 beta release that we made available at https://www.dropbox.com/sh/wme9k0z86sytg33/AAAT6wrHp2eeJHK-dYoW1Um4a?dl=0.

@kroybal said in CAN connectivity: Duet 3 MB6HC to EXP3HC:

The diag LED did not power on whatsoever, upon pressing the reset key and restarting power dozens of times the light finally sync'd with the mainboard and installed the firmware successfully which was verified with M115 B126 command.

I have been investigating another instance of this problem today, and identified the cause (crystal oscillator taking a long time to start up, so that the frequency measuring code in the 3.2 release gives up). I have updated the EXP3HC firmware beta at https://www.dropbox.com/sh/wme9k0z86sytg33/AAAT6wrHp2eeJHK-dYoW1Um4a?dl=0. This update resolves the problem on the board that I was testing today.

Please note, the 3.3beta tool and expansion board firmware will only sync with 3.3beta main board firmware. However, even without connecting a main board running 3.3beta firmware, if the fix in the 3.3beta EXP3HC firmware works on your board then the LED on the 3HC board should start flashing as soon as you apply power.

kroybal

@pkos

sure thing

My Mainboard

M122 B0
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.2 running on Duet 3 MB6HC v1.01 or later (standalone mode)
Board ID: 08DJM-956L2-G43S8-6JKD8-3S06K-9B2GD
Used output buffers: 3 of 40 (38 max)
=== RTOS ===
Static ram: 149788
Dynamic ram: 94800 of which 40 recycled
Never used RAM 114204, free system stack 120 words
Tasks: NETWORK(ready,171) ETHERNET(blocked,109) HEAT(blocked,271) CanReceiv(blocked,848) CanSender(blocked,344) CanClock(blocked,352) TMC(blocked,19) MAIN(running,1091) IDLE(ready,19)
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 00:50:11 ago, cause: software
Last software reset at 2021-01-26 12:16, reason: User, GCodes spinning, available RAM 114244, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
Aux0 errors 0,0,0
Aux1 errors 0,0,0
MCU temperature: min 44.3, current 44.4, max 44.5
Supply voltage: min 24.3, current 24.3, max 24.4, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 11.0, current 12.4, max 12.8, under voltage events: 0
Driver 0: position 0, standstill, reads 18998, writes 0 timeouts 0, SG min/max not available
Driver 1: position 5384, standstill, reads 18998, writes 0 timeouts 0, SG min/max not available
Driver 2: position 29174, standstill, reads 18998, writes 0 timeouts 0, SG min/max not available
Driver 3: position 47088, standstill, reads 18999, writes 0 timeouts 0, SG min/max not available
Driver 4: position 0, standstill, reads 18998, writes 0 timeouts 0, SG min/max not available
Driver 5: position 0, standstill, reads 18998, writes 0 timeouts 0, SG min/max not available
Date/time: 2021-01-26 13:07:01
Slowest loop: 7.73ms; fastest: 0.06ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 2.2ms, write time 1.1ms, max retries 0
=== Move ===
DMs created 125, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 19726, completed moves 19726, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = 3 -1 -1 -1
Heater 0 is on, I-accum = 0.0
Heater 2 is on, I-accum = 0.2
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is ready with "M122 B0" in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 24.77ms; fastest: 0.02ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions
HTTP sessions: 2 of 8

Ethernet -
State: active
Error counts: 0 0 0 0 0
Socket states: 5 2 2 2 2 0 0 0
=== Filament sensors ===
Extruder 0: pos 3.32, errs: frame 1 parity 0 ovrun 8 pol 0 ovdue

My Expansion board

M122 B15
Diagnostics for board 15:
Duet EXP3HC firmware version 3.2 (2021-01-05)
Bootloader ID: not available
Never used RAM 154796, free system stack 154 words
HEAT 83 CanAsync 94 CanRecv 78 TMC 30 MAIN 323 AIN 257
Last reset 00:50:15 ago, cause: software
Last software reset data not available
Driver 0: position -36704, 80.9 steps/mm, standstill, reads 53420, writes 22 timeouts 0, SG min/max 0/121
Driver 1: position 57777, 80.9 steps/mm, standstill, reads 53422, writes 22 timeouts 0, SG min/max 0/130
Driver 2: position 0, 80.0 steps/mm, standstill, reads 53437, writes 11 timeouts 0, SG min/max 0/0
Moves scheduled 18712, completed 18712, in progress 0, hiccups 0
No step interrupt scheduled
VIN: 24.4V, V12: 12.2V
MCU temperature: min 43.6C, current 43.9C, max 44.1C
Ticks since heat task active 240, ADC conversions started 3015982, completed 3015980, timed out 0
Last sensors broadcast 0x00000000 found 0 243 ticks ago, loop time 0
CAN messages queued 95, send timeouts 0, received 45924, lost 0, free buffers 36

kroybal

@dc42 good to hear. I will read that thread after lunch.

@dc42 said in CAN connectivity: Duet 3 MB6HC to EXP3HC:

@kroybal said in CAN connectivity: Duet 3 MB6HC to EXP3HC:

The diag LED did not power on whatsoever, upon pressing the reset key and restarting power dozens of times the light finally sync'd with the mainboard and installed the firmware successfully which was verified with M115 B126 command.

I have been investigating another instance of this problem today, and identified the cause (crystal oscillator taking a long time to start up, so that the frequency measuring code in the 3.2 release gives up). I have updated the EXP3HC firmware beta at https://www.dropbox.com/sh/wme9k0z86sytg33/AAAT6wrHp2eeJHK-dYoW1Um4a?dl=0. This update resolves the problem on the board that I was testing today.

Please note, the 3.3beta tool and expansion board firmware will only sync with 3.3beta main board firmware. However, even without connecting a main board running 3.3beta firmware, if the fix in the 3.3beta EXP3HC firmware works on your board then the LED on the 3HC board should start flashing as soon as you apply power.

You're recommending updating all boards to the 3.3beta?

dc42

@kroybal said in CAN connectivity: Duet 3 MB6HC to EXP3HC:

@dc42 good to hear. I will read that thread after lunch.

@dc42 said in CAN connectivity: Duet 3 MB6HC to EXP3HC:

@kroybal said in CAN connectivity: Duet 3 MB6HC to EXP3HC:

The diag LED did not power on whatsoever, upon pressing the reset key and restarting power dozens of times the light finally sync'd with the mainboard and installed the firmware successfully which was verified with M115 B126 command.

I have been investigating another instance of this problem today, and identified the cause (crystal oscillator taking a long time to start up, so that the frequency measuring code in the 3.2 release gives up). I have updated the EXP3HC firmware beta at https://www.dropbox.com/sh/wme9k0z86sytg33/AAAT6wrHp2eeJHK-dYoW1Um4a?dl=0. This update resolves the problem on the board that I was testing today.

Please note, the 3.3beta tool and expansion board firmware will only sync with 3.3beta main board firmware. However, even without connecting a main board running 3.3beta firmware, if the fix in the 3.3beta EXP3HC firmware works on your board then the LED on the 3HC board should start flashing as soon as you apply power.

You're recommending updating all boards to the 3.3beta?

Start by just updating just the 3HC board, and test whether after the update the LED on that board always starts flashing as soon as you apply power (but don't expect it to sync with the main board). If it does, upgrade the other boards too.

kroybal

@dc42 said in CAN connectivity: Duet 3 MB6HC to EXP3HC:

Start by just updating just the 3HC board, and test whether after the update the LED on that board always starts flashing as soon as you apply power (but don't expect it to sync with the main board). If it does, upgrade the other boards too.

So upon updating EXP3HC to 3.3beta and keeping the MB6HC at 3.2 and resetting the power resulted in the LED not blinking upon startup. Should I move forward with running a full 3.3beta?

kroybal

Running 3.3beta on the EXP3HC and 3.2 on the MB6HC resulted in warning message appearing periodically in the console upon a G28 homing command. Both my x-axis motors and their endstops are connected to the 3HC board.

1/26/2021, 1:47:36 PM Warning: Discarded msg src=15 typ=4510 RID=4095 exp 33

1/26/2021, 1:47:09 PM Warning: Discarded msg src=15 typ=4510 RID=4095 exp 31

Warning: Discarded msg src=15 typ=4510 RID=4095 exp 31

1/26/2021, 1:46:54 PM Warning: Discarded msg src=15 typ=4510 RID=4095 exp 28

1/26/2021, 1:46:47 PM Warning: Discarded msg src=15 typ=4510 RID=4095 exp 26

Warning: Discarded msg src=15 typ=4510 RID=4095 exp 26

1/26/2021, 1:46:32 PM Warning: Discarded msg src=15 typ=4510 RID=4095 exp 23

1/26/2021, 1:46:26 PM Warning: Discarded msg src=15 typ=4510 RID=4095 exp 21

Warning: Discarded msg src=15 typ=4510 RID=4095 exp 21

1/26/2021, 1:46:11 PM Warning: Discarded msg src=15 typ=4510 RID=4095 exp 18

1/26/2021, 1:45:47 PM Warning: Discarded msg src=15 typ=4510 RID=4095 exp 15

Warning: Discarded msg src=15 typ=4510 RID=4095 exp 15

dc42

Those messages are expected because of differences in the CAN protocol between 3.2 and 3.3.

kroybal

@dc42 Understood, I ended up updating to the 3.3beta firmware on both boards. It still requires a bit of patience to let the boards sync up.

jrockland

@kroybal hmm but why 15 ? If you have just a 6hc+3hc shouldnt it be address to 1 ? Or im missing something here

dc42

@kroybal said in CAN connectivity: Duet 3 MB6HC to EXP3HC:

@dc42 Understood, I ended up updating to the 3.3beta firmware on both boards. It still requires a bit of patience to let the boards sync up.

How long does it take? If the fix is working then the LED on the expansion board should start flashing as soon as it received power (as evidenced by the other LEDs on the board), but may take about 2 seconds before it settles down to the slow flash indicating CAN sync. If the other LEDs on the expansion board light up immediately power is applied but the status LED still does not flash until many seconds later, then I need to revisit the fix.

kroybal

@dc42 Following a full update to 3.3beta seemed to be working at this point. It doesn't light up immediately but it flashes about two seconds later (w.r.t. the other LEDs) overall takes less than 10 seconds to sync with the MB. I cannot complain!

Cheers

Edit: I should have bit my tongue, I tried another time and the LED failed to turn on all together following power up. It has been about a minute and no light.

kroybal

@jrockland said in CAN connectivity: Duet 3 MB6HC to EXP3HC:

@kroybal hmm but why 15 ? If you have just a 6hc+3hc shouldnt it be address to 1 ? Or im missing something here

Addressing is arbitrary right? I certainly do not have tons of boards and channel 1 is free but I found it easiest to just flip them all to on in one motion instead of trying to be precise with flipping one switch.

Edit: Not totally arbitrary, but no right or wrong answer as long as its logical

kroybal

Very coarse statistics on my setup suggest about a 1/15 shot at lighting up upon startup, and 2 minutes light-up-and-sync time in the case of a startup failure.

Does this have to do with the crystal oscillators not powering up still or simply waiting for the signal from the boards to be in phase?

jrockland

@kroybal not a clue, it is why I was asking. I tough I was wrong doing it 1-2-3...

dc42

@kroybal said in CAN connectivity: Duet 3 MB6HC to EXP3HC:

@dc42 Following a full update to 3.3beta seemed to be working at this point. It doesn't light up immediately but it flashes about two seconds later (w.r.t. the other LEDs) overall takes less than 10 seconds to sync with the MB. I cannot complain!

Cheers

Edit: I should have bit my tongue, I tried another time and the LED failed to turn on all together following power up. It has been about a minute and no light.

Thanks for the update. I'll continue working on this and provide another update tomorrow.

dc42

@kroybal, I've put an updated 3.3beta Duet3Firmware-EXP3HC.bin file at https://www.dropbox.com/sh/wme9k0z86sytg33/AAAT6wrHp2eeJHK-dYoW1Um4a?dl=0. Please install it and see whether the delay from power up to when the LED on the 3HC starts flashing is eliminated.

kroybal

@dc42 I have installed the updated EXP3HC and it seems like the problem is still persisting.

Here is my M122 command for both boards

1/28/2021, 10:01:52 AM === Diagnostics ===

RepRapFirmware for Duet 3 MB6HC version 3.3beta running on Duet 3 MB6HC v1.01 or later (standalone mode)

Board ID: 08DJM-956L2-G43S8-6JKD8-3S06K-9B2GD

Used output buffers: 1 of 40 (13 max)

=== RTOS ===

Static ram: 149784

Dynamic ram: 92536 of which 0 recycled

Never used RAM 108352, free system stack 200 words

Tasks: NETWORK(ready,254) ETHERNET(blocked,111) HEAT(blocked,305) CanReceiv(blocked,909) CanSender(blocked,365) CanClock(blocked,334) TMC(blocked,52) MAIN(running,1120) IDLE(ready,20)

Owned mutexes: LwipCore(NETWORK) HTTP(MAIN)

=== Platform ===

Last reset 00:00:13 ago, cause: software

Last software reset at 2021-01-28 09:57, reason: User, GCodes spinning, available RAM 108312, slot 2

Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a

Error status: 0x00

Aux0 errors 0,0,0

Aux1 errors 0,0,0

MCU temperature: min 38.6, current 40.3, max 40.5

Supply voltage: min 24.3, current 24.3, max 24.4, under voltage events: 0, over voltage events: 0, power good: yes

12V rail voltage: min 10.8, current 12.6, max 12.8, under voltage events: 0

Driver 0: position 0, standstill, reads 10313, writes 14 timeouts 0, SG min/max 0/0

Driver 1: position 0, standstill, reads 10313, writes 14 timeouts 0, SG min/max 0/0

Driver 2: position 0, standstill, reads 10313, writes 14 timeouts 0, SG min/max 0/0

Driver 3: position 0, standstill, reads 10313, writes 14 timeouts 0, SG min/max 0/0

Driver 4: position 0, standstill, reads 10314, writes 14 timeouts 0, SG min/max 0/0

Driver 5: position 0, standstill, reads 10314, writes 14 timeouts 0, SG min/max 0/0

Date/time: 2021-01-28 10:01:53

Slowest loop: 4.11ms; fastest: 0.05ms

=== Storage ===

Free file entries: 10

SD card 0 detected, interface speed: 25.0MBytes/sec

SD card longest read time 2.3ms, write time 0.0ms, max retries 0

=== Move ===

DMs created 125, maxWait 0ms, bed compensation in use: none, comp offset 0.000

=== MainDDARing ===

Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1

=== AuxDDARing ===

Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1

=== Heat ===

Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = 3 -1 -1 -1

Heater 0 is on, I-accum = 0.0

=== GCodes ===

Segments left: 0

Movement lock held by null

HTTP is ready with "M122 B0" in state(s) 0

Telnet is idle in state(s) 0

File is idle in state(s) 0

USB is idle in state(s) 0

Aux is idle in state(s) 0

Trigger is idle in state(s) 0

Queue is idle in state(s) 0

LCD is idle in state(s) 0

SBC is idle in state(s) 0

Daemon is idle in state(s) 0

Aux2 is idle in state(s) 0

Autopause is idle in state(s) 0

Code queue is empty.

=== Filament sensors ===

Extruder 0: pos 0.45, errs: frame 0 parity 0 ovrun 0 pol 0 ovdue 0

Extruder 1: pos 3.66, errs: frame 0 parity 0 ovrun 0 pol 0 ovdue 0

=== CAN ===

Messages queued 74, send timeouts 0, received 20, lost 0, longest wait 1ms for reply type 6018, peak Tx sync delay 493, free buffers 48 (min 47)

=== Network ===

Slowest loop: 6.40ms; fastest: 0.02ms

Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions

HTTP sessions: 1 of 8

Ethernet -

State: active

Error counts: 0 0 1 0 0

Socket states: 5 2 2 2 2 0 0 0

1/28/2021, 10:02:36 AM M122 B15

Diagnostics for board 15:

Duet EXP3HC firmware version 3.3beta (2021-01-28 09:11:08)

Bootloader ID: not available

Never used RAM 155236, free system stack 0 words

Move 160 HEAT 103 CanAsync 72 CanRecv 84 CanClock 74 TMC 64 MAIN 339 AIN 260

Last reset 00:00:13 ago, cause: software

Last software reset data not available

Driver 0: position 0, 80.9 steps/mm, standstill, reads 46408, writes 16 timeouts 0, SG min/max 0/0, steps req 0 done 0

Driver 1: position 0, 80.9 steps/mm, standstill, reads 46410, writes 16 timeouts 0, SG min/max 0/0, steps req 0 done 0

Driver 2: position 0, 80.0 steps/mm, standstill, reads 46419, writes 11 timeouts 0, SG min/max 0/0, steps req 0 done 0

Moves scheduled 0, completed 0, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0

Peak sync jitter 12, peak Rx sync delay 174, resyncs 0, no step interrupt scheduled

VIN: 24.4V, V12: 12.2V

MCU temperature: min 43.6C, current 43.6C, max 43.9C

Ticks since heat task active 90, ADC conversions started 13832, completed 13831, timed out 0

Last sensors broadcast 0x00000000 found 0 94 ticks ago, loop time 0

CAN messages queued 33, send timeouts 0, received 136, lost 0, free buffers 36, min 36, error reg 110067

dup 0, oos 0, bm 0, wbm 0

kroybal

One difference I have noticed is that my EXP3HC board flashes without sync now and does not connect to the mainboard whatsoever even after waiting over five minutes at this point.

dc42

@kroybal said in CAN connectivity: Duet 3 MB6HC to EXP3HC:

One difference I have noticed is that my EXP3HC board flashes without sync now and does not connect to the mainboard whatsoever even after waiting over five minutes at this point.

Please run M115 and M115 B1 and confirm that you see the following firmware versions and dates reported:

M115
FIRMWARE_NAME: RepRapFirmware for Duet 3 MB6HC FIRMWARE_VERSION: 3.3beta ELECTRONICS: Duet 3 MB6HC v0.6 or 1.0 FIRMWARE_DATE: 2021-01-21 09:00:34

M115 B1
Duet EXP3HC firmware version 3.3beta (2021-01-28 09:11:08)

(the M115 B1 command may work even though you don't have CAN sync).

Does the LED on the 3HC start flashing immediately when you power it up?
What board revision is your 3HC?

Thanks for your patience.

dc42

Hi @kroybal,

I guess you had better things to do over the weekend than work on this!

In the absence of feedback from you, I've taken a guess at what might be causing the loss of sync, and put new firmware binaries (both 6HC and 3HC) with today's build date at https://www.dropbox.com/sh/qr98k8fbkj5ue0k/AABPawUF99QVzDrheBQBDSxia?dl=0. Please try them and let me know whether they resolve the loss of sync.

Which country are you in?