Trouble connecting Toolboards 1LC to Duet 3 6HC

MathV

Hi,

I am trying to connect 2 toolboards 1LC to a Duet 3 6HC.

Configuration:

To do so, I am not using the Duet 3 Tool Distribution Board, but I chained my 2 toolboards 1LC with the 6HC as below:
CAN1_L (6HC) --> CAN_L (1LC#1) --> CAN_L (1LC#2) --> R120 --> CAN_H (1LC#2) --> CAN_H (1LC#1) --> CAN1_H (6HC)
I am using non twisted pair cable (total length 6m).
The toolboards 1LC are V1.0 and the mainboard 6HV is V1.01.
I am powering the toolboards with 24V from the OUT1 (1LC#1) and OUT2 (1LC#2).
I am using a SBC to pilot the mainboard 6HC, and the SBC powers the 6HC with 5V.
The SBC and the 6HC are not powered from a common power supply, so there are 2 power supplies (220V/5V for the SBC and 220V/24V for the 6HC)
I am using the 3.2.2 firmware release.

What I did before:
I succeeded in "pinging" the two boards sparetly with a M115 B121.
I changed the ID of the boards to 20 and 21 with M952 B121 A20 and M952 B121 A21, rebooted the boards with M999 B121, and controled the ID change by powering them both with a M115.

Issue:
The next day (machine switched off over night), I struggle to connect with the toolboards: the red LED blink rapidely on one tool, and/or the other.
The connection issue is confirmed when I send a M409 K"boards" (only the 6HC visible, and the 1LC eventually connected).
Since then, I do not succeed in establishing a connection between the 6HC and the two 1LC (not even one of them any more).

Investigation:
Here are the operations I realised in order to find the root cause of the issue:

I checked my cables: CAN_H connected together, CAN_L connected together and 120ohm on the RJ11 connector. --> Cable mapping OK
I connected only one 1LC with a short cable (~60cm, but still untwisted pair cable). --> No effect
I changed the 1LC (I purchased 4 of them). --> No effect
I resetted the 1LC by powering it up with both buttons pushed. --> No effect
I installed the latest duet firmware version (3.3RC3). --> No effect

To be tested:
I will now try to power the SBC with a 24V/5V converter, so both the 6HC and the SBC are powered by the 24V power supply (even though indirectly for the SBC).

I start to be out of idea, and a bit of help would be very welcome.
Thanks a lot.

Mathieu

PaulHew

@mathv First thing I noticed was this.
I am powering the toolboards with 24V from the OUT1 (1LC#1) and OUT2 (1LC#2).

Why not take a separate feed from the 24V PSU via an inline fuse or two to the 2 1LC's??
They then have a permanent power feed and are also protected if you use the inline fuse.

Everytime you make a config change I am guessing it drops the power to those outputs until it gets to the part where the 'Heaters' are turned back on.

Your CAN bus seems sound.

MathV

@paulhew said in Trouble connecting Toolboards 1LC to Duet 3 6HC:

Why not take a separate feed from the 24V PSU via an inline fuse or two to the 2 1LC's??

I configured OUT1 and OUT2 as outputs (not heater), so I can power the tools even before configuring the drivers.
The idea behind it is to be able to change the tools while the printer is running.

MathV

@mathv said in Trouble connecting Toolboards 1LC to Duet 3 6HC:

To be tested:
I will now try to power the SBC with a 24V/5V converter, so both the 6HC and the SBC are powered by the 24V power supply (even though indirectly for the SBC).

I just tested it. --> No effect
I also tested to power the SBC with the 6HC. --> No effect

MathV

@mathv said in Trouble connecting Toolboards 1LC to Duet 3 6HC:

Investigation:

In case it could give more information about the issue:
I tried to set back the CAN bus data rate (in case I would have change it by mistake) with:

M953 S1.0

The board responds:
Error: M953: Command is not supported

Phaedrux

Can you please post your full config.g and the results of M122 for the 6HC and hopefully for the tool boards as well? The firmware on all 3 must match.

@mathv said in Trouble connecting Toolboards 1LC to Duet 3 6HC:

I struggle to connect with the toolboards: the red LED blink rapidely on one tool, and/or the other

Is this still the case? Just rapid blinking? You may need to add a G4 S5 delay near the start of config.g to allow for time for the tool boards to startup before any commands can be sent to them.

MathV

Thanks @phaedrux

Please find below my actual config.g file (I will replace the 1s delay by a 5s delay, thanks for this information):

; Configuration file for Duet 3 (firmware version 3)
; executed by the firmware on start-up
;
; generated by RepRapFirmware Configuration Tool v3.2.3 on Tue Apr 06 2021 17:21:14 GMT+0200 (CEST)

; General preferences
G90                                                       ; send absolute coordinates...
M83                                                       ; ...but relative extruder moves
M550 P"ubuntu"                                            ; set printer name
G21                                                       ; work in millimetres
												          
; Tools                                                   
M950 P0 C"0.out1"                                         ; create the power port of the tool left on P0
M950 P1 C"0.out2"                                         ; create the power port of the tool right on P1
M42 P0 S1.0                                               ; switch on tool 0 (left)
M42 P1 S1.0                                               ; switch on tool 1 (right)
G4 S1                                                     ; wait 1 second for the tools to init
												          
; Drives                                                  
M569 P0.5 S1                                              ; physical drive 0.5 goes forwards
M569 P0.4 S0                                              ; physical drive 0.4 goes forwards
M569 P0.3 S1                                              ; physical drive 0.3 goes forwards
M569 P0.2 S1                                              ; physical drive 0.2 goes forwards
M569 P0.1 S1                                              ; physical drive 0.1 goes forwards
M569 P0.0 S1                                              ; physical drive 0.0 goes forwards
M584 X0.3 Y0.2 Z0.1:0.0 U0.4 V0.5                         ; set drive mapping
M350 X16 Y16 Z16 U16 I1                                   ; configure microstepping with interpolation
M92 X53.33 Y53.33 Z800.00 U53.33 V163.00                  ; set steps per mm
M566 X900.00 Y900.00 Z60.00 U900.00 V900.00               ; set maximum instantaneous speed changes (mm/min)
M203 X18000.00 Y18000.00 Z1000.00 U18000.00 V5000.00      ; set maximum speeds (mm/min)
M201 X2000.00 Y2000.00 Z20.00 U2000.00 V250.00            ; set accelerations (mm/s^2)
M906 X1150 Y1500 Z1150 U1150 V1200 I60                    ; set motor currents (mA) and motor idle factor in per cent
M84 S0                                                    ; Disable idle timeout

; Axis Limits
M208 X-150 Y-150 Z0 U-150 V0 S1                           ; set axis minima
M208 X150 Y150 Z300 U150 V10000 S0                        ; set axis maxima

; Endstops
M574 X1 S1 P"!0.io4.in"                                   ; configure active-high endstop for low end on X via pin !io3.in
M574 Y2 S1 P"!0.io2.in"                                   ; configure active-high endstop for high end on Y via pin !io2.in
M574 Z2 S1 P"!0.io0.in+!0.io1.in"                         ; configure active-high endstop for high end on Z via pin !io0.in
M574 U2 S1 P"!0.io3.in"                                   ; configure active-high endstop for high end on U via pin !io4.in
M574 V1 S1 P"^0.io5.in"                                   ; configure active-high endstop for low end on V via pin !^0.io5.in

; Heaters                                                       
M308 S0 P"0.temp0" Y"thermistor" A"Bed" T100000 B3950     ; configure sensor 0 as thermistor on pin temp0
M950 H0 C"0.out7" T0                                      ; create bed heater output on out7 and map it to sensor 0
M307 H0 B0 S1.00                                          ; disable bang-bang mode for the bed heater and set PWM limit
M140 H0                                                   ; map heated bed to heater 0
M307 H0 R0.791 C392.0 D4.77 S1.00                         ; set the heating parameters of the bed
M143 H0 P0 S110                                           ; raise a heater fault if heater 0 exceeds 110C
M143 H0 P1 S120 A3                                        ; shut down the printer if heater 0 exceeds 120C
													  
; Fans                                                    
M950 F2 C"0.out9" Q500                                    ; create fan 2 on pin out9 and set its frequency
M106 P2 S1.0 H-1 C"Material and Electronics"              ; set fan 2 value. Thermostatic control is turned off
														  
; Light                                                   
M950 P2 C"0.out3"                                         ; create the power port of the light on P2
M42 P2 S1.0                                               ; set P2 to 100%
														  														  													  														  
; Miscellaneous                                           
M911 S21 R22 P"M913 X0 Y0 G91 M83 G1 Z3 E-5 F1000"        ; set voltage thresholds and actions to run on power loss

I, unfortunately, lately cannot communicate with the tools. I already had some issues when I was on firmware version 3.2.2, and no connection at all since I upgraded to 3.3RC3.
Please find below the M112 result (6HC only here with firmware 3.3RC3):

01/06/2021, 08:45:35: m122: === Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.3RC3 (2021-05-26 12:29:42) running on Duet 3 MB6HC v0.6 or 1.0 (SBC mode)
Board ID: 08DJM-956BA-NA3TJ-6J1DD-3S86T-KU9AS
Used output buffers: 1 of 40 (12 max)
=== RTOS ===
Static ram: 150784
Dynamic ram: 61428 of which 0 recycled
Never used RAM 141980, free system stack 200 words
Tasks: SBC(ready,5.1%,298) HEAT(delaying,0.0%,301) Move(notifyWait,0.0%,352) CanReceiv(notifyWait,0.0%,945) CanSender(notifyWait,0.0%,374) CanClock(delaying,0.0%,333) TMC(notifyWait,7.6%,93) MAIN(running,87.2%,1272) IDLE(ready,0.0%,29), total 100.0%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 00:04:40 ago, cause: power up
Last software reset at 2021-05-31 14:50, reason: User, none spinning, available RAM 141372, slot 2
Software reset code 0x0012 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0044a000 BFAR 0x00000000 SP 0x00000000 Task SBC Freestk 0 n/a
Error status: 0x00
Aux0 errors 0,0,0
Aux1 errors 0,0,0
Step timer max interval 139
MCU temperature: min 23.1, current 35.8, max 35.9
Supply voltage: min 24.0, current 24.1, max 24.2, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.1, current 12.1, max 12.2, under voltage events: 0
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Driver 0: position 0, standstill, reads 59521, writes 14 timeouts 0, SG min/max 0/0
Driver 1: position 0, standstill, reads 59521, writes 14 timeouts 0, SG min/max 0/0
Driver 2: position 0, standstill, reads 59521, writes 14 timeouts 0, SG min/max 0/0
Driver 3: position 0, standstill, reads 59521, writes 14 timeouts 0, SG min/max 0/0
Driver 4: position 0, standstill, reads 59522, writes 14 timeouts 0, SG min/max 0/0
Driver 5: position 0, standstill, reads 59523, writes 13 timeouts 0, SG min/max 0/0
Date/time: 2021-06-01 06:45:32
Slowest loop: 0.46ms; fastest: 0.04ms
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 37.5MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = 3 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP* is doing "M122" in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== CAN ===
Messages queued 2338, send timeouts 2332, received 0, lost 0, longest wait 0ms for reply type 0, peak Tx sync delay 0, free buffers 49 (min 49), ts 1402/0/0
Last cancelled message type 4514 dest 127
=== SBC interface ===
State: 4, failed transfers: 0
Last transfer: 1ms ago
RX/TX seq numbers: 8996/8996
SPI underruns 0, overruns 0
Number of disconnects: 0, IAP RAM available 0x2c8bc
Buffer RX/TX: 0/0-0
=== Duet Control Server ===
Duet Control Server v3.3-rc3
Code buffer space: 4096
Configured SPI speed: 8000000 Hz
Full transfers per second: 0.16
Codes per second: 0.00
Maximum length of RX/TX data transfers: 2948/692

I also stored an old (most probably irrelevant) M112 B121 answer (so for the toolboard only) when I was running firmware 3.2.2. If needed, I can transfert it too.
I can downgrade the firmware version if needed. I will be working on it all day, so I will try to be as reactive as possible.

@phaedrux said in Trouble connecting Toolboards 1LC to Duet 3 6HC:

Is this still the case? Just rapid blinking?

Yes, both of them are blinking rapidely.

MathV

@phaedrux
I just catched a connection with a toolboard, but only managed to send a M409 K"boards", which result is:

01/06/2021, 10:08:10:
m409 K"boards":
{"key":"boards","flags":"","result":
[{"canAddress":0,
"firmwareDate":"2021-05-26",
"firmwareFileName":"Duet3Firmware_MB6HC.bin",
"firmwareName":"RepRapFirmware for Duet 3 MB6HC",
"firmwareVersion":"3.3RC3",
"iapFileNameSBC":"Duet3_SBCiap32_MB6HC.bin",
"iapFileNameSD":"Duet3_SDiap32_MB6HC.bin",
"mcuTemp":{"current":37.5,"max":37.9,"min":28.7},
"name":"Duet 3 MB6HC",
"shortName":"MB6HC",
"uniqueId":"08DJM-956BA-NA3TJ-6J1DD-3S86T-KU9AS",
"v12":{"current":12.1,"max":12.2,"min":12.1},
"vIn":{"current":24.1,"max":24.1,"min":24.0}},
{"canAddress":121,
"firmwareFileName":"Duet3Firmware_TOOL1LC.bin",
"firmwareVersion":"3.1.0 (2020-05-15b1)",
"mcuTemp":{"current":0,"max":0,"min":0},
"shortName":"TOOL1LC",
"state":"running",
"v12":{"current":0,"max":0,"min":0},
"vIn":{"current":0,"max":0,"min":0}}],
"next":0}

dc42

@paulhew said in Trouble connecting Toolboards 1LC to Duet 3 6HC:

I am powering the toolboards with 24V from the OUT1 (1LC#1) and OUT2 (1LC#2).

That is not a valid way of powering the tool boards for two reasons;

The OUT ports are ground-side switched. As a result, when a tool board is not powered, its ground will be at +24V. This is likely to feed a common-mode bias back through the CAN bus, which may disrupt CAN traffic between other boards that are powered up.
The tool board has 100uF of capacitance connected between +24V and ground, and another 100uF connected indirectly between +24V and ground. There will be a substantial current surge when you turn on power via the OUT port. That will cause a transient and possibly a brownout. if repeated many times, it may damage the OUT port mosfet.

If you really want to be able to switch power to the tool boards, you need to switch the high side instead of the ground side, and use a slow turn-on to avoid a large current surge. After applying power to the board you will need to delay for a few seconds, then execute the commands to configure that tool board.

MathV

@dc42 Thanks a lot.
As I understand, it is going to be risky to perform this on the 6HC directly (unless I missed something).
I will connect the tools to the 24V PSU with external relays on the high side (in addiction with G4 S5, also suggested by @Phaedrux ). That way, the current surge while switching on the tools shouldn't be a problem anymore.

I have now plugged the tools directly on the PSU, and indeed, some strange behaviours disappeared.
The connection issue is still there, but I noticed the tools were running the firmware 3.1.0 while the 6HC was running on 3.3RC3.
I will downgrade the firmware to 3.1.0, and then try to connect again via the CAN bus.
Is there an update order to respect? (Tools first, and the 6HC at the end?)

T3P3Tony

@mathv said in Trouble connecting Toolboards 1LC to Duet 3 6HC:

but I noticed the tools were running the firmware 3.1.0 while the 6HC was running on 3.3RC3.
I will downgrade the firmware to 3.1.0, and then try to connect again via the CAN bus.

I would not downgrade the toolboards to 3.1.0 better to upgrade the 6HC to at least 3.2.2 and if possible 3.3RC3

MathV

@t3p3tony , Can the toolboards 1LC communicate with the 6HC, enven if their firmware version are different? (6HC-->3.3RC3 and 1LC-->3.1.0)
Just in case my explanation was not very clear:

the 6HC is currently in 3.3RC3
the 1LC are currently in 3.1.0 (I cannot communicate with them, so I cannot update the firmware)
the idea was to downgrade the 6HC to match the 1LC version, then, if the communication is back again, upgrade the 1LC to 3.3RC3 and finish with the upgrade of the 6HC to 3.3RC3.
But may be I am misunderstanding something.

dc42

@mathv said in Trouble connecting Toolboards 1LC to Duet 3 6HC:

Can the toolboards 1LC communicate with the 6HC, enven if their firmware version are different? (6HC-->3.3RC3 and 1LC-->3.1.0)

You won't get full CAN sync or motion if you mix 3.3 with earlier versions, however some commands will still work, in particular M115 and M997.

Version 1.1 tool boards will not run any firmware earlier than 3.3RC.

MathV

@dc42 @t3p3tony @phaedrux, Thanks a lot for your help and time. It is now much more clear.

I managed to catch a sync between toolboard and 6HC, I ran an M997 B121, and it is now working perfectly.

Just to know, is there a procedure to flash the firmware (I only found it for the bootloader) directly to the 1LC with an ATMEL ICE?

dc42

@mathv said in Trouble connecting Toolboards 1LC to Duet 3 6HC:

Just to know, is there a procedure to flash the firmware (I only found it for the bootloader) directly to the 1LC with an ATMEL ICE?

It should be possible to do that, although I've not tried that except for debug firmware that doesn't use the bootloader. Don't remove the the bootloader protection, leave it in place so that the Erase operation doesn't erase it. Tell the ICE to install the binary at address 0x4000.

MathV

@dc42 Thanks again.
I will purchase the ATMEL ICE and try this procedure keeping the bootloader.

dc42

@mathv said in Trouble connecting Toolboards 1LC to Duet 3 6HC:

@dc42 Thanks again.
I will purchase the ATMEL ICE and try this procedure keeping the bootloader.

There's no real need for an ICE unless you want to modify and debug the firmware.