Tool distribution board problem?

Marcossf

Hi everyone,

We have a strange problem in our setup we're building.

Once configured and working all 1XD boards for axis (XYYZUV) we continue with the printheads that have 3x 1LC boards connected to a 1 Distribution board.

We have directly connected the distribution board to the 6HC and each 1LC in it, to configure the CAN addresses of each head and we have done it with each one of them without problem (CAN20,21,22).

The problem arises when we connect to the bus of the rest of the 1XD boards, which with the distribution board connected, it does not find any of the boards.
All boards are updated to the latest firmware version 3.3.0 and have their corresponding CAN address. Everything works perfect if the distribution board is not connected

If I disconnect the distribution board (the white wire in the photo with the yellow arrow) the other boards come to life. When I connect it again, they are all lost and simultaneously the red status LED lights up quickly.

If I connect just the distribution board directly to the 6HC, without other board in the CANbus, the 1LCs in the dsitribution board are detected without problem, so I rule out that none of these boards are bad.

What could be happening?

We have tried everything!

Marcossf

Another thing:

If I disconnect one of the 1XD from the bus (interchangeably) and connect the distribution board instead, it works and ok is detected.

Marcossf

@dc42 Could you have a look at this problem, please?

There are some limitations in the number of Can boards? There are 9 in this case, but not working with only 7 but Ok when it's 6 boards connected in any order.

All boards are checked for FW version and all ot them are correct (3.3.0). Not tried without distribution board (1LC direct connect for cabling route issues and its not a solution for us).
All CAN cables tested and working, no power problems, no bad connectors....

When the distribution board its added to the CAN Chain with all 1XD boards connected, the red LED behaviour is:
Flashing continuously and rapidly: board is running but does not have CAN communication with the main board

Why?

dc42

@marcossf my best guess is that the wiring between the tool distribution board and the tool board isn't correct. Please check that the wiring between the JST ZH connectors on the cable connecting the tool board and the tool distribution board really is straight through and that there are no breaks in that cable. Also check that you have removed the two jumpers on the tool distribution board next to the socket that you have plugged the cable into.

Here's another way to test that you have continuity. Disconnect the CAN cable to the Duet. Then with a multimeter, measure the resistance between the outer 2 pins of the termination jumper block of the first 1XD board. It should measure 120 ohms (the resistance of the termination resistor on the distribution board).

PS - are you using two twisted pairs between the two JST ZH connectors? How long is that cable?

dc42

@marcossf I just spotted in the photo: you have not removed the CAN termination jumpers from any of the 1XD boards. The rule is that you must remove the termination jumpers from all boards except the last one (the one that has only 1 CAN cable connected to it). In this case, the last board is the tool distribution board. Currently, you have 8 termination resistors on the CAN bus instead of just 2 (one on the Duet at one end of the bus, and one on the distribution board at the other end). See https://duet3d.dozuki.com/Wiki/CAN_connection.

Marcossf

@dc42 Thank you.

Tomorrow I'll check again the jumpers following your advice. The cable and connectors were tested throughout and exchanged with other ones, so don't expect to be the problem there.

Probably the jumper in one 1XD board can cause that as you said, that it wasn't checked.

dc42

@marcossf said in Tool distribution board problem?:

Probably the jumper in one 1XD board can cause that as you said, that it wasn't checked.

From your photo it appears to me that all the 1XD boards have the two termination jumpers fitted. With the tool distribution board at the end of the bus, none of the 1XD boards should have termination jumpers fitted.

Marcossf

@dc42 Yes the 1XD jumpers weren't removed at all. Right now we will probe it without any 1XD jumpers.

By the way, when the distribution board have less toolboard connected (retired printhead) it's necesary to put the jumpers in it or the termination is done by the toolboard itself if present?

Marcossf

CORRECT! You're right @dc42

It works with the jumpers removed from 1XD boards:

24/6/2021 12:22:47	m115 b20
Duet TOOL1LC firmware version 3.3 (2021-06-15 16:12:58)
24/6/2021 12:22:45	m115 b45
Duet EXP1XD firmware version 3.3 (2021-06-15 16:12:29)
24/6/2021 12:22:43	m115 b44
Duet EXP1XD firmware version 3.3 (2021-06-15 16:12:29)
24/6/2021 12:22:41	m115 b43
Duet EXP1XD firmware version 3.3 (2021-06-15 16:12:29)
24/6/2021 12:22:39	m115 b42
Duet EXP1XD firmware version 3.3 (2021-06-15 16:12:29)
24/6/2021 12:22:37	m115 b41
Duet EXP1XD firmware version 3.3 (2021-06-15 16:12:29)
24/6/2021 12:22:35	m115 b40
Duet EXP1XD firmware version 3.3 (2021-06-15 16:12:29)
24/6/2021 12:22:32	m115
FIRMWARE_NAME: RepRapFirmware for Duet 3 MB6HC FIRMWARE_VERSION: 3.3 ELECTRONICS: Duet 3 MB6HC v1.01 or later FIRMWARE_DATE: 2021-06-15 21:45:56

Three persons here couldn't think about bus temination...

I'll check if the distribution board can work with a removed prinhead 1LC and opened jumpers. If not, we have to ride with another problem...

Thank you very much. You can close as resolved.

dc42

@marcossf said in Tool distribution board problem?:

I'll check if the distribution board can work with a removed prinhead 1LC and opened jumpers. If not, we have to ride with another problem...

With opened jumpers on the distribution board next to a tool board connector, and no tool board plugged in, the CAN signal won't propagate along the distribution board any further. So any further tool boards will be cut off, and the bus won't be terminated at that end. So there will be reflections from that end of the bus, which may disrupt communication.

Depending on the length of the cable to the tool boards and how many of them you have, you might be able to operate the tool boards with the bypass jumpers in place.

Marcossf

@dc42 There is just one tool distribution board, so it can hold 1 to 3 printheads but could be that one or two may not be present (using only one printhead for example). The canbus cable from the last 1XD board to the distribution board is about 3m long

I've tested with the tool distribution board in 0 connector and it works with just this printhead connected, despite being the jumpers removed from distribution board (jumpers 0, 1 and 2 off and 3 on, our printheads carriage holds up to 3 printheads).

Next test is if one printhead can be detected in any of the other positions while the other has been removed. We are cabling the remaining two printheads and can be test it very soon.

dc42

@marcossf you should always ensure that the CAN bus is terminated at the end. This is especially important when you have a lot of boards connected to the CAN bus. Without proper termination, the system may appear to work, but there will be additional retries and possibly even lost CAN packets, which may cause issues with printing.

Marcossf

@dc42 I've tested with two printheads connected to the distribution board. It seems to work if the order of the board is followed: 0 and 1 populated works but not detect second printhead if it's fitted in 0 and 2 positions without any printhead in the 1 position. I guess it's because the detected 1LC's terminates the bus at first position.

How can I diag the canbus packets and efficiency apart from M122 B#? How interprete the data given?
Is there any way to show all current CAN detected in the all the Canbus-FD and their status?

dc42

@marcossf you must have the two bypass jumpers installed on the tool distribution board next to any of the 4 small CAN connectors that do not have a tool board attached.

You may find that a connected tool works even if the bypass jumpers for its connector are in place. But we can't guarantee that.

Marcossf

@dc42 This way is how we did:

And it seems to works.

The customer will not have access to this board (or any other) so it isn't possible to remove or put jumpers depending on the head installed. Thinking in doing it by switches ruins the user experience.

Is there a problem it can stay like this?
It would be better to leave the jumpers 0 removed and the rest set in, due to the possibility that the heads are present or not in 1 and 2 positions? Obviously in 0 connector there will always be one, and position 3 never will have one.

dc42

@marcossf said in Tool distribution board problem?:

The customer will not have access to this board (or any other) so it isn't possible to remove or put jumpers depending on the head installed. Thinking in doing it by switches ruins the user experience.

Does the customer have the ability to add or remove tool heads? If so, what process does the customer go through to do that?

Marcossf

@dc42

The user can choose a tool, connect it (power and canbus connectors) and lock it in any printhead craddle. The 1LC is inside each printhead. The distribution board is in the back of ZUV carriage. Could be one, two or three prinheads at any given time.

Would it be better to leave the jumpers closed? If it detects any header connected to the distribution board anyway it would be a good solution.

As you may know as we asked in other post, we installed two microswitches on each toolhead to know where the printhead is and which type is. Still not implemented any software to do this, just fitted the hw at this moment.

Marcossf

Diag B20 to B21 with distribution board with all bypass jumpers fitted:

Last software reset time unknown, reason: AssertionFailed, available RAM 4320, slot 0
Software reset code 0x0120 ICSR 0x00000000 SP 0x20003dbc Task   Freestk 130 bad marker
Stack: 00000544 000230c4 00019c25 20003154 00016da3 20003154 00016475 20000ed0 00000000 00000001 0000828d 200071c8 200071c8 200071e0 00000000 20000f50 0001165f 0002247c 00022538 00021b8c 00019bc5 200071c8 200071c8 20000f50 00008405 200071d8 000009c6
Driver 0: position 0, 80.0 steps/mm, standstill, SG min/max 0/0, read errors 0, write errors 0, ifcnt 9, reads 35805, writes 0, timeouts 0, DMA errors 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter 0/9, peak Rx sync delay 197, resyncs 0/0, no step interrupt scheduled
VIN: 23.8V
MCU temperature: min 24.3C, current 34.9C, max 35.1C
Ticks since heat task active 247, ADC conversions started 423990, completed 423989, timed out 0, errs 0
Last sensors broadcast 0x00000000 found 0 2 ticks ago, loop time 0
CAN messages queued 33, send timeouts 0, received 1022, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
Accelerometer detected: no
I2C bus errors 4619, naks 4619, other errors 4619


25/6/2021 19:24:19: m122 b21: Diagnostics for board 21:
Duet TOOL1LC firmware version 3.3 (2021-06-15 16:12:58)
Bootloader ID: not available
Never used RAM 4044, free system stack 2789 words
Tasks: Move(notifyWait<null>,0.0%,153) HEAT(delaying<null>,0.0%,117) CanAsync(notifyWait<null>,0.0%,65) CanRecv(notifyWait<null>,0.0%,76) CanClock(notifyWait<null>,0.0%,65) TMC(notifyWait<null>,2.9%,57) MAIN(running<null>,92.2%,436) IDLE(ready<null>,0.0%,41) AIN(delaying<null>,4.9%,142), total 100.0%
Last reset 00:07:06 ago, cause: power up
Last software reset data not available
Driver 0: position 0, 80.0 steps/mm, standstill, SG min/max 0/0, read errors 0, write errors 0, ifcnt 9, reads 36041, writes 0, timeouts 0, DMA errors 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter 0/9, peak Rx sync delay 210, resyncs 0/0, no step interrupt scheduled
VIN: 24.1V
MCU temperature: min 20.5C, current 31.3C, max 31.3C
Ticks since heat task active 117, ADC conversions started 426110, completed 426109, timed out 0, errs 0
Last sensors broadcast 0x00000000 found 0 122 ticks ago, loop time 0
CAN messages queued 27, send timeouts 0, received 1024, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
Accelerometer detected: no
I2C bus errors 4619, naks 4619, other errors 4619


25/6/2021 19:24:20: m122 b22: Diagnostics for board 22:
Duet TOOL1LC firmware version 3.3 (2021-06-15 16:12:58)
Bootloader ID: not available
Never used RAM 4044, free system stack 2789 words
Tasks: Move(notifyWait<null>,0.0%,153) HEAT(delaying<null>,0.0%,117) CanAsync(notifyWait<null>,0.0%,65) CanRecv(notifyWait<null>,0.0%,76) CanClock(notifyWait<null>,0.0%,65) TMC(delaying<null>,2.9%,57) MAIN(running<null>,92.2%,436) IDLE(ready<null>,0.0%,41) AIN(delaying<null>,4.9%,142), total 100.0%
Last reset 00:07:08 ago, cause: power up
Last software reset data not available
Driver 0: position 0, 80.0 steps/mm, standstill, SG min/max 0/0, read errors 0, write errors 0, ifcnt 9, reads 35817, writes 0, timeouts 0, DMA errors 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter 0/8, peak Rx sync delay 209, resyncs 0/0, no step interrupt scheduled
VIN: 24.2V
MCU temperature: min 23.5C, current 33.7C, max 33.8C
Ticks since heat task active 87, ADC conversions started 428080, completed 428080, timed out 0, errs 0
Last sensors broadcast 0x00000000 found 0 92 ticks ago, loop time 0
CAN messages queued 27, send timeouts 0, received 1022, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
Accelerometer detected: no
I2C bus errors 4619, naks 4619, other errors 4619

Same diag but with the 0, 1 and 2 bypass jumpers removed, 3 fitted.

25/6/2021 19:27:10: m122 b20: Diagnostics for board 20:
Duet TOOL1LC firmware version 3.3 (2021-06-15 16:12:58)
Bootloader ID: not available
Never used RAM 4044, free system stack 2789 words
Tasks: Move(notifyWait<null>,0.0%,153) HEAT(delaying<null>,0.0%,117) CanAsync(notifyWait<null>,0.0%,65) CanRecv(notifyWait<null>,0.0%,76) CanClock(notifyWait<null>,0.0%,65) TMC(delaying<null>,2.9%,63) MAIN(running<null>,92.2%,436) IDLE(ready<null>,0.0%,41) AIN(delaying<null>,4.9%,142), total 100.0%
Last reset 00:00:29 ago, cause: power up
Last software reset time unknown, reason: AssertionFailed, available RAM 4320, slot 0
Software reset code 0x0120 ICSR 0x00000000 SP 0x20003dbc Task   Freestk 130 bad marker
Stack: 00000544 000230c4 00019c25 20003154 00016da3 20003154 00016475 20000ed0 00000000 00000001 0000828d 200071c8 200071c8 200071e0 00000000 20000f50 0001165f 0002247c 00022538 00021b8c 00019bc5 200071c8 200071c8 20000f50 00008405 200071d8 000009c6
Driver 0: position 0, 80.0 steps/mm, standstill, SG min/max 0/0, read errors 0, write errors 0, ifcnt 9, reads 14501, writes 9, timeouts 0, DMA errors 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter 0/8, peak Rx sync delay 193, resyncs 0/0, no step interrupt scheduled
VIN: 23.8V
MCU temperature: min 30.4C, current 32.2C, max 32.2C
Ticks since heat task active 49, ADC conversions started 29042, completed 29041, timed out 0, errs 0
Last sensors broadcast 0x00000000 found 0 54 ticks ago, loop time 0
CAN messages queued 33, send timeouts 0, received 166, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
Accelerometer detected: no
I2C bus errors 4619, naks 4619, other errors 4619


25/6/2021 19:27:13: m122 b21: Diagnostics for board 21:
Duet TOOL1LC firmware version 3.3 (2021-06-15 16:12:58)
Bootloader ID: not available
Never used RAM 4044, free system stack 2789 words
Tasks: Move(notifyWait<null>,0.0%,153) HEAT(delaying<null>,0.0%,117) CanAsync(notifyWait<null>,0.0%,65) CanRecv(notifyWait<null>,0.0%,76) CanClock(notifyWait<null>,0.0%,65) TMC(delaying<null>,2.9%,57) MAIN(running<null>,92.2%,436) IDLE(ready<null>,0.0%,41) AIN(delaying<null>,4.9%,144), total 100.0%
Last reset 00:00:32 ago, cause: power up
Last software reset data not available
Driver 0: position 0, 80.0 steps/mm, standstill, SG min/max 0/0, read errors 0, write errors 0, ifcnt 9, reads 16024, writes 9, timeouts 0, DMA errors 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter 0/8, peak Rx sync delay 195, resyncs 0/0, no step interrupt scheduled
VIN: 24.1V
MCU temperature: min 26.8C, current 28.8C, max 28.8C
Ticks since heat task active 93, ADC conversions started 32086, completed 32086, timed out 0, errs 0
Last sensors broadcast 0x00000000 found 0 98 ticks ago, loop time 0
CAN messages queued 27, send timeouts 0, received 175, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
Accelerometer detected: no
I2C bus errors 4619, naks 4619, other errors 4619


25/6/2021 19:27:15: m122 b22: Diagnostics for board 22:
Duet TOOL1LC firmware version 3.3 (2021-06-15 16:12:58)
Bootloader ID: not available
Never used RAM 4044, free system stack 2789 words
Tasks: Move(notifyWait<null>,0.0%,153) HEAT(delaying<null>,0.0%,117) CanAsync(notifyWait<null>,0.0%,65) CanRecv(notifyWait<null>,0.0%,76) CanClock(notifyWait<null>,0.0%,65) TMC(delaying<null>,2.9%,63) MAIN(running<null>,92.2%,436) IDLE(ready<null>,0.0%,41) AIN(delaying<null>,4.9%,144), total 100.0%
Last reset 00:00:34 ago, cause: power up
Last software reset data not available
Driver 0: position 0, 80.0 steps/mm, standstill, SG min/max 0/0, read errors 0, write errors 0, ifcnt 9, reads 17189, writes 9, timeouts 0, DMA errors 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter 0/8, peak Rx sync delay 195, resyncs 0/0, no step interrupt scheduled
VIN: 24.2V
MCU temperature: min 29.5C, current 31.7C, max 31.7C
Ticks since heat task active 174, ADC conversions started 34418, completed 34417, timed out 0, errs 0
Last sensors broadcast 0x00000000 found 0 179 ticks ago, loop time 0
CAN messages queued 28, send timeouts 0, received 193, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
Accelerometer detected: no
I2C bus errors 4619, naks 4619, other errors 4619

I don't see relevant differences, and I don't know how to interpret most of the values. It's in the wiki?

Removing a printhead (powered off, of course) while respect the order (0,1,none) gives same results except for the missing printhead.

So, would left jumpers out on 0,1,2 and 3 IN? Could be damage?

Marcossf

At this moment we have it like this:

dc42

@marcossf said in Tool distribution board problem?:

So, would left jumpers out on 0,1,2 and 3 IN? Could be damage?

No, that will not cause damage.

A better way than leaving the jumpers fitted on the tool board would be to connect the two CANL pins together on your user-accessible connector, and also connect the two CANH pins together. In other words, move the bypass links closer to the tools. That will make the CAN 'stubs' shorter by the length of the cable from the tool board to the user-accessible connector.

An even better way would be to provide dummy tool plugs that connect the two CANL pins together and the two CANH pins together, and instruct the user to fit a dummy plug in any position that does not have a tool connected.