CAN Failure after bootloader update

psyblue

After testing several times, I suspect bootloader v2.2.

Any advice would be appreciated.

Environment :

Board configuration: MB6HC----------3HC----------1LC
Frimware: v3.2
Bootloader: v2.2
CAN Bus termination: open on 3HC, 120 Ohm termination on 1LC, cables are good condition.

Symptom :

No CAN-diagnsis-red-LED blicking on 3HC after power-on and communication time-out for'M122 B1' command.
If CAN cable between 3HC and 1LC is unplugged, CAN-diagnsis-LED(red) on 3HC starts blinking.
Then operates normally after plugged.
Before update the bootload, it worked good.

Diagnosis results

Before update

M122 B121
Diagnostics for board 121:
Board TOOL1LC firmware 3.1.0 (2020-05-15b1)
Never used RAM 8.7Kb, max stack 352b
HEAT 1184 CanAsync 1460 CanRecv 1416 TMC 204 AIN 524 MAIN 2308
Last reset 00:00:51 ago, cause: power up
Driver 0: standstill, SG min/max 0/0, read errors 0, write errors 0, ifcount 9, reads 25502, writes 9, timeouts 1, DMA errors 0, failedOp 0x80
Moves scheduled 0, completed 0, hiccups 0
VIN: 24.1V
MCU temperature: min 13.6C, current 17.7C, max 17.7C
Ticks since heat task active 58, ADC conversions started 50881, completed 50880, timed out 0
Last sensors broadcast 00000000 found 0 61 ticks ago
Free CAN buffers: 36
NVM user row b1d088a9 fffff8bb 20002a18 b
TSENS 0006f9 GAIN 015ac2 OFFS 00241e CAL 1f2a

After update

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.2 running on Duet 3 MB6HC v0.6 or 1.0 (standalone mode)
Board ID: 08DJM-956BA-NA3TJ-6JKDG-3S86R-TU9UT
Used output buffers: 1 of 40 (22 max)
=== RTOS ===
Static ram: 149788
Dynamic ram: 92720 of which 76 recycled
Never used RAM 116248, free system stack 188 words
Tasks: NETWORK(ready,163) ETHERNET(blocked,118) HEAT(blocked,302) CanReceiv(blocked,834) CanSender(blocked,352) CanClock(blocked,356) TMC(blocked,19) MAIN(running,1095) IDLE(ready,19)
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 00:01:22 ago, cause: power up
Last software reset at 2021-03-03 11:41, reason: User, GCodes spinning, available RAM 116152, slot 2
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
Aux0 errors 0,0,0
Aux1 errors 0,0,0
MCU temperature: min 17.0, current 17.3, max 17.4
Supply voltage: min 23.8, current 23.8, max 23.9, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.0, current 12.0, max 12.1, under voltage events: 0
Driver 0: position 8, standstill, reads 15712, writes 0 timeouts 0, SG min/max not available
Driver 1: position 8, standstill, reads 15712, writes 0 timeouts 0, SG min/max not available
Driver 2: position 20727, standstill, reads 15712, writes 0 timeouts 0, SG min/max not available
Driver 3: position 0, standstill, reads 15713, writes 0 timeouts 0, SG min/max not available
Driver 4: position 0, standstill, reads 15712, writes 0 timeouts 0, SG min/max not available
Driver 5: position 0, standstill, reads 15712, writes 0 timeouts 0, SG min/max not available
Date/time: 2021-03-03 14:59:00
Slowest loop: 22.45ms; fastest: 0.05ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 2.2ms, write time 16.4ms, max retries 0
=== Move ===
DMs created 125, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 1, completed moves 1, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is ready with "M122 " in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 28.11ms; fastest: 0.02ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- Ethernet -
State: active
Error counts: 0 0 1 0 0
Socket states: 5 2 2 2 2 0 0 0
=== CAN ===
Messages queued 11, send timeouts 0, received 36, lost 0, longest wait 0ms for reply type 0, free buffers 48



M122 B1
Diagnostics for board 1:
Duet EXP3HC firmware version 3.2 (2021-01-05)
Bootloader ID: SAME5x bootloader version 2.2 (2021-01-16b1)
Never used RAM 154896, free system stack 200 words
HEAT 102 CanAsync 94 CanRecv 87 TMC 64 MAIN 299 AIN 257
Last reset 00:00:07 ago, cause: software
Last software reset data not available
Driver 0: position 0, 80.0 steps/mm,  standstill, reads 35212, writes 11 timeouts 0, SG min/max 0/0
Driver 1: position 0, 80.0 steps/mm,  standstill, reads 35214, writes 11 timeouts 0, SG min/max 0/0
Driver 2: position 0, 80.0 steps/mm,  standstill, reads 35217, writes 11 timeouts 0, SG min/max 0/0
Moves scheduled 0, completed 0, in progress 0, hiccups 0
No step interrupt scheduled
VIN: 24.3V, V12: 12.3V
MCU temperature: min 28.1C, current 28.1C, max 28.1C
Ticks since heat task active 30, ADC conversions started 7772, completed 7771, timed out 0
Last sensors broadcast 0x00000000 found 0 33 ticks ago, loop time 0
CAN messages queued 18, send timeouts 0, received 108, lost 0, free buffers 36



M122 B121
Diagnostics for board 121:
Duet TOOL1LC firmware version 3.2 (2021-01-05)
Bootloader ID: SAMC21 bootloader version 2.2 (2021-01-16b1)
Never used RAM 4140, free system stack 96 words
HEAT 90 CanAsync 89 CanRecv 83 TMC 60 MAIN 218 AIN 70
Last reset 00:00:45 ago, cause: power up
Last software reset data not available
Driver 0: position 0, 420.0 steps/mm, standstill, SG min/max 0/0, read errors 0, write errors 0, ifcnt 13, reads 22960, writes 13, timeouts 0, DMA errors 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0
No step interrupt scheduled
VIN: 24.1V
MCU temperature: min 6.1C, current 10.1C, max 10.1C
Ticks since heat task active 227, ADC conversions started 45840, completed 45839, timed out 0
Last sensors broadcast 0x00000002 found 1 230 ticks ago, loop time 0
CAN messages queued 422, send timeouts 0, received 385, lost 0, free buffers 36

dc42

This is an interesting one! It could be either the bootloader or the firmware not starting.

If you temporarily downgrade the 3HC firmware to 3.1.1, does it start up and flash the LED with the tool board connected?
If it still doesn't start up with the tool board connected, then I suggest you downgrade the bootloader to version 1.3. When that is working and booting the firmware, put firmware 3.2.2 back on the 3HC.

psyblue

@dc42

As a result, The worst has happened. All three HC3 boards have been ICE-state(red diagnostic LED off, No response).
No response for M122 B1. No response to the reset button

I tested several things using the HC3 board I had.
RRF3.1 + Bootloader1.3 (estimated, No ID on diagnostics) -> good
RRF3.2.2 + Bootloader1.3 -> good
RRF3.2.2 + Bootloader2.2 -> Abnormal
RRF3.2.2 + Bootloader1.3 Down grade -> HC3 ICE-state

No problem on 1LC board.

I need the ATMEL ICE tool.

dc42

Please try the following:

Upload EXP3HC firmware version 3.1.1 to the Duet
Do a factory reset of one of the 3HC boards (set all 4 switches off and then turn on the power).

psyblue

@dc42

Dear dc42,

First of all, thank you very much for the always quick and kind reply.

I've tried following your advice, One of three boards could be made with firmware v3.1.1 + Bootloader v1.3.

I repeated the test with this.

I thought it was OK with firmware v3.1.1 + Bootloader v1.3, but there was a problem.
When 3HC is disconnected from every cable (include CAN) and only power is connected, does not the DIAG red LED flash at all, just off state.
Repeated plugging in and unplugging the CAN cable, the red LED starts blinking.

The other 2 3HC are still ICE"ICE-state" (I don't know what to call it in English. I call it'ICE'. power-on, no response, no blinking, do nothing state)
Disconnected the 3HC board from CAN and set the address to zero (all switches off), does not the DIAG red LED flash at all, just off state.
The 3HC is not connected to CAN in any way.

In my experience, I suspect the bootloader 2.2. the bootloader procedure seems to be stuck in the CAN init loop.
Even when it worked normally in the past, sometimes (30% probability) there was a situation where it was suspected that booting was delayed.
Because I found a case where the contents of config.g were not passed to 3HC.
So, I put "G4 S2" (delay 2 seconds) at the beginning of config.g.

I hope you try to reproduce the same situation as me.

And 1LC looks fine.

dc42

@psyblue, which country are you in, and which reseller did you buy the boards from?

psyblue

@dc42

I am Korean in Korea, bought at filastruder.com

Isn't it a known problem?
It could be an important source of potential bugs in the future.
I can send you one, if you want. Of course, I want to get it back with the improved one.

However, I have anxiety that seems to work normally in your hand as always

Or, I ordered ATMEL ICE. I will reprogram it and report the result.
It will take about 2 weeks.

And short question,
Is there any changed or added pull-up/ pull-down resistor on 3HC board?

psyblue

@dc42

All of the above symptoms are considered a problem with Bootloader 2.2.

After updating to Bootloader 2.3 via ATMEL-ICE, everything works fine.

Thank you for the bootloader 2.3.

dc42

@psyblue, I'm glad you solved it with bootloader 2.3. I'm sorry that it was necessary for you to get an ICE to fix it.

psyblue

@dc42, I am also glad that the problem disappeared with the bootloader 2.3, but I am curious if the above symptoms match the potential problem with Bootloader 2.2

dc42

@psyblue the main change between bootloader 2.2 and 2.3 is that in 2.2 if it didn't get a satisfactory frequency reading within a certain time, its reset the processor to start again. In 2.3 it just keeps on taking readings, in effect allowing the crystal oscillator as long as it needs to stabilise.