• Tags
  • Documentation
  • Order
  • Register
  • Login
Duet3D Logo Duet3D
  • Tags
  • Documentation
  • Order
  • Register
  • Login

CAN Failure after bootloader update

Scheduled Pinned Locked Moved
Using Duet Controllers
2
11
552
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • undefined
    psyblue
    last edited by psyblue 3 Mar 2021, 12:55 3 Mar 2021, 12:54

    After testing several times, I suspect bootloader v2.2.

    Any advice would be appreciated.

    Environment :

    • Board configuration: MB6HC----------3HC----------1LC
    • Frimware: v3.2
    • Bootloader: v2.2
    • CAN Bus termination: open on 3HC, 120 Ohm termination on 1LC, cables are good condition.

    Symptom :

    1. No CAN-diagnsis-red-LED blicking on 3HC after power-on and communication time-out for'M122 B1' command.
    2. If CAN cable between 3HC and 1LC is unplugged, CAN-diagnsis-LED(red) on 3HC starts blinking.
    3. Then operates normally after plugged.
    4. Before update the bootload, it worked good.

    Diagnosis results

    • Before update
    M122 B121
    Diagnostics for board 121:
    Board TOOL1LC firmware 3.1.0 (2020-05-15b1)
    Never used RAM 8.7Kb, max stack 352b
    HEAT 1184 CanAsync 1460 CanRecv 1416 TMC 204 AIN 524 MAIN 2308
    Last reset 00:00:51 ago, cause: power up
    Driver 0: standstill, SG min/max 0/0, read errors 0, write errors 0, ifcount 9, reads 25502, writes 9, timeouts 1, DMA errors 0, failedOp 0x80
    Moves scheduled 0, completed 0, hiccups 0
    VIN: 24.1V
    MCU temperature: min 13.6C, current 17.7C, max 17.7C
    Ticks since heat task active 58, ADC conversions started 50881, completed 50880, timed out 0
    Last sensors broadcast 00000000 found 0 61 ticks ago
    Free CAN buffers: 36
    NVM user row b1d088a9 fffff8bb 20002a18 b
    TSENS 0006f9 GAIN 015ac2 OFFS 00241e CAL 1f2a
    
    • After update
    M122
    === Diagnostics ===
    RepRapFirmware for Duet 3 MB6HC version 3.2 running on Duet 3 MB6HC v0.6 or 1.0 (standalone mode)
    Board ID: 08DJM-956BA-NA3TJ-6JKDG-3S86R-TU9UT
    Used output buffers: 1 of 40 (22 max)
    === RTOS ===
    Static ram: 149788
    Dynamic ram: 92720 of which 76 recycled
    Never used RAM 116248, free system stack 188 words
    Tasks: NETWORK(ready,163) ETHERNET(blocked,118) HEAT(blocked,302) CanReceiv(blocked,834) CanSender(blocked,352) CanClock(blocked,356) TMC(blocked,19) MAIN(running,1095) IDLE(ready,19)
    Owned mutexes: HTTP(MAIN)
    === Platform ===
    Last reset 00:01:22 ago, cause: power up
    Last software reset at 2021-03-03 11:41, reason: User, GCodes spinning, available RAM 116152, slot 2
    Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a
    Error status: 0x00
    Aux0 errors 0,0,0
    Aux1 errors 0,0,0
    MCU temperature: min 17.0, current 17.3, max 17.4
    Supply voltage: min 23.8, current 23.8, max 23.9, under voltage events: 0, over voltage events: 0, power good: yes
    12V rail voltage: min 12.0, current 12.0, max 12.1, under voltage events: 0
    Driver 0: position 8, standstill, reads 15712, writes 0 timeouts 0, SG min/max not available
    Driver 1: position 8, standstill, reads 15712, writes 0 timeouts 0, SG min/max not available
    Driver 2: position 20727, standstill, reads 15712, writes 0 timeouts 0, SG min/max not available
    Driver 3: position 0, standstill, reads 15713, writes 0 timeouts 0, SG min/max not available
    Driver 4: position 0, standstill, reads 15712, writes 0 timeouts 0, SG min/max not available
    Driver 5: position 0, standstill, reads 15712, writes 0 timeouts 0, SG min/max not available
    Date/time: 2021-03-03 14:59:00
    Slowest loop: 22.45ms; fastest: 0.05ms
    === Storage ===
    Free file entries: 10
    SD card 0 detected, interface speed: 25.0MBytes/sec
    SD card longest read time 2.2ms, write time 16.4ms, max retries 0
    === Move ===
    DMs created 125, maxWait 0ms, bed compensation in use: none, comp offset 0.000
    === MainDDARing ===
    Scheduled moves 1, completed moves 1, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
    === AuxDDARing ===
    Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
    === Heat ===
    Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
    Heater 1 is on, I-accum = 0.0
    === GCodes ===
    Segments left: 0
    Movement lock held by null
    HTTP is ready with "M122 " in state(s) 0
    Telnet is idle in state(s) 0
    File is idle in state(s) 0
    USB is idle in state(s) 0
    Aux is idle in state(s) 0
    Trigger is idle in state(s) 0
    Queue is idle in state(s) 0
    LCD is idle in state(s) 0
    SBC is idle in state(s) 0
    Daemon is idle in state(s) 0
    Aux2 is idle in state(s) 0
    Autopause is idle in state(s) 0
    Code queue is empty.
    === Network ===
    Slowest loop: 28.11ms; fastest: 0.02ms
    Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions
    HTTP sessions: 1 of 8
    - Ethernet -
    State: active
    Error counts: 0 0 1 0 0
    Socket states: 5 2 2 2 2 0 0 0
    === CAN ===
    Messages queued 11, send timeouts 0, received 36, lost 0, longest wait 0ms for reply type 0, free buffers 48
    
    
    
    M122 B1
    Diagnostics for board 1:
    Duet EXP3HC firmware version 3.2 (2021-01-05)
    Bootloader ID: SAME5x bootloader version 2.2 (2021-01-16b1)
    Never used RAM 154896, free system stack 200 words
    HEAT 102 CanAsync 94 CanRecv 87 TMC 64 MAIN 299 AIN 257
    Last reset 00:00:07 ago, cause: software
    Last software reset data not available
    Driver 0: position 0, 80.0 steps/mm,  standstill, reads 35212, writes 11 timeouts 0, SG min/max 0/0
    Driver 1: position 0, 80.0 steps/mm,  standstill, reads 35214, writes 11 timeouts 0, SG min/max 0/0
    Driver 2: position 0, 80.0 steps/mm,  standstill, reads 35217, writes 11 timeouts 0, SG min/max 0/0
    Moves scheduled 0, completed 0, in progress 0, hiccups 0
    No step interrupt scheduled
    VIN: 24.3V, V12: 12.3V
    MCU temperature: min 28.1C, current 28.1C, max 28.1C
    Ticks since heat task active 30, ADC conversions started 7772, completed 7771, timed out 0
    Last sensors broadcast 0x00000000 found 0 33 ticks ago, loop time 0
    CAN messages queued 18, send timeouts 0, received 108, lost 0, free buffers 36
    ​
    
    ​
    M122 B121
    Diagnostics for board 121:
    Duet TOOL1LC firmware version 3.2 (2021-01-05)
    Bootloader ID: SAMC21 bootloader version 2.2 (2021-01-16b1)
    Never used RAM 4140, free system stack 96 words
    HEAT 90 CanAsync 89 CanRecv 83 TMC 60 MAIN 218 AIN 70
    Last reset 00:00:45 ago, cause: power up
    Last software reset data not available
    Driver 0: position 0, 420.0 steps/mm, standstill, SG min/max 0/0, read errors 0, write errors 0, ifcnt 13, reads 22960, writes 13, timeouts 0, DMA errors 0
    Moves scheduled 0, completed 0, in progress 0, hiccups 0
    No step interrupt scheduled
    VIN: 24.1V
    MCU temperature: min 6.1C, current 10.1C, max 10.1C
    Ticks since heat task active 227, ADC conversions started 45840, completed 45839, timed out 0
    Last sensors broadcast 0x00000002 found 1 230 ticks ago, loop time 0
    CAN messages queued 422, send timeouts 0, received 385, lost 0, free buffers 36
    
    1 Reply Last reply Reply Quote 0
    • undefined
      dc42 administrators
      last edited by dc42 3 Mar 2021, 17:39 3 Mar 2021, 17:37

      This is an interesting one! It could be either the bootloader or the firmware not starting.

      1. If you temporarily downgrade the 3HC firmware to 3.1.1, does it start up and flash the LED with the tool board connected?

      2. If it still doesn't start up with the tool board connected, then I suggest you downgrade the bootloader to version 1.3. When that is working and booting the firmware, put firmware 3.2.2 back on the 3HC.

      Duet WiFi hardware designer and firmware engineer
      Please do not ask me for Duet support via PM or email, use the forum
      http://www.escher3d.com, https://miscsolutions.wordpress.com

      undefined 1 Reply Last reply 4 Mar 2021, 06:20 Reply Quote 0
      • undefined
        psyblue @dc42
        last edited by psyblue 3 Apr 2021, 08:13 4 Mar 2021, 06:20

        @dc42

        As a result, The worst has happened. All three HC3 boards have been ICE-state(red diagnostic LED off, No response).
        No response for M122 B1. No response to the reset button

        I tested several things using the HC3 board I had.
        RRF3.1 + Bootloader1.3 (estimated, No ID on diagnostics) -> good
        RRF3.2.2 + Bootloader1.3 -> good
        RRF3.2.2 + Bootloader2.2 -> Abnormal
        RRF3.2.2 + Bootloader1.3 Down grade -> HC3 ICE-state

        No problem on 1LC board.

        I need the ATMEL ICE tool.

        1 Reply Last reply Reply Quote 0
        • undefined
          dc42 administrators
          last edited by 4 Mar 2021, 09:12

          Please try the following:

          • Upload EXP3HC firmware version 3.1.1 to the Duet
          • Do a factory reset of one of the 3HC boards (set all 4 switches off and then turn on the power).

          Duet WiFi hardware designer and firmware engineer
          Please do not ask me for Duet support via PM or email, use the forum
          http://www.escher3d.com, https://miscsolutions.wordpress.com

          undefined 2 Replies Last reply 5 Mar 2021, 02:49 Reply Quote 0
          • undefined
            psyblue @dc42
            last edited by psyblue 3 May 2021, 09:55 5 Mar 2021, 02:49

            @dc42

            Dear dc42,

            First of all, thank you very much for the always quick and kind reply.

            I've tried following your advice, One of three boards could be made with firmware v3.1.1 + Bootloader v1.3.

            I repeated the test with this.

            I thought it was OK with firmware v3.1.1 + Bootloader v1.3, but there was a problem.
            When 3HC is disconnected from every cable (include CAN) and only power is connected, does not the DIAG red LED flash at all, just off state.
            Repeated plugging in and unplugging the CAN cable, the red LED starts blinking.

            The other 2 3HC are still ICE"ICE-state" (I don't know what to call it in English. I call it'ICE'. power-on, no response, no blinking, do nothing state)
            Disconnected the 3HC board from CAN and set the address to zero (all switches off), does not the DIAG red LED flash at all, just off state.
            The 3HC is not connected to CAN in any way.

            In my experience, I suspect the bootloader 2.2. the bootloader procedure seems to be stuck in the CAN init loop.
            Even when it worked normally in the past, sometimes (30% probability) there was a situation where it was suspected that booting was delayed.
            Because I found a case where the contents of config.g were not passed to 3HC.
            So, I put "G4 S2" (delay 2 seconds) at the beginning of config.g.

            I hope you try to reproduce the same situation as me.

            And 1LC looks fine.

            1 Reply Last reply Reply Quote 0
            • undefined
              dc42 administrators
              last edited by dc42 3 May 2021, 12:31 5 Mar 2021, 12:30

              @psyblue, which country are you in, and which reseller did you buy the boards from?

              Duet WiFi hardware designer and firmware engineer
              Please do not ask me for Duet support via PM or email, use the forum
              http://www.escher3d.com, https://miscsolutions.wordpress.com

              undefined 1 Reply Last reply 6 Mar 2021, 07:02 Reply Quote 0
              • undefined
                psyblue @dc42
                last edited by psyblue 3 Jul 2021, 10:34 6 Mar 2021, 07:02

                @dc42

                I am Korean in Korea, bought at filastruder.com

                Isn't it a known problem?
                It could be an important source of potential bugs in the future.
                I can send you one, if you want. Of course, I want to get it back with the improved one.

                However, I have anxiety that seems to work normally in your hand as always

                Or, I ordered ATMEL ICE. I will reprogram it and report the result.
                It will take about 2 weeks.

                And short question,
                Is there any changed or added pull-up/ pull-down resistor on 3HC board?

                1 Reply Last reply Reply Quote 0
                • undefined
                  psyblue @dc42
                  last edited by psyblue 13 May 2021, 04:57

                  @dc42

                  All of the above symptoms are considered a problem with Bootloader 2.2.

                  After updating to Bootloader 2.3 via ATMEL-ICE, everything works fine.

                  Thank you for the bootloader 2.3.

                  1 Reply Last reply Reply Quote 2
                  • undefined
                    dc42 administrators
                    last edited by 13 May 2021, 07:18

                    @psyblue, I'm glad you solved it with bootloader 2.3. I'm sorry that it was necessary for you to get an ICE to fix it.

                    Duet WiFi hardware designer and firmware engineer
                    Please do not ask me for Duet support via PM or email, use the forum
                    http://www.escher3d.com, https://miscsolutions.wordpress.com

                    undefined 1 Reply Last reply 18 May 2021, 06:09 Reply Quote 0
                    • undefined
                      psyblue @dc42
                      last edited by 18 May 2021, 06:09

                      @dc42, I am also glad that the problem disappeared with the bootloader 2.3, but I am curious if the above symptoms match the potential problem with Bootloader 2.2

                      undefined 1 Reply Last reply 18 May 2021, 07:28 Reply Quote 0
                      • undefined
                        dc42 administrators @psyblue
                        last edited by 18 May 2021, 07:28

                        @psyblue the main change between bootloader 2.2 and 2.3 is that in 2.2 if it didn't get a satisfactory frequency reading within a certain time, its reset the processor to start again. In 2.3 it just keeps on taking readings, in effect allowing the crystal oscillator as long as it needs to stabilise.

                        Duet WiFi hardware designer and firmware engineer
                        Please do not ask me for Duet support via PM or email, use the forum
                        http://www.escher3d.com, https://miscsolutions.wordpress.com

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post
                        Unless otherwise noted, all forum content is licensed under CC-BY-SA