• Tags
  • Documentation
  • Order
  • Register
  • Login
Duet3D Logo Duet3D
  • Tags
  • Documentation
  • Order
  • Register
  • Login

1LC Multiple Disconnects

Scheduled Pinned Locked Moved Solved
Duet Hardware and wiring
5
26
1.0k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • undefined
    dc42 administrators @curieos
    last edited by 1 Feb 2024, 19:25

    @curieos said in 1LC Multiple Disconnects:

    Checking now with a volt meter (the board is currently locked up), VIN is reading fine, 27V. 3.3V from the IO 1 connector reads 3.3V, and 5V from the IO 2 connector reads just shy of 5V (around 4.95), which I believe is also fine.

    Yes those are fine; but that doesn't prove that they are always fine. If you have a bad crimp or a fractured wire then the power is likely to disconnect occasionally depending on the movement of the power cable, which likely depends on the position of the print head. If the disconnection is short then the capacitors in the power circuit may supply sufficient power for long enough until the power connection is restored.

    The fact that you saw the red LED gradually dim is a sure sign that the 5V supply was gradually lost. Once power is lost, when it is restored the configuration parameters will be lost, so you will need to reboot the machine or at least run config.g to restore normal operation.

    Have you provided any strain relief on the power cable, after it has exited the JST VH connector?

    Are you powering anything from OUT0 ?

    Duet WiFi hardware designer and firmware engineer
    Please do not ask me for Duet support via PM or email, use the forum
    http://www.escher3d.com, https://miscsolutions.wordpress.com

    undefined 1 Reply Last reply 1 Feb 2024, 19:31 Reply Quote 0
    • undefined
      curieos @dc42
      last edited by 1 Feb 2024, 19:31

      @dc42 The red status light, not the red 5V light. The red 5V light stays brightly lit the whole time. The board is locked up, I can not communicate with it in any form. When I send M122 B21 it does not respond. This isn't a momentary issue.

      Yes, there is plenty of strain relief. A zip tie secures the power and CAN cables after they exit the chain for both toolheads. There's also cable management to make sure the cover doesn't squish any wires.

      OUT0 is hooked up to two 50 watt Slice heaters. During this last print that caused this issue, those heaters were never activated. That tool was unused and idle the whole time.

      She/Her
      I work at a local 3D printing shop.
      Printers: Micron+ w/Duet 3 Mini, in-progress adaptation of the Jubilee REL onto an E3D MS, Prusa i3 MK3S.

      undefined 1 Reply Last reply 1 Feb 2024, 19:44 Reply Quote 0
      • undefined
        curieos @curieos
        last edited by curieos 2 Jan 2024, 20:10 1 Feb 2024, 19:44

        @dc42 Here's some pictures of the toolboard, right now. Both voltage lights are lit up. The CAN activity and status lights are off. It's difficult to illustrate this without taking a video.

        IMG_20240201_133214.jpg IMG_20240201_133201.jpg

        I'm trying to get some thermal shots of it to see if there's any hotspots I can't detect, but our thermal camera is refusing to cooperate. edit: I got a thermal camera image. There's a bit of an offset due to how close I have the camera, but you can get an idea of where the hot spots are based on the ghost of the wires and mounting holes:
        FLIR_20240201_015736_627.jpg

        I believe the hotspot is U3, which is the 12V buck converter.

        She/Her
        I work at a local 3D printing shop.
        Printers: Micron+ w/Duet 3 Mini, in-progress adaptation of the Jubilee REL onto an E3D MS, Prusa i3 MK3S.

        undefined 1 Reply Last reply 2 Feb 2024, 08:35 Reply Quote 0
        • undefined
          dc42 administrators @curieos
          last edited by 2 Feb 2024, 08:35

          @curieos thanks for the photos. If the hotspot is that area indicated by the red circle and cross, that's diode D5 which feeds the output of the 12V regulator to the VOUT pin of the OUT1 and OUT2 connectors. It would be odd if that component was generating a lot of heat. What I suspect is that the surrounding area is at more or less the same temperature and the heat is coming from both the 12V buck regulator and the 5V linear regulator.

          Can you confirm that you are not using the IO_0 connector to power anything? It's not connected in your photo.

          It's nor normal for the Status lights to be off and the power lights to be on. That condition should only occur briefly after power up when the crystal oscillator is starting and the board is booting up. So I suspect that the processor has gone into some sort of locked up state that even the watchdog timer can't recover from, or the oscillator has stopped. The only reasons I can think of for it locking up in this way are static discharge, a a loss of power that wasn't quite enough to cause a full shutdown, or a faulty tool board. We know that static discharge is common on hot ends, but we haven't know of it affecting a TOOL1LC in this way. Nevertheless, please ensure that the hot end metalwork is grounded. You can ground it to one of the mounting screws of the tool board.

          What was the reason for putting a heatsink on top of the MCU? How did you attach it? The MCU generates very little heat and certainly doesn't need a heatsink. The components that generate heat on the tool board are the 5V linear regulator, the TMC2209 driver (depending on motor current), and the 12V buck regulator (depending on VIN voltage and the current draw from OUT1 and OUT2).

          I'd suggest that we replace your tool board, but you said you have already swapped it for a different one.

          Duet WiFi hardware designer and firmware engineer
          Please do not ask me for Duet support via PM or email, use the forum
          http://www.escher3d.com, https://miscsolutions.wordpress.com

          undefined 3 Replies Last reply 2 Feb 2024, 13:28 Reply Quote 0
          • undefined
            curieos @dc42
            last edited by 2 Feb 2024, 13:28

            @dc42 Correct, IO_0 is not connected. You said OUT0 earlier, which was confusing.

            The hotend metalwork and toolboard are connected via the toolplate. The hotend has continuity with the plate, and the toolboard has a connection to the toolplate via the green wire in the upper right corner.

            I put the heatsink on the MCU because I noticed the previous toolboard's MCU was very hot to the touch before I swapped it out. I put it on that toolboard, and then transferred it to the new one when I swapped it out. Originally I thought it was just an overheating issue. The heatsink has preattached thermal adhesive. Not shown is a thin ceramic heatsink on the rear of the toolboard, mounted with some 3M thermal transfer adhesive. That covers the MCU and stepper driver.

            She/Her
            I work at a local 3D printing shop.
            Printers: Micron+ w/Duet 3 Mini, in-progress adaptation of the Jubilee REL onto an E3D MS, Prusa i3 MK3S.

            1 Reply Last reply Reply Quote 0
            • undefined
              curieos @dc42
              last edited by 8 Feb 2024, 17:34

              @dc42 Any other suggestions? I just replaced the power cable for the toolboard with brand new wire and this issue is still occurring.

              She/Her
              I work at a local 3D printing shop.
              Printers: Micron+ w/Duet 3 Mini, in-progress adaptation of the Jubilee REL onto an E3D MS, Prusa i3 MK3S.

              1 Reply Last reply Reply Quote 0
              • undefined
                curieos @dc42
                last edited by 20 Feb 2024, 15:14

                @dc42 I tried updating that machine to 3.5 rc3 and the issue persists. What physically can cause a 1LC to lock up until a full power cycle?

                She/Her
                I work at a local 3D printing shop.
                Printers: Micron+ w/Duet 3 Mini, in-progress adaptation of the Jubilee REL onto an E3D MS, Prusa i3 MK3S.

                1 Reply Last reply Reply Quote 0
                • undefined
                  Phaedrux Moderator
                  last edited by 20 Feb 2024, 19:22

                  Are the mainboard and toolboard powered from a common PSU? If not, are the grounds of the PSU tied together?

                  Z-Bot CoreXY Build | Thingiverse Profile

                  undefined 1 Reply Last reply 20 Feb 2024, 19:23 Reply Quote 0
                  • undefined
                    curieos @Phaedrux
                    last edited by 20 Feb 2024, 19:23

                    @Phaedrux Yes, all of the 24V system is connected to one supply.

                    She/Her
                    I work at a local 3D printing shop.
                    Printers: Micron+ w/Duet 3 Mini, in-progress adaptation of the Jubilee REL onto an E3D MS, Prusa i3 MK3S.

                    undefined 1 Reply Last reply 14 Mar 2024, 20:10 Reply Quote 0
                    • undefined
                      curieos @curieos
                      last edited by curieos 14 Mar 2024, 20:10

                      @dc42 I'm reporting in that this issue is still occurring. I have changed out the CAN data and power cable runs to this toolhead yet again, in fact the run is a little different this time. The tool distribution board is now mounted to the gantry, so the power and data cables are shorter than before.

                      The same toolboard locks up at around the same point in any print I've tried, typically around the 1 hour mark. It's been replaced once already. The board does not recover until a full power cycle of the machine is performed.

                      I have an M122 output from this latest event:

                      === Diagnostics ===
                      RepRapFirmware for Duet 3 MB6XD version 3.5.0-rc.3 (2024-01-24 17:59:29) running on Duet 3 MB6XD v1.01 or later (SBC mode)
                      Board ID: 0JD2M-999AL-D25SW-6JKD0-3S06R-95ZB1
                      Used output buffers: 1 of 40 (40 max)
                      === RTOS ===
                      Static ram: 153600
                      Dynamic ram: 92392 of which 0 recycled
                      Never used RAM 96400, free system stack 122 words
                      Tasks: SBC(2,ready,14.4%,407) HEAT(3,nWait 6,0.2%,322) Move(4,nWait 6,8.7%,213) CanReceiv(6,nWait 1,28.2%,770) CanSender(5,nWait 7,0.1%,325) CanClock(7,delaying,0.1%,346) MAIN(2,running,47.7%,444) IDLE(0,ready,0.6%,30), total 100.0%
                      Owned mutexes: HTTP(MAIN)
                      === Platform ===
                      Last reset 03:30:24 ago, cause: power up
                      Last software reset at 2024-03-07 09:55, reason: User, Gcodes spinning, available RAM 95784, slot 0
                      Software reset code 0x6003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task SBC Freestk 0 n/a
                      Error status: 0x04
                      Aux0 errors 0,0,0
                      MCU temperature: min 30.9, current 48.1, max 49.9
                      Supply voltage: min 25.8, current 26.0, max 26.1, under voltage events: 0, over voltage events: 0, power good: yes
                      12V rail voltage: min 12.1, current 12.1, max 12.2, under voltage events: 0
                      Heap OK, handles allocated/used 99/38, heap memory allocated/used/recyclable 2048/1388/712, gc cycles 4
                      Events: 3 queued, 3 completed
                      Driver 0: ok
                      Driver 1: ok
                      Driver 2: ok
                      Driver 3: ok
                      Driver 4: ok
                      Driver 5: ok
                      Date/time: 2024-03-14 14:58:02
                      Slowest loop: 1000.79ms; fastest: 0.05ms
                      === Storage ===
                      Free file entries: 20
                      SD card 0 not detected, interface speed: 37.5MBytes/sec
                      SD card longest read time 0.0ms, write time 0.0ms, max retries 0
                      === Move ===
                      DMs created 125, segments created 31, maxWait 3494446ms, bed compensation in use: none, height map offset 0.000, max steps late 1, ebfmin 0.00, ebfmax 0.00
                      no step interrupt scheduled
                      Moves shaped first try 9729, on retry 4260, too short 16445, wrong shape 17913, maybepossible 1481
                      === DDARing 0 ===
                      Scheduled moves 59311, completed 59311, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 4], CDDA state -1
                      === DDARing 1 ===
                      Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
                      === Heat ===
                      Bed heaters 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
                      === GCodes ===
                      Movement locks held by null, null
                      HTTP* is doing "M122" in state(s) 0
                      Telnet is idle in state(s) 0
                      File* is idle in state(s) 0
                      USB is idle in state(s) 0
                      Aux is idle in state(s) 0
                      Trigger* is idle in state(s) 0
                      Queue* is idle in state(s) 0
                      LCD is idle in state(s) 0
                      SBC* is idle in state(s) 0
                      Daemon is idle in state(s) 0
                      Aux2 is idle in state(s) 0
                      Autopause* is idle in state(s) 0
                      File2* is idle in state(s) 0
                      Queue2 is idle in state(s) 0
                      Q0 segments left 0, axes/extruders owned 0x0000000
                      Code queue 0 is empty
                      Q1 segments left 0, axes/extruders owned 0x0000000
                      Code queue 1 is empty
                      === Filament sensors ===
                      check 0 clear 0
                      Extruder 0: no data received, errs: frame 0 parity 0 ovrun 0 pol 0 ovdue 0
                      Extruder 1: no data received, errs: frame 0 parity 0 ovrun 0 pol 0 ovdue 0
                      === CAN ===
                      Messages queued 189586, received 21212642, lost 0, errs 586337, boc 48
                      Longest wait 208ms for reply type 6033, peak Tx sync delay 1409, free buffers 50 (min 47), ts 63121/63096/0
                      Tx timeouts 0,0,0,0,0,0
                      === SBC interface ===
                      Transfer state: 5, failed transfers: 0, checksum errors: 0
                      RX/TX seq numbers: 57670/57670
                      SPI underruns 0, overruns 0
                      State: 5, disconnects: 0, timeouts: 0 total, 0 by SBC, IAP RAM available 0x2600c
                      Buffer RX/TX: 0/0-0, open files: 0
                      === Duet Control Server ===
                      Duet Control Server version 3.5.0-rc.3 (2024-01-26 12:34:19)
                      File+ProcessInternally:
                      > Busy
                      Failed to deserialize the following properties:
                      - Board -> BoardState from "timedOut"
                      Code buffer space: 4096
                      Configured SPI speed: 8000000Hz, TfrRdy pin glitches: 1
                      Full transfers per second: 39.34, max time between full transfers: 73.4ms, max pin wait times: 69.8ms/18.6ms
                      Codes per second: 13.12
                      Maximum length of RX/TX data transfers: 5088/3144
                      

                      I'm going to try disconnecting all connections to the board except for the endstop required for homing, and then I'll run another print. Let me know if there's any additional steps I should try.

                      edit: Here's the M122 for the board that locked up after a restart

                      Diagnostics for board 21:
                      Duet TOOL1LC rev 1.1 or later firmware version 3.5.0-rc.3 (2024-01-24 17:55:14)
                      Bootloader ID: SAMC21 bootloader version 2.8 (2023-07-25)
                      All averaging filters OK
                      Never used RAM 2936, free system stack 104 words
                      Tasks: Move(3,nWait 7,0.0%,135) HEAT(2,nWait 6,0.2%,127) CanAsync(5,nWait 4,0.0%,51) CanRecv(3,nWait 1,0.0%,71) CanClock(5,nWait 1,0.0%,59) ACCEL(3,nWait 6,0.0%,53) TMC(2,delaying,3.4%,53) MAIN(1,running,91.7%,315) IDLE(0,ready,0.0%,27) AIN(2,delaying,4.6%,112), total 100.0%
                      Owned mutexes:
                      Last reset 00:02:17 ago, cause: power up
                      Last software reset at 2024-02-01 11:32, reason: HardFault, available RAM 2016, slot 0
                      Software reset code 0x0060 ICSR 0x00000003 SP 0x20002f08 Task MAIN Freestk 813 ok
                      Stack: 200048c8 00000000 00000001 4aa36799 00000000 0000f3cb 4aa36798 21000000 20001488 0000f00d a5a5a5a5 0080bdc1 00000089 000001f4 20001ce0 0001285f 00000000 00000000 20004370 a5a5a5a5 a5a5a5a5 a5a5a5a5 a5a5a5a5 a5a5a5a5 a5a5a5a5 a5a5a5a5 a5a5a5a5
                      Driver 0: pos 0, 404.8 steps/mm, standstill, SG min 0, read errors 0, write errors 0, ifcnt 12, reads 2957, writes 12, timeouts 0, DMA errors 0, CC errors 0, steps req 0 done 0
                      Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
                      Peak sync jitter 0/4, peak Rx sync delay 207, resyncs 0/0, no timer interrupt scheduled
                      VIN voltage: min 26.7, current 26.8, max 26.8
                      MCU temperature: min 34.8C, current 41.4C, max 41.4C
                      Last sensors broadcast 0x00018004 found 3 35 ticks ago, 0 ordering errs, loop time 0
                      CAN messages queued 2530, send timeouts 0, received 2038, lost 0, errs 0, boc 0, free buffers 18, min 18, error reg 0
                      dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
                      Accelerometer: LIS3DH, status: 00
                      I2C bus errors 0, naks 3, contentions 0, other errors 0
                      === Filament sensors ===
                      Interrupt 5726621 to 0us, poll 6 to 542us
                      Driver 0: no data received, errs: frame 0 parity 0 ovrun 0 pol 0 ovdue 0
                      

                      She/Her
                      I work at a local 3D printing shop.
                      Printers: Micron+ w/Duet 3 Mini, in-progress adaptation of the Jubilee REL onto an E3D MS, Prusa i3 MK3S.

                      1 Reply Last reply Reply Quote 0
                      • undefined
                        dc42 administrators @curieos
                        last edited by dc42 15 Mar 2024, 09:36

                        @curieos said in 1LC Multiple Disconnects:

                        The only things that changed between the toolboard working fine and it having issues were a grounding wire run to the toolhead plate, and a new CAN cable with crimped terminations instead of the soldered pigtails.

                        What is the grounding wire connected to? If it's connected to mains ground, is Duet ground also connected to mains ground?

                        Is the extruder motor body also grounded?

                        Duet WiFi hardware designer and firmware engineer
                        Please do not ask me for Duet support via PM or email, use the forum
                        http://www.escher3d.com, https://miscsolutions.wordpress.com

                        undefined 1 Reply Last reply 15 Mar 2024, 12:39 Reply Quote 0
                        • undefined
                          curieos @dc42
                          last edited by 15 Mar 2024, 12:39

                          @dc42 The grounding wire is connected to earth ground, and it screws into an aluminum toolhead plate. The grounding wire connected to the 1LC via one of the mounting holes is also connected to the toolhead plate. I can see if there's a low impedance connection between DC ground and earth ground when I get into work today, I assume this would be an issue.

                          The extruder motor body is physically touching said toolhead plate, and the mounting screws go through the toolhead plate. The toolhead plate is anodized, so it's not the most solid connection, but I've checked with a multimeter and there is continuity. The extruder is an LGX Pro, so the hotend mount is made of anodized aluminum, and the hotend metalwork is connected to the toolhead plate through the extruder mounting. I've checked with a multimeter and it detects continuity. If this isn't good enough I can run a discrete grounding wire from the toolhead plate to the coldside of the hotend.

                          She/Her
                          I work at a local 3D printing shop.
                          Printers: Micron+ w/Duet 3 Mini, in-progress adaptation of the Jubilee REL onto an E3D MS, Prusa i3 MK3S.

                          undefined 1 Reply Last reply 20 Mar 2024, 15:48 Reply Quote 0
                          • undefined
                            curieos @curieos
                            last edited by 20 Mar 2024, 15:48

                            @dc42 I may have fixed it.

                            Turns out on this unit, there was a zero impedance connection between DC ground and Earth ground. I found the culprit, the cutout for the ethernet extension plugged into the pi was made upside down. This meant the shielding on the plug was in contact with the aluminum frame panel. The panel is powder coated, but it appears the coating wore away in this area. I flipped it upside down and now the grounds aren't shorted.

                            I ran a 6 hour print last night and there were no issues. The longest I've seen it go without locking up was 3-4 hours, so I think this means the problem is solved. I'm running a slightly longer print (22.5 hours) now, so that will definitely be definitive.

                            She/Her
                            I work at a local 3D printing shop.
                            Printers: Micron+ w/Duet 3 Mini, in-progress adaptation of the Jubilee REL onto an E3D MS, Prusa i3 MK3S.

                            undefined 1 Reply Last reply 20 Mar 2024, 16:25 Reply Quote 1
                            • undefined
                              dc42 administrators @curieos
                              last edited by dc42 20 Mar 2024, 16:25

                              @curieos I'm glad you appear to have found the cause! Ground loops can cause various odd behaviour, just like ESD can.

                              I'll mark this thread as solved but feel free to revert it if the problem recurs.

                              Duet WiFi hardware designer and firmware engineer
                              Please do not ask me for Duet support via PM or email, use the forum
                              http://www.escher3d.com, https://miscsolutions.wordpress.com

                              1 Reply Last reply Reply Quote 0
                              • undefined dc42 has marked this topic as solved 20 Mar 2024, 16:25
                              • First post
                                Last post
                              Unless otherwise noted, all forum content is licensed under CC-BY-SA