Duet 3 MB6HC Stuck Off

kurt5105

Hi, I've been trying to get berd air to work with my Duet 3 with raspberry pi 4 for hours now, and now the board is just stuck saying "off" and accepting commands and just permanently spinning on them. The reset switch does nothing, toggling the PSU and/or the raspberry pi does nothing to fix this state. All 4 bottom LEDs are on and the top one is flashing red on a regular interval. The board was previously working. I'm using the berd air with the ez switch.

I have all 6 motors wired, a bltouch, a 24v always on fan and two noctua fans (out4,5) connected to the board. Out4-6 are set to 12v and out7-9 are VIN, which is 24v (in other words the default jumper layout).

I'll start from the top. I first started by following the berd air diagram on their site to set up the switch. I was using out7 to control the switch. The Duet would boot and everything was fine, except the air pump was always on. Even when I unplugged the wires, the switch still somehow was supplying power. I wasn't sure if this was the switch or not, but I found in a thread that the out7 and not the V_OUTLC2 was switched for PWM. So I tried to use out6 gnd and out6 to control the switch. This gave a boot loop.

Then I switched to another mosfet board that I had. If I connect the new mosfet in any way, the board gets stuck in a boot loop, none of the lights turn on. This mosfet is the HiLetgo IRF520 MOSFET Driver Module, it has a SIG, VCC and GND pin on the board instead of just a + and -. I first wired this to out6 GND, V_OUTLC1 and out6 (out6 to SIG). Boot loop. Disconnect it, and the air pump is stuck on and the Duet 3 would boot. I then realized that the VCC was 5v, oops.

Tried a new mosfet module except connected it to the 5v PWM. Boot loop when connected, but when disconnected the board is stuck off in its status. An error popped up once that mentioned no headers but I double checked that the ribbon cable was secure along with all the power cables. Despite the board being unresponsive, all 4 bottom LEDs are on and the top LED blinks.

I can't use gcode commands to output debug info with the board stuck in the off state. The firmware is 3.2.2. No errors are present in the console, with the cable plugged in I get DCS has stopped spammed.

Any ideas on how to fix this (or get more info as to what's happening)?

dc42

@kurt5105 how are you providing power to the switch that feeds power to the Berd Air? The 12V regulator on the Duet cannot supply the amount of current that it needs. If you overload the 12V rail then the Duet status will read Off.

kurt5105

@dc42 I supply the power from the power supply directly. So the board and the berd air get their own power from the PSU, then I haven't had much success using one of the fan outputs to control things.

I saw another thread that mentioned PWM didn't work unless it was 20k and under and I noticed other mosfet modules saying 20khz max. I gave that a shot on the ez switch and initially had more problems pop up. My raspberry pi ended up refusing to boot. I swapped it out for another one, and I did an apt-get update and apt-get upgrade. I went to restart and this one refused to boot too. Eventually I found out if I disconnected the power and the touch usb connector of my display, the pi booted.

Once I got that up and running again, the Duet 3 was set to idle. I was able to verify that changing the fan % actually made the brightness of the LED on the ez switch change, and at 0% it was off. However, the air pump stays on at 100%. I also had the Duet 3 restart randomly. Finally, while I was writing this, the Duet 3 returned to the "off" state, adding this to the console: Warning: Lost connection to Duet (Board is not available (no header response))

Since the board was actually running before, I was able to put in m122 and m122 "DSF" both times. I'm having problems uploading it directly here, so here's the logs: https://pastebin.com/5MeGenLP

Any ideas on how to fix this or what I can do to get more debug output now that the Duet is off and won't respond to m122?

kurt5105

Here is a diagram to hopefully clear things up on the wiring side and the wiring used for my previous attempts: https://i.imgur.com/D6WWR7b.png

To clear up my current problems (see previous posts for more in depth of the process/other symptoms that might help diagnosing):

no header response error for duet board, status reporting as off afterwards
fan pwm is changing switch LED intensity, but the air pump doesn't seem to react at all and 0% doesn't turn it off

Given the LED changes intensity maybe I'm making some wiring or config mistake. Here is how I'm setting up the fan:

M950 F0 C"out7" Q20000
M106 P0 S1 H-1

The board behavior with my past attempts is also odd, why does having these fan pins connected cause the board to disconnect/boot loop? I'd be inclined to blame the ez switch itself but I tried a mosfet module that I know works with another printer and that caused a boot loop.

Phaedrux

@kurt5105 said in Duet 3 MB6HC Stuck Off:

Eventually I found out if I disconnected the power and the touch usb connector of my display, the pi booted.

How are you powering the Pi and the Duet? Do they each have their own independent supply, or are you trying to power one from the other?

kurt5105

@phaedrux The Duet 3 is powered by a Meanwell PSU while the pi is powered by a 5v 2a power adapter. I have a touchscreen that is also powered by some power adapter that has some USB connector that plugs into the pi to enable touch.

The pi is connected to the Duet via the included ribbon cable, so I'm not sure if there is power draw from that too, but the pi is able to remain on regardless of the Duet since it has its own independent power supply. Not really sure why the touch connector was causing the boot problems for the pi, but it doesn't seem to have affected the Duet at all.

Any extra info I can grab from the pi itself like some debug logs on why the Duet 3 is showing up as "off" but has all the power LEDs on and the diagnostic LED blinking?

Phaedrux

Ok your power setup sounds ok. The Duet can provide power to the SBC, but it's not recommended for the Pi4 because it can draw too much especially with a display connected.

The Duet has a jumper to configure the 5v power distribution. How do you have yours configured?

https://duet3d.dozuki.com/Wiki/Duet_3_Mainboard_6HC_Hardware_Overview#Section_5V

@kurt5105 said in Duet 3 MB6HC Stuck Off:

Any extra info I can grab from the pi itself like some debug logs on why the Duet 3 is showing up as "off" but has all the power LEDs on and the diagnostic LED blinking?

Indeed there is: https://duet3d.dozuki.com/Wiki/Getting_Started_With_Duet_3#Section_Monitoring_optional

One other thing to check, do you have an SD card in the Duet? When using the SBC option there should not be.

Another way to simplify the setup and troubleshoot is to see if you can get it running in standalone mode first without the SBC attached at all. This way you can confirm if the Duet3 itself is functioning correctly.

https://duet3d.dozuki.com/Wiki/Getting_Started_With_Duet_3#Section_Running_in_standalone_mode

kurt5105

@phaedrux I only have Int 5V EN jumped, and no SD card in the Duet. I have tried 2 different pi 4's by swapping the SD card and they both hit the same error in the web control. Thanks to the logging info you linked, I was able to see on one board what was happening!

Here are the logs:
logs.txt

The useful part is:

[fatal] SPI task faulted
   System.Exception: RepRapFirmware refused message format
   at DuetControlServer.SPI.DataTransfer.ExchangeHeader() in /home/christian/Duet3D/DuetSoftwareFramework/src/DuetControlServer/SPI/DataTransfer.cs:line 1283
   at DuetControlServer.SPI.DataTransfer.PerformFullTransfer(Boolean connecting) in /home/christian/Duet3D/DuetSoftwareFramework/src/DuetControlServer/SPI/DataTransfer.cs:line 163
   at DuetControlServer.SPI.Interface.Run() in /home/christian/Duet3D/DuetSoftwareFramework/src/DuetControlServer/SPI/Interface.cs:line 903

So the communication for some reason doesn't seem to be working well, and it's 2 pis vs 1 Duet in this case. The CRC32 hashes are completely incorrect many times so the control server is throwing. I've double checked that the ribbon is connected firmly on both ends. I could try another cable?

Anything else I might be missing here? Why would this connection suddenly be a problem after I previously had this working for multiple hours? I changed the extruder motor and added the Berd Air, but none of that seems like it should have an effect on the SPI.

Phaedrux

Can you go through this as well?

https://github.com/Duet3D/DuetSoftwareFramework/wiki/SBC-Setup-Guide#troubleshooting

chrishamm

@kurt5105 If you got CRC32 errors in the past and if you can confirm the firmware on the Duet matches the DSF version on your Pi, it's definitely worth a try to replace the ribbon cable (for testing jumper wires should do). Performing the troubleshooting steps as suggested by @Phaedrux is a good idea too.

kurt5105

@phaedrux done, it seems like the original ribbon cable has continuity between each corresponding pin, though I'm not actually sure how I would test the quality of the signal.

@chrishamm I replaced the cable and swapped the pi and so far so good. I'm going to keep it running a bit and try printing something. m122 reports the SBC state 4 with 0 disconnects and the rx/tx numbers are the same (I'm guessing this is indicative of the communication being stable). This pi was having issues yesterday too, so I'm inclined to say it was something with the cable.

Oddly enough though, I still have that same issue with the berd air/switch. I can change the fan % of fan 0, the switch LED changes brightness, but the berd air keeps spinning at the same speed. Any easy way to check if the switch is working correctly with a multimeter? I could try the other version of relay (the one in my wiring diagram with sig/vcc/gnd) in the 5v PWM slot if that seems reasonable (and I take it that it would be C"out9" in config despite there being a fan slot labeled out 9 as well).

chrishamm

@kurt5105 No disconnects and equal rx/tx sequence numbers are indeed a good sign. We recently had a user with a damaged SD card that affected the communication as well (just an idea).

If you're worried about a particular output and still have a spare one, perhaps just try to swap it (temporarily). If you need to turn off an output but it remains on when a heater is hot, you need to disable thermostatic control of the fan.

kurt5105

@chrishamm I got a no data response once (but the board immediately reconnected and wasn't in the off state) so far but that's it. I'll give the new relay a shot tomorrow and see if that works and doesn't cause the weird boot loop I was seeing before. I'm also getting a proper 26 to 40 pin cable as the cable I initially got was 26 to 26 which I soon found out the reason for the blank 14 pins and had to jump the rest. Maybe that will fix the no data response. It's still quite odd to me that the cable decided to fail at the exact time I made changes to the printer, and I'm not really sure what I could have done to affect it.

It makes sense to me why I was asked about the SD card so much, that's definitely a really interesting failure. I've personally never had an SD card fail on me and from what I understand they are super resilient.

For the berd air I don't have thermostatic control enabled, this is the gcode:

M950 F0 C"out7" Q20000
M106 P0 S1 H-1

I've definitely struggled setting up the fans on this board. Previously when I added my 2 noctua fans, they just completely stopped spinning and I couldn't control them. I think the default pinout is correct on the connectors, so I'm not really sure why they spin when not added in the config but refuse when added. The docs say "Mixed-voltage setups are not directly supported" but does that mean that having the first row of 4 pin fan slots at 12v and the next row of 2 pins at VIN (in my case 24v) is not supported (thus this fan issue is just undefined behavior)?

chrishamm

@kurt5105 From my experience some fans may struggle with high PWM frequencies like yours. That's why we recommend a default frequency of 500Hz or possibly even less. If they still refuse to turn on at low PWM levels, you may want to increase the blip time a bit.

kurt5105

@chrishamm so I tried the new mosfet board and that still didn't affect whether or not the berd air motor was running. I was getting several DCS stopped errors too. I got logging enabled again, only to see bad CRC32 warnings appearing in the console. When it started to print, I got a warning that phase A in one of my motors was disconnected. It's a good thing I got the warning, since I have a triple axis bed and that would have collided otherwise.

I didn't touch the motor at all, and it definitely is connected. The ribbon cable is the same one that was seemingly working too. Could this somehow be related to the mosfet board? I didn't try the one that was causing a constant boot loop yet, but perhaps that would give us more info? Could this fan output be somehow messing with the rest of the board?