Strange Motor Controller Failures

[BPisLife]

I got a real strange one:

I noticed my extruder suddenly stopped responding properly (ie very weak) so I tried changing the motor. when that didn't work I moved the motor to a different duetwifi where it worked like a charm. I tried driver 2-4, and 8 and 9 (only free ones) and all where weak in driving the motor. Now one thing that’s interesting is that we did have a power brown out here from a bad rain storm and I wonder if that cooked any motor driver that didn’t have a large motor (inductance) on it as the motor that is used is the e3d small motor. This is a huge guess by the way as I have no way of knowing as the printer has been sitting for the last few weeks. Also I never connected or disconnected the motor when power was on.

It literally seems like the motor is running very weak as if it isn't wired properly but I confirmed it on a different duet board and with the exact same cable and it worked perfect. Just odd that every controller is behaving this way except for those with large steppers plugged into it (including those on Duex5).

[dc42]

It sounds to me like one of the following:

• Configuration issue. Are you sure you haven't accidentally reduced the motor current setting?
• Power issue. Check that the VIN terminal block screws are still tight on the Duet and the DueX, and use M122 to check whether the power voltage is stable when you try to extrude.

[BPisLife]

@dc42 Didn't change the config from the last time I used it and it was working great with dozens of successful prints. double checked.

M906 X1000 Y1000 Z1000 E1000:1000:1000:1000:1000:1000:1000:1000:1000 I60 ; Set motor currents (mA)

M122:
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 2.01(RTOS) running on Duet WiFi 1.02 or later + DueX5
Board ID: 08DGM-956GU-DJMSJ-6J9FG-3SJ6J-9AQZF
Used output buffers: 1 of 20 (12 max)
=== RTOS ===
Static ram: 28476
Dynamic ram: 96240 of which 16 recycled
Exception stack ram used: 328
Never used ram: 6012
Tasks: NETWORK(ready,460) HEAT(blocked,1192) MAIN(running,3476)
Owned mutexes:
=== Platform ===
Last reset 00:00:35 ago, cause: software
Last software reset at 2018-07-27 11:27, reason: User, spinning module GCodes, available RAM 6012 bytes (slot 2)
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00417000 BFAR 0xe000ed38 SP 0xffffffff Task 0x4e49414d
Error status: 0
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest block write time: 0.0ms, max retries 0
MCU temperature: min 34.5, current 35.7, max 36.9
Supply voltage: min 24.1, current 24.3, max 24.5, under voltage events: 0, over voltage events: 0
Driver 0: standstill, SG min/max not available
Driver 1: standstill, SG min/max not available
Driver 2: standstill, SG min/max not available
Driver 3: standstill, SG min/max not available
Driver 4: standstill, SG min/max not available
Driver 5: standstill, SG min/max not available
Driver 6: standstill, SG min/max not available
Driver 7: standstill, SG min/max not available
Driver 8: standstill, SG min/max not available
Driver 9: standstill, SG min/max not available
Expansion motor(s) stall indication: no
Date/time: 2018-07-27 11:28:24
Slowest loop: 5.56ms; fastest: 0.08ms
=== Move ===
Hiccups: 0, StepErrors: 0, LaErrors: 0, FreeDm: 240, MinFreeDm: 240, MaxWait: 0ms, Underruns: 0, 0
Scheduled moves: 0, completed moves: 0
Bed compensation in use: none
Bed probe heights: 0.000 0.000 0.000 0.000 0.000
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1
Heater 1 is on, I-accum = 0.6
=== GCodes ===
Segments left: 0
Stack records: 1 allocated, 0 in use
Movement lock held by null
http is idle in state(s) 0
telnet is idle in state(s) 0
file is idle in state(s) 0
serial is idle in state(s) 0
aux is idle in state(s) 0
daemon is idle in state(s) 0
queue is idle in state(s) 0
autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 23.14ms; fastest: 0.01ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 1 of 8

• WiFi -
  Network state is running
  WiFi module is connected to access point
  Failed messages: pending 0, notready 0, noresp 0
  WiFi firmware version 1.21
  WiFi MAC address 5c:cf:7f:76:6d:51
  WiFi Vcc 3.39, reset reason Turned on by main processor
  WiFi flash size 4194304, free heap 15456
  WiFi IP address 172.24.1.104
  WiFi signal strength -39dBm, reconnections 0, sleep mode modem
  Socket states: 0 0 0 0 0 0 0 0
  === Expansion ===
  DueX I2C errors 0

Started the motor turning, well not really turning but it should be
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 2.01(RTOS) running on Duet WiFi 1.02 or later + DueX5
Board ID: 08DGM-956GU-DJMSJ-6J9FG-3SJ6J-9AQZF
Used output buffers: 3 of 20 (17 max)
=== RTOS ===
Static ram: 28476
Dynamic ram: 96240 of which 16 recycled
Exception stack ram used: 328
Never used ram: 6012
Tasks: NETWORK(ready,460) HEAT(blocked,1192) MAIN(running,3476)
Owned mutexes:
=== Platform ===
Last reset 00:01:31 ago, cause: software
Last software reset at 2018-07-27 11:27, reason: User, spinning module GCodes, available RAM 6012 bytes (slot 2)
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00417000 BFAR 0xe000ed38 SP 0xffffffff Task 0x4e49414d
Error status: 0
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest block write time: 0.0ms, max retries 0
MCU temperature: min 35.7, current 35.7, max 36.0
Supply voltage: min 24.2, current 24.3, max 24.5, under voltage events: 0, over voltage events: 0
Driver 0: standstill, SG min/max not available
Driver 1: standstill, SG min/max not available
Driver 2: open-load-A open-load-B, SG min/max not available
Driver 3: standstill, SG min/max not available
Driver 4: standstill, SG min/max not available
Driver 5: standstill, SG min/max not available
Driver 6: standstill, SG min/max not available
Driver 7: standstill, SG min/max not available
Driver 8: standstill, SG min/max not available
Driver 9: standstill, SG min/max not available
Expansion motor(s) stall indication: no
Date/time: 2018-07-27 11:29:21
Slowest loop: 1.73ms; fastest: 0.08ms
=== Move ===
Hiccups: 0, StepErrors: 0, LaErrors: 0, FreeDm: 239, MinFreeDm: 239, MaxWait: 0ms, Underruns: 0, 0
Scheduled moves: 1, completed moves: 0
Bed compensation in use: none
Bed probe heights: 0.000 0.000 0.000 0.000 0.000
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1
Heater 1 is on, I-accum = 0.3
=== GCodes ===
Segments left: 0
Stack records: 1 allocated, 0 in use
Movement lock held by null
http is idle in state(s) 0
telnet is idle in state(s) 0
file is idle in state(s) 0
serial is idle in state(s) 0
aux is idle in state(s) 0
daemon is idle in state(s) 0
queue is idle in state(s) 0
autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 178.62ms; fastest: 0.08ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 1 of 8

• WiFi -
  Network state is running
  WiFi module is connected to access point
  Failed messages: pending 0, notready 0, noresp 0
  WiFi firmware version 1.21
  WiFi MAC address 5c:cf:7f:76:6d:51
  WiFi Vcc 3.39, reset reason Turned on by main processor
  WiFi flash size 4194304, free heap 16920
  WiFi IP address 172.24.1.104
  WiFi signal strength -40dBm, reconnections 0, sleep mode modem
  Socket states: 0 0 0 0 0 0 0 0
  === Expansion ===
  DueX I2C errors 0

[BPisLife]

Ok, despite everyone in my lab thinking I am taking this too personal, I decided to break out the o-scope and start to take some measurements. I am not familiar with what the waveform is supposed to be as I usually don't work on stepper motor drivers. I only have 2 o-scope leads as the other two are being used in an experiment that will last a few days.

First picture is taken 2A to 1A.

Second: 1A to 1B

third is across 2A to 2B:

What I noticed is that the frequency across the two windings are off by about 600Hz. Is that supposed to be that way?

[dc42]

@bpislife said in Strange Motor Controller Failures:

What I noticed is that the frequency across the two windings are off by about 600Hz. Is that supposed to be that way?

AFAIK that's entirely possible, because each winding has its own chopper. If you change the motor position a little, the frequencies will probably change a little.

It's probably best to use the 2 channels in differential mode so that you see the voltage across the stepper motor winding.

[BPisLife]

Ah good one. Let me try that and I will report back.

[BPisLife]

@dc42 Ok here is the output.

Across 2A/2B:

Across 1A/1B:

[BPisLife]

@bpislife I noticed that the square wave period never changes. On a good stepper driver as the motor moves the period adjusts. On the bad drivers the period remains the same even if I physically turn the motor manually while it tries to turn.

[BPisLife]

@bpislife
As a side note: I was able to run the motor no problem my plugging it into a port with a motor that was working. This definitely confirms there is something wrong with the drivers themselves or the hardware that supports it.

[dc42]

It does sound as though there is something wrong with the drivers, but it's a very odd fault. Does the mark-space ratio change if you vary the current using M906 or M913, or if you command the motor to move by very small amounts?

[BPisLife]

@dc42 no, speed doesn’t seem to matter. I changed current and also no change. I did adjust the steps/mm and the space between sqaure waves changed.

Edit: the still image doesn’t do it justice as it is very much a twitching unclean square wave so the controller at least appears to be trying to move but can’t. I won’t if the logic in the motor control burned out? Like I was saying before we did get a series of brown-outs/surge from a really bad storm. It’s odd that only the controllers with large motors survived. That said it was only this printer that had an issue. My other 3 have no problems.

[BPisLife]

This post is deleted!

[BPisLife]

@bpislife I also just realized that are it standard QFP with no thermal pad. I am going to replace the main driver ICs as those are a pretty straightforward board level repair if this isn’t covered under warranty (only a month old).

[dc42]

Let me see whether I can sort out a warranty repair or replacement. Which country are you in?

[BPisLife]

@dc42 USA, already reached out to Filastruder.

[BPisLife]

@bpislife Just to close this out, the warranty process is taking too long for me so I decided to replace the motor controllers on the Duex 5 first just to prove it was in fact the motor controllers (before I take on the DuetWifi).

Just to be clear that isn't a complaint at the warranty process but more my own impatience.

Replacing the motor controllers resolves the problem on the Duex 5 and I am sure it will resolve the issue on the DuetWifi also. 2 down 3 more to go. Will report back here once I replace the final 3.

[dc42]

Thanks for the update. This is a really strange fault, we've not seen anything like it before. The only way we've seen these stepper drivers fail before is open-circuit or short-circuit output mosfets. Very occasionally we've seen incorrect status reports that we suspect was caused by a bad soldered joint on one of the SPI pins.

[BPisLife]

@dc42 well after all that it worked for the duex5 but when I changed all on the duetwifi and ran the motor, I had to upgrade the wiring on the motor and when I turned the unit off and back on, all 5 MCs we’re toast. I fixed one MC in the duex5 again but it didn’t solve it this time.

Turns out that the solder joints on the Vin for the duex5 were cracked (ie connector loose) so I resoldered them but MCs are stil actng badly. I will attempt one more MC replacement. Strange that it worked now it’s not.

if this continues I will remove everything and go one at a time. I also plan on testing the power supply to make sure it is in fact functioning properly and not providing a power spike.

My thought on the MC is that the logic of the MC is getting cooked.

[BPisLife]

@dc42 Well this is embarrassing/hilarious/sad etc. see below.

The issue was the motor current (original M906 statement) and nothing at all to do with the drivers. No idea why it worked after replacing the drivers before then failing but whatever, it works now.

After going to the latest firmware (though downgrading back to 2.0 didn't fix it) the issue showed up as I descibed in this post. It dawned on me (light dawns over marble head) to just verify the M906 statement was read and behold this statement :

M906
Motor current (mA) - X:1000, Y:1000, Z:1000, E:0, idle factor 60%

Yup, must not be valid and to be honest, I don't remember where it came from, why it isn't valid, why it would work for a month then suddenly not work, but whatever the reason I don't need it.

M906 X1000 Y1000 Z1000 E1000:1000:1000:1000:1000:1000:1000:1000:1000 I60 ; Set motor currents (mA)

I changed it to just M906 X1000 Y1000 Z1000 E1000 and boom everything works.

So now that I spent time replacing 5 motor controllers for nothing, I can now say that I at least replaced them successfully, haha! If anyone ever needs it done, just PM me. I am IPC certified.

[dc42]

OK, I can see what happened. Prior to firmware 2.01, after allocating drivers to X, Y and Z all the remaining 9 drivers out of the possible 12 total used to default to extruders. This led to problems if you sent e.g. M350 E64 because drivers 10 and 11 are external drivers only, so 64x microstepping can't be applied to them by the firmware, resulting in an error message. So in firmware 2.01 I changed it so that only 7 drivers default to extruders. But I didn't reckon on anyone listing 9 extruder drives individually in M906!

I'll add an item in the upgrade notes.