Duet sometimes really slow? - I2C error or?

dc42

@gizmotronx5000 said in Duet sometimes really slow? - I2C error or?:

I was under the impression that when the problem occurs, no I2C commands get through at all. Do you have any evidence that any I2C commands are getting through? I2C commands are needed to set the fan RPM and to read the endstops. They are not needed to drive the motors or the heaters.

When I get this error, everything still works, including Duex5 endstops for homing. I haven't tried changing fan RPM during this error though.

That's very interesting, because if there was a total breakdown in I2C communication then the endstops should stop working. Next time it happens, please can you pause the print and then:

Send M122 to get the I2C error counts.
Toggle the DueX5 endstops and check whether the Duet can see the changes.
Send M122 to get the error counts again. [When you toggle a DueX5 endstop, it sends an interrupt to the Duet, which uses I2C to read the new states.]
Send some M106 commands to alter the state of fans/LEDs connected to the DueX5 fan outputs, and see if that works.
Send M122 and get the I2C error counts again.
Also send M122 a few more times, to see if when the machine is idle and no endstops are changing, the I2C error count goes on increasing.

GizmotronX5000

@dc42 Can do. It doesn't happen often, but the next time it occurs I'll be sure to reply here with that information.

TBS

Here is my short update:
I have updated to 2.02. The error has recurred. This time again at the start of printing.

dc42

@tbs said in Duet sometimes really slow? - I2C error or?:

Here is my short update:
I have updated to 2.02. The error has recurred. This time again at the start of printing.

Thanks, please see my post addressed to @GizmotronX5000 for useful (to me) tests you can do when this problem occurs.

deckingman

I've just had this slow down happen mid print - first time for me - it's only ever happened at first boot until now,. The only thing I have not yet done is DC42s suggestion of fitting resistors which was mentioned in another thread.

The manifestation is that the printer carries out one or 2 moves, then pauses for several seconds before continuing. M122 shows I2C nak errors 0, send timeouts 34803, receive timeouts 0, finish timeouts 34803. Also reported were:

"Warning: motor phase A may be disconnected reported by driver(s) 5 7" (Never seen that before).

I have the full M122 report if there is any other info that might be needed.

I've sent M106 Pn H-1 to a few of the fans connected to the Duex5 then tried a few M106 Pnn Snnn commands with no reaction on the fans themselves. So, I guess that means a total breakdown in I2C comms?

I've taken a video of what happens with the printer. I've cancelled the print but I'll leave it in this state for a few hours in case anyone wants me to do any tests or provide any more information.

dc42

@deckingman, thanks for keeping your machine on. Please see my response to @GizmotronX500 a few posts back and run the tests there.

deckingman

@dc42 said in Duet sometimes really slow? - I2C error or?:

@deckingman, thanks for keeping your machine on. Please see my response to @GizmotronX500 a few posts back and run the tests there.

Yes tried that. As I said, the fans are all configured as thermostatic so I tried sending M106 Pn H-1 first, then a few M106 Pnn Snnn commands and nothing happened as far as the fans were concerned.

I don't have any Duex end stops configured so I can't test them. That's not strictly true. I do have micro switches connected to E2 and E3 but no axes configured to use those switches....... Just tried toggling those switches but the status on machine properties page remains unchanged.

So I'm guess that I2C comms have completely stopped in my case. M122 now reports I2C send and finish timeouts as 1861859.

I'll leave the printer powered up for a while yet.

dc42

@deckingman, thanks. What were the results of #3 and #6? I'd like to know whether the I2C error count increases (a) continuously, (b) [if not continuously] when you try to change the speed of a DueX fan, (c) [if not continuously] when you toggle the state of a DueX endstop switch.

deckingman

@dc42 said in Duet sometimes really slow? - I2C error or?:

@deckingman, thanks. What were the results of #3 and #6? I'd like to know whether the I2C error count increases (a) continuously, (b) [if not continuously] when you try to change the speed of a DueX fan, (c) [if not continuously] when you toggle the state of a DueX endstop switch.

Damm. Sorry I didn't pay enough attention to what you were asking. The only info I have is that when I did the first M122 during the print, the send timeouts were 34803 and some time later, they were 1861859 which isn't much use to you. I turned the printer off before I went to bed last night and of course, the bloody thing is behaving itself again now.

I'll pay more attention next time it happens.

dc42

@deckingman, if they went up that much then I suspect they were continuously incrementing.

What I suspect happened is that the interrupt signal from the SX1509B on the DueX went high, indicating a change in an endstop input signal (real or imagined). That causes the Duet main processor to try to read the SX1509B input registers, which will also make the interrupt signal go away. But if I2C communication with the SX1509B has broken down, the interrupt won't go away and the Duet will keep trying to read it - which accounts for the slowdown.

deckingman

@dc42 Cheers. I'm just in the process of making up a plug with the resistors you mentioned elsewhere. I'll try that and I'll replace the ribbon cable between the Duet and Duex5 as well. Open to any other suggestions.

dc42

@deckingman said in Duet sometimes really slow? - I2C error or?:

@dc42 Cheers. I'm just in the process of making up a plug with the resistors you mentioned elsewhere. I'll try that and I'll replace the ribbon cable between the Duet and Duex5 as well. Open to any other suggestions.

Please try just the resistors. The ribbon cable is unlikely to be the problem, and I'd like to know whether the resistors fix it.

deckingman

@dc42 said in Duet sometimes really slow? - I2C error or?:

@deckingman said in Duet sometimes really slow? - I2C error or?:

@dc42 Cheers. I'm just in the process of making up a plug with the resistors you mentioned elsewhere. I'll try that and I'll replace the ribbon cable between the Duet and Duex5 as well. Open to any other suggestions.

Please try just the resistors. The ribbon cable is unlikey to be the problem, and I'd like to know whether the resistors fix it.

Sure thing - will do.

Martin1454

I will try the next time it occurs to plug on my logic analyzer and see what I can see.

deckingman

@dc42

Had to improvise a bit (as well as make allowances for my failing eye site and shaky hands ) but it'll do the job. The resistors are 1.8K so pretty much in the middle of the range you specified. We'll see what happens....

Interestingly enough, now that the machine is not in an error state, those switches that are connected to the Duex End stops do now respond, even though the axes they were originally assigned to are no longer configured - (I guess you knew that though).

deckingman

@dc42,

David. As I'm waiting for some filament to arrive, the machine is just sitting here idle. Is there anything I can do to attempt to provoke the misbehaviour? I was thinking along the lines of writing a little script that would generate a gcode file that simply toggled one of the duex5 fans on and off. Or may send multiple M260 commands. Is there any point in me doing that?

dc42

@deckingman said in Duet sometimes really slow? - I2C error or?:

@dc42,

David. As I'm waiting for some filament to arrive, the machine is just sitting here idle. Is there anything I can do to attempt to provoke the misbehaviour? I was thinking along the lines of writing a little script that would generate a gcode file that simply toggled one of the duex5 fans on and off. Or may send multiple M260 commands. Is there any point in me doing that?

Yes, you could try that.

GizmotronX5000

@dc42

M122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 2.02(RTOS) running on Duet Ethernet 1.02 or later + DueX5
Board ID: 08DDM-9FAM2-LW4SD-6J9F2-3S46R-K2XBW
Used output buffers: 1 of 20 (14 max)
=== RTOS ===
Static ram: 25524
Dynamic ram: 98292 of which 0 recycled
Exception stack ram used: 384
Never used ram: 6872
Tasks: NETWORK(ready,544) HEAT(blocked,1232) MAIN(running,3812) IDLE(ready,200)
Owned mutexes:
=== Platform ===
Last reset 00:16:58 ago, cause: power up
Last software reset at 2019-01-23 14:44, reason: User, spinning module GCodes, available RAM 6776 bytes (slot 3)
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0441f000 BFAR 0xe000ed38 SP 0xffffffff Task 0x4e49414d
Error status: 0
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest block write time: 15.9ms, max retries 0
MCU temperature: min 31.3, current 31.5, max 31.9
Supply voltage: min 12.0, current 12.3, max 12.4, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: standstill, SG min/max 37/510
Driver 1: standstill, SG min/max 70/544
Driver 2: standstill, SG min/max 63/552
Driver 3: standstill, SG min/max not available
Driver 4: standstill, SG min/max not available
Driver 5: standstill, SG min/max not available
Driver 6: standstill, SG min/max 0/462
Driver 7: standstill, SG min/max 0/0
Driver 8: standstill, SG min/max 0/0
Driver 9: standstill, SG min/max not available
Date/time: 2019-01-23 15:17:17
Cache data hit count 4294967295
Slowest loop: 59.17ms; fastest: 29.15ms
I2C nak errors 0, send timeouts 63402, receive timeouts 0, finishTimeouts 63402
=== Move ===
Hiccups: 0, StepErrors: 0, LaErrors: 0, FreeDm: 240, MinFreeDm: 237, MaxWait: 181633ms, Underruns: 0, 0
Scheduled moves: 21, completed moves: 21
Bed compensation in use: none
Bed probe heights: 0.000 0.000 0.000 0.000 0.000
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1
Heater 0 is on, I-accum = 0.0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Stack records: 1 allocated, 0 in use
Movement lock held by null
http is idle in state(s) 0
telnet is idle in state(s) 0
file is idle in state(s) 0
serial is idle in state(s) 0
aux is idle in state(s) 0
daemon is idle in state(s) 0
queue is idle in state(s) 0
autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 63.08ms; fastest: 0.03ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 1 of 8
Interface state 5, link 100Mbps full duplex
The Duex5 endstops do not work as I had previously thought. The endstops on the Duet work, but not the Duex5.
M122 right after pressing the endstops a few times (most lines removed):
M122
Date/time: 2019-01-23 15:18:05
Cache data hit count 4294967295
Slowest loop: 59.10ms; fastest: 0.08ms
I2C nak errors 0, send timeouts 4797, receive timeouts 0, finishTimeouts 4797
No fans are actually connected to the Duex5, so I can't test this. I can configure a dummy fan to the Duex5 for next time though. All fans on the Duet work fine.
M122 (after messing around with duet fans)
Date/time: 2019-01-23 15:22:28
Cache data hit count 4294967295
Slowest loop: 59.11ms; fastest: 0.08ms
I2C nak errors 0, send timeouts 17551, receive timeouts 0, finishTimeouts 17551
No changes. Just sending M122 every few seconds.

M122
Date/time: 2019-01-23 15:23:31
Cache data hit count 4294967295
Slowest loop: 59.11ms; fastest: 29.47ms
I2C nak errors 0, send timeouts 10480, receive timeouts 0, finishTimeouts 10480

M122
Date/time: 2019-01-23 15:24:00
Cache data hit count 4294967295
Slowest loop: 59.10ms; fastest: 29.46ms
I2C nak errors 0, send timeouts 4900, receive timeouts 0, finishTimeouts 4900

M122
Date/time: 2019-01-23 15:24:42
Cache data hit count 4294967295
Slowest loop: 59.10ms; fastest: 29.47ms
I2C nak errors 0, send timeouts 6885, receive timeouts 0, finishTimeouts 6885

I'm assuming the error count starts over each time I send M122. But the number is constantly changing. The lowest I saw was 195 from sending M122 back to back as fast as I could manually from the console. Looks like about 160-170 errors per second on average (164 seems to be the usual) regardless of what commands I send or which endstops I press.

Edit:
I let it idle in this state for about 15 minutes. The average was 166.2 errors per second. It's very consistent.

deckingman

I can add a bit to what @GizmotronX5000 has come with in that I do have fans connected to the Duex5 and that they did not respond to any commands while the machine was in error state (neither did the duex endstops).

dc42

Thanks, you have both confirmed that the DueX5 isn't responding to I2C traffic at all, and the Duet is repeatedly trying (and failing) to communicate with it.