Duet 2 Ethernet WC 3.3.0 crashes, have to reset to reconnect

[airscapes]

@phaedrux What is the name of the Terminal program for usb connection. I have it someplace as I used it when I was setting this up but don't recall the name.
Output when working
I did not try a new network cable but will also eliminate that next time it fails.

1/21/2022, 2:45:24 PM	M122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.3 (2021-06-15 21:44:54) running on Duet Ethernet 1.02 or later
Board ID: 0JD0M-9P6B2-NJ4S8-6J1F2-3S46M-KU66M
Used output buffers: 1 of 24 (11 max)
=== RTOS ===
Static ram: 23876
Dynamic ram: 70772 of which 0 recycled
Never used RAM 19768, free system stack 188 words
Tasks: NETWORK(ready,27.9%,233) HEAT(delaying,0.0%,336) Move(notifyWait,0.1%,319) MAIN(running,70.3%,441) IDLE(ready,1.7%,29), total 100.0%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 00:00:52 ago, cause: software
Last software reset at 2022-01-21 14:44, reason: User, GCodes spinning, available RAM 16880, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
Step timer max interval 0
MCU temperature: min 38.1, current 38.9, max 39.2
Supply voltage: min 24.2, current 24.3, max 24.3, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Driver 0: position 0, standstill, SG min/max not available
Driver 1: position 0, standstill, SG min/max not available
Driver 2: position 0, standstill, SG min/max not available
Driver 3: position 0, standstill, SG min/max not available
Driver 4: position 0, standstill, SG min/max not available
Driver 5: position 0
Driver 6: position 0
Driver 7: position 0
Driver 8: position 0
Driver 9: position 0
Driver 10: position 0
Driver 11: position 0
Date/time: 2022-01-21 14:45:23
Cache data hit count 1979751074
Slowest loop: 5.32ms; fastest: 0.17ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 3.3ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is ready with "M122 " in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 83.42ms; fastest: 0.02ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
Interface state active, link 100Mbps full duplex

[Phaedrux]

YAT for yet another terminal

Also try pinging it if it stops responding so we can see if the IP address is still responding or not.

Another option may be to replace the SD card.

How are you using octoprint?

[airscapes]

@phaedrux said in Duet 2 Ethernet WC 3.3.0 crashes, have to reset to reconnect:

YAT for yet another terminal

Also try pinging it if it stops responding so we can see if the IP address is still responding or not.

Another option may be to replace the SD card.

How are you using octoprint?

I was using Octoprint via usb which did continue to work when the WC was not responding. Thinking it was octoprint, I stopped using it. I can try to connect via Octoprint next time the WC fails and if that does not work go straight from a laptop with YAT. Also after last print finished move the bed as I would normally do if need be and had no failure.

[airscapes]

@airscapes 2 print jobs in, make a change restarted the board and would not connect via network. Plugged in the pi and Octoprint connected on the first try and this is the output of M122
Sorry forgot to ping it, but did replace the cable before this as when I first connected it said it failed with network error and retied..

Send: M122
Recv: === Diagnostics ===
Recv: RepRapFirmware for Duet 2 WiFi/Ethernet version 3.3 (2021-06-15 21:44:54) running on Duet Ethernet 1.02 or later
Recv: Board ID: 0JD0M-9P6B2-NJ4S8-6J1F2-3S46M-KU66M
Recv: Used output buffers: 1 of 24 (21 max)
Recv: === RTOS ===
Recv: Static ram: 23876
Recv: Dynamic ram: 70804 of which 0 recycled
Recv: Never used RAM 16880, free system stack 134 words
Recv: Tasks: NETWORK(ready,29.1%,233) HEAT(delaying,0.0%,330) Move(notifyWait,0.1%,313) MAIN(running,70.7%,441) IDLE(ready,0.2%,29), total 100.0%
Recv: Owned mutexes: USB(MAIN)
Recv: === Platform ===
Recv: Last reset 00:09:13 ago, cause: software
Recv: Last software reset time unknown, reason: User, GCodes spinning, available RAM 19768, slot 0
Recv: Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Recv: Error status: 0x00
Recv: Step timer max interval 0
Recv: MCU temperature: min 45.5, current 46.3, max 47.5
Recv: Supply voltage: min 23.7, current 23.9, max 24.3, under voltage events: 0, over voltage events: 0, power good: yes
Recv: Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/30/30, gc cycles 0
Recv: Driver 0: position 0, standstill, SG min/max 12/348
Recv: Driver 1: position 10586, standstill, SG min/max 0/333
Recv: Driver 2: position 14109, standstill, SG min/max 0/108
Recv: Driver 3: position 0, standstill, SG min/max 0/212
Recv: Driver 4: position 0, standstill, SG min/max not available
Recv: Driver 5: position 0
Recv: Driver 6: position 0
Recv: Driver 7: position 0
Recv: Driver 8: position 0
Recv: Driver 9: position 0
Recv: Driver 10: position 0
Recv: Driver 11: position 0
Recv: Date/time: 2022-01-22 18:09:50
Recv: Cache data hit count 4294967295
Recv: Slowest loop: 47.47ms; fastest: 0.09ms
Recv: I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
Recv: === Storage ===
Recv: Free file entries: 10
Recv: SD card 0 detected, interface speed: 20.0MBytes/sec
Recv: SD card longest read time 3.9ms, write time 0.0ms, max retries 0
Recv: === Move ===
Recv: DMs created 83, maxWait 84260ms, bed compensation in use: none, comp offset 0.000
Recv: === MainDDARing ===
Recv: Scheduled moves 129, completed moves 129, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
Recv: === AuxDDARing ===
Recv: Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
Recv: === Heat ===
Recv: Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
Recv: Heater 0 is on, I-accum = 0.6
Recv: Heater 1 is on, I-accum = 0.4
Recv: === GCodes ===
Recv: Segments left: 0
Recv: Movement lock held by null
Recv: HTTP is idle in state(s) 0
Recv: Telnet is idle in state(s) 0
Recv: File is idle in state(s) 0
Recv: USB is ready with "M122" in state(s) 0
Recv: Aux is idle in state(s) 0
Recv: Trigger is idle in state(s) 0
Recv: Queue is idle in state(s) 0
Recv: LCD is idle in state(s) 0
Recv: Daemon is idle in state(s) 0
Recv: Autopause is idle in state(s) 0
Recv: Code queue is empty.
Recv: === Network ===
Recv: Slowest loop: 48.68ms; fastest: 0.02ms
Recv: Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
Recv: HTTP sessions: 0 of 8
Recv: Interface state active, link 100Mbps full duplex
Recv: ok
[...]

[airscapes]

@airscapes Opps! The thermostatic controlled fan was not running, seems like a connection issue or a cracked board. Had the X endstop not working the other week, reseated the connector and that has not happened but the connector was not loose and it was not a connector I replaced.. Will update with ping status if this happens again.

[airscapes]

@airscapes pint job done, did not look at the dang gui first, went and pulled the bed out on the Y axis to remove the print. Went to pc and was disconnected. Could not ping
oot@ [/root] # ping 192.168.1.245
PING 192.168.1.245 (192.168.1.245) 56(84) bytes of data.
From 192.168.1.101 icmp_seq=1 Destination Host Unreachable
From 192.168.1.101 icmp_seq=2 Destination Host Unreachable
From 192.168.1.101 icmp_seq=3 Destination Host Unreachable
From 192.168.1.101 icmp_seq=4 Destination Host Unreachable
From 192.168.1.101 icmp_seq=5 Destination Host Unreachable
From 192.168.1.101 icmp_seq=6 Destination Host Unreachable
From 192.168.1.101 icmp_seq=7 Destination Host Unreachable
From 192.168.1.101 icmp_seq=8 Destination Host Unreachable

Still have flashing green light and orange link light.
Had to push da reset button.. again.

[airscapes]

@Phaedrux
Just found one thing that "Might" be causing this issue but not positive. I had the network set to DHCP and I also had forgotten to add a Fixed entry for the host in the router. I am doubtful this is it, because I would not think the gui would return me a message that a network error had occurred if it just get a new IP and did not look when it was offline. I now have the Duet set to a fixed IP and the router set so that mac always get that same IP or should I say, so it does not give that IP to anyone else.
Is there a way to add additional logging and is there a CLI to the OS from the USB connections to see if the Web interface is still running should I get another failure??

BTW I did verify that disconnecting the network and reconnecting works just fine.. i.e. remove the cable wait a few moment, plug it back in, gui still works as it should. So other than an IP address no longer being on the interface or the IP I was expecting.. I don't think this is related to anything external in the network.
Thanks!

[Phaedrux]

If you get a disconnection again use the USB terminal again as you did when you got the M122 report but this time send M552 and see what IP address it has.

[airscapes]

@phaedrux Yep, will do! Have not used the printer but have several things to print today. Seems like my configuation failures would very likely cause the symptoms.. The nic was not dead, link and trafic lights still working so no IP or different IP makes perfect sense.

[airscapes]

@Phaedrux So much for that.. just lost connection after canceling a print job and homing Z. Ran the 552 and 122 again, also started the job from the usb connection without resetting the board if there is something else I can do to try and figure this out..

Recv: Network is enabled, configured IP address: 192.168.1.245, actual IP address: 192.168.1.245
Recv: ok
From pc to printer
Pinging 192.168.1.245 with 32 bytes of data:
Reply from 192.168.1.161: Destination host unreachable.

Ping statistics for 192.168.1.245:
Packets: Sent = 1, Received = 1, Lost = 0 (0% loss),
Control-C

Here is m122 again
Send: M122
Recv: === Diagnostics ===
Recv: RepRapFirmware for Duet 2 WiFi/Ethernet version 3.3 (2021-06-15 21:44:54) running on Duet Ethernet 1.02 or later
Recv: Board ID: 0JD0M-9P6B2-NJ4S8-6J1F2-3S46M-KU66M
Recv: Used output buffers: 1 of 24 (22 max)
Recv: === RTOS ===
Recv: Static ram: 23876
Recv: Dynamic ram: 70804 of which 320 recycled
Recv: Never used RAM 16560, free system stack 112 words
Recv: Tasks: NETWORK(ready,762.2%,225) HEAT(delaying,0.9%,330) Move(notifyWait,3.5%,313) MAIN(running,302.5%,420) IDLE(ready,0.2%,29), total 1069.2%
Recv: Owned mutexes: USB(MAIN)
Recv: === Platform ===
Recv: Last reset 03:56:56 ago, cause: power up
Recv: Last software reset at 2022-01-23 12:09, reason: User, GCodes spinning, available RAM 16560, slot 2
Recv: Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Recv: Error status: 0x00
Recv: Step timer max interval 0
Recv: MCU temperature: min 20.4, current 54.9, max 59.3
Recv: Supply voltage: min 23.7, current 23.8, max 24.4, under voltage events: 0, over voltage events: 0, power good: yes
Recv: Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/120/120, gc cycles 0
Recv: Driver 0: position 0, standstill, SG min/max 0/355
Recv: Driver 1: position 178, standstill, SG min/max 0/360
Recv: Driver 2: position 213533, standstill, SG min/max 0/1023
Recv: Driver 3: position 0, standstill, SG min/max 0/1023
Recv: Driver 4: position 0, standstill, SG min/max not available
Recv: Driver 5: position 0
Recv: Driver 6: position 0
Recv: Driver 7: position 0
Recv: Driver 8: position 0
Recv: Driver 9: position 0
Recv: Driver 10: position 0
Recv: Driver 11: position 0
Recv: Date/time: 2022-01-25 11:36:51
Recv: Cache data hit count 4294967295
Recv: Slowest loop: 90.23ms; fastest: 0.09ms
Recv: I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
Recv: === Storage ===
Recv: Free file entries: 10
Recv: SD card 0 detected, interface speed: 20.0MBytes/sec
Recv: SD card longest read time 4.0ms, write time 40.0ms, max retries 0
Recv: === Move ===
Recv: DMs created 83, maxWait 9607499ms, bed compensation in use: none, comp offset 0.000
Recv: === MainDDARing ===
Recv: Scheduled moves 1, completed moves 1, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 2], CDDA state -1
Recv: === AuxDDARing ===
Recv: Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
Recv: === Heat ===
Recv: Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
Recv: Heater 0 is on, I-accum = 0.8
Recv: Heater 1 is on, I-accum = 0.3
Recv: === GCodes ===
Recv: Segments left: 0
Recv: Movement lock held by null
Recv: HTTP is idle in state(s) 0
Recv: Telnet is idle in state(s) 0
Recv: File is idle in state(s) 0
Recv: USB is ready with "M122" in state(s) 0
Recv: Aux is idle in state(s) 0
Recv: Trigger is idle in state(s) 0
Recv: Queue is idle in state(s) 0
Recv: LCD is idle in state(s) 0
Recv: Daemon is idle in state(s) 0
Recv: Autopause is idle in state(s) 0
Recv: Code queue is empty.
Recv: === Network ===
Recv: Slowest loop: 193.31ms; fastest: 0.02ms
Recv: Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
Recv: HTTP sessions: 0 of 8
Recv: Interface state active, link 100Mbps full duplex
Recv: ok

[Phaedrux]

Thanks.

When and where did you purchase your Duet?

[airscapes]

@phaedrux MatterHackers.com Nov 20 2021

[Phaedrux]

Please send an email to warranty@duet3d.com and CC your reseller. Include a link to this forum thread and the details of your original purchase. You'll receive a reply with a form to fill out.

[airscapes]

@phaedrux Ok, thanks! Will post how it goes. Email sent

[airscapes]

@airscapes said in Duet 2 Ethernet WC 3.3.0 crashes, have to reset to reconnect:

@phaedrux Ok, thanks! Will post how it goes. Email sent

The return has been started, still waiting to hear details from Matterhackers about what to do next but they did email me and said I would be contacted.

I have been using the printer today with no issues until I smacked a print with the spatula to remove it, when I returned to the desk the network connection was gone. Guess the vibrations is all it took as I did not move any cables or touch the board enclosure.

[airscapes]

@Phaedrux Return initiated and shipping shortly!!
This has been one of the best trouble shooting and return process I have ever had. Top notch and thanks you very much!

[airscapes]

@Phaedrux
Wow this is not good.. Replacement board installed on the weekend. Did my first print today. PETG so no heat in the enclosure, 35 minute print job, finished, when into the other room pulled the bed out manually (been doing that since I got it as the manufacture said it is fine to do. removed the print, returned to desk and I am now disconnected. Checked m552 still had an IP, network module still flashing lights. connected to usb with octoprint, head bead still on as it should have been as I did not turn it off at the end of the job. Hit reset button, and after boot was back on the network. So what is the next step. Guess I will email Matterhackers and let them know still have the same problem?? Is there any more trouble shooting we can do. can not recreate problem on demand.

[Phaedrux]

It's pretty unlikely that the new board has an issue with ethernet as well. I think it's more likely that the issue is on the network hardware side. The Duet appears to still have an address and seeing activity.

What network hardware is between your PC and the Duet? Do you have any means of testing something alternative?

[airscapes]

@phaedrux I have tested it the with the first card. Cable was sent directly from the failed duet to the switch in a different port. Remember it can be correct by pushing the reset button on the board, so how would that fix my network? I would like it to be a bad cable, but it is not, that was swapped out with the old board and no change. so far I have not had a second failure. I used it all day yesterday and today. Without being able to reproduce the problem on demand it is going to be very difficult to track down.

Are there any other logs that can be configured on the duet?

I had thought it was maybe triggered by heat, but when it happened with the new board, I was printing PETG. Today all Day had the enclosure running at 100F printing ABS with no failures.. I am willing to try anything.
Questions; If next time it fails, I remove the network cable from the car and plug in a different cable, running it directly to the switch and into an unused port, would you assume that eliminates my network?

Currently the network path is; switch to jack, cable through wall to jack, Jack to 8 port switch which has the pi (octoprint) and the duet plugged into it. Booth switches are Dlink Unmanaged switches, a 24 port and 8port, DHCP is from the WAN router which is a Fios Gateway. I have many devices on the network with no issues on any of them.

Is there a way to ping the gateway from the USB connection using octoprint next time the failure occurs?

Also not sure on how the serial numbers work, but the first one was w09221 and the replacement is w09219. I am guessing they were built with all the same lot of parts and could wave to each other on the production line.

I just want to get to the root cause and correct it, be it local network or bad solder joint between the module and board etc.

After having 6 hard drives go bad in new IBM P950 enterprise servers servers within a week, I know even new enterprise class hardware can be manufactured with defects, and of course even technical users do make dumb mistakes, so please any other ideas on ways to track this next time it happens will be much appreciated!

Just thought of something the maybe I am doing wrong? My assumption was I can access the WC from multiple remote locations at the same time. For instance, I login from the browser on the Pi that is plugged into the same 8 port switch were the Duet it connected and sits next to the printer to load filament. I also connect from my PC in the other room. When the failure happens both browsers are disconnected. Now this is not what is causing my issue is it?? I mean I do this with the pi all the time when I was using octoprint so I assumed it is just a web interface and would not matter how many connections I make?

[airscapes]

@Phaedrux OK in failed state after first print of the day finished. I ran a known good network cable from an open port of the 24 port switch directly to the duet.
The duet shows link, and shows activity light, switch end show link but no activity. Moved same cable from duet to pi, get link and activity at switch.

Using the USB connection to Octoprint I disabled the networking on the Duet and then Enabled it setting the IP as it was prior. Refreshed the browser and after a few seconds got connected to WC!

So what does this tell you? Seems like software/frimware process died and needed to be restarted? Again I will ask is there a way to see a process table from USB or from other means I may have access to?