DuetEth not responding to Pings until after reboot

BlueDust

My Duet2 board is randomly not talking on the network. As I have had random disconnects for a while, I don't know if this started with different firmwares, or the DWC 2 RCx... But not (I believe after a recent update), I have to reboot my Duet2 to get it talking again. So far the reboots appear to only occur if I am actually using the DWC, when it will stop responding.
I wish I thought to test with pings before I upgraded to DWC 2, to see how severe it was, but only thought of it now, and see it doesn't respond to Pings. When this is severe (doesn't respond to DWC for minutes), I have to reboot the Duet2 to get it working on the network again. Generally it DWC comes back within <30 seconds, or not at all.
The touch screen was working during this time, but didn't check the console to see if it had anything useful before cutting power - my bad....

I have DuetEth board so WiFi isn't in the equation.

Duet Web Control 2.0.0-RC3

Electronics: Duet Ethernet 1.02 or later + DueX5
Firmware: RepRapFirmware for Duet 2 WiFi/Ethernet 2.02(RTOS) (2018-12-24b1)

I have the eventlog recording data, but it didn't catch anything.... Is there a more detailed/verbose debug I can use that will not affect printing?

2019-01-05 17:23:45 Finished printing file 0:/gcodes/3D_Printer_test_fixed_stl_3rd_gen.gcode, print time was 3h 57m
---- This is where the network stopped responding -----
power up + 00:00:00 Event logging started

Thank you!

dc42

When you lose the connection:

If you can connect a PC via USB and send M122 from it, the diagnostic report may reveal something.
Have you tried sending M552 S-1 followed by M552 S1 from PanelDue, to see if it then reconnects? Those commands will reset and restart the network interface.

BlueDust

I am assuming you would like the Diagnostic report when the issue occurs, and not now, after I rebooted it. Next time it happens I will setup a laptop to connect to it and pull a diagnostic report via USB.

Is there a way to Run the Output of M122 to a file?

I didn't know how to reset the NIC. I was looking for a way to just check the current IP using the PanelDue, but didn't find that info until shortly after I already powered off/on the printer. I will try that next time.

Thanks @dc42!

BlueDust

@dc42 said in DuetEth not responding to Pings until after reboot:

M552

Happened again, but wasn't able to connect directly to the Duet2. I was able to get the NIC working again but not sure if it was because I used M552 or not because it didn't how up on the screen when I submitted it to confirm it worked (so it could have been a coincidence?)

In any case, I ran M122 right after I reconnected using a web browser.

M122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 2.02(RTOS) running on Duet Ethernet 1.02 or later + DueX5
Board ID: 08DGM-9T6BU-FG3S0-7JTDL-3SN6N-TS6VG
Used output buffers: 3 of 20 (20 max)
=== RTOS ===
Static ram: 25524
Dynamic ram: 98316 of which 0 recycled
Exception stack ram used: 624
Never used ram: 6608
Tasks: NETWORK(ready,528) HEAT(blocked,1232) MAIN(running,3796) IDLE(ready,200)
Owned mutexes:
=== Platform ===
Last reset 190:56:21 ago, cause: software
Last software reset at 2019-01-13 17:06, reason: User, spinning module GCodes, available RAM 6720 bytes (slot 0)
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0441f000 BFAR 0xe000ed38 SP 0xffffffff Task 0x4e49414d
Error status: 28
Free file entries: 9
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest block write time: 1.1ms, max retries 0
MCU temperature: min 29.6, current 30.0, max 30.3
Supply voltage: min 21.2, current 23.9, max 24.6, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: standstill, SG min/max not available
Driver 1: standstill, SG min/max not available
Driver 2: standstill, SG min/max not available
Driver 3: standstill, SG min/max not available
Driver 4: standstill, SG min/max not available
Driver 5: standstill, SG min/max not available
Driver 6: standstill, SG min/max not available
Driver 7: standstill, SG min/max not available
Driver 8: standstill, SG min/max not available
Driver 9: standstill, SG min/max not available
Date/time: 2019-01-21 16:03:15
Cache data hit count 4294967295
Slowest loop: 62.62ms; fastest: 0.08ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0
=== Move ===
Hiccups: 0, StepErrors: 0, LaErrors: 0, FreeDm: 240, MinFreeDm: 240, MaxWait: 0ms, Underruns: 0, 0
Scheduled moves: 10, completed moves: 10
Bed compensation in use: mesh
Bed probe heights: 0.000 0.000 0.000 0.000 0.000
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1
Heater 0 is on, I-accum = 0.3
Heater 1 is on, I-accum = 0.6
=== GCodes ===
Segments left: 0
Stack records: 2 allocated, 0 in use
Movement lock held by null
http is idle in state(s) 0
telnet is idle in state(s) 0
file is idle in state(s) 0
serial is idle in state(s) 0
aux is idle in state(s) 0
daemon is idle in state(s) 0
queue is idle in state(s) 0
autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 7.30ms; fastest: 0.00ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 1 of 8
Interface state 5, link 100Mbps full duplex

dc42

What would be more useful is a M122 report obtained using the USB interface, when you find that you can't ping the Duet.

Are you certain that the IP address of the Duet isn't changing?

BlueDust

I am actually connecting to the Duet2 by DNS name, and my DHCP server is set to always give the Duet2 NIC the same IP. If the 3d Printer was closer to my Desktop Computer I could connect to it all the time, but its about 10 feet away. My laptop actually is setup as a server (more like a small desktop). I will set it up next time this happens to connect directly to the Duet2.
Also, one thing to note, is that this "seems" to only happen after a long print (over 24 hours) completes, but not every time. I completed a 38 hour print, and then a 27 hour print, and when I went to start another, I saw I couldn't connect. I haven't completed too many prints lately that took more then 6 or 8 hours until these 2 prints, and as such, I am wondering if this is a trigger or coincidence.

Thanks

BlueDust

It happened again. Consoled in with USB right now, while I am not able ping the DuetEth. I will wait until tomorrow just for the chance you may see this and ask me to do something else before I reset the NIC or reboot DuetEth.
Thanks!

M122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 2.02(RTOS) running on Duet Ethernet 1.02 or later + DueX5
Board ID: 08DGM-9T6BU-FG3S0-7JTDL-3SN6N-TS6VG
Used output buffers: 1 of 20 (20 max)
=== RTOS ===
Static ram: 25524
Dynamic ram: 98316 of which 0 recycled
Exception stack ram used: 592
Never used ram: 6640
Tasks: NETWORK(ready,528) HEAT(blocked,1232) MAIN(running,3788) IDLE(ready,200)
Owned mutexes:
=== Platform ===
Last reset 291:01:51 ago, cause: software
Last software reset at 2019-02-03 18:09, reason: User, spinning module GCodes, available RAM 6632 bytes (slot 3)
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0441f000 BFAR 0xe000ed38 SP 0xffffffff Task 0x4e49414d
Error status: 24
Free file entries: 9
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest block write time: 0.0ms, max retries 0
MCU temperature: min 29.5, current 29.8, max 30.2
Supply voltage: min 24.1, current 24.1, max 24.2, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: standstill, SG min/max not available
Driver 1: standstill, SG min/max not available
Driver 2: standstill, SG min/max not available
Driver 3: standstill, SG min/max not available
Driver 4: standstill, SG min/max not available
Driver 5: standstill, SG min/max not available
Driver 6: standstill, SG min/max not available
Driver 7: standstill, SG min/max not available
Driver 8: standstill, SG min/max not available
Driver 9: standstill, SG min/max not available
Date/time: 2019-02-15 21:11:49
Cache data hit count 4294967295
Slowest loop: 1.19ms; fastest: 0.08ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0
=== Move ===
Hiccups: 0, StepErrors: 0, LaErrors: 0, FreeDm: 240, MinFreeDm: 240, MaxWait: 0ms, Underruns: 0, 0
Scheduled moves: 0, completed moves: 0
Bed compensation in use: mesh
Bed probe heights: -2.519 -2.671 -3.411 -3.669 -3.286
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1
Heater 0 is on, I-accum = 0.3
Heater 1 is on, I-accum = 0.7
=== GCodes ===
Segments left: 0
Stack records: 2 allocated, 0 in use
Movement lock held by null
http is idle in state(s) 0
telnet is idle in state(s) 0
file is idle in state(s) 0
serial is ready with "M122" in state(s) 0
aux is idle in state(s) 0
daemon is idle in state(s) 0
queue is idle in state(s) 0
autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 1.17ms; fastest: 0.02ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 0 of 8
Interface state 5, link 100Mbps full duplex

M552
Network is enabled, configured IP address: 0.0.0.0, actual IP address: 10.1.10.78

dc42

Please confirm, is 10.1.10.78 the usual IP address of the Duet, and does "ping 10.1.10.78" fail at present?
If you disconnect the Ethernet cable for several seconds and then reconnect it, are you then able to ping the Duet?
If you send M552 S-1 followed by M552 S1, are you then able to ping and connect to the Duet?

BlueDust

The Duet2 NIC shows two lights orange is solid, and green is blinking before I start the below steps.

C:\Users\XXXXX>nslookup duet3d
Server: XXXX.XXXXXXXX.XX
Address: 10.1.10.1

Name: dusty3d.XXXXXXXX.XX
Address: 10.1.10.78

C:\Users\XXXXX>ping 10.1.10.78

Pinging 10.1.10.78 with 32 bytes of data:
Reply from 10.1.10.61: Destination host unreachable.
Reply from 10.1.10.61: Destination host unreachable.
Reply from 10.1.10.61: Destination host unreachable.
Reply from 10.1.10.61: Destination host unreachable.

Ping statistics for 10.1.10.78:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

I Pulled the Ethernet Cable out and plugged it back in. The screen showed me the correct IP of the printer. I went to Ping and Pings still fail.
M552
Network is enabled, configured IP address: 0.0.0.0, actual IP address: 10.1.10.78
After I plugged back in the cable, the green light flashed much faster giving me hope, but quickly slowed down to its exact same pattern of 2 quick flashes and a short pause.

M552
Network is enabled, configured IP address: 0.0.0.0, actual IP address: 10.1.10.78
M552 S-1

M552 S1

M552
Network is enabled, configured IP address: 0.0.0.0, actual IP address: 10.1.10.78

I tried M552 S-1/S1 a few times, and still not able to Ping the Duet2.

I have stuff to print, but will wait a little while for your reply before I try rebooting it in case you want me to try something else first.

Thanks!

dc42

Thanks for doing those tests. I don't understand why you are unable to ping the Duet, unless for some reason your router has decided not to talk to it. Removing the cable will have reset the physical link. Sending ~~M997 S-1~~ M552 S-1 will have reset the network controller chip completely.

Please try restarting your router, and see whether pings work after that. If they still don't, reset the Duet and try again.

BlueDust

Just mentioning it to reduce potential confusion. I made a typo in my host name above (should be the same in both places) and not able to fix it. An edited post keeps getting marked as SPAM when I try to fix it.

just to mention it, M997 S-1 isn't what I used.

I have a managed switch, and can see all the ports on the switch and what MAC is on each port. The Duet2 is plugged into port 8 on my switch, and I do not see a MAC on that port. How can I find out the MAC of the Duet2's NIC? I just want to confirm the entire ARP table before I reboot he switch.
Thanks!

dc42

My bad, I meant M552 S-1.

dc42

M540 without parameters will print the MAC address.

Dougal1957

your ping is actually not working by the look of it you are pinging.78 but looking for replies from .61 that doesn't look right to me but I am no expert with that side of things.

BlueDust

@dougal1957
.61 is the IP of the computer I am using. In Windows 10 (maybe in Win7 and 8 too), it was changed to show the local IP when it fails and then says detestation host unreachable.... , but have no idea why Microsoft changed to say this. At a first quick glance, its just confusing. If you read it, it looks like its saying your local computer is telling you the destination device in not responding pings. But telling me the ping failed/dropped like it used to in Win XP would tell me the same thing... If Microsoft wanted to be fancy, they could simply say ICMP failed, and force people to Google it, and make them learn something.

ping 10.1.10.78

Pinging 10.1.10.78 with 32 bytes of data:
Reply from 10.1.10.61: Destination host unreachable.
Reply from 10.1.10.61: Destination host unreachable.
Reply from 10.1.10.61: Destination host unreachable.
Reply from 10.1.10.61: Destination host unreachable.

Ping statistics for 10.1.10.78:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

BlueDust

@dc42
I am testing this from 2 different computers.
I wasn't able to find the MAC on the switch or my computers arp tables.
I rebooted the switch and that didn't fix it.

I rebooted the DuetEth board, and everything is working again.

ping 10.1.10.78

Pinging 10.1.10.78 with 32 bytes of data:
Reply from 10.1.10.78: bytes=32 time=91ms TTL=128
Reply from 10.1.10.78: bytes=32 time<1ms TTL=128
Reply from 10.1.10.78: bytes=32 time<1ms TTL=128
Reply from 10.1.10.78: bytes=32 time<1ms TTL=128

Ping statistics for 10.1.10.78:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 91ms, Average = 22ms

I can also now see the MAC on the switch, and it also shows up again in the arp tables on both computers.

Unless you have a different idea/test, next time this happens, I will add the MAC as static to port 8 on my switch and the arp table with IP on my PC to see if it works again. I don't know much about this side of things, but if that works, I am guessing the NIC isn't sending packets out with its mac, or stopped sending packets all together (if it doesn't work). If it stopped sending packets, I am not sure why the green light was blinking at all, unless that is only for receiving packets (and could have been broadcasts from other devices on my network).

Thanks for your help.

Dougal1957

@bluedust Well you learn something every day lol I don't use Win10 so hadn't seen that change before how confusing! Have you tried setting a fixed IP on the Duet and I think I read somewhere that MDNS could be flakey.

In fact the following thread says mDNS is not supported on the Ethernet boards.
mDNS
although it is an old thread so things may have changed!

BlueDust

@dougal1957
I don't know about the Duet, but I use my DNS server to assign DNS names to IPs. And my DHCP server to assign IPs to MACs. Most things on my network are DHCP, but set them to always receive the same IPs.
Even after I do the above, I still test by IPs and MACs... etc... Google OSI Model...Most of the time checking layers 1 to 3, and somethings 4 (IPs, routing) in that order will help resolve most networking problems.

Dougal1957

@bluedust As i say I am no expert in these matters I use a duet Wifi (i do have a ethernet for an in progress build) but I always give them static IP's and connect using they IP Address and have never come across this issue. But I do wish you luck in tracking it down).

I did have an issue a year or so ago when the Wifi connection suddenly stopped working and that was traced to the router not playing nicely at all used a cheap Tp-Link travel router as a AP Repeater and problem resolved since then I have replaced my main router and I am back to normal.

The router that was causing the issues was a nether one by the way

BlueDust

@dougal1957
In this case, no matter what, static on the Duet or not, the Duet MAC has to be on the switch port at all times. If not, the Switch will not know where/what port to send the packets. And your PC keeps an ARP table with every IP address and MAC of local devices, or MAC of Gateway for all devices that need to be reach via gateway/router. I maybe speaking a foreign language now.... But if the MAC isn't on the switch (and the switch is working properly), something maybe wrong with the NIC on the Duet. Funny thing is, it appears to stop working every 20 to 25 days and maybe more often during extended time prints over/around 30 hours. (I also do not shut my 3d Printer off)