Best posts made by ProteanReverie

@droftarts said in Duet 2 WiFi network slowdowns and board restarts:

@ProteanReverie It's a long shot, but... it does seem possible that Duets are a bit susceptible to malformed Ethernet packets coming from the network. Some other users have reported similar issues, and the Hardfaults seem to be related to networking, and occasionally oddly setup browsers. @dc42 has often asked what extra languages are installed in these cases, as it seems foreign characters or character sets can trigger the Duet crash.

What made me think of this is when you say you have '3 blink cameras' on your network. Security cameras are a favourite target for hackers, that can then be used to snoop on the local network (strange packets) as well as launch DoS attacks. The Blink camera does seem susceptible: https://www.electronicshub.org/can-blink-cameras-be-hacked/

Running a WireShark scan of the network for a while, and capturing any strange packets, particularly at the point when the Duet crashes, may be useful. It would be great to track this one down!

Ian

No extra languages are set up on any of the PCs or tablets, or any other devices to my knowledge.

I am aware to some extent the security issues present with networked cameras and will keep an eye on that. Nothing that I have seen so far has indicated to me that they have been compromised at any point, but those kinds of things not being able to be noticed is usually a key point so can't say for sure.

I have got wireshark set up, and next time I see issues I will run a capture focused on the IP of one of the machines to share here. I plan to take it through a file upload (which will likely fail) as well as some network interface resets which hopefully cause the restart while doing that capture. If there are other actions I should take during that capture, please let me know.

@dc42 said in Duet 2 WiFi network slowdowns and board restarts:

@ProteanReverie it might be worth setting up a macro that disables wifi using M552 S-1, delays for a few seconds using G4, then restarts wifi using M552 S1. Resetting the connection in this way may be simpler than resetting the Duet or the router.

I actually wrote that exact macro a couple of years ago when having other issues which were later resolved. I still use it from time to time when I see this issue crop up, though often times when the issue is present the interface will be so inaccessible that I cannot activate the macro so it is easier to connect via serial to send commands or just power cycle the machine. Thank you for the recommendation.

Another occurance over the weekend.

M122

=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.1 (2024-04-19 14:41:25) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: 65LYX-8Q6KL-K65J0-409N0-NP02Z-ZFK3T
Used output buffers: 14 of 40 (18 max)
=== RTOS ===
Static ram: 103232
Dynamic ram: 119916 of which 24 recycled
Never used RAM 15388, free system stack 150 words
Tasks: NETWORK(2,nWait 7,580.1%,218) HEAT(3,nWait 6,0.4%,372) Move(4,nWait 6,0.1%,355) CanReceiv(6,nWait 1,0.6%,808) CanSender(5,nWait 7,0.0%,336) CanClock(7,delaying,0.3%,348) TMC(4,nWait 6,34.6%,102) MAIN(1,running,35.1%,677) IDLE(0,ready,0.3%,30) AIN(4,delaying,36.4%,260), total 688.1%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 11:24:14 ago, cause: software
Last software reset at 2024-06-23 21:22, reason: HardFault bfarValid precise, Gcodes spinning, available RAM 15364, slot 0
Software reset code 0x0063 HFSR 0x40000000 CFSR 0x00008200 ICSR 0x00000803 BFAR 0x00000004 SP 0x20011ff0 Task NETW Freestk 483 ok
Stack: 00000160 00000002 200014e8 0000015f ffffffff 000a07ef 0002ff8e 810f0000 0002ff85 00000000 00000000 00000000 2002fdcc 00000800 20035308 2002c5f8 200187fc 2002c41e 200187fc 2001e9f8 0003011f 00000000 00000000 00000000 200120a0 00000014 ffffffff
Error status: 0x00
MCU revision 3, ADC conversions started 41054418, completed 41054418, timed out 0, errs 0
MCU temperature: min 38.2, current 39.6, max 41.0
Supply voltage: min 24.1, current 24.1, max 24.2, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/12/12, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 1, ifcnt 43, reads 63568, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 1, ifcnt 43, reads 63568, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 43, reads 63568, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 43, reads 63568, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 1, ifcnt 43, reads 63568, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2024-06-24 08:46:40
Cache data hit count 4294967295
Slowest loop: 7.67ms; fastest: 0.16ms
=== Storage ===
Free file entries: 19
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 3.2ms, write time 3.8ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, min interval 0, bad calcs 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000000
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 205271, received 328436, lost 0, errs 1, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 390, free buffers 26 (min 26), ts 205271/205270/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 7.95ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(1) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1.0
MAC address e8:68:e7:e1:4e:d0
Module reset reason: Power up, Vcc 3.42, flash size 2097152, free heap 29568
WiFi IP address 10.1.10.46
Signal strength -59dBm, channel 11, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 3 0 0 0 0 0 0 0

Relevant log file section

2024-06-21 20:23:42 [warn] HTTP client 10.1.10.9 login succeeded (session key 2318516549)
2024-06-22 12:46:56 [warn] HTTP client 10.1.10.9 login succeeded (session key 2498040479)
2024-06-22 12:51:11 [warn] HTTP client 10.1.10.9 login succeeded (session key 2882910232)
2024-06-23 12:58:37 [warn] HTTP client 10.1.10.9 login succeeded (session key 2274061159)
2024-06-23 13:14:38 [warn] HTTP client 10.1.10.9 login succeeded (session key 4219374896)
2024-06-23 13:14:58 [warn] HTTP client 10.1.10.9 login succeeded (session key 1587684493)
2024-06-23 13:15:18 [warn] HTTP client 10.1.10.9 login succeeded (session key 871240276)
2024-06-23 18:32:45 [warn] HTTP client 10.1.10.9 login succeeded (session key 640519980)
2024-06-23 20:18:14 [warn] HTTP client 10.1.10.9 login succeeded (session key 1301043269)
2024-06-23 20:19:52 [warn] HTTP client 10.1.10.9 login succeeded (session key 3916440988)
2024-06-23 20:20:20 [warn] HTTP client 10.1.10.9 login succeeded (session key 3232781641)
2024-06-23 20:20:40 [warn] HTTP client 10.1.10.9 login succeeded (session key 1725357550)
2024-06-23 20:20:56 [warn] HTTP client 10.1.10.9 login succeeded (session key 2497689280)
power up + 00:00:00 [info] Event logging started at level debug
power up + 00:00:00 [info] Running: Duet 3 Mini5plus WiFi: 3.5.1 (2024-04-19 14:41:25)
power up + 00:00:00 [debug] Done!
power up + 00:00:00 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:00 [warn] WiFi module started
power up + 00:00:08 [warn] WiFi module is connected to access point SD3D-24, IP address 10.1.10.46
power up + 00:00:08 [warn] HTTP client 10.1.10.9 login succeeded (session key 3424165787)
2024-06-23 21:22:35 [warn] Date and time set at power up + 00:00:08
2024-06-23 21:54:45 [warn] HTTP client 10.1.10.9 login succeeded (session key 835669197)
2024-06-23 21:54:54 [warn] HTTP client 10.1.10.9 login succeeded (session key 1712060142)
2024-06-23 21:55:04 [warn] HTTP client 10.1.10.9 login succeeded (session key 2620441938)
2024-06-23 21:55:12 [warn] HTTP client 10.1.10.9 login succeeded (session key 2547895376)
2024-06-23 21:55:35 [warn] HTTP client 10.1.10.9 login succeeded (session key 519733317)
2024-06-24 00:17:19 [warn] HTTP client 10.1.10.9 login succeeded (session key 1091016330)
2024-06-24 00:17:48 [warn] HTTP client 10.1.10.9 login succeeded (session key 598095062)
2024-06-24 00:18:08 [warn] HTTP client 10.1.10.9 login succeeded (session key 994352261)
2024-06-24 00:57:15 [warn] HTTP client 10.1.10.9 login succeeded (session key 852304322)
2024-06-24 04:59:31 [warn] HTTP client 10.1.10.9 login succeeded (session key 713145612)
2024-06-24 08:46:40 [debug] === Diagnostics ===

No occurrences through the week last week, but did have another over the weekend.

M122

=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.1 (2024-04-19 14:41:25) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: 65LYX-8Q6KL-K65J0-409N0-NP02Z-ZFK3T
Used output buffers: 1 of 40 (18 max)
=== RTOS ===
Static ram: 103232
Dynamic ram: 119940 of which 24 recycled
Never used RAM 15364, free system stack 156 words
Tasks: NETWORK(1,ready,419.7%,218) HEAT(3,nWait 6,0.7%,356) Move(4,nWait 6,0.2%,341) CanReceiv(6,nWait 1,1.0%,808) CanSender(5,nWait 7,0.0%,336) CanClock(7,delaying,0.6%,348) TMC(4,nWait 6,57.1%,102) MAIN(1,running,87.1%,677) IDLE(0,ready,0.5%,30) AIN(4,delaying,60.0%,260), total 627.0%
Owned mutexes:
=== Platform ===
Last reset 20:58:29 ago, cause: software
Last software reset at 2024-06-30 11:35, reason: HardFault bfarValid precise, Platform spinning, available RAM 15364, slot 1
Software reset code 0x0060 HFSR 0x40000000 CFSR 0x00008200 ICSR 0x00000803 BFAR 0x00000004 SP 0x20011ff0 Task NETW Freestk 483 ok
Stack: 00000160 00000002 200014e8 0000015f ffffffff 000a07ef 0002ff8e 810f0000 0002ff85 00000000 00000000 00000000 200305e4 00000800 2002c5f8 2002c5f8 00000001 2002c41e 200187fc 2001e9f8 0003011f 00000000 00000000 00000000 200120a0 00000014 20468121
Error status: 0x00
MCU revision 3, ADC conversions started 75510090, completed 75510090, timed out 0, errs 0
MCU temperature: min 38.7, current 39.8, max 41.4
Supply voltage: min 24.0, current 24.1, max 24.2, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/12/12, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 1, ifcnt 54, reads 42014, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 1, ifcnt 54, reads 42014, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 54, reads 42014, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 54, reads 42014, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 1, ifcnt 54, reads 42015, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2024-07-01 08:34:24
Cache data hit count 4294967295
Slowest loop: 7.01ms; fastest: 0.16ms
=== Storage ===
Free file entries: 19
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 0.7ms, write time 2.6ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, min interval 0, bad calcs 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000000
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 377549, received 604082, lost 0, errs 1, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 297, free buffers 26 (min 26), ts 377549/377548/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 7.38ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1.0
MAC address e8:68:e7:e1:4e:d0
Module reset reason: Power up, Vcc 3.39, flash size 2097152, free heap 42784
WiFi IP address 10.1.10.46
Signal strength -60dBm, channel 11, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

Log.txt

2024-06-26 13:58:26 [warn] HTTP client 10.1.10.9 login succeeded (session key 3446445976)
2024-06-27 07:44:43 [warn] HTTP client 10.1.10.9 login succeeded (session key 2598975707)
2024-06-27 11:34:42 [warn] HTTP client 10.1.10.9 login succeeded (session key 3419397635)
2024-06-27 12:00:55 [warn] HTTP client 10.1.10.9 login succeeded (session key 3802106080)
2024-06-27 14:17:11 [warn] HTTP client 10.1.10.9 login succeeded (session key 3855690080)
power up + 00:00:00 [info] Event logging started at level debug
power up + 00:00:00 [info] Running: Duet 3 Mini5plus WiFi: 3.5.1 (2024-04-19 14:41:25)
power up + 00:00:00 [debug] Done!
power up + 00:00:00 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:00 [warn] WiFi module started
power up + 00:00:07 [warn] WiFi module is connected to access point SD3D-24, IP address 10.1.10.46
power up + 00:00:08 [warn] HTTP client 10.1.10.9 login succeeded (session key 526990551)
2024-06-30 11:36:03 [warn] Date and time set at power up + 00:00:08
2024-06-30 13:26:46 [warn] HTTP client 10.1.10.9 login succeeded (session key 2188381128)
2024-06-30 13:26:51 [warn] HTTP client 10.1.10.9 login succeeded (session key 1375067276)
2024-06-30 13:26:54 [warn] HTTP client 10.1.10.9 login succeeded (session key 587158328)
2024-06-30 13:27:19 [warn] HTTP client 10.1.10.9 login succeeded (session key 3511153781)
2024-06-30 14:25:24 [warn] HTTP client 10.1.10.9 login succeeded (session key 2762896906)
2024-07-01 01:18:55 [warn] HTTP client 10.1.10.9 login succeeded (session key 96132253)
2024-07-01 01:20:58 [warn] HTTP client 10.1.10.9 login succeeded (session key 1951105233)
2024-07-01 08:34:24 [debug] === Diagnostics ===

Wanted to report in that following the changing of the SD cards I mentioned around 7/30/24 my benchtop board has not seen a single restart and currently has over 1000 hours of uptime. The other board which is running in a machine has not seen any issues since my last reported crash in this thread on 8/1/24, which was a crash that happened after the SD cards were changed. That machine has seen various restarts and downtime since then, due to other factors.