Duet2 wifi stuck repeating Wifi Error.

zBeeble

@rechrtb said in Duet2 wifi stuck repeating Wifi Error.:

@zBeeble The instructions for updating WiFi firmware only is at: https://docs.duet3d.com/User_manual/RepRapFirmware/Updating_firmware#updating-wifi-firmware

You might want to clarify there a bit. There's nothing wrong with that recomendation, but you should go a step further. For instance, minicom does everything required and runs in a terminal. tmux or screen can manage long running terminals when the other computer might reboot or suffer some other calamity (like being a laptop).

If you add minicom, you might note that from stock config, turning on 'add carriage return' and turning on 'line wrap' and turning on 'local echo' are really good settings.

zBeeble

OK. The crazy that needed to finish is done. The new wifi binary is loaded and I have a minicom terminal in a tmux that should live just fine when the laptop leaves for the nation's capital at the end of the week.

I think a short hiccup with the wifi occurred. I see some disconnects in the log... but this was before the log-log.

I'm kinda relying on you guys that this setup is correct for logging. I see a print stopping, but not starting on the console and I don't see, say, if I run m111 on the web console. I do see a result to running an m command (like m111) on the serial console, tho... so... fingers crossed.

zBeeble

Hey... is it supposed to say this:

m111
Debugging enabled for modules:
Debugging disabled for modules: Platform(0) Network(1) Webserver(2) Gcodes(3) Move(4) Heat(5) Kinematics(6) InputShaping(7) unused(8) PrintMonitor(9) Storage(10) PortControl(11) DuetExpansion(12) FilamentSensors(13) WiFi(14) Display(15) SbcInterface(16) CAN(17) Expansion(18)

... debugging not on wifi?

zBeeble

The bottom bit of m122 is as follows (and it's a bit of pain that this code changes in my head to 112)

=== WiFi ===
Interface state: active
Module is connected to access point
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1.0d
MAC address 60:01:94:73:54:4c
Module reset reason: Power up, Vcc 3.44, flash size 4194304, free heap 42540
WiFi IP address 192.168.221.183
Signal strength -44dBm, channel 6, mode 802.11n, reconnections 0
Clock register 00002002
Socket states: 0 0 0 0 0 0 0 0
ok
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.5.1
Executing config.g... HTTP is enabled on port 80
FTP is disabled
TELNET is disabled
Done!

droftarts

@zBeeble

M115
FIRMWARE_NAME: RepRapFirmware for Duet 3 Mini 5+ FIRMWARE_VERSION: 3.5.1 ELECTRONICS: Duet 3 Mini5plus WiFi FIRMWARE_DATE: 2024-04-19 14:41:25
M111
Debugging enabled for modules:
Debugging disabled for modules: Platform(0) Network(1) Webserver(2) Gcodes(3) Move(4) Heat(5) Kinematics(6) InputShaping(7) unused(8) PrintMonitor(9) Storage(10) PortControl(11) DuetExpansion(12) FilamentSensors(13) WiFi(14) Display(15) SbcInterface(16) CAN(17) Expansion(18)

WiFi is module 14, Network is module 1 (at least for me on a Duet 3 Mini 5+) so send M111 P14 S1 and M111 P1 S1 at startup to enable logging of those, with your serial terminal connected.

@zBeeble said in Duet2 wifi stuck repeating Wifi Error.:

If you add minicom, you might note that from stock config, turning on 'add carriage return' and turning on 'line wrap' and turning on 'local echo' are really good settings.

We do! See https://docs.duet3d.com/en/How_to_guides/Getting_connected/Getting_connected_to_your_Duet#minicom

Press CTRL-A then W for linewrap, CTRL-A then U for CR at end of line, and CTRL-A then E for local echo. CTRL-A then X quits Minicom, releasing the port.

Ian

zBeeble

@droftarts said in Duet2 wifi stuck repeating Wifi Error.:

WiFi is module 14, Network is module 1 (at least for me on a Duet 3 Mini 5+) so send M111 P14 S1 and M111 P1 S1 at startup to enable logging of those, with your serial terminal connected.

OK. That works, but I don't see leaving network on --- it's chatty even when the printer is not doing anything. That's just a disk filling service.

zBeeble

So... I'm preparing to leave for BSDCan. Printer has been idle for about 24h. I was in the area and saw this:

https://nextcloud.towernet.ca/s/FNtRo4MyJHiG7PR

So then I checked the console...

WiFi: I (142798071) wifi:bcn_timout,ap_probe_send_start
WiFi: I (142818052) wifi:state: 5 -> 0 (6c0)
ESP reported status change
Error: WiFi module reported: Lost connection, auto reconnecting
ESP reported status change
WiFi: /home/renz/Projects/Duet3D/WiFiSocketServerRTOS/src/SocketServer.cpp(179): wifi evt: WIFI_EVENT id: 5
ESP reported status change
WiFi: /home/renz/Projects/Duet3D/WiFiSocketServerRTOS/src/SocketServer.cpp(199): disconnect reason: 6
WiFi: /home/renz/Projects/Duet3D/WiFiSocketServerRTOS/src/SocketServer.cpp(469): Lost connection to AP
WiFi: I (142820117) wifi:state: 0 -> 2 (b0)
WiFi: I (142820123) wifi:state: 2 -> 3 (0)
WiFi: I (142820130) wifi:state: 3 -> 5 (10)
WiFi: I (142820174) wifi:connected with ac.DaveG.ca, aid = 1, channel 6, HT20, bssid = b4:fb:e4:d7:be:cc
WiFi: /home/renz/Projects/Duet3D/WiFiSocketServerRTOS/src/SocketServer.cpp(179): wifi evt: WIFI_EVENT id: 4
ESP reported status change
Error: WiFi module reported: Auto reconnect succeeded
ESP reported status change
WiFi: /home/renz/Projects/Duet3D/WiFiSocketServerRTOS/src/SocketServer.cpp(179): wifi evt: IP_EVENT id: 0

followed by...

WiFi: I (190597586) wifi:bcn_timout,ap_probe_send_start

Now... this wasn't a repeating error, but I'm reporting it anyways.

rechrtb

@zBeeble Seems like the disconnect was detected at the lower levels of the ESP WiFi stack. The upper layers did what it's supposed to and attempted a reconnect. I don't think this was what causes the repeated SPI timeout error - but noted nontheless.

zBeeble

Would you guys expected this debug wifi to be significantly slower?

rechrtb

@zBeeble You mean in terms of like upload/download speed? Debug builds would be 'unoptimized' so yes.
Also, by the way, did the repeated "SPI timeout" issue not appear till now?

zBeeble

@rechrtb said in Duet2 wifi stuck repeating Wifi Error.:

@zBeeble You mean in terms of like upload/download speed? Debug builds would be 'unoptimized' so yes.
Also, by the way, did the repeated "SPI timeout" issue not appear till now?

I didn't notice it until now, but I can't answer with certainty.

zBeeble

OK. So ... printers been idle for a week or two --- serial console connected to system that stays online. Nothing to note.

However, was just printing something and the web UI had a few seconds where it was complaining no connection. This is certainly something I noticed before the larger problems.

No activity on the serial console, tho. Should I be entering something to turn that up?

rechrtb

@zBeeble Nothing beyond turning on the debug messages for network and WiFi. If I'm correct, you turned off the network debug messages though since it was too noisy? Also, please make sure it's logging to file so we don't miss anything.

zBeeble

@rechrtb said in Duet2 wifi stuck repeating Wifi Error.:

@zBeeble Nothing beyond turning on the debug messages for network and WiFi. If I'm correct, you turned off the network debug messages though since it was too noisy? Also, please make sure it's logging to file so we don't miss anything.

Yeah... the networking was generating a line every so often ... printer on or not.

But the machine is connected to the home server ... which is always on. I'm using minicom ... which is set to log all traffic

zBeeble

I generally have at least 2 computers (so I expect that to be 4 connections from time-to-time)... go this just now:

WiFi: /home/renz/Projects/Duet3D/WiFiSocketServerRTOS/src/Connection.cpp(578): refused conn on port 80 already 4 conns
WiFi: /home/renz/Projects/Duet3D/WiFiSocketServerRTOS/src/Connection.cpp(578): refused conn on port 80 already 4 conns
WiFi: /home/renz/Projects/Duet3D/WiFiSocketServerRTOS/src/Connection.cpp(578): refused conn on port 80 already 4 conns

... but can have as many as 3 or 4. Is this number tunable?

This doesn't seem to be my problem, tho... as it should recover.

droftarts

@zBeeble From the https://docs.duet3d.com/en/User_manual/Troubleshooting/WiFi_troubleshooting page, two things that might be relevant:

In Duet Web Control, in the Settings > Machine specific > Communication section, check that "Maximum number of AJAX retries" is at least 3.
Is the connection stable when the printer is idle, but unreliable when printing? If so then there may be too few CPU cycles to service the network interface, because of an excessive step pulse rate.
- Check the M122 report after a disconnection during a print, or after completing the print, and look at the number of 'hiccups' ('MaxReps' in old firmware versions) figure in the 'DDARing' or 'Move' section of the diagnostics.
- This value should be kept below about 50. If it is higher, reduce either microstepping (M350) or maximum speed (M203).
- Please note, hiccups and MaxReps is reset when you run M122 so only the value the first time you run M122 after a disconnection or completion of a print is significant.

Edit: Sorry, the above is old info, and MaxReps isn't reported in the current firmware M122 report. I've asked @dc42 how this information is reported now.
Edit: Updated the above with info from dc42.

Ian

rechrtb

@zBeeble said in Duet2 wifi stuck repeating Wifi Error.:

as generating a line every so often ... printer on or not.

But the machine is connected to the home server ... which is always on. I'm using minicom ... which is set to log all traffic

Reply

Those "already 4 conns" should be normal, especially when there are multiple clients with the web interface open.

zBeeble

So... been printing OK... for awhile. Just started acting up today again. Oddly, the serial output is "chunky" .... and incomplete. One recent query:

m122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.5.1 (2024-04-19 14:40:46) running on Duet WiFi 1.02 or later
Board ID: 08DDM-9FAM2-LW4SD-6J9F8-3SN6J-T3ZRY
Used output buffers: 1 of 26 (26 max)
=== RTOS ===
Static ram: 23256
Dynamic ram: 72928 of which 12 recycled
Never used RAM 13540, free system stack 110 words
Tasks: NETWORK(1,ready,15.7%,175) HEAT(3,nWait 5,0.1%,307) Move(4,nWait 5,2.9%,298) MAIN(1,running,81.3%,711) IDLE(0,ready,0.0%,
29), total 100.0%
Owned mutexes: WiFi(NETWORK) USB(MAIN)
=== Platform ===
Last reset 10:03:58 ago, cause: power up
Last software reset at 2024-07-29 10:46, reason: User, Gcodes spinning, available RAM 17140, slot 2
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x04
Aux0 errors 0,0,0
MCU temperature: min 35.4, current 35.9, max 36.9
Supply voltage: min 11.0, current 12.2, max 12.3, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/6, heap memory allocated/used/recyclable 2048/932/816, gc cycles 1
Events: 0 queued, 0 completed
Driver 0: ok, SG min 0
Driver 1: ok, SG min 0
Driver 2: ok, SG min 0
Driver 3: ok, SG min 0
Driver 4: standstill, SG min n/a
Driver 5:
Driver 6:
Driver 7:
Driver 8:
Driver 9:
Driver 10:
Driver 11:
Date/time: 2024-07-30 01:04:58
Cache data hit count 4294967295
Slowest loop: 11.10ms; fastest: 0.20ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 9
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 1.5ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, segments created 23, maxWait 448ms, bed compensation in use: mesh, height map offset -0.031, max steps late 0, m
in interval 0, bad calcs 0, ebfmin 0.00, ebfmax 1.00
next step interrupt due in 20 ticks, disab

zBeeble

Notably, on one output not shown, "slowest loop" was around 146ms.

zBeeble

I've turned on the debugging for 14 and 1 (wifi and network). Lots of chatter, but the printer is definitely acting up. The serial is no longer chunky, but at one point, I disconnected the serial ... which would reset the driver on the server.... and probably the duet, too.

m111 and chatter look like:

Debugging enabled for modules: Network(1 - 0xffffffff) WiFi(14 - 0xffffffff)
Debugging disabled for modules: Platform(0) Webserver(2) Gcodes(3) Move(4) Heat(5) Kinematics(6) InputShaping(7) unused(8) Print
Monitor(9) Storage(10) PortControl(11) DuetExpansion(12) FilamentSensors(13) Display(15) SbcInterface(16) CAN(17) Expansion(18)
New conn on socket 0 for local port 80
Found responder
Received 367 bytes
New conn on socket 1 for local port 80
Found responder
Received 329 bytes
New conn on socket 0 for local port 80
Found responder
New conn on socket 1 for local port 80
Found responder
Received 329 bytes
Received 367 bytes
New conn on socket 0 for local port 80