Disconnects and freezing

Nxt-1

For the last few weeks I am experiencing issues with the occasional connection lost without getting auto reconnected. For my limited testing there seem to be be three scenario's (and I sadly have not found a structure in them ) :

1) Frequent connection lost but auto reconnected

I have only seen this once I believe. When it happened, disconnects were coming up 10-20 times per minute. This could easy be just a fluke for whatever reason.

2) Lost connection - can reset wifi module via Panel Due

I this state the Duet is not reachable from from my browser (Chrome) when I try to connect to it's IP (static). I can also confirm that I cannot ping to the Duet in this state. The easiest way out of this state seems to be to reset the wifi module, for which I have a macro:

M291 P"Resetting wifi module..." S3 T5

M552 S0 			; disconnect and shutdown wifi services
M587 				; list access points, with luck this will wait for the shutdown to complete
M552 S-1			; Disable wifi module
M552 S1				; Enable networking as client

M291 P"Wifi module reset. Check console or DWC." S2

M122 dump from when the Duet is in this state:

M122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 2.05.1 running on Duet WiFi 1.0 or 1.01
Board ID: 08D6M-91AST-L23S4-7JTD0-3S86K-1NX1K
Used output buffers: 1 of 24 (15 max)
=== RTOS ===
Static ram: 25712
Dynamic ram: 93428 of which 12 recycled
Exception stack ram used: 392
Never used ram: 11528
Tasks: NETWORK(ready,628) HEAT(blocked,1232) MAIN(running,3816) IDLE(ready,160)
Owned mutexes:
=== Platform ===
Last reset 18:30:46 ago, cause: software
Last software reset at 2020-06-09 13:25, reason: User, spinning module GCodes, available RAM 11432 bytes (slot 3)
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x04433000 BFAR 0xe000ed38 SP 0xffffffff Task 0x4e49414d
Error status: 0
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest block write time: 0.0ms, max retries 0
MCU temperature: min 24.6, current 24.8, max 25.0
Supply voltage: min 1.2, current 1.3, max 1.3, under voltage events: 1, over voltage events: 0, power good: no
Driver 0: standstill, SG min/max not available
Driver 1: standstill, SG min/max not available
Driver 2: standstill, SG min/max not available
Driver 3: standstill, SG min/max not available
Driver 4: standstill, SG min/max not available
Date/time: 2020-06-10 07:56:25
Cache data hit count 4294967295
Slowest loop: 1.05ms; fastest: 0.06ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Move ===
Hiccups: 0, FreeDm: 160, MinFreeDm: 160, MaxWait: 0ms
Bed compensation in use: none, comp offset 0.000
=== DDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Stack records: 3 allocated, 0 in use
Movement lock held by null
http is idle in state(s) 0
telnet is idle in state(s) 0
file is idle in state(s) 0
serial is ready with "M122" in state(s) 0
aux is idle in state(s) 0
daemon is idle in state(s) 0
queue is idle in state(s) 0
autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 0.17ms; fastest: 0.00ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 0 of 8
- WiFi -
Network state is changing mode
WiFi module is idle
Failed messages: pending 0, notready 0, noresp 3
Failed to get WiFi status
Socket states: 0 0 0 0 0 0 0 0
=== Filament sensors ===
Extruder 0: no data received
ok

3) Lost connection - PanelDue unresponsive (mostly)

When in this state, the previously mentioned macro does not help, as a matter of fact is does not even run. The only way out of this state AFAIK (without power cycling) is to hit the stop button on the PanelDue, this still works.
I preparation for this post I wanted to get an M122 dump while in this state, yet to my surprise, I wasn't even able to connect via USB using YAT. And I did get connection after a stop, just to verify the issue not YAT/USB related here.
I can also confirm that I can still ping to the Duet in this state.

Other info

My machine is a delta style printer with a Duet Wifi on board. Below a M122 dump from when it seems to be working fine, as well as my config.g:

M122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 2.05.1 running on Duet WiFi 1.0 or 1.01
Board ID: 08D6M-91AST-L23S4-7JTD0-3S86K-1NX1K
Used output buffers: 3 of 24 (11 max)
=== RTOS ===
Static ram: 25712
Dynamic ram: 93428 of which 12 recycled
Exception stack ram used: 392
Never used ram: 11528
Tasks: NETWORK(ready,628) HEAT(blocked,1232) MAIN(running,3816) IDLE(ready,160)
Owned mutexes:
=== Platform ===
Last reset 03:24:29 ago, cause: software
Last software reset at 2020-06-09 13:25, reason: User, spinning module GCodes, available RAM 11432 bytes (slot 3)
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x04433000 BFAR 0xe000ed38 SP 0xffffffff Task 0x4e49414d
Error status: 0
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest block write time: 108.2ms, max retries 0
MCU temperature: min 28.8, current 29.8, max 32.7
Supply voltage: min 1.0, current 1.3, max 24.5, under voltage events: 1, over voltage events: 0, power good: no
Driver 0: standstill, SG min/max 0/115
Driver 1: standstill, SG min/max 0/112
Driver 2: standstill, SG min/max 0/117
Driver 3: standstill, SG min/max not available
Driver 4: standstill, SG min/max 0/106
Date/time: 2020-06-09 16:50:07
Cache data hit count 4294967295
Slowest loop: 149.17ms; fastest: 0.06ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Move ===
Hiccups: 0, FreeDm: 160, MinFreeDm: 156, MaxWait: 343853ms
Bed compensation in use: none, comp offset 0.000
=== DDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Stack records: 3 allocated, 0 in use
Movement lock held by null
http is idle in state(s) 0
telnet is idle in state(s) 0
file is idle in state(s) 0
serial is idle in state(s) 0
aux is idle in state(s) 0
daemon is idle in state(s) 0
queue is idle in state(s) 0
autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 202.11ms; fastest: 0.08ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 1 of 8
- WiFi -
Network state is running
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 1
WiFi firmware version 1.23
WiFi MAC address a0:20:a6:16:ec:ed
WiFi Vcc 3.36, reset reason Turned on by main processor
WiFi flash size 4194304, free heap 23040
WiFi IP address 192.168.1.55
WiFi signal strength -55dBm, reconnections 0, sleep mode modem
Socket states: 0 0 0 0 0 0 0 0
=== Filament sensors ===
Extruder 0: no data received

; Configuration file for Deltex

; ===================== GENERAL SETTINGS =====================
M111 S0                             													; Debug off
M550 PDeltex - V6																		; Machine name and Netbios name
M555 P2                                   												; Repetier Output
M575 P1 B57600 S1                          												; PanelDue Comm Setup
M540 P0xBE:0xEF:0xDE:0xAD:0xFE:0xEA 													; MAC Address
M552 S1																					; Enable WiFi
M586 P1 S0																				; Disable FTP
M586 P2 S0																				; Disable Telnet

; ===================== MOVEMENT SETTINGS ====================
G21                                 													; Set Units to Millimeters
G90                            															; Set to Absolute Positioning
M83                                	 													; Set extruder to relative mode

M584 X0 Y1 Z2 U4 E3																		; Assign drives to each motor
M569 P0 S0																				; Drive 0 (X) goes backwards
M569 P1 S0																				; Drive 1 (Y) goes backwards
M569 P2 S0																				; Drive 2 (Z) goes backwards
M569 P3 S1																				; Drive 3 (E) goes forwards
M569 P4 S0																				; Drive 4 (4th) goes backwards
M574 X2 Y2 Z2 U2 S1																		; Endstop configuration (all endstops at high end, active high)
M665 L500.000:500.000:500.000:224.000 R236.244 H642.108 B200.0 X0.266 Y-0.108 Z0.000	; Diagonal rod length, delta radius, printable radius and homed height
M669 K3 X0 Y0																			; Location of the 4th axis tower
M666 X-0.054 Y3.533 Z-3.478 A0.00 B0.00													; Endstop adjustments
M350 X16 Y16 Z16 E16 U16 I1    															; Set 16x microstepping with interpolation
M92 X200 Y200 Z200 U80 E414																; Set steps/mm
M906 X2050 Y2050 Z2050 U800 E750 I60													; Set motor currents (mA) and increase idle current to 60%
M201 X1500 Y1500 Z1000 U900 E3000														; Accelerations (mm/s^2)
M203 X10000 Y10000 Z10000 U10000 E20000													; Maximum speeds (mm/min)
M566 X400 Y400 Z250 U200 E3600 P0														; Maximum instant speed changes (mm/min)
M591 D0 P5 C4 R40:120 E3.0 S1															; Duet3D laser sensor for extruder drive 0, connected to endstop input 4 (E1), tolerance 40 to 120%, 3mm comparison length, enabled

; ================ HEATING AND COOLING SETTINGS ==============
; Thermistors
M305 S"Bed" P0 B3950																	; Bed temperature sensor parameters
M305 S"Hotend" P1 B4725 C7.060000e-8													; Hotend temperature sensor parameters 
M305 S"Water" P2 X2 T10000 B3950														; Water temperature sensor parameters
M570 S180 T15 S4320																		; Configure heater fault detection, allow a fault to persist for 180 seconds (+/- 15°). Cancel the print if the fault is not cleared after 4320 minutes (3 days).
M307 H0 A408.0 C897.0 D2.1 S1.00 V24.2 B0												; Set bed heating process parameters and use PID at 10Hz
M307 H1 A559.0 C285.9 D5.3 S1.00 V24.2 B0												; Set hotend heating process parameters and use PID

M143 H0 S200																			; Set maximum bed temperature to 200

; Fans
M106 P0 I0 H-1 																			; disable thermostatic mode for fan 0 (part cooling)
M106 P1 S255 T45 H1																		; enable thermostatic mode for fan 1 (hotend)

; ====================== TOOL SETTINGS =======================
M563 P0 D0 H1 S"Hotend"              													; Define hotend tool
G10 P0 S0 R0                        													; Set tool 0 operating and standby temperatures
M207 S0	R0 F3000 T1200 Z0																; Set default retraction settings

M563 P1 S"Water" H2																		; Set up an extra tool to display heater temperature

; =================== CALIBRATION SETTINGS ===================
M558 P8 F1200 H3 R0.4 T10000 A10 S0.04													; Z Probe type 8, input 0, invert no, dive height 3.0mm, probe speed 1200mm/min, travel speed 10000mm/min, recovery time 0.40 sec, heaters normal, max taps 5, max diff 0.04
G31 X0 Y0 Z-0.15 P100  		        													; Set the zprobe height and threshold (more negative values translate into a higher nozzle)
M557 R200 P21																			; Define probing mesh grid


M912 P0 S-12.35																			; Set temperature offset for MCU
T0																						; Select hot end
M501																					; Load settings from config-override.g

JoergS5

@Nxt-1 said in Disconnects and freezing:

Supply voltage: min 1.0, current 1.3, max 24.5, under voltage events: 1

I would suspect it has something to do with your voltage drop. If it's a comparable situation like at ESP32 chips, Wifi needs some power.

Nxt-1

@JoergS5 said in Disconnects and freezing:

@Nxt-1 said in Disconnects and freezing:

Supply voltage: min 1.0, current 1.3, max 24.5, under voltage events: 1

I would suspect it has something to do with your voltage drop. If it's a comparable situation like at ESP32 chips, Wifi needs some power.

I am using an external 5V PSU, I am not sure that voltage is monitored.

Nxt-1

Update: when I woke up this morning, I noticed a loss of connection that turned out to be of variety 2. I updated the original post with an M122 dump from when the machine was unreachable. As typical for the case, I was able to reestablish connection by using the reset wifi module macro. One other thing to note, I was unable to ping the Duet in this case. Not surprisingly, as the wifi module seems to be disabled for some reason.

Phaedrux

Can you try reflashing the wifi server firmware?

Nxt-1

@Phaedrux A long print I was doing finally finished, so I can update the wifi server bin file.

I should add that I have a dual sys directory setup, both for the same firmware, yet different config.g's. I am not sure what the correct procedure is to update firmware files in this case, as just uploading and installing them places the files just in /sys instead of /sys/DeltextV6 for example. I cant just copy the files manually, as DWC does not let you overwrite system files like that.

Phaedrux

I think in that case you'd just upload to the normal /sys folder.

Nxt-1

@Phaedrux I updated the wifiserver (in /sys). I am not sure which bin file the Duet now uses as there is one in /sys, /sys/DeltexV6 (and /sys/DeltexSV). That aside, I am having another case of situation 3). So no connection, no panel Due wifi reset, no YAT but is available to ping.

Phaedrux

The wifiserver bin is flashed to the wifi chip, so once you've flashed it, it's on the chip and the file doesn't get used. The duet will use /sys at boot. So download the proper bin file again just to make sure it's the right version.

https://github.com/Duet3D/RepRapFirmware/releases/download/2.05.1/DuetWiFiServer-1.23.bin

Rename it to DuetWiFiServer.bin (remove the version number) and place it in /sys.

Then send M997 S1 over USB terminal to start the wifi server update.

Then let's see what happens.

Nxt-1

@Phaedrux For some reason I missed your last reply, sorry for the long wait. Meanwhile, I upgraded to RRF3.1.1 so I guess that covers updating the wifi server as well

Yesterday I had two disconnects that I could recover from (so type 2). From the last instance, I connected via USB before resetting the wifi module and here is the output:

M115
FIRMWARE_NAME: RepRapFirmware for Duet 2 WiFi/Ethernet FIRMWARE_VERSION: 3.1.1 ELECTRONICS: Duet WiFi 1.0 or 1.01 FIRMWARE_DATE: 2020-05-19b2
ok

M122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.1.1 running on Duet WiFi 1.0 or 1.01
Board ID: 08D6M-91AST-L23S4-7JTD0-3S86K-1NX1K
Used output buffers: 1 of 24 (22 max)
=== RTOS ===
Static ram: 27980
Dynamic ram: 93632 of which 20 recycled
Exception stack ram used: 592
Never used ram: 8848
Tasks: NETWORK(ready,368) HEAT(blocked,1224) MAIN(running,612) IDLE(ready,80)
Owned mutexes:
=== Platform ===
Last reset 34:26:56 ago, cause: power up
Last software reset at 2020-06-23 16:38, reason: User, spinning module GCodes, available RAM 8560 bytes (slot 1)
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0441f000 BFAR 0xe000ed38 SP 0xffffffff Task MAIN
Error status: 10
MCU temperature: min 29.1, current 30.9, max 40.9
Supply voltage: min 0.5, current 0.7, max 24.5, under voltage events: 2, over voltage events: 0, power good: no
Driver 0: standstill, SG min/max 0/199
Driver 1: standstill, SG min/max 0/251
Driver 2: standstill, SG min/max 0/179
Driver 3: standstill, SG min/max 0/1023
Driver 4: standstill, SG min/max 0/316
Date/time: 2020-06-25 08:03:58
Cache data hit count 4294967295
Slowest loop: 79.38ms; fastest: 0.09ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 2.0ms, write time 507.0ms, max retries 0
=== Move ===
Hiccups: 2864(0), FreeDm: 169, MinFreeDm: 104, MaxWait: 6777066ms
Bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves: 478071, completed moves: 478071, StepErrors: 0, LaErrors: 0, Underruns: 167, 0  CDDA state: -1
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
Heater 0 is on, I-accum = 0.0
Heater 1 is on, I-accum = 0.3
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is ready with "M122" in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 509.82ms; fastest: 0.00ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 0 of 8
- WiFi -
Network state is changingMode
WiFi module is idle
Failed messages: pending 0, notready 0, noresp 1
Failed to get WiFi status
Socket states: 0 0 0 0 0 0 0 0
=== Filament sensors ===
Extruder 0: no data received
ok

Here is the output from after a wifi module reset:

WiFi module stopped
WiFi module started
WiFi module is connected to access point FritsLiving, IP address 192.168.1.55
M122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.1.1 running on Duet WiFi 1.0 or 1.01
Board ID: 08D6M-91AST-L23S4-7JTD0-3S86K-1NX1K
Used output buffers: 1 of 24 (22 max)
=== RTOS ===
Static ram: 27980
Dynamic ram: 93632 of which 20 recycled
Exception stack ram used: 592
Never used ram: 8848
Tasks: NETWORK(ready,368) HEAT(blocked,1224) MAIN(running,612) IDLE(ready,80)
Owned mutexes:
=== Platform ===
Last reset 34:41:48 ago, cause: power up
Last software reset at 2020-06-23 16:38, reason: User, spinning module GCodes, available RAM 8560 bytes (slot 1)
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0441f000 BFAR 0xe000ed38 SP 0xffffffff Task MAIN
Error status: 10
MCU temperature: min 31.6, current 32.0, max 32.4
Supply voltage: min 0.6, current 0.7, max 0.8, under voltage events: 2, over voltage events: 0, power good: no
Driver 0: standstill, SG min/max not available
Driver 1: standstill, SG min/max not available
Driver 2: standstill, SG min/max not available
Driver 3: standstill, SG min/max not available
Driver 4: standstill, SG min/max not available
Date/time: 2020-06-25 08:18:50
Cache data hit count 4294967295
Slowest loop: 5.16ms; fastest: 0.09ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 1.0ms, write time 0.0ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 169, MinFreeDm: 169, MaxWait: 0ms
Bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves: 478071, completed moves: 478071, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
Heater 0 is on, I-accum = 0.0
Heater 1 is on, I-accum = 0.3
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is ready with "M122" in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 201.06ms; fastest: 0.08ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- WiFi -
Network state is active
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 1
WiFi firmware version 1.23
WiFi MAC address a0:20:a6:16:ec:ed
WiFi Vcc 3.36, reset reason Unknown
WiFi flash size 4194304, free heap 24504
WiFi IP address 192.168.1.55
WiFi signal strength -54dBm, reconnections 0, sleep mode modem
Socket states: 0 0 0 0 0 0 0 0
=== Filament sensors ===
Extruder 0: no data received
ok