Duet 3 Wifi module firmware 2.1b4 - random disconnects

[NeoDue]

My Wiif module suffers from random disconnects - which let it fail to reconnect until the Duet 3 6HC itself is rebooted. This happens about once a day at the moment, at completely erratic times. The Wifi connection is very stable apart from when this happens, the access point the printer connects to is very close.

I managed to get an M122 log out of the Duet without rebooting it via the PanelDue:

2023-10-07 15:38:07 [info] Event logging started at level debug
2023-10-07 15:38:07 [info] Running: Duet 3 MB6HC v1.02 or later: 3.5.0-rc.1 (2023-08-31 16:19:24)
2023-10-07 15:38:18 [debug] === Diagnostics ===
2023-10-07 15:38:18 [debug] RepRapFirmware for Duet 3 MB6HC version 3.5.0-rc.1 (2023-08-31 16:19:24) running on Duet 3 MB6HC v1.02 or later (standalone mode)
2023-10-07 15:38:18 [debug] Board ID: 08DJM-956BA-NA3TN-6JTDL-3SN6L-998UU
2023-10-07 15:38:18 [debug] Used output buffers: 3 of 40 (40 max)
2023-10-07 15:38:18 [debug] === RTOS ===
2023-10-07 15:38:18 [debug] Static ram: 154852
2023-10-07 15:38:18 [debug] Dynamic ram: 123836 of which 0 recycled
2023-10-07 15:38:18 [debug] Never used RAM 63536, free system stack 136 words
2023-10-07 15:38:18 [debug] Tasks:
2023-10-07 15:38:18 [debug]  NETWORK(1,ready,69.1%,187)
2023-10-07 15:38:18 [debug]  HEAT(3,nWait,0.2%,323)
2023-10-07 15:38:18 [debug]  Move(4,nWait,7.7%,214)
2023-10-07 15:38:18 [debug]  CanReceiv(6,nWait,0.0%,941)
2023-10-07 15:38:18 [debug]  CanSender(5,nWait,0.0%,335)
2023-10-07 15:38:18 [debug]  CanClock(7,delaying,0.1%,343)
2023-10-07 15:38:18 [debug]  TMC(4,nWait,43.7%,59)
2023-10-07 15:38:18 [debug]  MAIN(1,running,340.6%,137)
2023-10-07 15:38:18 [debug]  IDLE(0,ready,1.4%,30)
2023-10-07 15:38:18 [debug] , total 462.7%
Owned mutexes:
2023-10-07 15:38:18 [debug]  Aux(MAIN)
2023-10-07 15:38:18 [debug] === Platform ===
2023-10-07 15:38:18 [debug] Last reset 02:01:45 ago, cause: power up
2023-10-07 15:38:18 [debug] Last software reset at 2023-10-06 01:50, reason: User, Gcodes spinning, available RAM 62972, slot 2
2023-10-07 15:38:18 [debug] Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a
2023-10-07 15:38:18 [debug] Error status: 0x04
2023-10-07 15:38:18 [debug] Aux0 errors 0,0,0
2023-10-07 15:38:18 [debug] MCU temperature: min 20.6, current 47.7, max 49.5
2023-10-07 15:38:18 [debug] Supply voltage: min 23.6, current 23.9, max 24.2, under voltage events: 0, over voltage events: 0, power good: yes
2023-10-07 15:38:18 [debug] 12V rail voltage: min 12.0, current 12.4, max 12.8, under voltage events: 0
2023-10-07 15:38:18 [debug] Heap OK, handles allocated/used 99/5, heap memory allocated/used/recyclable 2048/1192/1092, gc cycles 22
2023-10-07 15:38:18 [debug] Events: 0 queued, 0 completed
2023-10-07 15:38:18 [debug] Driver 0: standstill, SG min 0, mspos 344, reads 48608, writes 35 timeouts 0
2023-10-07 15:38:18 [debug] Driver 1: standstill, SG min 0, mspos 280, reads 48608, writes 35 timeouts 0
2023-10-07 15:38:18 [debug] Driver 2: standstill, SG min 0, mspos 72, reads 48622, writes 28 timeouts 0
2023-10-07 15:38:18 [debug] Driver 3: standstill, SG min 0, mspos 776, reads 48619, writes 31 timeouts 0
2023-10-07 15:38:18 [debug] Driver 4: standstill, SG min 0, mspos 414, reads 48627, writes 24 timeouts 0
2023-10-07 15:38:18 [debug] Driver 5: standstill, SG min 0, mspos 214, reads 48627, writes 24 timeouts 0
2023-10-07 15:38:18 [debug] Date/time: 
2023-10-07 15:38:18 [debug] 2023-10-07 15:38:18
2023-10-07 15:38:18 [debug] Slowest loop: 224.94ms; fastest: 0.05ms
2023-10-07 15:38:18 [debug] === Storage ===
Free file entries: 19
2023-10-07 15:38:18 [debug] SD card 0 detected, interface speed: 25.0MBytes/sec
2023-10-07 15:38:18 [debug] SD card longest read time 7.1ms, write time 82.2ms, max retries 0
2023-10-07 15:38:18 [debug] === Move ===
DMs created 125, segments created 38, maxWait 147189ms, bed compensation in use: none, height map offset 0.000, ebfmin -1.00, ebfmax 1.00
2023-10-07 15:38:18 [debug] no step interrupt scheduled
2023-10-07 15:38:18 [debug] Moves shaped first try 96, on retry 2566, too short 21650, wrong shape 83887, maybepossible 10136
2023-10-07 15:38:18 [debug] === DDARing 0 ===
Scheduled moves 174214, completed 174214, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
2023-10-07 15:38:18 [debug] === DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
2023-10-07 15:38:18 [debug] === Heat ===
2023-10-07 15:38:18 [debug] Bed heaters 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
2023-10-07 15:38:18 [debug] Heater 0 is on, I-accum = 0.5
2023-10-07 15:38:18 [debug] Heater 2 is on, I-accum = 0.5
2023-10-07 15:38:18 [debug] === GCodes ===
2023-10-07 15:38:18 [debug] Movement locks held by null, null
2023-10-07 15:38:18 [debug] HTTP is idle in state(s) 0
2023-10-07 15:38:18 [debug] Telnet is idle in state(s) 0
2023-10-07 15:38:18 [debug] File is idle in state(s) 0
2023-10-07 15:38:18 [debug] USB is idle in state(s) 0
2023-10-07 15:38:18 [debug] Aux is ready with "M122" in state(s) 0
2023-10-07 15:38:18 [debug] Trigger is idle in state(s) 0
2023-10-07 15:38:18 [debug] Queue is idle in state(s) 0
2023-10-07 15:38:18 [debug] LCD is idle in state(s) 0
2023-10-07 15:38:18 [debug] SBC is idle in state(s) 0
2023-10-07 15:38:18 [debug] Daemon is idle in state(s) 0
2023-10-07 15:38:18 [debug] Aux2 is idle in state(s) 0
2023-10-07 15:38:18 [debug] Autopause is idle in state(s) 0
2023-10-07 15:38:18 [debug] File2 is idle in state(s) 0, sync state 2
2023-10-07 15:38:18 [debug] Queue2 is idle in state(s) 0
2023-10-07 15:38:18 [debug] Q0 segments left 0, axes/extruders owned 0x4000000a
2023-10-07 15:38:18 [debug] Code queue 0 is empty
2023-10-07 15:38:18 [debug] Q1 segments left 0, axes/extruders owned 0x0000000
2023-10-07 15:38:18 [debug] Code queue 1 is empty
2023-10-07 15:38:18 [debug] === Filament sensors ===
2023-10-07 15:38:18 [debug] Extruder 0 sensor: ok
2023-10-07 15:38:18 [debug] Extruder 1 sensor: ok
2023-10-07 15:38:18 [debug] === CAN ===
2023-10-07 15:38:18 [debug] Messages queued 65738, received 0, lost 0, boc 0
2023-10-07 15:38:18 [debug] Longest wait 0ms for reply type 0, peak Tx sync delay 0, free buffers 50 (min 50), ts 36527/0/0
2023-10-07 15:38:18 [debug] Tx timeouts 0,0,36526,0,0,29210 last cancelled message type 30 dest 127
2023-10-07 15:38:18 [debug] === Network ===
2023-10-07 15:38:18 [debug] Slowest loop: 91.91ms; fastest: 0.00ms
2023-10-07 15:38:18 [debug] Responder states:
2023-10-07 15:38:18 [debug]  MQTT(0)
2023-10-07 15:38:18 [debug]  HTTP(0)
2023-10-07 15:38:18 [debug]  HTTP(0)
2023-10-07 15:38:18 [debug]  HTTP(0)
2023-10-07 15:38:18 [debug]  HTTP(0)
2023-10-07 15:38:18 [debug]  HTTP(0)
2023-10-07 15:38:18 [debug]  HTTP(0)
2023-10-07 15:38:18 [debug]  FTP(0)
2023-10-07 15:38:18 [debug]  Telnet(0)
2023-10-07 15:38:18 [debug]  Telnet(0)
2023-10-07 15:38:18 [debug] HTTP sessions: 0 of 8
2023-10-07 15:38:18 [debug] = Ethernet =
Interface state: disabled
2023-10-07 15:38:18 [debug] Error counts: 0 0 0 0 0 0
Socket states:
2023-10-07 15:38:18 [debug]  0
2023-10-07 15:38:18 [debug]  0
2023-10-07 15:38:18 [debug]  0
2023-10-07 15:38:18 [debug]  0
2023-10-07 15:38:18 [debug]  0
2023-10-07 15:38:18 [debug]  0
2023-10-07 15:38:18 [debug]  0
2023-10-07 15:38:18 [debug]  0
2023-10-07 15:38:18 [debug] === WiFi ===
Interface state: changingMode
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
2023-10-07 15:38:18 [debug] Failed to get WiFi status
2023-10-07 15:38:18 [debug] Socket states:
2023-10-07 15:38:18 [debug]  0
2023-10-07 15:38:18 [debug]  0
2023-10-07 15:38:18 [debug]  0
2023-10-07 15:38:18 [debug]  0
2023-10-07 15:38:18 [debug]  0
2023-10-07 15:38:18 [debug]  0
2023-10-07 15:38:18 [debug]  0
2023-10-07 15:38:18 [debug]  0
2023-10-07 15:38:18 [debug] === Multicast handler ===
Responder is inactive, messages received 0, responses 0
2023-10-07 15:38:18 [debug] {"err":-1}
 
2023-10-07 15:38:30 [info] Event logging stopped

Is there anything I can try or do to provide you with additional data?

[NeoDue]

One more thing I tried: in that disconnected state, the Wifi module

• still has its green LED active and
• does not react to M552 commands.

Edit: the issue also seems to be something that happens after a longer period of the Duet running. Up to now, it never happened right after a boot, only after several hours of the printer being powered on.

[jens55]

@NeoDue, I had a similar issue where only a power cycle would bring the connection back. I reverted to an older wifi firmware because the problem started to happen after a firmware upgrade. I haven't seen the issue since. Due to the random nature of the fault I can't say if my reverting back to an older firmware fixed anything or not .... but it might be worthwhile for you to try that.

[NeoDue]

@jens55 thanks, but I would rather prefer that v.3.5.0 gets this issue solved - that firmware revision has some features I am quite happy about

Therefore I am rather living with this issue and try to help where I can to find its cause.

[jens55]

@NeoDue, I fully agree that the right thing to do is to get the issue resolved. Reverting to an older version of firmware is part of the process of discovering where the error showed up.
Note that you are mixing up firmware for the Duet boards (V3.5.0) and the wifi module firmware (2.1). In my case I still run duet 3.5.0 firmware and just reverted to an older wifi module firmware. I am not aware what the changes are between the two versions of wifi firmware so maybe there is a good reason to stay with 2.1 but that doesn't negate the fact that tracing the issue requires determining when the issue first crept up and what has changed between having a board that works and having a board that hangs intermittently.
In your case, if you were to revert to an older version of the wifi firmware, you would confirm my suspicion that the issue showed up with the new wifi firmware which in turn would help the developers.
As I get older, I am making more and more 'old fart' mistakes and I can't trust that my reverting to older wifi firmware fixed the issue - if you were to revert and the issue would go away then this would confirm my observations.

[NeoDue]

@jens55 ah, you were talking about the Wifi module firmware. Sorry, then I misunderstood you. The issue in my case is that I have the new wifi module for the Duet 3 6HC - where the Duet team states that at least firmware 2.1b3 must be used (https://docs.duet3d.com/en/Duet3D_hardware/Duet_3_family/Duet_3_Mainboard_6HC_Hardware_Overview#wifi-v102 ). Currently I am running 2.1b4 which also the version that is shipped with RRF3.5.0rc1.

Thus, I fear my options regarding downgrading only involve going back to 2.1b3 - but that is indeed something I will try, thanks!

[tas]

@NeoDue
Some 'external to Duet' things to try:

Someone else on this forum noticed that a wireless phone in the vicinity of the Duet could cause this. I had a Vtech phone in my printer room and removing it did indeed resolve most of my issues.

A second issue was with my router, not the Duet. There was a switchover frequency set in the router where if the 2.4ghz signal strength dropped below a certain level the router would autoswitch the Duet connection to 5ghz. To fix that I just had to change the switchover limit in the router to a level that would never happen like -70db. My signal strength is never lower than -57db.

[jens55]

@NeoDue I was not aware of requirement to run wifi firmware 2.1x x for the 6hc board. My issue was with a Duet2wifi board. The 6HC board I have is only capable of ethernet.

[jens55]

@tas, interesting to hear that the newer wifi modules support 5 GHz - did not know that.

[jay_s_uk]

@jens55 it doesn't support 5Ghz

[tas]

@jens55 Sorry to confuse the issue.

Duets do not support 5ghz. I was trying to point out that when my router switched to 5ghz the Duet lost connection as a result.

[NeoDue]

@jens55 Yes, my Duet 2 Wifi delivers a rock solid performance since day one which was about six years ago - I did not upgrade it since firmware 3.4.3 though since I pretty much focused on my new printer then.

Compared to that, I have to admit the experience with the Duet 3 so far leaves a bit to desire:

• the random lockups of the Wifi module,
• then there are rather frequent hiccups when the Wifi module connects after a boot (some seconds up to some minutes of recurring "authentication failed" messages on the PanelDue - and suddenly the thing connects anyway),
• and every now and then the Duet 3 rebooting unexpectedly with a "StuckInSpinLoop" error in M122. The latter is likely triggered by the web server somehow since you can severely improve that issue by disabling the Object Model Browser - but a unlucky hang of the web browser on the computer still seems to be capable of killing the Duet...

Anyway, I am confident the Duet team will sort these issues out eventually.

[NeoDue]

@tas thanks a lot, but I think I can rule these out:
there is no wireless phone (or anything else that disturbs Wifi) anywhere near the printer or in the same room. And the access point the printer connects to only has 2.4GHz - I never cared to upgrade that, since the Wifi frequencies are not that full here and I frankly do not care about the additional speed.

It might be worth noting that my old printer who runs on a Duet 2 Wifi was placed at exactly the same spot - and never had (or has, when I count the infrequent times when I turn it in at its current place under the table) any connection issues.

[NeoDue]

Just in case someone else follows this: I just saw a new Wifi module firmware (2.1beta6) has been released that addresses some Wifi connection issues if I understand the release notes correctly. I am currently giving it a try and will report what happens here.

[NeoDue]

Update: not a single disconnect or "does-not-connect-on-boot" issue with 2.1b6 for the last week, even after I removed the G4 command in config.g which before had reliably led to very long fruitless connection attempts.

I will mark this solved therefore.

[dc42]

@NeoDue I'm glad the new version fixed it for you. Thanks for reporting.