Wifi 2.1beta6 from 3.5.0-rc.2/3 still disconnecting

Chriss

Here the screenshot from wireshark while I try to ping the printer:

Cheers, Chriss

gloomyandy

@Chriss Can you confirm what version of the WiFi firmware you are running (It should be displayed as part of the M122 output, which we would also really like to see).

Chriss

@gloomyandy You will get that tomorrow noonish. I need the long USB cable first. You will get the M122 as soon as Amazon delivered +10Minutes.

oliof

@Chriss do you have docker running on your host by chance? Depending on how that's set up traffic may be routed into the docker network and then you won't be able to get to the wifi connected printer.

rechrtb

@Chriss Thanks for the reply.

I'm not 100% sure, so please do not nail me down to it. But I think that it is 3.5.0RC2 I had before on it. But I saw that behaviour with the reconnects in 3.4 something already. It never bother me because the printer did always reconnect.

That's ok. So you suspect it was 3.5.0RC2 previously, do you happen to remember also the WiFi firmware version?

My two other printers and my "on my desk test board" are on 3.4.6. And I have found a other printer which I did not upgraded to RC2, it is still 3.5.0rc2 and this one does not have the problem. And this is very much in use at the moment.

In the second sentence, do you mean to say "not upgraded to RC3, it is still 3.5.0rc2" or "not upgraded to RC2, it is still 3.5.0rc1 (or maybe 3.4.x)". Also, can you also confirm the WiFi firmware version of these boards?

Just to be clear here about the status and for unblocking my confusion:
1: RC3 with that problem, with PanelDue and printing (My Voron 2.4)
2: RC3 without problem, without PenalDue and not printing (VCast)
3: RC2 without problem, without PenalDue and printing (V0)
4: 3.4.7 without problem, without PenalDue and not printing (Workbench, VCore, Creality)

Can you also confirm the wifi firmware version for these boards?

You'll notice that I'm constantly asking for the WiFi firmware version, since flashing a certain RRF version does not guarantee a certain WiFi firmware version. That is, you might have upgraded the RRF to 3.5.0rc2 or rc3, but the wifi firmware version can still be on an older version (or the opposite can also be true - you might have new WiFi firmware + old RRF).

Chriss

@oliof said in Wifi 2.1beta6 from 3.5.0-rc.2/3 still disconnecting:

@Chriss do you have docker running on your host by chance? Depending on how that's set up traffic may be routed into the docker network and then you won't be able to get to the wifi connected printer.

For god sake... no Docker in my household. Seriously: I have a bridge device for other reasons. (And other hosts can not reach the board too)

Cheers, Chriss

Chriss

@gloomyandy @rechrtb

The cable just arrived, here is the status of my Voron 2.4 with 3.5.0rc3 and the corresponding Wifi version:

=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.3 (2024-01-24 17:56:48) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: V9NWJ-R296U-D65J0-40KM6-4113Z-HM83B
Used output buffers: 1 of 40 (40 max)
Error in macro line 21 while starting up: in file macro line 21 column 33: M558: unknown variable 'probe_settingsH'
=== RTOS ===
Static ram: 103200
Dynamic ram: 126032 of which 12 recycled
Never used RAM 8500, free system stack 122 words
Tasks: NETWORK(1,ready,103.0%,156) HEAT(3,nWait 6,0.9%,326) Move(4,nWait 6,29.9%,243) CanReceiv(6,nWait 1,1.5%,771) CanSender(5,nWait 7,0.8%,328) CanClock(7,delaying,0.2%,349) TMC(4,nWait 6,42.2%,102) MAIN(1,running,44.2%,500) IDLE(0,ready,6.2%,30) AIN(4,delaying,25.1%,260), total 254.2%
Owned mutexes: USB(MAIN)
=== Platform ===
Last reset 29:39:53 ago, cause: power up
Last software reset at 2024-02-04 02:34, reason: User, Gcodes spinning, available RAM 8420, slot 2
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00487000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 106795063, completed 106795063, timed out 0, errs 0
MCU temperature: min 31.3, current 42.6, max 50.6
Supply voltage: min 24.2, current 24.4, max 24.4, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/48, heap memory allocated/used/recyclable 2048/868/104, gc cycles 471
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 83, reads 7572, writes 83, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 0, ifcnt 66, reads 7589, writes 66, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 64, reads 7591, writes 64, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 0, ifcnt 10, reads 7644, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 0, ifcnt 79, reads 7576, writes 79, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 0, read errors 0, write errors 0, ifcnt 79, reads 7576, writes 79, timeouts 0, DMA errors 0, CC errors 0
Driver 6: standstill, SG min 0, read errors 0, write errors 0, ifcnt 78, reads 7577, writes 78, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2024-02-06 12:38:38
Cache data hit count 4294967295
Slowest loop: 527.32ms; fastest: 0.10ms
=== Storage ===
Free file entries: 20
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 11.5ms, write time 122.4ms, max retries 0
=== Move ===
DMs created 83, segments created 34, maxWait 4268975ms, bed compensation in use: mesh, height map offset 0.000, max steps late 1, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 913360, completed 913360, hiccups 0, stepErrors 0, LaErrors 0, Underruns [438, 0, 8], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is ready with "M122" in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000807
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 1964276, received 2196723, lost 0, errs 0, boc 0
Longest wait 7ms for reply type 6029, peak Tx sync delay 441, free buffers 26 (min 24), ts 533967/533966/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 522.93ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 0 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1beta6
MAC address f0:08:d1:02:e6:75
Module reset reason: Power up, Vcc 3.40, flash size 2097152, free heap 33592
WiFi IP address 192.168.1.69
Signal strength -64dBm, channel 6, mode 802.11n, reconnections 1
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

(The IP is correct, the MAC has a reservation in my DHCP server for that IP)

More to come soon....

Cheers, Chriss

Chriss

@rechrtb said in Wifi 2.1beta6 from 3.5.0-rc.2/3 still disconnecting:

I'm not 100% sure, so please do not nail me down to it. But I think that it is 3.5.0RC2 I had before on it. But I saw that behaviour with the reconnects in 3.4 something already. It never bother me because the printer did always reconnect.

That's ok. So you suspect it was 3.5.0RC2 previously, do you happen to remember also the WiFi firmware version?

I suspect that it was the version which comes with 3.5.0RC2. I'm a lazy dude here. I download all files from the releases page and upload all of them to the printer. I have the confidence that the board does picks the correct files.

My two other printers and my "on my desk test board" are on 3.4.6. And I have found a other printer which I did not upgraded to RC2, it is still 3.5.0rc2 and this one does not have the problem. And this is very much in use at the moment.

In the second sentence, do you mean to say "not upgraded to RC3, it is still 3.5.0rc2" or "not upgraded to RC2, it is still 3.5.0rc1 (or maybe 3.4.x)". Also, can you also confirm the WiFi firmware version of these boards?

Just to be clear here about the status and for unblocking my confusion:
1: RC3 with that problem, with PanelDue and printing (My Voron 2.4)
2: RC3 without problem, without PenalDue and not printing (VCast)
3: RC2 without problem, without PenalDue and printing (V0)
4: 3.4.7 without problem, without PenalDue and not printing (Workbench, VCore, Creality)

Can you also confirm the wifi firmware version for these boards?

I just saw a typo... sorry, my bad! No4 is not 3.4.7 they are on 3.4.6, is think that there is no 3.4.7 as far as I know. Sorry, typo.

The RC[2|3] have the versions from the releases in github. So RC2 and RC3 are on 2.1beta6.

You'll notice that I'm constantly asking for the WiFi firmware version, since flashing a certain RRF version does not guarantee a certain WiFi firmware version. That is, you might have upgraded the RRF to 3.5.0rc2 or rc3, but the wifi firmware version can still be on an older version (or the opposite can also be true - you might have new WiFi firmware + old RRF).

Got you point. Well, the situation is a bit strange, isn't it? We see rc RC2 board with 2.1beta6 working, but the RC3 board is failing. At least the one who was printing.
Correct me if I'm wrong but the DuetFW seams to work fine. The printer was printing, the PanelDue worked so that is cool. We see from my last post that the Formware sees a connected Wifi module etc, so the WiFi module reported (or is reporting) the current connection status.

My Access told me that the WiFi module is connected. But the module itself does not react on any IP traffic anymore.

Is there any M-Command to ping a ip from the Mini5+? (Like M4711 T192.168.1.1) I have a feature request if not.

If would be nice to see if it would help send some traffic from the WiFi to module too the AP and the wired network behind the AP.

I will walk the dogs for about an hour and will restart the disconnecting printer than. I hope that I bring him back to the failing mode than. Let me know if you need more info from the board in its current (disconnected) state.

Cheers, Chriss

gloomyandy

@moth4017 It seems like your problem is different to that being seen by @Chriss and he is also running Duet hardware, so perhaps we should continue looking into your problem on another thread. I've created one here: https://forum.duet3d.com/topic/34897/wifi-disconnects-but-connects-after-refresh

rechrtb

@Chriss Ok, I may finally have hit the same issue you're experiencing after running my setup for a few days.

The logs you would have should tell if it's the same issue. Please turn on debug outputs for network and wifi via M111 P1 S1 and M111 P14 S1 and also messages logging M929 P"eventlog.txt" S3.
Keep the serial terminal open and also log serial terminal to a file - I think most serial terminals support this while waiting for disconnect.

Once in the disconnected state:

See wifi module if LED is on.
Send M122 command and take note of the output (if the serial terminal is logging to file, the output should be in the file automatically).
Verify from access point if module is connected (the reason I ask is that you mentioned previously that the access point reports the module is connected, however in my case it's not - I was wondering if you perhaps mistook the DHCP lease list as the connected stations list?).

After the above, you can try reviving the module with M552 S-1 and M552 S1. Please post the serial terminal log and the eventlog.txt here afterwards.

Chriss

@rechrtb Thanks god, I'm not the only one! I was a bit scared that it is a race condition with my home AP.

Anyway, I executed the commands you mentioned. I was about to perform 3 "tiny" prints in a row since I have the usb cable. I need to leave now, more updates tomorrow.

Chriss

@rechrtb said in Wifi 2.1beta6 from 3.5.0-rc.2/3 still disconnecting:

I was wondering if you perhaps mistook the DHCP lease list as the connected stations list?).

Nope that is not the case, the DHCP server and my AP are two different devices.

(Still waiting for the disconnect)

Cheers, Chriss

Chriss

Hi @gloomyandy Hi @rechrtb

The board was again in the failed state when I came back to my office. The printer was not printing and not pingable anymore. So I went to the terminal session and did the requested "M122", and here is the outcome:

That is strange, I was able to send the M122 to the board yesterday when it was in the failed state before I connected the USB. Could the power via the USB have any influence here?

The board is still not reachable via the network. I have the feeling that the reset did not perform a "reset" on the wifi module. That would match my obeservations from the past. I had to power cycle the printer to bring it back online.

Cheers, Chriss

Chriss

Here are the requested log files. I was unable to upload them as txt or as bz2. So here is a tar file which I names .bin. Extract the tar file and you will find a bz2 which is the output from the serial since yesterday. And the eventlog.txt from yesterday till the it failed.

logs.bin

The :
M552 s-1
M552 s1

Brought back the Wifi access if this is what you want to read.

And one more word about my console logfile. You will see there at the end that the board did the reset when I did the M122. The 2nd M122 was successfully. (See my other post)

I hope that this help you guys to find the issue.

Do you want me to downgrade the board to RC2 and the older WiFI firmware now? Or Should I keep RC3 and downgrade WiFi only?

I will be off for 6 days from Friday onwards and can not test anything during this period.

Cheers, Chriss

droftarts

@Chriss From your log file, you sent M112 (which is emergency stop), not M122! After M112, you need to send M999 to reset. Did you notice if the WiFi LED was still on?

I changed the file ending of the log file from .bin to .txt to look at it. Usually the log files are in text format - did you change the ending to .bin? There's a lot of garbage text at the end of the log file. I don't know what that is, or if it is useful; one for @rechrtb to have a look at.

Ian

Chriss

@droftarts

You are right.... I did a 2nd typo... bloody hell... I think that we need to do the drill again, don't we?

I changed the file ending of the log file from .bin to .txt to look at it.

Not sure what you mean, the file I uploaded is a tar file which I renamed to .bin. There is one bz2 file in it (the logfile from the serial) and a .txt which is the enventlog from the sd card. You should not need to open any .bin file.

Just to be on the save site here: You call the "LED ESP" on the WiFi module "WiFi LED", don't you? Yes, that one was still on.

Cheers, Chriss

droftarts

@Chriss Sorry, didn't read your instructions about it being a tar file. I can read them now! Can't see anything immediately obvious, but hopefully there's something there for @rechrtb

Yes, by "WiFi LED" I mean the green (when on) LED next to the WiFi module, labelled "ESP" on the board silkscreen. The WiFi module itself doesn't have an LED on it.

Ian

Chriss

@droftarts

Good that it is clear now... I did not see any unexpected too. The module looks good, the AP thinks that there is a connection with the module etc. The only thing is that the module does not react anymore.

Thanks for the confirmation, we spoke about the same LED than. And it was on.. I was a bit confused by the wiring diagram it looks slightly as the LED is on the module but it is next to it and there is "ESP" printed next to it.

Please let me I will reboot the board now and will wait for the next disconnect to do the same drill, this time without the typo, hopefully. Let me know if I can stop that drill if you think that you have enough information.

Btw: The disconnect happened while the printer was not printing. Just fyi

Cheers, Chriss

Chriss

@droftarts @gloomyandy @rechrtb

Good morning... The printer is again in the failed state.... Before I forget it: The WiFi LED is blinking

It was off after "M552 s-1" (not a surprise) and turned back on after "M552 s1" which is not a surprise too. My apologies, I was not precise enough in the past about the LED, I wrote about "on or off" only. Bot about the blinking.

Do you guys want to have the log files again? (I can tell you so far that they do not look different.)

Please let me know, I will be off for 6 days from tomorrow early morning.

Cheers, Chriss

droftarts

@Chriss Yes, please post the logs, as this is different from what I have experienced (WiFi LED stays lit) and what others have experienced (lots of 'ResponseBusy' messages in the serial console). So any information would be useful. Hopefully you did a M122 from the serial console while it was in the failed state?

Ian