Wifi 2.1beta6 from 3.5.0-rc.2/3 still disconnecting

Chriss

@rechrtb said in Wifi 2.1beta6 from 3.5.0-rc.2/3 still disconnecting:

I'm not 100% sure, so please do not nail me down to it. But I think that it is 3.5.0RC2 I had before on it. But I saw that behaviour with the reconnects in 3.4 something already. It never bother me because the printer did always reconnect.

That's ok. So you suspect it was 3.5.0RC2 previously, do you happen to remember also the WiFi firmware version?

I suspect that it was the version which comes with 3.5.0RC2. I'm a lazy dude here. I download all files from the releases page and upload all of them to the printer. I have the confidence that the board does picks the correct files.

My two other printers and my "on my desk test board" are on 3.4.6. And I have found a other printer which I did not upgraded to RC2, it is still 3.5.0rc2 and this one does not have the problem. And this is very much in use at the moment.

In the second sentence, do you mean to say "not upgraded to RC3, it is still 3.5.0rc2" or "not upgraded to RC2, it is still 3.5.0rc1 (or maybe 3.4.x)". Also, can you also confirm the WiFi firmware version of these boards?

Just to be clear here about the status and for unblocking my confusion:
1: RC3 with that problem, with PanelDue and printing (My Voron 2.4)
2: RC3 without problem, without PenalDue and not printing (VCast)
3: RC2 without problem, without PenalDue and printing (V0)
4: 3.4.7 without problem, without PenalDue and not printing (Workbench, VCore, Creality)

Can you also confirm the wifi firmware version for these boards?

I just saw a typo... sorry, my bad! No4 is not 3.4.7 they are on 3.4.6, is think that there is no 3.4.7 as far as I know. Sorry, typo.

The RC[2|3] have the versions from the releases in github. So RC2 and RC3 are on 2.1beta6.

You'll notice that I'm constantly asking for the WiFi firmware version, since flashing a certain RRF version does not guarantee a certain WiFi firmware version. That is, you might have upgraded the RRF to 3.5.0rc2 or rc3, but the wifi firmware version can still be on an older version (or the opposite can also be true - you might have new WiFi firmware + old RRF).

Got you point. Well, the situation is a bit strange, isn't it? We see rc RC2 board with 2.1beta6 working, but the RC3 board is failing. At least the one who was printing.
Correct me if I'm wrong but the DuetFW seams to work fine. The printer was printing, the PanelDue worked so that is cool. We see from my last post that the Formware sees a connected Wifi module etc, so the WiFi module reported (or is reporting) the current connection status.

My Access told me that the WiFi module is connected. But the module itself does not react on any IP traffic anymore.

Is there any M-Command to ping a ip from the Mini5+? (Like M4711 T192.168.1.1) I have a feature request if not.

If would be nice to see if it would help send some traffic from the WiFi to module too the AP and the wired network behind the AP.

I will walk the dogs for about an hour and will restart the disconnecting printer than. I hope that I bring him back to the failing mode than. Let me know if you need more info from the board in its current (disconnected) state.

Cheers, Chriss

gloomyandy

@moth4017 It seems like your problem is different to that being seen by @Chriss and he is also running Duet hardware, so perhaps we should continue looking into your problem on another thread. I've created one here: https://forum.duet3d.com/topic/34897/wifi-disconnects-but-connects-after-refresh

rechrtb

@Chriss Ok, I may finally have hit the same issue you're experiencing after running my setup for a few days.

The logs you would have should tell if it's the same issue. Please turn on debug outputs for network and wifi via M111 P1 S1 and M111 P14 S1 and also messages logging M929 P"eventlog.txt" S3.
Keep the serial terminal open and also log serial terminal to a file - I think most serial terminals support this while waiting for disconnect.

Once in the disconnected state:

See wifi module if LED is on.
Send M122 command and take note of the output (if the serial terminal is logging to file, the output should be in the file automatically).
Verify from access point if module is connected (the reason I ask is that you mentioned previously that the access point reports the module is connected, however in my case it's not - I was wondering if you perhaps mistook the DHCP lease list as the connected stations list?).

After the above, you can try reviving the module with M552 S-1 and M552 S1. Please post the serial terminal log and the eventlog.txt here afterwards.

Chriss

@rechrtb Thanks god, I'm not the only one! I was a bit scared that it is a race condition with my home AP.

Anyway, I executed the commands you mentioned. I was about to perform 3 "tiny" prints in a row since I have the usb cable. I need to leave now, more updates tomorrow.

Chriss

@rechrtb said in Wifi 2.1beta6 from 3.5.0-rc.2/3 still disconnecting:

I was wondering if you perhaps mistook the DHCP lease list as the connected stations list?).

Nope that is not the case, the DHCP server and my AP are two different devices.

(Still waiting for the disconnect)

Cheers, Chriss

Chriss

Hi @gloomyandy Hi @rechrtb

The board was again in the failed state when I came back to my office. The printer was not printing and not pingable anymore. So I went to the terminal session and did the requested "M122", and here is the outcome:

That is strange, I was able to send the M122 to the board yesterday when it was in the failed state before I connected the USB. Could the power via the USB have any influence here?

The board is still not reachable via the network. I have the feeling that the reset did not perform a "reset" on the wifi module. That would match my obeservations from the past. I had to power cycle the printer to bring it back online.

Cheers, Chriss

Chriss

Here are the requested log files. I was unable to upload them as txt or as bz2. So here is a tar file which I names .bin. Extract the tar file and you will find a bz2 which is the output from the serial since yesterday. And the eventlog.txt from yesterday till the it failed.

logs.bin

The :
M552 s-1
M552 s1

Brought back the Wifi access if this is what you want to read.

And one more word about my console logfile. You will see there at the end that the board did the reset when I did the M122. The 2nd M122 was successfully. (See my other post)

I hope that this help you guys to find the issue.

Do you want me to downgrade the board to RC2 and the older WiFI firmware now? Or Should I keep RC3 and downgrade WiFi only?

I will be off for 6 days from Friday onwards and can not test anything during this period.

Cheers, Chriss

droftarts

@Chriss From your log file, you sent M112 (which is emergency stop), not M122! After M112, you need to send M999 to reset. Did you notice if the WiFi LED was still on?

I changed the file ending of the log file from .bin to .txt to look at it. Usually the log files are in text format - did you change the ending to .bin? There's a lot of garbage text at the end of the log file. I don't know what that is, or if it is useful; one for @rechrtb to have a look at.

Ian

Chriss

@droftarts

You are right.... I did a 2nd typo... bloody hell... I think that we need to do the drill again, don't we?

I changed the file ending of the log file from .bin to .txt to look at it.

Not sure what you mean, the file I uploaded is a tar file which I renamed to .bin. There is one bz2 file in it (the logfile from the serial) and a .txt which is the enventlog from the sd card. You should not need to open any .bin file.

Just to be on the save site here: You call the "LED ESP" on the WiFi module "WiFi LED", don't you? Yes, that one was still on.

Cheers, Chriss

droftarts

@Chriss Sorry, didn't read your instructions about it being a tar file. I can read them now! Can't see anything immediately obvious, but hopefully there's something there for @rechrtb

Yes, by "WiFi LED" I mean the green (when on) LED next to the WiFi module, labelled "ESP" on the board silkscreen. The WiFi module itself doesn't have an LED on it.

Ian

Chriss

@droftarts

Good that it is clear now... I did not see any unexpected too. The module looks good, the AP thinks that there is a connection with the module etc. The only thing is that the module does not react anymore.

Thanks for the confirmation, we spoke about the same LED than. And it was on.. I was a bit confused by the wiring diagram it looks slightly as the LED is on the module but it is next to it and there is "ESP" printed next to it.

Please let me I will reboot the board now and will wait for the next disconnect to do the same drill, this time without the typo, hopefully. Let me know if I can stop that drill if you think that you have enough information.

Btw: The disconnect happened while the printer was not printing. Just fyi

Cheers, Chriss

Chriss

@droftarts @gloomyandy @rechrtb

Good morning... The printer is again in the failed state.... Before I forget it: The WiFi LED is blinking

It was off after "M552 s-1" (not a surprise) and turned back on after "M552 s1" which is not a surprise too. My apologies, I was not precise enough in the past about the LED, I wrote about "on or off" only. Bot about the blinking.

Do you guys want to have the log files again? (I can tell you so far that they do not look different.)

Please let me know, I will be off for 6 days from tomorrow early morning.

Cheers, Chriss

droftarts

@Chriss Yes, please post the logs, as this is different from what I have experienced (WiFi LED stays lit) and what others have experienced (lots of 'ResponseBusy' messages in the serial console). So any information would be useful. Hopefully you did a M122 from the serial console while it was in the failed state?

Ian

Chriss

@droftarts

You will find the file attached, same drill as last time.
no3.bin

I'm not so convinced anymore that the WiFi LED was permanently on last time. It could be that it was blinking... My apologies but is was early in the morning before my first cup of tee and I wrote the message a bit later.

And one more thing: I have not seen that before... I the M552 drill, and the board is pingable since than. But I did not tried to reach a higher layer. I tried to download the enventlog for you the FTP server did not reacted really:

The port is obviously open but the application seams not to react anymore. Telnet has very much the same behaviour:

And the webui is very much the same, I can curl it, I get an connection but the daemon behind the port is not responding. I had to cut the power to solve that.

I have here a other mini 5+ on my work bench which does literally nothing. There is the stepper extension board attached but no sensor, no stepper. What do you think? Would it make sense to push RC3 to that board and copy the config, just to see whether the problem will pup up there too? Just to remove the dependency from my physical printer.
And I may be able to connect that board to a other wifi to remove that variable too.

Or we test with a very simple config file to strike that out. But I think that I should do all of that test with my test rick than with my main printer.

Cheers, Chriss

droftarts

@Chriss I haven't seen your config.g in a while, perhaps you can post that too? I assume FTP and telnet are turned on in that? That would use up a couple more sockets, though they are supposed to be released when not in use, for other connections. I'll pass on all the info to @rechrtb for him to look at.

Ian

Chriss

@droftarts Sure, the services are enabled, you will get a "connection refused" if not. And they work after the reboot. I wanted to indicate that it seems that the WiFi modules get stuck when this failed state happens. And the "turn in off and on again" drill did not brought it back to normal operations. It is better (back bingable) but not fully working. I have seen that behaviour often on Un*x aor Linux systems when the OS had not enough resources for the demon behind to port. So the TCP handshake worked but the process behind it was unable to respond.

Here is the current config.g. (Kindly ignore the typos, I'm German)

; Hardware: Duet Mini 5+ 
;           Toolboard 1.1 LC
; Stepper XY = LDO 0,9° 2Amax  LDO-42STH40-2004MAC
; Stepper Z = LDO 1,8°  2Amax  LDO-42STH48-2004AC
; Stepper E = LDO 1,8   1Amax  LDO-42STH20-1004ASH


; Enable network
if {network.interfaces[0].type = "ethernet"}
    M552 P0.0.0.0 S1
else
    M552 S1

; Network
M586 P0 S1  ; enable HTTP
M586 P1 S1  ; enable FTP
M586 P2 S1  ; enable Telnet

G90         ; send absolute coordinates...
M83         ; ...but relative extruder moves
M550 P"v2"  ; set printer name
M669 K1     ; 1=select CoreXY mode   0=Cadasian

;; Helpful Toolboards commands
; M115 B121  ; Show board 121
; M997 B121  ; Update tool 121
; M122 B121  ; Detailed status of toolboard

G4 S1        ; wait 1s for expansion boards to start

;;; Drives
;X
M569 P0.2 S1 D3         ; physical drive 0.2 goes forward
M584 X0.2               ; Map the stepper to X
;Y
M569 P0.1 S0 D3         ; physical drive 0.1 goes backward
M584 Y0.1               ; Map the stepper to Y

;; Z
; - front left
M569 P0.5 S1 D3          ; physical drive 0.5 goes forward
; - front right
M569 P0.6 S0 D3         ; physical drive 0.6 goes backward
; - back right
M569 P0.0 S1 D3         ; physical drive 0.0 goes forward
; - back left
M569 P0.4 S0 D3         ; physical drive 0.4 goes backward
M584 Z0.5:0.4:0.0:0.6   ; Mapping

;; E
M569 P121.0 S0 D3       ; Extruder stepper goes backward
M584 E121.0             ; Map the E stepper to E

; Stepper settings
M350 X16 Y16 Z16 E32 I1                         ; configure microstepping with interpolation
M92 X160 Y160 Z400 E823							; set steps per mm (800 from manuall, measured 823
M98 P"/macros/print_scripts/speed_printing.g"   ; Accelerations and speed
M906 X1400 Y1400 Z1000 E700 I30                 ; set motor currents (mA) and motor idle factor in per cent  (E stepper max 1A)
M84 S120                                        ; Idle timeout

; Axis Limits
M208 X0 Y0 Z0 S1                                ; set axis minima
M208 X250 Y258 Z210 S0                          ; set axis maxima

;; Endstops -- Display status with: M119
M574 Y2 S1 P"0.io5.in"  ; Y
M574 X2 S1 P"!0.io6.in" ; X
M574 Z0 P"nil"          ; No endstop we have the switch and a probe
M574 Z1 S2              ; configure Z-probe endstop for low end on Z
; Z probe
M98 P"/macros/print_scripts/activate_z_probe.g"

; Z-level settings
;M671 X-75:-75:288:289 Y0:320:320:0 S20 ; Define Z belts locations (Front_Left, Back_Left, Back_Right, Front_Right)
;M671 X-75:-75:288:289 Y0:328:328:0 S20 ; Define Z belts locations (Front_Left, Back_Left, Back_Right, Front_Right)
M671 X-75:-75:288:289 Y0:358:358:0 S20  ; Define Z belts locations (Front_Left, Back_Left, Back_Right, Front_Right)

;; Define the mesh
;M557 X5:245 Y22:245 S35    ; spacing
;M557 X5:245 Y22:245 P9     ; grid (points per axis)
M557 X5:245 Y22:220 P9      ; grid (points per axis)


;; Heaters :: Tune with: M303 H0 S110
; Bed
M308 S0 P"0.temp0" Y"thermistor" A"Bed" T100000 B4138	; configure sensor 0 as thermistor on pin temp0
M950 H0 C"out5+out6" T0 Q10            			        ; create bed heater outputs for both SSRs on out0 and map it to sensor 0
M307 H0 B0 S1.00                                		; disable bang-bang mode for the bed heater and set PWM limit
M140 H0                                         		; map heated bed to heater 0
M143 H0 S120                                    		; set temperature limit for heater 0 to 120C

;; Bed Corner temp sensor (2=Orange, 3=Brown, 4=Green, 5=Yellow, 6=Purple 7=Black, )
; Configure Bed corner temp sensor as thermistor on pin temp2
M308 S5 P"0.temp2" Y"thermistor" A"Bed-Corner" T100000 B4138


; Hotend
; Tune in with:  M303 H1 S270       (270=Temp) (M500 to save) 
; Show current settings M307 H1
;M308 S1 P"121.temp0" Y"thermistor" A"Hotend" T500000 B4702 C1.171057e-7	; configure sensor 1 as thermistor on pin temp1 Mosquito
;M308 S1 P"121.temp0" A"Hotend" Y"thermistor" T100000 B4725 C7.06e-8        ; define E0 temperature sensor Rapido Argo
M308 S1 P"121.temp0" A"Hotend" Y"thermistor" T100000 B4725 C7.060000e-8     ; define E0 temperature sensor e3d revo
M950 H1 C"121.out0" T1                      		            	    ; create nozzle heater output on 0.out3 and map it to sensor 1
M143 H1 S300                                        			        ; set temperature limit for heater 1 to 300C


;; Fans
; Fan for the printed part:
M950 F0 C"121.out1" Q500        ; create fan 0 on pin 0.out9 and set its frequency
M106 P0 S0 H-1 C"Part"          ; set fan 0 value. Thermostatic control is turned off

; Fan for the Hotend:
M950 F1 C"121.out2" Q500        ; create fan 1 on pin 0.out9 and set its frequency
M106 P1 S1 H1 T45 C"Hotend"     ; P="set fan 1" S="value" H="Thermostatic control Heater No." T=" is turned on at 45°C"


;; Tool
M563 P0 S"Tool" D0 H1 F0    ; define tool
G10 P0 X0 Y0 Z0             ; set tool 0 axis offsets
G10 P0 R0 S0                ; set initial tool 0 active and standby temperatures to 0C

; Filament sensor : Status  M591 D0
;M591 D0 P7 C"io4.in" L7 R50:150 E5 S0 ;pulse, disabled, 7 mm/pulse, measure every 22 sec, minimum 50 maximum 250, S1 = Enabled S0 = Disabled
;M591 D0 P1 C"io4.in" S1
M950 J3 C"!io4.in" ; Create a trigger on io4.in (NC)
M581 P3 T3 S0 R1   ; R1=Trigger only while printing

;; Chamber temp sensor
M308 S4 P"0.temp1" Y"thermistor" A"Chamber" T100000 B4138   ; configure Chamber temp sensor as thermistor on pin temp1

;; Input Shaping
; Accelerometer https://duet3d.dozuki.com/Wiki/Input_shaping
M955 P121.0 I05 ; specify orientation of accelerometer on Toolboard 1LC with CAN address 121
; Input Shaping
;M593 P"zvd" F40.5  ; use ZVD input shaping to cancel ringing at 40.5Hz
;M593 P"none"        ; disable input shaping
;M593 P"custom" H0.4:0.7 T0.0135:0.0135 ; use custom input shaping

; PA https://duet3d.dozuki.com/Wiki/Pressure_advance
M572 D0 S0.025


;;;;;;;;;;;; Setup Only
;M564 S0 H0             ; Allow movement over the endstops
;M302 P1 ; allow cold extrusion
;M302 S1 ; deny cold extrusion
;;;;;;;;;;;; Setup Only END

;; Case Cooling
; Temps
M308 S9 P"mcu-temp" Y"mcu-temp" A"Mainboard"  ; define sensor 9 to be mcu temperature

; Case Fans
M950 F3 C"!0.out3" Q50  ; Fan on out3 ground on top pin, plus on 3rd pin from top (V_OUTLC1)
M106 P3 C"Base" S120    ; Setup the FAN and slow it down

; Nevermore
m950 F4 C"0.out0" Q50
m106 P4 C"Nevermore" S0


; Define the LED stripe and turn it off
M950 F5 C"0.out1" Q100  ; LED on out1 
M106 P5 C"LED" S0       ; Make sure that the LEDs are off

; Trigger on the toolboard
;#M950 J5 C"^121.button0"
;#M581 P4 T5 S0



;########################################
M950 J1 C"^0.io1.in"
M581 P1 T2 S0


;M572 D0 S0.037    ; Set preasure Advance Gemessen




M501              ; Load config-override.g

;; Serial interface
; Duet
M575 P1 S1 B57600
;;;;;; Old Display
;M575 P1 B115200 S1
;; Mini 12864
;M918 P2
;M918 P2 E4 R3 C100
;M150 X2 R255 U255 B255 S3      ; set all 3 LEDs to white
;M150 X2 R0 U255 B0 S3          ; set all 3 LEDs to red

T0                      ; Select the tool 0 as default

; Make sure that all heaters are off
M104 S0 		; Extruder temp to 0
M568 P0 A0		; Extruder heater off
M140 S0 		; Set the bed temp to 0
M140 S-276		; Bed heater off


; Some variables for later
global tool_temp_initial=0
global bed_temp_initial=0
global debug=false

; AutoZ
global klicky_home=true
global qgl_done=false
global nozzle_cleaned=false
global Zswitch_homed=false
global probetype="euclid"
global clickystatus = "none"
global probe_settingsH=10
global probe_settingsA=1
global autoz_temp2=20


# Stealthburner LEDs:
global sb_leds="n-off"
M98 P"/macros/sb_leds/sb_leds.g"
set global.sb_leds="hot"
set global.sb_leds="n-off"

;set global.sb_logo="red"
;set global.sb_leds="n-off"
;global sb_nozzle="off"

; M307 H0 R0.327 C227.635:227.635 D5.48 S1.00 V24.4 B0 I0 ; R altered for a firmware bug

; EOF[chriss@leela sys]$

rechrtb

@Chriss Hello, I might have a fix for this issue. But as you can guess, this issue seems to be intermittent and highly network dependent. So I'd like your help in order to verify it really works.

Are you able to setup two boards:

One board has 2.1beta6 and 3.5rc3 from the release https://github.com/Duet3D/RepRapFirmware/releases/tag/3.5.0-rc.3
The other board also has 3.5rc3 from the release, but has experimental wifi server firmware: https://drive.google.com/file/d/1NgssWNSS3xGL99hWwfgYbY5YVXF-f4jH/view?usp=drive_link.

Please verify the versions are correct for each board; for the first board it should say "2.1beta6" and on the second one it should be "2.1beta7".

The idea is simple - to run and use these board normally and see if the 2.1beta7 board has none of these disconnections you previously encountered, compared to the 2.1beta6 one.

Sorry by the way for the delay on this issue. I was sick for the last few days (from last week) and was only able to resume work yesterday.

Chriss

@rechrtb

Glad to hear that you are recovered from your sick leave. I was on vacation in the meantime so I was not "waiting" for a reply.

Please give me some days to test with the new WiFi firmware. Do you remember that I had a other printer with RC3?:
2: RC3 without problem, without PenalDue and not printing (VCast)

That one is printing since yesterday and has developed the same problem since this morning. I think that I will upgrade this printer to beta7 first. I will use that printer more frequently in the next days, please let me know if you want me to stick on the other printer which had the problem first.
I have to admit that I'm more than happy that both printers with beta6 have the same problem now, I was a bit concerned about the observation that one encountered the problem while the other one was fine.

Cheers, Chriss

Chriss

@rechrtb I have no access to the file. I requested it a minute ago.

rechrtb

@Chriss Granted you access to the file. Tell me if you still have problems accessing it.

Regarding your question, I would advise to put beta7 on the board on which the issue seems to manifest most often.

I would also advise putting the two boards near each other if you can, so that they roughly get the same wifi signal strength, same wifi devices in proximity, etc. I recommend moving the beta6 board to the beta7 board location (again, because this might be a 'goldilocks' location w/ respect to the access point for the issue to manifest more frequently).