AJAX timeout error during a print

bot

Are you using that computer for anything? In my experience, albeit on a Duet 0.8.5, the web interface can disconnect/ajax error out if the computer it is running on has its cores maxed out (like when rendering, for example). Is anything like this possibly the case?

roboduet

@bot - no, computer's CPU isn't overloaded. I'm using Intel Core 2 Quad Q6660 @2.4 GHz, Win10, FF 49.

I had never seen before so many AJAX timeouts, connection was stable and connection speed was acceptable. Nothing changed at hardware side (router and printer stay in the same positions, no wiring changed, no new metal things lay near and so on).

I think (but I'm not sure) that this is due to interference in the WiFi radio link multiplied to handling of TCP retransmits and other TCP-specific things on the ESP8266 side.

bot

Are you doing ANYTHING with the computer, though? The computer has the ability to cause the ajax errors, not just the duet or the router.

Also, a Q6660 is nearly ten years old. How much ram do you have in that PC?

roboduet

@bot - no, nothing else CPU-hungry is running both in background and foreground. 8 GB RAM is installed and quad-core Q6600 is still fast enough to show DWC page in Firefox.

It seems that this timeouts appear during the day, not at night. Now (00:38 here) DWC running for about ~20 minutes without any issues.

I'm C++ programmer with 20+ years of experience, also with 18+ years of knowledge of network programming, TCP sockets, HTTP protocol and so on. I can post raw HTTP request using telnet with hidden eyes I'm able to distinguish browser timeouts caused by high CPU load and other errors. I'm quite comfortable in Wireshark and I saw TCP retransmits and other errors in logged traffic.

I suspect that this AJAX timeouts for me caused by high WiFi network load in my area AND incorrect handling of lost TCP packets and other TCP transmit errors in sockets library from ESP8266 side. But I have to thoroughly understand the problem before I'll be something to argue.

roboduet

It seems to me that "Maximum number of AJAX retries:" isn't working as expected for me (retry ajax requests N times on any failure) or I'm not using this right way.

I don't know why this happens but in 50%/50% issues after receiving empty response DWC shows AJAX error or just doing nothing and after this stops any further attempts to connect.

Sometimes answer from ESP8266 is really empty (ESP8266 just closes connection after receiving GET request body), sometimes ESP8266 sends correct answer but this answer appears as empty in Firefox browser. Both types of errors causes stop of any further attempts to connect.

I've never seen repeated attempts to make the request in spite of the this setting.

dc42

I think the retry count may only apply to the case where no response is received during the timeout period. I'll alert chrishamm to this thread.

chrishamm

Can you confirm that the same AJAX request isn't sent multiple times before an error is reported? You can use the developer tools in your browser (press F12) and go to Network/Traffic to see all the HTTP requests being sent.

Also be aware that the timeout for each request becomes smaller if you increase the number of AJAX retries. AFAIR the Duet WiFi has an HTTP session timeout of 20 seconds, so if you have configured 9 AJAX retries, the web interface will reduce the timeout of each request to 20/(n+1) = 2 seconds.

roboduet

@chrishamm thank you for your response.

Yes, I can confirm that AJAX request isn't sent multiple times (10 AJAX retries in settings) if received response is empty (i.e. without HTTP headers and body).

As I said before, after unsuccessful AJAX request DWC requests (purposes are unknown for me) the main (index) html page (GET / HTTP/1.1) and stops any further requests after this.

Call stack (provided by developer tools) for successful requests:

[[language]]
.send
.ajax
updateStatus

Call stack for index page request:

[[language]]
.send
.ajax
 <anonymous>bv.event.dispatch
bv.event.add/r.handle
.trigger
H
.ajax/d.abort
.ajax/P<</anonymous>

Request time for last unsuccessful AJAX request was 36 milliseconds so I don't think that this is timeout.

roboduet

Is there a chance that this will be fixed? Seems like I'm the only who experiencing this?

chrishamm

@roboduet, I'll look into it but I need some more information. When you start using DWC, could you record a Wireshark dump so I can get an idea what the communication between your Duet and your PC looks like? And when DWC has stopped updating, could you try to reconnect and send me the output of M122?

I figure the whole story is caused by some bad packets at the end of certain transmissions, but I'd like to investigate this once more.

roboduet

FF dev console, empty request:

FF dev console, request index page after previous empty request:

Wireshark:

Last successful request tcp stream number = 56
Empty request tcp stream number = 57
Index page request tcp stream number = 58

Also I noticed long pause between 56 and 57 streams (time column).

57 stream seems like complete (correct) AJAX JSON response.

After 58 stream (after requesting index page) DWC stops any further requests. DWC doesn't show any errors (i.e AJAX or just "disconnected") in DWC G-Code console tab and still think that it connected to Duet.

I have saved Wireshark dump in pcapng format, I can upload file if this can help.

roboduet

Sorry, forgot to attach M122 logs after reconnect:

[[language]]
M122
=== Diagnostics ===
Used output buffers: 2 of 32 (8 max)
=== Platform ===
Memory usage:
Program static ram used: 19284
Dynamic ram used: 83012
Recycled dynamic ram: 104
Current stack ram used: 2768
Maximum stack ram used: 3868
Never used ram: 24804
Last reset 01:58:35 ago, cause: power up
Error status: 0
Bed probe heights: 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest block write time: 0.0ms
MCU temperature: min 31.6, current 33.3, max 36.7
Supply voltage: min 12.1, current 12.1, max 12.2, under voltage events: 0, over voltage events: 0
Slowest main loop (seconds): 0.004883; fastest: 0.000000
=== Move ===
MaxReps: 0, StepErrors: 0, MaxWait: 0ms, Underruns: 0, 0
=== Heat ===
Bed heater = 0, chamber heater = -1
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Move available? no
Stack pointer: 0 of 5
macro is idle
http is ready with "M122"
telnet is idle
serial is idle
aux is idle
file is idle
=== Network ===
WiFiServer is running
SPI underruns 0, overruns 0
=== Webserver ===
HTTP sessions: 1 of 8

chrishamm

Hmm, just as I suspected. It looks like the WiFi module actually sends a non-empty response, but Firefox never receives and processes it. I'll look into the WiFi Server firmware later. Could you send me the Wireshark pcap file though?

roboduet

Yes, as I said before. For some unknown for me reasons Duet sends valid response, but Firefox "receives" empty response. This is freeze/hangup bug. But also sometimes DWC shows me AJAX error and properly shows disconnected state if it receives invalid or empty response. I don't yet know why sometimes DWC is just freezes and why sometimes it shows proper AJAX error.

I still think about big pause between last valid request and next empty request. Can it be that AJAX request timeout is interpreted as empty response?

I will ask your email in PM to send logs.

roboduet

Two hours without disconnects or connection freeze using latest 1.16 release firmware with @chrishamm web-server 1.03-ch firmware. Still looking at error logs, but it seems that this release is much stable.

Firmware Name: RepRapFirmware for Duet WiFi
Firmware Electronics: Duet WiFi 1.0
Firmware Version: 1.16 (2016-11-08)
WiFi Server Version: 1.03 (ch fork)
Web Interface Version: 1.13

roboduet

Seems like DWC connection freeze issue fixed. Three hours yesterday and one hour today without any connection freezes. Thank you!

chrishamm

That's great, thanks for confirming that my latest improvements are working well!

timpact

Hi

I'm new here and building 3d printer in my free time.

I just set up a Duet v0.6 with the latest firmware and Web Control:

Firmware Name: RepRapFirmware for Duet-0.6-0.8.5
Firmware Version: 1.16 (2016-11-08)
Web Interface Version: 1.13

After connecting to the duet via Web Control I'm getting the same error like appjaws after ~55 secconds while doing nothing:
Communication Error
An AJAX error has been reported, so the current session has been terminated.
Please check if your printer is still on and try to connect again.
Error reason: timeout

I reestablished the connection several times and got the error message every time after 55 secconds.
Thanks for every hint and help. Just tell me if you need further information.

Edit: I'm using windows 7 pro and the error appears in Firefox and Chrome

chrishamm

I recommend you try to reset your TCP/IP stack once and restart your computer, see https://support.microsoft.com/en-us/kb/299357

timpact

Well that solved the problem fast and perfect. Thank you!