AJAX timeout error during a print

dc42

We'd love to use an external antenna, and we produced a few pre-production boards with them. There are two issues that prevent us from making it standard:

1. The ESP8266 versions supporting external antenna are not FCC certified. I have read that it is impossible to get a wifi module with external antenna certified, you have to get the entire piece of equipment certified instead.

2. The version of the ESP8266 with external antenna is the ESP-08, but that has only 512kbytes or 1Mbyte flash memory, compared to the 4Mbyte flash on the ESP12F that we use. Currently we could manage with 1Mbyte, but it would limit the possibilities for firmware expansion. For the prototypes we changed the flash chip on the ESP-08, but this is not practical to do in production.

T3P3Tony

We might be able to supply a boards with no esp mounted from the next production run… If you then choose to mount an esp7 or 8 then that's fine. Changing the flash chip is fiddly but doable if you have a rework station.

W3DRK

I think it would be neat if the ESP's could be setup to run the web interface in a manor similar to OctoPrint so it could be used on various other controllers over a serial connection…

dc42

@W3DRK:

I think it would be neat if the ESP's could be setup to run the web interface in a manor similar to OctoPrint so it could be used on various other controllers over a serial connection…

It's already been done and there is a thread on the reprap.org forums about how to do it. But the file upload speed to SD card is very slow. I put a lot of effort into the Duet WiFi design and firmware into getting a good file upload speed, because I know that it matters to users.

roboduet

Nightmare. Wireshark shows me a lot of retransmissions. Duet WiFi beta board is in about 1 meter from Mikrotik router. Mikrotik shows from -27 to -31 dBm signal strength, everything else with WiFi (TV, two phones, tablet, notebook) is working very good without any troubles.

Firmware Name: RepRapFirmware for Duet WiFi
Firmware Version: 1.15e (2016-10-02)
WiFi Server Version: 1.02
Web Interface Version: 1.12

10 ajax retries in settings.

https://postimg.org/image/i9shy26hf/

bot

Are you using that computer for anything? In my experience, albeit on a Duet 0.8.5, the web interface can disconnect/ajax error out if the computer it is running on has its cores maxed out (like when rendering, for example). Is anything like this possibly the case?

roboduet

@bot - no, computer's CPU isn't overloaded. I'm using Intel Core 2 Quad Q6660 @2.4 GHz, Win10, FF 49.

I had never seen before so many AJAX timeouts, connection was stable and connection speed was acceptable. Nothing changed at hardware side (router and printer stay in the same positions, no wiring changed, no new metal things lay near and so on).

I think (but I'm not sure) that this is due to interference in the WiFi radio link multiplied to handling of TCP retransmits and other TCP-specific things on the ESP8266 side.

bot

Are you doing ANYTHING with the computer, though? The computer has the ability to cause the ajax errors, not just the duet or the router.

Also, a Q6660 is nearly ten years old. How much ram do you have in that PC?

roboduet

@bot - no, nothing else CPU-hungry is running both in background and foreground. 8 GB RAM is installed and quad-core Q6600 is still fast enough to show DWC page in Firefox.

It seems that this timeouts appear during the day, not at night. Now (00:38 here) DWC running for about ~20 minutes without any issues.

I'm C++ programmer with 20+ years of experience, also with 18+ years of knowledge of network programming, TCP sockets, HTTP protocol and so on. I can post raw HTTP request using telnet with hidden eyes I'm able to distinguish browser timeouts caused by high CPU load and other errors. I'm quite comfortable in Wireshark and I saw TCP retransmits and other errors in logged traffic.

I suspect that this AJAX timeouts for me caused by high WiFi network load in my area AND incorrect handling of lost TCP packets and other TCP transmit errors in sockets library from ESP8266 side. But I have to thoroughly understand the problem before I'll be something to argue.

roboduet

It seems to me that "Maximum number of AJAX retries:" isn't working as expected for me (retry ajax requests N times on any failure) or I'm not using this right way.

I don't know why this happens but in 50%/50% issues after receiving empty response DWC shows AJAX error or just doing nothing and after this stops any further attempts to connect.

Sometimes answer from ESP8266 is really empty (ESP8266 just closes connection after receiving GET request body), sometimes ESP8266 sends correct answer but this answer appears as empty in Firefox browser. Both types of errors causes stop of any further attempts to connect.

I've never seen repeated attempts to make the request in spite of the this setting.

dc42

I think the retry count may only apply to the case where no response is received during the timeout period. I'll alert chrishamm to this thread.

chrishamm

Can you confirm that the same AJAX request isn't sent multiple times before an error is reported? You can use the developer tools in your browser (press F12) and go to Network/Traffic to see all the HTTP requests being sent.

Also be aware that the timeout for each request becomes smaller if you increase the number of AJAX retries. AFAIR the Duet WiFi has an HTTP session timeout of 20 seconds, so if you have configured 9 AJAX retries, the web interface will reduce the timeout of each request to 20/(n+1) = 2 seconds.

roboduet

@chrishamm thank you for your response.

Yes, I can confirm that AJAX request isn't sent multiple times (10 AJAX retries in settings) if received response is empty (i.e. without HTTP headers and body).

As I said before, after unsuccessful AJAX request DWC requests (purposes are unknown for me) the main (index) html page (GET / HTTP/1.1) and stops any further requests after this.

Call stack (provided by developer tools) for successful requests:

[[language]]
.send
.ajax
updateStatus

Call stack for index page request:

[[language]]
.send
.ajax
 <anonymous>bv.event.dispatch
bv.event.add/r.handle
.trigger
H
.ajax/d.abort
.ajax/P<</anonymous>

Request time for last unsuccessful AJAX request was 36 milliseconds so I don't think that this is timeout.

roboduet

Is there a chance that this will be fixed? Seems like I'm the only who experiencing this?

chrishamm

@roboduet, I'll look into it but I need some more information. When you start using DWC, could you record a Wireshark dump so I can get an idea what the communication between your Duet and your PC looks like? And when DWC has stopped updating, could you try to reconnect and send me the output of M122?

I figure the whole story is caused by some bad packets at the end of certain transmissions, but I'd like to investigate this once more.

roboduet

FF dev console, empty request:

FF dev console, request index page after previous empty request:

Wireshark:

Last successful request tcp stream number = 56
Empty request tcp stream number = 57
Index page request tcp stream number = 58

Also I noticed long pause between 56 and 57 streams (time column).

57 stream seems like complete (correct) AJAX JSON response.

After 58 stream (after requesting index page) DWC stops any further requests. DWC doesn't show any errors (i.e AJAX or just "disconnected") in DWC G-Code console tab and still think that it connected to Duet.

I have saved Wireshark dump in pcapng format, I can upload file if this can help.

roboduet

Sorry, forgot to attach M122 logs after reconnect:

[[language]]
M122
=== Diagnostics ===
Used output buffers: 2 of 32 (8 max)
=== Platform ===
Memory usage:
Program static ram used: 19284
Dynamic ram used: 83012
Recycled dynamic ram: 104
Current stack ram used: 2768
Maximum stack ram used: 3868
Never used ram: 24804
Last reset 01:58:35 ago, cause: power up
Error status: 0
Bed probe heights: 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest block write time: 0.0ms
MCU temperature: min 31.6, current 33.3, max 36.7
Supply voltage: min 12.1, current 12.1, max 12.2, under voltage events: 0, over voltage events: 0
Slowest main loop (seconds): 0.004883; fastest: 0.000000
=== Move ===
MaxReps: 0, StepErrors: 0, MaxWait: 0ms, Underruns: 0, 0
=== Heat ===
Bed heater = 0, chamber heater = -1
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Move available? no
Stack pointer: 0 of 5
macro is idle
http is ready with "M122"
telnet is idle
serial is idle
aux is idle
file is idle
=== Network ===
WiFiServer is running
SPI underruns 0, overruns 0
=== Webserver ===
HTTP sessions: 1 of 8

chrishamm

Hmm, just as I suspected. It looks like the WiFi module actually sends a non-empty response, but Firefox never receives and processes it. I'll look into the WiFi Server firmware later. Could you send me the Wireshark pcap file though?

roboduet

Yes, as I said before. For some unknown for me reasons Duet sends valid response, but Firefox "receives" empty response. This is freeze/hangup bug. But also sometimes DWC shows me AJAX error and properly shows disconnected state if it receives invalid or empty response. I don't yet know why sometimes DWC is just freezes and why sometimes it shows proper AJAX error.

I still think about big pause between last valid request and next empty request. Can it be that AJAX request timeout is interpreted as empty response?

I will ask your email in PM to send logs.

roboduet

Two hours without disconnects or connection freeze using latest 1.16 release firmware with @chrishamm web-server 1.03-ch firmware. Still looking at error logs, but it seems that this release is much stable.

Firmware Name: RepRapFirmware for Duet WiFi
Firmware Electronics: Duet WiFi 1.0
Firmware Version: 1.16 (2016-11-08)
WiFi Server Version: 1.03 (ch fork)
Web Interface Version: 1.13