Cannot upload files. (Duet 2 Wifi)

Jenco

@Phaedrux well the largest file i could complete is right around the 1mb mark. Anything larger fails. the file progress progresses to about 35% and resets to 0 a few times before failing... I also tried uploading by ftp i can delete files on the duet from ftp but as soon as i initiate the upload it fails no matter the file size. Im logged in as anon so that could be it.

As for smaller files it takes around a minute to upload but they do succeed thru !DWC!

As for navigation. DWC seems sluggish and I can only connect to it from 1 computer at a time.

Oh once i get the gcode to the SD (as in copying it from my computer with an sd adapter). It prints beautifully.

sko

Having the same problem here. After ~40kB any upload stalls. Smaller files transfer just fine...

M122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.1.1 running on Duet WiFi 1.02 or later
Board ID: 08DLM-996AL-K6PSD-6J1DJ-3SD6L-99GGY
Used output buffers: 3 of 24 (10 max)
=== RTOS ===
Static ram: 27980
Dynamic ram: 93760 of which 56 recycled
Exception stack ram used: 240
Never used ram: 9036
Tasks: NETWORK(ready,384) HEAT(blocked,1320) MAIN(running,1912) IDLE(ready,80)
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:00:40 ago, cause: software
Last software reset at 2020-08-10 18:21, reason: User, spinning module GCodes, available RAM 8992 bytes (slot 1)
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x04433000 BFAR 0xe000ed38 SP 0xffffffff Task MAIN
Error status: 0
MCU temperature: min 37.0, current 37.7, max 37.8
Supply voltage: min 1.7, current 1.7, max 1.8, under voltage events: 0, over voltage events: 0, power good: no
Driver 0: ok, SG min/max not available
Driver 1: ok, SG min/max not available
Driver 2: ok, SG min/max not available
Driver 3: ok, SG min/max not available
Driver 4: ok, SG min/max not available
Date/time: 2020-08-10 18:22:06
Cache data hit count 65784672
Slowest loop: 5.57ms; fastest: 0.10ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 3.4ms, write time 0.0ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 169, MinFreeDm: 169, MaxWait: 0ms
Bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 15.93ms; fastest: 0.00ms
Responder states: HTTP(1) HTTP(2) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- WiFi -
Network state is active
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
WiFi firmware version 1.23
WiFi MAC address d8:bf:c0:fe:92:c1
WiFi Vcc 3.38, reset reason Unknown
WiFi flash size 4194304, free heap 24280
WiFi IP address 172.25.52.135
WiFi signal strength -43dBm, reconnections 0, sleep mode modem
Socket states: 2 4 0 0 0 0 0 0

Tried 2 brand new SD-cards with the same result, and as I can upload smaller files and edit the config through DWC I don't think its the SD card. Is there any max filesize or buffer defined in the Webserver that might interfere with uploads?

Phaedrux

@Jenco @sko

Would you both be able to test by rolling back to firmware 2.05.1 and DWC 2.0.7 and seeing if you can reliably upload?

Your config.g won't work very well with RRF2 but you should still be able to upload a file.

If it helps, we at least have a lead to follow. IF it doesn't help, then it might be a wifi module issue.

sko

I took the board with me to work today to flash the 2.05.1 firmware and here uploads are working (with DWC 3.1.1 !).
Just to verify I re-flashed with firmware 3.1.1 and uploads are still working!
Although upload speeds are a bit disappointing at ~500k max.

So it seems the duet wifi has problems with some wifi setups. I'll investigate this when I get home from work this evening, but right now I have no clue where I should start...
At work we have aruba APs and at home I'm running 2 cisco APs and never had any issues, especially not such specific ones like transfers only in one direction (downloading from the duet works just fine!) and only above a certain filesize...

I'll try to gather some useful information from the debug output later. Here on the network at work all seems fine (=no errors/warnings) when I enable debug for network and wifi (M111 P1 S1 && M111 P1 S14)

sko

Heres some debug output from the Webserver:

HTTP req, command words { POST /rr_upload HTTP/1.1 }, parameters { name=0:/gcodes/3D4U_Coffee_Clip.gcode time=2020-7-13T20:34:25 crc32=ba283f22 }
Start uploading file 0:/gcodes/3D4U_Coffee_Clip.gcode length 42855016
HTTP req, command words { GET /rr_reply HTTP/1.1 }, parameters { }
Sending reply, file = no
Sending G-Code reply to HTTP client 1 of 1 (length 2)
HTTP connection accepted
HTTP req, command words { GET /rr_model HTTP/1.1 }, parameters { flags=d99fn }
Sending JSON reply, length 1950
HTTP connection accepted
HTTP req, command words { GET /rr_model HTTP/1.1 }, parameters { key=volumes flags=d99vn }
Sending JSON reply, length 249
HTTP connection accepted
HTTP req, command words { GET /rr_reply HTTP/1.1 }, parameters { }
Sending reply, file = no
Sending G-Code reply to HTTP client 1 of 1 (length 1)
HTTP connection accepted
HTTP req, command words { GET /rr_model HTTP/1.1 }, parameters { flags=d99fn }
Sending JSON reply, length 1950
HTTP connection accepted
HTTP req, command words { GET /rr_model HTTP/1.1 }, parameters { key=volumes flags=d99vn }
Sending JSON reply, length 249
HTTP connection accepted
HTTP req, command words { GET /rr_reply HTTP/1.1 }, parameters { }
Sending reply, file = no
Sending G-Code reply to HTTP client 1 of 1 (length 1)
HTTP connection accepted
HTTP req, command words { GET /rr_model HTTP/1.1 }, parameters { flags=d99fn }
[...]

Right after the initial HTTP POST there's only the page refresh data. Nothing else about the file transfer.

Debug output from storage (P10) shows some writes right after the HTTP POST, but they disappear after a few seconds.

Network (P1) and WiFi (P14) don't show anything else but status/keepalive data. No errors or such (I can also post the whole, crowded output when logging everything together if requested...)

I also did a tcpdump while starting a transfer:

00:00:00.170736 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500)
    172.25.52.99.53451 > 172.25.52.135.80: Flags [.], cksum 0xc6eb (incorrect -> 0x05c8), seq 1:1461, ack 1, win 65535, length 1460: HTTP, length: 1460
        POST /rr_upload?name=0%3A%2Fgcodes%2F3D4U_Coffee_Clip.gcode&time=2020-7-13T20%3A34%3A25&crc32=ba283f22 HTTP/1.1
        Host: 172.25.52.135
        User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Firefox/68.0
        Accept: */*
        Accept-Language: en-US,en;q=0.5
        Accept-Encoding: gzip, deflate
        Referer: http://172.25.52.135/Files/Jobs
        Content-Type: application/json
        Content-Length: 42855016
        DNT: 1
        Connection: keep-alive

        M4010 X300 Y300
        M4010 I0 T25960 'ffff3fffffff3fffffff3fffffff3fffffff3fffffff3fffffff33190000290141c2628373048364300562834a0231410840ffff311c08004a0283448ba5300c83445a43398218c118a030030000ffff31100800314149c35203'
        M4010 I25960 T623 '18816ac48ba5301193c5ac459c058b857b044a021880ffff310c000039427b448b857b0473046ac48ba5301283447b048b8583448ba530035a431080ffff3109000052038ba5300372c45244834593858ba530127b0472c47b44'
        M4010 I26583 T582 '8ba5300483443141ffff310839828ba530056ac49bc5b485ac458ba5300e93c5ac45a4458b857b448ba530075a430840ffff309700000860300518c129213004086030050000ffff305d18817b458ba53007ac45cd06dd86cd06'
        M4010 I27165 T287 'b486a40593e530028ba5300393e53002ac45bcc6cd06d546c506a4058ba5300a83442101ffff308d000008002101290149c24a02628372e430027b048ba5300f834472e4300262834a0249c22921300208000000ffff30530000'
        M4010 I27452 T315 '62848ba5300aa445bcc6d546ee06fe47fe87ee47ee07ee47fea73002fe47f606c4c6a4058ba5300e52030000ffff308500000840290141825a436ac483448ba530217b0472c44a02314118800000ffff304e31428ba5300993a53006'
        M4010 I2[!http]
 00:00:00.003299 IP (tos 0x0, ttl 255, id 29367, offset 0, flags [none], proto TCP (6), length 40)
    172.25.52.135.80 > 172.25.52.99.53451: Flags [.], cksum 0xbfb1 (correct), ack 1461, win 5840, length 0
 00:00:00.000007 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500)
    172.25.52.99.53451 > 172.25.52.135.80: Flags [.], cksum 0xc6eb (incorrect -> 0xd43a), seq 1461:2921, ack 1, win 65535, length 1460: HTTP
 00:00:00.000004 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500)
    172.25.52.99.53451 > 172.25.52.135.80: Flags [.], cksum 0xc6eb (incorrect -> 0xafd8), seq 2921:4381, ack 1, win 65535, length 1460: HTTP
 00:00:00.017993 IP (tos 0x0, ttl 255, id 29368, offset 0, flags [none], proto TCP (6), length 40)
    172.25.52.135.80 > 172.25.52.99.53451: Flags [.], cksum 0xbfb1 (correct), ack 2921, win 4380, length 0
 00:00:00.000017 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500)
    172.25.52.99.53451 > 172.25.52.135.80: Flags [.], cksum 0xc6eb (incorrect -> 0x6e08), seq 4381:5841, ack 1, win 65535, length 1460: HTTP
 00:00:00.000005 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500)
    172.25.52.99.53451 > 172.25.52.135.80: Flags [.], cksum 0xc6eb (incorrect -> 0x3e4c), seq 5841:7301, ack 1, win 65535, length 1460: HTTP
 00:00:00.001360 IP (tos 0x0, ttl 255, id 29369, offset 0, flags [none], proto TCP (6), length 40)
    172.25.52.135.80 > 172.25.52.99.53451: Flags [.], cksum 0xbfb1 (correct), ack 2921, win 4380, length 0
 00:00:00.009468 IP (tos 0x0, ttl 255, id 29370, offset 0, flags [none], proto TCP (6), length 40)
    172.25.52.135.80 > 172.25.52.99.53451: Flags [.], cksum 0xb9fd (correct), ack 2921, win 5840, length 0
 00:00:00.026213 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 322)
    172.25.52.99.28373 > 172.25.52.135.80: Flags [P.], cksum 0xc251 (incorrect -> 0x4753), seq 1:283, ack 1, win 65535, length 282: HTTP, length: 282
        GET /rr_reply HTTP/1.1
        Host: 172.25.52.135
        User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Firefox/68.0
        Accept: */*
        Accept-Language: en-US,en;q=0.5
        Accept-Encoding: gzip, deflate
        Referer: http://172.25.52.135/Files/Jobs
        DNT: 1
        Connection: keep-alive

 00:00:00.003966 IP (tos 0x0, ttl 255, id 29371, offset 0, flags [none], proto TCP (6), length 239)
    172.25.52.135.80 > 172.25.52.99.28373: Flags [P.], cksum 0x3266 (correct), seq 1:200, ack 283, win 5558, length 199: HTTP, length: 199
        HTTP/1.1 200 OK
        Cache-Control: no-cache, no-store, must-revalidate
        Pragma: no-cache
        Expires: 0
        Access-Control-Allow-Origin: *
        Content-Type: text/plain
        Content-Length: 2
        Connection: close

So it starts to transmit the file (as plaintext) but the connection is dropped after a few lines and then only the keepalives are left...

I don't have more time right now (lunch break...) but I'll try to fire up another AP tonight to verify if it's a problem with how the wireless network is handled (by the duet?) or if there's another problem (e.g. MTU related...)

Phaedrux

@sko said in Cannot upload files. (Duet 2 Wifi):

Although upload speeds are a bit disappointing at ~500k max.

That would be normal speed. There isn't a lot of ram available to buffer the transfer.

A Former User

Uploads in conjunction with Raspberry Pi is almost an order of magnitude faster, might be worth while upgrade if upload speeds are an issue (Terms and conditions apply, subject to wifi coverage, wind direction and lunar phase)

(edit: and also subject to the stable release of RRF3.2 and some custom wiring, but potentially doable without soldering, ref older threads here)

sko

@Phaedrux said in Cannot upload files. (Duet 2 Wifi):

@sko said in Cannot upload files. (Duet 2 Wifi):

Although upload speeds are a bit disappointing at ~500k max.

That would be normal speed. There isn't a lot of ram available to buffer the transfer.

Totally fine for this use case. It's not that you have to push multiple GB per day through that connection...
One can also put the duet behind a nginx proxy that buffers the connection (and could deal with SSL/TLS and authentication) to increase speeds, although it would of course take the same time until the files are actually available for printing.
In fact I already did that to reach the duet from outside of my network - it doesn't solve the upload issue though...

I haven't had much time this evening to do anything meaningful. I hope to find some time tomorrow. I checked the connection and interface stats on the AP and everything seems fine, although dropped frames are a tad high considering the duet is lying ~1m from the AP. OTOH the Amazon Fire TV stick has a much higher rate of dropped frames (b/c its wifi is total crap... ) and still works just fine...

sko

I totally forgot yesterday to mention that the AP logs the following errors from time to time:

[...]
Aug 11 17:19:53.181: %DOT11-4-CCMP_REPLAY: Client d8bf.c0fe.92c1 had 1 AES-CCMP TSC replays
Aug 11 17:50:23.178: %DOT11-4-CCMP_REPLAY: Client d8bf.c0fe.92c1 had 1 AES-CCMP TSC replays
Aug 11 18:18:23.199: %DOT11-4-CCMP_REPLAY: Client d8bf.c0fe.92c1 had 1 AES-CCMP TSC replays
Aug 11 18:35:23.242: %DOT11-4-CCMP_REPLAY: Client d8bf.c0fe.92c1 had 1 AES-CCMP TSC replays
[...]

An actual replay attack can be dismissed - I live on the countryside and there's nobody in range to reach my networks and I highly doubt someone is hiding outside for >12 hours and counting...
There's a known bug with several Cisco APs and Intel WiFi Chipsets:
https://quickview.cloudapps.cisco.com/quickview/bug/CSCsx50775
https://quickview.cloudapps.cisco.com/quickview/bug/CSCur62548

I'll try another AP at home just to verify if the problem lies within the wireless connection.

@Jenco what type/brand of AP are you using? Are you getting the same errors (given it is capable of logging/showing such errors...)?

sko

I've now tried Wifi firmware versions 1.21 and 1.22 with similar results on the cisco APs.

Using another AP (FS AP1167C) however solves the problem (or at least hides it). From the various discussions I found in cisco support forums, the error message (CCMP TSC replays) is usually triggered by malformed packages, caused by some broken WiFi Drivers (notably some Intel Windows drivers, e.g. older Centrinos). For most users this got fixed by updating the drivers.
So it seems the duet is sometimes sending malformed frames that are being dropped by some APs (like cisco, causing above mentioned error message) but accepted by others...

If true, this might also explain issues with disconnections or not being able to connect, especially for APs that have "crude" mitigations for deauth-attacks implemented (-> they just kick out the client...)

droftarts

@sko This sounds similar to this issue with a new Vodafone router (perhaps Cisco make them?) https://forum.duet3d.com/topic/14993/cannot-connect-duet-wifi-to-new-wifi-router
There does seem to be a bug in the expressif SDK that was used to build the WiFi firmware, and causes some routers not to connect, and perhaps you're having similar issues. @dc42 has said this is a possible bug, and it's on his list to do a new wifi server using the newer SDK, but it's not a quick job.

Best current workaround is to use a different access point, connected via ethernet to the Cisco router.

Ian

sko

OK, so if this is an already known bug that is being worked on (soon-ish...) I'll cease further investigation and troubleshooting for now until there's a newer version...

Is there an open discussion about this issue I can follow or chime in to help if required? There's no related open issue at the DuetWiFiServer repo on github...