Cannot upload files. (Duet 2 Wifi)



  • Hello. After a few days of configuring and settin my Duet WiFi board. The only issue I'm having is not being able to upload my gcodes because of timeouts. Transfers start out great then drops speed into the kbps, then timeout. Here is my M122 report. Also, Ive changed my SD card to a different one. (both off brand, bought a brand name one off amazon. should be here in a couple days just to be sure) I've also re flashed this device just to make sure as well. Thanks again for this really well made board!

    M122
    === Diagnostics ===
    RepRapFirmware for Duet 2 WiFi/Ethernet version 3.1.1 running on Duet WiFi 1.02 or later
    Board ID: 08DLM-996AL-K65SN-6JTDJ-3SD6N-992UY
    Used output buffers: 3 of 24 (13 max)
    === RTOS ===
    Static ram: 27980
    Dynamic ram: 93840 of which 44 recycled
    Exception stack ram used: 560
    Never used ram: 8648
    Tasks: NETWORK(ready,384) HEAT(blocked,1224) MAIN(running,1848) IDLE(ready,80)
    Owned mutexes: WiFi(NETWORK)
    === Platform ===
    Last reset 00:33:17 ago, cause: software
    Last software reset at 2020-08-03 23:45, reason: User, spinning module GCodes, available RAM 9144 bytes (slot 1)
    Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0441f000 BFAR 0xe000ed38 SP 0xffffffff Task MAIN
    Error status: 0
    MCU temperature: min 32.5, current 33.5, max 35.7
    Supply voltage: min 24.1, current 24.3, max 24.7, under voltage events: 0, over voltage events: 0, power good: yes
    Driver 0: standstill, SG min/max 21/214
    Driver 1: ok, SG min/max 0/187
    Driver 2: ok, SG min/max 0/183
    Driver 3: ok, SG min/max 0/123
    Driver 4: standstill, SG min/max not available
    Date/time: 2020-08-04 00:18:23
    Cache data hit count 3329487922
    Slowest loop: 48.84ms; fastest: 0.12ms
    I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
    === Storage ===
    Free file entries: 9
    SD card 0 detected, interface speed: 20.0MBytes/sec
    SD card longest read time 2.8ms, write time 1.9ms, max retries 0
    === Move ===
    Hiccups: 0(0), FreeDm: 166, MinFreeDm: 144, MaxWait: 54149ms
    Bed compensation in use: mesh, comp offset 0.000
    === MainDDARing ===
    Scheduled moves: 3010, completed moves: 3006, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: 3
    === AuxDDARing ===
    Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
    === Heat ===
    Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
    Heater 0 is on, I-accum = 0.0
    Heater 1 is on, I-accum = 0.5
    === GCodes ===
    Segments left: 1
    Movement lock held by null
    HTTP is idle in state(s) 0
    Telnet is idle in state(s) 0
    File is doing "G1 X-48.479 Y-29.807 E0.00872" in state(s) 0
    USB is idle in state(s) 0
    Aux is idle in state(s) 0
    Trigger is idle in state(s) 0
    Queue is idle in state(s) 0
    Daemon is idle in state(s) 0
    Autopause is idle in state(s) 0
    Code queue is empty.
    === Network ===
    Slowest loop: 42.10ms; fastest: 0.00ms
    Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
    HTTP sessions: 1 of 8
    - WiFi -
    Network state is active
    WiFi module is connected to access point 
    Failed messages: pending 0, notready 0, noresp 0
    WiFi firmware version 1.23
    WiFi MAC address 50:02:91:6a:34:21
    WiFi Vcc 3.35, reset reason Unknown
    WiFi flash size 4194304, free heap 24072
    WiFi IP address 192.168.1.181
    WiFi signal strength -38dBm, reconnections 0, sleep mode modem
    Socket states: 0 0 0 0 0 0 0 0
    

  • Moderator

    Can you verify that your DWC version is also 3.1.1?

    @Jenco said in Cannot upload files. (Duet 2 Wifi):

    WiFi signal strength -38dBm

    Signal strength looks good. If you check it when the transfers are slow does it get worse? Intermittent interference from other devices could be harming your transfers.



  • DWC.jpg

    Signal stays constant. Ive also tried changing the wifi channel, alas no dice.


  • Moderator

    How is the SD card formatted?

    It could be a failing card, but having 2 failing cards seems unlikely.

    How large are the files you're trying to upload? Do they ever succeed? Do smaller files succeed? Are you able to stay connected to DWC and navigate normally? Have you tried a different browser?



  • @Phaedrux well the largest file i could complete is right around the 1mb mark. Anything larger fails. the file progress progresses to about 35% and resets to 0 a few times before failing... I also tried uploading by ftp i can delete files on the duet from ftp but as soon as i initiate the upload it fails no matter the file size. Im logged in as anon so that could be it.

    As for smaller files it takes around a minute to upload but they do succeed thru !DWC!

    As for navigation. DWC seems sluggish and I can only connect to it from 1 computer at a time.

    Oh once i get the gcode to the SD (as in copying it from my computer with an sd adapter). It prints beautifully.



  • Having the same problem here. After ~40kB any upload stalls. Smaller files transfer just fine...

    M122
    === Diagnostics ===
    RepRapFirmware for Duet 2 WiFi/Ethernet version 3.1.1 running on Duet WiFi 1.02 or later
    Board ID: 08DLM-996AL-K6PSD-6J1DJ-3SD6L-99GGY
    Used output buffers: 3 of 24 (10 max)
    === RTOS ===
    Static ram: 27980
    Dynamic ram: 93760 of which 56 recycled
    Exception stack ram used: 240
    Never used ram: 9036
    Tasks: NETWORK(ready,384) HEAT(blocked,1320) MAIN(running,1912) IDLE(ready,80)
    Owned mutexes: WiFi(NETWORK)
    === Platform ===
    Last reset 00:00:40 ago, cause: software
    Last software reset at 2020-08-10 18:21, reason: User, spinning module GCodes, available RAM 8992 bytes (slot 1)
    Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x04433000 BFAR 0xe000ed38 SP 0xffffffff Task MAIN
    Error status: 0
    MCU temperature: min 37.0, current 37.7, max 37.8
    Supply voltage: min 1.7, current 1.7, max 1.8, under voltage events: 0, over voltage events: 0, power good: no
    Driver 0: ok, SG min/max not available
    Driver 1: ok, SG min/max not available
    Driver 2: ok, SG min/max not available
    Driver 3: ok, SG min/max not available
    Driver 4: ok, SG min/max not available
    Date/time: 2020-08-10 18:22:06
    Cache data hit count 65784672
    Slowest loop: 5.57ms; fastest: 0.10ms
    I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
    === Storage ===
    Free file entries: 10
    SD card 0 detected, interface speed: 20.0MBytes/sec
    SD card longest read time 3.4ms, write time 0.0ms, max retries 0
    === Move ===
    Hiccups: 0(0), FreeDm: 169, MinFreeDm: 169, MaxWait: 0ms
    Bed compensation in use: none, comp offset 0.000
    === MainDDARing ===
    Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
    === AuxDDARing ===
    Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
    === Heat ===
    Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
    === GCodes ===
    Segments left: 0
    Movement lock held by null
    HTTP is idle in state(s) 0
    Telnet is idle in state(s) 0
    File is idle in state(s) 0
    USB is idle in state(s) 0
    Aux is idle in state(s) 0
    Trigger is idle in state(s) 0
    Queue is idle in state(s) 0
    Daemon is idle in state(s) 0
    Autopause is idle in state(s) 0
    Code queue is empty.
    === Network ===
    Slowest loop: 15.93ms; fastest: 0.00ms
    Responder states: HTTP(1) HTTP(2) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
    HTTP sessions: 1 of 8
    - WiFi -
    Network state is active
    WiFi module is connected to access point 
    Failed messages: pending 0, notready 0, noresp 0
    WiFi firmware version 1.23
    WiFi MAC address d8:bf:c0:fe:92:c1
    WiFi Vcc 3.38, reset reason Unknown
    WiFi flash size 4194304, free heap 24280
    WiFi IP address 172.25.52.135
    WiFi signal strength -43dBm, reconnections 0, sleep mode modem
    Socket states: 2 4 0 0 0 0 0 0
    

    Tried 2 brand new SD-cards with the same result, and as I can upload smaller files and edit the config through DWC I don't think its the SD card. Is there any max filesize or buffer defined in the Webserver that might interfere with uploads?


  • Moderator

    @Jenco @sko

    Would you both be able to test by rolling back to firmware 2.05.1 and DWC 2.0.7 and seeing if you can reliably upload?

    Your config.g won't work very well with RRF2 but you should still be able to upload a file.

    If it helps, we at least have a lead to follow. IF it doesn't help, then it might be a wifi module issue.



  • I took the board with me to work today to flash the 2.05.1 firmware and here uploads are working (with DWC 3.1.1 !).
    Just to verify I re-flashed with firmware 3.1.1 and uploads are still working!
    Although upload speeds are a bit disappointing at ~500k max.

    So it seems the duet wifi has problems with some wifi setups. I'll investigate this when I get home from work this evening, but right now I have no clue where I should start...
    At work we have aruba APs and at home I'm running 2 cisco APs and never had any issues, especially not such specific ones like transfers only in one direction (downloading from the duet works just fine!) and only above a certain filesize...

    I'll try to gather some useful information from the debug output later. Here on the network at work all seems fine (=no errors/warnings) when I enable debug for network and wifi (M111 P1 S1 && M111 P1 S14)



  • Heres some debug output from the Webserver:

    HTTP req, command words { POST /rr_upload HTTP/1.1 }, parameters { name=0:/gcodes/3D4U_Coffee_Clip.gcode time=2020-7-13T20:34:25 crc32=ba283f22 }
    Start uploading file 0:/gcodes/3D4U_Coffee_Clip.gcode length 42855016
    HTTP req, command words { GET /rr_reply HTTP/1.1 }, parameters { }
    Sending reply, file = no
    Sending G-Code reply to HTTP client 1 of 1 (length 2)
    HTTP connection accepted
    HTTP req, command words { GET /rr_model HTTP/1.1 }, parameters { flags=d99fn }
    Sending JSON reply, length 1950
    HTTP connection accepted
    HTTP req, command words { GET /rr_model HTTP/1.1 }, parameters { key=volumes flags=d99vn }
    Sending JSON reply, length 249
    HTTP connection accepted
    HTTP req, command words { GET /rr_reply HTTP/1.1 }, parameters { }
    Sending reply, file = no
    Sending G-Code reply to HTTP client 1 of 1 (length 1)
    HTTP connection accepted
    HTTP req, command words { GET /rr_model HTTP/1.1 }, parameters { flags=d99fn }
    Sending JSON reply, length 1950
    HTTP connection accepted
    HTTP req, command words { GET /rr_model HTTP/1.1 }, parameters { key=volumes flags=d99vn }
    Sending JSON reply, length 249
    HTTP connection accepted
    HTTP req, command words { GET /rr_reply HTTP/1.1 }, parameters { }
    Sending reply, file = no
    Sending G-Code reply to HTTP client 1 of 1 (length 1)
    HTTP connection accepted
    HTTP req, command words { GET /rr_model HTTP/1.1 }, parameters { flags=d99fn }
    [...]
    

    Right after the initial HTTP POST there's only the page refresh data. Nothing else about the file transfer.

    Debug output from storage (P10) shows some writes right after the HTTP POST, but they disappear after a few seconds.

    Network (P1) and WiFi (P14) don't show anything else but status/keepalive data. No errors or such (I can also post the whole, crowded output when logging everything together if requested...)

    I also did a tcpdump while starting a transfer:

    00:00:00.170736 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500)
        172.25.52.99.53451 > 172.25.52.135.80: Flags [.], cksum 0xc6eb (incorrect -> 0x05c8), seq 1:1461, ack 1, win 65535, length 1460: HTTP, length: 1460
            POST /rr_upload?name=0%3A%2Fgcodes%2F3D4U_Coffee_Clip.gcode&time=2020-7-13T20%3A34%3A25&crc32=ba283f22 HTTP/1.1
            Host: 172.25.52.135
            User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Firefox/68.0
            Accept: */*
            Accept-Language: en-US,en;q=0.5
            Accept-Encoding: gzip, deflate
            Referer: http://172.25.52.135/Files/Jobs
            Content-Type: application/json
            Content-Length: 42855016
            DNT: 1
            Connection: keep-alive
    
            M4010 X300 Y300
            M4010 I0 T25960 'ffff3fffffff3fffffff3fffffff3fffffff3fffffff3fffffff33190000290141c2628373048364300562834a0231410840ffff311c08004a0283448ba5300c83445a43398218c118a030030000ffff31100800314149c35203'
            M4010 I25960 T623 '18816ac48ba5301193c5ac459c058b857b044a021880ffff310c000039427b448b857b0473046ac48ba5301283447b048b8583448ba530035a431080ffff3109000052038ba5300372c45244834593858ba530127b0472c47b44'
            M4010 I26583 T582 '8ba5300483443141ffff310839828ba530056ac49bc5b485ac458ba5300e93c5ac45a4458b857b448ba530075a430840ffff309700000860300518c129213004086030050000ffff305d18817b458ba53007ac45cd06dd86cd06'
            M4010 I27165 T287 'b486a40593e530028ba5300393e53002ac45bcc6cd06d546c506a4058ba5300a83442101ffff308d000008002101290149c24a02628372e430027b048ba5300f834472e4300262834a0249c22921300208000000ffff30530000'
            M4010 I27452 T315 '62848ba5300aa445bcc6d546ee06fe47fe87ee47ee07ee47fea73002fe47f606c4c6a4058ba5300e52030000ffff308500000840290141825a436ac483448ba530217b0472c44a02314118800000ffff304e31428ba5300993a53006'
            M4010 I2[!http]
     00:00:00.003299 IP (tos 0x0, ttl 255, id 29367, offset 0, flags [none], proto TCP (6), length 40)
        172.25.52.135.80 > 172.25.52.99.53451: Flags [.], cksum 0xbfb1 (correct), ack 1461, win 5840, length 0
     00:00:00.000007 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500)
        172.25.52.99.53451 > 172.25.52.135.80: Flags [.], cksum 0xc6eb (incorrect -> 0xd43a), seq 1461:2921, ack 1, win 65535, length 1460: HTTP
     00:00:00.000004 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500)
        172.25.52.99.53451 > 172.25.52.135.80: Flags [.], cksum 0xc6eb (incorrect -> 0xafd8), seq 2921:4381, ack 1, win 65535, length 1460: HTTP
     00:00:00.017993 IP (tos 0x0, ttl 255, id 29368, offset 0, flags [none], proto TCP (6), length 40)
        172.25.52.135.80 > 172.25.52.99.53451: Flags [.], cksum 0xbfb1 (correct), ack 2921, win 4380, length 0
     00:00:00.000017 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500)
        172.25.52.99.53451 > 172.25.52.135.80: Flags [.], cksum 0xc6eb (incorrect -> 0x6e08), seq 4381:5841, ack 1, win 65535, length 1460: HTTP
     00:00:00.000005 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500)
        172.25.52.99.53451 > 172.25.52.135.80: Flags [.], cksum 0xc6eb (incorrect -> 0x3e4c), seq 5841:7301, ack 1, win 65535, length 1460: HTTP
     00:00:00.001360 IP (tos 0x0, ttl 255, id 29369, offset 0, flags [none], proto TCP (6), length 40)
        172.25.52.135.80 > 172.25.52.99.53451: Flags [.], cksum 0xbfb1 (correct), ack 2921, win 4380, length 0
     00:00:00.009468 IP (tos 0x0, ttl 255, id 29370, offset 0, flags [none], proto TCP (6), length 40)
        172.25.52.135.80 > 172.25.52.99.53451: Flags [.], cksum 0xb9fd (correct), ack 2921, win 5840, length 0
     00:00:00.026213 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 322)
        172.25.52.99.28373 > 172.25.52.135.80: Flags [P.], cksum 0xc251 (incorrect -> 0x4753), seq 1:283, ack 1, win 65535, length 282: HTTP, length: 282
            GET /rr_reply HTTP/1.1
            Host: 172.25.52.135
            User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Firefox/68.0
            Accept: */*
            Accept-Language: en-US,en;q=0.5
            Accept-Encoding: gzip, deflate
            Referer: http://172.25.52.135/Files/Jobs
            DNT: 1
            Connection: keep-alive
    
     00:00:00.003966 IP (tos 0x0, ttl 255, id 29371, offset 0, flags [none], proto TCP (6), length 239)
        172.25.52.135.80 > 172.25.52.99.28373: Flags [P.], cksum 0x3266 (correct), seq 1:200, ack 283, win 5558, length 199: HTTP, length: 199
            HTTP/1.1 200 OK
            Cache-Control: no-cache, no-store, must-revalidate
            Pragma: no-cache
            Expires: 0
            Access-Control-Allow-Origin: *
            Content-Type: text/plain
            Content-Length: 2
            Connection: close
    

    So it starts to transmit the file (as plaintext) but the connection is dropped after a few lines and then only the keepalives are left...

    I don't have more time right now (lunch break...) but I'll try to fire up another AP tonight to verify if it's a problem with how the wireless network is handled (by the duet?) or if there's another problem (e.g. MTU related...)


  • Moderator

    @sko said in Cannot upload files. (Duet 2 Wifi):

    Although upload speeds are a bit disappointing at ~500k max.

    That would be normal speed. There isn't a lot of ram available to buffer the transfer.



  • Uploads in conjunction with Raspberry Pi is almost an order of magnitude faster, might be worth while upgrade if upload speeds are an issue (Terms and conditions apply, subject to wifi coverage, wind direction and lunar phase)

    (edit: and also subject to the stable release of RRF3.2 and some custom wiring, but potentially doable without soldering, ref older threads here)



  • @Phaedrux said in Cannot upload files. (Duet 2 Wifi):

    @sko said in Cannot upload files. (Duet 2 Wifi):

    Although upload speeds are a bit disappointing at ~500k max.

    That would be normal speed. There isn't a lot of ram available to buffer the transfer.

    Totally fine for this use case. It's not that you have to push multiple GB per day through that connection...
    One can also put the duet behind a nginx proxy that buffers the connection (and could deal with SSL/TLS and authentication) to increase speeds, although it would of course take the same time until the files are actually available for printing.
    In fact I already did that to reach the duet from outside of my network - it doesn't solve the upload issue though...

    I haven't had much time this evening to do anything meaningful. I hope to find some time tomorrow. I checked the connection and interface stats on the AP and everything seems fine, although dropped frames are a tad high considering the duet is lying ~1m from the AP. OTOH the Amazon Fire TV stick has a much higher rate of dropped frames (b/c its wifi is total crap... 🙄 ) and still works just fine...



  • I totally forgot yesterday to mention that the AP logs the following errors from time to time:

    [...]
    Aug 11 17:19:53.181: %DOT11-4-CCMP_REPLAY: Client d8bf.c0fe.92c1 had 1 AES-CCMP TSC replays
    Aug 11 17:50:23.178: %DOT11-4-CCMP_REPLAY: Client d8bf.c0fe.92c1 had 1 AES-CCMP TSC replays
    Aug 11 18:18:23.199: %DOT11-4-CCMP_REPLAY: Client d8bf.c0fe.92c1 had 1 AES-CCMP TSC replays
    Aug 11 18:35:23.242: %DOT11-4-CCMP_REPLAY: Client d8bf.c0fe.92c1 had 1 AES-CCMP TSC replays
    [...]
    

    An actual replay attack can be dismissed - I live on the countryside and there's nobody in range to reach my networks and I highly doubt someone is hiding outside for >12 hours and counting...
    There's a known bug with several Cisco APs and Intel WiFi Chipsets:
    https://quickview.cloudapps.cisco.com/quickview/bug/CSCsx50775
    https://quickview.cloudapps.cisco.com/quickview/bug/CSCur62548

    I'll try another AP at home just to verify if the problem lies within the wireless connection.

    @Jenco what type/brand of AP are you using? Are you getting the same errors (given it is capable of logging/showing such errors...)?



  • I've now tried Wifi firmware versions 1.21 and 1.22 with similar results on the cisco APs.

    Using another AP (FS AP1167C) however solves the problem (or at least hides it). From the various discussions I found in cisco support forums, the error message (CCMP TSC replays) is usually triggered by malformed packages, caused by some broken WiFi Drivers (notably some Intel Windows drivers, e.g. older Centrinos). For most users this got fixed by updating the drivers.
    So it seems the duet is sometimes sending malformed frames that are being dropped by some APs (like cisco, causing above mentioned error message) but accepted by others...

    If true, this might also explain issues with disconnections or not being able to connect, especially for APs that have "crude" mitigations for deauth-attacks implemented (-> they just kick out the client...)


  • Moderator

    @sko This sounds similar to this issue with a new Vodafone router (perhaps Cisco make them?) https://forum.duet3d.com/topic/14993/cannot-connect-duet-wifi-to-new-wifi-router
    There does seem to be a bug in the expressif SDK that was used to build the WiFi firmware, and causes some routers not to connect, and perhaps you're having similar issues. @dc42 has said this is a possible bug, and it's on his list to do a new wifi server using the newer SDK, but it's not a quick job.

    Best current workaround is to use a different access point, connected via ethernet to the Cisco router.

    Ian



  • OK, so if this is an already known bug that is being worked on (soon-ish...) I'll cease further investigation and troubleshooting for now until there's a newer version...

    Is there an open discussion about this issue I can follow or chime in to help if required? There's no related open issue at the DuetWiFiServer repo on github...


Log in to reply