Lost connection when uploading Gcode file during print

MartinNYHC

Duet3 6HC with SBC (RPI 4 with 4GB RAM), Duet 3 3HC expansion, Duet 1LC Toolboard running RRF 3.3.0.

When uploading larger Gcode files during an active print connection to board is lost. Don't had this issue in the past. Any ideas please?

Screenshot 2021-07-06 at 16.14.40.png

06/07/2021, 16:15:04 	M122
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.3 (2021-06-15 21:45:47) running on Duet 3 MB6HC v1.01 or later (SBC mode)
Board ID: 08DJM-9P63L-DJ3T0-6J1DL-3SN6Q-KS33A
Used output buffers: 1 of 40 (40 max)
=== RTOS ===
Static ram: 150904
Dynamic ram: 62788 of which 0 recycled
Never used RAM 137644, free system stack 132 words
Tasks: SBC(ready,26.0%,324) HEAT(delaying,0.1%,325) Move(notifyWait,1.5%,250) CanReceiv(notifyWait,0.1%,774) CanSender(notifyWait,0.1%,362) CanClock(delaying,0.0%,339) TMC(notifyWait,45.2%,59) MAIN(running,27.0%,922) IDLE(ready,0.1%,29), total 100.0%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 01:56:36 ago, cause: software
Last software reset at 2021-07-06 14:18, reason: User, none spinning, available RAM 137644, slot 0
Software reset code 0x0012 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0043c000 BFAR 0x00000000 SP 0x00000000 Task SBC Freestk 0 n/a
Error status: 0x04
Aux0 errors 0,1,0
Step timer max interval 273
MCU temperature: min 33.1, current 33.6, max 46.6
Supply voltage: min 23.6, current 23.9, max 24.0, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.1, current 12.2, max 12.2, under voltage events: 0
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/496/496, gc cycles 0
Driver 0: position 25594, standstill, reads 7311, writes 39 timeouts 0, SG min/max 0/1023
Driver 1: position -18492, standstill, reads 7311, writes 39 timeouts 0, SG min/max 0/1023
Driver 2: position 5712, standstill, reads 7303, writes 47 timeouts 0, SG min/max 0/1023
Driver 3: position 0, standstill, reads 7304, writes 47 timeouts 0, SG min/max 0/1023
Driver 4: position 0, standstill, reads 7304, writes 47 timeouts 0, SG min/max 0/1023
Driver 5: position 0, standstill, reads 7304, writes 47 timeouts 0, SG min/max 0/1023
Date/time: 2021-07-06 16:15:04
Slowest loop: 262.72ms; fastest: 0.03ms
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 37.5MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, maxWait 584309ms, bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves 149157, completed moves 149157, hiccups 0, stepErrors 0, LaErrors 0, Underruns [3, 0, 32], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
Heater 0 is on, I-accum = 0.4
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP* is doing "M122" in state(s) 0
Telnet is idle in state(s) 0
File* is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue* is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== CAN ===
Messages queued 189088, received 85214, lost 0, longest wait 2ms for reply type 6049, peak Tx sync delay 274, free buffers 49 (min 26), ts 34982/34981/0
Tx timeouts 0,0,0,0,0,0
=== SBC interface ===
State: 4, failed transfers: 1, checksum errors: 0
Last transfer: 1ms ago
RX/TX seq numbers: 51651/13836
SPI underruns 0, overruns 0
Disconnects: 1, timeouts: 1, IAP RAM available 0x2c810
Buffer RX/TX: 0/0-0
=== Duet Control Server ===
Duet Control Server v3.3.0
Code buffer space: 4096
Configured SPI speed: 8000000Hz
Full transfers per second: 44.89, max wait times: 50.6ms/0.0ms
Codes per second: 23.58
Maximum length of RX/TX data transfers: 3216/1560

Jul 06 16:08:22 Voron2 DuetWebServer[736]: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 GET http://192.168.255.38/machine/directory/0:%2Fgcodes%2FMMU%2FCa
Jul 06 16:08:24 Voron2 DuetWebServer[736]: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 PUT http://192.168.255.38/machine/file/0:%2Fgcodes%2FMMU%2FCarrot
Jul 06 16:08:24 Voron2 DuetWebServer[736]: Microsoft.AspNetCore.Cors.Infrastructure.CorsService[5] CORS policy execution failed.
Jul 06 16:08:24 Voron2 DuetWebServer[736]: Microsoft.AspNetCore.Cors.Infrastructure.CorsService[6] Request origin http://192.168.255.38 does not have permission to access the reso
Jul 06 16:08:24 Voron2 DuetWebServer[736]: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'DuetWebServer.Controllers.MachineController.UploadFile (DuetWebSe
Jul 06 16:08:24 Voron2 DuetWebServer[736]: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[3] Route matched with {action = "UploadFile", controller = "Machine"}. E
Jul 06 16:08:33 Voron2 DuetControlServer[6350]: [info] System time has been changed
Jul 06 16:08:33 Voron2 DuetControlServer[6350]: [warn] Lost connection to Duet (Timeout while waiting for transfer ready pin)
Jul 06 16:08:33 Voron2 DuetControlServer[6350]: [info] Connection to Duet established
Jul 06 16:08:33 Voron2 DuetWebServer[736]: Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor[1] Executing CreatedResult, writing value of type 'null'.
Jul 06 16:08:33 Voron2 DuetWebServer[736]: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[2] Executed action DuetWebServer.Controllers.MachineController.UploadFil
Jul 06 16:08:33 Voron2 DuetWebServer[736]: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint 'DuetWebServer.Controllers.MachineController.UploadFile (DuetWebSer
Jul 06 16:08:33 Voron2 DuetWebServer[736]: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 PUT http://192.168.255.38/machine/file/0:%2Fgcodes%2FMMU%2FCarrot
Jul 06 16:08:33 Voron2 DuetControlServer[6350]: [warn] Controller has been reset
Jul 06 16:08:34 Voron2 DuetWebServer[736]: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 GET http://192.168.255.38/machine/directory/0:%2Fgcodes%2FMMU%2FCa
Jul 06 16:08:34 Voron2 DuetWebServer[736]: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'DuetWebServer.Controllers.MachineController.GetFileList (DuetWebS
Jul 06 16:08:34 Voron2 DuetWebServer[736]: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[3] Route matched with {action = "GetFileList", controller = "Machine"}.
Jul 06 16:08:34 Voron2 DuetWebServer[736]: Microsoft.AspNetCore.Mvc.Infrastructure.ContentResultExecutor[1] Executing ContentResult with HTTP Response ContentType of application/j
Jul 06 16:08:34 Voron2 DuetWebServer[736]: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[2] Executed action DuetWebServer.Controllers.MachineController.GetFileLi
Jul 06 16:08:34 Voron2 DuetWebServer[736]: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint 'DuetWebServer.Controllers.MachineController.GetFileList (DuetWebSe
Jul 06 16:08:34 Voron2 DuetWebServer[736]: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 GET http://192.168.255.38/machine/directory/0:%2Fgcodes%2FMMU%2FCa
Jul 06 16:08:34 Voron2 DuetWebServer[736]: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 GET http://192.168.255.38/machine/fileinfo/0:%2Fgcodes%2FMMU%2FCar
Jul 06 16:08:34 Voron2 DuetWebServer[736]: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'DuetWebServer.Controllers.MachineController.GetFileInfo (DuetWebS
Jul 06 16:08:34 Voron2 DuetWebServer[736]: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[3] Route matched with {action = "GetFileInfo", controller = "Machine"}.
Jul 06 16:08:34 Voron2 DuetControlServer[6350]: [info] Aborted job file

MartinNYHC

Just had the connection reset when uploading a file but not printing.

Phaedrux

What kind of SD card are you using? Do you have an A2 class to try instead?
How large is the file you're uploading?

MartinNYHC

I'm using PXE boot with GigabitEthernet on a very performent NAS.

ls -alh carrot_patch_red_set.gcode
-rw-r--r--@ 1 xxx.yyy  staff    44M Jul  6 20:05 carrot_patch_red_set.gcode

chrishamm

These two lines

Jul 06 16:08:33 Voron2 DuetControlServer[6350]: [info] System time has been changed
Jul 06 16:08:33 Voron2 DuetControlServer[6350]: [warn] Controller has been reset

indicate that the IO throughput is too low and/or the CPU load too high. If you really need to run your Pi from a PXE source, I suggest you copy DSF to memory (e.g. /tmp) to keep the latencies as low as possible.

MartinNYHC

thanks. ok, makes sense.

can you please explain what you exactly mean with "DSF"? the complete folder? what i'm thinking about is the fact the the "gcodes" foder si also a subfolder

chrishamm

@martinnyhc I'd start with the /opt/dsf/bin folder and check if that improves things. If you have macro files which are called frequently from G/M-codes, I'd copy the system folder as well (/opt/dsf/sd/sys).

PS: It might be possible to set up cached folders in your PXE system config (comparable to offline files on Windows) but I can't say if that is possible. Might be worth checking though.

MartinNYHC

allright, that helps. will try things out and let you know.

MartinNYHC

All of a sudden I get the following error message:

Jul 07 16:39:38 Voron2 DuetControlServer[952]: [warn] IPC#14: Client with outdated protocol version connected (got 7, want 11)
Jul 07 16:39:40 Voron2 DuetControlServer[952]: [info] Received file abort request on channel HTTP for all files
Jul 07 16:39:40 Voron2 DuetControlServer[952]: [info] Aborted macro file homeall.g

Any ideas?

chrishamm

@martinnyhc There should be something above that log part showing the reason for the file aborts. Possible reasons may include bad probe readings, emergency stops, stack overflows (by M-codes or M120/M121), or G-code exceptions (e.g. from bad expressions).

That first warning message from your excerpt comes from a plugin that doesn't use the latest API version.

MartinNYHC

@chrishamm said in Lost connection when uploading Gcode file during print:

@martinnyhc There should be something above that log part showing the reason for the file aborts. Possible reasons may include bad probe readings, emergency stops, stack overflows (by M-codes or M120/M121), or G-code exceptions (e.g. from bad expressions).

That first warning message from your excerpt comes from a plugin that doesn't use the latest API version.

Yeah, in the meantime I've found out that the Z endstop was the cause. Replaced it and everthing's good again

Will check the SBC tmpfs thing today and let you know the results.

MartinNYHC

Moving dsf/bin to tmpfs didn't solved the problem. But that's fine. Just don't upload large files during a print. Need a reminder

MartinNYHC

I decided to use a SD card again and only mounting the macro and sys folder via network. That should solve the issue and I don't need to take care of backing up these files.