Firmware restart while refreshing DWC and pausing
-
Please post the GCode file that caused the reset when you simulated it.
Duet WiFi hardware designer and firmware engineer
Please do not ask me for Duet support via PM or email, use the forum
http://www.escher3d.com, https://miscsolutions.wordpress.com -
@dc42 This file is quite big
https://www.dropbox.com/s/twebf0ajjvn0bdd/moon6inches.gcode?dl=0 -
@dc42 I was digging in the firmware in order to find solution. I have no idea why Network task cannot Give SD0 mutex when external SD is simulating. It is happen when refresh DWC. Firmware want to send all files to the browser and it got stuck in void NetworkResponder::SendData()
I also don't have old releases duet PCB with JTAG socket so I'm not able to use debugger.
-
@Veti said in Firmware restart while refreshing DWC and pausing:
M350 X32 Y32 Z32 E16 I1
@felek Rather than remove the I1 to stop the interpolation, better to use x16 microstepping with interpolation enabled to begin with.
@felek said in Firmware restart while refreshing DWC and pausing:
PanelDue(3.2.9)
Would you be able to test again with PanelDue 1.24?
-
@Phaedrux Ok, I changed microstepping to 16 and enabled interpolation.
Here is my config.g:
; Configuration file for Duet WiFi (firmware version 3) ; executed by the firmware on start-up ; ; generated by RepRapFirmware Configuration Tool v3.1.10 on Sat Dec 05 2020 19:46:54 GMT+0100 (Central European Standard Time) ; General preferences G90 ; send absolute coordinates... M83 ; ...but relative extruder moves M550 P"TT Sapphire" ; set printer name M669 K1 ; select CoreXY mode ; Network M552 S1 ; enable network M586 P0 S1 ; enable HTTP M586 P1 S1 ; enable FTP M586 P2 S0 ; disable Telnet M575 P1 B57600 S1 ; Turn PanelDue on ; Drives M569 P0 S0 ; physical drive 0 goes forwards M569 P1 S0 ; physical drive 1 goes forwards M569 P2 S0 ; physical drive 2 goes forwards M569 P3 S0 ; physical drive 3 goes forwards M584 X0 Y1 Z2 E3 ; set drive mapping ;M350 X32 Y32 Z32 E16 I1 ; configure microstepping with interpolation M350 X16 Y16 Z16 E16 I1 ; configure microstepping with interpolation M92 X160.60 Y160.60 Z3200.00 E392.40 ; set steps per mm M566 X800.00 Y800.00 Z60.00 E420.00 P1 ; set maximum instantaneous speed changes (mm/min) M203 X18000.00 Y18000.00 Z1200.00 E30000.00 ; set maximum speeds (mm/min) M201 X2500.00 Y2500.00 Z200.00 E10000.00 ; set accelerations (mm/s^2) M204 P2000 T4000 ; set acceleration for printing moves M906 X1200 Y1200 Z1200 E800 I50 ; set motor currents (mA) and motor idle factor in per cent M84 S30 ; Set idle timeout ; Axis Limits M208 X0 Y0 Z0 S1 ; set axis minima M208 X223 Y223 Z235 S0 ; set axis maxima ; Endstops M574 X1 S1 P"!xstop" M574 Y2 S1 P"!ystop" M574 Z1 S1 P"!zstop" ; configure active-high endstop for low end on Z via pin zstop ; Z-Probe M558 P1 C"zprobe.in" H5 F120 T6000 ; set Z probe type to unmodulated and the dive height + speeds G31 P500 X0 Y0 Z2.5 ; set Z probe trigger value, offset and trigger height M557 X15:215 Y15:195 S20 ; define mesh grid ; Heaters M308 S0 P"bedtemp" Y"thermistor" T100000 B4138 ; configure sensor 0 as PT1000 on pin bedtemp M950 H0 C"bedheat" T0 ; create bed heater output on bedheat and map it to sensor 0 M307 H0 B0 S1.00 ; disable bang-bang mode for the bed heater and set PWM limit M140 H0 ; map heated bed to heater 0 M143 H0 S120 ; set temperature limit for heater 0 to 120C M308 S1 P"e0temp" Y"thermistor" T100000 B4138 ; configure sensor 1 as thermistor on pin e0temp M950 H1 C"e0heat" T1 ; create nozzle heater output on e0heat and map it to sensor 1 M307 H1 B0 S1.00 ; disable bang-bang mode for heater and set PWM limit M143 H1 S280 ; set temperature limit for heater 1 to 280C ; Fans M950 F0 C"fan0" Q500 ; create fan 0 on pin fan0 and set its frequency M106 P0 S0 H-1 ; set fan 0 value. Thermostatic control is turned off M950 F1 C"fan1" Q500 ; create fan 1 on pin fan1 and set its frequency M106 P1 S1 H-1 ; set fan 1 value. Thermostatic control is turned off ; Tools M563 P0 D0 H1 F0 ; define tool 0 G10 P0 X0 Y0 Z0 ; set tool 0 axis offsets G10 P0 R0 S0 ; set initial tool 0 active and standby temperatures to 0C ; Custom settings are not defined ; Miscellaneous M501 ; load saved parameters from non-volatile memory M593 F51 ; cancel ringing at 51Hz ;M572 D0 S0.7I downgraded PanelDue. Now I'm using 1.24 version but the issue still exist.
Here is M122:
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.2 running on Duet WiFi 1.02 or later Board ID: 0JD0M-9K662-MGPSS-6J1FJ-3SN6M-9SSUW Used output buffers: 3 of 24 (11 max) === RTOS === Static ram: 23460 Dynamic ram: 73392 of which 0 recycled Never used RAM 15228, free system stack 195 words Tasks: NETWORK(ready,182) HEAT(blocked,295) MAIN(running,466) IDLE(ready,19) Owned mutexes: WiFi(NETWORK) === Platform === Last reset 00:00:19 ago, cause: software Last software reset at 2021-02-01 09:45, reason: StuckInSpinLoop, GCodes spinning, available RAM 15192, slot 1 Software reset code 0x4083 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f80f BFAR 0xe000ed38 SP 0x20003ea4 Task MAIN Freestk 641 ok Stack: 0042885f 0042885e 21060000 3f800000 00000000 41200000 3d20a0a1 40f6fb48 44e0f026 44e0f026 3dbdae8d 44e0f026 3ca0a0a1 00000000 00000000 00000000 38533333 3f800000 3f800000 60000010 200025a8 000773d0 200025a8 20000f0c 200025a8 200025b8 200025b2 Error status: 0x00 Aux0 errors 0,0,0 MCU temperature: min 19.9, current 20.8, max 21.1 Supply voltage: min 24.1, current 24.2, max 24.2, under voltage events: 0, over voltage events: 0, power good: yes Driver 0: position 0, standstill, SG min/max not available Driver 1: position 0, standstill, SG min/max not available Driver 2: position 0, standstill, SG min/max not available Driver 3: position 0, standstill, SG min/max not available Driver 4: position 0, standstill, SG min/max not available Driver 5: position 0 Driver 6: position 0 Driver 7: position 0 Driver 8: position 0 Driver 9: position 0 Driver 10: position 0 Driver 11: position 0 Date/time: 2021-02-01 09:46:30 Cache data hit count 22936358 Slowest loop: 2.95ms; fastest: 0.20ms I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0 === Storage === Free file entries: 10 SD card 0 detected, interface speed: 20.0MBytes/sec SD card longest read time 0.7ms, write time 0.0ms, max retries 0 === Move === DMs created 83, maxWait 0ms, bed compensation in use: none, comp offset 0.000 === MainDDARing === Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === AuxDDARing === Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === Heat === Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1 === GCodes === Segments left: 0 Movement lock held by null HTTP is idle in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 Daemon is idle in state(s) 0 Autopause is idle in state(s) 0 Code queue is empty. === Network === Slowest loop: 139.39ms; fastest: 0.00ms Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions HTTP sessions: 1 of 8 - WiFi - Network state is active WiFi module is connected to access point Failed messages: pending 0, notready 0, noresp 0 WiFi firmware version 1.25beta0 WiFi MAC address 40:f5:20:3a:0b:d7 WiFi Vcc 3.29, reset reason Power up WiFi flash size 4194304, free heap 21968 WiFi IP address 192.168.1.28 WiFi signal strength -70dBm, mode 802.11n, reconnections 0, sleep mode modem Clock register 00002002 Socket states: 0 0 0 0 0 0 0 0 -
Here is different reset reason:
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.2 running on Duet WiFi 1.02 or later Board ID: 0JD0M-9K662-MGPSS-6J1FJ-3SN6M-9SSUW Used output buffers: 3 of 24 (11 max) === RTOS === Static ram: 23460 Dynamic ram: 73392 of which 0 recycled Never used RAM 15228, free system stack 200 words Tasks: NETWORK(ready,192) HEAT(blocked,309) MAIN(running,448) IDLE(ready,19) Owned mutexes: WiFi(NETWORK) === Platform === Last reset 00:00:25 ago, cause: software Last software reset at 2021-02-01 09:51, reason: HardFault bfarValid precise, PrintMonitor spinning, available RAM 15228, slot 1 Software reset code 0x4069 HFSR 0x40000000 CFSR 0x00008200 ICSR 0x0441f803 BFAR 0x9d03b488 SP 0x200043c0 Task MAIN Freestk 968 ok Stack: 20020000 00000000 2000444f 9d03b470 00000046 0041fbb5 00428840 610e0000 bf800000 00000000 3edb6db7 b8ac9c00 3331bb4c 40000000 3f317200 b5ddea0e 388a851a bb35d196 3d5dbc4d 00000000 00000000 38533333 00000000 00000000 60000010 200025a8 0007a0e7 Error status: 0x00 Aux0 errors 0,0,0 MCU temperature: min 20.7, current 21.4, max 21.7 Supply voltage: min 24.1, current 24.2, max 24.2, under voltage events: 0, over voltage events: 0, power good: yes Driver 0: position 0, standstill, SG min/max not available Driver 1: position 0, standstill, SG min/max not available Driver 2: position 0, standstill, SG min/max not available Driver 3: position 0, standstill, SG min/max not available Driver 4: position 0, standstill, SG min/max not available Driver 5: position 0 Driver 6: position 0 Driver 7: position 0 Driver 8: position 0 Driver 9: position 0 Driver 10: position 0 Driver 11: position 0 Date/time: 2021-02-01 09:51:32 Cache data hit count 30320382 Slowest loop: 3.20ms; fastest: 0.20ms I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0 === Storage === Free file entries: 10 SD card 0 detected, interface speed: 20.0MBytes/sec SD card longest read time 0.7ms, write time 0.0ms, max retries 0 === Move === DMs created 83, maxWait 0ms, bed compensation in use: none, comp offset 0.000 === MainDDARing === Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === AuxDDARing === Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === Heat === Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1 === GCodes === Segments left: 0 Movement lock held by null HTTP is idle in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 Daemon is idle in state(s) 0 Autopause is idle in state(s) 0 Code queue is empty. === Network === Slowest loop: 140.03ms; fastest: 0.00ms Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions HTTP sessions: 1 of 8 - WiFi - Network state is active WiFi module is connected to access point Failed messages: pending 0, notready 0, noresp 0 WiFi firmware version 1.25beta0 WiFi MAC address 40:f5:20:3a:0b:d7 WiFi Vcc 3.29, reset reason Power up WiFi flash size 4194304, free heap 25632 WiFi IP address 192.168.1.28 WiFi signal strength -69dBm, mode 802.11n, reconnections 0, sleep mode modem Clock register 00002002 Socket states: 0 0 0 0 0 0 0 0 -
@felek said in Firmware restart while refreshing DWC and pausing:
@dc42 This file is quite big
https://www.dropbox.com/s/twebf0ajjvn0bdd/moon6inches.gcode?dl=0Wow, 106Mb gcode file. I'll try simulating it myself.
-
I was able to simulate that gcode file on a Duet2 with fw 3.2 on the internal SD card without error. I don't have the ribbon cable connected for the PanelDue SD card slot, so I can't test that.
You could try reducing the baud rate on the PanelDue and in your
M575 P1 B57600 S1command to see if that improves it. -
@Phaedrux internal SD card works great but external SD has issues.
PanelDue is not problem in this case.
I enabled debugPrint in CoreNG and M111 S1 P10 (Storage) in order to find bug.
This log below is while simulating and refreshing DWC (sd_mmc errors)
Read 1 1 97118<LF> Read 1 1 97119<LF> Read 1 1 9508<LF> Read 1 1 97120<LF> Read 1 1 97121<LF> Read 1 1 97122<LF> Read 1 1 97123<LF> Read 1 1 97124<LF> Read 1 1 97125<LF> Read 1 1 97126<LF> Read 1 1 97127<LF> Read 1 1 9508<LF> Read 1 1 97128<LF> Read 1 1 97129<LF> Read 1 1 97130<LF> Read 1 1 97131<LF> Read 1 1 97132<LF> Read 1 1 97133<LF> Read 0 1 16384<LF> Read 0 1 16385<LF> Read 0 1 35584<LF> Read 0 1 16384<LF> Read 0 1 16385<LF> Read 0 1 35584<LF> Read 0 1 16384<LF> Read 0 1 16385<LF> Read 0 1 35584<LF> Read 0 1 35840<LF> Read 1 1 97134<LF> Read 1 1 97135<LF> Read 1 1 9508<LF> Read 1 1 97136<LF> Read 1 1 97137<LF> Read 0 1 16384<LF> Read 0 1 16385<LF> Read 0 1 35584<LF> Read 0 1 35968<LF> Read 0 1 16384<LF> Read 0 1 16385<LF> Read 0 1 35584<LF> Read 0 1 41088<LF> Read 0 4 36096<LF> Read 0 4 41216<LF> Read 0 4 36100<LF> sd_mmc_spi_adtc_start: cmd 12, arg 0x00000000, R1 timeout<LF> <CR>sd_mmc_spi_adtc_start: cmd 12, arg 0x00000000, R1 timeout<LF> <CR>Read 0 4 41220<LF> Read 1 1 97138<LF> Read 1 1 97139<LF> Read 1 1 97140<LF> Read 1 1 97141<LF> Read 1 1 97142<LF> Read 1 1 97143<LF> Read 1 1 9508<LF> Read 1 1 97144<LF> Read 1 1 97145<LF>I figured out that ctrl_access.c is vulnerable and two task might have access in the same time so there can be a race condition.
I also noticed that there is support for FreeRTOS but is disabled by default. I enabled FreeRTOS from FreeRTOS project and now it works flawlessly.
I'm testing this solution on my desk now, but I looks promising.
Now I can simulating and refreshing in the same time and I works stable. I can even upload files on internal SD card during simulating from external SD card - it takes a bit longer but it is still stable.I did this modyfication on 2.05 firmware.
-
I spotted a week or two ago that the Atmel/Microchip SD card stack shares data across multiple devices (i.e. HSMCI and SPI) and flagged this for investigation. So yes, I do think there is likely to be an issue with concurrent access to the external and on-board SD card. My preference is to stop using shared data, because there is already a per-volume mutex, and I'd prefer that using the external SD card for something didn't slow down the internal one. I'll schedule this for implementation in firmware 3.3.
Duet WiFi hardware designer and firmware engineer
Please do not ask me for Duet support via PM or email, use the forum
http://www.escher3d.com, https://miscsolutions.wordpress.com -
@dc42 Okey, It sounds great.
So what do you prefer? Should I write some bypass (just omit ctrl_access.c) for this library and use direct fuctions between FatFS and HSMCI/SPI?
In this way I don't need to use extra mutex and internal SD card won't slow down.Do you see any dangerous?
Are you going to implement this only for 3.3? -
I think the problem is the static variables in sd_mmc.c:
//! SD/MMC card list //DC added __nocache for SAME70 because 'csd' is read by DMA static __nocache struct sd_mmc_card sd_mmc_cards[SD_MMC_MEM_CNT]; //! Index of current slot selected static uint8_t sd_mmc_slot_sel; //! Pointer on current slot selected static struct sd_mmc_card *sd_mmc_card; //! Number of block to read or write on the current transfer static uint16_t sd_mmc_nb_block_to_tranfer = 0; //! Number of block remaining to read or write on the current transfer static uint16_t sd_mmc_nb_block_remaining = 0;The first one should be OK because there is one entry per SD card, but the other four are not, assuming they are used for both HSMCI and SPI transfers. So I think that sd_mmc_slot_sel needs to be passed around as a parameter, then the second one replaced by indexing into sd_mmc_cards (i.e. initialise a local sd_mmc_card variable at the start of each function that uses it), then the other two can be turned into arrays indexed by slot (strictly speaking, one element per channel would be enough, but one per slot is simpler).
I will probably only implement it for 3.3, although it's looking likely that there will be a 3.2.1, so the fix may get back-ported to that if it is simple enough.
Duet WiFi hardware designer and firmware engineer
Please do not ask me for Duet support via PM or email, use the forum
http://www.escher3d.com, https://miscsolutions.wordpress.com -
