Firmware restart while refreshing DWC and pausing

felek

Hi!

Probably I've found firmware bug which can restart duet2.

My setup:
Duet2 Ethernet - firmware 2.05.1 (flashed in factory)
SD card has the same factory files (I've only changed IP address)
PanelDue 1.24 (flashed in factory)

Reproduction:

Select SD1 card (external) and start simulation (I use PanelDue for that)
During simulation try to refres DWC. Probably it got stuck.
Simulation still works fine, however I can't get access to DWC
Put M122 via USB (You get information that Owned Mutexes: SD0(Network))
Using USB send M25 which should pause print but it hangs firmware
After ~15s firmware do software reset with stack trace.

Using map file it looks like it got stuck in sd_mmc library and cannot give mutex.
When we send M25 firmware wants to write/read something from SD0 and mutex causes hang.
I tried to add extra mutex for HSMCI but It doesn't work and I have no idea how to fix it.
@dc42 Any idea how to help me?
Don't hesitate if you want more info.

First of all, I don't want to go to firmware 3.x because 2.x works fine for me.
I also suspect that firmware 3.x is vulnerable.

I attached log file from YAT. There is M122 after refresh DWC then M25 and M122 after reboot

Network running, IP address = 192.168.60.14<LF>
RepRapFirmware for Duet 2 WiFi/Ethernet Version 2.05.1 dated 2020-02-09b1<LF>
<LF>
Executing config.g...Error: Bad command: FOR TESTING ONLY! USE https://configurator.reprapfirmware.org/ to generate configuration files for your printer!<LF>
Warning: Heater 0 appears to be over-powered. If left on at full power, its temperature is 
M122<LF>
=== Diagnostics ===<LF>
RepRapFirmware for Duet 2 WiFi/Ethernet version 2.05.1 running on Duet Ethernet 1.02 or later<LF>
Board ID: 0JD0M-9P6M2-NWNS4-7J1DG-3S86K-9A1RL<LF>
Used output buffers: 1 of 24 (11 max)<LF>
=== RTOS ===<LF>
Static ram: 25712<LF>
Dynamic ram: 92504 of which 120 recycled<LF>
Exception stack ram used: 288<LF>
Never used ram: 12448<LF>
Tasks: NETWORK(ready,628) HEAT(blocked,912) MAIN(running,3800) IDLE(ready,160)<LF>
Owned mutexes: SD0(NETWORK)<LF>
=== Platform ===<LF>
Last reset 00:00:46 ago, cause: software<LF>
Last software reset at 2021-01-28 11:40, reason: User, spinning module GCodes, available RAM 12472 bytes (slot 0)<LF>
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x04433000 BFAR 0xe000ed38 SP 0xffffffff Task 0x4e49414d<LF>
Error status: 0<LF>
Free file entries: 7<LF>
SD card 0 detected, interface speed: 20.0MBytes/sec<LF>
SD card longest block write time: 0.0ms, max retries 0<LF>
MCU temperature: min 40.8, current 41.1, max 41.6<LF>
Supply voltage: min 1.7, current 1.7, max 1.8, under voltage events: 0, over voltage events: 0, power good: no<LF>
Driver 0: ok, SG min/max not available<LF>
Driver 1: ok, SG min/max not available<LF>
Driver 2: ok, SG min/max not available<LF>
Driver 3: ok, SG min/max not available<LF>
Driver 4: ok, SG min/max not available<LF>
Date/time: 2021-01-28 11:40:52<LF>
Cache data hit count 79851251<LF>
Slowest loop: 149.06ms; fastest: 0.05ms<LF>
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0<LF>
=== Move ===<LF>
Hiccups: 0, FreeDm: 160, MinFreeDm: 160, MaxWait: 29752ms<LF>
Bed compensation in use: none, comp offset 0.000<LF>
=== DDARing ===<LF>
Scheduled moves: 6233, completed moves: 6193, StepErrors: 0, LaErrors: 0, Underruns: 0, 0<LF>
=== Heat ===<LF>
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1<LF>
Heater 1 is on, I-accum = 0.0<LF>
=== GCodes ===<LF>
Segments left: 0<LF>
Stack records: 1 allocated, 0 in use<LF>
Movement lock held by null<LF>
http is idle in state(s) 0<LF>
telnet is idle in state(s) 0<LF>
file is idle in state(s) 0<LF>
serial is ready with "M122" in state(s) 0<LF>
aux is idle in state(s) 0<LF>
daemon is idle in state(s) 0<LF>
queue is idle in state(s) 0<LF>
autopause is idle in state(s) 0<LF>
Code queue is empty.<LF>
=== Network ===<LF>
Slowest loop: 6.01ms; fastest: 0.01ms<LF>
Responder states: HTTP(2) HTTP(2) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)<LF>
HTTP sessions: 0 of 8<LF>
Interface state 5, link 100Mbps full duplex<LF>
ok<LF>
M25<LF>
RepRapFirmware for Duet 2 WiFi/Ethernet Version 2.05.1 dated 2020-02-09b1<LF>
<LF>
Executing config.g...Error: Bad command: FOR TESTING ONLY! USE https://configurator.reprapfirmware.org/ to generate configuration files for your printer!<LF>
Warning: Heater 0 appears to be over-powered. If left on at full power, its temperature is 
M122<LF>
=== Diagnostics ===<LF>
RepRapFirmware for Duet 2 WiFi/Ethernet version 2.05.1 running on Duet Ethernet 1.02 or later<LF>
Board ID: 0JD0M-9P6M2-NWNS4-7J1DG-3S86K-9A1RL<LF>
Used output buffers: 1 of 24 (3 max)<LF>
=== RTOS ===<LF>
Static ram: 25712<LF>
Dynamic ram: 92460 of which 164 recycled<LF>
Exception stack ram used: 256<LF>
Never used ram: 12480<LF>
Tasks: NETWORK(ready,1616) HEAT(blocked,912) MAIN(running,3824) IDLE(ready,160)<LF>
Owned mutexes:<LF>
=== Platform ===<LF>
Last reset 00:00:06 ago, cause: software<LF>
Last software reset at 2021-01-28 11:40, reason: Stuck in spin loop, spinning module GCodes, available RAM 12076 bytes (slot 1)<LF>
Software reset code 0x0043 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0043380f BFAR 0xe000ed38 SP 0x20001efc Task 0x5754454e<LF>
Stack: 00446a47 0044a4ca 21000000 00000000 4031b980 00000000 bff00000 3331bb4c 40000000 3f317200 b5ddea0e 388aa908 4225826c 00000000 476a0b00 37d33333 3c113333 00000000 3ffc5c00 60000010 00000004 00000000 00000004<LF>
Error status: 0<LF>
Free file entries: 10<LF>
SD card 0 detected, interface speed: 20.0MBytes/sec<LF>
SD card longest block write time: 0.0ms, max retries 0<LF>
MCU temperature: min 40.9, current 41.3, max 41.5<LF>
Supply voltage: min 1.7, current 1.7, max 1.7, under voltage events: 0, over voltage events: 0, power good: no<LF>
Driver 0: ok, SG min/max not available<LF>
Driver 1: ok, SG min/max not available<LF>
Driver 2: ok, SG min/max not available<LF>
Driver 3: ok, SG min/max not available<LF>
Driver 4: ok, SG min/max not available<LF>
Date/time: 1970-01-01 00:00:00<LF>
Cache data hit count 9762783<LF>
Slowest loop: 1.02ms; fastest: 0.05ms<LF>
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0<LF>
=== Move ===<LF>
Hiccups: 0, FreeDm: 160, MinFreeDm: 160, MaxWait: 0ms<LF>
Bed compensation in use: none, comp offset 0.000<LF>
=== DDARing ===<LF>
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0<LF>
=== Heat ===<LF>
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1<LF>
Heater 1 is on, I-accum = 0.0<LF>
=== GCodes ===<LF>
Segments left: 0<LF>
Stack records: 1 allocated, 0 in use<LF>
Movement lock held by null<LF>
http is idle in state(s) 0<LF>
telnet is idle in state(s) 0<LF>
file is idle in state(s) 0<LF>
serial is ready with "M122" in state(s) 0<LF>
aux is idle in state(s) 0<LF>
daemon is idle in state(s) 0<LF>
queue is idle in state(s) 0<LF>
autopause is idle in state(s) 0<LF>
Code queue is empty.<LF>
=== Network ===<LF>
Slowest loop: 0.44ms; fastest: 0.01ms<LF>
Responder states: HTTP(1) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)<LF>
HTTP sessions: 0 of 8<LF>
Interface state 5, link 100Mbps full duplex<LF>
ok<LF>

felek

@Phaedrux My mistake. Can you change category to Firmware Developers?

Phaedrux

@felek said in Firmware restart while refreshing DWC and pausing:

First of all, I don't want to go to firmware 3.x because 2.x works fine for me.

Except for the bug you mean?

I also suspect that firmware 3.x is vulnerable.

I don't understand. What is vulnerable?

2.05 is over a year old now. All work on RRF2 has stopped and moved to RRF3. The bug you've found is likely fixed in 3.2.

If you still have access to DWC. Upload these 3 zip files, one at a time. Don't extract them. Reboot after each. Use M115 to verify the firmware has been applied.
https://github.com/Duet3D/RepRapFirmware/releases/download/2.05.1/Duet2Firmware-2.05.1.zip
https://github.com/Duet3D/RepRapFirmware/releases/download/3.0/Duet2and3Firmware-3.0.zip
https://github.com/Duet3D/RepRapFirmware/releases/download/3.2/Duet2and3Firmware-3.2.zip
That will get your firmware and DWC up to date.

For your config, might be a good idea to run through the configurator tool and generate a fresh set for RRF3.
https://configtool.reprapfirmware.org/Start

felek

@Phaedrux Thanks!

So, I've updated RepRap firmware(3.2), PanelDue(3.2.9) and DWC(3.2.0) in standalone mode (without SBC)

I put micro SD card to the PanelDue and started simulation. After a few seconds I lost DWC communication and main board rebooted. I tested different sd cards, cables and it doesn't work.

I repeated this a few times and firmware still reboot.
External SD card is slow but it shouldn't cause a reboot.

config.g

; Configuration file for Duet WiFi (firmware version 3)
; executed by the firmware on start-up
;
; generated by RepRapFirmware Configuration Tool v3.1.10 on Sat Dec 05 2020 19:46:54 GMT+0100 (Central European Standard Time)

; General preferences
G90                                           ; send absolute coordinates...
M83                                           ; ...but relative extruder moves
M550 P"TT Sapphire"                        ; set printer name
M669 K1                                       ; select CoreXY mode

M552 P192.168.60.14						; Set IP address
M553 P255.255.255.0						; Set Netmask
M552 S1								; Turn network on
M575 P1 B57600 S1					; Turn PanelDue on

; Drives
M569 P0 S0                                    ; physical drive 0 goes forwards
M569 P1 S0                                    ; physical drive 1 goes forwards
M569 P2 S0                                    ; physical drive 2 goes forwards
M569 P3 S0                                    ; physical drive 3 goes forwards
M584 X0 Y1 Z2 E3                              ; set drive mapping
M350 X32 Y32 Z32 E16 I1                       ; configure microstepping with interpolation
M92 X160.60 Y160.60 Z3200.00 E392.40          ; set steps per mm
M566 X800.00 Y800.00 Z60.00 E420.00 P1      ; set maximum instantaneous speed changes (mm/min)
M203 X18000.00 Y18000.00 Z1200.00 E30000.00   ; set maximum speeds (mm/min)
M201 X2500.00 Y2500.00 Z200.00 E10000.00      ; set accelerations (mm/s^2)
M204 P2000 T4000							  ; set acceleration for printing moves
M906 X1200 Y1200 Z1200 E800 I50               ; set motor currents (mA) and motor idle factor in per cent
M84 S30                                       ; Set idle timeout

; Axis Limits
M208 X0 Y0 Z0 S1                              ; set axis minima
M208 X223 Y223 Z235 S0                        ; set axis maxima

; Endstops
M574 X1 S1 P"!xstop" 
M574 Y2 S1 P"!ystop" 
M574 Z1 S1 P"!zstop"                           ; configure active-high endstop for low end on Z via pin zstop

; Z-Probe
M558 P1 C"zprobe.in" H5 F120 T6000            ; set Z probe type to unmodulated and the dive height + speeds
G31 P500 X0 Y0 Z2.5                           ; set Z probe trigger value, offset and trigger height
M557 X15:215 Y15:195 S20                      ; define mesh grid

; Heaters
M308 S0 P"bedtemp" Y"thermistor" T100000 B4138    ; configure sensor 0 as PT1000 on pin bedtemp
M950 H0 C"bedheat" T0                         ; create bed heater output on bedheat and map it to sensor 0
M307 H0 B0 S1.00                              ; disable bang-bang mode for the bed heater and set PWM limit
M140 H0                                       ; map heated bed to heater 0
M143 H0 S120                                  ; set temperature limit for heater 0 to 120C
M308 S1 P"e0temp" Y"thermistor" T100000 B4138     ; configure sensor 1 as thermistor on pin e0temp
M950 H1 C"e0heat" T1                          ; create nozzle heater output on e0heat and map it to sensor 1
M307 H1 B0 S1.00                              ; disable bang-bang mode for heater  and set PWM limit
M143 H1 S280                                  ; set temperature limit for heater 1 to 280C

; Fans
M950 F0 C"fan0" Q500                          ; create fan 0 on pin fan0 and set its frequency
M106 P0 S0 H-1                                ; set fan 0 value. Thermostatic control is turned off
M950 F1 C"fan1" Q500                          ; create fan 1 on pin fan1 and set its frequency
M106 P1 S1 H-1                                ; set fan 1 value. Thermostatic control is turned off

; Tools
M563 P0 D0 H1 F0                              ; define tool 0
G10 P0 X0 Y0 Z0                               ; set tool 0 axis offsets
G10 P0 R0 S0                                  ; set initial tool 0 active and standby temperatures to 0C

; Custom settings are not defined

; Miscellaneous
M501                                          ; load saved parameters from non-volatile memory

M593 F51	  	; cancel ringing at 51Hz
;M572 D0 S0.7

M122 after reboot

RepRapFirmware for Duet 2 WiFi/Ethernet version 3.2 running on Duet Ethernet 1.02 or later
Board ID: 0JD0M-9P6M2-NWNS4-7J1DG-3S86K-9A1RL
Used output buffers: 3 of 24 (15 max)
=== RTOS ===
Static ram: 23460
Dynamic ram: 68656 of which 0 recycled
Never used RAM 19964, free system stack 208 words
Tasks: NETWORK(ready,186) HEAT(blocked,331) MAIN(running,474) IDLE(ready,19)
Owned mutexes:
=== Platform ===
Last reset 00:00:13 ago, cause: software
Last software reset at 2021-01-29 10:32, reason: StuckInSpinLoop, GCodes spinning, available RAM 19964, slot 1
Software reset code 0x4083 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0040080f BFAR 0xe000ed38 SP 0x20003424 Task IDLE Freestk 33 ok
Stack: 00454679 004550bc 61000000 a5a5a5a5 00454679 a5a5a5a5 a5a5a5a5 20003e04 0000714e 200046e8 20001804 20003440 200046e0 00000004 2000cbe0 2000cbe0 20003440 2000cbd8 00000001 200034a0 4e49414d 00000000 00000000 00000001 00000000 00000001 00000003
Error status: 0x00
Aux0 errors 0,0,0
MCU temperature: min 41.8, current 42.3, max 42.5
Supply voltage: min 1.7, current 1.7, max 1.8, under voltage events: 0, over voltage events: 0, power good: no
Driver 0: position 0, ok, SG min/max not available
Driver 1: position 0, ok, SG min/max not available
Driver 2: position 0, ok, SG min/max not available
Driver 3: position 0, ok, SG min/max not available
Driver 4: position 0, ok, SG min/max not available
Driver 5: position 0
Driver 6: position 0
Driver 7: position 0
Driver 8: position 0
Driver 9: position 0
Driver 10: position 0
Driver 11: position 0
Date/time: 2021-01-29 10:32:39
Cache data hit count 20356317
Slowest loop: 3.95ms; fastest: 0.13ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 3.2ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 6.41ms; fastest: 0.01ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
Interface state active, link 100Mbps full duplex

Veti

@felek said in Firmware restart while refreshing DWC and pausing:

Supply voltage: min 1.7, current 1.7, max 1.8, under voltage events: 0, over voltage events: 0, power good: no

how are you powering the duet?

felek

@Veti I test setup on my desk

Veti

have you tested with a real power supply?

also
M350 X32 Y32 Z32 E16 I1 ; configure microstepping with interpolation
this is not valid for the duet 2. interpolation only works at x16

felek

@Veti I removed I1 from config.g

I tested on 3d printer and It still ocurrs

Simulation works fine on 2.05.1 firmware, however there is issue when I want to load DWC (or refres).

M122

RepRapFirmware for Duet 2 WiFi/Ethernet version 3.2 running on Duet WiFi 1.02 or later
Board ID: 0JD0M-9K662-MGPSS-6J1FJ-3SN6M-9SSUW
Used output buffers: 5 of 24 (11 max)
=== RTOS ===
Static ram: 23460
Dynamic ram: 73392 of which 0 recycled
Never used RAM 15228, free system stack 193 words
Tasks: NETWORK(ready,179) HEAT(blocked,295) MAIN(running,452) IDLE(ready,19)
Owned mutexes:
=== Platform ===
Last reset 00:00:11 ago, cause: software
Last software reset at 2021-01-29 11:37, reason: StuckInSpinLoop, GCodes spinning, available RAM 15228, slot 0
Software reset code 0x4083 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f80f BFAR 0xe000ed38 SP 0x20003424 Task IDLE Freestk 33 ok
Stack: 00454679 004550bc 61000000 a5a5a5a5 00454679 a5a5a5a5 a5a5a5a5 20003eac 00012d50 200046e8 20001804 20003440 200046e0 00000004 20001140 20001140 20003440 20001138 00000001 200034a0 4e49414d 00000000 00000000 00000001 00000000 00000001 00000003
Error status: 0x00
Aux0 errors 0,0,0
MCU temperature: min 22.1, current 22.8, max 22.9
Supply voltage: min 24.1, current 24.1, max 24.3, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: position 0, standstill, SG min/max not available
Driver 1: position 0, standstill, SG min/max not available
Driver 2: position 0, standstill, SG min/max not available
Driver 3: position 0, standstill, SG min/max not available
Driver 4: position 0, standstill, SG min/max not available
Driver 5: position 0
Driver 6: position 0
Driver 7: position 0
Driver 8: position 0
Driver 9: position 0
Driver 10: position 0
Driver 11: position 0
Date/time: 2021-01-29 11:37:47
Cache data hit count 12795529
Slowest loop: 99.99ms; fastest: 0.20ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 0.8ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 39.85ms; fastest: 0.00ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- WiFi -
Network state is active
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
WiFi firmware version 1.25beta0
WiFi MAC address 40:f5:20:3a:0b:d7
WiFi Vcc 3.29, reset reason Power up
WiFi flash size 4194304, free heap 22048
WiFi IP address 192.168.1.28
WiFi signal strength -75dBm, mode 802.11n, reconnections 0, sleep mode modem
Clock register 00002002
Socket states: 0 0 0 0 0 0 0 0

felek

I tried to print but duet rebooted

AFAIK when duet has issue with external SD card it should indicate error like: "Cannot read file, error code X"

29.01.2021, 12:00:41: m122: === Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.2 running on Duet WiFi 1.02 or later
Board ID: 0JD0M-9K662-MGPSS-6J1FJ-3SN6M-9SSUW
Used output buffers: 3 of 24 (14 max)
=== RTOS ===
Static ram: 23460
Dynamic ram: 73392 of which 0 recycled
Never used RAM 15228, free system stack 189 words
Tasks: NETWORK(ready,186) HEAT(blocked,309) MAIN(running,452) IDLE(ready,19)
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:00:35 ago, cause: software
Last software reset at 2021-01-29 11:59, reason: StuckInSpinLoop, PrintMonitor spinning, available RAM 15192, slot 0
Software reset code 0x4089 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f80f BFAR 0xe000ed38 SP 0x20003424 Task IDLE Freestk 33 ok
Stack: 00454679 004550bc 61000000 a5a5a5a5 00454679 a5a5a5a5 a5a5a5a5 20004334 0005f2d0 200046e8 20001804 20003440 200046e0 00000004 2000cbe0 2000cbe0 20003440 2000cbd8 00000001 200034a0 4e49414d 00000000 00000000 00000001 00000000 00000001 00000002
Error status: 0x00
Aux0 errors 0,0,0
MCU temperature: min 24.7, current 25.5, max 25.8
Supply voltage: min 24.0, current 24.1, max 24.3, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: position 0, standstill, SG min/max not available
Driver 1: position 0, standstill, SG min/max not available
Driver 2: position 0, standstill, SG min/max not available
Driver 3: position 0, standstill, SG min/max not available
Driver 4: position 0, standstill, SG min/max not available
Driver 5: position 0
Driver 6: position 0
Driver 7: position 0
Driver 8: position 0
Driver 9: position 0
Driver 10: position 0
Driver 11: position 0
Date/time: 2021-01-29 12:00:41
Cache data hit count 42753898
Slowest loop: 96.96ms; fastest: 0.20ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 0.5ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 39.81ms; fastest: 0.00ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- WiFi -
Network state is active
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
WiFi firmware version 1.25beta0
WiFi MAC address 40:f5:20:3a:0b:d7
WiFi Vcc 3.29, reset reason Power up
WiFi flash size 4194304, free heap 24616
WiFi IP address 192.168.1.28
WiFi signal strength -72dBm, mode 802.11n, reconnections 0, sleep mode modem
Clock register 00002002
Socket states: 0 0 0 0 0 0 0 0

dc42

Please post the GCode file that caused the reset when you simulated it.

felek

@dc42 This file is quite big
https://www.dropbox.com/s/twebf0ajjvn0bdd/moon6inches.gcode?dl=0

felek

@dc42 I was digging in the firmware in order to find solution. I have no idea why Network task cannot Give SD0 mutex when external SD is simulating. It is happen when refresh DWC. Firmware want to send all files to the browser and it got stuck in void NetworkResponder::SendData()

I also don't have old releases duet PCB with JTAG socket so I'm not able to use debugger.

Phaedrux

@Veti said in Firmware restart while refreshing DWC and pausing:

M350 X32 Y32 Z32 E16 I1

@felek Rather than remove the I1 to stop the interpolation, better to use x16 microstepping with interpolation enabled to begin with.

@felek said in Firmware restart while refreshing DWC and pausing:

PanelDue(3.2.9)

Would you be able to test again with PanelDue 1.24?

felek

@Phaedrux Ok, I changed microstepping to 16 and enabled interpolation.

Here is my config.g:

; Configuration file for Duet WiFi (firmware version 3)
; executed by the firmware on start-up
;
; generated by RepRapFirmware Configuration Tool v3.1.10 on Sat Dec 05 2020 19:46:54 GMT+0100 (Central European Standard Time)

; General preferences
G90                                           ; send absolute coordinates...
M83                                           ; ...but relative extruder moves
M550 P"TT Sapphire"                        ; set printer name
M669 K1                                       ; select CoreXY mode

; Network
M552 S1                                       ; enable network
M586 P0 S1                                    ; enable HTTP
M586 P1 S1                                    ; enable FTP
M586 P2 S0                                    ; disable Telnet
M575 P1 B57600 S1					; Turn PanelDue on

; Drives
M569 P0 S0                                    ; physical drive 0 goes forwards
M569 P1 S0                                    ; physical drive 1 goes forwards
M569 P2 S0                                    ; physical drive 2 goes forwards
M569 P3 S0                                    ; physical drive 3 goes forwards
M584 X0 Y1 Z2 E3                              ; set drive mapping
;M350 X32 Y32 Z32 E16 I1	                      ; configure microstepping with interpolation
M350 X16 Y16 Z16 E16 I1	                      ; configure microstepping with interpolation
M92 X160.60 Y160.60 Z3200.00 E392.40          ; set steps per mm
M566 X800.00 Y800.00 Z60.00 E420.00 P1      ; set maximum instantaneous speed changes (mm/min)
M203 X18000.00 Y18000.00 Z1200.00 E30000.00   ; set maximum speeds (mm/min)
M201 X2500.00 Y2500.00 Z200.00 E10000.00      ; set accelerations (mm/s^2)
M204 P2000 T4000							  ; set acceleration for printing moves
M906 X1200 Y1200 Z1200 E800 I50               ; set motor currents (mA) and motor idle factor in per cent
M84 S30                                       ; Set idle timeout

; Axis Limits
M208 X0 Y0 Z0 S1                              ; set axis minima
M208 X223 Y223 Z235 S0                        ; set axis maxima

; Endstops
M574 X1 S1 P"!xstop" 
M574 Y2 S1 P"!ystop" 
M574 Z1 S1 P"!zstop"                           ; configure active-high endstop for low end on Z via pin zstop

; Z-Probe
M558 P1 C"zprobe.in" H5 F120 T6000            ; set Z probe type to unmodulated and the dive height + speeds
G31 P500 X0 Y0 Z2.5                           ; set Z probe trigger value, offset and trigger height
M557 X15:215 Y15:195 S20                      ; define mesh grid

; Heaters
M308 S0 P"bedtemp" Y"thermistor" T100000 B4138    ; configure sensor 0 as PT1000 on pin bedtemp
M950 H0 C"bedheat" T0                         ; create bed heater output on bedheat and map it to sensor 0
M307 H0 B0 S1.00                              ; disable bang-bang mode for the bed heater and set PWM limit
M140 H0                                       ; map heated bed to heater 0
M143 H0 S120                                  ; set temperature limit for heater 0 to 120C
M308 S1 P"e0temp" Y"thermistor" T100000 B4138     ; configure sensor 1 as thermistor on pin e0temp
M950 H1 C"e0heat" T1                          ; create nozzle heater output on e0heat and map it to sensor 1
M307 H1 B0 S1.00                              ; disable bang-bang mode for heater  and set PWM limit
M143 H1 S280                                  ; set temperature limit for heater 1 to 280C

; Fans
M950 F0 C"fan0" Q500                          ; create fan 0 on pin fan0 and set its frequency
M106 P0 S0 H-1                                ; set fan 0 value. Thermostatic control is turned off
M950 F1 C"fan1" Q500                          ; create fan 1 on pin fan1 and set its frequency
M106 P1 S1 H-1                                ; set fan 1 value. Thermostatic control is turned off

; Tools
M563 P0 D0 H1 F0                              ; define tool 0
G10 P0 X0 Y0 Z0                               ; set tool 0 axis offsets
G10 P0 R0 S0                                  ; set initial tool 0 active and standby temperatures to 0C

; Custom settings are not defined

; Miscellaneous
M501                                          ; load saved parameters from non-volatile memory

M593 F51	  	; cancel ringing at 51Hz
;M572 D0 S0.7

I downgraded PanelDue. Now I'm using 1.24 version but the issue still exist.

Here is M122:

RepRapFirmware for Duet 2 WiFi/Ethernet version 3.2 running on Duet WiFi 1.02 or later
Board ID: 0JD0M-9K662-MGPSS-6J1FJ-3SN6M-9SSUW
Used output buffers: 3 of 24 (11 max)
=== RTOS ===
Static ram: 23460
Dynamic ram: 73392 of which 0 recycled
Never used RAM 15228, free system stack 195 words
Tasks: NETWORK(ready,182) HEAT(blocked,295) MAIN(running,466) IDLE(ready,19)
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:00:19 ago, cause: software
Last software reset at 2021-02-01 09:45, reason: StuckInSpinLoop, GCodes spinning, available RAM 15192, slot 1
Software reset code 0x4083 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f80f BFAR 0xe000ed38 SP 0x20003ea4 Task MAIN Freestk 641 ok
Stack: 0042885f 0042885e 21060000 3f800000 00000000 41200000 3d20a0a1 40f6fb48 44e0f026 44e0f026 3dbdae8d 44e0f026 3ca0a0a1 00000000 00000000 00000000 38533333 3f800000 3f800000 60000010 200025a8 000773d0 200025a8 20000f0c 200025a8 200025b8 200025b2
Error status: 0x00
Aux0 errors 0,0,0
MCU temperature: min 19.9, current 20.8, max 21.1
Supply voltage: min 24.1, current 24.2, max 24.2, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: position 0, standstill, SG min/max not available
Driver 1: position 0, standstill, SG min/max not available
Driver 2: position 0, standstill, SG min/max not available
Driver 3: position 0, standstill, SG min/max not available
Driver 4: position 0, standstill, SG min/max not available
Driver 5: position 0
Driver 6: position 0
Driver 7: position 0
Driver 8: position 0
Driver 9: position 0
Driver 10: position 0
Driver 11: position 0
Date/time: 2021-02-01 09:46:30
Cache data hit count 22936358
Slowest loop: 2.95ms; fastest: 0.20ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 0.7ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 139.39ms; fastest: 0.00ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- WiFi -
Network state is active
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
WiFi firmware version 1.25beta0
WiFi MAC address 40:f5:20:3a:0b:d7
WiFi Vcc 3.29, reset reason Power up
WiFi flash size 4194304, free heap 21968
WiFi IP address 192.168.1.28
WiFi signal strength -70dBm, mode 802.11n, reconnections 0, sleep mode modem
Clock register 00002002
Socket states: 0 0 0 0 0 0 0 0

felek

@Phaedrux

Here is different reset reason:

RepRapFirmware for Duet 2 WiFi/Ethernet version 3.2 running on Duet WiFi 1.02 or later
Board ID: 0JD0M-9K662-MGPSS-6J1FJ-3SN6M-9SSUW
Used output buffers: 3 of 24 (11 max)
=== RTOS ===
Static ram: 23460
Dynamic ram: 73392 of which 0 recycled
Never used RAM 15228, free system stack 200 words
Tasks: NETWORK(ready,192) HEAT(blocked,309) MAIN(running,448) IDLE(ready,19)
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:00:25 ago, cause: software
Last software reset at 2021-02-01 09:51, reason: HardFault bfarValid precise, PrintMonitor spinning, available RAM 15228, slot 1
Software reset code 0x4069 HFSR 0x40000000 CFSR 0x00008200 ICSR 0x0441f803 BFAR 0x9d03b488 SP 0x200043c0 Task MAIN Freestk 968 ok
Stack: 20020000 00000000 2000444f 9d03b470 00000046 0041fbb5 00428840 610e0000 bf800000 00000000 3edb6db7 b8ac9c00 3331bb4c 40000000 3f317200 b5ddea0e 388a851a bb35d196 3d5dbc4d 00000000 00000000 38533333 00000000 00000000 60000010 200025a8 0007a0e7
Error status: 0x00
Aux0 errors 0,0,0
MCU temperature: min 20.7, current 21.4, max 21.7
Supply voltage: min 24.1, current 24.2, max 24.2, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: position 0, standstill, SG min/max not available
Driver 1: position 0, standstill, SG min/max not available
Driver 2: position 0, standstill, SG min/max not available
Driver 3: position 0, standstill, SG min/max not available
Driver 4: position 0, standstill, SG min/max not available
Driver 5: position 0
Driver 6: position 0
Driver 7: position 0
Driver 8: position 0
Driver 9: position 0
Driver 10: position 0
Driver 11: position 0
Date/time: 2021-02-01 09:51:32
Cache data hit count 30320382
Slowest loop: 3.20ms; fastest: 0.20ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 0.7ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 140.03ms; fastest: 0.00ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- WiFi -
Network state is active
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
WiFi firmware version 1.25beta0
WiFi MAC address 40:f5:20:3a:0b:d7
WiFi Vcc 3.29, reset reason Power up
WiFi flash size 4194304, free heap 25632
WiFi IP address 192.168.1.28
WiFi signal strength -69dBm, mode 802.11n, reconnections 0, sleep mode modem
Clock register 00002002
Socket states: 0 0 0 0 0 0 0 0

Phaedrux

@felek said in Firmware restart while refreshing DWC and pausing:

@dc42 This file is quite big
https://www.dropbox.com/s/twebf0ajjvn0bdd/moon6inches.gcode?dl=0

Wow, 106Mb gcode file. I'll try simulating it myself.

Phaedrux

I was able to simulate that gcode file on a Duet2 with fw 3.2 on the internal SD card without error. I don't have the ribbon cable connected for the PanelDue SD card slot, so I can't test that.

You could try reducing the baud rate on the PanelDue and in your M575 P1 B57600 S1 command to see if that improves it.

felek

@Phaedrux internal SD card works great but external SD has issues.

PanelDue is not problem in this case.

I enabled debugPrint in CoreNG and M111 S1 P10 (Storage) in order to find bug.

This log below is while simulating and refreshing DWC (sd_mmc errors)

Read 1 1 97118<LF>
Read 1 1 97119<LF>
Read 1 1 9508<LF>
Read 1 1 97120<LF>
Read 1 1 97121<LF>
Read 1 1 97122<LF>
Read 1 1 97123<LF>
Read 1 1 97124<LF>
Read 1 1 97125<LF>
Read 1 1 97126<LF>
Read 1 1 97127<LF>
Read 1 1 9508<LF>
Read 1 1 97128<LF>
Read 1 1 97129<LF>
Read 1 1 97130<LF>
Read 1 1 97131<LF>
Read 1 1 97132<LF>
Read 1 1 97133<LF>
Read 0 1 16384<LF>
Read 0 1 16385<LF>
Read 0 1 35584<LF>
Read 0 1 16384<LF>
Read 0 1 16385<LF>
Read 0 1 35584<LF>
Read 0 1 16384<LF>
Read 0 1 16385<LF>
Read 0 1 35584<LF>
Read 0 1 35840<LF>
Read 1 1 97134<LF>
Read 1 1 97135<LF>
Read 1 1 9508<LF>
Read 1 1 97136<LF>
Read 1 1 97137<LF>
Read 0 1 16384<LF>
Read 0 1 16385<LF>
Read 0 1 35584<LF>
Read 0 1 35968<LF>
Read 0 1 16384<LF>
Read 0 1 16385<LF>
Read 0 1 35584<LF>
Read 0 1 41088<LF>
Read 0 4 36096<LF>
Read 0 4 41216<LF>
Read 0 4 36100<LF>
sd_mmc_spi_adtc_start: cmd 12, arg 0x00000000, R1 timeout<LF>
<CR>sd_mmc_spi_adtc_start: cmd 12, arg 0x00000000, R1 timeout<LF>
<CR>Read 0 4 41220<LF>
Read 1 1 97138<LF>
Read 1 1 97139<LF>
Read 1 1 97140<LF>
Read 1 1 97141<LF>
Read 1 1 97142<LF>
Read 1 1 97143<LF>
Read 1 1 9508<LF>
Read 1 1 97144<LF>
Read 1 1 97145<LF>

I figured out that ctrl_access.c is vulnerable and two task might have access in the same time so there can be a race condition.

I also noticed that there is support for FreeRTOS but is disabled by default. I enabled FreeRTOS from FreeRTOS project and now it works flawlessly.

I'm testing this solution on my desk now, but I looks promising.
Now I can simulating and refreshing in the same time and I works stable. I can even upload files on internal SD card during simulating from external SD card - it takes a bit longer but it is still stable.

I did this modyfication on 2.05 firmware.

dc42

I spotted a week or two ago that the Atmel/Microchip SD card stack shares data across multiple devices (i.e. HSMCI and SPI) and flagged this for investigation. So yes, I do think there is likely to be an issue with concurrent access to the external and on-board SD card. My preference is to stop using shared data, because there is already a per-volume mutex, and I'd prefer that using the external SD card for something didn't slow down the internal one. I'll schedule this for implementation in firmware 3.3.

felek

@dc42 Okey, It sounds great.
So what do you prefer? Should I write some bypass (just omit ctrl_access.c) for this library and use direct fuctions between FatFS and HSMCI/SPI?
In this way I don't need to use extra mutex and internal SD card won't slow down.

Do you see any dangerous?
Are you going to implement this only for 3.3?