RRF36 disconnecting mid-print

justGuner

Spoke too soon... It disconnected again

Before troubleshooting again, is there a way to resume the print from the paused state after restarting the machine? I don't want to throw away the current job

Edit: after deciding to just reprint the whole module again, the toolhead disconnected again. This time a few layers higher

justGuner

I attempted to print the same file without any filament loaded at 200% speed. This time, it fails after about 20-30 layers.

Here are the outputs of M122 sent during and after the fail.

M122 during print job:

M122
=== Diagnostics ===
RepRapFirmware for STM32F4 based Boards (biquskr_2) version 3.5.0-rc.3+101 (2024-01-28 17:56:35) running on STM32F4 (standalone mode)
Board ID: X00A4-0N2J2-D0PSN-6J18D-DFARH-40000
Used output buffers: 2 of 40 (31 max)
=== RTOS ===
Static ram: 20140
Dynamic ram: 93464 of which 0 recycled
Never used RAM 16440, free system stack 113 words
CCMRam static ram: 24628 dynamic ram: 38940 free ram 1964
Tasks: NETWORK(2,nWait 7,10.7%,204) HEAT(3,nWait 6,0.1%,298) Move(4,nWait 6,2.7%,243) CanReceiv(6,delaying,2.4%,297) CanSender(5,nWait 7,0.3%,375) CanClock(7,delaying,0.2%,372) TMC22xx(4,nWait 6,1.9%,103) FSWRITE(2,nWait 4,0.0%,495) MAIN(1,running,81.6%,526) IDLE(0,ready,0.2%,29), total 100.0%
Owned mutexes: BITIO(TMC22xx) HTTP(MAIN)
=== Platform ===
Last reset 00:09:30 ago, cause: power on/off
Last software reset at 2024-04-21 14:00, reason: User, Gcodes spinning, available RAM 17176, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00454000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
MCU temperature: min 39.2, current 39.9, max 40.0
Supply voltage: min 24.0, current 24.0, max 24.0, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/1, heap memory allocated/used/recyclable 2048/1644/1612, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: ok 2209, SG min 0, reads 3827, writes 0
Driver 1: ok 2209, SG min 0, reads 3827, writes 0
Driver 2: ok 2209, SG min 0, reads 3827, writes 0
Driver 3: ok 2209, SG min 0, reads 3827, writes 0
Driver 4: not present
Driver 5: 
Driver 6: 
Driver 7: 
Driver 8: 
Driver 9: 
Driver 10: 
Driver 11: 
Driver 12: 
Driver 13: 
Date/time: 2024-04-21 16:58:33
Slowest loop: 7.42ms; fastest: 0.07ms
=== Storage ===
Free file entries: 18
SD card 0 detected
SD card longest read time 1.7ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, segments created 30, maxWait 225ms, bed compensation in use: mesh, height map offset 0.000, max steps late 1, ebfmin 0.00, ebfmax 0.00
next step interrupt due in 10863 ticks, disabled
Moves shaped first try 22, on retry 80, too short 470, wrong shape 530, maybepossible 35
=== DDARing 0 ===
Scheduled moves 5074, completed 5034, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state 3
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.3
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is ready with "M122" in state(s) 0
File is doing "G1 X136.262 Y123.064 E0.04591" in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0, sync state 1
Queue2 is idle in state(s) 0
Q0 segments left 1, axes/extruders owned 0x0004007
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 2026, received 920, lost 0, errs 0, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 327 free buffers 50 (min 38), ts 230/230/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 40.68ms; fastest: 0.08ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0)
HTTP sessions: 1 of 8
Uploads/Errors: 0/0
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Bad header: 0/0
Firmware version 1.27-01S32-D
MAC address 68:58:11:44:3b:b0
Module reset reason: Power up, Vcc 0.00, flash size 0, free heap 146880
WiFi IP address 192.168.1.10
Signal strength -55dBm, channel 0, mode 802.11n, reconnections 0
Clock register 00002003
Socket states: 0 0 0 0 0 0 0 0

M122 after fail:

M122
=== Diagnostics ===
RepRapFirmware for STM32F4 based Boards (biquskr_2) version 3.5.0-rc.3+101 (2024-01-28 17:56:35) running on STM32F4 (standalone mode)
Board ID: X00A4-0N2J2-D0PSN-6J18D-DFARH-40000
Used output buffers: 17 of 40 (31 max)
=== RTOS ===
Static ram: 20140
Dynamic ram: 93536 of which 0 recycled
Never used RAM 16368, free system stack 89 words
CCMRam static ram: 24628 dynamic ram: 38988 free ram 1916
Tasks: NETWORK(2,nWait 7,15.5%,204) HEAT(3,delaying,6.7%,292) Move(4,nWait 6,1.2%,243) CanReceiv(6,delaying,1.3%,297) CanSender(5,nWait 7,0.9%,369) CanClock(7,delaying,11.2%,372) TMC22xx(4,nWait 6,2.4%,103) FSWRITE(2,nWait 4,0.0%,487) MAIN(1,running,60.6%,526) IDLE(0,ready,0.1%,29), total 100.0%
Owned mutexes: WiFi(NETWORK) HTTP(MAIN)
=== Platform ===
Last reset 00:18:03 ago, cause: power on/off
Last software reset at 2024-04-21 14:00, reason: User, Gcodes spinning, available RAM 17176, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00454000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
MCU temperature: min 38.9, current 39.7, max 40.1
Supply voltage: min 24.0, current 24.0, max 24.0, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/1, heap memory allocated/used/recyclable 2048/1644/1612, gc cycles 0
Events: 1 queued, 1 completed
Driver 0: standstill 2209, SG min 0, reads 42737, writes 0
Driver 1: standstill 2209, SG min 0, reads 42737, writes 0
Driver 2: standstill 2209, SG min 0, reads 42737, writes 0
Driver 3: standstill 2209, SG min 0, reads 42738, writes 0
Driver 4: not present
Driver 5: 
Driver 6: 
Driver 7: 
Driver 8: 
Driver 9: 
Driver 10: 
Driver 11: 
Driver 12: 
Driver 13: 
Date/time: 2024-04-21 17:07:05
Slowest loop: 55.65ms; fastest: 0.06ms
=== Storage ===
Free file entries: 18
SD card 0 detected
SD card longest read time 5.1ms, write time 5.1ms, max retries 0
=== Move ===
DMs created 83, segments created 32, maxWait 303ms, bed compensation in use: mesh, height map offset 0.000, max steps late 1, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 129, on retry 232, too short 2672, wrong shape 3689, maybepossible 392
=== DDARing 0 ===
Scheduled moves 13667, completed 13667, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 72, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.2
=== GCodes ===
Movement locks held by null, null
HTTP is ready with "M122" in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0004007
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 9917, received 4473, lost 0, errs 0, boc 1
Longest wait 4ms for reply type 6024, peak Tx sync delay 201 free buffers 50 (min 19), ts 2564/1112/0
Tx timeouts 105,0,1452,0,0,777 last cancelled message type 8191 dest 127
=== Network ===
Slowest loop: 40.04ms; fastest: 0.08ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0)
HTTP sessions: 1 of 8
Uploads/Errors: 0/0
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Bad header: 0/0
Firmware version 1.27-01S32-D
MAC address 68:58:11:44:3b:b0
Module reset reason: Power up, Vcc 0.00, flash size 0, free heap 126576
WiFi IP address 192.168.1.10
Signal strength -55dBm, channel 0, mode 802.11n, reconnections 0
Clock register 00002003
Socket states: 0 0 0 0 0 0 0 0

M122 B124 during print job:

Diagnostics for board 124:
Duet FLY36RRF firmware version 3.5.0-rc.3+101 (2024-01-28 17:59:13)
Bootloader ID: 
All averaging filters OK
Never used RAM 195856, free system stack 392 words
Tasks: Move(3,nWait 7,0.5%,71) HEAT(2,nWait 6,0.3%,109) CanAsync(5,nWait 4,0.0%,65) CanRecv(3,nWait 1,0.2%,357) CanClock(5,nWait 1,0.0%,335) ACCEL(3,nWait 6,0.0%,53) TMC(2,delaying,1.8%,53) MAIN(1,running,94.7%,422) IDLE(0,ready,0.0%,27) USBD(2,blocked,0.0%,151) AIN(2,delaying,2.4%,248), total 100.0%
Owned mutexes:
Last reset 00:11:31 ago, cause: power up or brownout
Last software reset data not available
Driver 0: pos 0, 6501.0 steps/mm, ok, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 37455, writes 0, timeouts 0, DMA errors 0, CC errors 0, steps req 0 done 3150223
Moves scheduled 7008, completed 7001, in progress 1, hiccups 41, segs 29, step errors 0, maxLate 1 maxPrep 507, maxOverdue 39, maxInc 34, mcErrs 0, gcmErrs 0, ebfmin -0.94 max 1.00
Peak sync jitter -14/26, peak Rx sync delay 432, resyncs 0/0, next timer interrupt due in 14 ticks, enabled
VIN voltage: min 23.5, current 23.7, max 24.0
MCU temperature: min 45.4C, current 48.7C, max 48.7C
Last sensors broadcast 0x00000012 found 2 33 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 8312, send timeouts 0, received 7054, lost 0, errs 0, boc 0, free buffers 38, min 38, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 592, adv 34763/74418
Accelerometer: LIS3DH, status: 00

M122 B124 after fail:

CAN response timeout: board 124, req type 6024, RID 159

justGuner

This post is deleted!

gloomyandy

@justGuner When the print fails, what is happening with the flashing led on the RRF36? Is it flashing quickly, slowly or not at all?

After the print fails if you reset the mainboard using the reset button on the skr2 but do not powercycle it (and especially don't powercycle the RRF36), do not do anything that moves the print head (like homing) does the led on the toolboard come back into sync with the mianboard? Can you get a M122 B124 after doing that reset? If not power cycle the printer, do not move anything, does the RRF36 sync with the mainbord, if so can you get a M122 B122 from it.

justGuner

@gloomyandy

@gloomyandy said in RRF36 disconnecting mid-print:

When the print fails, what is happening with the flashing led on the RRF36? Is it flashing quickly, slowly or not at all?

The led flashes rapidly.

@gloomyandy said in RRF36 disconnecting mid-print:

After the print fails if you reset the mainboard using the reset button on the skr2 but do not powercycle it (and especially don't powercycle the RRF36), do not do anything that moves the print head (like homing) does the led on the toolboard come back into sync with the mianboard?

As far as I remember, after pressing the reset button on the mainboard, the led on the toolboard would come back to synk with the mainboard and regain all functionality. I started another print job to verify the behaviour.

I should also mention that after the fail point, the hotend seems to hold it's determined temperature, because it doesn't cool down and it doesn't seem to cook the filament in the nozzle either.

gloomyandy

@justGuner That sounds like the RRF36 did not crash, I would be looking closely at the CAN bus wiring between the two boards. The mainboard was showing CAN Tx timeouts and a bus off error, that usually indicates a problem with the CAN bus wiring.

Messages queued 9917, received 4473, lost 0, errs 0, boc 1
Longest wait 4ms for reply type 6024, peak Tx sync delay 201 free buffers 50 (min 19), ts 2564/1112/0
Tx timeouts 105,0,1452,0,0,777 last cancelled message type 8191 dest 127

I assume that you have the terminator enabled on the toolboard?

justGuner

@gloomyandy I pressed the reset button on the board, and now it is stuck in a boot-loop. After power cycling the board, it remains in boot-loop. The contents of the sd card are still readable on my laptop.

gloomyandy

@justGuner How do you know it is in a bootloop? What did you do differently to the last time you reset the board, you said above "after pressing the reset button". Doing a hardware reset will not normally cause any sort of loop.

justGuner

@gloomyandy this is the behavior that I'm associating with boot loop:

https://drive.google.com/file/d/1EX8ncjLaL5Gg9t0ZIwL676H8gxXVDZhE/view?usp=drivesdk

The difference between what I did now and before is that before when problem occurred, I would first pause the print, and then press the button. This time I just pressed the button without stopping the print.

gloomyandy

@justGuner I've never seen pushing reset cause any sort of loop and I push reset a lot when debugging/developing the firmware. Are you sure that there is not some other problem like an issue with the power supply? I can't see your video it is still be processed.

The only thing I can suggest is that you try reflashing the firmware to the board from the sd card.

justGuner

@gloomyandy I've changed the link, you should be able to see the video now.

Reflashing the firmware did not solve the issue.

gloomyandy

@justGuner What is the Led that sort of fades out? Is that the normal status LED? If you put a firmware.bin file on the root of the sd card and reboot the board do you see any leds flicker for a short while after power on? Also if you check the card after a waiting a short while has the firmware.bin has the name changed to something else?

justGuner

@gloomyandy If you mean the green one I think it's a power indicator.

If you put a firmware.bin file on the root of the sd card and reboot the board do you see any leds flicker for a short while after power on?

Yes, and after that it repeats the previous behaviour.

Also if you check the card after a waiting a short while has the firmware.bin has the name changed to something else?

It turns into a cursor file.

gloomyandy

@justGuner So it sounds like you still have a bootloader. If that is a power led then it looks like something is pulling the power supply down very low and that is causing the board to reboot. You could try removing everything from your config.g file (keep a copy) other than the commands to setup the wifi and see if that allows the board to boot. Are you sure you have not made any other changes, for instance to any of the wiring or to the config.g file?

If the above does not work then I'd suggest disconnecting everything from the board and see if that allows it to start.

justGuner

@gloomyandy ok, removing everything from the board (except the esp32) after deleting everything in the config file except the networking stuff lets the board boot up. I can acces the board through web control. I guess I will shut it down and start plugging in stuff untill it stops working again.

gloomyandy

@justGuner Yep that's the process. Good luck!

justGuner

@gloomyandy Everything works fine untill I connect the efd to the board. After that, it's the same story again...

Closer inspection of the efd doesn't reveal any burns. It also passes the smell test (no burnt electronics smell)

gloomyandy

@justGuner Sorry what is the "efd" in this context? How is it connected to the board.

justGuner

@gloomyandy isn't that what the can adapter from mellow is called? The one connected to the exp ports

gloomyandy

@justGuner Ok so it is the CAN-FD interface. Do you still have the CANBUS connected to the board? If you disconnect that does it still cause problems?