Inconsistent state of printer with SBC

gnydick

Printer web interface b0rked, shows disconnected, and some wrong sensor info, but the paneldue is fully functional and everything shows right.

I've restarted the duetwebserver on the sbc to no avail.

Red Underlines

should be connected
should show 4 fan RPMs
should show heater offline
should show 45 degrees on first tool

M122

=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.4.5 (2022-11-30 19:35:23) running on Duet 3 MB6HC v1.01 (SBC mode)
Board ID: 0JD2M-999AL-D25SW-6J1FL-3SN6S-1RXZ2
Used output buffers: 5 of 40 (40 max)
=== RTOS ===
Static ram: 152760
Dynamic ram: 69820 of which 0 recycled
Never used RAM 124252, free system stack 114 words
Tasks: SBC(ready,0.2%,532) HEAT(notifyWait,0.0%,322) Move(notifyWait,0.0%,245) CanReceiv(notifyWait,0.1%,772) CanSender(notifyWait,0.0%,326) CanClock(delaying,0.0%,339) TMC(notifyWait,8.2%,57) MAIN(running,91.4%,923) IDLE(ready,0.0%,30), total 100.0%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 07:28:19 ago, cause: power up
Last software reset details not available
Error status: 0x04
Aux0 errors 0,0,0
Step timer max interval 134
MCU temperature: min 44.3, current 44.4, max 44.7
Supply voltage: min 24.1, current 24.1, max 24.2, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.0, current 12.0, max 12.1, under voltage events: 0
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/132/132, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min n/a, mspos 840, reads 13293, writes 0 timeouts 0
Driver 1: standstill, SG min n/a, mspos 584, reads 13293, writes 0 timeouts 0
Driver 2: standstill, SG min n/a, mspos 592, reads 13293, writes 0 timeouts 0
Driver 3: standstill, SG min n/a, mspos 712, reads 13293, writes 0 timeouts 0
Driver 4: standstill, SG min n/a, mspos 808, reads 13293, writes 0 timeouts 0
Driver 5: standstill, SG min n/a, mspos 280, reads 13293, writes 0 timeouts 0
Date/time: 2022-12-18 19:01:09
Slowest loop: 1.72ms; fastest: 0.06ms
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 37.5MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 41, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 1315011, completed 1315011, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP* is doing "M122" in state(s) 0
Telnet is idle in state(s) 0
File* is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue* is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty
=== CAN ===
Messages queued 1082, received 9620, lost 0, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 6, free buffers 50 (min 50), ts 601/601/0
Tx timeouts 0,0,0,0,0,0
=== SBC interface ===
Transfer state: 5, failed transfers: 0, checksum errors: 0
RX/TX seq numbers: 15031/52346
SPI underruns 0, overruns 0
State: 5, disconnects: 0, timeouts: 0 total, 0 by SBC, IAP RAM available 0x2ad48
Buffer RX/TX: 0/0-0, open files: 0
=== Duet Control Server ===
Duet Control Server v3.4.5
Code buffer space: 4096
Configured SPI speed: 8000000Hz, TfrRdy pin glitches: 4
Full transfers per second: 39.98, max time between full transfers: 76.6ms, max pin wait times: 30.3ms/18.1ms
Codes per second: 0.01
Maximum length of RX/TX data transfers: 4692/1764

I then restarted duetcontrolserver and the DWC came back. There is literally nothing other than Daemon.g start/stop messages in the log.

Phaedrux

Can you try increasing the log level on DCS?
https://github.com/Duet3D/DuetSoftwareFramework/wiki/SBC-Setup-Guide#increasing-log-level

gnydick

@Phaedrux sure.

chrishamm

@gnydick It would be helpful to inspect the journal of DuetControlServer (sudo journalctl -u duetcontrolserver -e, then scroll up). Can you see any unusual log entries there? If not, please share your config file and daemon.g so I can attempt to reproduce it.

gnydick

@chrishamm nothing useful in the logs and hasn't happened again.

chrishamm

@gnydick I have identified a possible cause and prepared a solution in RRF (although I believe it's unlikely to happen). If this problem occurs again, please open the JS console by pressing F12 and check potential errors in there, too.

gnydick

@chrishamm will do. Can you describe what it is? I'm a software engineer among other things, so you can be as technical as you need.

chrishamm

@gnydick I think I could reproduce this problem today, it doesn't look like my previous idea was the cause. I have prepared fixes for 3.5-b1 and backported it to our v3.4 branch. That bug fix will be included in 3.5-b1 and 3.4.6.

At first I thought the response of an object model request could be too long which would result in endless resends of the OM request. This (yet unlikely) bug is now fixed. The other issue I found today could only occur if an OM request was sent from DCS to RRF and RRF lost connection afterwards due to a timeout. Then DSF would not invalidate the pending OM request so it could not be retried. In this scenario the UI would display "disconnected" all the time and no more updates would be processed, although the response of G/M/T-codes could still be seen in the console. That's now fixed, too.

gnydick

@chrishamm that behavior seems exactly like what I experienced. It still interacted some, but still showed Disconnected. It was resolved by restarting duetcontrolserver. Does that match the bug you fixed, in terms of at least working as a recovery step?

chrishamm

@gnydick Yes, that's right. Emergency Stop usually restarts DCS, too.

gnydick

@chrishamm emergency stop hasn't resolved things for me. It seems to be the frequent cause of these issues.

gnydick

Here's another occasion. Yesterday, I noticed I could not reach the DWC, so used the PanelDue + M586 and saw the http was disabled. So I enabled it there. I then used the printer as usual.

Today, I went to use the printer and saw the screen attached after submitting a print from my slicer, but the DWC is stuck at a "Paused" state, again with no tools, sensors, etc.

Here are log files. It's actually a zip file, but it wouldn't let me upload a zip.

logs.zip.txt

nothing happening.jpg