DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1

Garfield

@bearer said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

it would terminate after 30-60 seconds or so as a timeout

Yup it does ...

A Former User

(i wonder if setting process affinity could isolate the hanging to leave a core running ssh etc, if possible in raspbian - anyways thats it for me today)

Garfield

I found this in the duet web server log

Apr 26 19:19:11 duet3 DuetWebServer[1106]: info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[3]
Apr 26 19:19:11 duet3 DuetWebServer[1106]:       Route matched with {action = "Get", controller = "WebSocket"}. Executing controller action with signature System.Threading.Tasks.Task Get() on controller DuetWebServer.Controllers.WebSocketController (DuetWe
Apr 26 19:19:11 duet3 DuetWebServer[1106]: fail: DuetWebServer.Controllers.WebSocketController[0]
Apr 26 19:19:11 duet3 DuetWebServer[1106]:       [WebSocketController] DCS is not started
Apr 26 19:19:11 duet3 DuetWebServer[1106]: info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[2]
Apr 26 19:19:11 duet3 DuetWebServer[1106]:       Executed action DuetWebServer.Controllers.WebSocketController.Get (DuetWebServer) in 339302.5643ms
Apr 26 19:19:11 duet3 DuetWebServer[1106]: info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
Apr 26 19:19:11 duet3 DuetWebServer[1106]:       Executed endpoint 'DuetWebServer.Controllers.WebSocketController.Get (DuetWebServer)'
Apr 26 19:19:11 duet3 DuetWebServer[1106]: info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Apr 26 19:19:11 duet3 DuetWebServer[1106]:       Request finished in 339446.6255ms 101
Apr 26 19:19:11 duet3 DuetWebServer[1106]: warn: DuetWebServer.Services.ModelObserver[0]
Apr 26 19:19:11 duet3 DuetWebServer[1106]:       Failed to synchronize machine model
Apr 26 19:19:11 duet3 DuetWebServer[1106]: System.Net.Sockets.SocketException (107): Transport endpoint is not connected
Apr 26 19:19:11 duet3 DuetWebServer[1106]:    at DuetAPI.Utility.JsonHelper.ReceiveUtf8Json(Socket socket, CancellationToken cancellationToken) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Utility/JsonHelper.cs:line 154
Apr 26 19:19:11 duet3 DuetWebServer[1106]:    at DuetAPIClient.BaseConnection.ReceiveJson(CancellationToken cancellationToken) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPIClient/BaseConnection.cs:line 294
Apr 26 19:19:11 duet3 DuetWebServer[1106]:    at DuetAPIClient.SubscribeConnection.GetMachineModelPatch(CancellationToken cancellationToken) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPIClient/SubscribeConnection.cs:line 100
Apr 26 19:19:11 duet3 DuetWebServer[1106]:    at DuetWebServer.Services.ModelObserver.Execute() in /home/christian/duet/DuetSoftwareFramework/src/DuetWebServer/Services/ModelObserver.cs:line 156
Apr 26 19:19:11 duet3 DuetWebServer[1106]: info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[2]

Apr 26 19:59:29 duet3 DuetWebServer[1106]: warn: DuetWebServer.Services.ModelObserver[0]
Apr 26 19:59:29 duet3 DuetWebServer[1106]:       Failed to synchronize machine model
Apr 26 19:59:29 duet3 DuetWebServer[1106]: System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (99): Cannot assign requested address /var/run/dsf/dcs.sock
Apr 26 19:59:29 duet3 DuetWebServer[1106]:    at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
Apr 26 19:59:29 duet3 DuetWebServer[1106]:    at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
Apr 26 19:59:29 duet3 DuetWebServer[1106]:    at DuetAPIClient.BaseConnection.Connect(ClientInitMessage initMessage, String socketPath, CancellationToken cancellationToken) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPIClient/BaseConnection.cs:l
Apr 26 19:59:29 duet3 DuetWebServer[1106]:    at DuetWebServer.Services.ModelObserver.Execute() in /home/christian/duet/DuetSoftwareFramework/src/DuetWebServer/Services/ModelObserver.cs:line 131
Apr 26 19:59:30 duet3 DuetWebServer[1106]: info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Apr 26 19:59:30 duet3 DuetWebServer[1106]:       Request starting HTTP/1.1 GET http://10.100.2.225/machine
Apr 26 19:59:30 duet3 DuetWebServer[1106]: info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
Apr 26 19:59:30 duet3 DuetWebServer[1106]:       Executing endpoint 'DuetWebServer.Controllers.WebSocketController.Get (DuetWebServer)'
Apr 26 19:59:30 duet3 DuetWebServer[1106]: info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[3]
Apr 26 19:59:30 duet3 DuetWebServer[1106]:       Route matched with {action = "Get", controller = "WebSocket"}. Executing controller action with signature System.Threading.Tasks.Task Get() on controller DuetWebServer.Controllers.WebSocketController (DuetWe
Apr 26 19:59:30 duet3 DuetWebServer[1106]: fail: DuetWebServer.Controllers.WebSocketController[0]
Apr 26 19:59:30 duet3 DuetWebServer[1106]:       [WebSocketController] DCS is not started
Apr 26 19:59:30 duet3 DuetWebServer[1106]: info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[2]
Apr 26 19:59:30 duet3 DuetWebServer[1106]:       Executed action DuetWebServer.Controllers.WebSocketController.Get (DuetWebServer) in 6.6056ms
Apr 26 19:59:30 duet3 DuetWebServer[1106]: info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
Apr 26 19:59:30 duet3 DuetWebServer[1106]:       Executed endpoint 'DuetWebServer.Controllers.WebSocketController.Get (DuetWebServer)'
Apr 26 19:59:30 duet3 DuetWebServer[1106]: info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Apr 26 19:59:30 duet3 DuetWebServer[1106]:       Request finished in 7.2117ms 101
Apr 26 19:59:32 duet3 DuetWebServer[1106]: info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Apr 26 19:59:32 duet3 DuetWebServer[1106]:       Request starting HTTP/1.1 GET http://10.100.2.225/machine
Apr 26 19:59:32 duet3 DuetWebServer[1106]: info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
Apr 26 19:59:32 duet3 DuetWebServer[1106]:       Executing endpoint 'DuetWebServer.Controllers.WebSocketController.Get (DuetWebServer)'
Apr 26 19:59:32 duet3 DuetWebServer[1106]: info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[3]
Apr 26 19:59:32 duet3 DuetWebServer[1106]:       Route matched with {action = "Get", controller = "WebSocket"}. Executing controller action with signature System.Threading.Tasks.Task Get() on controller DuetWebServer.Controllers.WebSocketController (DuetWe
Apr 26 19:59:32 duet3 DuetWebServer[1106]: fail: DuetWebServer.Controllers.WebSocketController[0]
Apr 26 19:59:32 duet3 DuetWebServer[1106]:       [WebSocketController] DCS is not started

Same error even in RC9

Danal

@Garfield said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

I feel the need for a compatibility matrix for the 3 main components - which versions of RRF work wich versions of DWC.

Interesting. Duet3 + Pi 4B, 4 gig. I've been having random hangs that take a power cycle to clear. I am also on RC10, as of mid evening yesterday. I was not certain this was happening, nor certain that it started at RC whatever, so I have not reported anything, yet.

Now that I think about it, it came on hard when I switched to RC10. I had to power cycle at least eight or ten times last night.

I typically have a DWC, a VNC and a SSH running. They all just hang. Attempting to start a new SSH also hangs (note, not refused, connects and never gets a password prompt).

I will see what data I can gather.

gtj0

Can I ask all you guys with issues, if the DWC is NOT connected, does it still lock up?

Garfield

I've never run without it, only ever connected via WiFi, would take me a while to set up if the way to test is to put the SD card into the duet itself.

Danal

I believe he's saying, "start a job, and then close DWC".

Yes, I tried that. SSH only, no VNC, no DWC. Still locked within a few minutes.

gtj0

@Danal said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

I believe he's saying, "start a job, and then close DWC".

Yes, I tried that. SSH only, no VNC, no DWC. Still locked within a few minutes.

Yeah that was it. I wanted to make sure it wasn't DWC related. I've been running RC10 + 2.1.1 and printing fine but I don't use the DWC.

Something to try... The systemd service file for the DCS was changed to set

CPUSchedulingPolicy=fifo
CPUSchedulingPriority=20

which may be contributing to the problem.

Edit /lib/systemd/system/duetcontrolserver.service and remove those 2 lines, then reboot and see if that helps.

Danal

@gtj0 said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

Edit /lib/systemd/system/duetcontrolserver.service and remove those 2 lines, then reboot and see if that helps.

Will do. I've made a bunch of other changes, so let me re-verify the hangs are real, then I will try that. THANKS!

Wally

I'm a little late to the reporting; to help confirm, I too have seen full system lock ups and print connection losses too with RRF 3.01-R10, DWC 2.1.5 and DSF 2.1.1.

I was able to overcome and get working with:

sudo system duetcontrolserver restart

No system hardware changes, ribbons, or otherwise, just the new Beta install.

The duetcontrolserver would go to 400% CPU usage, and painstakingly getting to the terminal (I have screen direct on my Pi) was able to get a terminal open and issue the fix. SSH & web access were dead - couldn't remote in.

After the duetcontrolserver restart, system will work but still get random disconnects. I will try the modification above and report back too.

Danal

Well... just finished a two hour print, no changes, completed OK.

I will keep data points coming, in either direction.

chas2706

Another fantastic build.

It is like playing the lottery. Sometimes works fine, other times loads of disconnections from DWC and/or error messages like "homing failed" or heaters take forever to turn on!

Back to RRF3.01_RC3!!!!

gtj0

@chas2706 Are you experiencing this specific issue?

chas2706

@gtj0 said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

Are you experiencing this specific issue?

Which issue are you refering to because there are many.
Has anyone tried to upload a new slicer gcode with this version because when I do it the DWC only gives me access to the folders on the SBC!!

A Former User

@chas2706 said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

Which issue are you refering to because there are many.

i suspect he was, in a slightly more diplomatic tone, trying to say unless you have something constructive to add to the issue in this particular forum topic you might be well advised to go elsewhere looking for some cheese to go with that whine.

yes there are a bunch of issues, and you're not the only one thats frustrated, but how you deal with it is only up to you.

i think you should take a page or two out of your own book..

chas2706

@bearer said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

i suspect he was, in a slightly more diplomatic tone, trying to say unless you have something constructive to add to the issue in this particular forum topic you might be well advised to go elsewhere looking for some cheese to go with that whine.

Ok cheers. Won't bother posting anything anymore bye bye.

Garfield

Well I got the Pi back, I can get into SSH, I wiped the SD card with the image and instruction found here >>

[https://duet3d.dozuki.com/Wiki/Getting_Started_With_Duet_3]

I used the duetpi (with GUI).

And it does NOT work, can't find a config.g anywhere in it, even if I add one it doesn't work.

When connecting to the web page it sits there connecting, an error message bottom right states DCS not available.

systemctl reports both DCS and DWC as Active and Running. Log report in uploaded file.

Please make an SD card image available that works. It would be kind of nice if such an image existed for each RC, a list of files and their correct locations would also help enormously, it really shouldn't need to be this difficult.

DCS - DWC Log.txt

gtj0

@Garfield The SD card image has the "stable" versions of the DSF and needs the "unstable" versions OR you need to downgrade your Duet3 to the firmware located on the card image.

Danal

@Garfield said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

Please make an SD card image available that works

Unfortunately, that is not anywhere near as easy as it sounds. The software on the SD has to match the firmware on the board well enough for them to talk. There are so many variants that a ton of images might actually become more confusing than not. Over time, this will all get better as 'apt update' and 'apt upgrade' get better and better at keeping everything in sync.

But... right now today, because you chose to "force" the card into a given state, by wiping and re-etching, then the burden also falls on you to "force" the board into a matching state, via bossa.

Step 1 is to get to get duetsoftwareframework on the SD to a whatever level you prefer.

sudo apt list duetsoftwareframework -a

Will show you all available levels, as well as what is installed.

sudo apt remove duetsoftwareframework
sudo apt -s autoremove                            (-s is simulate, be sure it is removing what you want, then run it again without the -s)
sudo apt install duetsofwareframework=1.3.2       (or whatever you want)

Will put everything at a certain level.

Step 2 is to bossa with the file found at /opt/dsf/sd/sys/Duet3Firmware_MB6HC.bin This ensures that the SD and the board firmware will match.

P.S. I prefer to bossa from the Pi itself. Instructions are on the dozuki.
P.P.S When picking the release you want in step 1... At this moment, I know of no way to correlate "duetsoftwareframework release x.y.z" to "firmware V3.01 RCxyz". I'm working on finding a way.

Garfield

@Danal said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

because you chose to "force" the card into a given state

I did NOT choose to - I was given no option because I could not start / access the system hardwired or otherwise !!

I have no desire to fall out over this, the code is RC after all, you expect glitches but things just really shouldn't be this difficult, I should NOT need to use BOSSA - another mechanism is required because I have ZERO intention of making up yet more cables.

Same principle as the Pi would be optimal - blank SD card, formatted FAT32, firmware image in root - same way a huge amount of embedded systems are updated - it isn't difficult.