DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1

Garfield

pr 27 11:15:05 duet3 DuetControlServer[3136]: [info] Application has shut down
Apr 27 11:15:05 duet3 systemd[1]: duetcontrolserver.service: Succeeded.
Apr 27 11:15:10 duet3 systemd[1]: duetcontrolserver.service: Service RestartSec=5s expired, scheduling restart.
Apr 27 11:15:10 duet3 systemd[1]: duetcontrolserver.service: Scheduled restart job, restart counter is at 57.
Apr 27 11:15:10 duet3 systemd[1]: Stopped Duet Control Server.
Apr 27 11:15:10 duet3 systemd[1]: Started Duet Control Server.
Apr 27 11:15:11 duet3 DuetControlServer[3175]: Duet Control Server v2.1.1
Apr 27 11:15:11 duet3 DuetControlServer[3175]: Written by Christian Hammacher for Duet3D
Apr 27 11:15:11 duet3 DuetControlServer[3175]: Licensed under the terms of the GNU Public License Version 3
Apr 27 11:15:11 duet3 DuetControlServer[3175]: [info] Settings loaded
Apr 27 11:15:11 duet3 DuetControlServer[3175]: [info] Environment initialized
Apr 27 11:15:11 duet3 DuetControlServer[3175]: [info] Connection to Duet established
Apr 27 11:15:11 duet3 DuetControlServer[3175]: [info] IPC socket created at /var/run/dsf/dcs.sock
Apr 27 11:15:12 duet3 DuetControlServer[3175]: [fatal] Abnormal program termination
Apr 27 11:15:12 duet3 DuetControlServer[3175]: [fatal] Update task faulted
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    System.ArgumentException: Property set method not found.
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    at System.Reflection.RuntimePropertyInfo.SetValue(Object obj, Object value, BindingFlags invokeAttr, Binder binder, Object[] index, CultureInfo culture)
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    at System.Reflection.RuntimePropertyInfo.SetValue(Object obj, Object value, Object[] index)
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    at System.Reflection.PropertyInfo.SetValue(Object obj, Object value)
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    at DuetAPI.Machine.ModelObject.UpdateFromJson(JsonElement jsonElement, Boolean ignoreSbcProperties) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Base/ModelObject.cs:line 237
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    at DuetAPI.Machine.FilamentMonitor.UpdateFromJson(JsonElement jsonElement, Boolean ignoreSbcProperties) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Sensors/FilamentMonitors/FilamentMon
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    at DuetAPI.Machine.ModelObject.UpdateFromJson(JsonElement jsonElement) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Base/ModelObject.cs:line 209
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    at DuetAPI.Machine.FilamentMonitor.UpdateFromJson(JsonElement jsonElement, Boolean ignoreSbcProperties) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Sensors/FilamentMonitors/FilamentMon
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    at DuetAPI.Machine.ModelCollectionHelper.UpdateFromJson(IList list, Type itemType, JsonElement jsonElement, Boolean ignoreSbcProperties) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Bas
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    at DuetAPI.Machine.ModelObject.UpdateFromJson(JsonElement jsonElement, Boolean ignoreSbcProperties) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Base/ModelObject.cs:line 264
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    at DuetAPI.Machine.MachineModel.UpdateFromFirmwareModel(String key, JsonElement jsonElement) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/MachineModel.cs:line 149
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    at DuetControlServer.Model.Updater.Run() in /home/christian/duet/DuetSoftwareFramework/src/DuetControlServer/Model/Updater.cs:line 182
Apr 27 11:15:12 duet3 DuetControlServer[3175]: [fatal] Update task faulted
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    System.ArgumentException: Property set method not found.
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    at System.Reflection.RuntimePropertyInfo.SetValue(Object obj, Object value, BindingFlags invokeAttr, Binder binder, Object[] index, CultureInfo culture)
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    at System.Reflection.RuntimePropertyInfo.SetValue(Object obj, Object value, Object[] index)
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    at System.Reflection.PropertyInfo.SetValue(Object obj, Object value)
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    at DuetAPI.Machine.ModelObject.UpdateFromJson(JsonElement jsonElement, Boolean ignoreSbcProperties) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Base/ModelObject.cs:line 237
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    at DuetAPI.Machine.FilamentMonitor.UpdateFromJson(JsonElement jsonElement, Boolean ignoreSbcProperties) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Sensors/FilamentMonitors/FilamentMon
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    at DuetAPI.Machine.ModelObject.UpdateFromJson(JsonElement jsonElement) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Base/ModelObject.cs:line 209
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    at DuetAPI.Machine.FilamentMonitor.UpdateFromJson(JsonElement jsonElement, Boolean ignoreSbcProperties) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Sensors/FilamentMonitors/FilamentMon
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    at DuetAPI.Machine.ModelCollectionHelper.UpdateFromJson(IList list, Type itemType, JsonElement jsonElement, Boolean ignoreSbcProperties) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Bas
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    at DuetAPI.Machine.ModelObject.UpdateFromJson(JsonElement jsonElement, Boolean ignoreSbcProperties) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Base/ModelObject.cs:line 264
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    at DuetAPI.Machine.MachineModel.UpdateFromFirmwareModel(String key, JsonElement jsonElement) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/MachineModel.cs:line 149
Apr 27 11:15:12 duet3 DuetControlServer[3175]:    at DuetControlServer.Model.Updater.Run() in /home/christian/duet/DuetSoftwareFramework/src/DuetControlServer/Model/Updater.cs:line 182
Apr 27 11:15:12 duet3 DuetControlServer[3175]: [info] Application has shut down
Apr 27 11:15:12 duet3 systemd[1]: duetcontrolserver.service: Succeeded.

SSH connections are stable though ...

A Former User

@Garfield it needs to evolve a little, but it has a sibling that will identify and flash any duet 2 wifi/ethernet/maestro or duet 3 with the correct firmware in any state as long as usb is working. that might be proposed for the doczuki/wiki when polished.

anyways, glad you found it helpful in the end, the goal is in the end to take the guesswork out of the flashing when necessary.

Garfield

spoke too soon - ssh eventually dies .... so we are back on topic - same issue as we started with ...

ChrisP

@Garfield said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

.... so we are back on topic - same issue as we started with ...

Excellent... ish

Is the log you posted above what you see when DCS crashes? I still haven't managed to get anything interesting in the logs.

Garfield

Yup - thats the crash log ...

Danal

I'm the one that leans toward bossa from the Pi. However, I'm just an end-user that's trying to dig another end user out of a hole. If someone else, prefers windows, great!

I prefer the Pi for two reasons:

It does avoid ground loops, to a somewhat greater extent than a separate computer.
Windows bossa works quite a bit of the time... and other times, windows gets really weird about USB ports.

Again, I'm just making suggestions, if someone prefers one tool over another, fantastic, use that!

gtj0

@Garfield @ChrisP Have you tried my suggestion above to remove the two CPUScheduling lines from /lib/systemd/system/duetcontrolserver.service and rebooting?

Danal

Data point: I have not changed the CPU scheduling and I am now about 14 hours into a print, with no issues.

Running DuetLapse on the same Pi. (Which I was not doing the other day).

I don't know what changed, if anything, from the point where I couldn't get it to run for 15 minutes at a time.

Garfield

I've not tried the scheduling - can't get DCS to start and stay running, first time it came up and was fine - until I threw my .g files at it and then BOOM ....

I need to get the whole thing back to RC6 and stable but right now it is all turned off (day job - that and run out of patience for now).

Could really do with that 'version compatibility' matrix ....

gtj0

@Danal When your print is done it's be helpful if you could test without DuetLapse and see if you can reproduce the issue, then try removing the CPUScheduling lines.

gtj0

@Garfield Understood. If you do get a chance, try removing those 2 lines.

A Former User

@Garfield said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

back to RC6

Could really do with that 'version compatibility' matrix ....

@bearer said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

RRF3.01-RC10     duetsoftwareframework 2.1.1      reprapfirmware 2.1.1-1
RRF3.01-RC9      duetsoftwareframework 2.1.0      reprapfirmware 2.1.0-1
RRF3.01-RC8      duetsoftwareframework 2.0.0      reprapfirmware 2.0.0-1
RRF3.01-RC7      duetsoftwareframework 1.3.2      reprapfirmware 1.3.2-1
RRF3.01-RC6      duetsoftwareframework 1.3.1      reprapfirmware 1.3.1-1
RRF3.01-RC6      duetsoftwareframework 1.3.0      reprapfirmware 1.3.0-1
RRF3.01-RC4      duetsoftwareframework 1.2.5.0    reprapfirmware 1.2.5.0-1
RRF3.0           duetsoftwareframework 1.2.4.0    reprapfirmware 1.2.4.0-1
RRF3.0           duetsoftwareframework 1.2.3.1    reprapfirmware 1.2.3.1-1
RRF3.0           duetsoftwareframework 1.2.3.0    reprapfirmware 1.2.3.0-1
RRF3.0           duetsoftwareframework 1.2.2.1    reprapfirmware 1.2.2.1-1
RRF3.0RC2+1      duetsoftwareframework 1.2.2.0    reprapfirmware 1.2.2.0-1
RRF3.0RC1        duetsoftwareframework 1.2.1.0    reprapfirmware 1.2.1.0-1
RRF3.0RC1        duetsoftwareframework 1.2.0.0    reprapfirmware 1.2.0.0-1
RRF3.0beta11     duetsoftwareframework 1.1.0.5    reprapfirmware 1.1.0.5-1
RRF3.0beta11     duetsoftwareframework 1.1.0.4    reprapfirmware 1.1.0.4-1
RRF3.0beta10+2   duetsoftwareframework 1.1.0.3    reprapfirmware 1.1.0.3-1
RRF3.0beta10+2   duetsoftwareframework 1.1.0.2    reprapfirmware 1.1.0.2-1
RRF3.0beta10+2   duetsoftwareframework 1.1.0.1    reprapfirmware 1.1.0.1-1
RRF3.0beta10+2   duetsoftwareframework 1.1.0.0    reprapfirmware 1.1.0.0-1
RRF3.0beta10+1   duetsoftwareframework 1.0.4.1    reprapfirmware 1.0.4.1-1

depcheck()
{
  local NODE=$1
  local VER=$2
  local pattern="\ ([a-z]*)\ \([0-9]+\ ([0-9\.]+-?[0-9]{0,2})\)"
  local IFS=$'\n'
  s=$(apt-cache showpkg $NODE |  grep "^$VER - ." )
  for package in $(echo $s | grep -Eo  $pattern)
  do
    [[ $package =~ $pattern ]]
    [ "${BASH_REMATCH[1]}" == "${BASH_REMATCH[2]}" ] || echo -en "  ${BASH_REMATCH[1]}=${BASH_REMATCH[2]} $EOL"
    [ "${BASH_REMATCH[1]}" == "duetcontrolserver" ] && depcheck ${BASH_REMATCH[1]} ${BASH_REMATCH[2]}
  done
}

[ $# -gt 0 ] && {
V=$1
PARENT=duetsoftwareframework
[ $# -gt 1 ] && PARENT=$2
EOL=""
[ -t 1 ] && EOL="\\ \n"
echo  -en "sudo apt install $EOL"
depcheck $PARENT $V
[ -t 1 ] && EOL="\n"
echo  -en "  $PARENT=$V $EOL"
} ||  echo $0 version ;

save as dsfdep.sh and run like bash dsfdep.sh 1.3.1 to restore to 3.01-RC6
copy and paste the suggest command and hope for the best (or put in backticks if braver than most)

pi@raspberrypi:~ $ bash dsfdep.sh 1.3.1
sudo apt install \
  duetcontrolserver=1.3.1 \
  duetruntime=1.3.1 \
  duetsd=1.0.6 \
  duettools=1.3.1 \
  duetwebserver=1.3.1 \
  duetwebcontrol=2.1.1 \
  reprapfirmware=1.3.1-1 \
  duetsoftwareframework=1.3.1

Danal

@gtj0 said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

@Danal When your print is done it's be helpful if you could test without DuetLapse and see if you can reproduce the issue, then try removing the CPUScheduling lines.

Will do.

May do a different print, this one is running MUCH longer than I anticipated. It is a toolchanger multi-material, and "estimators" do not (yet) account for the time spent changing tools. There are 1045 changes in this print.

Danal

@Garfield said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

Could really do with that 'version compatibility' matrix ....

I'm working on a script that generates one. No promises as to when... lots of learning and research on my part.

In fact, if anyone else is working on one, I'll stop.

gloomyandy

Just as another data point. After my initial crashes I went on to print (for a couple of hours or so) with no problems and things seemed pretty stable. I wonder if forcing a reload of DWC had anything to do with the problem?

jay_s_uk

@gtj0 said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

@Danal said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

I believe he's saying, "start a job, and then close DWC".

Yes, I tried that. SSH only, no VNC, no DWC. Still locked within a few minutes.

Yeah that was it. I wanted to make sure it wasn't DWC related. I've been running RC10 + 2.1.1 and printing fine but I don't use the DWC.

Something to try... The systemd service file for the DCS was changed to set
CPUSchedulingPolicy=fifo
CPUSchedulingPriority=20
which may be contributing to the problem.

Edit /lib/systemd/system/duetcontrolserver.service and remove those 2 lines, then reboot and see if that helps.

@gtj0

I've checked the above folder and I don't have a duetcontrolserver.service file to edit

gtj0

@jay_s_uk said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

@gtj0 said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:
@Danal said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

I believe he's saying, "start a job, and then close DWC".

Yes, I tried that. SSH only, no VNC, no DWC. Still locked within a few minutes.

Yeah that was it. I wanted to make sure it wasn't DWC related. I've been running RC10 + 2.1.1 and printing fine but I don't use the DWC.

Something to try... The systemd service file for the DCS was changed to set
CPUSchedulingPolicy=fifo
CPUSchedulingPriority=20
which may be contributing to the problem.

Edit /lib/systemd/system/duetcontrolserver.service and remove those 2 lines, then reboot and see if that helps.
@gtj0

I've checked the above folder and I don't have a duetcontrolserver.service file to edit

See if it's in /usr/lib/systemd/system.

jay_s_uk

@gtj0

Thats where they are. I have commented them out and will report back

jay_s_uk

@gtj0

I have managed a small print without any failures.
So far so good.

A Former User

@Danal said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

In fact, if anyone else is working on one, I'll stop.

gave up on recursion seems all reprapfirmware from 1.0.4.1-1 to 1.2.5.0-1 has a dependency to duetcontrolserver 1.0.3.4 which doesn't exist (which probably means or greater but )