Power failure and resurrection on a CNC machine

infiniteloop

@nightowl999

It's my understanding the principles are the same for an unexpected and a manual shutdown

That depends on how you arrange your PSUs. If you have a separate (small) power supply for the duet (5V), you can

control the power for the mechanics with GCodes (ATX Power on/off)
use a tiny UPS (or a large capacitor) to keep the board alive for some seconds at power outages
collect data from the object model in order to create a macro which can be called (even after days) to restore the previous state of the machine

As long as the duet is powered, you can save the state of your machine (which is, in my understanding, a bit different for a mill), be it for a pause, for an emergency stop, or at the event of a power outage.

That’s a question of software (i.e. macros), which you seem to elaborate in another thread with the qualified help of @OwenD.

The general idea is to write macros who in turn generate macro files who, if called, execute the Gcodes needed to restore the desired machine state. You will want to generate different macros for pause-resume and for emergency stops: in the latter case (and on power loss), you have to re-home the mill first (with a 3D print on the bed, this can be quite a challenge).

The thing is: You will have to write a lot of code, and, as neither me nor @OwenD have a milling machine, it’s up to you to explore the object model's properties - where to find the spindle’s RPM, which entry reflects a setting, which one the current state (that depends and is not always easy to find out). I bet that, when you’ve accomplished this task, you’ll speak Gcode-ish fluently.

When you have made up your decision how to proceed, I can at least provide some code snippets to demonstrate most of the techniques you need for writing macros who spit out macros onto the SD card. Just ask.

Nightowl

@infiniteloop said in Power failure and resurrection on a CNC machine:

it’s up to you to explore the object model's properties - where to find the spindle’s RPM, which entry reflects a setting, which one the current state (that depends and is not always easy to find out)

I'm on that now, but some sort of reference book wouldn't go amiss!

@infiniteloop said in Power failure and resurrection on a CNC machine:

I bet that, when you’ve accomplished this task, you’ll speak Gcode-ish fluently.

May that day come soon!

@infiniteloop said in Power failure and resurrection on a CNC machine:

When you have made up your decision how to proceed, I can at least provide some code snippets to demonstrate most of the techniques you need for writing macros who spit out macros onto the SD card. Just ask.

Thank you.

infiniteloop

@nightowl999

Some sort of reference book wouldn't go amiss!

You will certainly know the Object Model view of DWC (under „Settings“). This is the first place to go, because it allows you to observe kind of "live" the current settings and, even more important, how values change during operation.

Then, I found this documentation on the web - it’s automatically generated (don’t know how frequently it is updated).

This in turn links to the ultimate source: the Object Model Documentation on GitHub.

Hope that helps

Nightowl

@infiniteloop
Blimey, that's not a 30 minute read

I had a wander through the Object Model and realised there's a lot of information in there, but interacting with it I wasn't sure about. It's clear where 'things' are defined, like a spindle or a heater, but finding non-permanent values took a little getting used to.

What didn't help (me) was having several web browser windows open to try and see the impact of what I was doing, but every time I ran a file, it seemed all the windows went to the Dashboard screen - and what showed up in the Console screen was a bit random, too, appearing in one screen but not the other!

At least the Pause and Resume files are working now, with just one small glitch I think I can iron out, and then it's on to the next project

infiniteloop

@nightowl999 said in Power failure and resurrection on a CNC machine:

Blimey, that's not a 30 minute read

… but a reference I use almost all of the time

Nightowl

@infiniteloop
Actually, I had a look when trying to resolve the pause/ressume issues I was having, and it helped.

Thanks for the pointer!

dc42

@nightowl999 said in Power failure and resurrection on a CNC machine:

@infiniteloop

Thanks for that. Is this also the case with a 24v powered machine?

With 24V power the Duet should be able to save the important data to the resurrect.g file. However, there may not be enough power to complete movement commands in the M911 power fail script.

Nightowl

@dc42
The only 'mechanical' things, as I see it, would be to raise the Z height to clear the workpiece (same as a printer I guess), and switch the spindle off.

The problem might be how to switch the spindle on, at the correct speed, from within the resurrect-prologue.g file. I appreciate a variable is probably volatile so I wouldn't be able to get the spindle speed from there, but would it be saved in state.restorePoints[].spindleSpeeds[] of the Object Browser on a power fail - unexpected or otherwise?

Thanks

Nightowl

On from this a bit, as I've just had two power failures and hadn't configured M911 in the config.g file, so don't have a resurrect.g to recover from (I have now!) I'd be interested in looking at your...

...'customised recovery file generator'...

... @infiniteloop to see if I can find a way of saving the value of a variable in the resurrect.g file

Nightowl

@infiniteloop

I found the printed version...

Screenshot 2022-05-29 at 17.41.12.png

infiniteloop

@nightowl999

to see if I can find a way of saving the value of a variable in the resurrect.g file

That’s difficult. If you have M911 configured near the end of your config.g, it will automatically generate resurrect.g and write it on the SD card. In a second step, M911 will execute the commands you told it with the P parameter. There, you might be able to invoke your own macro generator - although, I’m not quite sure about how to put a macro call into the P string …

Depending on whether resurrect.g has already been closed at this stage, you might be able to append own gCode commands - at the end of the file, which is the wrong place for what I think you want to achieve.

Another possibility is to save your own file, which can then be called from resurrect-prologue.g. To get an idea where in the recovery process resurrect-prologue.g is called, please have a look at this document.

The third option is to ignore resurrect.g completely and to generate your own recovery file instead.

Aside from the document linked above, you should thoroughly study the resurrect.g file you now have on your SD card. Best is to draw a copy onto your PC and to comment each single gCode you find. Also take care of the parameters and their meaning. Use the GCode Dictionary to spell the lines, until you can imagine what resurrect.g does and why so.

Then, you are finally able to decide whether you can get along with some supplemental file which is called from within resurrect-prologue.g, or if you have to bite the apple and write your own recovery mechanism.

In the meantime, I’ll prepare some code samples for you. Sorry, but this may take until tomorrow, as I have an Arduino on my desk which needs a severe debugging treatment

Nightowl

@infiniteloop
Thanks for that, and I'm pretty sure I can wait until tomorrow!

Yeah, I didn't realise the resurrect.g file was overwritten every time, but I should have!

Now my thinking is when the power fails the relevant information should be saved into
state.restorePoint[0].spindleSpeed and state.restorePoints[0].toolNumber, which I thought it would make sense for it to do that anyway, but it seems not.

Maybe I could write something into the M911 string to save that info into the restore points and 'recover' it from the resurrect-prologue.g file?

infiniteloop

@nightowl999

when the power fails the relevant information should be saved into
state.restorePoint[0].spindleSpeed and state.restorePoints[0].toolNumber

Sorry, but the object model is read-only. This said, I think I know what you mean: these are the properties you want to restore on top of resurrect.g.

Maybe I could write something into the M911 string to save that info into the restore points and 'recover' it from the resurrect-prologue.g file?

You must write a macro to generate a file with these data points. That file, in turn, is a macro itself, containing the data as arguments of GCode instructions who put these data into place.

Whether you can call this file from resurrect-prologue.g or not depends on the required order of commands: for instance, you should not turn the spindle on before the proper tool has been selected. That’s why you should study your resurrect.g in detail.

infiniteloop

@nightowl999 This time, the debugging session took not as long as I expected, so here’s some fish for the cat:

; --- just a sample resurrect.g file: don't ever execute! ---

√M141 P0 S50.0										; activate chamber heater
√G29 S1												; load the height map from file and activate bed compensation
√T-1 P0												; de-select tools, don't run macros
√G92 X-101.068 Y6.281 Z9.100							; set position of head without moving it
√G60 S1												; store current position in slot #1
M98 P"resurrect-prologue.g"
√M116												; wait for all temperatures to reach their set values
√M290 X0.000 Y0.000 Z0.100 R0						; restore babystepping offsets
√G10 L2 P1 X0.00 Y0.00 Z0.00							; L=2: this sets the origin of the coordinate system number
G10 L2 P2 X0.00 Y0.00 Z0.00							;	 specified by the P parameter (1 to 9) to the specified X, Y, X... values
G10 L2 P3 X0.00 Y0.00 Z0.00
G10 L2 P4 X0.00 Y0.00 Z0.00
G10 L2 P5 X0.00 Y0.00 Z0.00
G10 L2 P6 X0.00 Y0.00 Z0.00
G10 L2 P7 X0.00 Y0.00 Z0.00
G10 L2 P8 X0.00 Y0.00 Z0.00
G10 L2 P9 X0.00 Y0.00 Z0.00
G54													; select coordinate system #1
√G10 P1 S0 R0										; set active(S) and standby(R) temperatures for tool #1
√T1 P0												; select tool #1, don't run macros
√G10 P0 S230 R225									; set active(S) and standby(R) temperatures for tool #0
√T0 P6												; select tool #0, run macros (0b0110) - tfree=1, tpre=2 and tpost=4
M106 S0.70											; set fan speed (fan defaults to #0)
√M106 P0 S0.70										; 	now, with dedication to every possible fan
√M106 P1 S0.00
M106 P2 S0.00
M106 P5 S0.00
M106 P7 S0.00
M106 P8 S0.00
√M116												; wait for all temperatures to reach their set values
√G92 E0.00000										; set position of extruder without moving it
√M83													; extruder to relative
√M23 "0:/gcodes/Grundplatte II.gcode"				; select file for printing
√M26 S8015948 P0.000									; set pointer into file (P=proportion of the first move to be skipped, default 0.0, must be less than 1.0)
√G0 F6000 Z11.200									; place Z 2 mm above the printing position
√G0 F6000 X-91.068 Y6.281							; drive X and Y to the printing position
√G0 F6000 Z9.200										; lower Z for 2 mm into its printing position
√G1 F3000.0 P0										; use speed F, set port bits to 0 (remainder from RRF 2?)
G21													; set units to mm
√M24													; resume print (after a pause)

That’s an older resurrect.g file that I used as a first template for my own recovery macro. I’ve added comments to understand what’s going on, every GCode I then implemented in my counterpart has a checkmark.

Depending on the editor and font used, the comments are badly misaligned. For better readability, you might want to paste the content of the black box into your editor of choice and ameliorate the text at will.

Recently generated "resurrects" look a bit different, so I suggest you to publish your latest resurrect.g - I'd like to have a look at that.

Nightowl

@infiniteloop

Here is the latest resurrect.g file:

; File "0:/gcodes/Calibration Toolpath.gcode" resume print after power failure at 2022-05-29 16:49
G21
G92 X182.895 Y658.930 Z66.000
G60 S1
T-1 P0
M98 P"resurrect-prologue.g"
M116
M290 X0.000 Y0.000 Z0.000 R0
; Workplace coordinates
G10 L2 P1 X147.00 Y584.00 Z67.00
G10 L2 P2 X0.00 Y0.00 Z0.00
G10 L2 P3 X0.00 Y0.00 Z0.00
G10 L2 P4 X0.00 Y0.00 Z0.00
G10 L2 P5 X0.00 Y0.00 Z0.00
G10 L2 P6 X0.00 Y0.00 Z0.00
G10 L2 P7 X0.00 Y0.00 Z0.00
G10 L2 P8 X0.00 Y0.00 Z0.00
G10 L2 P9 X0.00 Y0.00 Z0.00
G54
M106 S0.50
M106 P0 S0.50
M116
G92 E0.00000
M83
M486 S-1
G17
M23 "0:/gcodes/Calibration Toolpath.gcode"
M26 S249
G0 F6000 Z68.000
G0 F6000 X182.895 Y658.930
G0 F6000 Z66.000
G1 F2500.0 P0
G21
M24

I was under the impression this file was overwritten with a new one every time there was a 'power failure', and it wasn't user-editable?

infiniteloop

@nightowl999 said in Power failure and resurrection on a CNC machine:

I was under the impression this file was overwritten with a new one every time there was a 'power failure', and it wasn't user-editable?

Your first impression is correct. Whether resurrect.g is write-protected or not? Can't tell you as I currently have no such file at hand …

Nevertheless, here's the first lesson - by the way, it demonstrates how M911 generates resurrect.g, up to the fact that the generated file will always be overwritten when we call M98 P"MWriter.g" again

; MWriter aka macro writer
; Creates the macro "HelloWorld.g" on the SD card which in turn can be executed
; Requirements: RRF v3.4

; To keep things simple, we use the /sys directory - which makes sense as any recovery macros should live in there.
; For development and tests, I use DWC, mainly with the "System" tab

var fName = "HelloWorld.g"			; this is the filename for the macro to create


; Here, we create the file - note the use of a single ">" with echo: this overwrites any existing file with this name.
echo >{var.fName} "; HelloWorld.g"
; with ">>", echo appends something to the now existing file:
echo >>{var.fName} "; Macro file created by macro ""MWriter.g"""	; we always write strings - note how to put a string into it (""MWriter.g"")
echo >>{var.fName} ""												; empty line for better readability
echo >>{var.fName} "M291 S2 R""Hello World"" P""Moin Moin"""		; message box, blocking (S2), with title (R) and message (P), localized :-)
echo >>{var.fName} ""
echo >>{var.fName} ";EOF"											; ... old habit of mine

;EOF																; same habit ...

Oh, and thanks for your resurrect.g - I'll try to study it. Did you make a copy and added comments?

Nightowl

@infiniteloop

Yes, here it is, but I'm not sure if I've done it correctly...

; File "0:/gcodes/Calibration Toolpath.gcode" resume print after power failure at 2022-05-29 16:49
G21					; sets units to mm
G92 X182.895 Y658.930 Z66.000		; sets the position for X, Y and Z
G60 S1					; save memory slot number
T-1 P0					; deselects the tool
M98 P"resurrect-prologue.g"		; runs the user-defined resurrect-prologue.g
M116					; wait for all temperatures to reach set value
M290 X0.000 Y0.000 Z0.000 R0		; sets babystepping (not CNC)
; Workplace coordinates
G10 L2 P1 X147.00 Y584.00 Z67.00	; origin of coordinate system number 
G10 L2 P2 X0.00 Y0.00 Z0.00		; as above for 3D printers
G10 L2 P3 X0.00 Y0.00 Z0.00		; as above for 3D printers
G10 L2 P4 X0.00 Y0.00 Z0.00		; as above for 3D printers
G10 L2 P5 X0.00 Y0.00 Z0.00		; as above for 3D printers
G10 L2 P6 X0.00 Y0.00 Z0.00		; as above for 3D printers
G10 L2 P7 X0.00 Y0.00 Z0.00		; as above for 3D printers
G10 L2 P8 X0.00 Y0.00 Z0.00		; as above for 3D printers
G10 L2 P9 X0.00 Y0.00 Z0.00		; as above for 3D printers
G54					; select coordinate system 1
M106 S0.50				; turns the fan on at half speed
M106 P0 S0.50				; turns user-configured fan on
M116					; wait for all temperatures to reach set value
G92 E0.00000				; sets the position for the extruder
M83					; sets extruder to relative mode
M486 S-1				; Indicate a non-object, purge tower, or other global feature
G17					; select XY plane for arc moves
M23 "0:/gcodes/Calibration Toolpath.gcode"	; selects the file previously running
M26 S249				; set file offset from the start of the interrupted file
G0 F6000 Z68.000			; rapid move to above saved Z height
G0 F6000 X182.895 Y658.930		; rapid move to saved XY axes
G0 F6000 Z66.000			; reaped move to saved Z height
G1 F2500.0 P0				; controlled linear move  relating to output pins
G21					; sets units to mm
M24					; start/resume SD print

infiniteloop

@nightowl999 said in Power failure and resurrection on a CNC machine:

Yes, here it is, but I'm not sure if I've done it correctly...

Thx, and well done! From a first glance, I get the impression that this resurrect.g is missing all the things which are specific to CNC routers.

Could you please try to edit this commented version and insert comment lines with the CNC specific commands you miss? Think carefully, the sequence of commands is essential, else you can wreck your machine in the worst case.

With this "script" ready, we can see if the required additions can be called from resurrect-prologue.g or if we have to write the whole M911 stuff ourselves.

Nightowl

@infiniteloop Whew, let's see how this goes...

; File "0:/gcodes/Calibration Toolpath.gcode" resume print after power failure at 2022-05-29 16:49
G21					; sets units to mm
G92 X182.895 Y658.930 Z66.000		; sets the position for X, Y and Z
G60 S1					; I think a user-defined save slot can be 0, 3, 4 or 5 (0 is default)
? T-1 P0				; I'm not sure why the tool is deselected
Assuming this data has been saved in the state.restorePoints nodes...
T0					; select the last tool used - this should be stored in the relevant state.restorePoints[*].toolNumber
Set savedSpindleSpeed			; select the last tool's spindle speed - this should be stored in the relevant state.restorePoints[0].spindleSpeeds
M98 P"resurrect-prologue.g"		; runs the user-defined resurrect-prologue.g
; M116					; not relevant for CNC
; M290 X0.000 Y0.000 Z0.000 R0		; not relevant for CNC
; Workplace coordinates
G10 L2 P1 X147.00 Y584.00 Z67.00	; I think this is necessary 
; G10 L2 P2 X0.00 Y0.00 Z0.00		; not relevant for CNC
; G10 L2 P3 X0.00 Y0.00 Z0.00		; not relevant for CNC
; G10 L2 P4 X0.00 Y0.00 Z0.00		; not relevant for CNC
; G10 L2 P5 X0.00 Y0.00 Z0.00		; not relevant for CNC
; G10 L2 P6 X0.00 Y0.00 Z0.00		; not relevant for CNC
; G10 L2 P7 X0.00 Y0.00 Z0.00		; not relevant for CNC
; G10 L2 P8 X0.00 Y0.00 Z0.00		; not relevant for CNC
; G10 L2 P9 X0.00 Y0.00 Z0.00		; not relevant for CNC
? G54					; select coordinate system 1
; M106 S0.50				; not relevant for my CNC
M106 P0 S0.50				; turns user-configured fan on (I've got a case fan configured)
; M116					; not relevant for CNC
; G92 E0.00000				; not relevant for CNC
; M83					; not relevant for CNC
? M486 S-1				; not relevant for CNC
? G17					; not sure if this is relevant for CNC - it's not in the config.g file
M23 "0:/gcodes/Calibration Toolpath.gcode"
M26 S249				; set file offset from the start of the interrupted file
G0 F6000 Z68.000			; rapid move to above saved Z height
G0 F6000 X182.895 Y658.930		; rapid move to saved XY axes
G0 F6000 Z66.000			; reaped move to saved Z height
? G1 F2500.0 P0				; controlled linear move relating to output pins
G21					; sets units to mm
M24					; start/resume SD print

Before the power failure happens, I know where the spindle information is held, as I use that in my pause.g (which includes a test M28/M29 script) and resume.g files. These may help, so I've added them below...

; pause.g
M28 savedCNC.g
echo "Duet and RepRap are fantastic"
M29
set global.savedSpindleSpeed = spindles[0].active ; sets the global variable
echo "Spindle speed saved at " ^ {global.savedSpindleSpeed} ^ "RPM" ; shows saved spindle speed in the Console
G1 Z{max(move.axes[2].userPosition+5,move.axes[2].max-5)} F2400 ; move the Z axis to a safe height		
G0 X273.5 Y560 ; move XY to a safe place
M5 ; turn the spindle off

; resume.g
if state.currentTool =-1
	echo "No tool active.  Selecting tool zero"
	T0 ; select tool zero
if state.currentTool >= 0
	echo "Spindle state on tool ", state.currentTool, " is ", spindles[state.currentTool].state
if {state.currentTool >= 0} & {tools[state.currentTool].spindleRpm >= 0}
	M3 S{global.savedSpindleSpeed} ; resume saved spindle speed
	G4 S1 ; wait 1 second to allow the spindle to spin up
	echo "Spindle speed resumed at " ^ {global.savedSpindleSpeed} ^ "RPM" ; this should show the setting was successful
G0 R1 X0 Y0 ; move X and Y back to saved work XY location
G1 R1 Z0 F240 ; move Z slowly down to saved work Z location

infiniteloop

@nightowl999 Some remarks to your resurrect.g:

G60 S1 - this saves the current coordinate values in the memory slot which is later referred to from M24 (resume print). For this to work, the coordinates must have been set before. That’s the G92 line, in front of G60.
M486 S-1 - the context is object cancellation, but I don’t quite understand the meaning. In my resurrect.g, the parameter was T-1, which is not documented but could have meant something like „don’t cancel objects“.
G1 F2500.0 P0 - this one sets the feed rate, but the P0 is special - from the GCode Reference: „Pnnnn (supported only in some builds of RepRapFirmware)“. I once figured out the port bits (IOBITS), but I have no guess whether P0 is used by our RRF or simply ignored.

All of the other entries have a potential meaning with 3D printers, depending on the setup, but I have not the faintest idea which entries you really need for your mill: do you heat a nozzle, use multiple coordinate systems or do you extrude something? OK, just see your latest post - I’ll go through that in a minute … [contemplating] …