[SBC only] Solution For Persistent Global Variables

oozeBot

We wanted to share our solution for setting persistent global variables in a SQLite database on the SBC. It uses ExecOnMCode to create three new mCodes and DSF-Python to return global variables within the ObjectModel. This isn't yet 100% bulletproof, and I'm sure we'll be expanding the functionality that we can share, but it is solid enough that we have now implemented it in our machines.

This requires an understanding of installing/using ExecOnMCode, Python, SQLite, Linux, etc that we really don't have the time to teach. However, we will try to explain this in detail below.

The three new mCodes are:

M9050 - load all variables
- Inputs:
  - none
- Description: Loads all saved variables to the ObjectModel
- Use Case: Should be used in config.g to load all global variables after reset
- Example: M9050
M9055 - set variable
- Inputs:
  - N = name of variable
  - V = value of variable
- Description: Creates/Updates variable within the database and the ObjectModel
- Use Case: Should be used any time a persistent global variable is set
- Example: M9055 N"wookie" V"Chewbacca"
M9060 - delete variable
- Inputs:
  - N = name of variable
- Description: Deletes variable within the database and sets it to Null within the ObjectModel
- Use Case: Should be used any time a persistent global variable is removed
- Example: M9055 N"wookie"

A SQLite database needs to be created. There are plenty of resources on the web to explain how to do this. In the Python scripts that follow, the path/name of your database will need to be updated. Once created, connect to the database and run the following command within SQLite to create the variables table.

create table variables(name varchar(50) PRIMARY KEY NOT NULL, value varchar(255) NOT NULL);

oozeBot

Service Code for M9050

[Unit]
Description=M9050 - Load All Global Variables
After=duetcontrolserver.service
Requires=duetcontrolserver.service

[Service]
ExecStart=/usr/local/bin/execonmcode -mCode 9050 -command "./mCodes/M9050.py" -returnOutput
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Python script for M9050.py

#!/usr/bin/env python3

import sys
import sqlite3

from dsf.connections import CommandConnection
from dsf.commands.basecommands import MessageType
from dsf.commands.basecommands import LogLevel

Database = "/oozeBot/sql.db"

########################################################################################################

def UpdateVariable(name:str, value:str):
  command_connection = CommandConnection()
  command_connection.connect()
  command_connection.perform_simple_code("if !exists(global." + name + ") || global." + name + "=null")
  command_connection.perform_simple_code(" global " + name + "=null")
  command_connection.perform_simple_code("else")
  command_connection.perform_simple_code(" set global." + name + "=null")
  command_connection.perform_simple_code("set global." + name + "=\"" + value + "\"")
  command_connection.close()

########################################################################################################

def LoadAll():
  con = sqlite3.connect(Database)
  cur = con.cursor()
  res = cur.execute("select * from variables;").fetchall()
  for row in res:
    UpdateVariable(row[0] , row[1])
  con.close()

########################################################################################################

def RecordsExists():
  con = sqlite3.connect(Database)
  cur = con.cursor()
  res = cur.execute("select count(*) from variables;").fetchone()
  return bool(res[0])
  con.close()

########################################################################################################

if __name__ == "__main__":
  if (RecordsExists()):
    LoadAll()

oozeBot

Service Code for M9055

[Unit]
Description=M9055 - Set Global Variable
After=duetcontrolserver.service
Requires=duetcontrolserver.service

[Service]
ExecStart=/usr/local/bin/execonmcode -mCode 9055 -command "./mCodes/M9055.py %%N %%V" -returnOutput
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Python script for M9055.py

#!/usr/bin/env python3

import sys
import sqlite3

from dsf.connections import CommandConnection
from dsf.commands.basecommands import MessageType
from dsf.commands.basecommands import LogLevel

Database = "/oozeBot/sql.db"

########################################################################################################

def UpdateVariable(name:str, value:str):
  command_connection = CommandConnection()
  command_connection.connect()
  command_connection.perform_simple_code("if !exists(global." + name + ") || global." + name + "=null")
  command_connection.perform_simple_code(" global " + name + "=null")
  command_connection.perform_simple_code("else")
  command_connection.perform_simple_code(" set global." + name + "=null")
  command_connection.perform_simple_code("set global." + name + "=\"" + value + "\"")
  command_connection.close()

########################################################################################################

def RecordExists(name:str):
  con = sqlite3.connect(Database)
  cur = con.cursor()
  res = cur.execute("select count(*) from variables where name= \"" + name + "\";").fetchone()
  return bool(res[0])
  con.close()

########################################################################################################

def UpdateRecord(name:str, value:str):
  con = sqlite3.connect(Database)
  cur = con.cursor()
  cur.execute("update variables set value=\"" + value + "\" where name=\"" + name + "\";")
  con.commit()
  con.close()

########################################################################################################

def InsertRecord(name:str, value:str):
  con = sqlite3.connect(Database)
  cur = con.cursor()
  cur.execute("insert into variables (name, value) values (\"" + name + "\", \"" + value + "\");")
  con.commit()
  con.close()

########################################################################################################

if __name__ == "__main__":
  if (RecordExists(sys.argv[1])):
    UpdateRecord(sys.argv[1], sys.argv[2])
  else:
    InsertRecord(sys.argv[1], sys.argv[2])
  UpdateVariable(sys.argv[1], sys.argv[2])

oozeBot

Service Code for M9060

[Unit]
Description=M9060 - Delete Global Variable
After=duetcontrolserver.service
Requires=duetcontrolserver.service

[Service]
ExecStart=/usr/local/bin/execonmcode -mCode 9060 -command "./mCodes/M9060.py %%N" -returnOutput
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Python script for M9060.py

#!/usr/bin/env python3

import sys
import sqlite3

from dsf.connections import CommandConnection
from dsf.commands.basecommands import MessageType
from dsf.commands.basecommands import LogLevel

Database = "/oozeBot/sql.db"

########################################################################################################

def NullVariable(name:str):
  command_connection = CommandConnection()
  command_connection.connect()
  command_connection.perform_simple_code("if !exists(global." + name + ") || global." + name + "=null")
  command_connection.perform_simple_code(" global " + name + "=null")
  command_connection.perform_simple_code("else")
  command_connection.perform_simple_code(" set global." + name + "=null")
  command_connection.close()

########################################################################################################

def RecordExists(name:str):
  con = sqlite3.connect(Database)
  cur = con.cursor()
  res = cur.execute("select count(*) from variables where name= \"" + name + "\";").fetchone()
  return bool(res[0])
  con.close()

########################################################################################################

def DeleteRecord(name:str):
  con = sqlite3.connect(Database)
  cur = con.cursor()
  cur.execute("delete from variables where name=\"" + name + "\";")
  con.commit()
  con.close()

########################################################################################################

if __name__ == "__main__":
  if (RecordExists(sys.argv[1])):
    DeleteRecord(sys.argv[1])
    NullVariable(sys.argv[1])

oozeBot

Example:

M9055 N"wookie" V"Chewbacca"
echo global.wookie
M9055 N"wookie" V"Chewy"
echo global.wookie
M9060 N"wookie"
echo global.wookie

Console Output:

resam

@oozebot great idea of putting this into a small sqlite DB!

If you want to simplify the files and all the exec-on-m-code invocations a bit, you can directly subscribe to M-codes from a single Python script started by a single systemd service unit:

from dsf.commands.basecommands import MessageType
from dsf.connections import CommandConnection, InterceptConnection, SubscribeConnection
from dsf.initmessages.clientinitmessages import InterceptionMode, SubscriptionMode

...

            filters = ("M9055", "M9060")
            intercept_connection = InterceptConnection(
                InterceptionMode.PRE, filters=filters
            )
            intercept_connection.connect()
            while True:
                cde = intercept_connection.receive_code()
                if cde.majorNumber == 9055:
                     print("do something with parameter B", cde.parameter("B", 0))
                     intercept_connection.resolve_code()
               elif cd.majorNumber == 9060:
                    print("do something else")
                    intercept_connection.resolve_code()
               ...

oozeBot

@resam Thanks for the pointer. I knew of this but hadn't had a chance to dig into it yet. I'm guessing this is pretty close to how ExecOnMCode works. I'll give this a go here soon!

mikeabuilder

This is and interesting thread. I'm not using an SBC but I've recently implemented a new macro that I call "create_global_variables.g". I run this macro as one of the earliest lines in config.g. What I have in the file is a bunch of "global <variable> = whatever" lines. I try not to create global variables elsewhere.

I did this for a few reasons:

I can set the value of a global in any macro without first testing that it has been created. Makes easier to read gcode.
I can more easily manage my globals and not have duplicates. I add new variables in alphabetic order.
This file has "the feel" of a C header file. I'm not a C coder (I like python), but I do understand the header file concept.

It would be easy (but cumbersome and eventually a headache) to programmatically add new variables (or to change values) in this macro file, by using the "echo >>" (append to file) command in RRF3.4. New entries would not be in alphabetical order and eventually changes in values would leave a long collection of useless code. I suppose daemon.g could kick off another macro for cleanup of the file...

Clearly not as useful as a real database, but can get the job done.

oozeBot

@mikeabuilder This was posted, in part, to show a proof of concept in returning external data into DSF. Setting persistent global variables is a good use case, but I already have several more complex ideas in mind!