Logging

fma

Logs are a good idea. But I vote to have them turned off by default, to avoid SD card writings if not necessary.

Having them enable/disabled on the fly would be great, to allow logging of just a part of a print, for exemple.

Do you plan to have configurable logs level (DEBUG/INFO/WARNING/ERROR/CRITICAL)?

WalterSKW

Logs :
High level log : keeps track of startup, printing a object …
kind of low level log for root-cause analysis. I would not opt for having different levels of logging in that one. All or nothing.

IMHO : size should be limited. One could switch between 2 files with a fixed max size and have an indicator to the active one.

Time :
Why not having the printer time automagically set from the DWC ? PC-clock -> printer clock ...

Starting and configuring with G-code (what else?) is a good idea. Size, max number of files...
Fixed location on SD card \logs\ ore something likely

Start/stop of the detailed logging from the DWC

DjDemonD

@dc42:

A major problem with auto archiving is that the date and time are not always available. In particular, when logging is started (which is the obvious time to decide whether the old log file should be archived and a new one created), it generally isn't.

Tab should be usable as a field separator, although it may cause issues depending on what program is used to display the log. Comma might be problematic because an error message might contain one.

Maybe each log entry could just be indexed with a number from 00000 onwards in the same way a digital camera numbers photographs taken, then if date and time are unavailable it will still be clear that print/session 00001 occurred before 00005 even if 00005 has a date/time field filled and 0001 does not.

Would love to see "global statistics" for the printer, run hours, filament used, etc… And for this to be able to be backed up so if I had to change duet card or upgrade I could continue the log.

The sample log entry looks good. Maybe dump an M122 to the end of it after each print, that wouldn't be too onerous, would it?

Definitely have the option to turn logging on/off and configure the level of detail logged, although I'm assuming you aren't too concerned the mighty duet will struggle to process/store this data even if it is verbose. But on by default on new units, as this would be a huge benefit in finding faults for new users.

T3P3Tony

@WalterSKW:

Logs :

Time :
Why not having the printer time automagically set from the DWC ? PC-clock -> printer clock …

People, including myself, use the Panel Due to run prints without connecting via DWC each time. This is especially the case when reprinting the same file a number of times over a number of days.

While time can't be guaranteed it should be used where possible.

zov

@T3P3Tony:

@WalterSKW:

Logs :

Time :
Why not having the printer time automagically set from the DWC ? PC-clock -> printer clock …

People, including myself, use the Panel Due to run prints without connecting via DWC each time. This is especially the case when reprinting the same file a number of times over a number of days.

While time can't be guaranteed it should be used where possible.

Maybe NTP might be an answer (printer is permanently connected to net).

dc42

I need a solution that works for everyone - even those few people who never use the web interface.

fma

A 3D printer is not really a security system as a firewall is. So, is it really important to have a lot of previous logs?

I would just do some simple log rotate based on number of files and size of each file. Both customisable with G-Codes (duet.log for current file, duet.log.1 for previous one, duet.log.2 for previous of previous…)

Then, for the date in the log, use real date if available (ntp), or date from Duet start on if not. Like:

[[language]]
2017-09-19 21:05:27::INFO::Start printing xxx
2017-09-19 21:05:29::LOG::my_var=3
...
2017-09-21 05:08:47::INFO::End of print xxx

or:

[[language]]
0000-00-00 00:00:00::INFO::Start printing xxx
0000-00-00 00:00:02::LOG::my_var=3
...
0000-00-01 08:03:20::INFO::End of print xxx

WalterSKW

@dc42:

I need a solution that works for everyone

Only one solution then : a RTC.
For NTP the printer has to reach out to the internet. I guess not everyone allows that.
Date/time setting could also be implemented as a function on the PanelDue for those not using DWC

Edit : Adding a small RTC board stacked on the PT100/Thermocouple board connector? (http://www.st.com/content/ccc/resource/technical/document/datasheet/group2/40/3f/91/e4/91/a1/47/7c/CD00129012/files/CD00129012.pdf/jcr:content/translations/en.CD00129012.pdf)

zov

But nowadays almost all consumer router have built-in NTP server so no need for WAN access for 3d printer. And I think that NTP would be allright for a big part of users.
Added: Who doesn't need DWC is no need for a precise time in log if nedd log at all. They can be provided with local time of Duet (time from board power up). Any decent investigation of a problem would require time stamp of events IMO.

Whitewolf

Possibly add Riemann logging ability… this way a farm of printers can send these streams to a remote system for logging and monitoring with dashboard alerts and metrics etc.

Riemann is an agentless monitoring system that would work perfectly for something like this and is recommended by and supported by Docker:
https://blog.docker.com/2016/03/realtime-cluster-monitoring-docker-swarm-riemann/

Why Riemann in particular?
There are a variety of reasons to use Riemann. Here are a few of the most obvious.
Riemann’s push-based model lets you detect problems and verify that fixes are working in near real-time.
Traditional monitoring systems such as Nagios usually work with a poll-based model: They wake up periodically (say, every 5 minutes), run a series of health checks to verify that everything in the system is in the desired state (e.g. your services are up), and report the results to you via e-mail or another method. Riemann requires that services send their own events, and consequently outages or network partitions can be detected as soon as the events stop flowing. Likewise, we can collect and process events of any type, opening the doors to proactively providing value through monitoring instead of simply fighting fires.
Riemann’s stream processing allows for powerful transformations of incoming data.
For instance, even if an exception is logged 1000 times in an hour, Riemann can roll up all exceptions of that type into just a few e-mails with information about how many times the exception occured. This helps combat alarm fatigue. Another example- Riemann makes measuring metrics in percentiles quite trivial. This allows to gain true insight into the operations of our systems without having important information masked behind aggregations such as averages.
Riemann is easy to get data into or out of, allowing interesting use cases such as storing event data for analysis later on.
Clients for Riemann can send events using a thin protocol buffers layer over TCP or UDP. Riemann can emit events and metrics from its index into a variety of backends including Graphite, InfluxDB, Librato, and more. Riemann also has built-in support for notification via SMS, e-mail, Slack, etc.
Riemann’s architectural model is simple, preventing the potentially ironic situation of having monitoring software that is difficult to operate and/or failing often.
Riemann is designed from the ground up to be straightforward to operate and reason about. While this has some tradeoffs (for instance, it makes it impossible to distribute a single Riemann instance safely), in practice Riemann’s philosophy is largely that imperfect information right now is better than perfect information which never arrives. If the Riemann server crashes, usually a simple restart will remedy the problem; no finicky reconciling of distributed state needs to occur.
Likewise, the push-based model helps to alleviate the “who monitors the monitoring software” question (i.e. how do we detect when our monitoring itself has issues). We can simply set up a few Riemann servers and forward events to the downstream servers. If the upstream server goes down, the downstream servers will notice this and alert you. Especially running Docker across clouds, this could be particularly valuable.
It’s rather fast.
From the landing page: “Throughput depends on what your streams do with events, but a stock Riemann config on commodity x86 hardware can handle millions of events per second at sub-ms latencies, with 99ths around 5ms. Riemann is fully parallel and leverages Clojure and JVM concurrency primitives throughout.”