Posts made by BryanH

Thank you, I'm running DCS in debug mode - and will be running jobs beginning tomorrow morning with the last jobs to see if I can reproduce the issue.

Syslog with the observed issue https://drive.google.com/file/d/1qGrD7uS_jXZGs1kyyUrE6AGbiEL8Fvxc/view?usp=sharing

Great, that's from the kernel log - I'm pulling the syslog to see if there's anything there.

JubileePiLogDump.txt

@Phaedrux Thanks for the info - I'm running the debug monitoring now - Found what looks like an out of memory dump in the Pi's logs at or about the time of the last failure - Should I post it here?
Thanks!

@Wally Thanks for the suggestion but I was really hoping for some diagnostics / troubleshooting. Does anyone know if we can connect the Pi to the Duet3 via USB while in operation to capture the debug output on the USB port? I've already set the DSF / DWC to debug logging and will be trying some increasingly longer prints to see if I can reproduce the issue.

@Wally Pi is powered by the Duet3 per the documentation.

Page reloads did not work.
Browser restart did not work.
Browser cache clear and restart did not work.

Power cycle of entire system required to regain access to DWC as stated in initial post.

No errors recorded in eventlog.txt

No errors recorded in console.

DWC hung again - required reboot of pi to restore access

m122
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.1.1 running on Duet 3 MB6HC v0.6 or 1.0 (SBC mode)
Board ID: 08DJM-956L2-G43S4-6J9DL-3SJ6S-1866H
Used output buffers: 1 of 40 (14 max)
=== RTOS ===
Static ram: 154604
Dynamic ram: 163732 of which 20 recycled
Exception stack ram used: 520
Never used ram: 74340
Tasks: NETWORK(ready,1972) HEAT(blocked,1088) CanReceiv(suspended,3420) CanSender(suspended,1428) CanClock(blocked,1436) TMC(blocked,68) MAIN(running,4936) IDLE(ready,76)
Owned mutexes:
=== Platform ===
Last reset 09:06:56 ago, cause: power up
Last software reset at 2020-07-22 19:13, reason: User, spinning module LinuxInterface, available RAM 74300 bytes (slot 2)
Software reset code 0x0010 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0444a000 BFAR 0x00000000 SP 0xffffffff Task MAIN
Error status: 0
MCU temperature: min 18.1, current 39.6, max 41.1
Supply voltage: min 23.9, current 24.1, max 24.2, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 11.9, current 12.0, max 12.1, under voltage events: 0
Driver 0: standstill, reads 26677, writes 27 timeouts 0, SG min/max 0/1023
Driver 1: standstill, reads 26681, writes 23 timeouts 0, SG min/max 0/55
Driver 2: standstill, reads 26678, writes 27 timeouts 0, SG min/max 0/1023
Driver 3: standstill, reads 26678, writes 27 timeouts 0, SG min/max 0/1023
Driver 4: standstill, reads 26679, writes 27 timeouts 0, SG min/max 0/1023
Driver 5: standstill, reads 26679, writes 27 timeouts 0, SG min/max 0/1023
Date/time: 2020-07-25 19:05:43
Slowest loop: 10.81ms; fastest: 0.14ms
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 37.5MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 375, MinFreeDm: 322, MaxWait: 3079109ms
Bed compensation in use: mesh, comp offset 0.008
=== MainDDARing ===
Scheduled moves: 45996, completed moves: 45996, StepErrors: 0, LaErrors: 0, Underruns: 0, 0 CDDA state: -1
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0 CDDA state: -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
Heater 0 is on, I-accum = 0.1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP* is ready with "M122" in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon* is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 2.21ms; fastest: 0.01ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions
HTTP sessions: 0 of 8

• Ethernet -
  State: disabled
  Error counts: 0 0 0 0 0
  Socket states: 0 0 0 0 0 0 0 0
  === CAN ===
  Messages sent 175312, longest wait 2ms for type 6011
  === Linux interface ===
  State: 0, failed transfers: 0
  Last transfer: 17ms ago
  RX/TX seq numbers: 6486/19231
  SPI underruns 0, overruns 0
  Number of disconnects: 8
  Buffer RX/TX: 0/0-0
  === Duet Control Server ===
  Duet Control Server v3.1.1
  Code buffer space: 4096
  Configured SPI speed: 8000000 Hz
  Full transfers per second: 29.63

config.g

I've run into this again today where the machine stopped mid-print and DWC became unresponsive. Heaters and fans remained on. No errors generated in the console or in DWC. Was able to ssh into pi, was able to VNC into pi, DWC completely unresponsive. CAN-FD sync light blinking in time, no error lights on Duet3.

Required power cycle to regain access to DWC - print was lost.

How do I troubleshoot this issue - the console log does not appear to be functioning properly as there are no entries.
Can I connect the Pi to the Duet USB and set up debugging and capture the USB output on the PI?
I currently do not trust the setup for unattended operation.

system configuration:
Duet3 6HC (SBC Pi4 4GB) + 3HC + 4 x Tool boards (with distro board)
RRF 3.1.1 + DuetPi with updates
CoreXY
XYZE is x16 microstepping Interpolated
U is x4 microstepping interpolated

I'd like to add that I am experiencing the same issue - Currently running current code on a Duet3 6HC (SBC Pi4 4GB) + 3HC + 4 x tool boards with distro board. Upon powering on the system the eventlog.txt records:

2020-07-24 16:21:59 Event logging started
2020-07-24 16:21:59 Event logging stopped

Here I issued a manual M929 S1

2020-07-24 19:21:31 Event logging started
2020-07-24 19:21:31 Event logging stopped

Thank you,

Warranty claim submitted.

Hi,
Wanted to check in and see what the next steps should be?
Thanks

Solved!

Found that a set screw had backed out that was holding the fixed pulley in place on the stepper shaft.

We found it odd In the way it was manifesting.

Thank you everyone for the assistance.

ok, it seems it's getting worse.

Just ran the job wilst recording - caught on tape was a failed lock -

• The machine attempted to pick up T0, the lock didn't finish the rotation, I can't tell if the limit switch triggered or not. I tightened up the lock by hand and noted at that time there was no holding torque on the U stepper - it was off. It finished the layer and go to the point where the tool change was called (15:42:32)

• The carriage moved T0 to the park position and placed the tool on the post then tried to run the tool_unlock.g macro, nothing happened. There was no movement of the lock. There were no reported errors at this point.

• The carriage did not pull back to move to the T1 pickup position.

• The error was reported twice at 15:43:01 and T1 became the active tool, T0 was still mounted on the carriage, then the carriage pulled back and continued the job as T1.

• T1 operated as if it were mounted on the carriage, extruding with an active part fan.

• At the end of the current layer when the next tool change was called the system attempted to part T0 in the T1 parking location - The E-stop did not immediately respond and I pulled the plug.

Is there a better logging option than the console?

Thanks,

@JoergS5 Done - let's see what happens

@JoergS5 Any reason that the M208 U0:200 can't be commented out? The Remote Elastic Lock is a dual limit switch environment and all moves are seeking an endstop.

@JoergS5 My hardware is a bit different - I've got a spring buffer on my tool posts that gets compressed during the tpost0.g code - it's something that @Miasmictruth and I were working on to see if we get a better lockup - as an aside the "TODO: read sensors" line in my macros/tool_lock.g is for the switch built into the front carriage plate that detects when all three balls of the Maxwell coupling are seated. You can see the io created in the config.g along with some general concepts related to a pause if the tool is disconnected during a print.

Just got away from work and running the job now.

@T3P3Tony Thanks, @poofjunior and I were working on it last night before I posted here. he’s thinking there’s some firmware issue since I’m using the same macros that the rest of the community is using. The major difference is that I’m using a duet3(sbc).

Thank you for the feedback, here’s some additional information.
Here is a photo of the three failures I’ve seen so far. The first two were with the M208s that are commented out in place. These limits are to keep the mounted tools from impacting the frame. The third attempt failed early.

Once the error occurs the job continues without picking up the tool. The tools continue to extrude filament and the carriage goes through the motions of each layer but just stops trying to rotate the lock.

I’ve set up a camera and am trying again while recording the activities.

Machine: Jubilee Toolchanger
Control Boards: Duet3 6HC + 3HC + Tool Distribution Board + 2 x Tool Boards
SBC Mode Pi4 4GB
Firmware: 3.1.1

Issue: Error: G1/G2/G3: Intermediate position outside machine boundaries will be generated and print will continue without picking up any tools following error. Does not appear to be invoking the macros/tool_lock.g file - Tool locking macro.

files are located here: https://github.com/BryanHaven/Jubilee-error.git

Error occurs seeming randomly

Any thoughts?