Duet3D Logo Duet3D
    • Tags
    • Documentation
    • Order
    • Register
    • Login

    Duet 2.05 memory leak?

    Scheduled Pinned Locked Moved
    Firmware installation
    9
    132
    6.8k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • kazolarundefined
      kazolar
      last edited by

      @T3P3Tony I started setting up Cura. It's taking some time to get it setup with the extruders -- I kinda remember why I paid for S3D, cura is slow now as it was 5 years ago. I can see it's a more complex geometry -- I'll try to re-slice it in Cura for next print and see where it goes

      1 Reply Last reply Reply Quote 0
      • kazolarundefined
        kazolar
        last edited by

        @T3P3Tony so just moving the basics into cura -- it's estimating the time spot on -- but it's generating a gcode that is 50% bigger -- probably that will cause more problems than fix them? I could try it, but my thinking was to eliminate small movements -- well I adjusted the mesh in Fusion 360 also to be less dense, and cut the gcode down a bit - but it still hit underrun errors. I am not sure the point in trying a different slicer if the gcode I am getting is bigger -- either it will do a lot more travels, or create even more small movements

        1 Reply Last reply Reply Quote 0
        • dc42undefined
          dc42 administrators
          last edited by

          @kazolar, when the problem occurs, have you tried uploading a different file to SD card, and then printing the original file again? I'm wondering whether it is the act of writing to the SD card that clears the problem.

          Also, are you running with logging (M929) enabled or disabled? Logging will generate occasional writes to the SD card, e.g. when a print starts or finishes.

          Duet WiFi hardware designer and firmware engineer
          Please do not ask me for Duet support via PM or email, use the forum
          http://www.escher3d.com, https://miscsolutions.wordpress.com

          1 Reply Last reply Reply Quote 0
          • kazolarundefined
            kazolar
            last edited by

            @dc42 i've been trying different things all night -- doesn't seem to be sd card specific. I tried printing from octoprint -- bypassing the SD card, and still hit underruns. I don't have M929 enabled. It seems that a full reset does clear the cobwebs -- I kept trying to preheat then clean the nozzle -- then print, but the print that is working fine -- 2+ hours into it now, I preheated, cleaned the nozzle, then hit reset -- then print -- that worked.

            dc42undefined 1 Reply Last reply Reply Quote 0
            • dc42undefined
              dc42 administrators @kazolar
              last edited by dc42

              @kazolar said in Duet 2.05 memory leak?:

              It seems that a full reset does clear the cobwebs

              Is that with or without powering down and up again?

              @kazolar said in Duet 2.05 memory leak?:

              I tried printing from octoprint -- bypassing the SD card, and still hit underruns.

              Underruns are likely when printing over USB if the GCode file contains sequences of short segments, unless the sending program knows that when talking to a Duet, it doesn't need to wait for an OK reply to each command. I've seen many reports (not from Duet users) that printing from Octoprint is sometimes slower than printing from SD card. So results using Octoprint are only significant if you compare multiple runs using Octoprint.

              Duet WiFi hardware designer and firmware engineer
              Please do not ask me for Duet support via PM or email, use the forum
              http://www.escher3d.com, https://miscsolutions.wordpress.com

              1 Reply Last reply Reply Quote 0
              • kazolarundefined
                kazolar
                last edited by

                @dc42 i was able to get it going on again by simply hitting reset -- no power down, and immediately select a file to print. That's how it's printing now. It seems that if I do anything prior to initiating the print -- it starts to show significant max times in loop and then at the 30 minute mark it can't keep up and it starts getting underruns. I tweaked octoprint settings not to wait for OK before I started because it was timing out on basic, stuff -- it still hit underrun at about the same point. And I could see in DWC console when I did m122 -- I'd see loop times in the 50ms -- so it clearly wasn't going well. I check the model and the file for small segments, and there aren't any. I took that set of models into cura with adjustments for minimum travels, and it generated a bigger gcode file than S3D -- I'll try prusa slicer -- not sure what else to try. The problem has gotten progressively worse. The gcode file I'm printing 12 mb, the cura generated one with adjustments for minimum segments is 19mb -- I can print that file next and see what happens?

                1 Reply Last reply Reply Quote 0
                • T3P3Tonyundefined
                  T3P3Tony administrators
                  last edited by

                  The different size could be Cura being more verbose with comments so its still worth trying.

                  www.duet3d.com

                  kazolarundefined 1 Reply Last reply Reply Quote 0
                  • kazolarundefined
                    kazolar @T3P3Tony
                    last edited by

                    @T3P3Tony I did a Prusa Slicer version -- it's a little bit bigger, but the paths look good -- I'm trying it now -- the previous print just finished -- I only added my custom header and tail to the prusa version to make sure the right stuff happens -- did not reset the board -- just uploaded the prusa version and see what happens.

                    1 Reply Last reply Reply Quote 0
                    • kazolarundefined
                      kazolar
                      last edited by

                      @T3P3Tony Prusa version was doing funny things with perimeters -- sorta failed on it's own, kinda chewed up the perimeters of the model, I canceled it -- I'd need to mess with it to make it work -- Cura is next.

                      1 Reply Last reply Reply Quote 0
                      • kazolarundefined
                        kazolar
                        last edited by

                        @T3P3Tony ok so the cura version -- 3rd print without a reset - right as it got to post5.g -- it was doing it in slow motion -- doing a move -- thinking doing another -- move. So I reset it -- will try again

                        1 Reply Last reply Reply Quote 0
                        • kazolarundefined
                          kazolar
                          last edited by

                          @T3P3Tony ok -- so cura sliced code seemed a bit more sane than Prusa, but it didn't look good, but I let it run -- and it just failed -- underruns and stuttering.
                          I optimized it for short paths to simplify it -- still...same result. Gonna clean the bed. Reset and kick of the S3D print -- that actually looks good and works after a reset (fingers crossed)

                          1 Reply Last reply Reply Quote 0
                          • kazolarundefined
                            kazolar
                            last edited by

                            @T3P3Tony @dc42
                            Here is where I am at --

                            1. I am running 2.05RC -- 2.05.1 is a lot less predictable -- it seems to work ok, then stops working and gets into stuttering underrun condition. I can't trust it from print to print, so I went back to 2.05RC -- it's at repeatable, and I can get shields out now without checking on it every 5 minutes.
                            2. I can tell right away from m122 after the print starts, actually starts laying down plastic -- if I run M122 two times in 30 second intervals, if the 2nd time I get
                              Slowest loop: 3.99ms; fastest: 0.08ms
                              Then it will work through the end of the print -- I can tell that within 5 minutes now of starting the print before waiting the full 30 minutes to get underruns ans stuttering .
                              If I get
                              Slowest loop: 63.64ms; fastest: 0.07ms
                              Then it will underrun by 30 minute mark.
                            3. This is what works. I preheated the bed, nozzles -- clean everything off - press reset -- then start the S3D print -- and this works. The print will get slowest loop in 5ms range and will run through to conclusion - 5hrs 15 min give or take.
                            4. This procedure works -- and has worked for the entire month I've been printing shields until I started messing around with 2.05.1. I don't have to copy the file over, I can keep using the same file.
                            5. If I do anything prior to starting the print without pressing reset -- I will get the high slowest loop numbers and inevitable underrun.
                            6. The behavior is absolutely repeatable with S3D or Cura prints (in terms of underrun condition). I am sure I can tune the Cura version to look better, and probably the Prusa version, both looked like trash on layer one to varying degrees -- but both were optimized to eliminate small movements, S3D doesn't have that feature, but this print doesn't have any -- or many.

                            Question is -- is this hardware or software. I have done triplicate prints before without having to do the "three finger salute" before starting it -- so either something died in the hardware, David's documentation on underrruns talks about voltage regulators, etc. I have no indication that something is wrong there, but the only alteration from then and now, is I switched to optical end stops on XYUVWA axes - the others (for the bed -- are still mechanical). I also went from 32 teeth pulleys to 20 teeth. I have done a 40 hour long multi extrusion print with the current setup with no issues.

                            1 Reply Last reply Reply Quote 1
                            • kazolarundefined
                              kazolar
                              last edited by

                              Another update -- I did not power down -- did the exact some procedure -- I removed the shields -- set bed temp, set extruder temp (keep bed hot and set extruders to 150 so I can pull of any stuck on filament) -- then after alcohol wipe down of the build area, i pressed stop on LCD, the select the print -- it's working perfectly -- THE only way I can print in triplicate reliably is to do this process. I don't need to delete and reupload the file. No other tricks involved. If I try printing it again 50/50 chance it will spin up the loop time and will fail due to underruns. If I keep doing this process, I can keep doing a lot of shields with no interruption.
                              There is a leak, or some missed clean up routine in this triplicate mode -- i have offsets specified such that my bed 0 left front -- I don't use 0,0 at center -- to convoluted on non-delta machines. So my offsets correspond with 0,0 at front left, triplication works perfectly fine -- mirror works fine -- so far as I can demonstrate, triplication in 2.05RC works only after a reset, and in 2.05.1 it works kinda better, but only that i can do more prints before a failure, and I can't predict when it will happen, so kinda not very helpful -- I am doing the shields while doing my day job -- so I can't be monitoring M122 results every few seconds -- it's not practical -- do the setup -- reset the firmware -- press print -- walk away -- that works, rinse repeat.

                              droftartsundefined 1 Reply Last reply Reply Quote 0
                              • dc42undefined
                                dc42 administrators
                                last edited by dc42

                                Just to confirm: if you do a software reset of the Duet, for example using the Emergency Stop button or M999, it clears the problem sufficiently to do another print?

                                If that's the case, please can you do the following next time:

                                • Send M122 after powering up and resetting the printer
                                • After the print has finished, send M122 again
                                • Then send M21 and see whether that is sufficient to allow a new print to succeed. If the M21 command reports an error, send M22 and then M21.

                                Post the M122 results here.

                                Duet WiFi hardware designer and firmware engineer
                                Please do not ask me for Duet support via PM or email, use the forum
                                http://www.escher3d.com, https://miscsolutions.wordpress.com

                                1 Reply Last reply Reply Quote 0
                                • droftartsundefined
                                  droftarts administrators @kazolar
                                  last edited by

                                  @kazolar I had a thought. This only happens when you run the 3rd hot end, in triplication, and not in mirror/duplication. Is the 3rd hot end heater connected to the Duex? I wonder if something about the PWM on the 3rd heater is interfering with the SD card. One obvious candidate is that you have not tied the grounds from the Duet and Duex together; see https://duet3d.dozuki.com/Wiki/Duex2_and_Duex5_Features#Section_Wiring

                                  Alternatively, there may be subtle differences between the firmwares that cause the PWM frequency to be different, or the signalling between the boards to shift, enough to cause a stutter in the command queue (as you have had the same problem streaming from Octoprint/USB). This is just a guess; I think the PWM code has been unchanged for a long time.

                                  Ian

                                  Bed-slinger - Mini5+ WiFi/1LC | RRP Fisher v1 - D2 WiFi | Polargraph - D2 WiFi | TronXY X5S - 6HC/Roto | CNC router - 6HC | Tractus3D T1250 - D2 Eth

                                  1 Reply Last reply Reply Quote 0
                                  • kazolarundefined
                                    kazolar
                                    last edited by kazolar

                                    @dc42 I'll run the test this weekend, Sunday most likely. I need to get as many shields made as possible today and tomorrow -- I have a pickup of 200 shields for a local hospital on Sunday morning.
                                    @droftarts I actually had some issues with i2c a while back, and David suggested to thicken up the grounding wire between duet and duex5 -- I got the thickest wire that could be possibly inserted into a ferrule and did that -- also David had added some code to reset i2c if it gets an issue -- but my m122 has not shown any i2c errors since I improved the grounding -- but trust me there is a very short 14awg silicone shielded wire crimped in ferrules that is connecting my 2 boards -- so my grounding between them is no longer an issue.

                                    1 Reply Last reply Reply Quote 0
                                    • kazolarundefined
                                      kazolar
                                      last edited by kazolar

                                      @dc42 M21 and combined with M22 do not help, I just tried it after I replaced the 4 Z stepper connectors, recall a while back I had an issue with the included stepper connectors getting singed, you suggest molex, and I had those in place for well over a year with no singing until today -- this time the singing was very bad melting the housing. I ended up taking duex5 out and I soldered wires with heavy duty JS style connectors in the place of the Z stepper connectors (I have 4 lead screws -- upon further inspection 2 had developed some singe marks.
                                      Singing.png

                                      I didn't save M122 -- I will make sure to save it on Sunday, I hope not to be dealing with unrelated electrical issues. What was telling that even with M21 being run, the loop times were high -- max loop times until stuttering started were 50-70ms. None were lower than 12ms -- I check M122 every minute or so. When stuttering started I canceled the print -- kept the bed at temperature, cleared everything, then I reset the board -- just M999 (or stop button in this case) -- then I started the same print -- it's running now -- all loop times are 5ms and lower.

                                      1 Reply Last reply Reply Quote 0
                                      • kazolarundefined
                                        kazolar
                                        last edited by

                                        @dc42
                                        here is the M122 right before it starts failing
                                        M122
                                        What's interesting -- which makes this look more like a hardware issue is that the first print after the machine has been off for a couple of hours fails. High loop times -- then underruns. Then prints done afterwards work. I did not do anything different than preheat -- reset, hit print -- first print fails the same fashion

                                        === Diagnostics ===
                                        RepRapFirmware for Duet 2 WiFi/Ethernet version 2.05 running on Duet Ethernet 1.02 or later + DueX5
                                        Board ID: 08DGM-9T6BU-FG3S0-7JTD4-3S06K-1A4ZD
                                        Used output buffers: 1 of 24 (21 max)
                                        === RTOS ===
                                        Static ram: 25708
                                        Dynamic ram: 96332 of which 0 recycled
                                        Exception stack ram used: 472
                                        Never used ram: 8560
                                        Tasks: NETWORK(ready,616) HEAT(blocked,1144) DUEX(blocked,164) MAIN(running,1668) IDLE(ready,156)
                                        Owned mutexes: I2C(DUEX)
                                        === Platform ===
                                        Last reset 00:20:52 ago, cause: software
                                        Last software reset at 2020-04-25 11:06, reason: User, spinning module GCodes, available RAM 8504 bytes (slot 1)
                                        Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f000 BFAR 0xe000ed38 SP 0xffffffff Task 0x4e49414d
                                        Error status: 0
                                        Free file entries: 9
                                        SD card 0 detected, interface speed: 20.0MBytes/sec
                                        SD card longest block write time: 0.0ms, max retries 0
                                        MCU temperature: min 25.6, current 26.0, max 26.2
                                        Supply voltage: min 24.1, current 24.4, max 24.6, under voltage events: 0, over voltage events: 0, power good: yes
                                        Driver 0: ok, SG min/max 0/313
                                        Driver 1: standstill, SG min/max not available
                                        Driver 2: standstill, SG min/max 0/252
                                        Driver 3: ok, SG min/max 0/305
                                        Driver 4: ok, SG min/max 0/332
                                        Driver 5: standstill, SG min/max not available
                                        Driver 6: standstill, SG min/max 64/229
                                        Driver 7: standstill, SG min/max 144/295
                                        Driver 8: standstill, SG min/max 88/251
                                        Driver 9: standstill, SG min/max 41/218
                                        Date/time: 2020-04-25 11:27:19
                                        Cache data hit count 2554842663
                                        Slowest loop: 47.93ms; fastest: 0.08ms
                                        I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
                                        === Move ===
                                        Hiccups: 0, FreeDm: 154, MinFreeDm: 8, MaxWait: 0ms
                                        Bed compensation in use: none, comp offset 0.000
                                        === DDARing ===
                                        Scheduled moves: 8715, completed moves: 8675, StepErrors: 0, LaErrors: 0, Underruns: 6, 0
                                        === Heat ===
                                        Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1
                                        Heater 0 is on, I-accum = 1.0
                                        Heater 1 is on, I-accum = 0.6
                                        Heater 2 is on, I-accum = 0.7
                                        Heater 4 is on, I-accum = 0.5
                                        === GCodes ===
                                        Segments left: 1
                                        Stack records: 2 allocated, 0 in use
                                        Movement lock held by null
                                        http is idle in state(s) 0
                                        telnet is idle in state(s) 0
                                        file is doing "G1 X228.135 Y232.279 E1.2574" in state(s) 0
                                        serial is idle in state(s) 0
                                        aux is idle in state(s) 0
                                        daemon is idle in state(s) 0
                                        queue is idle in state(s) 0
                                        autopause is idle in state(s) 0
                                        Code queue is empty.
                                        === Network ===
                                        Slowest loop: 7.26ms; fastest: 0.06ms
                                        Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
                                        HTTP sessions: 2 of 8
                                        Interface state 5, link 100Mbps full duplex

                                        1 Reply Last reply Reply Quote 0
                                        • dc42undefined
                                          dc42 administrators
                                          last edited by

                                          Is there any metalwork (e.g. printer enclosure) close to the SD card socket? If so, is that metalwork connected to Duet ground, either directly or through a resistor? I'm wondering whether static buildup might be a factor.

                                          Duet WiFi hardware designer and firmware engineer
                                          Please do not ask me for Duet support via PM or email, use the forum
                                          http://www.escher3d.com, https://miscsolutions.wordpress.com

                                          1 Reply Last reply Reply Quote 0
                                          • kazolarundefined
                                            kazolar
                                            last edited by

                                            @dc42 nope, printer is in a plastic enclosure -- no metal parts touch the network jack
                                            Here is the actual failure:
                                            12:22:43 PMM122
                                            === Diagnostics ===
                                            RepRapFirmware for Duet 2 WiFi/Ethernet version 2.05 running on Duet Ethernet 1.02 or later + DueX5
                                            Board ID: 08DGM-9T6BU-FG3S0-7JTD4-3S06K-1A4ZD
                                            Used output buffers: 1 of 24 (22 max)
                                            === RTOS ===
                                            Static ram: 25708
                                            Dynamic ram: 96332 of which 0 recycled
                                            Exception stack ram used: 448
                                            Never used ram: 8584
                                            Tasks: NETWORK(ready,616) HEAT(blocked,1144) DUEX(suspended,164) MAIN(running,1668) IDLE(ready,156)
                                            Owned mutexes:
                                            === Platform ===
                                            Last reset 00:37:46 ago, cause: software
                                            Last software reset at 2020-04-25 11:44, reason: User, spinning module GCodes, available RAM 8748 bytes (slot 0)
                                            Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0441f000 BFAR 0xe000ed38 SP 0xffffffff Task 0x4e49414d
                                            Error status: 0
                                            Free file entries: 9
                                            SD card 0 detected, interface speed: 20.0MBytes/sec
                                            SD card longest block write time: 0.0ms, max retries 0
                                            MCU temperature: min 25.9, current 26.2, max 26.6
                                            Supply voltage: min 24.1, current 24.3, max 24.6, under voltage events: 0, over voltage events: 0, power good: yes
                                            Driver 0: ok, SG min/max 0/333
                                            Driver 1: standstill, SG min/max not available
                                            Driver 2: ok, SG min/max 0/1023
                                            Driver 3: ok, SG min/max 0/721
                                            Driver 4: ok, SG min/max 0/332
                                            Driver 5: standstill, SG min/max not available
                                            Driver 6: standstill, SG min/max 26/241
                                            Driver 7: standstill, SG min/max 135/297
                                            Driver 8: standstill, SG min/max 71/249
                                            Driver 9: standstill, SG min/max 29/222
                                            Date/time: 2020-04-25 12:22:39
                                            Cache data hit count 4294967295
                                            Slowest loop: 151.20ms; fastest: 0.08ms
                                            I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
                                            === Move ===
                                            Hiccups: 0, FreeDm: 120, MinFreeDm: 6, MaxWait: 0ms
                                            Bed compensation in use: none, comp offset 0.000
                                            === DDARing ===
                                            Scheduled moves: 26743, completed moves: 26706, StepErrors: 0, LaErrors: 0, Underruns: 69, 78
                                            === Heat ===
                                            Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1
                                            Heater 0 is on, I-accum = 1.0
                                            Heater 1 is on, I-accum = 0.6
                                            Heater 2 is on, I-accum = 0.6
                                            Heater 4 is on, I-accum = 0.5
                                            === GCodes ===
                                            Segments left: 0
                                            Stack records: 2 allocated, 0 in use
                                            Movement lock held by null
                                            http is idle in state(s) 0
                                            telnet is idle in state(s) 0
                                            file is idle in state(s) 0
                                            serial is idle in state(s) 0
                                            aux is idle in state(s) 0
                                            daemon is idle in state(s) 0
                                            queue is idle in state(s) 0
                                            autopause is idle in state(s) 0
                                            Code queue is empty.
                                            === Network ===
                                            Slowest loop: 77.03ms; fastest: 0.06ms
                                            Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
                                            HTTP sessions: 1 of 8
                                            Interface state 5, link 100Mbps full duplex

                                            dc42undefined kazolarundefined 2 Replies Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Unless otherwise noted, all forum content is licensed under CC-BY-SA