Cannot read file, error code 1 (5 times in 2 days)

Vlad

@dc42 Also, if it helps, I always use the button "print another" as I am printing 200pcs. Maybe it has something to do with that.

Here is M122:

6/26/2019, 7:16:37 PM: Connected to 192.168.1.145
6/26/2019, 7:19:06 PM: Upload of B-V1.0.gcode successful after 2s
6/26/2019, 7:19:06 PM: M32 "0:/gcodes/B-V1.0.gcode": File 0:/gcodes/B-V1.0.gcode selected for printing
6/26/2019, 7:19:33 PM: : Warning: M73 command is not supported
6/26/2019, 7:21:05 PM: M566 X600 Y600 Z8 E150
6/26/2019, 7:48:08 PM: : Finished printing file 0:/gcodes/B-V1.0.gcode, print time was 0h 29m
6/26/2019, 7:56:43 PM: M32 "0:/gcodes/B-V1.0.gcode": File 0:/gcodes/NV-Box-V1.0.gcode selected for printing
6/26/2019, 7:58:01 PM: : Warning: M73 command is not supported
6/26/2019, 8:26:37 PM: : Finished printing file 0:/gcodes/NV-Box-V1.0.gcode, print time was 0h 30m
6/26/2019, 8:29:48 PM: M32 "0:/gcodes/B-V1.0.gcode": File 0:/gcodes/B-V1.0.gcode selected for printing
6/26/2019, 8:30:45 PM: : Warning: M73 command is not supported
6/26/2019, 8:31:51 PM: M566 X400 Y400 Z8 E150
6/26/2019, 8:45:16 PM: : Error: Cannot read file, error code 1
Cancelled printing file 0:/gcodes/B-V1.0.gcode, print time was 0h 15m
6/26/2019, 8:48:41 PM: M120
6/26/2019, 8:49:00 PM: M122: === Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 2.03 running on Duet WiFi 1.02 or later
Board ID: 08DGM-917DA-G4MS8-6JKD0-3S86J-K9SBA
Used output buffers: 3 of 24 (20 max)
=== RTOS ===
Static ram: 25680
Dynamic ram: 93664 of which 0 recycled
Exception stack ram used: 452
Never used ram: 11276
Tasks: NETWORK(ready,524) HEAT(blocked,1236) MAIN(running,3748) IDLE(ready,160)
Owned mutexes:
=== Platform ===
Last reset 01:35:17 ago, cause: power up
Last software reset at 2019-06-23 22:49, reason: User, spinning module GCodes, available RAM 11380 bytes (slot 3)
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0441f000 BFAR 0xe000ed38 SP 0xffffffff Task 0x4e49414d
Error status: 0
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest block write time: 13.4ms, max retries 0
MCU temperature: min 27.2, current 47.0, max 47.5
Supply voltage: min 23.7, current 24.1, max 24.4, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: standstill, SG min/max 0/348
Driver 1: standstill, SG min/max 0/321
Driver 2: standstill, SG min/max 0/643
Driver 3: standstill, SG min/max not available
Driver 4: standstill, SG min/max not available
Date/time: 2019-06-26 20:48:58
Cache data hit count 4294967295
Slowest loop: 546.79ms; fastest: 0.07ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Move ===
Hiccups: 0, FreeDm: 169, MinFreeDm: 106, MaxWait: 580296ms
Bed compensation in use: none, comp offset 0.000
=== DDARing ===
Scheduled moves: 0, completed moves: 27, StepErrors: 0, LaErrors: 0, Underruns: 0, 2
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1
Heater 0 is on, I-accum = 0.0
Heater 1 is on, I-accum = 0.3
=== GCodes ===
Segments left: 0
Stack records: 1 allocated, 1 in use
Movement lock held by null
http is idle in state(s) 0 0
telnet is idle in state(s) 0
file is idle in state(s) 0
serial is idle in state(s) 0
aux is idle in state(s) 0
daemon is idle in state(s) 0
queue is idle in state(s) 0
autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 56.64ms; fastest: 0.00ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 1 of 8

WiFi -
Network state is running
WiFi module is connected to access point
Failed messages: pending 0, notready 0, noresp 0
WiFi firmware version 1.23
WiFi MAC address b4:e6:2d:53:16:0a
WiFi Vcc 3.43, reset reason Turned on by main processor
WiFi flash size 4194304, free heap 25704
WiFi IP address 192.168.1.145
WiFi signal strength -59dBm, reconnections 0, sleep mode modem
Socket states: 0 0 0 0 0 0 0 0

Vlad

@dc42 for the record, my first duet board which failed ran at exact same setup and I never had such problem on that one.

What is strange, is that I checked retries on this report and you can see it is zero there.

SD card longest block write time: 13.4ms, max retries 0

dc42

Yes, that is odd.

As I said before, I'll support your warranty claim if you want to exchange the Duet.

Vlad

@dc42 Ok, how do I proceed with it, what is the process? I purchased this at matterhackers. I will be at their store next week.

dc42

@vlad, I suggest you email them and direct them to this thread.

Matterhackers, please exchange Vlad's board.

Vlad

@dc42 ok, will do, thanks!

Vlad

Hi David @dc42 While I am still trying to find a time to drive to matterhackers, I installed another Duet board on this machine that I had handy, and it gives me exact same issue. I am starting to believe that there is no reason to exchange the board if it's gonna have exact same issue.

Is it possible that I am exceeding a cap for number of commands or reads per second for this MCU?

Also is it possible to have a workaround in firmware where it reattempts the read again instead of freezing forever? Possibly increase command buffer maybe? Occasional stall wouldn't be so disastrous if firmware could resume printing at least. I usually print high resolution models and at relatively high speeds, but nothing crazy. It froze now at 220mm/s at the curve/fillet location. Never freezes at straight line, so I guess it has to do with some processing cap being reached.

As a side note, I already reached a cap of print speed with 256 microstepping at 200mm/s. So I have to drop microstepping to 128, and 256 is not really achievable in this case, even on my simple Cartesian machine. Hope you guys soon come up with faster chip

P.S: I can also provide a factory file from S3D if you want to try to replicate the issue on your machine.

Vlad

@dc42 another thought: has anyone actually printed fast with this board? maybe I am overloading the MCU with 150mm/s? But to be honest my average speeds that are in 150mm/s area, are not really any fast... Reducing STL resolution seems to do nothing. I am out of thoughts. Also I closed web server window in last 4 prints just to see, and all 4 completed fine. Regardless I think this is serious and valid firmware bug that it simply cancels everything instead of continuing the print after reattempting to read card. MKS SBASE was lagging in similar way, it could freeze for up to 20 seconds, but it will eventually continue and print wasn't wasted.

dc42

Firmware 2.0 and later already retries reading from the SD card several times. See lines 192 onwards at https://github.com/dc42/RepRapFirmware/blob/dev/src/Libraries/Fatfs/diskio.cpp. The constant MaxSdCardTries is set to 3 and SdCardRetryDelay is set to 30.

dc42

@vlad said in Cannot read file, error code 1 (5 times in 2 days):

As a side note, I already reached a cap of print speed with 256 microstepping at 200mm/s. So I have to drop microstepping to 128, and 256 is not really achievable in this case, even on my simple Cartesian machine.

That is expected. The Trinamic chips can interpolate x16 microstepping to x256, and that is the setting we recommend for most purposes.

Vlad

@dc42 As soon as printer freezes, I get error message about SD card and everything is cancelled that exact moment, and it is never retried.

I have the latest firmware (just installed this week). And error message says it is cancelled and that happens immediately. No way it is retrying anything. Another thought, is that my web interface is very jerky when it comes to uploading files (cursor is acting weird), so I tried to close web interface in my browser while printing, and so far 5 prints went without problem with web interface closed. Do you think web client can cause this possibly?
I tried to play with enabling interpolation, but it seems to do nothing. Maybe I am wrong, but I expected interpolation to have the same effect on stepper noise. For example 16x is noisy, 256x is silent. So I was expecting 16x with interpolation enabled to become more silent vs 16x without interpolation, which is not the case. I even tested on super noisy 2x setting and enabling interpolation does nothing to noise, which I thought it should.
Another problem that I reported about a year ago still there too - that is incorrect values for current/accel/jerk on a first printer move after power on. Even if I set for example 0.2A current, and restart the machine, it will be very obviously getting some "default" setting instead, until the first command, after that command everything is read front he config file normally. Any progress on this bug at all? This also persists on all boards, my friends confirmed the same behavior on their Duets.

Vlad

@dc42

Doesn't look like it was retrying anything at all David. Will this bug be looked at at all? My board is unusable pretty much now and exchanging it is not going to do anything, as I mentioned before. This doesn't look like the issue only on this board.

Tried enabling logging, does nothing, not catching any useful information at all... Not even sure what it is trying to write during printing...

Vlad

@dc42 also regarding MaxSdCardTries, I am not a coder and not very familiar with cpp, but are you sure this variable is defined somewhere? I wasn't able to find the definition in that file. Also, when it retries, is there a command to resume the print? Same for SdCardRetryDelay, I see you reading those, but where from? Maybe it is not set in my firmware?

deckingman

@vlad said in Cannot read file, error code 1 (5 times in 2 days):

@dc42 another thought: has anyone actually printed fast with this board?

Yup - printing at up to 300mm/sec with non-print moves set to 350mm/sec - see here https://somei3deas.wordpress.com/2018/10/14/real-3d-printing-at-high-speeds-and-even-higher-melt-rates-with-a-large-nozzle/

....and the video that goes with that blog post here https://www.youtube.com/watch?v=rUV5IZxfAxU&feature=youtu.be

Vlad

@deckingman I am trying now to change memory block size on my SD card. It is all either super mess or super confusing. In Github code, which David gave, it has this code:

#define SECTOR_SIZE_DEFAULT 512

/** Supported sector size. These values are based on the LUN function:

mem_sector_size(). */
#define SECTOR_SIZE_512 1
#define SECTOR_SIZE_1024 2
#define SECTOR_SIZE_2048 4
#define SECTOR_SIZE_4096 8

However @dc42 always suggests FAT32 for FS option for duet cards. The only problem here is that FAT32 can not be formatted with 512 block size, and my particular card was formatted with 32kb block size. Right now I am checking whether it was too much to handle for duet controller and I changed block size to minimum now that FAT32 allows: 2048 bits. Gonna test and see if it freezes again tomorrow. But something tells me this actually could be the reason. Will see.

I wanted to especially thank @vlad for keeping looking into a problem and trying to find a solution for this flaw. because board makers are apparently clueless and/or not interested in finding a solution for a very real problem with their product.

droftarts

It's probably a problem of using large capacity SD cards, and the manufacturer formatting them to make sure the full capacity of the card is available, with the limitation that FAT32 can only address 268,435,445 clusters (2pow28-1 as FAT32 only uses 28 bits not 32, with the other 4 bits "reserved for future use"). See the 'Default cluster sizes for FAT32' of this page: https://support.microsoft.com/en-gb/help/140365/default-cluster-size-for-ntfs-fat-and-exfat
Currently RRF would seem to support up to 8GB cards (with 4k cluster size), so larger cards with 8k+ cluster size are not reading correctly. Maybe @dc42 can add the larger cluster sizes to the firmware? Unless that has an impact on memory usage?

Ian

dc42

@vlad said in Cannot read file, error code 1 (5 times in 2 days):

As soon as printer freezes, I get error message about SD card and everything is cancelled that exact moment, and it is never retried.

That's because it has already done the retries and they all failed.

dc42

@vlad said in Cannot read file, error code 1 (5 times in 2 days):

@deckingman I am trying now to change memory block size on my SD card. It is all either super mess or super confusing. In Github code, which David gave, it has this code:

#define SECTOR_SIZE_DEFAULT 512

/** Supported sector size. These values are based on the LUN function:

mem_sector_size(). */
#define SECTOR_SIZE_512 1
#define SECTOR_SIZE_1024 2
#define SECTOR_SIZE_2048 4
#define SECTOR_SIZE_4096 8

However @dc42 always suggests FAT32 for FS option for duet cards. The only problem here is that FAT32 can not be formatted with 512 block size, and my particular card was formatted with 32kb block size. Right now I am checking whether it was too much to handle for duet controller and I changed block size to minimum now that FAT32 allows: 2048 bits. Gonna test and see if it freezes again tomorrow. But something tells me this actually could be the reason. Will see.

I wanted to especially thank @vlad for keeping looking into a problem and trying to find a solution for this flaw. because board makers are apparently clueless and/or not interested in finding a solution for a very real problem with their product.

The sector size must be 512. You can use either FAT16 or FAT32 format. You can't use EXFAT because it is patented and attracts a royalty payment, so RRF doesn't support it. FAT16 is only available on low capacity SD cards.

Larger cluster sizes give better file upload speeds, but waste more space on the SD card.

Vlad

@dc42 I am not worried about wasting space on modern cards. This is why all my cards were formatted in 32 for speed and performance. This, however, may be taking much more on board memory to read and therefore could possibly be causing my problem with freezing. And i ran tests on SD card while printer is stuck, SD card is fully accessible from web client and from paneldue also. Super weird. Dropping microstepping further down to 64 now too (it was 128 before). This was tested on two duet controllers so far with the same problem persisting on both. I have about 30 more people in my group with duets, I am gonna send my factory file to them to see if issue is easy to replicate on their machines.

@dc42 said in Cannot read file, error code 1 (5 times in 2 days):

That's because it has already done the retries and they all failed.

How long is the pause between retries? Is it in milliseconds? Because it does look as instant cancellation to me when it prints. And there seems to be no way to track down if retries actually took place.

A Former User

@vlad said in Cannot read file, error code 1 (5 times in 2 days):

How long is the pause between retries? Is it in milliseconds? Because it does look as instant cancellation to me when it prints. And there seems to be no way to track down if retries actually took place.

MaxSdCardTries is defined in Configuration.h and is set to 30 milliseconds.

If the retries show 0 in M122 the after the error occurs, then disk_read may be aborting before it gets to the retry code, i.e. fails the valid address check: https://github.com/dc42/RepRapFirmware/blob/dev/src/Libraries/Fatfs/diskio.cpp#L189 . You could add in a debug line and to see if that's actually occurring on your setup.

If you print the same file again (without resetting the duet) after it has cancelled does it also fail again the next time? and same spot? Does the same file work again after resetting the board? Or do you need to re-upload the file again?