Cannot read file, error code 1 (5 times in 2 days)

dc42

@vlad, I suggest you email them and direct them to this thread.

Matterhackers, please exchange Vlad's board.

Vlad

@dc42 ok, will do, thanks!

Vlad

Hi David @dc42 While I am still trying to find a time to drive to matterhackers, I installed another Duet board on this machine that I had handy, and it gives me exact same issue. I am starting to believe that there is no reason to exchange the board if it's gonna have exact same issue.

Is it possible that I am exceeding a cap for number of commands or reads per second for this MCU?

Also is it possible to have a workaround in firmware where it reattempts the read again instead of freezing forever? Possibly increase command buffer maybe? Occasional stall wouldn't be so disastrous if firmware could resume printing at least. I usually print high resolution models and at relatively high speeds, but nothing crazy. It froze now at 220mm/s at the curve/fillet location. Never freezes at straight line, so I guess it has to do with some processing cap being reached.

As a side note, I already reached a cap of print speed with 256 microstepping at 200mm/s. So I have to drop microstepping to 128, and 256 is not really achievable in this case, even on my simple Cartesian machine. Hope you guys soon come up with faster chip

P.S: I can also provide a factory file from S3D if you want to try to replicate the issue on your machine.

Vlad

@dc42 another thought: has anyone actually printed fast with this board? maybe I am overloading the MCU with 150mm/s? But to be honest my average speeds that are in 150mm/s area, are not really any fast... Reducing STL resolution seems to do nothing. I am out of thoughts. Also I closed web server window in last 4 prints just to see, and all 4 completed fine. Regardless I think this is serious and valid firmware bug that it simply cancels everything instead of continuing the print after reattempting to read card. MKS SBASE was lagging in similar way, it could freeze for up to 20 seconds, but it will eventually continue and print wasn't wasted.

dc42

Firmware 2.0 and later already retries reading from the SD card several times. See lines 192 onwards at https://github.com/dc42/RepRapFirmware/blob/dev/src/Libraries/Fatfs/diskio.cpp. The constant MaxSdCardTries is set to 3 and SdCardRetryDelay is set to 30.

dc42

@vlad said in Cannot read file, error code 1 (5 times in 2 days):

As a side note, I already reached a cap of print speed with 256 microstepping at 200mm/s. So I have to drop microstepping to 128, and 256 is not really achievable in this case, even on my simple Cartesian machine.

That is expected. The Trinamic chips can interpolate x16 microstepping to x256, and that is the setting we recommend for most purposes.

Vlad

@dc42 As soon as printer freezes, I get error message about SD card and everything is cancelled that exact moment, and it is never retried.

I have the latest firmware (just installed this week). And error message says it is cancelled and that happens immediately. No way it is retrying anything. Another thought, is that my web interface is very jerky when it comes to uploading files (cursor is acting weird), so I tried to close web interface in my browser while printing, and so far 5 prints went without problem with web interface closed. Do you think web client can cause this possibly?
I tried to play with enabling interpolation, but it seems to do nothing. Maybe I am wrong, but I expected interpolation to have the same effect on stepper noise. For example 16x is noisy, 256x is silent. So I was expecting 16x with interpolation enabled to become more silent vs 16x without interpolation, which is not the case. I even tested on super noisy 2x setting and enabling interpolation does nothing to noise, which I thought it should.
Another problem that I reported about a year ago still there too - that is incorrect values for current/accel/jerk on a first printer move after power on. Even if I set for example 0.2A current, and restart the machine, it will be very obviously getting some "default" setting instead, until the first command, after that command everything is read front he config file normally. Any progress on this bug at all? This also persists on all boards, my friends confirmed the same behavior on their Duets.

Vlad

@dc42

Doesn't look like it was retrying anything at all David. Will this bug be looked at at all? My board is unusable pretty much now and exchanging it is not going to do anything, as I mentioned before. This doesn't look like the issue only on this board.

Tried enabling logging, does nothing, not catching any useful information at all... Not even sure what it is trying to write during printing...

Vlad

@dc42 also regarding MaxSdCardTries, I am not a coder and not very familiar with cpp, but are you sure this variable is defined somewhere? I wasn't able to find the definition in that file. Also, when it retries, is there a command to resume the print? Same for SdCardRetryDelay, I see you reading those, but where from? Maybe it is not set in my firmware?

deckingman

@vlad said in Cannot read file, error code 1 (5 times in 2 days):

@dc42 another thought: has anyone actually printed fast with this board?

Yup - printing at up to 300mm/sec with non-print moves set to 350mm/sec - see here https://somei3deas.wordpress.com/2018/10/14/real-3d-printing-at-high-speeds-and-even-higher-melt-rates-with-a-large-nozzle/

....and the video that goes with that blog post here https://www.youtube.com/watch?v=rUV5IZxfAxU&feature=youtu.be

Vlad

@deckingman I am trying now to change memory block size on my SD card. It is all either super mess or super confusing. In Github code, which David gave, it has this code:

#define SECTOR_SIZE_DEFAULT 512

/** Supported sector size. These values are based on the LUN function:

mem_sector_size(). */
#define SECTOR_SIZE_512 1
#define SECTOR_SIZE_1024 2
#define SECTOR_SIZE_2048 4
#define SECTOR_SIZE_4096 8

However @dc42 always suggests FAT32 for FS option for duet cards. The only problem here is that FAT32 can not be formatted with 512 block size, and my particular card was formatted with 32kb block size. Right now I am checking whether it was too much to handle for duet controller and I changed block size to minimum now that FAT32 allows: 2048 bits. Gonna test and see if it freezes again tomorrow. But something tells me this actually could be the reason. Will see.

I wanted to especially thank @vlad for keeping looking into a problem and trying to find a solution for this flaw. because board makers are apparently clueless and/or not interested in finding a solution for a very real problem with their product.

droftarts

It's probably a problem of using large capacity SD cards, and the manufacturer formatting them to make sure the full capacity of the card is available, with the limitation that FAT32 can only address 268,435,445 clusters (2pow28-1 as FAT32 only uses 28 bits not 32, with the other 4 bits "reserved for future use"). See the 'Default cluster sizes for FAT32' of this page: https://support.microsoft.com/en-gb/help/140365/default-cluster-size-for-ntfs-fat-and-exfat
Currently RRF would seem to support up to 8GB cards (with 4k cluster size), so larger cards with 8k+ cluster size are not reading correctly. Maybe @dc42 can add the larger cluster sizes to the firmware? Unless that has an impact on memory usage?

Ian

dc42

@vlad said in Cannot read file, error code 1 (5 times in 2 days):

As soon as printer freezes, I get error message about SD card and everything is cancelled that exact moment, and it is never retried.

That's because it has already done the retries and they all failed.

dc42

@vlad said in Cannot read file, error code 1 (5 times in 2 days):

@deckingman I am trying now to change memory block size on my SD card. It is all either super mess or super confusing. In Github code, which David gave, it has this code:

#define SECTOR_SIZE_DEFAULT 512

/** Supported sector size. These values are based on the LUN function:

mem_sector_size(). */
#define SECTOR_SIZE_512 1
#define SECTOR_SIZE_1024 2
#define SECTOR_SIZE_2048 4
#define SECTOR_SIZE_4096 8

However @dc42 always suggests FAT32 for FS option for duet cards. The only problem here is that FAT32 can not be formatted with 512 block size, and my particular card was formatted with 32kb block size. Right now I am checking whether it was too much to handle for duet controller and I changed block size to minimum now that FAT32 allows: 2048 bits. Gonna test and see if it freezes again tomorrow. But something tells me this actually could be the reason. Will see.

I wanted to especially thank @vlad for keeping looking into a problem and trying to find a solution for this flaw. because board makers are apparently clueless and/or not interested in finding a solution for a very real problem with their product.

The sector size must be 512. You can use either FAT16 or FAT32 format. You can't use EXFAT because it is patented and attracts a royalty payment, so RRF doesn't support it. FAT16 is only available on low capacity SD cards.

Larger cluster sizes give better file upload speeds, but waste more space on the SD card.

Vlad

@dc42 I am not worried about wasting space on modern cards. This is why all my cards were formatted in 32 for speed and performance. This, however, may be taking much more on board memory to read and therefore could possibly be causing my problem with freezing. And i ran tests on SD card while printer is stuck, SD card is fully accessible from web client and from paneldue also. Super weird. Dropping microstepping further down to 64 now too (it was 128 before). This was tested on two duet controllers so far with the same problem persisting on both. I have about 30 more people in my group with duets, I am gonna send my factory file to them to see if issue is easy to replicate on their machines.

@dc42 said in Cannot read file, error code 1 (5 times in 2 days):

That's because it has already done the retries and they all failed.

How long is the pause between retries? Is it in milliseconds? Because it does look as instant cancellation to me when it prints. And there seems to be no way to track down if retries actually took place.

A Former User

@vlad said in Cannot read file, error code 1 (5 times in 2 days):

How long is the pause between retries? Is it in milliseconds? Because it does look as instant cancellation to me when it prints. And there seems to be no way to track down if retries actually took place.

MaxSdCardTries is defined in Configuration.h and is set to 30 milliseconds.

If the retries show 0 in M122 the after the error occurs, then disk_read may be aborting before it gets to the retry code, i.e. fails the valid address check: https://github.com/dc42/RepRapFirmware/blob/dev/src/Libraries/Fatfs/diskio.cpp#L189 . You could add in a debug line and to see if that's actually occurring on your setup.

If you print the same file again (without resetting the duet) after it has cancelled does it also fail again the next time? and same spot? Does the same file work again after resetting the board? Or do you need to re-upload the file again?

Vlad

@sdavi Hi I am not a coder nor did this board say I will need to be a coder to make it not throw errors everywhere and that I have to redo the whole firmware myself. So, no, I have no idea how to add debug lines. Nor do I have time to mess with this myself on $200 product to be honest. It does seem to fail before it even retries, and you seem to have looked much more into my issue than @dc42 did actually. Answering your second message, it fails absolutely randomly and it is usually always in the section of small tubes etc, where it needs to process a lot of commands per second. As said earlier, I reformatted the SD card into 2048 sector size and will try to see if that could help. I am running out of options pretty much to be honest.

dc42

@vlad said in Cannot read file, error code 1 (5 times in 2 days):

@dc42 I am not worried about wasting space on modern cards. This is why all my cards were formatted in 32 for speed and performance. This, however, may be taking much more on board memory to read and therefore could possibly be causing my problem with freezing.

The cluster size does not affect the amount of memory needed for reading or writing. Only the sector size affects that, but the sector size is fixed at 512 bytes.

And i ran tests on SD card while printer is stuck, SD card is fully accessible from web client and from paneldue also. Super weird. Dropping microstepping further down to 64 now too (it was 128 before). This was tested on two duet controllers so far with the same problem persisting on both.

It appears that the problem is transient - the Duet is unable to read the SD card for a short while, then it starts working again. One possibility is that it has decided to move some data around to do some wear levelling. I think that should normally occur only if you are writing to the SD card as well as reading from it. That might happen if you have enabled logging and you are getting a lot of stalls logged, or something like that.

@dc42 said in Cannot read file, error code 1 (5 times in 2 days):

That's because it has already done the retries and they all failed.

How long is the pause between retries? Is it in milliseconds? Because it does look as instant cancellation to me when it prints. And there seems to be no way to track down if retries actually took place.

In file Configuration.h:

constexpr unsigned int MaxSdCardTries = 3;				// Number of read or write attempts before giving up
constexpr uint32_t SdCardRetryDelay = 30;				// Number of milliseconds delay between SD transfer retries. Looks like 10ms may be too low.

Retries that resulted in success are logged and reported in M122. Retries that resulted in failure produce the error message that you have seen.

I can try increasing the maximum number of retries, and making the delay between retries increase with the number of retries.

Vlad

@dc42 nope, logging is disabled already. I tried it but it didn't give any useful info. If I can edit that file myself, I will try that as well. I would increase the delay significantly, to 500 or so and do 5 retries. I will give it a try if I can. M122 last time reported absolutely nothing, like it wasn't even retried once.

Ok just checked, I can't. I guess it needs to be compiled into .bin then, I have no way to do that.