Completely replace SD Card? (duet ethernet)

dc42

Thanks David, I think you missed my last edit. I don't suppose you could share some more details on how exactly the cards are formatted to ensure best performance. When I tried to manually format for largest cluster size I got worse performance than when I used the SD Formatter.

We don't format the cards specially, we take them as they come.

I am very surprised that using a larger cluster size made the performance worse, and I suggest you re-test this in case a variation in WiFi performance affected the speed.

It makes sense to me that cards optimised for random access would perform better than cards optimised for digital cameras.

The next RC firmware will include the cluster size in the M39 response.

Phaedrux

I re-ran the tests a little more methodically. Here are the results and how I got them.

I used the SD associations SD Formatter tool to format the cards for defaults. It uses 32k cluster size for cards 32gb and under.
To get 64k cluster sizes I used the OSX terminal to run sudo newfs_msdos -F 32 -c 128 -v DUETWIFI disk1
It is interesting to note that the stock SD card that comes with the Duet is actually formatted with 32k cluster sizes. I was actually unable to format it using that command, as it gave me an error. 61416 clusters too few clusters for FAT32, need 65525. Perhaps 4gb is too small to use 64k clusters?
Cluster sizes were verified on a windows system using chkdsk.

To test the transfer speeds I used an 18Mb gcode file uploaded from my laptop over wifi to the duet. The router is about 20 feet away. I probably could have gone wired for the laptop to take that out of the equation, but I wouldn't be doing that on a regular basis anyway, so this test was more realistic. 18Mb is a very large gcode file for me. I originally was going to test with a 200Mb file, but it was taking too long. I uploaded the file 3 times in a row and timed it using my smartphone timer and averaging the times.

Here are the results.
[c]
Transcend 4gb Class 4 FAT32 with 32kb clusters 1:00 (stock)
Transcend 4gb Class 4 FAT32 with 64kb clusters Could not format
Sandisk Ultra 32gb Class 10 A1 FAT32 with 32kb clusters 1:10
Sandisk Ultra 32gb Class 10 A1 FAT32 with 64kb clusters 1:00
Kingston 8gb Class 10 FAT32 with 32kb clusters 0:55
Kingston 8gb Class 10 FAT32 with 64kb clusters 0:56
[/c]

As you can see, not very exciting, and probably more limited by wifi than the SD card. Very little difference between cluster sizes, except in the case of the larger 32Gb card. Most surprisingly for me, the 8gb class 10 card performed terribly in synthetic benchmarks, but once formatted properly (I think it was a bit messed up at first) it turned in the best real world results.

I should also note that cards over 32Gb will default to being formatted in exFAT, and the SD Formatter tool will format it as such. You'd have to use the command line or another tool to format those any other way.

Also, if you are going to use a very large card, look for one with the A1 classification for improved random write performance. That may become more important over time as the large card gets filled up with more files and things get fragmented.

dc42

For 4Gb and under cards, use FAT16 format. FAT16 provides an additional performance boost over FAT32.

Phaedrux

Good to know. I'll try retesting with the 4GB class 4 card that came with the Duet with FAT16 and 64k clusters. Is that how they are supposed to come formatted? Mine was FAT32 with 32k clusters.

T3P3Tony

We have been using FAT32 for the 4GB cards. I was not actually aware of the FAT16 performance increase that David mentioned.

Something to change on the next batch.

Phaedrux

One final data point then.

[c]
Transcend 4gb Class 4 FAT16 with 64kb clusters 00:50
[/c]

10 seconds off the FAT32 format.

So there you go. For best speeds the stock card formatted at FAT16 with 64k clusters is the way to go. This might require you to reformat the card yourself though, as out of the box mine was FAT32 and 32k clusters.

If you need a larger card, FAT32 with 64kb clusters and A1 classification.

I may try again with an ideal wireless situation with the router directly line of site to the printer and my laptop on a wired connection, just to isolate the SD card performance. In actual practice there isn't much difference.

EasyTarget

Sorry to be pedantic, would using fAT16 provide a significant upload speed boost?

If I am driving prints remotely (http xfer gcode to Duet, start print remotely, from a script on a server that it will automatically remove completed print files too); Would it provide a useful/noteworthy speed boost to the upload If I reformatted my card to FAT16.

(I'm suggesting not using the whole drive.. eg just force a FAT16 filesystem onto the first four gigs and ignore the rest.. I've done something similar to this before.)

Edit: Looking at the above.. I guess not.

T3P3Tony

@Phaedrux:

One final data point then.

[c]
Transcend 4gb Class 4 FAT16 with 64kb clusters 00:50
[/c]

10 seconds off the FAT32 format.

So there you go. For best speeds the stock card formatted at FAT16 with 64k clusters is the way to go. This might require you to reformat the card yourself though, as out of the box mine was FAT32 and 32k clusters.

If you need a larger card, FAT32 with 64kb clusters and A1 classification.

I may try again with an ideal wireless situation with the router directly line of site to the printer and my laptop on a wired connection, just to isolate the SD card performance. In actual practice there isn't much difference.

Thanks for running the tests!

Phaedrux

@EasyTarget:

Sorry to be pedantic, would using fAT16 provide a significant upload speed boost?

If I am driving prints remotely (http xfer gcode to Duet, start print remotely, from a script on a server that it will automatically remove completed print files too); Would it provide a useful/noteworthy speed boost to the upload If I reformatted my card to FAT16.

(I'm suggesting not using the whole drive.. eg just force a FAT16 filesystem onto the first four gigs and ignore the rest.. I've done something similar to this before.)

Edit: Looking at the above.. I guess not.

Short stroking a drive like that used to be popular with spinning platters back when they topped out at 120ish gigs, but for an SD card it might actually degrade performance further due to the wear leveling algorithm having less space to work with. If there is some performance to be gained from FAT16 it would probably be lost from that.

As it is, I think unless you have a really poor SD card, you're probably more limited by wireless transfer than write speed.

dc42

The wear levelling algorithm works at a level underneath the FAT format, so it shouldn't be affected.

dc42

I have tried to reproduce this issue, but so far without success. I copied a 384Mbyte file over wifi to the Duet. The speed started at over 1Mbyte/sec but had dropped to 657kbytes/sec by the time it finished. DWC shows file information for it. Then I tacked half a megabyte of useless comments on the end so that DWC would have to search a long way back from the end of the file without finding what it wanted, and uploaded that. Again, no problems but file info is not displayed because the Duet does not search back as far as half a megabyte from the end. When I refresh the file list, it pauses on that file for 4 seconds, but carries on after that.

So I conclude that you can use files up to at least 385Mbytes long if your choice of SD card and format is suitable. This card is a Sandisk 16Gb card with a class 4 speed rating, formatted to FAT32 with 64Kb clusters. In DWC I have the status update interval set to 250ms which I think is the default, and the maximum number of AJAX retries set to 1 which is wrong - I recommend at least 3 when using WiFi.

EasyTarget

Thanks for the feedback, and the testing! super helpful

I'm certainly not going this route for a modest gain, I guess I was optimistically hoping for a dramatic (2x or whatever) speedup

Once this weekend's re-organising and wiring is complete I should have my Pi acting as the AP for my print cupboard, I just, errm, 'acquired' a 10m CAT5 cable for this purpose from the nest at work. This will put the AP ~10cm from the Duet in line-of-sight; so I'm hoping to push the speed if I can, and see if this solves the disconnects that plague me from webcontrol. (yes, I know that may be too close; I'll play with separating them if I see weirdness before complaining here)

dc42

If you are getting disconnects, check that you have the retry count set to at least 3 in DWC.

EasyTarget

20 enough?I've seen all the advice, but since I have a signal strength that flaps between -50 and -70 I'll wait till that is stable before passing judgement.. at the moment I'm happy to continue blaming my end of the network.

dc42

With the signal strength changing that much, I suspect you are getting interference from another device in the same band. Have you tried changing the WiFi channel?

SpoonUnit

Just found something odd. I have near identical printers setup, one which has a Duet Wifi, and one which is a Duet Ethernet. Using DWC to upload a 30MB file shows the performance is severely different between the two, but not in a way that I would expect. The Wifi connected Duet can be uploaded to significantly quicker. The Duet Wifi seems to top out at around 580k/sec, whereas the Ethernet only gets to about 340k/s. Could this be a feature of the firmware on the Ethernet version? Does anyone else have similar results?

The SD card in both is a brand new 16GB SanDisk Ultra SDHC Class 10.

T3P3Tony

@spoonunit are both SD cards formatted with 64 KB blocks? That can make a difference.

dc42

You can run M39 to check the cluster size.

The other thing that can make a difference is fragmentation of the SD card. A heavily-fragmented SD card tends to slow down the speed of writing to it.

In my tests, both WiFi (with a good signal strength)and Ethernet top out at about 1Mbps if the SD card has a large cluster size and is not heavily fragmented. For very large files it drops to around 800kbytes/sec.

SpoonUnit

Will double check. Both were straight out the packet, so I assume they're identically formatted. Assumptions can be wrong though.

dc42

It may be worth checking that the Ethernet physical link has connected at 100Mbps, not 10Mbps.