Possible FTP Upload Problem

fcwilt

Hi,

As the capabilities of the Duet firmware have grown, my set of configuration files have increased in number and complexity.

I am finding the DWC Editor is no longer efficient.

So I have turned to keeping and editing the files on my computer and uploading them to the Duet via FTP as needed.

Most of the time this works just fine.

However on occasion I am seeing (using firmware 3.4.1) that the file on the Duet ends up truncated.

I have no idea where the problem may lie

User error?
FTP client?
Duet FTP server?

I have tried different FTP clients and that has not made a difference.

Any ideas? Thanks much.

Frederick

Phaedrux

I think this is an old issue. https://forum.duet3d.com/topic/5992/files-uploaded-over-ftp-occasionally-blank?_=1654876890534

fcwilt

@phaedrux said in Possible FTP Upload Problem:

I think this is an old issue. https://forum.duet3d.com/topic/5992/files-uploaded-over-ftp-occasionally-blank?_=1654876890534

Well that suggests what I am seeing is a Duet related problem - not something at the the "computer" end of things.

Could you pass my issue along to the appropriate folks?

Thanks much.

Frederick

OwenD

@fcwilt
I found lots of similar issues (empty files) when using FileZilla as the client.
I think the main problem is that the duet will only accept a single connection/transfer and most modern ftp clients try to establish multiple concurrent connections.
I think I had to also limit the transfer rate.

fcwilt

@owend said in Possible FTP Upload Problem:

@fcwilt
I found lots of similar issues (empty files) when using FileZilla as the client.
I think the main problem is that the duet will only accept a single connection/transfer and most modern ftp clients try to establish multiple concurrent connections.
I think I had to also limit the transfer rate.

I thought it might be something like that. The FTP client I am hoping to use is part of my favorite editor.

I don't see any settings for the FTP client part of the editor that mention connections or transfer speed.

I'm seeing the problem with three different FTP clients so I think the problem lies in the Duet code.

Thanks.

Frederick

chrishamm

@fcwilt The FTP server in RepRapFirmware is very limited. You must limit the number of concurrent transfers to 1, else certain problems will show up (although I wouldn't have thought empty files would be a symptom). WinSCP has that setting:

And so does FileZilla:

fcwilt

@chrishamm

I installed FileZilla Pro and made the settings you mentioned regards number of transfers.

Doing a upload from FileZilla:

I still got empty files on the Duet.
Once there was an empty file it could not be overwritten but had to be deleted before the non-empty file could be uploaded.

I did several tests.

Hopefully there is a bug that can be found and fixed in the Duet?

Thanks.

Frederick

Phaedrux

@chrishamm It happens even with the FTP client set to a single file at a time.

It's a bit insidious, since the file will be created, but it is 0 bytes. So unless you're checking the file size you can miss that there is a problem.

chrishamm

@fcwilt What Duet board are you using?

fcwilt

@chrishamm said in Possible FTP Upload Problem:

@fcwilt What Duet board are you using?

I am using a variety: Duet 2 WiFi/Duex 5 combo, Duet 3 Mini 5+ and Duet 3 6HC/3HC combo.

The Duet 2s are running 3.4.0.

The Duet 3 Mini 5+ and the Duet 3 6HC/3HC are running 3.4.1

At the risk of repeating myself, the things I am seeing:

files of size 0
files of size 0 that could not be updated via FileZilla - they had to be deleted. FileZilla reported no errors.
files of truncated size

To date I have not found any pattern.

Frederick

Phaedrux

Uploading these seems to be a good way to provoke it.

https://forum.duet3d.com/topic/6181/tuning-macros-menus-accel-jerk-retraction-pressure-advance

Sometimes they look like they are full size files, but on refreshing the directory they are 0 byte.

fcwilt

@chrishamm

I contacted the developer of the editor I am using that has the FTP client built-in.

He said it always uses just a single connection and always transfer just one file at a time.

Frederick

wschadow

I use filezilla in many cases But together with teh Duet boards I found that WinSCP is a bit more reliable. I have not looked into details.

chrishamm

@fcwilt I could reproduce those problems on a Duet 3 Mini 5+ WiFi and on a MB6HC. For both board types I have firmware fixes ready but I still need to check if the Duet 2 Ethernet is affected, too.

fcwilt

@chrishamm said in Possible FTP Upload Problem:

@fcwilt I could reproduce those problems on a Duet 3 Mini 5+ WiFi and on a MB6HC. For both board types I have firmware fixes ready but I still need to check if the Duet 2 Ethernet is affected, too.

Would the Duet 2 WiFi also be fixed by the changes for a Duet 2 Ethernet?

Frederick

chrishamm

@fcwilt Yes.

paunboli2

@chrishamm I'm also getting problem with old file explorer

chrishamm

@fcwilt @Phaedrux @wschadow You can find my updated v3.4.1+1 firmware builds for standalone mode here: https://pkg.duet3d.com/RRF-3.4.1plus1.zip Please let me know if they fix the FTP transfer problems. The issue where uploading config.g resulted in an empty file may be fixed by these builds, too.

fcwilt

@chrishamm said in Possible FTP Upload Problem:

@fcwilt @Phaedrux @wschadow You can find my updated v3.4.1+1 firmware builds for standalone mode here: https://pkg.duet3d.com/RRF-3.4.1plus1.zip Please let me know if they fix the FTP transfer problems. The issue where uploading config.g resulted in an empty file may be fixed by these builds, too.

I'm out of town for a few days.

I will test this new firmware as soon as I return home.

Thanks much.

Frederick

OwenD

@chrishamm
I just tried the following using FileZilla set to single connection and transfer limited to 500kb/s
On past usage all of these tests gave multiple zero kb files both directions

Bulk upload of all my filaments (25 directories each with 3 files) - OK
Bulk download of same - OK
Bulk download of complete system (700 files in 25 directories) - Only one failed transfer.
File was 0/www/css/OnScreenKeyboar
It was not 0kb, it failed to transfer. Re-queuing worked.
FileZilla didn't give a clear reason why the transfer of that file failed, but as it was reported I would call the test successful.