You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When uploading a larger file (~6GB) I noticed the upload was becoming quite slow due to how the chunks were being processed. It seems that currently a full chunk is uploaded and stored in the server's memory, then written to disk once the whole chunk has been received and only then does the upload of the next chunk begin. This means the client is just waiting for the server to finish until it begins uploading the next chunk. If instead we uploaded multiple at once say for example 3 at a time then the server could be writing chunk 1 to disk while it has chunk 2 queued up ready in memory to be written and chunk 3 is being uploaded from the client, this way there's much less time when the client is just not uploading chunks/waiting for the current chunk to be processed/written to disk. A chunk configuration option to choose the number of chunks to upload at once would be nice and it could warn users that you will need chunk size * number of chunks memory free on the server.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
When uploading a larger file (~6GB) I noticed the upload was becoming quite slow due to how the chunks were being processed. It seems that currently a full chunk is uploaded and stored in the server's memory, then written to disk once the whole chunk has been received and only then does the upload of the next chunk begin. This means the client is just waiting for the server to finish until it begins uploading the next chunk. If instead we uploaded multiple at once say for example 3 at a time then the server could be writing chunk 1 to disk while it has chunk 2 queued up ready in memory to be written and chunk 3 is being uploaded from the client, this way there's much less time when the client is just not uploading chunks/waiting for the current chunk to be processed/written to disk. A chunk configuration option to choose the number of chunks to upload at once would be nice and it could warn users that you will need chunk size * number of chunks memory free on the server.
Beta Was this translation helpful? Give feedback.
All reactions