有关穿线/背景工人的问题

Question

I have a question around threading and background workers that I hope you can help with.

I plan on making an ftp application to upload a file to 50 servers. Rather than the user having to wait for each upload to finish before the next one starts I was looking at threading/background workers. Once an upload finishes I want to report the status of the upload "completed/failed" back to the UI. From my understanding, I will need to use background workers for this so I know when the task has completed. I know with threading I can use producer/consumer queue or a semaphore to run a given amount of threads at once but I am not quite sure how I can achieve this with back ground workers.

So my question is, what would be a sensible number of background workers controlling uploading to run at once and what would be the best way to queue the rest?

There is no limit on the size of the upload file so this could be quite small or up to a few MB.

Thanks in advance.

Edit - I tested out one backgroundworker for each server running simultaneousness. The results where faster than just a single backgroundworker but I can't say that i was fully comfortable with running 50 plus background workers at once and since the server count may increase in the future, I decided to stick with just the one, which seems to be fast enough. I may in future look at increasing the count of workers to 2 or 3 but currently 1 seems to be adequate. Thanks for everyones help.

Thanks

Answer 1

I'd go in a completely different direction with it, tbh. Your app should take the file and store it once, responding to the client that it's got it. The file should then be propagated to the other servers. You can do this many ways, but if you want it controlled by the same application (ie not done using a windows service or the like) then a good way would be to use a message queue (either MSMQ or one of the OS ones).

Answer 2

The bottleneck here is going to be your network bandwidth. If your local upstream connection is so fast that you can saturate the incoming connections on two or more remote hosts, then you'll benefit from running multiple uploads in parallel. If not, then it makes very little difference to the total upload time, since it'll be dictated by (file size * number of uploads) / (local bandwidth). In other words - if you do 20 uploads one at a time, it'll take an hour; if you do 20 uploads in parallel, it'll still take an hour. The advantage of the first approach is that if you lose connectivity you'll only need to resume/restart a single upload - whichever one was in progress when the connection was lost.

I'd therefore use a single background thread to sequentially upload the file to each server in turn. If you're using the .NET BackgroundWorker to do this, you can get it to ReportProgress at the end of each file (and you know in advance how many files are to be uploaded so you can calculate progress as a percentage), and attach some custom state to the progress update to inform the user whether the last upload succeeded or not.

Answer 3

The only way to know for sure is to test and measure, but it can be different from machine to machine, mostly depending on uplink speed.

Starting 50 backgroundworkers at the same time is a bit on the high end, but is not incredibly many. A simple approach would be to start 50 all at the same time and measure memory consumption and upload speed.

If the FTP servers are each much faster than the client uplink speed the most efficient would be to just upload one (or possibly two) at a time.

Answer 4

This is much easier than using a semaphore or producer-consumer queue.

Put all your tasks in a queue (doesn't need to be a thread-safe queue, it will only be used from the UI thread).

Loop from 1 to N, taking out a task and starting a BackgroundWorker . (Be sure to handle the empty queue, when there were less than N tasks to begin with). In the RunWorkerCompleted event, update your UI, dequeue another task, and start another BackgroundWorker .