Download very large files (8GB+) - combining WCF and WebClient?

Question

I have quite some trouble creating a WCF service that supports downloads of very large files. I have read a lot of guides on how to set transferMode attribute to Streamed, increase all messageSize and bufferSize attributes to Int32.MaxValue, and still I have no luck. (I am also returning the stream as the message body via the MessageBodyMember attribute, and metadata is sent via the headers using the MessageHeader attributes).

If I set all these attributes, I can download smaller files fine, but when I try to download 1-2GB files I simply get a 400 bad request error which makes it rather hard to debug...

My service should ideally support file sizes of at least 8GB. Is this even doable with WCF? The various messageSize attributes of the web.config file seem to be limited to Int32.MaxValue which equals a maximum file size of 2GB.

From my studies I have found that it seems I will have to use WebClient.DownloadFile instead. Files should only be available for download to users who have the required rights. With WCF my download method could take a token-parameter that the server could check and return the stream only if the user had rights to download the requested file. This does not seem straight forward using the WebClient approach. If anyone has some guidelines on how to do this (via the WebClient), I would very much appreciate it.

Ideally my WCF service should administer and provide user tokens and somehow bind to every individual file what tokens are currently legal (tokens should be usable only once). Download should then happen via the WebClient.

Thanks in advance for any clues.

Answer 1

You can do this in WCF. Many moons ago I built a service that did this (we didn't have a web server as part of our configuration). We used WCF streaming:

http://msdn.microsoft.com/en-us/library/ms733742.aspx

The strategy to deal with large payloads is streaming. While messages, especially those expressed in XML, are commonly thought of as being relatively compact data packages, a message might be multiple gigabytes in size and resemble a continuous data stream more than a data package. When data is transferred in streaming mode instead of buffered mode, the sender makes the contents of the message body available to the recipient in the form of a stream and the message infrastructure continuously forwards the data from sender to receiver as it becomes available.