C# HttpWebRequest - 发送巨大的 stream 到服务器

Question

UPD：事实证明（从这里的代码https://source.dot.net/#System.Net.Requests/System/Net/HttpWebRequest.cs,1135 ） HttpWebRequest不支持流式传输，它在 memory 和仅在GetResponse上发送。

因此，将研究其他选择。

原始问题

我有非常巨大的 stream，长度未知（如“来自传感器的数据”），需要通过 HTTPS 发送到服务器。

我正在尝试使用HttpWebRequest并使用它提供的 stream 发送数据。

考虑到下面的代码（“模拟”要发送的大数据），我预计HttpWebRequest早在r.GetResponse()之前就开始发送数据。

stream 数据，或者如果https://some.url/endpoint不可用，或者证书错误，或者无法发送数据，则抛出异常。

但是，我得到的是“流太长”的例外。 服务器永远不会被调用。

我做错了什么？ 我应该以某种方式明确告诉HttpWebRequest开始通信吗？

编码：

static void Main(string[] args)
        {
            byte[] buff = new byte[64 * 1024];
            try
            {
                HttpWebRequest r = WebRequest.CreateHttp("https://some.url/endpoint");
                r.Method = "POST";
                r.AllowWriteStreamBuffering = false; // please do not do bufferring
                r.ContentType = "application/octet-stream";
                r.SendChunked = true;
                using Stream stream = r.GetRequestStream();
                for (int i = 0; i < 1000000000; i++) // really, really large stream
                {
                    stream.Write(buff, 0, buff.Length);
                }
                var resp = r.GetResponse();
                // ...
            } catch (Exception e)
            {
                Console.WriteLine($"Exception: {e.Message}"); // Stream was too long.
            }

        }

正如@Jimi 在评论中所要求的，完整的代码片段， .Net Core 3.1 ：

using System;
using System.IO;
using System.Net;

namespace HttpWebRequestConsoleTest
{
    class ConsoleTest
    {

        static void Main(string[] args)
        {
            byte[] buff = new byte[64 * 1024];
            try
            {
                HttpWebRequest r = WebRequest.CreateHttp("https://some.invalid.url/endpoint");
                r.Method = "POST";
                r.AllowWriteStreamBuffering = false; // please do not bufferring
                r.ContentType = "application/octet-stream";
                r.SendChunked = true;
                using Stream stream = r.GetRequestStream();
                for (int i = 0; i < 1000000000; i++) // really, really large stream
                {
                    stream.Write(buff, 0, buff.Length);
                    //Thread.Sleep(10);
                }
                var resp = r.GetResponse();
                // ...
            } catch (Exception e)
            {
                Console.WriteLine($"Exception: {e.Message}"); // Stream was too long.
            }

        }
    }
}

Answer 1

这是我过去如何做的一个例子。

当您要发送长度未知的数据时，您必须将其分块发送。 没有其他办法。

为了测试这一点，我设置了一个新的 ASP.NET Core 3.1 项目并添加了这个 controller：

using Microsoft.AspNetCore.Mvc;
using System.Threading.Tasks;
using WebApplication1.Utilities;

namespace WebApplication1.Controllers
{
    [ApiController, Produces("application/json"), Route("[controller]")]
    public sealed class StreamingController : ControllerBase
    {
        [HttpPost, Route("consumeStream"), DisableRequestSizeLimit]
        public async Task<IActionResult> ConsumeStreamAsync()
        {
            using (var ns = new NullStream())
            {
                await Request.Body.CopyToAsync(ns);
                return Ok(new { ns.Length });
            }
        }
    }
}

在上面的例子中， NullStream是一个 object，它继承自System.IO.Stream并简单地丢弃写入它的长度数据，但会跟踪它。

这是客户端代码：

using System;
using System.IO;
using System.Net;
using System.Threading.Tasks;

namespace ConsoleApp1
{
    class Program
    {
        static async Task Main()
        {
            var req = (HttpWebRequest)WebRequest.Create("https://localhost:44366/Streaming/consumeStream");
            req.Method = "POST";
            req.SendChunked = true;
            req.AllowWriteStreamBuffering = false;
            req.AllowReadStreamBuffering = false;
            req.ContentType = "application/octet-stream";

            var buf = new byte[64 * 1024];
            using(var s = await req.GetRequestStreamAsync())
            {
                for(long i = 0; i < 100_000_000; ++i)
                {
                    await s.WriteAsync(buf, 0, buf.Length);

                    if ((i & 10000) == 0)
                    {
                        Console.WriteLine($"Wrote {i * buf.Length:n0} bytes...");
                    }
                }
            }

            using (var resp = await req.GetResponseAsync())
            using (var s = resp.GetResponseStream())
            using (var sr = new StreamReader(s))
            {
                Console.WriteLine(await sr.ReadToEndAsync());
            }

            Console.WriteLine("Finished");
            Console.ReadKey(true);
        }
    }
}

为了完整起见，这里是NullStream ：

using System;
using System.IO;

namespace WebApplication1.Utilities
{
    public sealed class NullStream : Stream
    {
        private long _length;
        public override bool CanRead => false;
        public override bool CanSeek => false;
        public override bool CanWrite => true;
        public override long Length => _length;
        public override long Position
        { 
            get => throw new NotImplementedException();
            set => throw new NotImplementedException();
        }
        public override void Flush() { }
        public override int Read(byte[] buffer, int offset, int count) =>
            throw new NotImplementedException();
        public override long Seek(long offset, SeekOrigin origin) =>
            throw new NotImplementedException();
        public override void SetLength(long value) =>
            _length = value;
        public override void Write(byte[] buffer, int offset, int count)
        {
            _length += count;
            System.Diagnostics.Debug.WriteLine($"Write: {count:n0}; Length: {_length:n0}");
        }
    }
}

output 示例：

...
Write: 65,536; Length: 4,514,222,022
Write: 57,344; Length: 4,514,279,366
Write: 65,536; Length: 4,514,344,902
Write: 65,536; Length: 4,514,410,438
Write: 65,536; Length: 4,514,475,974
Write: 57,344; Length: 4,514,533,318
Write: 61,440; Length: 4,514,594,758
Write: 65,536; Length: 4,514,660,294
Write: 65,536; Length: 4,514,725,830
Write: 65,536; Length: 4,514,791,366
Write: 65,536; Length: 4,514,856,902
...