WebRTC从画布卡纸发送流

Question

好吧，这个问题非常罕见。 我创建了Angular应用程序，以通过WebRTC在两个Peer之间进行通信。 该体系结构是简单的2个对等方，它们从摄像机发送视频流并从其他对等方接收流。 简单且有效。

现在，我想添加一些要处理的流，因此向第一个对等方添加如下所示的canvas元素：

    this.localCameraVideoStream = document.createElement('video');
    this.localCameraVideoStream.srcObject = stream;
    this.localCameraVideoStream.muted = true;
    this.localCameraVideoStream.play();

    this.canvas = document.createElement('canvas');
    this.canvas.width = 1280;
    this.canvas.height = 720;
    this.canvasStream = this.canvas.captureStream();

    this.localVideoElement.nativeElement.srcObject = this.canvasStream;
    this.localVideoElement.nativeElement.muted = true;
    this.localVideoElement.nativeElement.play();
    this.redrawStreamToCanvas();

再绘制方法将流从<video>元素绘制到画布：

 private redrawStreamToCanvas(){
    const ctx = this.canvas.getContext('2d');
    const width = 1280;
    const height = 720;
    const combinedVideoStream = this.localCameraVideoStream;

    function drawVideo() {
      ctx.clearRect(0, 0, width, height);
      ctx.drawImage(combinedVideoStream, 0, 0, width, height);
      requestAnimationFrame(drawVideo);
    }

    requestAnimationFrame(drawVideo);
  }

只是为了消除任何疑问。 localCameraVideoStream是创建的<video>元素，用于保存摄像机流，而<canvas>可以从<video>获取流。 canvas是<canvas> localVideoElement只是.html中的<video> ，并显示当前的“本地”流。

问题在于，在本地预览中，一切正常。 当我将canvasStream发送给其他对等方并将其显示在某些<video>我只会得到一帧而已。

你知道什么发生了吗？ 奇怪的是，在预览中还可以，没有这种组合，rtc连接也可以。

Answer 1

这很可能是由于Chrome的 [功能] 错误 ，一旦他们离开视口，它们就会暂停静音的 <video> 。

如果确实如此，那么这与MediaStream无关，并且已经在“ 绘制画布”操作中发生：

 var ctx = canvas.getContext('2d'); if ((videoin.buffered && !videoin.buffered.length) || videoin.paused) { videoin.onloadedmetadata = videoin.onplaying = begin; } else { begin(); } function begin() { videoin.onloadedmetadata = videoin.onplaying = null; canvas.width = videoin.videoWidth; canvas.height = videoin.videoHeight; drawToCanvas(); } function drawToCanvas() { ctx.drawImage(videoin, 0, 0); requestAnimationFrame(drawToCanvas); } 

 body { margin-bottom: 100vh; } 

 <p> Scroll down until the &lt;video&gt; element be out. </p> <video crossorigin id="videoin" src="https://upload.wikimedia.org/wikipedia/commons/transcoded/2/22/Volcano_Lava_Sample.webm/Volcano_Lava_Sample.webm.360p.webm" muted autoplay></video> <canvas id="canvas"></canvas> 

因此，如果您未设置muted属性，则可以使用：

 var ctx = canvas.getContext('2d'); if ((videoin.buffered && !videoin.buffered.length) || videoin.paused) { videoin.onloadedmetadata = videoin.onplaying = begin; } else { begin(); } function begin() { videoin.volume = 0; // does the same you'd say? videoin.onloadedmetadata = videoin.onplaying = null; canvas.width = videoin.videoWidth; canvas.height = videoin.videoHeight; drawToCanvas(); } function drawToCanvas() { ctx.drawImage(videoin, 0, 0); requestAnimationFrame(drawToCanvas); } 

 body { margin-bottom: 100vh; } 

 <p> Scroll down until the &lt;video&gt; element be out. </p> <video crossorigin id="videoin" src="https://upload.wikimedia.org/wikipedia/commons/transcoded/2/22/Volcano_Lava_Sample.webm/Volcano_Lava_Sample.webm.360p.webm" autoplay></video> <canvas id="canvas"></canvas> 

甚至，即使您没有将其附加在文档中，它也可以正常工作（在用户手势之后。）。

 var ctx = canvas.getContext('2d'); var videoin = document.createElement('video'); videoin.onloadedmetadata = videoin.onplaying = begin; videoin.muted = true; // even if 'muted' videoin.src = 'https://upload.wikimedia.org/wikipedia/commons/transcoded/2/22/Volcano_Lava_Sample.webm/Volcano_Lava_Sample.webm.360p.webm'; function begin() { videoin.onplay = null; canvas.width = videoin.videoWidth; canvas.height = videoin.videoHeight; drawToCanvas(); } function drawToCanvas() { ctx.drawImage(videoin, 0, 0); requestAnimationFrame(drawToCanvas); } play_btn.onclick = e => { videoin.play(); } 

 <button id="play_btn">click to start playing the video</button> <p> This contains only the canvas element </p> <canvas id="canvas"></canvas> 

现在，我不禁在这个答案中说，在调用它的captureStream方法之前，您必须已经初始化了画布上下文（甚至可能已经在其上绘制过）。 否则，将导致Firefox中出现NS异常，并且IIRC符合规范。