HTML5音频流：精确测量延迟？

Question

I'm building a cross-platform web app where audio is generated on-the-fly on the server and live streamed to a browser client, probably via the HTML5 audio element. 我正在构建一个跨平台的Web应用程序，其中音频在服务器上即时生成，并且可能通过HTML5音频元素直播到浏览器客户端。 On the browser, I'll have Javascript-driven animations that must precisely sync with the played audio. 在浏览器上，我将使用Javascript驱动的动画，必须与播放的音频精确同步。 "Precise" means that the audio and animation must be within a second of each other, and hopefully within 250ms (think lip-syncing). “精确”意味着音频和动画必须在彼此之间，并且希望在250ms内（想想唇形同步）。 For various reasons, I can't do the audio and animation on the server and live-stream the resulting video. 由于各种原因，我无法在服务器上进行音频和动画，并对生成的视频进行实时流式传输。

Ideally, there would be little or no latency between the audio generation on the server and the audio playback on the browser, but my understanding is that latency will be difficult to control and probably in the 3-7 second range (browser-, environment-, network- and phase-of-the-moon-dependent). 理想情况下，服务器上的音频生成和浏览器上的音频播放之间几乎没有或没有延迟，但我的理解是延迟将难以控制，并且可能在3-7秒范围内（浏览器，环境 - ，网络和月相依赖）。 I can handle that, though, if I can precisely measure the actual latency on-the-fly so that my browser Javascript knows when to present the proper animated frame. 但是，我可以处理这个问题，如果我可以在运行中精确测量实际延迟，以便我的浏览器Javascript知道何时呈现正确的动画帧。

So, I need to precisely measure the latency between my handing audio to the streaming server (Icecast?), and the audio coming out of the speakers on the computer hosting the speaker. 那么，我需要精确测量我将音频传输到流媒体服务器（Icecast？）和来自扬声器主机上扬声器的音频之间的延迟。 Some blue-sky possibilities: 一些蓝天的可能性：

Add metadata to the audio stream, and parse it from the playing audio (I understand this isn't possible using the standard audio element) 将元数据添加到音频流，并从播放音频中解析它（我知道使用标准音频元素是不可能的）
Add brief periods of pure silence to the audio, and then detect them on the browser (can audio elements yield the actual audio samples?) 为音频添加短暂的纯静音时段，然后在浏览器上检测它们（音频元素可以产生实际的音频样本吗？）
Query the server and the browser as to the various buffer depths 查询服务器和浏览器的各种缓冲区深度
Decode the streamed audio in Javascript and then grab the metadata 在Javascript中解码流式音频，然后获取元数据

Any thoughts as to how I could do this? 有关如何做到这一点的任何想法？

Answer 1

Utilize timeupdate event of <audio> element, which is fired three to four times per second, to perform precise animations during streaming of media by checking .currentTime of <audio> element. 利用<audio>元素的timeupdate事件，每秒触发三到四次，通过检查<audio>元素的.currentTime ，在媒体流传输过程中执行精确的动画。 Where animations or transitions can be started or stopped up to several times per second. 动画或过渡可以每秒开始或停止多次。

If available at browser, you can use fetch() to request audio resource, at .then() return response.body.getReader() which returns a ReadableStream of the resource; 如果在浏览器中可用，则可以使用fetch()来请求音频资源，在.then()返回response.body.getReader() ，它返回资源的ReadableStream ; create a new MediaSource object, set <audio> or new Audio() .src to objectURL of the MediaSource ; 创建一个新的MediaSource对象，将<audio>或new Audio() .src objectURL为MediaSource objectURL ; append first stream chunks at .read() chained .then() to sourceBuffer of MediaSource with .mode set to "sequence" ; 追加第一流块在.read()链.then()到sourceBuffer的MediaSource与.mode设置为"sequence" ; append remainder of chunks to sourceBuffer at sourceBuffer updateend events. 在sourceBuffer updateend事件sourceBuffer剩余的块附加到sourceBuffer 。

If fetch() response.body.getReader() is not available at browser, you can still use timeupdate or progress event of <audio> element to check .currentTime , start or stop animations or transitions at required second of streaming media playback. 如果fetch() response.body.getReader()在浏览器中不可用，您仍然可以使用timeupdate或<audio>元素的progress事件来检查.currentTime ，在所需的第二个流媒体播放时启动或停止动画或转换。

Use canplay event of <audio> element to play media when stream has accumulated adequate buffers at MediaSource to proceed with playback. 当流在MediaSource上累积了足够的缓冲区以继续播放时，使用canplay事件的<audio>元素播放媒体。

You can use an object with properties set to numbers corresponding to .currentTime of <audio> where animation should occur, and values set to css property of element which should be animated to perform precise animations. 您可以使用一个对象，其属性设置为与<audio> .currentTime相对应的.currentTime ，其中应该出现动画，并将值设置为元素的css属性，该元素应该被动画化以执行精确的动画。

At javascript below, animations occur at every twenty second period, beginning at 0 , and at every sixty seconds until the media playback has concluded. 在下面的javascript中，动画每20秒发生一次，从0开始，每60秒一次，直到媒体播放结束。

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">    
<head>
  <meta charset="utf-8" />
  <title></title>
  <style>
    body {
      width: 90vw;
      height: 90vh;
      background: #000;
      transition: background 1s;
    }

    span {
      font-family: Georgia;
      font-size: 36px;
      opacity: 0;
    }
  </style>
</head>

<body>
  <audio controls></audio>
  <br>
  <span></span>
  <script type="text/javascript">
    window.onload = function() {
      var url = "/path/to/audio";
      // given 240 seconds total duration of audio 
      // 240/12 = 20
      // properties correspond to `<audio>` `.currentTime`,
      // values correspond to color to set at element
      var colors = {
        0: "red",
        20: "blue",
        40: "green",
        60: "yellow",
        80: "orange",
        100: "purple",
        120: "violet",
        140: "brown",
        160: "tan",
        180: "gold",
        200: "sienna",
        220: "skyblue"
      };
      var body = document.querySelector("body");
      var mediaSource = new MediaSource;
      var audio = document.querySelector("audio");
      var span = document.querySelector("span");
      var color = window.getComputedStyle(body)
                  .getPropertyValue("background-color");
      //console.log(mediaSource.readyState); // closed
      var mimecodec = "audio/mpeg";

      audio.oncanplay = function() {
        this.play();
      }

      audio.ontimeupdate = function() {         
        // 240/12 = 20
        var curr = Math.round(this.currentTime);

        if (colors.hasOwnProperty(curr)) {
          // set `color` to `colors[curr]`
          color = colors[curr]
        }
        // animate `<span>` every 60 seconds
        if (curr % 60 === 0 && span.innerHTML === "") {
          var t = curr / 60;
          span.innerHTML = t + " minute" + (t === 1 ? "" : "s") 
                           + " of " + Math.round(this.duration) / 60 
                          + " minutes of audio";
          span.animate([{
              opacity: 0
            }, {
              opacity: 1
            }, {
              opacity: 0
            }], {
              duration: 2500,
              iterations: 1
            })
            .onfinish = function() {
              span.innerHTML = ""
            }
        }
        // change `background-color` of `body` every 20 seconds
        body.style.backgroundColor = color;
        console.log("current time:", curr
                   , "current background color:", color
                  , "duration:", this.duration);
      }
      // set `<audio>` `.src` to `mediaSource`
      audio.src = URL.createObjectURL(mediaSource);
      mediaSource.addEventListener("sourceopen", sourceOpen);

      function sourceOpen(event) {
        // if the media type is supported by `mediaSource`
        // fetch resource, begin stream read, 
        // append stream to `sourceBuffer`
        if (MediaSource.isTypeSupported(mimecodec)) {
          var sourceBuffer = mediaSource.addSourceBuffer(mimecodec);
          // set `sourceBuffer` `.mode` to `"sequence"`
          sourceBuffer.mode = "sequence";

          fetch(url)
          // return `ReadableStream` of `response`
          .then(response => response.body.getReader())
          .then(reader => {

            var processStream = (data) => {
              if (data.done) {
                  return;
              }
              // append chunk of stream to `sourceBuffer`
              sourceBuffer.appendBuffer(data.value);
            }
            // at `sourceBuffer` `updateend` call `reader.read()`,
            // to read next chunk of stream, append chunk to 
            // `sourceBuffer`
            sourceBuffer.addEventListener("updateend", function() {
              reader.read().then(processStream);
            });
            // start processing stream
            reader.read().then(processStream);
            // do stuff `reader` is closed, 
            // read of stream is complete
            return reader.closed.then(() => {
              // signal end of stream to `mediaSource`
              mediaSource.endOfStream();
              return  mediaSource.readyState;
            })
          })
          // do stuff when `reader.closed`, `mediaSource` stream ended
          .then(msg => console.log(msg))
        } 
        // if `mimecodec` is not supported by `MediaSource`  
        else {
          alert(mimecodec + " not supported");
        }
      };
    }
  </script>
</body>
</html>

plnkr http://plnkr.co/edit/fIm1Qp?p=preview plnkr http://plnkr.co/edit/fIm1Qp?p=preview

Answer 2

There no way for you to measure latency directly, but any AudioElement generate events like 'playing' if it just played (fired quite often), or 'stalled' if stoped streaming, or 'waiting' if data is loading. 您无法直接测量延迟，但任何AudioElement都会生成“播放”等事件，如果它刚播放（经常触发），或者如果停止流式传输则“停顿”，或者如果数据正在加载则“等待”。 So what you can do, is to manipulate your video based on this events. 所以你能做的就是根据这些事件操纵你的视频。

So play while stalled or waiting is fired, then continue playing video if playing fired again. 因此，在停顿或等待被解雇时播放，然后如果再次发射则继续播放视频。

But I advice you check other events that might affect your flow (error for example would be important for you). 但我建议您检查可能影响您的流量的其他事件（例如，错误对您很重要）。

https://developer.mozilla.org/en-US/docs/Web/API/HTMLAudioElement https://developer.mozilla.org/en-US/docs/Web/API/HTMLAudioElement

Answer 3

What i would try is first create a timestamp with performance.now , process the data, and record it in a blob with the new web recorder api. 我会尝试首先创建一个带有performance.now的时间戳，处理数据，并使用新的web记录器api将其记录在blob中。

The web recorder will ask user access to his audio card, this can be a problem for your app, but it look like mandatory to get the real latency. 网络录音机将要求用户访问他的声卡，这可能是您的应用程序的问题，但它似乎是必须获得真正的延迟。

As soon this done, there is many way to measure the actual latency between the generation and the actual rendering. 一旦完成，有很多方法可以测量生成和实际渲染之间的实际延迟。 Basically, a sound event. 基本上是一个声音事件。

For further reference and example: 有关进一步参考和示例：

Recorder demo 录音机演示

https://github.com/mdn/web-dictaphone/ https://github.com/mdn/web-dictaphone/

https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder_API/Using_the_MediaRecorder_API https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder_API/Using_the_MediaRecorder_API

HTML5音频流：精确测量延迟？

问题描述

3 个解决方案

解决方案1
7 2016-08-09 05:19:14

解决方案2
1 2016-08-08 14:31:44

解决方案3
0 2016-08-13 06:05:11

HTML5音频流：精确测量延迟？

问题描述

3 个解决方案

解决方案1 7 2016-08-09 05:19:14

解决方案2 1 2016-08-08 14:31:44

解决方案3 0 2016-08-13 06:05:11

解决方案1
7 2016-08-09 05:19:14

解决方案2
1 2016-08-08 14:31:44

解决方案3
0 2016-08-13 06:05:11