将 numpy 数组发送到 Bokeh 回调以作为音频播放

Question

I'm currently trying to write a script to display spectrograms of (multichannel) audio in Bokeh. Since I am doing some processing on the audio, I can't easily save them as files on the computer, so I'm trying to remain in Python.

The idea is to create a plot where each column corresponds to an audio sample, and each row corresponds to a channel.

Now I want to be able to listen to the corresponding audio when clicking on a subplot. I've managed to do the non-interactive part of displaying the spectrograms, written a callback to play audio, and applied it to each callback.

Here is a minimal working example of the code:

import numpy as np
from bokeh.plotting import figure, output_file, show
from bokeh.models import ColumnDataSource, CustomJS
from bokeh.palettes import Viridis256
from bokeh.layouts import gridplot


def bokeh_subplots(specs, wavs):
    channels = max([s.shape[0] for s in specs])

    def inner(p, s, w):
        # p is the plot, s is the spectrogram, and w is the numpy array representing the sound
        source = ColumnDataSource(data=dict(raw=w))
        callback = CustomJS(args=dict(source=source),
                            code =
                            """
                            function typedArrayToURL(typedArray, mimeType) {
                                return URL.createObjectURL(new Blob([typedArray.buffer], {type: mimeType}))
                            }

                            const bytes = new Float32Array(source.data['raw'].length);

                            for(let i = 0; i < source.data['raw'].length; i++) {
                                bytes[i] = source.data['raw'][i];
                            }

                            const url = typedArrayToURL(bytes, 'audio/wave');

                            var audio = new Audio(url);
                            audio.src = url;
                            audio.load();
                            audio.play();
                            """ % w)
        # we plot the actual spectrogram here, which works fine
        p.image([s], x=0, y=0, dw=s.shape[1], dh=s.shape[0], palette = Viridis256)
        # then we add the callback to be triggered on a click within the plot
        p.js_on_event('tap', callback)
        return p
    
    # children will contain the plots in the form of a list of list
    children = []
    for s, w in zip(specs, wavs):
        # initialise the plots for each channel of a spectrogram, missing channels will be left blank
        glyphs = [None] * channels
        for i in range(s.shape[0]):
            # apply the function above to create the plots
            glyphs[i] = inner(figure(x_range=(0, s[i].shape[1]), y_range=(0, s[i].shape[0])),
                              s[i], w[i])
        children.append(glyphs)

    # finally, create the grid of plots and display
    grid = gridplot(children=children, plot_width=250, plot_height=250)
    show(grid)

# Generate some random data for illustration
random_specs = [np.random.uniform(size=(4, 80, 800)), np.random.uniform(size=(2, 80, 750))]
random_wavs = [np.random.uniform(low=-1, high=1, size=(4, 96*800)), np.random.uniform(low=-1, high=1, size=(2, 96*750))]

# This creates a plot with 2 rows and 4 columns
bokeh_subplots(specs=random_specs, wavs=random_wavs)

I basically copied this page to write the callback, but unfortunately it does not appear to be correct for my use case, as when I run the script, the plots generate correctly but the audio does not play. I have also attempted to create a data URI after encoding the array to base64 like here and here , with the same results. When trying the same with a simpler callback providing the path to a local file it works fine

callback = CustomJS(code = """var audio = new Audio("path/to/my/file.wav");
                              audio.play();
                           """)

This works but is not flexible enough for my purpose (since I either need to save a separate file for each channel, or have to forsake selecting the channel entirely).

I am extremely new in both JavaScript and Bokeh, so I am at a bit of loss as to what is wrong here. Fromthe above page I assume it has to do with the way I provide the array to the callback, but I have no idea how to fix it. (For that matter, I don't know if populating the 'bytes' array elementwise is an efficient way to go about it, but for now I'd settle on having the script work.)

Does anyone have any pointers regarding what's going on here?

Answer 1

So I ended up going another route with the callback after checking some more stuff in JavaScript, namely here , which ended up working with minimal alterations. The power of searching...

It's not necessarily the most efficient way of doing it, but it works, which is good enough for me right now.

I'm posting the full function here in case someone ever comes across it. The code should work as is, and I left some comments to explain what goes where.

import itertools
from bokeh.plotting import figure, show
from bokeh.models import ColumnDataSource, CustomJS
from bokeh.palettes import Viridis256
from bokeh.layouts import gridplot

def bokeh_subplots(specs,           # spectrograms to plot. List of numpy arrays of shape (channels, time, frequency). Heterogenous number of channels (e.g. one with 2, another with 4 channels) are handled by leaving blank spaces where required
                   wavs,            # sounds you want to play, there should be a 1-1 correspondence with specs. List of numpy arrays (tested with float32 values) of shape (channels, samples)
                   sr=48000,        # sampling rate in Hz
                   hideaxes=True,   # If True, the axes will be suppressed
                   ):
    # not really required, but helps with setting the number of rows of the final plot
    channels = max([s.shape[0] for s in specs])

    def inner(p, s, w):
        # this inner function is just for (slight) convenience
        source = ColumnDataSource(data=dict(raw=w))
        callback = CustomJS(args=dict(source=source),
                            code=
                            """
                            var audioCtx = new (window.AudioContext || window.webkitAudioContext)();
                            var myArrayBuffer = audioCtx.createBuffer(1, source.data['raw'].length, %d);

                            for (var channel = 0; channel < myArrayBuffer.numberOfChannels; channel++) {
                                  var nowBuffering = myArrayBuffer.getChannelData(channel);
                                  for (var i = 0; i < myArrayBuffer.length; i++) {
                                        nowBuffering[i] = source.data['raw'][i];
                                    }
                                }

                            var source = audioCtx.createBufferSource();
                            // set the buffer in the AudioBufferSourceNode
                            source.buffer = myArrayBuffer;
                            // connect the AudioBufferSourceNode to the
                            // destination so we can hear the sound
                            source.connect(audioCtx.destination);
                            // start the source playing
                            source.start();
                            """ % sr)
                            # Just need to specify the sampling rate here
        p.image([s], x=0, y=0, dw=s.shape[1], dh=s.shape[0], palette=Viridis256)
        p.js_on_event('tap', callback)
        return p

    children = []
    for s, w in zip(specs, wavs):
        glyphs = [None] * channels
        for i in range(s.shape[0]):
            glyphs[i] = figure(x_range=(0, s[i].shape[1]), y_range=(0, s[i].shape[0]))
            if hideaxes:
                glyphs[i].axis.visible = False
            glyphs[i] = inner(glyphs[i], s[i], w[i])
        children.append(glyphs)

    # we transpose the list so that each column corresponds to one (multichannel) spectrogram and each row corresponds to a channel of it
    children = list(map(list, itertools.zip_longest(*children, fillvalue=None)))
    grid = gridplot(children=children, plot_width=100, plot_height=100)
    show(grid)