Syncronized recording from a microphone array using the JavaSound API

Question

I've gone through the tutorials for the Java Sound API and I've successfully read off data from my microphone.

I would now like to go a step further and get data synchronously from multiple microphones in a microphone array (like a PS3 Eye or Respeaker)

I could get a TargetDataLine for each microphone and open/start/write the input to buffers - but I don't know how to do this in a way that will give me data that I can then line up time-wise (I would like to eventually do beamforming)

When reading from something like ALSA I would get the bytes from the different microphone simultaneously, so I know that each byte from each microphone is from the same time instant - but the Java Sound API seems to have an abstration that obfuscates this b/c you are just dumping/writing data out of separate line buffers and processing it and each line is acting separately. You don't interact with the whole device/mic-array at once

However I've found someone who managed to do beamforming in Java with the Kinect 1.0 so I know it should be possible. The problem is that the secret sauce is inside a custom Mixer object inside a .jar that was pulled out of some other software.. So I don't have any easy way to figure out how they pulled it off

Answer 1

You will only be able to align data from multiple sources with the time synchronous accuracy to perform beam-forming if this is supported by the underlying hardware drivers.

If the underlying hardware provides you with multiple, synchronised, data-streams (eg recording in 2 channels - in stereo), then your array data will be time synchronised.

If you are relying on the OS to simply provide you with two independent streams, then maybe you can rely on timestamping. Do you get the timestamp of the first element? If so, then you can re-align data by dropping samples based on your sample rate. There may be a final difference (delta-t) that you will have factor in to your beam-forming algorithm.

Reading about the PS3 Eye (which has an array of microphones), you will be able to do this if the audio driver provides all the channels at once.

For Java, this probably means "Can you open the channel with an AudioFormat that includes 4 channels"? If yes, then your samples will contain multiple frames and the decoded frame data will (almost certainly) be time aligned. To quote the Java docs : "A frame contains the data for all channels at a particular time".

Answer 2

IDK what "beamforming" is, but if there is hardware that can provide synchronization, using that would obviously be the best solution.

Here, for what it is worth, is what should be a plausible algorithmic way to manage synchronization.

(1) Set up a frame counter for each TargetDataLine . You will have to convert bytes to PCM as part of this process.

(2) Set up some code to monitor the volume level on each line, some sort of RMS algorithm I would assume, on the PCM data.

(3) Create a loud, instantaneous burst that reaches each microphone at the same time, one that the RMS algorithm is able to detect and to give the frame count for the onset.

(4) Adjust the frame counters as needed, and reference them going forward on each line of incoming data.

Rationale: Java doesn't offer real-time guarantees, as explained in this article on real-time, low latency audio processing . But in my experience, the correspondence between the byte data and time (per the sample rate) is very accurate on lines closest to where Java interfaces with external audio services.

How long would frame counting remain accurate without drifting? I have never done any tests to research this. But on a practical level, I have coded a fully satisfactory "audio event" scheduler based on frame-counting, for playing multipart scores via real-time synthesis (all done with Java), and the timing is impeccable for the longest compositions attempted (6-7 minutes in length).