Java Sound transmission over UDP - Sound errors when implementing input level checking method

Question

i'm writing a program for simple voice transmission via udp. This works fine, until I implement a method to check the average volume level of a sample-to-be-sent.

Here I have included the Audio class and the Receiver class and cut out some unimportant stuff.

public class Audio extends Thread
{
    private AudioFormat defaultAudioFormat = null;
    private TargetDataLine inputLine = null;
    private byte[] sample = new byte[1024];
    private SourceDataLine speakers = null;
    private int voiceActivationLevel = 35;
    private SourceDataLine speakers = null;
    private boolean running = true;
    private VOIPSocket sender = null;
    
    public Audio(int voiceActivationLevel, VOIPSocket socket) throws LineUnavailableException
    {
        this.voiceActivationLevel = voiceActivationLevel;
        this.defaultAudioFormat = new AudioFormat(8000f, 16, 1, true, true);
        
        DataLine.Info info = new DataLine.Info(TargetDataLine.class, this.defaultAudioFormat);
        this.inputLine = (TargetDataLine) AudioSystem.getLine(info);
        this.inputLine.open(defaultAudioFormat);
        this.sender = socket;
        
        DataLine.Info dataLineInfo = new DataLine.Info(SourceDataLine.class, this.defaultAudioFormat);
        this.speakers = (SourceDataLine) AudioSystem.getLine(dataLineInfo);
        this.speakers.open();
        this.speakers.start();
    }
    
    public void run()
    {
        DataLine.Info = new DataLine.Info(TargetDataLine.class, this.defaultAudioFormat);
        this.inputLine.start();
        while(running)
        {
            if(AudioSystem.isLineSupported(info))
            {
                int data = inputLine.read(this.sample, 0, sample.length);
                int voiceLevel = calculateVolumeLevel(this.sample);
                if(voiceLevel >= this.voiceActivationLevel)
                    this.sender.sendData(this.sample); //
            }
        }
    }
    
    public int calculateVolumeLevel(byte[] audioData)
    {
        long l = 0;
        for(int i = 0; i < audioData.length; i++)
            l = l + audioData [i];
        double avg = l / audioData.length;
        double sum = 0d;
        
        for(int j = 0; j < audioData.length; j++)
            sum += Math.pow(audioData[j] - avg, 2d);
        
        double averageMeanSquare = sum / audioData.length;
        return (int)(Math.pow(averageMeanSquare, 0.5d) + 0.5);
    }
    
    public void playSound(byte[] data)
    {
        synchronized(this.speakers)
            this.speakers.write(data, 0, data.length);
    }
}

Note that calculateVolumeLevel does NOT modify voiceData, just returns an average volume level as integer.

public class Receiver extends Thread
{
    private VOIPSocket socket = null; //Just a simple class with some helper functions for the socket, not important for this
    private byte[] buffer = new byte[4096];
    private boolean isRunning = true;
    private Audio audio = null;
    
    public Receiver(VOIPSocket socket, Audio audio) throws SocketException
    {
        this.socket = socket;
        this.audio = audio;
    }
    
    public void run()
    {
        DatagramPacket packet = new DatagramPacket(this.buffer, this.buffer.length);
        while(isRunning)
        {
            if(!socket.getSocket.isClosed())
            {
                socket.getSocket.receive(packet);
                byte data = packet.getData();
                this.audio.playSound(data);
            }
        }
    }
}

As soon as I include the check for the volume level, the sound is stuttering, repeating over and over and other mistakes, until I flush or drain the speakers-dataline.

The data transmission via UDP is working correctly and needs no further investigation in my opinion.

My read is, that as soon as I implement check for the voice volume, the byte-data somehow is corrupted or important parts of sample[] are not transmitted. This somehow puts errors on the speakers-dataline.

I don't know how to solve this. Any ideas?

edit:

According to https://docs.oracle.com/javase/10/troubleshoot/java-sound.htm#JSTGD490 , some over- or underrun condition comes in I guess. If I dont have the volume check enabled, a continuous data stream is provided for the speakers-dataline. If it is enabled, this stream of course is interrupted most of the time, leading to corrupted data in either the input dataline or the speakers-dataline (or both). This can be solved by flushing and closing the datalines, then opening them again. This unfortunately is not a suitable solution, as flushing can sometimes take up to 1 second, where no data can be played, which is not acceptable, as I would need to flush very often (like every time, there is silence).

Any ideas on this?

Answer 1

I would consider buffering the volume-checking. For example, I wrote a line that takes input from a live microphone's TargetDataLine , converts to PCM and passes the data on to an input track of an audio mixing tool. The key to getting this to work in my situation was providing an intermediate, concurrent-safe FIFO queue. I used a ConcurrentLinkedQueue<Float[]> with the float[] arrays holding signed, normalized float "packets" of PCM.

A more experienced coder than myself used to continuously harp on the following general principle in working with audio: never block. I haven't read through the code you provided thoroughly enough to know if this is arising in your case, but seeing the keyword synchronized in the playSound method reminds me of his advice.

The only blocking when working with audio, I think, should be that which is built into the blocking queues employed by TargetDataLine or SourceDataLine read and write methods. I blindly pass on this advice and encourage you to inspect your code for blocking and find alternatives in any situation in your code where a block might occur.