Live audio stream java

Question

I am implementing live streaming from MIC to java server at another PC. But I am only hearing a white noise.

I have attached both client and server program

Client:

import java.io.IOException;
import java.net.DatagramPacket;
import java.net.DatagramSocket;
import java.net.InetAddress;
import java.net.SocketException;
import java.net.UnknownHostException;

import javax.sound.sampled.AudioFormat;
import javax.sound.sampled.AudioInputStream;
import javax.sound.sampled.AudioSystem;
import javax.sound.sampled.DataLine;
import javax.sound.sampled.LineUnavailableException;
import javax.sound.sampled.TargetDataLine;

public class Mic 
{
    public byte[] buffer;
    private int port;
    static AudioInputStream ais;

    public static void main(String[] args)
    {
        TargetDataLine line;
        DatagramPacket dgp; 

        AudioFormat.Encoding encoding = AudioFormat.Encoding.PCM_SIGNED;
        float rate = 44100.0f;
        int channels = 2;
        int sampleSize = 16;
        boolean bigEndian = true;
        InetAddress addr;


        AudioFormat format = new AudioFormat(encoding, rate, sampleSize, channels, (sampleSize / 8) * channels, rate, bigEndian);

        DataLine.Info info = new DataLine.Info(TargetDataLine.class, format);
        if (!AudioSystem.isLineSupported(info)) {
            System.out.println("Line matching " + info + " not supported.");
            return;
        }

        try
        {
            line = (TargetDataLine) AudioSystem.getLine(info);

            int buffsize = line.getBufferSize()/5;
            buffsize += 512; 

            line.open(format);

            line.start();   

            int numBytesRead;
            byte[] data = new byte[buffsize];

            addr = InetAddress.getByName("127.0.0.1");
            DatagramSocket socket = new DatagramSocket();
            while (true) {
                   // Read the next chunk of data from the TargetDataLine.
                   numBytesRead =  line.read(data, 0, data.length);
                   // Save this chunk of data.
                   dgp = new DatagramPacket (data,data.length,addr,50005);

                   socket.send(dgp);
                }

        }catch (LineUnavailableException e) {
            e.printStackTrace();
        }catch (UnknownHostException e) {
            // TODO: handle exception
        } catch (SocketException e) {
            // TODO: handle exception
        } catch (IOException e2) {
            // TODO: handle exception
        }
    }
}

and the server side is no issue. It is running perfectly with android client AudioRecord.

Server:

import java.io.ByteArrayInputStream;
import java.net.DatagramPacket;
import java.net.DatagramSocket;

import javax.sound.sampled.AudioFormat;
import javax.sound.sampled.AudioInputStream;
import javax.sound.sampled.AudioSystem;
import javax.sound.sampled.DataLine;
import javax.sound.sampled.SourceDataLine;

public class Server {

    AudioInputStream audioInputStream;
    static AudioInputStream ais;
    static AudioFormat format;
    static boolean status = true;
    static int port = 50005;
    static int sampleRate = 44100;

    static DataLine.Info dataLineInfo;
    static SourceDataLine sourceDataLine;

    public static void main(String args[]) throws Exception 
    {
        System.out.println("Server started at port:"+port);

        DatagramSocket serverSocket = new DatagramSocket(port);

        /**
         * Formula for lag = (byte_size/sample_rate)*2
         * Byte size 9728 will produce ~ 0.45 seconds of lag. Voice slightly broken.
         * Byte size 1400 will produce ~ 0.06 seconds of lag. Voice extremely broken.
         * Byte size 4000 will produce ~ 0.18 seconds of lag. Voice slightly more broken then 9728.
         */

        byte[] receiveData = new byte[4096];

        format = new AudioFormat(sampleRate, 16, 1, true, false);
        dataLineInfo = new DataLine.Info(SourceDataLine.class, format);
        sourceDataLine = (SourceDataLine) AudioSystem.getLine(dataLineInfo);
        sourceDataLine.open(format);
        sourceDataLine.start();

        //FloatControl volumeControl = (FloatControl) sourceDataLine.getControl(FloatControl.Type.MASTER_GAIN);
        //volumeControl.setValue(1.00f);

        DatagramPacket receivePacket = new DatagramPacket(receiveData, receiveData.length);

        ByteArrayInputStream baiss = new ByteArrayInputStream(receivePacket.getData());

        while (status == true) 
        {
            serverSocket.receive(receivePacket);
            ais = new AudioInputStream(baiss, format, receivePacket.getLength());
            toSpeaker(receivePacket.getData());
        }

        sourceDataLine.drain();
        sourceDataLine.close();
    }

    public static void toSpeaker(byte soundbytes[]) {
        try 
        {
            System.out.println("At the speaker");
            sourceDataLine.write(soundbytes, 0, soundbytes.length);
        } catch (Exception e) {
            System.out.println("Not working in speakers...");
            e.printStackTrace();
        }
    }
}

Answer 1

So, I filled the microphone with a sine wave (or something which, in some vague sense, resembles a sine wave), and your program works fine.

My specific changes were thus:

package audioclient;

import java.io.*;
import java.net.*;
import java.nio.ByteBuffer;

import javax.sound.sampled.*;

public class Mic {
    public byte[] buffer;
    private int port;
    static AudioInputStream ais;

        public static void main(String[] args) {
        TargetDataLine line;
        DatagramPacket dgp;

        AudioFormat.Encoding encoding = AudioFormat.Encoding.PCM_SIGNED;
        float rate = 44100.0f;
        int channels = 2;
        int sampleSize = 16;
        boolean bigEndian = true;
        InetAddress addr;

        AudioFormat format = new AudioFormat(encoding, rate, sampleSize, channels, (sampleSize / 8) * channels, rate, bigEndian);

        DataLine.Info info = new DataLine.Info(TargetDataLine.class, format);
        if (!AudioSystem.isLineSupported(info)) {
            System.out.println("Line matching " + info + " not supported.");
            return;
        }

        try {
            line = (TargetDataLine) AudioSystem.getLine(info);

            //TOTALLY missed this.
            int buffsize = line.getBufferSize() / 5;
            buffsize += 512;

            line.open(format);

            line.start();

            int numBytesRead;
            byte[] data = new byte[buffsize];

            /*
             * MICK's injection: We have a buffsize of 512; it is best if the frequency
             * evenly fits into this (avoid skips, bumps, and pops). Additionally, 44100 Hz,
             * with two channels and two bytes per sample. That's four bytes; divide
             * 512 by it, you have 128.
             * 
             * 128 samples, 44100 per second; that's a minimum of 344 samples, or 172 Hz.
             * Well within hearing range; slight skip from the uneven division. Maybe
             * bump it up to 689 Hz.
             * 
             * That's a sine wave of shorts, repeated twice for two channels, with a
             * wavelength of 32 samples.
             * 
             * Note: Changed my mind, ignore specific numbers above.
             * 
             */
            {
                final int λ = 16;
                ByteBuffer buffer = ByteBuffer.allocate(λ * 2 * 8);
                for(int j = 0; j < 2; j++) {
                    for(double i = 0.0; i < λ; i++) {
                        System.out.println(j + " " + i);
                        //once for each sample
                        buffer.putShort((short)(Math.sin(Math.PI * (λ/i)) * Short.MAX_VALUE));
                        buffer.putShort((short)(Math.sin(Math.PI * (λ/i)) * Short.MAX_VALUE));
                    }
                }

                data = buffer.array();
            }

            addr = InetAddress.getByName("127.0.0.1");
            try(DatagramSocket socket = new DatagramSocket()) {
                while (true) {
                    for(byte b : data) System.out.print(b + " ");

                    // Read the next chunk of data from the TargetDataLine.
//                  numBytesRead = line.read(data, 0, data.length);

                    for(int i = 0; i < 64; i++) {
                        byte b = data[i];
                        System.out.print(b + " ");
                    }
                    System.out.println();

                    // Save this chunk of data.
                    dgp = new DatagramPacket(data, data.length, addr, 50005);    

                    for(int i = 0; i < 64; i++) {
                        byte b = dgp.getData()[i];
                        System.out.print(b + " ");
                    }
                    System.out.println();

                    socket.send(dgp);
                }
            }

        } catch (LineUnavailableException e) {
            e.printStackTrace();
        } catch (UnknownHostException e) {
            // TODO: handle exception
        } catch (SocketException e) {
            // TODO: handle exception
        } catch (IOException e2) {
            // TODO: handle exception
        }
    }
}

Obviously I misinterpreted it as a 512-byte-long piece and botched the sine wave, but the thing is, it produced exactly the sound that it was meant to--a mind-numbing rattle at a specific pitch.

This in mind, I don't suspect that the problem is explicitly in your code. The first thing I would check is which line your system is tapping for audio. Do you have multiple microphones hooked up? A webcam mic, maybe? You might grab a utility like PulseAudio Volume Control to check. If you haven't already checked on the functionality of your microphone, you might do that too; they do have a lifespan on them.

It isn't uncommon at all to scramble the bits in an audio stream, nor is it difficult; but I don't see anywhere where you could be doing that.

One thought might be to modify your program to attempt to play the sound locally, before sending it over to the server. That way, you can at least determine if the problem is pre- or post-Mic.

Answer 2

When client and server use data buffers of different sizes one will get truncated and may cause one or both to produce artifacts.

Your server buffer size is set to byte[] receiveData = new byte[4096];

Your client buffer size is for some reason dynamic, and set to byte[] data = new byte[buffsize];

Set client buffer size to a static 4096 to match the server: byte[] data = new byte[4096];

Or just make sure they are both the same size...

Answer 3

So this is an old question but solving this helped me somewhat and I suppose what I found might help others so.. this is how I solved the issues you described:

On my machine, changing

boolean bigEndian = true;

to

boolean bigEndian = false;

solved the white noise issue (it was obviously a byte order issue)

If this is the only change you make, the resulting audio is going to have a low pitch, this is due to the fact that on the Mic side you collect 2 channels and on the Speaker side you play through one channel.

To solve that simply change this line:

format = new AudioFormat(sampleRate, 16, 1, true, false);

to

format = new AudioFormat(sampleRate, 16, 2, true, false);

And then the audio should be clear and understandable

Answer 4

I suggest that you first write to a file the audio that is recorder on the client. This will enable you to verify if the captured audio is OK. You can convert the PCM to WAV using utilities like sox.

Answer 5

It is important to match the audio format on both client and server, for instance change the one in Client.java to: format = new AudioFormat(sampleRate, 16, 1, true, false); You also need to use the same buffer size on both programs.