Delay with voice streams on java

Question

I've been working on a java project that makes calls from an usb modem. The application works pefrectly on my computer, but when I tryed to run it on a lower-specs one, the audio stream of the person calling from the pc goes out perfectly and it's perfectly heared on the phone called. But the audio that should be heared by the pc user gets delayed (3 to 5 secs), with white noise, and makes literally impossible to make a conversation.

Some things to take in mind:

• My computer is an i3 4gb RAM notebook, and the low specs are Pentium 4 1gb RAM desktop.
• I tested the CPU and RAM usages, the application consumes 20 - 25% of the cpu on my computer, almost 100% on the low-specs one, and about 30 - 40mb from RAM on both cases.
• The application also has a call recording feature, and for some reason the output files are written perfectly (no delays or iterferences).

Any clue on what could be the problem or how can it be solved?

Class made to handle the audio after I start the new thread:(ingoing call audio)

public class SerialVoiceReader implements Runnable{

    /** The running. */
private volatile boolean running = true;

/** The in. */
DataInputStream in;

/** The af. */
AudioFormat af;

/** The samples per frame. */
private int samplesPerFrame = 160; 

/** The audio buffer size. */
private int audioBufferSize = samplesPerFrame * 2 ; //20ms delay

private String tel;

private String timestamp;

public SerialVoiceReader ( DataInputStream in,  AudioFormat af){
    this.in = in;
    this.af = af;
}

public void run (){
        try
        {
            Info infos = new Info(SourceDataLine.class, af);
            SourceDataLine dataLine  = (SourceDataLine) AudioSystem.getLine(infos);
            dataLine.open(dataLine.getFormat(),audioBufferSize *2);                     
            dataLine.start();   
// set the volume up
            if (dataLine.isControlSupported(FloatControl.Type.MASTER_GAIN)) {
                FloatControl volume = (FloatControl) dataLine.getControl(FloatControl.Type.MASTER_GAIN);
                volume.setValue(volume.getMaximum());
            }
// get a field from GUI to set as part of the file name
            tel = CallGUI.telField.getText();
            timestamp = new SimpleDateFormat("yyyyMMddHHmmss").format(Calendar.getInstance().getTime());

            // save the stream to a file to later set the header and make it .wav format
            FileOutputStream fos = new FileOutputStream("Llamadas/" + timestamp + "-" + tel + "-OUT.raw");
            // the audio buffer writing (this is the audio that goes out on the call)
            while (running){
                byte[] buffer = new byte[audioBufferSize];
                int offset = 0;
                int numRead = 0;
                while (running && (offset < buffer.length && (numRead = this.in.read(buffer, offset, buffer.length - offset)) >= 0)) 
                {
                    offset += numRead;
                }
                if(running && offset>=0){
                    dataLine.write(buffer, 0, offset);
                    fos.write(buffer);
                }
            }   
            dataLine.stop();
            dataLine.drain();
            dataLine.close();
            fos.close();

        }
        catch ( Exception e )
        {
        }          
    }

---

Class made to handle the audio after I start the new thread:(outgoing call audio)

public class SerialVoiceWriter implements Runnable{

    /** The running. */
    private volatile boolean running = true;

    /** The out. */
    DataOutputStream out;

    /** The af. */
    AudioFormat af;

    /** The samples per frame. */
    private int samplesPerFrame = 160; 

    /** The audio buffer size. */
    private int audioBufferSize = samplesPerFrame * 2; //20ms delay

    private String tel;

    private String timestamp;

    public SerialVoiceWriter ( DataOutputStream out, AudioFormat af, Boolean playMessage)
    {
        this.out = out;
        this.af = af;
    }

    public void run ()
    {   
        try
        {   
                Info infos = new Info(TargetDataLine.class, af);
                TargetDataLine dataLine  = (TargetDataLine) AudioSystem.getLine(infos);
                dataLine.open(dataLine.getFormat(),audioBufferSize*2 );
                dataLine.start();

                tel = CallGUI.telField.getText();
                timestamp = new SimpleDateFormat("yyyyMMddHHmmss").format(Calendar.getInstance().getTime());

                FileOutputStream fis = new FileOutputStream("Llamadas/" + timestamp + "-" + tel + "-IN.raw");
                while (running){
                    byte[] audioBuffer = new byte[audioBufferSize];
                    int offset = 0;
                    int numRead = 0;
                    while (running && (offset < audioBuffer.length && (numRead = dataLine.read(audioBuffer, offset, audioBuffer.length - offset)) > 0)) 
                    {
                        offset += numRead;
                    }
                    if(running && offset>=0){
                        this.out.write(audioBuffer);
                        fis.write(audioBuffer);
                    }
                }               
                    dataLine.flush();   
                    dataLine.stop();
                    dataLine.close();
                    fis.close();
                    dataLine = null;                

        }
        catch (Exception e )
        {
        }            
    }

Thank you in advice

Answer 1

The steps you need to take are:

1. Profile/Sample the application and find out where the time is really being spent. VisualVM is powerful and free and comes as part of the JDK. Start your application. Start VisualVM. Have VisualVM connect to your application. Go to the Sampler tab and start sampling CPU usage. After a couple minutes take a snapshot. Look at it. If you can't figure it out, post something here.
2. Move the audio buffer initialization out of the loop. If your buffer is 20ms the byte array is being allocated and garbage collected 50x per second. This is obvious and easy to do, but probably won't solve your problem.
3. Wrap your FileOutputStreams with BufferedOutputStreams . Like this: OutputStream fos = new BufferedOutputStream( new FileOutputStream("Llamadas/" + timestamp + "-" + tel + "-OUT.raw")); You'll get hugely improved performance. The way it is now every iteration of the loop waits for the buffer to finish being written to disk. Physical disks are slow and that adds up to alot of waiting.
4. Get rid of the inner while loop. Its not important to actually fill up the buffer. When the inner while loop is filling up that buffer you are getting out of sync. What you want to do is try to read from the input stream once and if something was read then write whatever was read to the output stream. Instead of calling write(byte[]) call DataOutputStream write(byte[], off, len)

5. This will take a little more work: Instead of writing to dataLine AND THEN writing to fos sequentially, write to them in parallel. They each take a certain amount of time to write data to their respective destinations. If fos takes X microseconds and dataLine takes Y your current code takes X + Y microseconds. If you do it in parallel you could end up only waiting max(X, Y). `

    ExecutorService es = Executors.newFixedThreadPool(2); Callable<Void>[] calls = new Callable[2]; //... your other code here... if (running && offset >= 0) { final int finalOffset = offset; Callable<Void> call1 = new Callable<Void>() { @Override public Void call() throws Exception { dataLine.write(buffer, 0, finalOffset); return null; } }; Callable<Void> call2 = new Callable<Void>() { @Override public Void call() throws Exception { fos.write(buffer); // or however you need to write. return null; } }; calls[0] = call1; calls[1] = call2; List<Callable<Void>> asList = Arrays.asList(calls); es.invokeAll(asList); // invokeAll will block until both callables have completed. } 

   `

6. If the improvement in #5 isn't good enough, you can move the writes into the background. Once you've read the first piece of data you then kick off the writes in separate threads - but don't wait for the writes to finish. Immediately start reading the next piece of data. Once you have the next bit of data you then wait for the first writes to finish and then you start the second write in the background.

Answer 2

I think that whatever is causing the close to 100% CPU is the culprit. But that doesn't really tell you anything specific too look at. First thing is that since the problem is with playback on the lower end PC, you may want to check if audio drivers on that device are up to date. After that, I'd look at optimizing the portion of code that is handling the receiving of audio back. Although the older PC is lower specced, I don't think it should have any problems with what you are trying to implement. I suggest running a performance profiler while you app is running to see what is taking long.

Update: You could try upping the audioBufferSize to see if it has any effect, 20ms seems low. The code provided is just for audio sent from the pc. What about the audio received from the phone?