InputStream音频混合（MODE_STREAM）

Question

I'm making a drum sequencer in Android...

I'm writing to an AudioTrack in MODE_STREAM , so that I can achieve synchronized audio playback with all InputStreams (availible via a list of 'active' InputStreams, activeStreams in the code below)

The audio is always: PCM (WAV), 16bit Stereo 44100 Hz.

Obviously, I can't composite audio in real time on the UI thread, so I'm using an AsyncTask to queue up all the audio buffering.

I got buffered playback working, but when it comes to merging the buffers of two (or more) InputStream 's, the internet seems to be in some kind of debate of what to do next. "Convert the byte[] to short[]!", "No, do the bit mixing on-the-fly!", "But if you don't use shorts the byte Endianness is ignored!", "It gets ignored anyway!" - I don't even know any more.

How do I mix the buffer of two or more InputStreams? I don't understand why my current implementation is failing

I've tried like, 4 different StackOverflow solutions to convert the byte[] to short[] so I can add the samples together, but the conversion always instantly crashes Java with some cryptic error message that I can't get my head around. So now I give up. Here's my code implementing one such StackOverflow solution ...

protected Long doInBackground(Object ... Object) {

  int bytesWritten = 0;
  InputStream inputStream;
  int si = 0, i = 0;

  //The combined buffers. The 'composition'
  short[] cBuffer = new short[Synth.AUDIO_BUFFER_SIZE];

  //The 'current buffer', the segment of inputStream audio.
  byte[] bBuffer = new byte[Synth.AUDIO_BUFFER_SIZE];

  //The 'current buffer', converted to short?
  short[] sBuffer = new short[Synth.AUDIO_BUFFER_SIZE];

  int curStreamNum;
  int numStreams = activeStreams.size();
  short mix;

  //Start with an empty 'composition'
  cBuffer = new short[Synth.AUDIO_BUFFER_SIZE];

  boolean bufferEmpty = false;
  try {
    while(true) { // keep going forever, until stopped or paused.
      for(curStreamNum = 0;curStreamNum < numStreams;curStreamNum++){
        inputStream = activeStreams.get(curStreamNum);
        i = inputStream.read(bBuffer);
        bufferEmpty = i<=-1;
        if(bufferEmpty){
          //Input stream buffer was empty. It's out of audio. Close and remove the stream.
          inputStream.close();
          activeStreams.remove(curStreamNum);
          curStreamNum--; numStreams--; continue; // hard continue.
        }else{
          //Take the now-read buffer, and convert to shorts.
          ByteBuffer.wrap(bBuffer).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(sBuffer);

          //Take the short buffer, merge into composition buffer.
          //TODO: Optimize by making the 'first layer' of the composition the first buffer, on its own.
          for(si=0;si<Synth.AUDIO_BUFFER_SIZE;si++){
            mix = (short) (sBuffer[si] + cBuffer[si]);
            //This part is probably completely wrong too. I'm not up to here yet to evaluate whats needed...
            if(mix >= 32767){
              mix = 32767;
            }else if (mix <= -32768){
              mix = -32768;
            }
            cBuffer[si] = mix;
          }
        }
      }
      track.write(sBuffer, 0, i);

      //It's always full; full buffer of silence, or of composited audio.
      totalBytesWritten += Synth.AUDIO_BUFFER_SIZE;

      //.. queueNewInputStreams ..
      publishProgress(totalBytesWritten);
      if (isCancelled()) break;
    }
  } catch (IOException e) {e.printStackTrace();}
  return Long.valueOf(totalBytesWritten);
}

I'm currently getting a BufferUnderflowException on this line: ByteBuffer.wrap(bBuffer).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(sBuffer); .

How is it possible to have buffer underrun? I'm only converting a byte[] to a short[].

Please help!

I've posted my whole function in the hopes that this more complete code sample and fairly adaptable usage can help other people out there.

(PS the byte[] to short[] conversion is followed by some flimsy hard clipping which I'm not even up to debugging yet, but advice there would also be appreciated)

Answer 1

Your solution seems almost good, I see two issues and a potential one:

1. the length of the short array: it MUST be the half of the byte array, otherwise you get the underflow
2. the sum of the short must be the average of the shorts and not just the sum, or you'll get just noise
3. (potential issue) the length of the array you read by InputStream cannot be totally free, since you have to sum 2bytes for every InputStream (then it must be an even array) and you should take care of mono vs. stereo audio files (if stereo you have 2bytes for the left channel and 2bytes for the right channel interleaved)

Here you can find a snippet that I would use to sum of two WAV array (16bit, mono)

    Random random = new Random();

    int bufferLength = 20;

    byte[] is1 = new byte[bufferLength];
    byte[] is2 = new byte[bufferLength];
    byte[] average = new byte[bufferLength];

    random.nextBytes(is1);
    random.nextBytes(is2);

    short[] shorts1 = new short[bufferLength/2];
    ByteBuffer.wrap(is1).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(shorts1);

    short[] shorts2 = new short[bufferLength/2];
    ByteBuffer.wrap(is2).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(shorts2);

    short[] result = new short[bufferLength/2];

    for (int i=0; i<result.length; i++) {
        result[i] = (short) ((shorts1[i] + shorts2[i])/2);
    }

    ByteBuffer.wrap(average).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().put(result);

For a 32bit stereo, the solution could be

    Random random = new Random();

    int bufferLength = 8 * 50;

    byte[] is1 = new byte[bufferLength];
    byte[] is2 = new byte[bufferLength];
    byte[] average = new byte[bufferLength];

    random.nextBytes(is1);
    random.nextBytes(is2);

    System.out.println(bytesToHex(is1));
    System.out.println(bytesToHex(is2));

    int[] ints1 = new int[bufferLength/4];
    ByteBuffer.wrap(is1).order(ByteOrder.LITTLE_ENDIAN).asIntBuffer().get(ints1);

    int[] ints2 = new int[bufferLength/4];
    ByteBuffer.wrap(is2).order(ByteOrder.LITTLE_ENDIAN).asIntBuffer().get(ints2);

    int[] result = new int[bufferLength/4];

    for (int i=0; i<result.length; i++) {
        result[i] = ((ints1[i] + ints2[i])/2);
    }

    ByteBuffer.wrap(average).order(ByteOrder.LITTLE_ENDIAN).asIntBuffer().put(result);