Mixing two16-bit encoded stereo PCM samples causing noise and distortion in the resulting audio

问题

I get two different audio samples from two sources.

For microphone sound:

audioRecord =
         new AudioRecord(MediaRecorder.AudioSource.DEFAULT, 44100, AudioFormat.CHANNEL_IN_STEREO, AudioFormat.ENCODING_PCM_16BIT,
                 (AudioRecord.getMinBufferSize(44100, AudioFormat.CHANNEL_IN_STEREO, AudioFormat.ENCODING_PCM_16BIT)*5));

For Internal sound:

audioRecord = new AudioRecord.Builder()
                 .setAudioPlaybackCaptureConfig(config)
                 .setAudioFormat(new AudioFormat.Builder()
                         .setEncoding(AudioFormat.ENCODING_PCM_16BIT)
                         .setSampleRate(44100)
                         .setChannelMask(AudioFormat.CHANNEL_IN_STEREO)
                         .build())
                 .setBufferSizeInBytes((AudioRecord.getMinBufferSize(44100, AudioFormat.CHANNEL_IN_STEREO, AudioFormat.ENCODING_PCM_16BIT)*5))
                 .build();

For reading from the audioRecord object we create individual frame objects(Custom objects called frame)-

private ByteBuffer pcmBuffer = ByteBuffer.allocateDirect(4096);
private Frame read() {
  pcmBuffer.rewind();
  int size = audioRecord.read(pcmBuffer, pcmBuffer.remaining());
  if (size <= 0) {
   return null;
  }
    return new Frame(pcmBuffer.array(),
           pcmBuffer.arrayOffset(), size);
}

We create two separate LL(Linked List) for adding these frames that we get from read function.

private LinkedList internalAudioQueue = new LinkedList<>(); private LinkedList microphoneAudioQueue = new LinkedList<>();

public void onFrameReceived(Frame frame, boolean isInternalAudio) {
    if (isInternalAudio) {
        internalAudioQueue.add(frame);
    } else {
        microphoneAudioQueue.add(frame);
    }
    checkAndPoll();
}

Every time we add a frame in the respective LL we call the following checkAndPoll() function and depending upon the case pass the frame to the audioEncoder.

public void checkAndPoll() {
    Frame frame1 = internalAudioQueue.poll();
    Frame frame2 = microphoneAudioQueue.poll();
    if (frame1 == null && frame2 != null) {
        audioEncoder.inputPCMData(frame2);
    } else if (frame1 != null && frame2 == null) {
        audioEncoder.inputPCMData(frame1);
    } else if (frame1 != null && frame2 != null) {
        Frame frame = new Frame(PCMUtil.mix(frame1.getBuffer(), frame2.getBuffer(), frame1.getSize(), frame2.getSize(), false), frame1.getOrientation(), frame1.getSize());
        audioEncoder.inputPCMData(frame);
    }
}

Now we mix the audio samples in form of ByteBuffer from the two sources in this way taking Hendrik's help from this link.

public static byte[] mix(final byte[] a, final byte[] b, final boolean bigEndian) {
    final byte[] aa;
    final byte[] bb;

    final int length = Math.max(a.length, b.length);
    // ensure same lengths
    if (a.length != b.length) {
        aa = new byte[length];
        bb = new byte[length];
        System.arraycopy(a, 0, aa, 0, a.length);
        System.arraycopy(b, 0, bb, 0, b.length);
    } else {
        aa = a;
        bb = b;
    }

    // convert to samples
    final int[] aSamples = toSamples(aa, bigEndian);
    final int[] bSamples = toSamples(bb, bigEndian);

    // mix by adding
    final int[] mix = new int[aSamples.length];
    for (int i=0; i<mix.length; i++) {
        mix[i] = aSamples[i] + bSamples[i];
        // enforce min and max (may introduce clipping)
        mix[i] = Math.min(Short.MAX_VALUE, mix[i]);
        mix[i] = Math.max(Short.MIN_VALUE, mix[i]);
    }

    // convert back to bytes
    return toBytes(mix, bigEndian);
}

private static int[] toSamples(final byte[] byteSamples, final boolean bigEndian) {
    final int bytesPerChannel = 2;
    final int length = byteSamples.length / bytesPerChannel;
    if ((length % 2) != 0) throw new IllegalArgumentException("For 16 bit audio, length must be even: " + length);
    final int[] samples = new int[length];
    for (int sampleNumber = 0; sampleNumber < length; sampleNumber++) {
        final int sampleOffset = sampleNumber * bytesPerChannel;
        final int sample = bigEndian
                ? byteToIntBigEndian(byteSamples, sampleOffset, bytesPerChannel)
                : byteToIntLittleEndian(byteSamples, sampleOffset, bytesPerChannel);
        samples[sampleNumber] = sample;
    }
    return samples;
}

private static byte[] toBytes(final int[] intSamples, final boolean bigEndian) {
    final int bytesPerChannel = 2;
    final int length = intSamples.length * bytesPerChannel;
    final byte[] bytes = new byte[length];
    for (int sampleNumber = 0; sampleNumber < intSamples.length; sampleNumber++) {
        final byte[] b = bigEndian
                ? intToByteBigEndian(intSamples[sampleNumber], bytesPerChannel)
                : intToByteLittleEndian(intSamples[sampleNumber], bytesPerChannel);
        System.arraycopy(b, 0, bytes, sampleNumber * bytesPerChannel, bytesPerChannel);
    }
    return bytes;
}

// from https://github.com/hendriks73/jipes/blob/master/src/main/java/com/tagtraum/jipes/audio/AudioSignalSource.java#L238
private static int byteToIntLittleEndian(final byte[] buf, final int offset, final int bytesPerSample) {
    int sample = 0;
    for (int byteIndex = 0; byteIndex < bytesPerSample; byteIndex++) {
        final int aByte = buf[offset + byteIndex] & 0xff;
        sample += aByte << 8 * (byteIndex);
    }
    return (short)sample;
}

// from https://github.com/hendriks73/jipes/blob/master/src/main/java/com/tagtraum/jipes/audio/AudioSignalSource.java#L247
private static int byteToIntBigEndian(final byte[] buf, final int offset, final int bytesPerSample) {
    int sample = 0;
    for (int byteIndex = 0; byteIndex < bytesPerSample; byteIndex++) {
        final int aByte = buf[offset + byteIndex] & 0xff;
        sample += aByte << (8 * (bytesPerSample - byteIndex - 1));
    }
    return (short)sample;
}

private static byte[] intToByteLittleEndian(final int sample, final int bytesPerSample) {
    byte[] buf = new byte[bytesPerSample];
    for (int byteIndex = 0; byteIndex < bytesPerSample; byteIndex++) {
        buf[byteIndex] = (byte)((sample >>> (8 * byteIndex)) & 0xFF);
    }
    return buf;
}

private static byte[] intToByteBigEndian(final int sample, final int bytesPerSample) {
    byte[] buf = new byte[bytesPerSample];
    for (int byteIndex = 0; byteIndex < bytesPerSample; byteIndex++) {
        buf[byteIndex] = (byte)((sample >>> (8 * (bytesPerSample - byteIndex - 1))) & 0xFF);
    }
    return buf;
}

The mixed samples that I am getting have both distortion and noise. Not able to figure out what needs to be done to remove it. Any help here is appreciated. Thanks in Advance!

回答1:

I think if you're mixing, you should take the (weighted) average of both.

If you've got a sample 128 and 128, the result would be still 128, not 256, which could be out-of-range.

So just change your code to:

// mix by adding
final int[] mix = new int[aSamples.length];
for (int i=0; i<mix.length; i++) {
    // calculating the average
    mix[i] = (aSamples[i] + bSamples[i]) >> 1;
}

Does that work for you?

来源：https://stackoverflow.com/questions/65662025/mixing-two16-bit-encoded-stereo-pcm-samples-causing-noise-and-distortion-in-the

标签

android

signal-processing

pcm

audio-processing

android-audiorecord