Why do different sounds play depending on the playback audio device? (Mixing audio into video on Android)

Question

I have a m4a music file and a mp4 video file. I want to mux music file and video file in to a mp4 file and save it.

Using this article as a reference, I made my own code. My code almost works fine but different sounds play depending on the playback device .

Android (Pixel 3a): Music ＋ Video(without Audio)

iOS (Mac QuickTime Player, iPhone 11Pro): Music ＋ Video(with Audio) ← I want to do this!

Is there any solution??

my code:

    fun mux(audioFile: String, videoFile: String, outFile: String) {

        // video
        val videoExtractor = MediaExtractor()
        videoExtractor.setDataSource(videoFile)
        videoExtractor.selectTrack(0)
        val videoFormat = videoExtractor.getTrackFormat(0)

        // audio
        val soundExtractor = MediaExtractor()
        soundExtractor.setDataSource(videoFile)
        soundExtractor.selectTrack(1)
        val soundFormat = soundExtractor.getTrackFormat(1)

        // music
        val audioExtractor = MediaExtractor()
        audioExtractor.setDataSource(audioFile)
        audioExtractor.selectTrack(0)
        val audioFormat = audioExtractor.getTrackFormat(0)

        val muxer: MediaMuxer

        val videoMetaData = MediaMetadataRetriever()
        videoMetaData.setDataSource(videoFile)
        val degreeString = videoMetaData.extractMetadata(MediaMetadataRetriever.METADATA_KEY_VIDEO_ROTATION)
        val videoDegree = degreeString?.toInt() ?: 0
        muxer = MediaMuxer(outFile, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4)
        muxer.setOrientationHint(videoDegree)

        val videoIndex = muxer.addTrack(videoFormat)
        val soundIndex = muxer.addTrack(soundFormat)
        val audioIndex = muxer.addTrack(audioFormat)
        muxer.start()

        val maxChunkSize = 1024 * 1024
        val buffer = ByteBuffer.allocate(maxChunkSize)
        val videoBufferInfo = MediaCodec.BufferInfo()
        val soundBufferInfo = MediaCodec.BufferInfo()
        val audioBufferInfo = MediaCodec.BufferInfo()

        // copy video
        while (true) {
            val chunkSize = videoExtractor.readSampleData(buffer, 0)

            if (chunkSize > 0) {
                videoBufferInfo.presentationTimeUs = videoExtractor.sampleTime
                videoBufferInfo.flags = videoExtractor.sampleFlags
                videoBufferInfo.size = chunkSize

                muxer.writeSampleData(videoIndex, buffer, videoBufferInfo)

                videoExtractor.advance()
            } else {
                break
            }
        }

        // copy audio
        while (true) {
            val chunkSize = soundExtractor.readSampleData(buffer, 0)

            if (chunkSize > 0) {
                soundBufferInfo.presentationTimeUs = soundExtractor.sampleTime
                soundBufferInfo.flags = soundExtractor.sampleFlags
                soundBufferInfo.size = chunkSize

                muxer.writeSampleData(soundIndex, buffer, soundBufferInfo)
                soundExtractor.advance()
            } else {
                break
            }
        }

        // copy music
        while (true) {
            val chunkSize = audioExtractor.readSampleData(buffer, 0)

            if (chunkSize > 0 && videoBufferInfo.presentationTimeUs > audioBufferInfo.presentationTimeUs) {
                audioBufferInfo.presentationTimeUs = audioExtractor.sampleTime
                audioBufferInfo.flags = audioExtractor.sampleFlags
                audioBufferInfo.size = chunkSize

                muxer.writeSampleData(audioIndex, buffer, audioBufferInfo)
                audioExtractor.advance()
            } else {
                break
            }
        }


        muxer.stop()
        muxer.release()

        videoExtractor.release()
        soundExtractor.release()
        audioExtractor.release()
    }

Answer 1

The problem is that the Audio Stream is split into two parts . When I use ffmpeg to combine AudioStream into one and then mux audio file and video file in to a mp4 file, the problem is solved.