Audio and video development journey (6) MediaCodec hard compilation process and practice

Audio and video development journey (6) MediaCodec hard compilation process and practice

table of Contents

Introduction to MediaCodec

Working principle and basic process

Data Format

Life cycle

Synchronous and asynchronous modes

Flow Control

Practice: AAC decoding is two realizations of PCM synchronous and asynchronous

Problems encountered

reference

reward

1. Introduction Android bottom multimedia module uses the OpenMax framework, and the implementation parties must follow the OpenMax standard. By default, Google provides a series of implementations of soft programming and software solutions, while hard programming and hard solutions are completed by chip manufacturers. Therefore, the implementation and performance of hard programming and hard solutions for mobile phones with different chips will be different. For example, the codec implementation part of my mobile phone is as follows

MediaCodec provides a set of interfaces for accessing the underlying multimedia modules and the application layer realizes the coding and decoding functions.

2. Working principle and basic process The basic process used by MediaCodec is as follows:

  • createByCodeName/createEncoderByType/createDecoderByType: (Static factory constructs MediaCodec object) --Uninitialized state
  • configure: (configuration) - configure state
  • start (start)-enter the Running state
  • while(1) {try{-dequeueInputBuffer (getting the input buffer from the codec)-queueInputBuffer (the buffer is filled up by the client and submitted to the codec)-dequeueOutputBuffer (getting the output buffer from the codec) -releaseOutputBuffer (the consumer client releases it to the compiler after consumption)} catch(Error e){-error (enter the error state when an exception occurs)}

}

  • stop (After the encoding and decoding is complete, release the codec)
  • release

Combining the basic process with the code, the two Buffer queue diagrams and the life cycle diagram together, the whole process will be very clear.

Below we focus on the operation of the core part of the Buffer queue. MediaCodec uses 2 buffer queues (inputBuffer and outputBuffer) to process data asynchronously,

  1. The data producer (Client on the left) applies for an empty buffer from the input buffer queue "dequeueinputBuffer
  2. The data producer (the client on the left) copies the data that needs to be coded and decoded to the empty buffer, and then to the input buffer queue "queueInputBuffer
  3. MediaCodec takes a frame from the input buffer queue for encoding and decoding processing
  4. After the encoding and decoding process, MediaCodec sets the original input buffer to empty and puts it back into the input buffer queue on the left, and puts the encoded and decoded data into the output buffer queue on the right
  5. Consumer Client (Client on the right) applies for the encoded and decoded buffer from the output buffer queue "dequeueOutputBuffer
  6. The consumer client (the client on the right) renders or plays the encoded and decoded buffer
  7. After rendering/playing is complete, the consumer Client puts the buffer back into the output buffer queue "releaseOutputBuffer

3. Data format Mediacodec accepts three data formats and two carriers respectively as follows: Compressed data, original audio data and original video data use Surface as the carrier or ByteBuffer as the carrier

Compressed data Compressed data can be used as the input of the decoder and the output of the encoder. The format of the data needs to be specified, so that the codec knows how to process the compressed data

Raw audio data PCM audio data frame

The commonly used color formats supported by the original video data video decoding are native raw video format and flexible YUV buffers native raw video format: COLOR_FormatSurface, which can be used to process surface mode data input and output. Flexible YUV buffers: For example, COLOR_FormatYUV420Flexible, which can be used to process the output output in surface mode. When using the ByteBuffer mode, you can use the getInput/OutputImage(int) method to get the image data.

4. the life cycle

MediaCodec life cycle status is divided into three kinds of Stopped, Executing and Released. In the first part of the working principle and basic process, we also briefly mentioned. Below we will elaborate on the three sub-states of Stopped, Uninitialized (initialized state), Configured Configuration state), Error (abnormal state) Executing also contains three sub-states Flushed (refreshing state), Running (running state) and EOS (stream end state). Let s focus on the Executing state after calling the mediacodec.start() method. The encoder immediately enters the Flush sub-state of the Executing state. At this time, the codec will have all the inputBuffer. Once the first input buffer inputbuffer is removed from the queue (ie: queueInputBuffer), the codec turns to the Running state, and most of the codec The life cycle will be in this state. When the inputBuffer marked with end-of-stream enters the queue (queueInputBuffer), the codec will switch to EOS state. In this state, the codec no longer receives a new inputBuffer, but still generates an outputBuffer until the end-of-stream mark reaches the output. You can call flush() at any time under Executiong to return the codec to the flushed state.

V. Synchronous and asynchronous mode Buffer production and consumption has a mode, one is the synchronous mode, which is the process method introduced in the first part of this article. Asynchronous mode was introduced from android5.0 google, and the production and consumption operation of the buffer was performed by setting the callback setCalback to the codec. (img) The official code is as follows:

MediaCodec codec = MediaCodec.createByCodecName(name); MediaFormat mOutputFormat;//member variable codec.setCallback(new MediaCodec.Callback() {@Override void onInputBufferAvailable(MediaCodec mc, int inputBufferId) {ByteBuffer inputBuffer = codec.getInputBuffer(inputBufferId);//fill inputBuffer with valid data codec.queueInputBuffer(inputBufferId, );}

@Override void onOutputBufferAvailable(MediaCodec mc, int outputBufferId, ) {ByteBuffer outputBuffer = codec.getOutputBuffer(outputBufferId); MediaFormat bufferFormat = codec.getOutputFormat(outputBufferId);//option A//bufferFormat is equivalent to mOutputFormat//outputBuffer is ready to be processed or rendered. codec.releaseOutputBuffer(outputBufferId, );}

@Override void onOutputFormatChanged(MediaCodec mc, MediaFormat format) {//Subsequent data will conform to new format.//Can ignore if using getOutputFormat(outputBufferId) mOutputFormat = format;//option B}

@Override void onError( ) { } }); codec.configure(format, ); mOutputFormat = codec.getOutputFormat();//option B codec.start();//wait for processing to complete codec.stop (); codec.release(); 6. MediaCodec flow control encoder can set a target bit rate, but the actual output bit rate of the encoder will not completely match the setting, and the actual output that can be controlled during the encoding process is not the final output The code rate is a quantization parameter (Quantiaztion Parameter QP) in the encoding process. It has no fixed relationship with the code rate, but depends on the image content. There are two modes of android code rate control. One is to set the target code rate and code rate control mode when setting cofigure.

mediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, bitRate); mediaFormat.setInteger(MediaFormat.KEY_BITRATE_MODE, MediaCodecInfo.EncoderCapabilities.BITRATE_MODE_VBR); mVideoCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE); There are three code rate control modes: CQ Do not control the bit rate at all, try to ensure the image quality as much as possible. This strategy can be selected when the quality requirements are high, the bandwidth is not concerned, and the decoder supports drastic fluctuations in the bit rate; CBR means that the encoder will try to control the output bit rate to the setting Constant value, the output code rate will fluctuate within a certain range. For small shaking, the block effect will be improved, but there is still nothing to do with severe shaking. Continuously reducing the code rate will cause the code rate to drop sharply. If this problem is unacceptable, Then VBR is not a good choice; VBR means that the encoder will dynamically adjust the output bit rate according to the complexity of the image content (actually the amount of change between frames), the bit rate is high if the image is complex, and the bit rate is low if the image is simple , The advantage is that it is stable and controllable, which is helpful for real-time guarantee. Therefore, CBR is generally used in WebRTC development;

The other is to dynamically adjust the target bit rate.

Bundle param = new Bundle(); param.putInt(MediaCodec.PARAMETER_KEY_VIDEO_BITRATE, bitrate); mediaCodec.setParameters(param); 7. Practice: AAC decoding is realized by PCM synchronous and asynchronous (audio decoding). Purpose: through this function The practice, familiar with the process of mediaCodec, and the implementation of both synchronous and asynchronous methods are as follows:

package com.av.mediajourney.mediacodec;

import android.content.Context; import android.media.MediaCodec; import android.media.MediaExtractor; import android.media.MediaFormat; import android.os.Build; import android.os.Environment; import android.text.TextUtils; import android.util.Log; import android.widget.Toast;

import androidx.annotation.NonNull;

import java.io.File; import java.io.FileNotFoundException; import java.io.FileOutputStream; import java.io.IOException; import java.nio.ByteBuffer;

public class AACToPCMDelegate {

private static final String TAG = "AACToPCMDelegate";
private static final long TIMEOUT_US = 0;
private Context mContext;

private ByteBuffer[] inputBuffers;
private ByteBuffer[] outputBuffers;
private MediaExtractor mediaExtractor;
private MediaCodec decodec;

private FileOutputStream fileOutputStream;

public AACToPCMDelegate(MediaCodecActivity context) {
    this.mContext = context;
}

/**
 *  aacToPCM  mediaCodec 
 */
void aacToPCM() {
    boolean isAsync = true;
    if (isAsync && Build.VERSION.SDK_INT < Build.VERSION_CODES.LOLLIPOP) {
        isAsync = false;
    }

   //1. initFile
    File file = initFile(isAsync);
    if (file == null) {
        return;
    }

   //2.  mediaCodec
    initMediaCodec(file.getAbsolutePath(), isAsync);
    if (decodec == null) {
        Toast.makeText(mContext, "decodec is null", Toast.LENGTH_SHORT).show();
        return;
    }

    if (!isAsync) {
       //
        decodecAacToPCMSync();
    }
}

private File initFile(boolean isAsync) {
    String child = isAsync ? "aacToPcmAsync.pcm" : "aacToPcmSync.pcm";
    File outputfile = new File(mContext.getExternalFilesDir(Environment.DIRECTORY_MUSIC), child);
    if (outputfile.exists()) {
        outputfile.delete();
    }
    try {
        fileOutputStream = new FileOutputStream(outputfile.getAbsolutePath());
    } catch (FileNotFoundException e) {
        e.printStackTrace();
    }
    File file = new File(mContext.getExternalFilesDir(Environment.DIRECTORY_MUSIC), "sanguo.aac");
    if (!file.exists()) {
        Toast.makeText(mContext, " ", Toast.LENGTH_SHORT).show();
        return null;
    }
    return file;
}

private void initMediaCodec(String path, boolean isASync) {
    mediaExtractor = new MediaExtractor();
    try {
        mediaExtractor.setDataSource(path);
        int trackCount = mediaExtractor.getTrackCount();
        for (int i = 0; i < trackCount; i++) {

            MediaFormat trackFormat = mediaExtractor.getTrackFormat(i);
            String mime = trackFormat.getString(MediaFormat.KEY_MIME);
            if (TextUtils.isEmpty(mime)) {
                continue;
            }
            Log.i(TAG, "initMediaCodec: mime=" + mime);
            if (mime.startsWith("audio/")) {
                mediaExtractor.selectTrack(i);
            }
           //MediaCodec Uninitialized 
            decodec = MediaCodec.createDecoderByType(mime);
           //configure  Configured 
            decodec.configure(trackFormat, null, null, 0);

            if (isASync) {
                setAsyncCallBack();
            }
           //Excuting  Flushed 
            decodec.start();
            if (Build.VERSION.SDK_INT < Build.VERSION_CODES.LOLLIPOP) {
                inputBuffers = decodec.getInputBuffers();
                outputBuffers = decodec.getOutputBuffers();
                Log.i(TAG, "initMediaCodec: inputBuffersSize=" + inputBuffers.length + " outputBuffersSize=" + outputBuffers.length);
            }
        }
    } catch (IOException e) {
        e.printStackTrace();
    }
}


/**
 *  
 *  pcm ffplay pcm   aac 
 * ffplay -ar 44100 -channels 2 -f s16le -i/Users/yabin/Desktop/tmp/aacToPcm.pcm
 */
private void decodecAacToPCMSync() {
    boolean isInputBufferEOS = false;
    boolean isOutPutBufferEOS = false;

    while (!isOutPutBufferEOS) {
        if (!isInputBufferEOS) {
           //1.  codecInputBuffer empty input buffer index
            int index = decodec.dequeueInputBuffer(TIMEOUT_US);
            if (index >= 0) {
                ByteBuffer inputBuffer;
               //2.  index inputBuffer
                if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) {
                    inputBuffer = decodec.getInputBuffer(index);
                } else {
                    inputBuffer = inputBuffers[index];
                }
                if (inputBuffer != null) {
                    Log.i(TAG, "decodecAacToPCMSync: " + "  index=" + index);

                    inputBuffer.clear();
                }
               //extractor sampleData
                int sampleSize = mediaExtractor.readSampleData(inputBuffer, 0);
               //3.  EOS buffer  code inputbuffer
                Log.i(TAG, "decodecAacToPCMSync: sampleSize=" + sampleSize);
                if (sampleSize < 0) {
                    isInputBufferEOS = true;
                    decodec.queueInputBuffer(index, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM);
                } else {
                    decodec.queueInputBuffer(index, 0, sampleSize, mediaExtractor.getSampleTime(), 0);
                   //
                    mediaExtractor.advance();
                }
            }
        }

        MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
       //4.  Client  outputbuffer index
        int index = decodec.dequeueOutputBuffer(bufferInfo, TIMEOUT_US);
        if (index < 0) {
            continue;
        }
        ByteBuffer outputBuffer;
       //5.  index inputBuffer
        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) {
            outputBuffer = decodec.getOutputBuffer(index);
        } else {
            outputBuffer = outputBuffers[index];
        }

        Log.i(TAG, "decodecAacToPCMSync: outputbuffer index=" + index + " size=" + bufferInfo.size + " flags=" + bufferInfo.flags);
       //FileOutputStream
        byte[] bytes = new byte[bufferInfo.size];
        outputBuffer.get(bytes);
        try {
            fileOutputStream.write(bytes);
            fileOutputStream.flush();
        } catch (IOException e) {
            e.printStackTrace();
        }

       //6.  outputbuffer codec outputbuffer
        decodec.releaseOutputBuffer(index, false);
        if ((bufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
            isOutPutBufferEOS = true;
        }
    }

    close();
}

private void close() {
    mediaExtractor.release();
    decodec.stop();
    decodec.release();
    try {
        fileOutputStream.close();
    } catch (IOException e) {
        e.printStackTrace();
    }
}

private void setAsyncCallBack() {
    if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) {
        decodec.setCallback(new MediaCodec.Callback() {
            @Override
            public void onInputBufferAvailable(@NonNull MediaCodec codec, int index) {
                Log.i(TAG, "setAsyncCallBack - onInputBufferAvailable: index=" + index);
               //1.  codecInputBuffer empty input buffer index
                if (index >= 0) {
                    ByteBuffer inputBuffer;
                   //2.  index inputBuffer
                    inputBuffer = decodec.getInputBuffer(index);
                    if (inputBuffer != null) {
                        Log.i(TAG, "setAsyncCallBack- onInputBufferAvailable: " + "  index=" + index);
                        inputBuffer.clear();
                    }
                   //extractor sampleData
                    int sampleSize = mediaExtractor.readSampleData(inputBuffer, 0);
                   //3.  EOS buffer  code inputbuffer
                    Log.i(TAG, "setAsyncCallBack- onInputBufferAvailable: sampleSize=" + sampleSize);
                    if (sampleSize < 0) {
                        decodec.queueInputBuffer(index, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM);
                    } else {
                        decodec.queueInputBuffer(index, 0, sampleSize, mediaExtractor.getSampleTime(), 0);
                       //
                        mediaExtractor.advance();
                    }
                }
            }

            @Override
            public void onOutputBufferAvailable(@NonNull MediaCodec codec, int index, @NonNull MediaCodec.BufferInfo bufferInfo) {
                Log.i(TAG, "setAsyncCallBack - onOutputBufferAvailable: index=" + index + " size=" + bufferInfo.size + " flags=" + bufferInfo.flags);
               //4.  Client  outputbuffer index
                if (index >= 0) {
                    ByteBuffer outputBuffer;
                   //5.  index inputBuffer
                    outputBuffer = decodec.getOutputBuffer(index);

                    Log.i(TAG, "setAsyncCallBack - onOutputBufferAvailable: outputbuffer index=" + index + " size=" + bufferInfo.size + " flags=" + bufferInfo.flags);
                   //FileOutputStream
                    byte[] bytes = new byte[bufferInfo.size];
                    outputBuffer.get(bytes);
                    try {
                        fileOutputStream.write(bytes);
                        fileOutputStream.flush();
                    } catch (IOException e) {
                        e.printStackTrace();
                    }

                   //6.  outputbuffer codec outputbuffer
                    decodec.releaseOutputBuffer(index, false);
                    if ((bufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
                        close();
                    }
                }

            }

            @Override
            public void onError(@NonNull MediaCodec codec, @NonNull MediaCodec.CodecException e) {
                Log.e(TAG, "setAsyncCallBack - onError: ");
            }

            @Override
            public void onOutputFormatChanged(@NonNull MediaCodec codec, @NonNull MediaFormat format) {
                Log.i(TAG, "setAsyncCallBack - onOutputFormatChanged: ");

            }
        });
    }

}
 

} 8. the problems encountered

java.lang.IllegalStateException at android.media.MediaCodec.getBuffer(Native Method) at android.media.MediaCodec.getInputBuffer(MediaCodec.java:3246)

int index = decodec.dequeueInputBuffer(TIMEOUT_US); _ After that, the inputbuffer is processed directly without judging whether the index is valid (index>=0) 9. Refer to the official documents of MeidaCodec

Android audio development (5): encoding and decoding of audio data

MediaCodec performs AAC encoding and decoding (file format conversion)

Android MediaCodec stuff

10. Gains It is strongly recommended to learn MediaCodec from the sample code instead of trying to figure it out from the documentation.

Understand the working principle and basic process of MediaCodec

Application understanding of life cycle

Synchronous and asynchronous usage and scenarios of codec

Flow control settings

Through AAC decoding into PCM, synchronous and asynchronous realization step by step understanding of the principle process

Summary and review of the problems encountered

Thank you for reading. In the next article, we will learn and practice SurfaceView, GLSurfaceView, TextureView, SurfaceTexture, Surface, and their relationships, usage scenarios, and advantages and disadvantages.

Welcome to follow the "Audio and Video Development Journey", your "watching", "like" and "sharing" are all great support. See you in the next article.

View original text