NuPlayer source code analysis (4) NuPlayer difference between Android Q and Android 5.1 versions

1. Introduction:
Because it was convenient for local debugging before, the analysis of nuplayer is based on Android5.1the version. After getting familiar with the whole architecture, AndroidQI analyzed all the processes of nuplayer in the version and found that the changes are not particularly large, mainly reflected in three Aspects:

1.将5.1版本用于协调解码及渲染的主动循环改为了MediaCodec的消息回调方式；
2.handleAnInputBuffer和handleAnOutputBuffer均变为了MediaCodec的消息回调方式；
3.修正了视频帧校准的bug；

2. Analysis of message callback:
The AndroidQ version onConfigurefunction has the following code fragments:

onConfigure@NuPlayerDecoder.cpp：
void NuPlayer::Decoder::onConfigure(const sp<AMessage> &format) {
    
    
	...
    sp<AMessage> reply = new AMessage(kWhatCodecNotify, this);
    mCodec->setCallback(reply);
    ...
}

Look at the implementation of the setCallback function in MediaCodec:

status_t MediaCodec::setCallback(const sp<AMessage> &callback) {
    
    
    sp<AMessage> msg = new AMessage(kWhatSetCallback, this);
    msg->setMessage("callback", callback);

    sp<AMessage> response;
    return PostAndAwaitResponse(msg, &response);
}

Here, the incoming kWhatCodecNotifymessage will be written to kWhatSetCallback, see kWhatSetCallbackthe message processing below:

 case kWhatSetCallback:
 {
    
    
     uint32_t replyID;
     CHECK(msg->senderAwaitsResponse(&replyID));

     if (mState == UNINITIALIZED
             || mState == INITIALIZING
             || isExecuting()) {
    
    
         // callback can't be set after codec is executing,
         // or before it's initialized (as the callback
         // will be cleared when it goes to INITIALIZED)
         PostReplyWithError(replyID, INVALID_OPERATION);
         break;
     }

     sp<AMessage> callback;
     CHECK(msg->findMessage("callback", &callback));

	 /* 保存kWhatCodecNotify这个消息 */
     mCallback = callback;

     if (mCallback != NULL) {
    
    
         ALOGI("MediaCodec will operate in async mode");
         mFlags |= kFlagIsAsync;
     } else {
    
    
         mFlags &= ~kFlagIsAsync;
     }

     sp<AMessage> response = new AMessage;
     response->postReply(replyID);
     break;
 }

We can see that onConfigurethe function will kWhatCodecNotifypass this message to MediaCodec. When there is an input and output buffer available, OMX will call the sum
onInputBufferAvailablefunction of MediaCodec onOutputBufferAvailableto send this message:

void MediaCodec::onInputBufferAvailable() {
    
    
    int32_t index;
    while ((index = dequeuePortBuffer(kPortIndexInput)) >= 0) {
    
    
    	/* 深拷贝该消息 */
        sp<AMessage> msg = mCallback->dup();
        /* 注明是处理input/output */
        msg->setInt32("callbackID", CB_INPUT_AVAILABLE);
        msg->setInt32("index", index);
        msg->post();
    }
}

NuPlayerDecoder.cppThe processing of the message will go to call handleAnInputBufferand handleAnInputBuffer.

3. Modification of synchronization calibration:
The 5.1 version of the code will calibrate the synchronization time point before sending it to the display. As a result, when it is time to send it to the display, this value is not used, so it is not calibrated, but I see the logic on AndroidQ and change it. Now, I will calibrate it when I really want to send it to the display:

onDrainVideoQueue@NuPlayerRenderer.cpp：
void NuPlayer::Renderer::onDrainVideoQueue() {
    
    
	...
	/* 1.获得预估送显时间 */
	realTimeUs = getRealTimeUs(mediaTimeUs, nowUs);
	...
	/* 2.校准送显时间 */
	realTimeUs = mVideoScheduler->schedule(realTimeUs * 1000) / 1000;
	...
}

4. AndroidQ version logic:

There is no difference in the logic call graph except for the message mechanism used for the final processing of decoding and rendering;