Audio and video synchronization problem (first bullet)

Foreword:

Hello everyone, because I have encountered some problems about audio and video synchronization in my work recently, such as the audio cannot keep up with the playback speed of the video; based on this, today I will share with you an issue about audio and video synchronization. This series of articles will be divided into A few articles to share, starting with the basic theory of audio and video synchronization, and then based on the actual source code in ffplay to deepen the understanding of audio and video synchronization!

1. Audio and video synchronization basis:

First of all, let's first understand what is audio and video synchronization. I believe that friends who have read the previous articles about ffplay source code analysis should know that in the process of decoding and getting data from a player, audio and video have their own Threads process their corresponding data!

Then there may be problems in this link. For example, the audio thread and the video thread may not decode the audio, video and video frames at the same time, then this will cause different problems in subsequent playback! There will also be problems with different audio and video. To put it bluntly, audio and video synchronization is like we usually use a player to play a video file (including audio and video!), and there will be video images and voices of characters speaking in the video images. Failure to do so will result in a very poor experience.

Therefore, in order to better solve practical development problems in the future, we need to master some basic theoretical knowledge before that. Let’s take a look at what PTS, DTS, and timebase are. Let’s talk about theoretical concepts first, and then understand them through actual combat later. At present, you only need to know that there are these three things:

• DTS (Decoding Time Stamp), which is the decoding timestamp, will tell the player when to decode the data of this frame.

• PTS (Presentation Time Stamp), which is to display the timestamp, then it will also tell the player when to display the data of this frame.

• timebase time base: the real time unit of the PTS value, usually seconds; at the same time, we should note that in ffplay, the data type of the PTS value is double. For example, I define an audio PTS variable and initialize it below:

double audio_pts = 0.020;

At the same time, in ffplay, the type of timebase is the structure AVRational (one of the structure members represents the numerator, and the other represents the denominator!), the specific code is as follows:

typedef struct AVRational{
  int num;
  int den;
}AVRational;

So how to calculate the display time of PTS through this?

For example, timebase={1,100} means 100 milliseconds, if pts=1000, then 1000*1/100=10 seconds, which means that this frame of data needs to be displayed at the 10th second!

Therefore, in ffmpeg, there is an interface implementation about converting AVRational to double type. It is very convenient for us to use this interface to calculate the display time of pts:

/**
 * Convert an AVRational to a `double`.
 * @param a AVRational to convert
 * @return `a` in floating-point form
 * @see av_d2q()
 */
static inline double av_q2d(AVRational a){
    return a.num / (double) a.den;
}

So the time calculation to display the timestamp is equal to:

pts * av_q2d(参数)；

2. Commonly used audio and video synchronization strategy theoretical solutions:

There are generally three solutions:

• (1): Based on the audio, synchronize the video to the audio; if the video is slow, discard some video frames (frame skipping will occur!); if the video is fast, continue to render the previous frame.

• (2) Based on the video, synchronize the audio to the video; if the audio is slow, speed up the playback speed, or discard some video frames, but there will be a staccato; if the audio is slow, slow down the playback speed, or repeat previous frame. But changing the playback speed here will involve resampling!

• (3) Based on the external clock, synchronize audio and video to the external clock

3. Summary:

Today's content is simply shared here. It is more about how to quickly solve this problem if we encounter similar problems in the actual development process is the key. This also requires our usual basic accumulation!

In the next issue, let's continue!