Problems encountered:
Use wireshark to analyze the successful and failed rtmp packets, and find that the first successful videoData has data, but the first failed videoData has no data.
success:
fail:
Is there a problem here?
During the study, refer to the following blogs:
Tear the details of the rtmp protocol by hand, this series is very good and basic
The rtmp protocol is an application layer protocol based on tcp, so there is a three-way handshake. After the tcp is established, the rtmp protocol level handshake is performed
The handshake process mainly completes two tasks, one is to verify the version of rtmp, and the other is to send some random data for network status detection
After the handshake of the Rtmp protocol is completed, data interaction can be carried out, but the exchanged data format requires an organization's standard. The sender assembles the data according to the standard, and the receiver disassembles the data according to the standard, so as to complete the communication.
-
rtmp header
-
rtmp body
Let's take a look at the createStream message first. The RTMP client sends this message to the server to create a logical channel for message communication. Audio, video, and metadata all interact through the data channel created by createStream, and releaseStream corresponds to createStream. Why do you sometimes come to releaseStream before createStream ? 这个问题也困扰了我，让我一度怀疑是否是这里的问题This is like in many of our service implementations, a stop is performed first, and then a start is performed. Because every time we start a new process, we cannot ensure whether the previous process is completed normally, whether there is an abnormal situation, whether the abnormal situation has been processed, etc., so, to do an operation similar to restoring the initial state, releaseStream is this effect.
Like the picture above, many rtmp packets are put together, you have to click to open to see which ones are there, and you can’t see createStream if you don’t click.
Here you can see that there is a publish, which is what we need to use to push the stream. If there is a process of pulling the stream, then there should be a play .
Introduction to video package in Rtmp
RTMP live streaming - short book (jianshu.com)
h264 manually add sps and pps to AVCodecContext->extradata
Big data] FFmepg AV_CODEC_FLAG_GLOBAL_HEADER problem description
So I searched in these articles, and found the same problem in this article RTMP live streaming - Jianshu (jianshu.com) ! ! !
原来在调用avformat_write_header之前，就要把sps跟pps传入
The first is the introduction of sps and pps:
Sequence Parameter Sets (SPS) 和Picture Parameter Set (PPS)
For H264, SPS is the first frame after encoding . If it is a read H264 file, the length of the data between the first frame delimiter and the second frame delimiter is 4.
PPS is the second frame after encoding . If it is a read H264 file, the data length between the second frame delimiter and the third frame delimiter is not fixed.
In the header of nalu, it is stipulated that 67 is followed by sps, and 68 is followed by pps. I won’t say more about the specific values of height and width. You can find relevant content on the Internet. If you know more about this, you can Direct handwriting sps_pps array ( strongly not recommended )
The AMF encoding format is an encoding format of Adobe
Then there's the symptom of the problem:
In the player log of station b, a video track parse error appeared at the beginning and firstFrame timeout appeared later. From this, it can be judged that there should be a problem with the first videoData, and there is no information in the player information, indicating that The player does not get the video data it should have.
[External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly
After comparing with Douyu and Huya, I found that the phenomenon of being unable to play is also related to the player. Some players do not choose, such as Douyu’s player. It doesn’t matter whether your first player has sps and pps information. Strange, and even stranger is Huya, which receives the stream and keeps requesting data in the network, but! It can't be released, unlike station b, the player will repeat the request and cross-domain request if the player parses the error.
Big data] FFmepg AV_CODEC_FLAG_GLOBAL_HEADER problem description , this article said that by default, AVCodecContext extradata
will automatically fill SPS and PPS into the extradata string when detecting the code stream (avformat_find_stream_info), and pass the codecpar content to AVCodecContext through the avcodec_parameters_to_context function , which may also be an approach , but it does not apply to our scenario.
(119 messages) AVC sequence header & AAC sequence header_You Yu Yu You-CSDN Blog
It can also be seen from the above article that the first video frame, which is the AVC sequence header, is very important, and it is a necessary part of client-side decoding.
This is the reason why the player at station b cannot decode it.
https://blog.csdn.net/ai2000ai/article/details/85114110
The above article said that extradata can be updated through side_data, and he also tried to modify and update AVCodecContext without success. can try
上面这个方案不行，修改side_data是修改的packet的数据，我们这个是在pollAVPacket之前进行的操作，是AVFormat_write_header的操作，这个方法适用于中途修改sps和pps设置。
Try to write sps_pps information into stream->codecpar->extraData:
Writing sps_pps information into extraData before avFormat_write_header is still not possible. Although there is data in the body of videoData, some information is missing, and handwriting sps_pps is unrealistic, and it can only be used for testing without portability.
Missing information:
When it is also mentioned in this article that it is AVC, the subsequent data format is | AVCPacketType(8)| CompostionTime(24) | Data |
AVCPacketType is 00, the data type is AVCSequence Header,
AVCPacketType is 01, the data type is AVC NALU
So I know that the four 00s after 17 are AVCPacketType and CompostionTime, and I also find the explanation of the remaining bytes from the source code in ffmpeg.
Link: https://www.jianshu.com/p/e90184fe94f9
Because it is necessary to fill in sps and pps information before AVFormat_write_header and before sending rtmp packets, so the following attempts have been made:
- First modifying the discovery on the stream
flags |= AV_CODEC_FLAG_GLOBAL_HEADER;is useless; - Then manually writing sps_pps is not a good way, because there will be some information in front of sps and pps as mentioned above;
- Finally, by adding ctx to the encoder when the encoder is initialized
flags |= AV_CODEC_FLAG_GLOBAL_HEADER;, we can obtain the sps_pps information of the encoder when no data enters the encoder (that is, after initialization) ;
However, after setting this parameter, it will affect the subsequent data, that is, the subsequent I frames will no longer have sps and pps information, because the flv format only needs to use the sps and pps information of the first Video Tag to generate AVCC (AVCDecoderConfigurationRecord) , so the following I frame has no sps and pps has no effect on it, and the problem is solved.
Summarize:
To analyze problems, we should start from the basics, especially for some underlying knowledge or communication-related knowledge. You can read official documents about protocols or specifications . Secondly, you can also read some blogs or articles that analyze them . By analyzing the process Sort out to find the problem, which is a positioning problem.
Then to solve the problem, it is necessary to clarify the thinking. The important point for this problem is how to make the first videoData have sps and pps data, and can automatically obtain the correct data. In this case, it should be from At first , although I learned that it is necessary to set flags for AVCodecContext, if you are not familiar with the process, you may not be able to find the appropriate location to set it. Study the source code and get familiar with the process.