iOS提供AVPictureInPictureController用于画中画播放控制，但是只能绑定AVPlayer使用，对于开发者来说挺头痛的。在iOS 15.0后呼之欲出，支持SampleBufferDisplayLayer自定义数据源显示图层，意味着我们可以自定义第三方播放器实现画中画。以ijkplayer为例，让我们一起探索如何支持画中画播放。
目录
一、判断画中画支持
我们可以通过isPictureInPictureSupported() 判断当前设备是否支持画中画。
二、后台播放配置
1、配置AudioSession
在application创建时，配置AudioSession为playback模式，代码如下：
func application(_ application: UIApplication,
didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {
let audioSession = AVAudioSession.sharedInstance()
do {
try audioSession.setCategory(.playback)
} catch {
print("Setting category to AVAudioSessionCategoryPlayback failed.")
}
return true
}2、配置后台模式
勾选“Audio, Airplay, Picture in Picture”选项，如下图所示：
三、画中画生命周期
1、从全屏进入画中画
启动画中画时，首先执行willStartPictureInPicture()方法，此时动画还未开始。等到初始化完毕，执行didStartPictureInPicture()方法，完成从全屏到画中画的切换，进入后台状态。
2、退出画中画回到全屏
退出画中画时，从后台状态恢复app的UI，执行willStopPictureInPicture()方法。最终完成退出，执行didStopPictureInPicture()方法，回到全屏播放。
3、生命周期
画中画完整生命周期包括：从全屏到画中画，最终退出画中画回到全屏。即上面两个生命周期的结合，如下图所示：
四、画中画创建流程
1、初始化
初始化AVSampleBufferDisplayLayer，包括设置frame、opaque、position、videoGravity、controlTimebase。示例代码如下：
- (AVSampleBufferDisplayLayer *) createDisplayLayer
{
AVSampleBufferDisplayLayer *layer;
layer = [[AVSampleBufferDisplayLayer alloc] init];
layer.frame = self.bounds;
layer.opaque = YES;
layer.position = CGPointMake(CGRectGetMidX(self.bounds), CGRectGetMidY(self.bounds));
layer.videoGravity = AVLayerVideoGravityResizeAspect;
CMTimebaseRef controlTimebase;
CMTimebaseCreateWithSourceClock( CFAllocatorGetDefault(), CMClockGetHostTimeClock(), &controlTimebase);
layer.controlTimebase = controlTimebase;
return layer;
}使用AVSampleBufferDisplayLayer来初始化ContentSource，前面先判断设备是否支持画中画。示例代码如下：
- (BOOL)initPictureInPicture:(AVSampleBufferDisplayLayer *)displayLayer {
if ([AVPictureInPictureController isPictureInPictureSupported] == NO) {
NSLog(@"Sorry, don't support PictureInPicture mode!");
return NO;
}
if (@available(iOS 15.0, *)) {
_displayLayer = displayLayer;
_contentSource = [[AVPictureInPictureControllerContentSource alloc]
initWithSampleBufferDisplayLayer:displayLayer playbackDelegate:self];
return YES;
} else {
return NO;
}
}2、提供start和stop方法
开始画中画，包括初始化AVPictureInPictureController、设置delegate代理、设置播放速率、启动。。示例代码如下：
- (void)startPictureInPicture {
if (@available(iOS 15.0, *)) {
if (_pipController != nil) {
return;
}
_pipController = [[AVPictureInPictureController alloc] initWithContentSource:_contentSource];
_pipController.delegate = self;
CMTimebaseSetRate(_displayLayer.controlTimebase, 1);
CMTimebaseSetTime(_displayLayer.controlTimebase, CMTimeMake([_delegate getCurrentTime], 1));
// 延时启动，否则可能会启动失败
dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)(100 * NSEC_PER_MSEC)), dispatch_get_main_queue(), ^{
if ([self->_pipController isPictureInPicturePossible]) {
[self->_pipController startPictureInPicture];
NSLog(@"startPictureInPicture...");
}
});
}
}结束画中画， 示例代码如下：
- (void)stopPictureInPicture {
if (_pipController != nil && [_pipController isPictureInPictureActive]) {
[_pipController stopPictureInPicture];
}
}3、实现画中画代理方法
实现AVPictureInPictureControllerDelegate代理方法如下：
- (void)pictureInPictureControllerWillStartPictureInPicture:(AVPictureInPictureController *)pictureInPictureController {
NSLog(@"pictureInPictureControllerWillStart...");
}
- (void)pictureInPictureControllerDidStartPictureInPicture:(AVPictureInPictureController *)pictureInPictureController {
// 回调画中画开始
[_delegate onStartPictureInPicture:nil];
}
- (void)pictureInPictureController:(AVPictureInPictureController *)pictureInPictureController
failedToStartPictureInPictureWithError:(NSError *)error {
// 画中画开始失败
[_delegate onStartPictureInPicture:error];
}
- (void)pictureInPictureControllerWillStopPictureInPicture:(AVPictureInPictureController *)pictureInPictureController {
NSLog(@"pictureInPictureControllerWillStop...");
}
- (void)pictureInPictureControllerDidStopPictureInPicture:(AVPictureInPictureController *)pictureInPictureController {
[self stopPictureInPicture];
// 回调画中画结束
[_delegate onStopPictureInPicture];
}
- (void)pictureInPictureController:(AVPictureInPictureController *)pictureInPictureController
restoreUserInterfaceForPictureInPictureStopWithCompletionHandler:(void (^)(BOOL restored))completionHandler {
completionHandler(true);
// 恢复全屏界面
}4、更新缓冲区图层
在ijk解码出来后，封装成CVPixelBuffer，然后封装CMSampleBufferRef，存入displayLayer。示例代码如下：
-(void) enqueueBuffer:(CVPixelBufferRef)pixelBuffer
{
if (!_displayLayer || !pixelBuffer)
return;
@autoreleasepool {
CMSampleBufferRef sampleBuffer = nil;
CMVideoFormatDescriptionRef format = nil;
// 设置dts、pts
CMSampleTimingInfo timeInfo = {.presentationTimeStamp = kCMTimeInvalid, .decodeTimeStamp = kCMTimeInvalid};
OSStatus status = CMVideoFormatDescriptionCreateForImageBuffer(kCFAllocatorDefault, pixelBuffer, &format);
status = CMSampleBufferCreateForImageBuffer(kCFAllocatorDefault, pixelBuffer, true, NULL, NULL,
format, &timeInfo, &sampleBuffer);
if (format != nil) {
CFRelease(format);
}
if (sampleBuffer == nil || status != noErr) {
NSLog(@"sampleBuffer error!");
return;
}
CFArrayRef attachments = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, YES);
CFMutableDictionaryRef dict = (CFMutableDictionaryRef)CFArrayGetValueAtIndex(attachments, 0);
// 设置立即渲染
CFDictionarySetValue(dict, kCMSampleAttachmentKey_DisplayImmediately, kCFBooleanTrue);
[_displayLayer enqueueSampleBuffer:sampleBuffer];
CFRelease(sampleBuffer);
if (_displayLayer.status == AVQueuedSampleBufferRenderingStatusFailed) {
NSLog(@"displayLayer error=%@", [_displayLayer error]);
}
}
}5、更新播放状态
实现AVPictureInPictureSampleBufferPlaybackDelegate代理方法，更新播放状态、seek进度。示例代码如下：
- (void)pictureInPictureController:(AVPictureInPictureController *)pictureInPictureController setPlaying:(BOOL)playing {
_isPlaybackPaused = !playing;
// 回调播放/暂停
[_delegate setPlaying:playing];
if (@available(iOS 15.0, *)) {
[pictureInPictureController invalidatePlaybackState];
}
if (playing == NO) {
_playClickTime = [[NSDate date] timeIntervalSince1970] * 1000;
}
}
- (CMTimeRange)pictureInPictureControllerTimeRangeForPlayback:(AVPictureInPictureController *)pictureInPictureController {
// 计算time range：0～duration
NSTimeInterval durationTime = [_delegate getDurationTime];
return CMTimeRangeMake(kCMTimeZero, CMTimeMake(durationTime * 1000, 1000));
}
- (BOOL)pictureInPictureControllerIsPlaybackPaused:(AVPictureInPictureController *)pictureInPictureController {
return _isPlaybackPaused;
}
- (void)pictureInPictureController:(AVPictureInPictureController *)pictureInPictureController
didTransitionToRenderSize:(CMVideoDimensions)newRenderSize {
}
- (void)pictureInPictureController:(AVPictureInPictureController *)pictureInPictureController skipByInterval:(CMTime)skipInterval
completionHandler:(void (^)(void))completionHandler {
// 回调seek
[_delegate seekTo:CMTimeGetSeconds(skipInterval)];
if (@available(iOS 15.0, *)) {
// seek开始会暂停播放，seek成功后恢复播放状态
if (([[NSDate date] timeIntervalSince1970] * 1000 - _playClickTime) < 200) {
[_delegate setPlaying:YES];
}
// 更新displaylayer的controlTimebase时钟，从而更新进度
CMTimebaseSetTime(_displayLayer.controlTimebase, CMTimeMake([_delegate getCurrentTime] * 1000, 1000));
// 更新播放状态
[_pipController invalidatePlaybackState];
}
}ijkplayer提供play、seek、getCurrentPosition、getDuration、isPlaying等接口，给自定义的画中画Controller调用：
- (BOOL)isPlaybackPaused
{
return [self isPlaying] == NO;
}
- (void)setPlaying:(BOOL)playing
{
if (playing) {
[self play];
} else {
[self pause];
}
}
- (void)seekTo:(NSTimeInterval)relativePosition
{
NSTimeInterval position = [self currentPlaybackTime] + relativePosition;
[self setCurrentPlaybackTime:position];
}
- (NSTimeInterval)getCurrentTime
{
return [self currentPlaybackTime];
}
- (NSTimeInterval)getDurationTime
{
return [self duration];
}6、避坑指南
(1) 延时启动：大概要延时100ms左右启动AVPictureInPictureController，否则可能启动画中画失败，有点坑爹
(2) 设置时钟：需要设置dts、pts，设置立即渲染。坑爹+1
(3) 更新播放状态：seek开始时会暂停播放，等到seek成功要恢复播放状态，坑爹+2