最近输出的内容变少了，一是工作内容确实不少，二是生活方面。但都不应该是放弃学习的原因。废话不说本篇承接上篇内容，分析AndroidVideoDecoder & HardwareVideoEncoder
一、前提回顾
稍微回顾一下之前的内容，以解码DefaultVideoDeocderFactory，createDecoder是由HardwareVideoDecoderFactory 和 SoftwareVideoDecoderFactory各自创建出来，然后回传到PeerConnectionClient。
@Override
public @Nullable
VideoDecoder createDecoder(VideoCodecInfo codecType) {
VideoDecoder softwareDecoder = softwareVideoDecoderFactory.createDecoder(codecType);
final VideoDecoder hardwareDecoder = hardwareVideoDecoderFactory.createDecoder(codecType);
if (softwareDecoder == null && platformSoftwareVideoDecoderFactory != null) {
softwareDecoder = platformSoftwareVideoDecoderFactory.createDecoder(codecType);
}
if (hardwareDecoder != null && softwareDecoder != null) {
// Both hardware and software supported, wrap it in a software fallback
return new VideoDecoderFallback(
/* fallback= */ softwareDecoder, /* primary= */ hardwareDecoder);
}
return hardwareDecoder != null ? hardwareDecoder : softwareDecoder;
}对于HardwareVideoDecoderFactory createDecoder创建硬解器，具体如下：
@Nullable
@Override
public VideoDecoder createDecoder(VideoCodecInfo codecType) {
VideoCodecMimeType type = VideoCodecMimeType.valueOf(codecType.getName());
MediaCodecInfo info = findCodecForType(type);
if (info == null) {
return null;
}
CodecCapabilities capabilities = info.getCapabilitiesForType(type.mimeType());
return new AndroidVideoDecoder(new MediaCodecWrapperFactoryImpl(), info.getName(), type,
MediaCodecUtils.selectColorFormat(MediaCodecUtils.DECODER_COLOR_FORMATS, capabilities),
sharedContext);
}对是的，今天就是来看看这个AndroidVideoDecoder有什么值得去挖掘学习的。
二、AndroidVideoDecoder
看Android开源项目，建议大家善用AS窗口左下角的Structure标签，它会展现当前类的所有成员变量，方法，能給阅读者一个总体概述。
我们直接来聊聊关键点，initDeocde -> SurfaceTextureHelper
@Override
public VideoCodecStatus initDecode(Settings settings, Callback callback) {
this.decoderThreadChecker = new ThreadChecker();
this.callback = callback;
if (sharedContext != null) {
surfaceTextureHelper = createSurfaceTextureHelper();
surface = new Surface(surfaceTextureHelper.getSurfaceTexture());
surfaceTextureHelper.startListening(this);
}
return initDecodeInternal(settings.width, settings.height);
}private SurfaceTextureHelper(Context sharedContext, Handler handler, boolean alignTimestamps,
YuvConverter yuvConverter, FrameRefMonitor frameRefMonitor) {
if (handler.getLooper().getThread() != Thread.currentThread()) {
throw new IllegalStateException("SurfaceTextureHelper must be created on the handler thread");
}
this.handler = handler;
this.timestampAligner = alignTimestamps ? new TimestampAligner() : null;
this.yuvConverter = yuvConverter;
this.frameRefMonitor = frameRefMonitor;
// 1
eglBase = EglBase.create(sharedContext, EglBase.CONFIG_PIXEL_BUFFER);
try {
// Both these statements have been observed to fail on rare occasions, see BUG=webrtc:5682.
eglBase.createDummyPbufferSurface();
eglBase.makeCurrent();
} catch (RuntimeException e) {
// Clean up before rethrowing the exception.
eglBase.release();
handler.getLooper().quit();
throw e;
}
// 2
oesTextureId = GlUtil.generateTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES);
surfaceTexture = new SurfaceTexture(oesTextureId);
setOnFrameAvailableListener(surfaceTexture, (SurfaceTexture st) -> {
if (hasPendingTexture) {
Logging.d(TAG, "A frame is already pending, dropping frame.");
}
hasPendingTexture = true;
tryDeliverTextureFrame();
}, handler);
}1、EglBase.createDummyPbufferSurface 创建了 Pbuffer，有关pbuffer的知识点可以看我以前的相关文章。理解pbuffer和fbo的区别，我想这个问题已经很多知识博主说得很明白，然后面试的时候会经常问pbuffer能代替fbo吗？哪些情况用pbuffer？这里給出几年前给网友解疑的回答：
pbuffer适合大规模多GPU计算的工程，在移动端fbo还真的能替代pbuffer绝大多数的使用场景。
2、SurfaceTexture.FrameAvailableListener，SurfaceTexture的工作方式很多同学都知道，我就不展开阐述了。那这里我想说啥呢，就是很多同学问到的SurfaceTexture如何高效提取帧数据？
@Override
public VideoCodecStatus initDecode(Settings settings, Callback callback) {
... ...
return initDecodeInternal(settings.width, settings.height);
}
private VideoCodecStatus initDecodeInternal(int width, int height) {
codec = mediaCodecWrapperFactory.createByCodecName(codecName);
codec.start();
... ...
outputThread = createOutputThread();
outputThread.start();
... ...
return VideoCodecStatus.OK;
}
private Thread createOutputThread() {
return new Thread("AndroidVideoDecoder.outputThread") {
@Override
public void run() {
outputThreadChecker = new ThreadChecker();
while (running) {
deliverDecodedFrame();
}
releaseCodecOnOutputThread();
}
};
}这里车速有点快，请大家坐稳跟上节奏。回到AndroidVideoDecoder的 initDecoder -> initDecodeInternal -> createOutputThread。意思很明显，就是一个线程不断的提取传递解码后的数据，跟进deliverDecodedFrame()
protected void deliverDecodedFrame() {
... ...
MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
int result = codec.dequeueOutputBuffer(info, DEQUEUE_OUTPUT_BUFFER_TIMEOUT_US);
... ...
if (surfaceTextureHelper != null) {
deliverTextureFrame(result, info, rotation, decodeTimeMs);
} else {
deliverByteFrame(result, info, rotation, decodeTimeMs);
}
}
private void deliverTextureFrame(final int index, final MediaCodec.BufferInfo info,
final int rotation, final Integer decodeTimeMs) {
... ...
synchronized (renderedTextureMetadataLock) {
if (renderedTextureMetadata != null) {
codec.releaseOutputBuffer(index, false);
return; // still waiting for previous frame, drop this one.
}
surfaceTextureHelper.setTextureSize(width, height);
surfaceTextureHelper.setFrameRotation(rotation);
renderedTextureMetadata = new DecodedTextureMetadata(info.presentationTimeUs, decodeTimeMs);
codec.releaseOutputBuffer(index, /* render= */ true);
}
}
@Override
public void onFrame(VideoFrame frame) {
final VideoFrame newFrame;
final Integer decodeTimeMs;
final long timestampNs;
synchronized (renderedTextureMetadataLock) {
if (renderedTextureMetadata == null) {
throw new IllegalStateException(
"Rendered texture metadata was null in onTextureFrameAvailable.");
}
timestampNs = renderedTextureMetadata.presentationTimestampUs * 1000;
decodeTimeMs = renderedTextureMetadata.decodeTimeMs;
renderedTextureMetadata = null;
}
// Change timestamp of frame.
final VideoFrame frameWithModifiedTimeStamp =
new VideoFrame(frame.getBuffer(), frame.getRotation(), timestampNs);
callback.onDecodedFrame(frameWithModifiedTimeStamp, decodeTimeMs, null /* qp */);
}deliverTextureFrame非真的deliver Texture。关键是在 renderedTextureMetadata，看到在onFrame，把回调出来的VideoFrame替换renderedTextureMetadata的pts，最后才回调上去。那么这个onFrame是哪里的回调接口，就是SurfaceTextureHelper里面的SurfaceTexture.FrameAvailableListener抛上来的。
private void tryDeliverTextureFrame() {
updateTexImage();
final float[] transformMatrix = new float[16];
surfaceTexture.getTransformMatrix(transformMatrix);
long timestampNs = surfaceTexture.getTimestamp();
final TextureBuffer buffer =
new TextureBufferImpl(textureWidth, textureHeight, TextureBuffer.Type.OES, oesTextureId,
RendererCommon.convertMatrixToAndroidGraphicsMatrix(transformMatrix), handler,
yuvConverter, textureRefCountMonitor);
final VideoFrame frame = new VideoFrame(buffer, frameRotation, timestampNs);
listener.onFrame(frame);
frame.release();
}在tryDeliverTextureFrame函数当中，看到把 oesTextureId 封装成 TextureBuffer，再封装成VideoFrame回调給AndroidVideoDeocder，可以看到关键类 YuvConverter，显然这就是把Texture转换成yuv数据的辅助类。
接下来我就不装*了，直接看代码，挖掘有价值的知识。
// TextureBufferImpl
@Override
public VideoFrame.I420Buffer toI420() {
return ThreadUtils.invokeAtFrontUninterruptibly(
toI420Handler, () -> yuvConverter.convert(this));
}
// ThreadUtils
public static <V> V invokeAtFrontUninterruptibly(
final Handler handler, final Callable<V> callable) {
if (handler.getLooper().getThread() == Thread.currentThread()) {
try {
return callable.call();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
final Result result = new Result();
final CaughtException caughtException = new CaughtException();
final CountDownLatch barrier = new CountDownLatch(1);
handler.post(new Runnable() {
@Override
public void run() {
try {
result.value = callable.call();
} catch (Exception e) {
caughtException.e = e;
}
barrier.countDown();
}
});
awaitUninterruptibly(barrier);
if (caughtException.e != null) {
final RuntimeException runtimeException = new RuntimeException(caughtException.e);
runtimeException.setStackTrace(
concatStackTraces(caughtException.e.getStackTrace(), runtimeException.getStackTrace()));
throw runtimeException;
}
return result.value;
}2.1、学习过Kotlin的同学，或多或少都涉及过suspend修饰词——协程，简单理解就是业务代码执行流程在同步不阻塞的情况下，代码片A执行在指定线程A，代码片B执行在指定线程B，确保业务流程。那么在Java版本有无这样的舒爽的用法？肯定是有的！版本答案就在上面，其实就是利用了Java的CountDownLatch同步计数，awaitUninterruptibly(CountDownLatch)里在调用线程上循环等待，在返回执行结果，值得同学细品，并运用到实际项目当中。
2.2、打开YuvConveter，快速定位到convert方法，方法备注已经写得明明白白/*Converts the texture buffer to I420.*/，继续分析是如何转换的，代码分两部分理解：①texture按照yuv的格式draw到fbo上；②把fbo的内容提取出来；
三、Texture Conveter YUV420
public I420Buffer convert(TextureBuffer inputTextureBuffer) {
TextureBuffer preparedBuffer = (TextureBuffer) videoFrameDrawer.prepareBufferForViewportSize(
inputTextureBuffer, inputTextureBuffer.getWidth(), inputTextureBuffer.getHeight());
// We draw into a buffer laid out like
// +---------+
// | |
// | Y |
// | |
// | |
// +----+----+
// | U | V |
// | | |
// +----+----+
// In memory, we use the same stride for all of Y, U and V. The
// U data starts at offset |height| * |stride| from the Y data,
// and the V data starts at at offset |stride/2| from the U
// data, with rows of U and V data alternating.
//
// Now, it would have made sense to allocate a pixel buffer with
// a single byte per pixel (EGL10.EGL_COLOR_BUFFER_TYPE,
// EGL10.EGL_LUMINANCE_BUFFER,), but that seems to be
// unsupported by devices. So do the following hack: Allocate an
// RGBA buffer, of width |stride|/4. To render each of these
// large pixels, sample the texture at 4 different x coordinates
// and store the results in the four components.
//
// Since the V data needs to start on a boundary of such a
// larger pixel, it is not sufficient that |stride| is even, it
// has to be a multiple of 8 pixels.
// Note1
final int frameWidth = preparedBuffer.getWidth();
final int frameHeight = preparedBuffer.getHeight();
final int stride = ((frameWidth + 7) / 8) * 8;
final int uvHeight = (frameHeight + 1) / 2;
// Total height of the combined memory layout.
final int totalHeight = frameHeight + uvHeight;
// Viewport width is divided by four since we are squeezing in four color bytes in each RGBA pixel.
final int viewportWidth = stride / 4;
// Note2
// Produce a frame buffer starting at top-left corner, not bottom-left.
final Matrix renderMatrix = new Matrix();
renderMatrix.preTranslate(0.5f, 0.5f);
renderMatrix.preScale(1f, -1f);
renderMatrix.preTranslate(-0.5f, -0.5f);
// Note3
i420TextureFrameBuffer.setSize(viewportWidth, totalHeight);
// Bind our framebuffer.
GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, i420TextureFrameBuffer.getFrameBufferId());
GlUtil.checkNoGLES2Error("glBindFramebuffer");
// Note4
// Draw Y.
shaderCallbacks.setPlaneY();
VideoFrameDrawer.drawTexture(drawer, preparedBuffer, renderMatrix, frameWidth, frameHeight,
/* viewportX= */ 0, /* viewportY= */ 0, viewportWidth,
/* viewportHeight= */ frameHeight);
// Draw U.
shaderCallbacks.setPlaneU();
VideoFrameDrawer.drawTexture(drawer, preparedBuffer, renderMatrix, frameWidth, frameHeight,
/* viewportX= */ 0, /* viewportY= */ frameHeight, viewportWidth / 2,
/* viewportHeight= */ uvHeight);
// Draw V.
shaderCallbacks.setPlaneV();
VideoFrameDrawer.drawTexture(drawer, preparedBuffer, renderMatrix, frameWidth, frameHeight,
/* viewportX= */ viewportWidth / 2, /* viewportY= */ frameHeight, viewportWidth / 2,
/* viewportHeight= */ uvHeight);
GLES20.glReadPixels(0, 0, i420TextureFrameBuffer.getWidth(), i420TextureFrameBuffer.getHeight(),
GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE, i420ByteBuffer);
GlUtil.checkNoGLES2Error("YuvConverter.convert");
// Restore normal framebuffer.
GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, 0);
... ...
}Note1：上面的代码对应第一部分，开头的部分注解主要是说明8字节内存对齐，YUV420格式的数据存储到一整块pixel（RGBA）buffer上。
Note2：之后准备了一个renderMatrix，执行了三个左乘矩阵运算，最终的renderMatrix = 单位矩阵MIdentity*T*S*-T，关于左乘和右乘的区别，请看这里面的矩阵运算。这里的操作有点类似8字节内存对齐的套路，往下看就知道了。
Note3：不要看名字是i420TextureFrameBuffer，其实它格式是为GLES20.GL_RGBA的FrameBuffer，宽度是stride / 4，即内存对齐后的frameWidth，高度是frameHeight + uvHeight。这其实就是传统意义的yuv-size = width * height * 3 / 2的大小了。
private final GlTextureFrameBuffer i420TextureFrameBuffer =
new GlTextureFrameBuffer(GLES20.GL_RGBA);
final int frameWidth = preparedBuffer.getWidth();
final int frameHeight = preparedBuffer.getHeight();
final int stride = ((frameWidth + 7) / 8) * 8;
final int uvHeight = (frameHeight + 1) / 2;
final int viewportWidth = stride / 4;
final int totalHeight = frameHeight + uvHeight;
i420TextureFrameBuffer.setSize(viewportWidth, totalHeight);
Note4：往下走想看懂DrawY，DrawU，DrawV是什么骚操作，得先知道这里的ShaderCallbacks、GlGenericDrawer和VideoFrameDrawer，三者是如何关联工作的。
private final ShaderCallbacks shaderCallbacks = new ShaderCallbacks();
private final GlGenericDrawer drawer = new GlGenericDrawer(FRAGMENT_SHADER, shaderCallbacks);
public GlGenericDrawer(String genericFragmentSource, ShaderCallbacks shaderCallbacks) {
this(DEFAULT_VERTEX_SHADER_STRING, genericFragmentSource, shaderCallbacks);
}GlGenericDrawer就是Shader着色器的封装对象，然后呢这里的代码我觉得有点绕，搞不到为啥要这样写，得慢慢的一步步分析。
Note4.1：分析GlGenericDrawer构造传入的DEFAULT_VERTEX_SHADER_STRING 和 FRAGMENT_SHADER，注意这里不是全部的shader内容，往下看流程VideoFrameDrawer.drawTexture -> GlGenericDrawer.drawOes -> prepareShader->createShader
/**
* Draw an OES texture frame with specified texture transformation matrix. Required resources are
* allocated at the first call to this function.
*/
@Override
public void drawOes(int oesTextureId, float[] texMatrix, int frameWidth, int frameHeight,
int viewportX, int viewportY, int viewportWidth, int viewportHeight) {
prepareShader(
ShaderType.OES, texMatrix, frameWidth, frameHeight, viewportWidth, viewportHeight);
// Bind the texture.
GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, oesTextureId);
// Draw the texture.
GLES20.glViewport(viewportX, viewportY, viewportWidth, viewportHeight);
GLES20.glDrawArrays(GLES20.GL_TRIANGLE_STRIP, 0, 4);
// Unbind the texture as a precaution.
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0);
}
// Visible for testing.
GlShader createShader(ShaderType shaderType) {
return new GlShader(
vertexShader, createFragmentShaderString(genericFragmentSource, shaderType));
}
static String createFragmentShaderString(String genericFragmentSource, ShaderType shaderType) {
final StringBuilder stringBuilder = new StringBuilder();
if (shaderType == ShaderType.OES) {
stringBuilder.append("#extension GL_OES_EGL_image_external : require\n");
}
stringBuilder.append("precision mediump float;\n");
stringBuilder.append("varying vec2 tc;\n");
if (shaderType == ShaderType.YUV) {
stringBuilder.append("uniform sampler2D y_tex;\n");
stringBuilder.append("uniform sampler2D u_tex;\n");
stringBuilder.append("uniform sampler2D v_tex;\n");
// Add separate function for sampling texture.
// yuv_to_rgb_mat is inverse of the matrix defined in YuvConverter.
stringBuilder.append("vec4 sample(vec2 p) {\n");
stringBuilder.append(" float y = texture2D(y_tex, p).r * 1.16438;\n");
stringBuilder.append(" float u = texture2D(u_tex, p).r;\n");
stringBuilder.append(" float v = texture2D(v_tex, p).r;\n");
stringBuilder.append(" return vec4(y + 1.59603 * v - 0.874202,\n");
stringBuilder.append(" y - 0.391762 * u - 0.812968 * v + 0.531668,\n");
stringBuilder.append(" y + 2.01723 * u - 1.08563, 1);\n");
stringBuilder.append("}\n");
stringBuilder.append(genericFragmentSource);
} else {
final String samplerName = shaderType == ShaderType.OES ? "samplerExternalOES" : "sampler2D";
stringBuilder.append("uniform ").append(samplerName).append(" tex;\n");
// Update the sampling function in-place.
stringBuilder.append(genericFragmentSource.replace("sample(", "texture2D(tex, "));
}
return stringBuilder.toString();
}代码在createFragmentShaderString中追加了一些内容，所以完整的shader内容如下所示：
/* DEFAULT_VERTEX_SHADER_STRING */
varying vec2 tc;
attribute vec4 in_pos;
attribute vec4 in_tc;
uniform mat4 tex_mat;
void main() {
gl_Position = in_pos;
tc = (tex_mat * in_tc).xy;
}
/* FRAGMENT_SHADER */
#extension GL_OES_EGL_image_external : require
precision mediump float;
varying vec2 tc;
uniform samplerExternalOES tex;
// Difference in texture coordinate corresponding to one
// sub-pixel in the x direction.
uniform vec2 xUnit;
// Color conversion coefficients, including constant term
uniform vec4 coeffs;
void main() {
gl_FragColor.r = coeffs.a + dot(coeffs.rgb,
texture2D(tex, tc - 1.5 * xUnit).rgb);
gl_FragColor.g = coeffs.a + dot(coeffs.rgb,
texture2D(tex, tc - 0.5 * xUnit).rgb);
gl_FragColor.b = coeffs.a + dot(coeffs.rgb,
texture2D(tex, tc + 0.5 * xUnit).rgb);
gl_FragColor.a = coeffs.a + dot(coeffs.rgb,
texture2D(tex, tc + 1.5 * xUnit).rgb);
}
Note4.2：shader准备好之后，看绘制流程，既是DrawY，DrawU，DrawV，先看ShaderCallbacks代码，继续分析。
private static class ShaderCallbacks implements GlGenericDrawer.ShaderCallbacks {
// Y'UV444 to RGB888, see https://en.wikipedia.org/wiki/YUV#Y%E2%80%B2UV444_to_RGB888_conversion
// We use the ITU-R BT.601 coefficients for Y, U and V.
// The values in Wikipedia are inaccurate, the accurate values derived from the spec are:
// Y = 0.299 * R + 0.587 * G + 0.114 * B
// U = -0.168736 * R - 0.331264 * G + 0.5 * B + 0.5
// V = 0.5 * R - 0.418688 * G - 0.0813124 * B + 0.5
// To map the Y-values to range [16-235] and U- and V-values to range [16-240], the matrix has
// been multiplied with matrix:
// {
{219 / 255, 0, 0, 16 / 255},
// {0, 224 / 255, 0, 16 / 255},
// {0, 0, 224 / 255, 16 / 255},
// {0, 0, 0, 1}}
private static final float[] yCoeffs =
new float[]{0.256788f, 0.504129f, 0.0979059f, 0.0627451f};
private static final float[] uCoeffs =
new float[]{-0.148223f, -0.290993f, 0.439216f, 0.501961f};
private static final float[] vCoeffs =
new float[]{0.439216f, -0.367788f, -0.0714274f, 0.501961f};
public void setPlaneY() {
coeffs = yCoeffs;
stepSize = 1.0f;
}
public void setPlaneU() {
coeffs = uCoeffs;
stepSize = 2.0f;
}
public void setPlaneV() {
coeffs = vCoeffs;
stepSize = 2.0f;
}
@Override
public void onNewShader(GlShader shader) {
xUnitLoc = shader.getUniformLocation("xUnit");
coeffsLoc = shader.getUniformLocation("coeffs");
}
@Override
public void onPrepareShader(GlShader shader, float[] texMatrix, int frameWidth, int frameHeight,
int viewportWidth, int viewportHeight) {
GLES20.glUniform4fv(coeffsLoc, /* count= */ 1, coeffs, /* offset= */ 0);
// Matrix * (1;0;0;0) / (width / stepSize). Note that OpenGL uses column major order.
GLES20.glUniform2f(
xUnitLoc, stepSize * texMatrix[0] / frameWidth, stepSize * texMatrix[1] / frameWidth);
}
}
看备注可以了解大致思路：Wiki上的推导说是有误的，正确的RGB转YUV的公式如备注所示，
Y = 0.299 * R + 0.587 * G + 0.114 * B
U = -0.168736 * R - 0.331264 * G + 0.5 * B + 0.5
V = 0.5 * R - 0.418688 * G - 0.0813124 * B + 0.5
To map the Y-values to range [16-235] and U- and V-values to range [16-240], the matrix has been multiplied with matrix:
{
{219 / 255, 0, 0, 16 / 255},
{0, 224 / 255, 0, 16 / 255},
{0, 0, 224 / 255, 16 / 255},
{0, 0, 0, 1}}
接着说了一个转换关系，其实所表达的内容是，YUV的 Full range 转 TV range的过程。至于什么是Full range 和 TV range。这里简单介绍，详细请到这里。（这又可以引出说说BT601、BT709、BT2020的区别）
YUV 有多种表现形式
除了色彩空间， 还需要注意YUV 的多种表现形式， 比如：
YUV ： YUV是一种模拟型号， Y∈ [0,1] U,V∈[-0.5,0.5]
YCbCr ：也叫YCC或者Y'CbCr YCbCr 是数字信号， 它包含两种形式， 分别为TV range 和 full range， TV range 主要是广播电视采用的标准， full range 主要是pc 端采用的标准， 所以full range 有时也叫 pc range
TV range 的各个分量的范围为： YUV Y∈[16,235] Cb∈[16-240] Cr∈[16-240]
Full range 的各个分量的范围均为： 0-255
我们平时接触到的绝大多数都是 YCbCr （tv range） ， ffmpeg 解码出来的数据绝大多数也是这个， 虽然ffmpeg 里面将它的格式描述成YUV420P ， 实际上它是YCbCr420p tv range
YUV转tv range: Y' = 219.0*Y + 16 ; Cb = U * 224.0 + 128; Cr = V * 224.0 + 128;
最终yCoeffs、uCoeffs、vCoeffs（矩阵非齐次运算） =
最终结果 yCoeffs = {0.299, 0.587, 0.114} * (219 / 255) = {0.256788f, 0.504129f, 0.0979059f, 0.0627451f} uCoeffs、vCoeffs同理。
Note4.3：解析完coffs，再说说这个xUnit（setpSize），单纯从shader的内容上，能在第一时间能理解这个xUnit的作用是很难的。结合之前的renderMatrix 加以理解：
// Produce a frame buffer starting at top-left corner, not bottom-left.
final Matrix renderMatrix = new Matrix();
renderMatrix.preTranslate(0.5f, 0.5f);
renderMatrix.preScale(1f, -1f);
renderMatrix.preTranslate(-0.5f, -0.5f);
shaderCallbacks.setPlaneY();
VideoFrameDrawer.drawTexture(drawer, preparedBuffer, renderMatrix, frameWidth, frameHeight,
/* viewportX= */ 0, /* viewportY= */ 0, viewportWidth,
/* viewportHeight= */ frameHeight);
public static void drawTexture(RendererCommon.GlDrawer drawer, VideoFrame.TextureBuffer buffer,
Matrix renderMatrix, int frameWidth, int frameHeight, int viewportX, int viewportY,
int viewportWidth, int viewportHeight) {
Matrix finalMatrix = new Matrix(buffer.getTransformMatrix());
finalMatrix.preConcat(renderMatrix);
float[] finalGlMatrix = RendererCommon.convertMatrixFromAndroidGraphicsMatrix(finalMatrix);
switch (buffer.getType()) {
case OES:
drawer.drawOes(buffer.getTextureId(), finalGlMatrix, frameWidth, frameHeight, viewportX,
viewportY, viewportWidth, viewportHeight);
break;
case RGB:
drawer.drawRgb(buffer.getTextureId(), finalGlMatrix, frameWidth, frameHeight, viewportX,
viewportY, viewportWidth, viewportHeight);
break;
default:
throw new RuntimeException("Unknown texture type.");
}
}
// GlGenericDrawer.drawOes.prepareShader会触发ShaderCallbacks.onPrepareShader
@Override
public void onPrepareShader(GlShader shader, float[] texMatrix, int frameWidth, int frameHeight,
int viewportWidth, int viewportHeight) {
GLES20.glUniform4fv(coeffsLoc, /* count= */ 1, coeffs, /* offset= */ 0);
// Matrix * (1;0;0;0) / (width / stepSize). Note that OpenGL uses column major order.
GLES20.glUniform2f(
xUnitLoc, stepSize * texMatrix[0] / frameWidth, stepSize * texMatrix[1] / frameWidth);
}代码流程脉络如上：显然这个setpSize与renderMartix有莫大的关系，其中renderMartix有一句备注// Produce a frame buffer starting at top-left corner, not bottom-left. 意思是“从左上角而不是左下角开始生成帧缓冲区”，再结合GlGenericDrawer传入的最原始的纹理坐标FULL_RECTANGLE_TEXTURE_BUFFER
// Texture coordinates - (0, 0) is bottom-left and (1, 1) is top-right.
private static final FloatBuffer FULL_RECTANGLE_TEXTURE_BUFFER =
GlUtil.createFloatBuffer(new float[]{
0.0f, 0.0f, // Bottom left.
1.0f, 0.0f, // Bottom right.
0.0f, 1.0f, // Top left.
1.0f, 1.0f, // Top right.
});由于篇幅的关系，（涉及矩阵乘法原理请到这里）这里直接给出运算过程和结果：
//preTranslate的translate矩阵具体值，通过Androidrexf.com查找到的源码
static float[] setTranslate(float[] dest, float dx, float dy) {
dest[0] = 1;
dest[1] = 0;
dest[2] = dx;
dest[3] = 0;
dest[4] = 1;
dest[5] = dy;
dest[6] = 0;
dest[7] = 0;
dest[8] = 1;
return dest;
}
//preScale的scale矩阵具体值，通过Androidrexf.com查找到的源码
static float[] getScale(float sx, float sy) {
return new float[] { sx, 0, 0, 0, sy, 0, 0, 0, 1 };
}
0.0f, 0.0f, 0.0 1, 0, 0.5 0, 0, 0
1.0f, 0.0f, 0.0 preTranslate(0.5f, 0.5f) 即 0, 1, 0.5 = 1, 0, 0.5
0.0f, 1.0f, 0.0 0, 0, 1 0，1, 0.5
1.0f, 1.0f, 0.0 1, 1, 1
0, 0, 0 1, 0, 0 0, 0, 0
1, 0, 0.5 preScale(1f, -1f)即 0, -1, 0 = 1, 0, 0.5
0，1, 0.5 0, 0, 1 0, -1,0.5
1, 1, 1 1, -1, 1
0, 0, 0 0, 0, 0
1, 0, 0.5 preTranslate(-0.5f, -0.5f) 1, 0, -0.5 1, 0, 0
0, -1,0.5 即 0, 1, -0.5 = 0, -1, 1
1, -1, 1 0, 0, 1 1, -1, 1经过renderMatrix的转换后，纹理座标确确实实的向下翻转了。
再然后，DrawY的setpSize = 1；DrawU的setpSize = 2；DrawV的setpSize = 2；而且采集大小是正方形的（都是除frameWidth），结合YUV 4:2:0 采样理论可知（如下图其中，Y 分量用叉表示，UV 分量用圆圈表示。）其实xUnit/setpSize就是采样步阀。
然后还有一个问题要注意，为啥shader上输出rgba四个采样点，纹理采样分别是 (tc - 1.5 * xUnit) (tc - 0.5 * xUnit) (tc + 0.5 * xUnit) (tc + 1.5 * xUnit) 并不是(tc - 2 * xUnit) (tc - 1 * xUnit) (tc + 1 * xUnit) (tc + 2 * xUnit)？这是取每个纹素的中心颜色值作为采样色值。
好了，重点都解读完了。剩下的需要关注的提几点：drawU和drawV时候的viewport和窗口大小的确定；JavaI420Buffer的封装使用；再聊一点就是这里没有结合双pbo缓冲readpixel的（不理解说啥的同学自行百度，很多教学文章。）在技巧上巧妙的把rgba转成yuv420的draw，属于GPGPU的思想，大家可以细细体会。
下一章发掘HardwareVideoEncoder有用的知识点，之后就深入PeerConnectionFactory.createPeerConnection。