[HDR Learning] Apple EDR Technology Insights (2)

review

Apple uses the word EDR to distinguish it from HDR, because HDR may have different understandings in different scenarios:

HDR display: display bright and dark colors more vividly
HDR formats: HDR10, Dolby Vision
HDR conversion functions: PQ, HLG
Tone Mapping: HDR → SDR

EDR (Extended Dynamic Range) is a set of rendering pipeline technology introduced by Apple to support the correct display of SDR and HDR content on different screens at the same time. When displaying HDR content, EDR does not directly brighten the HDR area, but increases the overall screen brightness after recognizing the HDR content, while reducing the white point value of the non-HDR area, making it look less bright .

Technical solutions for EDR

SDR's floating-point representation of pixels ranges from [0.0, 1.0]0.0 to black and 1.0 to white. In the floating point representation of pixels in EDR, the part of SDR maps to [0.0, 1.0], and the part greater than 1.0 is the part of HDR that is brighter than SDR.

Unlike other HDR formats, EDR does not do Tone Mapping to map pixel values to [0.0, 1.0] the range. This means that when rendering, it has a new mechanism. When rendered, [0.0, 1.0] SDR content with pixel floating point values in the range is always rendered normally. (1.0, EDR headroom] Ranged HDR content is also renderable. However, the part that exceeds the EDR headroom will be discarded.

The presence of an EDR headroom enables brighter HDR content, but how much is it? In fact, EDR headroom is dynamic, and its value is affected by many factors, such as: the display technology of the device, the current display brightness, and so on.

We can usually use the following formula to roughly estimate the EDR headroom:

Headroom ≈ Display Peak / SDR

The maximum brightness level of the Pro Display XDR display that can be manually/automatically adjusted is 500nits (used in a well-lit environment), so take it as the default value of 1.0EDR, this range is the range of SDR, EDR value=1600nits/500nits=3.2; If the environment is dark, manually/automatically adjust the display brightness to 4nits, and the EDR value is 1600nits/4nits=400.

code explanation

1. Take CAMetalLayer, the preferred EDR framework, as an example. First look at the following 4 steps: Choose to use EDR-"Set the extended range color gamut-"Adjust the pixel format of the metaLayer to a floating-point format, such as RGBA16Float-"Actually generate EDR pixels

The code examples for the first three steps are as follows:

For the fourth step, we use ImageIO to import the HDR static image content and render it into an EDR texture. This process can be summarized as: first create a CGImage through the HDR image-"draw floating point bitmap-"create floating point texture-"convert Import the EDR bitmap into the texture-"render the texture into a metal pipeline available for EDR

1) Read the original HDR image and save it in CGImageRef format.

CGImageRef, this structure is used to create a pixel bitmap , and the image can be edited by manipulating the stored pixel bits . Its parameters are explained as follows:

sizt_t是定义的一个可移植性的单位，在64位机器中为8字节，32位位4字节。
width：图片宽度像素
height：图片高度像素
bitsPerComponent：每个颜色的比特数，例如在rgba-32模式下为8
bitsPerPixel：每个像素的总比特数
bytesPerRow：每一行占用的字节数，注意这里的单位是字节
space：颜色空间模式，例如const CFStringRef kCGColorSpaceGenericRGB 这个函数可以返回一个颜色空间对象。
bitmapInfo：位图像素布局，这是个枚举
provider：数据源提供者
decode[]：解码渲染数组
shouldInterpolate：是否抗锯齿
intent：图片相关参数

2) Draw a floating point bitmap

First read the width and height of the pixel bitmap, and then call CGBitmapContextCreate to create a drawing context (equivalent to a canvas) according to the bitmap composition information and color space (Display-P3 set before), and then call the CGContextDrawImage method to draw in the current context .

CGContextRef (The core API of Quartz 2D drawing is CGContextRef, which is specially used to draw various graphics.)

CGContextRef CGBitmapContextCreate (
void *data,
size_t width,
size_t height,
size_t bitsPerComponent,
size_t bytesPerRow,
CGColorSpaceRef colorspace,
CGBitmapInfo bitmapInfo
);
/**
参数：
data                    指向要渲染的绘制内存的地址。这个内存块的大小至少是（bytesPerRow*height）个字节。使用时可填NULL或unsigned char类型的指针。
width                   bitmap的宽度,单位为像素
height                  bitmap的高度,单位为像素
bitsPerComponent        内存中像素的每个组件的位数.例如，对于32位像素格式和RGB 颜色空间，你应该将这个值设为8。
bytesPerRow             bitmap的每一行在内存所占的比特数，一个像素一个byte。
colorspace              bitmap上下文使用的颜色空间。
bitmapInfo              指定bitmap是否包含alpha通道，像素中alpha通道的相对位置，像素组件是整形还是浮点型等信息的字符串。
*/

Explanation of CGBitmapInfo

CGBitmapInfo is composed of two parts or operations, one part is to specify the endian mode used by the cpu, and the other part is to specify the arrangement order of each bule green red alpha in the color space.

typedef CF_ENUM(uint32_t, CGImageByteOrderInfo) {
    kCGImageByteOrderMask     = 0x7000,
    kCGImageByteOrderDefault  = (0 << 12),
    kCGImageByteOrder16Little = (1 << 12),
    kCGImageByteOrder32Little = (2 << 12),
    kCGImageByteOrder16Big    = (3 << 12),
    kCGImageByteOrder32Big    = (4 << 12)
} CG_AVAILABLE_STARTING(10.0, 2.0);

typedef CF_ENUM(uint32_t, CGImageAlphaInfo) {
    kCGImageAlphaNone,               /* For example, RGB. */
    kCGImageAlphaPremultipliedLast,  /* For example, premultiplied RGBA */
    kCGImageAlphaPremultipliedFirst, /* For example, premultiplied ARGB */
    kCGImageAlphaLast,               /* For example, non-premultiplied RGBA */
    kCGImageAlphaFirst,              /* For example, non-premultiplied ARGB */
    kCGImageAlphaNoneSkipLast,       /* For example, RBGX. */
    kCGImageAlphaNoneSkipFirst,      /* For example, XRGB. */
    kCGImageAlphaOnly                /* No color data, alpha data only */
};

typedef CF_OPTIONS(uint32_t, CGBitmapInfo) {
    kCGBitmapAlphaInfoMask = 0x1F,

    kCGBitmapFloatInfoMask = 0xF00, 
    kCGBitmapFloatComponents = (1 << 8), // 浮点型表示

    kCGBitmapByteOrderMask     = kCGImageByteOrderMask,
    kCGBitmapByteOrderDefault  = kCGImageByteOrderDefault,  // 默认
    kCGBitmapByteOrder16Little = kCGImageByteOrder16Little, // 16 位小端
    kCGBitmapByteOrder32Little = kCGImageByteOrder32Little, // 32 位小端
    kCGBitmapByteOrder16Big    = kCGImageByteOrder16Big, // 16 位大端
    kCGBitmapByteOrder32Big    = kCGImageByteOrder32Big // 32 位大端
} CG_AVAILABLE_STARTING(10.0, 2.0);
// Big、Little 大端和小端分别
// 大端表示低字节放在高地址，高字节放在低地址
// 小端表示高字节放在高地址,低字节放在低地址

The format RGB of the color space must be sorted continuously. The only possible change is the storage location of A. There are two possibilities for the storage location of A:

Case 1: A is placed after RGB and RGBA corresponds to CGImageAlphaInfo of iOS as AlphaLast.

(For 32-bit images, 4 bytes represent a pixel, and every 8 bits represent a color.)

Case 2: A is placed in front of RGB and ARGB corresponds to CGImageAlphaInfo of iOS as AlphaFirst

Then contact the big and small endian, then

For case one: A is placed after RGBA

For big-endian-aligned CPUs, the pixel storage format is 0xRGBA

For the little-endian cpu, its pixel storage format is 0xABGR

For case two: A is placed in front of RGB ARGB

For big-endian-aligned CPUs, the pixel storage format is 0xARGB

For little-endian-aligned CPUs, the pixel storage format is 0xBGRA

Also: the role of the alpha channel

IOS is little endian, so kCGBitmapByteOrder16Host is kCGBitmapByteOrder16Little.

3) Create a texture object of type RGBA16Float (MTLTexture)

Use the newTextureWithDescriptor method to use a new piece of memory for storing texture image data to create a MTLTexture texture object, and the API uses MTLTextureDescriptor to describe the properties of the texture

MTLTextureDescriptor is used to define properties when creating MTLTexture, including image size (width, height, depth), pixel format, arrangement (array, cubemap) and the number of mipmaps. The value of MTLTextureDescriptor is useful when creating MTLTexture. After creation, changing the properties of MTLTextureDescriptor will have no effect on the created texture. That is to say, when the texture object is created, most of its attributes, such as size, new type, and pixel format, cannot be changed, but the pixel data of the texture can be changed.

Create a MTLTextureDescriptor that includes a texture property:
- textureType indicates the dimensionality and arrangement (array or cube) of the texture
- Width, height, and depth are used to indicate the pixel size of each dimension in the texture base level mipmap
- pixelFormat indicates the pixel storage method in texture
- arrayLength indicates the number of array elements of MTLTextureType1DArray or MTLTextureType2DArray type texture
- mipmapLevelCount indicates the number of texture mipmaps
- sampleCount indicates the number of samples corresponding to each pixel
- resourceOptions indicates the way memory is allocated
Create a texture based on MTLTextureDescriptor through the newTextureWithDescriptor: method of MTLDevice. After creation, if you want to copy the pixel data of the memory to the texture, call the replaceRegion :mipmapLevel:slice:withBytes:bytesPerRow:bytesPerImage: method to load the texture image data

4) Add the pixel data of EDR to the texture.

Use CGBitmapContextGetData to get the EDR image data, and use replaceRegion to set the EDR texture.

5) Use the metal pipeline to render EDR

【Related Links】

The text version description explained by bilibili: Overview of WWDC 2022 audio and video related sessions (EDR related)丨Audio and video engineering examples-Nuggets

bilibili explained: [Chinese subtitles] Apple: How does EDR work? The Way of HDR Rendering | Explore HDR rendering with EDR_哔哩哔哩_bilibili

w3 explains EDR: http://3ms.huawei.com/hi/group/1004055/wiki_6221842.html

Apple’s “EDR” Brings High Dynamic Range to Non-HDR Displays — Prolost

WeChat public account explanation: https://mp.weixin.qq.com/s/EgJkGimBs5AF1n3O4KqYog

[HDR Learning] Apple EDR Technology Insights (2)

review

Technical solutions for EDR

code explanation

Guess you like