Technically speaking, single-frame HDR is much simpler than multi-frame HDR. When the processing capacity of the early equipment is insufficient, the speed is fast, there is no smear, and the single-frame HDR with low performance requirements has more advantages. In the HDR+ era, single-frame HDR is gradually losing to multi-frame composite HDR. The iteration of technology is so cruel. These technologies were briefly used on some mobile phones, but they were quickly withdrawn from the market due to poor results.
The results of several attempts are very general, where is the way out for hardware HDR? In response to this question, mainstream manufacturers have also given their own answers.
1. Line interleaved HDR
The problem of multiple frames with different exposures mainly occurs in the inability to deal with moving objects and is easily affected by hand shake. Although the upper limit of the image quality of multiple frames with the same exposure is high, the cost is also very high. Is there a compromise method that can both How to better deal with moving objects and better save valuable computing resources and power?
The answer is line interleaved HDR.
To understand line interleaving HDR, we must first understand the working principle of CMOS. Modern mobile phones use Rolling Shutter, which means that its reading is not all pixels at the same time, but from top to bottom. Read line by line.
In traditional multi-frame HDR, each frame needs to be read after the previous frame is read. CMOS will first perform a long exposure, then read the data of the entire screen from top to bottom, then start the medium exposure, read it from top to bottom, and finally complete the short exposure in the same way. In other words, the time interval between two frames is the exposure time + the time for the sensor to complete reading one frame. After the first row of pixels is read out, it needs to wait for the pixels of other rows to be read out before starting mid-exposure.
If we can start the next exposure immediately after each row of pixels in the CMOS has finished reading, we can save the time waiting for other pixels to finish reading. After we let the CMOS finish the long exposure, when the first line is read, the first line of pixels starts to be exposed almost immediately, without waiting for the pixel reading of other lines to complete, and at this time, other lines are also reading Take - The work of the next exposure, each row is alternated with different exposures and readings, just like the interwoven yarns in a textile, we call this the row interweaving technique. The reading interval between frames is shortened, and artifacts caused by object motion are significantly improved.
Taking two exposures as an example, each line of the sensor will be output twice in sequence with long and short exposures. From the perspective of readout, the long-exposure frame line and the short-exposure frame line interleaving are output in sequence:
Of course, the solutions proposed by various manufacturers are somewhat different. Sony's is called DOL-HDR (Digital overlap), and OV's is called staggered HDR, which will be introduced separately below.
1. DOL-HDR
Sony supports 'quasi-simultaneous' output of multiple frames of images with different exposure times. After the ISP receives multiple frames of images, it can perform image fusion to generate HDR images.
When Sony launched DOL, it promoted DOL as 'quasi-simultaneous' output length exposure. Since it is 'quasi-simultaneous', it is not at the same time, so there will be some typical temporal multi-frame HDR image quality problems, and DOL also has some unique IQ (image quality) problems - HDR Transition Artefacts.
As can be seen from the noise profile of DOL hdr, as shown in the figure below, at the HDR splicing, you can see a significant change in SNR, which is called snr drop. When the SNR drop is large, there will be obvious edges, as shown in the figure above.
The smaller the exposure ratio, the smaller the SNR drop. It is conceivable that if the exposure ratio is 1, there will be no snr drop. Conversely, the larger the exposure ratio, the larger the dynamic range and the larger the snr drop.
2. Staggered HDR
The difference from Sony is that it supports up to 4:1 exposure output, namely long, medium, short, veryshort. OV supports up to 3:1 output, that is, long, medium, and short.
So if the exposure ratio is the same, the composite HDR image sony can achieve a larger dynamic range.
The synthesis method is the same, save the first output exposure line into the line buffer, and then do frame stitch to form the HDR output.
OV supports 4 output modes:
Supports outputting 3 frames of 12bit: L+M+S
stitches the images exposed by Long and Medium on the sensor to become a 16bit fused linear image, plus the original short exposure image (12bit).
The long exposure and medium exposure are fused into a 12bit image, and the original short exposure image (12bit) is added.
Two frames of 12bit original images, which can be long + short, or medium + short.
Compared with Sony's DOL-HDR, the output mode of OV is more flexible to configure to adapt to the limitation of transmission bandwidth.
Summary: When using the line interleaving HDR method, the interval of each frame is reduced from the exposure time + readout time of time domain HDR to only the exposure time. However, since the time interval between two adjacent rows is changed from reading only one short exposure previously to reading one short exposure and several long exposures, the readout time of row-interleaved HDR will be doubled (the multiple depends on how many times each row is exposures), the jelly effect is exacerbated.
In the daytime, when the short exposure time is very short, you can see that the exposure interval of each frame is almost negligible, but it is still not synchronized, the object will also be displaced, and the effect will not be satisfactory in low light.
In general, line-interleaved HDR is one of the best HDR synthesis methods in the general consumer field.
二、Dual Gain
As the number of pixels of the image sensor is getting higher and higher, the size of a single pixel becomes smaller and smaller, and the pixel pitch of the current 100 million pixel sensor has reached 0.8um. The decrease in full well capacity and SNR performance caused by the smaller Pixel size also greatly affects the dynamic range of the sensor. Therefore, mobile phone sensor factories need to adopt new technical means to solve and improve this problem.
We all know that a CMOS image sensor is essentially a device connected by photoelectric conversion and analog-to-digital conversion. If these two steps of conversion work in a state with a certain multiplier gain, there should be a one-to-one correspondence between how many photons are input and how much brightness is output. But in fact, anyone with photography knowledge knows that this is not the case. ISO can be set in digital cameras, which means "variable gain" - under a fixed number of photons, the output brightness can be inconsistent. Of course, this inconsistency is not entirely without cost.
For a typical CMOS sensor (built-in ADC), the process of outputting images can be summarized as the following steps:
1. The photodiode absorbs energy and is excited, producing electrons.
2. Electrons enter the charge trap.
3. Disconnect the photocell and there will be a voltage across the charge trap (essentially a capacitor).
4. This voltage signal is too weak and needs an analog amplification, generally using a Programmable Voltage Amplifier (PGA).
5. The amplified signal goes into analog-to-digital conversion.
Obviously in traditional CMOS technology, the steps to change the gain value (the ratio of output brightness to input brightness) are 4 and 5 - or to put it another way, changing the gain multiplier of 4 is equivalent to setting the ISO (analog) on the camera body. Gain), changing the gain multiplier of 5 (digital gain) is equivalent to using extended ISO or adding brightness in ACR later.
Secondly, a performance parameter called Full-Well Capacity (FWC) is often mentioned in sensor technology, and a concept called White Level is also used in digital images. In fact, for a camera and its output image, these two values should be in one-to-one correspondence - a full well means overexposure, and the overexposure output image is pure white. So the well capacity (the capacity of the capacitor in 3) will also affect the ISO.
Therefore, we can complete HDR by adjusting Gain to obtain pictures with different exposure values (under different ISOs). The position of changing the gain is different, and the technology involved is also different. The popular ones are: Dual Native ISO Technology (Dual Gain Amplifier) and Dual Conversion Gain Technology (Dual Conversion Gain, DCG). They are introduced separately below.
1. Dual Gain Amplifier
The Dual Gain Amplifier technology actually provides two circuits in the circuit to connect PGAs with different gains. In fact, more accurately, this technique should be called dual circuit gain. Since changing the analog gain is equivalent to adjusting the ISO, it is also called dual native ISO technology.
It is worth mentioning that in the main camera of the Xiaomi Mi 10 Extreme Commemorative Edition, Xiaomi and OmniVision proposed the "dual native ISO Fusion technology", which is compared with the dual native ISO technology as follows:
The specific technical principle is to be inquired.
2. Dual Conversion Gain
When we adjust the well capacitance to obtain exposure values of different gains, we use the double conversion gain technique: the pixels have two different FWCs. When the FWC is small, the noise floor is small, corresponding to a higher gain; when the FWC is large, the noise floor is large, corresponding to a lower gain. In this way, the proportion of noise is reduced, that is, the signal-to-noise ratio is improved.
Due to the need to change the well capacitance, the actual realization of this technology lies in the special pixel structure:
The scheme used adds a Dual Gain Switch between the FD node and the CFD charge trap. In the mode of High conversion gain, the DCG remains disconnected, so that the FD capacitor is equal to the small capacitor FD, so the conversion gain is large. When working in low conversion gain mode, DCG remains connected, the FD capacitor is equal to the large capacitor, and the corresponding CG is small.
3. Dual Gain+ line interleaving HDR
On the basis of the previous one, OV realizes dual gain that supports simultaneous reading on the main camera OV48C used in the Xiaomi Mi 10 Extreme Commemorative Edition, which can read a brighter and a darker picture at the same time. Based on this, a stronger implementation is obtained using this technique with row-interleaved HDR:
In the first frame, we use the same exposure time to read out two frames of high gain (High Conversion Gain, HCG) and low gain (Low Conversion Gain, LCG) at the same time to achieve the effect of long exposure and medium exposure, and then A short exposure can be added. If the light ratio is not too large, it is also possible to give up the short exposure in exchange for a shorter lag.
This enhanced version of interleaved HDR with no time lag between long and medium exposures further improves shooting of moving subjects.