Background
Sergei Mikhailovich Prokudin-Gorskii (1863-1944) was a photographer ahead of his time. He saw color photography as the wave of the future and came up with a simple idea to produce color photos: record three exposures of every scene onto a glass plate using a red, a green, and a blue filter and then project the monochrome pictures with correctly coloured light to reproduce the color image; color printing of photos was very difficult at the time. Due to the fame he got from his color photos, including the only color portrait of Leo Tolstoy (a famous Russian author), he won the Tzar's permission and funding to travel across the Russian Empire and document it in 'color' photographs. His RGB glass plate negatives were purchased in 1948 by the Library of Congress. They are now digitized and available on-line.
Photo Processing
Split and Merge
Because the images are, from top to bottom, in BGR order, I split each image into 3 parts and merge to get the colorized image.
Aligning
In order to overlay the 3 BGR images in alignment, there are some ways.
- Sum of squared differences (SSD) between the intensities of the two features and the latter computes the correlation between the intensity values of the two features.
- Mean squares method and the normalized correlation （NCC）
- Enhanced Correlation Coefficient (ECC)
I use ECC algorithm to align the images. The ECC image alignment algorithm introduced in OpenCV 3 is based on a 2008 paper titled Parametric Image Alignment using Enhanced Correlation Coefficient Maximization by Georgios D. Evangelidis and Emmanouil Z. Psarakis. They propose using a new similarity measure called Enhanced Correlation Coefficient (ECC) for estimating the parameters of the motion model. There are two advantages of using their approach.
- Unlike the traditional similarity measure of difference in pixel intensities, ECC is invariant to photometric distortions in contrast and brightness.
- Although the objective function is nonlinear function of the parameters, the iterative scheme they develop to solve the optimization problem is linear. In other words, they took a problem that looks computationally expensive on the surface and found a simpler way to solve it iteratively.
Automatic Cropping
Canny Edge Detector
Canny is a good way to extract useful structual information from different vision objects. I use it to get the outline before detecting the lines in margin. Because the goal is to crop the white, black or other color borders, I do not to detect the whole image. Instead, I split the graph into 4 parts: top, bottom, left, right. Each part contains only 10% of height or wide of the Step2 image.
Probabilistic Hough Transform
Hough is a technique to find shapes. If a shape can be described as a mathematical expression, then it can be detected in a image by Hough Transform. Straight lines are the simplest case. I use Probabilistic Hough Transform since it is a more efficient version of Hough Transform for curve detection.
To make the Hough Line Detection more intuitive, I detect the lines on the whole canny image and draw the lines on the colorized image. But in project, I detect the lines in 4 part (top, bottom, left, right) mentioned before.
Cropping
After detecting the lines, I choose the bottommost line on the top part, the topmost line on the bottom part, the rightmost lines on the left and the leftmost lines on the right part. Then cut the image.
Image Pyramid
When I try to run the above algorithm on a large image, the procedure was broken down. However, for a 500 X 500 image, it cost no more than 1 second. Thus, the idea is to resize the image by a factor of 0.5 recursively until the size of height is small than 500px and record the recurvise times N. If I resize the image 1 time by factor 0.5, the height and width [x, y] will become [0.5x, 0.5y]
After that, I get a small image and implement the above algorithm on it. Particularly, in ECC alignment, while getting a warp_matrix from small image, I mutilply it with a matric [[1 1 2]; [1 1 2]] N times and apply it on the large image to expand the X, Y offset.
Similarly, because I run the Canny algorithm on small image, we need to mutiply 2^N times on the X or Y of lines when crop the image.
There are some results:
Code:
https://gitee.com/sengo/Russian-Empire
Reference:
1.https://www.learnopencv.com/image-alignment-ecc-in-opencv-c-python/
2.OpenCV-Python 中文教程 OpenCV官方教程中文版（For Python）https://github.com/makelove/OpenCV-Python-Tutorial/blob/master/OpenCV-Python-Tutorial-%E4%B8%AD%E6%96%87%E7%89%88.pdf
3.http://robertpless.com/classes/ComputerVision/project1/description.html
4.https://en.wikipedia.org/wiki/Canny_edge_detector
5.https://en.wikipedia.org/wiki/Hough_transform
6.http://crcv.ucf.edu/videos/lectures/2012.php
7.http://www.jb51.net/article/33274.htm