计算机视觉学习（七）：由基础矩阵计算照相机矩阵

基础矩阵

原理

基本矩阵体现了两视图几何（对极几何，epipolar geometry）的内在射影几何（projective geometry）关系，基本矩阵只依赖于摄像机的内参KK和外参R,tR,t。

上图是一个两视图的几何描述，其中OO、O′O′是两个相机的光心，两点连线OO′OO′称为基线，基线与图像平面的交点ee、e′e′称为对极点，其中ll、l′l′分别是图像点x′x′、xx对应的对极线。

上图的左侧相机的图像平面上的一个点xx，反向投影得到射线OXOX。由于点的深度未知，图像平面上的点xx可能是射线上某一深度的3D点XX。而射线OXOX在第二个相机的图像平面上的投影为l′l′。也就是说，给定一对图像，第一幅图像上的每个点xx，在另外一幅图像上存在一条直线l′l′与之对应。换言之，第二幅图像上与点xx对应的点x′x′必定在线l′l′上。

我们可以看到这里存在一个从一副图像上的点到另外一幅图像与之对应的对极线的映射x→l′x→l′。而基本矩阵就表示了这种从点到直线的射影映射关系。

几何和代数推导

参考博客https://blog.csdn.net/kokerf/article/details/72191054#commentBox

性质

假设两幅图像由中心不重合的相机获得，则基本矩阵F为所对应点x↔x′x↔x′都满足下式的秩为22的3×33×3齐次线性矩阵。

有如下性质：

F是秩为22、自由度为77的齐次矩阵（3×33×3其次矩阵有88个独立比率，还有一个满足约束detF=0detF=0的约束，所以再减去一个自由度）
对应点：如果xx和x′x′是对应的图像点，那么x′TFx=0x′TFx=0。
对极线：

l′=Fxl′=Fx对应于xx的对极线
l=FTxl=FTx对应于x′x′的对极线
对极点：

Fe=0Fe=0
FTe=0FTe=0
有摄像机矩阵PP、P′P′计算：

一般相机

F=[e′]×P′P+F=[e′]×P′P+，其中P+P+是PP的伪逆，e′=P′Oe′=P′O且PO=0PO=0。

规范相机，P=[I|0]P=[I|0]，P′=[M|m]P′=[M|m]

F=[e′]×M=M′T[e′]×F=[e′]×M=M′T[e′]×，其中e′=me′=m且e=M−1me=M−1m

非无穷远相机P=K[I|0]P=K[I|0] ，P′=K′[R|t]P′=K′[R|t]

F=K′−T[t]×RK−1=[K′[t]×K′RK−1=K′−TRKT[KRTt]×

本次实验流程

用SIFT算法实现两幅图像的特征点检测，找到对应的匹配点并绘制出来
使用RANSAC方法估计最佳基础矩阵F，以及正确点的索引，求出不同图像对的基础矩阵
由基础矩阵计算照相机矩阵
从照相机矩阵的列表中，对正确点的三维点进行三角剖分，挑选出经过三角剖分后，在两个照相机前均含有最多场景点

代码


# coding: utf-8

# In[1]:

from PIL import Image
from numpy import *
from pylab import *
import numpy as np


# In[2]:

from PCV.geometry import camera
import homography
from PCV.geometry import sfm
from PCV.localdescriptors import sift
camera = reload(camera)
homography = reload(homography)
sfm = reload(sfm)
sift = reload(sift)


# In[3]:

# Read features
im1 = array(Image.open('D:\pythonxy\PCV-book-data\data\canss2.jpg'))
sift.process_image('D:\pythonxy\PCV-book-data\data\canss2.jpg', 'im1.sift')

im2 = array(Image.open('D:\pythonxy\PCV-book-data\data\canss3.jpg'))
sift.process_image('D:\pythonxy\PCV-book-data\data\canss3.jpg', 'im2.sift')


# In[4]:

l1, d1 = sift.read_features_from_file('im1.sift')
l2, d2 = sift.read_features_from_file('im2.sift')


# In[5]:

matches = sift.match_twosided(d1, d2)


# In[6]:

ndx = matches.nonzero()[0]
x1 = homography.make_homog(l1[ndx, :2].T)
ndx2 = [int(matches[i]) for i in ndx]
x2 = homography.make_homog(l2[ndx2, :2].T)

d1n = d1[ndx]
d2n = d2[ndx2]
x1n = x1.copy()
x2n = x2.copy()


# In[7]:

figure(figsize=(16,16))
sift.plot_matches(im1, im2, l1, l2, matches, True)
show()


# In[26]:

#def F_from_ransac(x1, x2, model, maxiter=5000, match_threshold=1e-6):
def F_from_ransac(x1, x2, model, maxiter=5000, match_threshold=1e-6):
    """ Robust estimation of a fundamental matrix F from point
    correspondences using RANSAC (ransac.py from
    http://www.scipy.org/Cookbook/RANSAC).

    input: x1, x2 (3*n arrays) points in hom. coordinates. """

    from PCV.tools import ransac
    data = np.vstack((x1, x2))
    d = 10 # 20 is the original
    # compute F and return with inlier index
    F, ransac_data = ransac.ransac(data.T, model,
                                   8, maxiter, match_threshold, d, return_all=True)
    return F, ransac_data['inliers']


# In[27]:

# find F through RANSAC
model = sfm.RansacModel()
F, inliers = F_from_ransac(x1n, x2n, model, maxiter=5000, match_threshold=1e-3)
print F


# In[28]:

P1 = array([[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0]])
P2 = sfm.compute_P_from_fundamental(F)


# In[29]:

print P2
print F


# In[30]:

# P2, F (1e-4, d=20)
# [[ -1.48067422e+00   1.14802177e+01   5.62878044e+02   4.74418238e+03]
#  [  1.24802182e+01  -9.67640761e+01  -4.74418113e+03   5.62856097e+02]
#  [  2.16588305e-02   3.69220292e-03  -1.04831621e+02   1.00000000e+00]]
# [[ -1.14890281e-07   4.55171451e-06  -2.63063628e-03]
#  [ -1.26569570e-06   6.28095242e-07   2.03963649e-02]
#  [  1.25746499e-03  -2.19476910e-02   1.00000000e+00]]


# In[31]:

# triangulate inliers and remove points not in front of both cameras
X = sfm.triangulate(x1n[:, inliers], x2n[:, inliers], P1, P2)


# In[32]:

# plot the projection of X
cam1 = camera.Camera(P1)
cam2 = camera.Camera(P2)
x1p = cam1.project(X)
x2p = cam2.project(X)


# In[33]:

figure(figsize=(16, 16))
imj = sift.appendimages(im1, im2)
imj = vstack((imj, imj))

imshow(imj)

cols1 = im1.shape[1]
rows1 = im1.shape[0]
for i in range(len(x1p[0])):
    if (0<= x1p[0][i]<cols1) and (0<= x2p[0][i]<cols1) and (0<=x1p[1][i]<rows1) and (0<=x2p[1][i]<rows1):
        plot([x1p[0][i], x2p[0][i]+cols1],[x1p[1][i], x2p[1][i]],'c')
axis('off')
show()


# In[34]:

d1p = d1n[inliers]
d2p = d2n[inliers]


# In[35]:

# Read features
im3 = array(Image.open('D:\pythonxy\PCV-book-data\data\canss3.jpg'))
sift.process_image('D:\pythonxy\PCV-book-data\data\canss3.jpg', 'im3.sift')
l3, d3 = sift.read_features_from_file('im3.sift')


# In[36]:

matches13 = sift.match_twosided(d1p, d3)


# In[37]:

ndx_13 = matches13.nonzero()[0]
x1_13 = homography.make_homog(x1p[:, ndx_13])
ndx2_13 = [int(matches13[i]) for i in ndx_13]
x3_13 = homography.make_homog(l3[ndx2_13, :2].T)


# In[38]:

figure(figsize=(16, 16))
imj = sift.appendimages(im1, im3)
imj = vstack((imj, imj))

imshow(imj)

cols1 = im1.shape[1]
rows1 = im1.shape[0]
for i in range(len(x1_13[0])):
    if (0<= x1_13[0][i]<cols1) and (0<= x3_13[0][i]<cols1) and (0<=x1_13[1][i]<rows1) and (0<=x3_13[1][i]<rows1):
        plot([x1_13[0][i], x3_13[0][i]+cols1],[x1_13[1][i], x3_13[1][i]],'c')
axis('off')
show()


# In[39]:

P3 = sfm.compute_P(x3_13, X[:, ndx_13])


# In[40]:

print P3


# In[41]:

print P1
print P2
print P3


# In[22]:

# Can't tell the camera position because there's no calibration matrix (K)