3D vision (4): ORB feature extraction and matching
According to Wikipedia's definition, image features are a set of information related to computing tasks, which depend on specific applications. In short, features are another form of representation of image information.
Digital images are stored in the computer as a matrix of gray values, so the simplest single image pixel is also a feature. However, in visual odometry, we hope that the feature points remain stable during camera movement, while the gray value is seriously affected by illumination, deformation, and object material, and varies greatly between different images, which is not stable enough.
Ideally, the algorithm would also be able to tell from the image where the same point is when the scene and camera perspective change by a small amount. Therefore, the gray value alone is not enough, we need to extract feature points from the image.
Article directory
- 3D vision (4): ORB feature extraction and matching
- 1. ORB feature principle
- 1.1. Key points of FAST
- 1.2. BRIEF descriptor
- 1.3. ORB feature matching
- 2. Algorithm process
- 2.1. The process of extracting ORB features
- 2.2. ORB feature matching process
- 3. Experimental process
- 4. Source code
- 5. Project link
1. ORB feature principle
Feature points are composed of two parts: Key-point and Descriptor. For example, when we say "calculate SIFT feature points in an image", we mean two things: "extract SIFT key points and calculate SIFT descriptors".
The key point refers to the position of the feature point in the image. The descriptor is usually a vector, which describes the information of the pixels around the key point in some artificially designed way. The descriptors are designed according to the principle that "features with similar appearance should have similar descriptors". Therefore, as long as the descriptors of two feature points have similar distances in the vector space, they can be considered as the same feature point.
ORB features are composed of key points and descriptors. Its key point is called "Oriented FAST", which is an improved FAST corner point, and its descriptor is called BRIEF, which is an extremely fast binary descriptor.
1.1. Key points of FAST
FAST is a kind of corner point, which mainly detects places where the local pixel gray level changes significantly, and is known for its fast speed. The idea is that if a pixel is very different (too bright or too dark) from its neighbors, then it is probably a corner. Compared with other corner detection algorithms, FAST only needs to compare the brightness of pixels, which is very fast.
The FAST corner detection process is as follows:
For the weakness of FAST corners that do not have directionality and scale, ORB adds scale and rotation descriptions. Scale invariance is achieved by building an image pyramid and detecting corners at each level of the pyramid. The feature rotation is realized by the gray-scale centroid method, which calculates the feature direction vector of each block, rotates the block accordingly, and then extracts the corresponding descriptor.
In terms of rotation, we compute the grayscale centroid of the image near the feature points. The so-called centroid refers to the center with the gray value of the image block as the weight. The specific steps of the gray scale centroid method are as follows:
1.2. BRIEF descriptor
BRIEF is a binary descriptor whose description vector consists of many 0s and 1s, where 0s and 1s encode the size relationship between two random pixels (such as p and q) near the key point: if p is larger than q, then take 1, otherwise take 0. If we take 128 such p and q, we can finally get a 128-dimensional vector composed of 0 and 1.
Note that although p and q are selected arbitrarily, we will use a fixed template in actual operation, and select the corresponding pixel index from this template. This template is carefully designed by researchers to ensure that the extracted BRIEF vector has a good effect.
BRIEF uses the comparison of randomly selected points, which is very fast, and because of the use of binary expression, it is also very convenient to store and is suitable for real-time image matching.
1.3. ORB feature matching
Feature matching is an extremely critical step in visual SLAM. It solves the data association problem (data association) in SLAM, that is, to determine the correspondence between the currently seen landmark and the previously seen landmark. By accurately matching the descriptors between images, it can reduce a lot of burden for subsequent pose estimation, optimization and other operations.
However, due to the local characteristics of image features, the situation of mismatching exists widely, and has not been effectively solved for a long time. It has been called a major bottleneck restricting performance improvement in visual SLAM. Part of the reason is that there are often a large number of repeated textures in the scene, making the feature descriptions very similar, and in this case, it is very difficult to solve the mismatch by using only local features.
2. Algorithm process
2.1. The process of extracting ORB features
Step 1: Extract FAST key points to obtain a series of landmark coordinate positions.
Step 2: For each landmark point, select a patch, calculate the moment of the image block, and obtain the rotation direction.
Step 3: Use the designed orb pattern to select a specific point pair to obtain the two-dimensional point coordinates.
Step 4: According to the rotation direction, rotate the two-dimensional point coordinates to realize the rotation invariance of feature extraction.
Step 5: Compare the pixel value of the two-dimensional point coordinate position, assign 0 or 1, repeat the step 256 times, and calculate the BRIEF feature.
2.2. ORB feature matching process
Step 1: Detect the respective Oriented FAST corner positions of the two images to obtain keypoints_1 and keypoints_2.
Step 2: Calculate the respective BRIEF descriptors according to the corner positions to obtain descriptors_1 and descriptors_2.
Step 3: Match the BRIEF descriptors in the two images, and obtain matching point pairs based on the Hamming distance.
Step 4: Filter out the matching point pairs, keep the correct matches, and get good_matches.
3. Experimental process
Two images are captured by the binocular camera:
Extract the respective ORB features to obtain the landmark position and descriptor:
Perform feature matching to obtain corresponding point pairs:
4. Source code
ORB feature extraction with OpenCV function:
#include <iostream>
#include <opencv2/core/core.hpp>
// core是opencv的主要头文件，包含数据结构，矩阵运算，数据变换，内存管理，文本、数学等功能
#include <opencv2/features2d/features2d.hpp>
// features2d模块，包含特征提取和匹配等功能
#include <opencv2/highgui/highgui.hpp>
// highgui模块，包含图形界面、视频图像处理等功能
#include <chrono>
// chrono是C++11新加入的方便时间日期操作的标准库，它既是相应的头文件名称，也是std命名空间下的一个子命名空间，所有时间日期相关定义均在std::chrono命名空间下
// 通过这个新的标准库，可以非常方便进行时间日期相关操作。
using namespace std;
using namespace cv;
// ORB特征匹配算法流程：
// 第1步：检测两张图片各自的Oriented FAST角点位置，得到keypoints_1, keypoints_2;
// 第2步：根据角点位置，计算各自的BRIEF描述子，得到descriptors_1, descriptors_2;
// 第3步：对两幅图像中的BRIEF描述子进行匹配，基于Hamming距离，得到匹配点对matches;
// 第4步：对匹配点对做筛选剔除，保留下正确的匹配，得到good_matches;
// 第5步：作图，绘制匹配结果
int main(int argc, char **argv) {
// 读取图像
Mat img_1 = imread("../left_3.jpg");
Mat img_2 = imread("../right_3.jpg");
assert(img_1.data != nullptr && img_2.data != nullptr);
// 创建vector容器，存储特征点
std::vector<KeyPoint> keypoints_1, keypoints_2;
// 创建Mat矩阵，存储特征点的描述子
Mat descriptors_1, descriptors_2;
// 创建特征检测器、描述子提取器、点对匹配器
Ptr<FeatureDetector> detector = ORB::create();
Ptr<DescriptorExtractor> descriptor = ORB::create();
Ptr<DescriptorMatcher> matcher = DescriptorMatcher::create("BruteForce-Hamming");
// 第一步:检测 Oriented FAST 角点位置
chrono::steady_clock::time_point t1 = chrono::steady_clock::now();
detector->detect(img_1, keypoints_1);
detector->detect(img_2, keypoints_2);
// 输出提取到的特征点位置
cout << "keypoints_1.size :" << keypoints_1.size() << " keypoints_2.size :" << keypoints_2.size() << endl;
// for(int k=0; k<keypoints_1.size();k++)
// {
// cout << keypoints_1[k].pt.x << " " << keypoints_1[k].pt.y << endl;
// }
chrono::steady_clock::time_point t2 = chrono::steady_clock::now();
chrono::duration<double> time_used = chrono::duration_cast<chrono::duration<double>>(t2 - t1);
cout << "检测特征点耗时: " << time_used.count() << "s" << endl;
// 作图显示
Mat outimg1, outimg2;
drawKeypoints(img_1, keypoints_1, outimg1, Scalar::all(-1), DrawMatchesFlags::DEFAULT);
drawKeypoints(img_2, keypoints_2, outimg2, Scalar::all(-1), DrawMatchesFlags::DEFAULT);
imshow("ORB features1", outimg1);
imshow("ORB features2", outimg2);
waitKey(0);
cv::imwrite("../demo/ORB_features1.png", outimg1);
cv::imwrite("../demo/ORB_features2.png", outimg2);
// 第二步:根据角点位置计算 BRIEF 描述子
descriptor->compute(img_1, keypoints_1, descriptors_1);
descriptor->compute(img_2, keypoints_2, descriptors_2);
// 这里descriptors是一个 500*32 的二维矩阵，表示有500个landmark，每个标志点特征用32维的向量记录
cout << "第1张图片检测出来的ORB特征的描述子: " << descriptors_1.rows << " " << descriptors_1.cols << endl;
cout << "第2张图片检测出来的ORB特征的描述子: " << descriptors_2.rows << " " << descriptors_2.cols << endl;
// 每个特征点位置，提取得到一个32维度的特征
// cout << descriptors_1 << endl;
// [232, 84, 213, 84, 104, 76, 109, 48, 49, 170, 98, 120, 115, 55, 133, 255, 20, 244, 108, 106, 105, 86, 225, 46, 23, 186, 101, 144, 2, 241, 70, 61;
// 138, 204, 146, 255, 85, 213, 190, 230, 31, 14, 84, 2, 172, 255, 33, 64, 249, 123, 147, 194, 98, 156, 90, 255, 103, 215, 158, 139, 22, 89, 61, 138;
// 11, 159, 50, 223, 53, 183, 30, 246, 95, 73, 92, 38, 172, 186, 166, 228, 223, 58, 150, 128, 134, 152, 113, 255, 47, 127, 154, 131, 22, 204, 184, 15;
// ... ... ...
// 181, 94, 110, 254, 102, 130, 79, 112, 212, 130, 246, 230, 115, 201, 177, 97, 126, 228, 22, 238, 233, 167, 12, 46, 114, 171, 227, 136, 125, 187, 112, 49;
// 66, 178, 187, 186, 41, 237, 57, 87, 175, 93, 177, 125, 190, 158, 74, 123, 167, 51, 253, 1, 151, 73, 178, 209, 211, 209, 122, 251, 8, 4, 135, 83]
// 第三步:对两幅图像中的BRIEF描述子进行匹配，使用 Hamming 距离
vector<DMatch> matches;
t1 = chrono::steady_clock::now();
matcher->match(descriptors_1, descriptors_2, matches);
// matches数据结构包含的内容有：
// size：配对成功的特征点对数
// queryIdx：当前“匹配点”在查询图像的特征在KeyPoints1向量中的索引号，可以据此找到匹配点在查询图像中的位置
// trainIdx：当前“匹配点”在训练（模板）图像的特征在KeyPoints2向量中的索引号，可以据此找到匹配点在训练图像中的位置
// distance：两个特征点之间的欧氏距离，越小表明匹配度越高
// imgIdx：当前匹配点对应训练图像（如果有若干个）的索引，如果只有一个训练图像跟查询图像配对，即两两配对，则imgIdx=0
// 输出匹配好的特征点对的距离和索引
// cout << matches.size() << endl; // 497
// for(int k=0; k<matches.size(); k++)
// {
// cout << matches[k].distance << " " << matches[k].queryIdx << " " << matches[k].trainIdx << " " << matches[k].imgIdx << endl;
// }
t2 = chrono::steady_clock::now();
time_used = chrono::duration_cast<chrono::duration<double>>(t2 - t1);
cout << "特征点对匹配耗时: " << time_used.count() << "s" << endl;
// 第四步:匹配点对筛选
// 计算最小距离和最大距离
// minmax_element函数，返回指定范围内的最大最小值的元素的迭代器组成的一个pair, 如果最值多于一个，firstf返回的是第一个出现的最小值的迭代器，second返回的是最后一个出现的最大值的迭代器
// 第三个参数cmp可写可不写，max_element() 和 min_element() 默认是从小到大排列，然后 max_element() 输出最后一个值，min_element() 输出第一个值
// 但是如果自定义的cmp函数写的是从大到小排列，那么会导致max_element()和min_element()的两个结果是对调的，这里利用lambda表达式定义了cmp函数
// []后面的代码称为lambda表达式，获取代码{}中的返回值，函数符返回值是一个bool
auto min_max = minmax_element(matches.begin(), matches.end(), [](const DMatch &m1, const DMatch &m2) {
return m1.distance < m2.distance; });
double min_dist = min_max.first->distance;
double max_dist = min_max.second->distance;
cout << "匹配点对的最大距离: " << max_dist << endl;
cout << "匹配点对的最小距离: " << min_dist << endl;
// 当描述子之间的距离大于两倍的最小距离时，即认为匹配有误，但有时候最小距离会非常小，设置一个经验值30作为下限
std::vector<DMatch> good_matches;
for (int i = 0; i < matches.size(); i++)
{
if (matches[i].distance <= max(2 * min_dist, 30.0))
{
good_matches.push_back(matches[i]);
}
}
// 第五步:绘制匹配结果
Mat img_match;
Mat img_goodmatch;
drawMatches(img_1, keypoints_1, img_2, keypoints_2, matches, img_match);
drawMatches(img_1, keypoints_1, img_2, keypoints_2, good_matches, img_goodmatch);
imshow("all matches", img_match);
imshow("good matches", img_goodmatch);
waitKey(0);
cv::imwrite("../demo/all_matches.png", img_match);
cv::imwrite("../demo/good_matches.png", img_goodmatch);
return 0;
}
Self-implemented ORB feature extraction:
#include <opencv2/opencv.hpp>
// opencv.hpp中包含了OpenCV各模块的头文件，如高层GUI图形用户界面模块头文件highgui.hpp、图像处理模块头文件imgproc.hpp、2D特征模块头文件features2d.hpp等
#include <string>
// string头文件基本已经包含在iostream中，但平时使用建议加上#include <string>，尤其是使用string类型、使用cin、cout语句输入输出string类型变量、使用strlen()、strcpy()等函数时
#include <nmmintrin.h>
// SSE指令集，能大幅优化文本处理速度
#include <chrono>
// chrono是C++11新加入的方便时间日期操作的标准库，它既是相应的头文件名称，也是std命名空间下的一个子命名空间，所有时间日期相关定义均在std::chrono命名空间下
using namespace std;
using namespace cv;
// 提取ORB特征具体细节：
// 1、利用OpenCV的cv::FAST函数，检测出图像中FAST特征标志点landmark;
// 2、对于每个landmark标志点，选定patch，计算图像块的矩，得到旋转方向;
// 3、利用已经设计好的orb pattern，选取特定点对，得到二维点坐标;
// 4、根据旋转方向，对二维点坐标进行旋转;
// 5、比较此时点对的像素值，赋予0或1，重复步骤256次，计算得到BRIEF特征
string first_file = "../1.png";
string second_file = "../2.png";
// 每个unsigned int型含32位，容器中存储8个，则可以表示256维的0-1的BRIEF特征
typedef vector<uint32_t> DescType; // Descriptor type
/**
* compute descriptor of orb keypoints
* @param img input image
* @param keypoints detected fast keypoints
* @param descriptors descriptors
*
* 也就是说，如果FAST算法检测到的landmark在边界附近，无法得到周边的邻域，则也无法计算出BRIEF描述子，此时该landmark将被舍弃忽略
*
* 如果关键点超出边界（8像素），描述符将不会被计算，并将保留为空
*
* NOTE: if a keypoint goes outside the image boundary (8 pixels), descriptors will not be computed and will be left as empty
*
*/
void ComputeORB(const cv::Mat &img, vector<cv::KeyPoint> &keypoints, vector<DescType> &descriptors);
/**
* 根据两幅图像各自提取出来的BRIEF特征描述子，进行特征匹配，并存储于matches中
*
* brute-force match two sets of descriptors
* @param desc1 the first descriptor
* @param desc2 the second descriptor
* @param matches matches of two images
*/
void BfMatch(const vector<DescType> &desc1, const vector<DescType> &desc2, vector<cv::DMatch> &matches);
int main(int argc, char **argv) {
// 加载图像
cv::Mat first_image = cv::imread(first_file);
cv::Mat second_image = cv::imread(second_file);
assert(first_image.data != nullptr && second_image.data != nullptr);
chrono::steady_clock::time_point t1 = chrono::steady_clock::now();
// 调用OpenCV库 FAST算法， 设置参数threshold=40，检测图片中的landmark标志点
vector<cv::KeyPoint> keypoints1;
cv::FAST(first_image, keypoints1, 40);
// 调用自定义的函数，根据检测得到的landmark，提取其对应的BRIEF特征描述子
vector<DescType> descriptor1;
ComputeORB(first_image, keypoints1, descriptor1);
// 同样进行landmark检测、并提取BRIEF特征描述子
vector<cv::KeyPoint> keypoints2;
vector<DescType> descriptor2;
cv::FAST(second_image, keypoints2, 40);
ComputeORB(second_image, keypoints2, descriptor2);
chrono::steady_clock::time_point t2 = chrono::steady_clock::now();
chrono::duration<double> time_used = chrono::duration_cast<chrono::duration<double>>(t2 - t1);
cout << "提取 ORB 特征耗时: " << time_used.count() << "s. " << endl;
// 根据两幅图像各自提取到的landmark标志点和BRIEF特征描述子，进行匹配
vector<cv::DMatch> matches;
t1 = chrono::steady_clock::now();
BfMatch(descriptor1, descriptor2, matches);
t2 = chrono::steady_clock::now();
time_used = chrono::duration_cast<chrono::duration<double>>(t2 - t1);
cout << "匹配 ORB 特征耗时: " << time_used.count() << "s. " << endl;
cout << "匹配成功的landmark点对数量: " << matches.size() << endl;
// 作图
cv::Mat image_show;
cv::drawMatches(first_image, keypoints1, second_image, keypoints2, matches, image_show);
cv::imshow("matches", image_show);
cv::imwrite("../matches.png", image_show);
cv::waitKey(0);
return 0;
}
// ORB pattern
// BRIEF算法在提取landmark特征描述子时，需要随机选取邻域附近不同的点对，比较其像素亮度，
// 选取的索引，通过ORB pattern每一行数组元素值实现，共256行，对应256个点对，两两比较得到256维度的0-1特征，
// 这些索引的取法是经过精心设计好的，能够保证良好的特征提取效果。
int ORB_pattern[256 * 4] = {
8, -3, 9, 5/*mean (0), correlation (0)*/,
4, 2, 7, -12/*mean (1.12461e-05), correlation (0.0437584)*/,
-11, 9, -8, 2/*mean (3.37382e-05), correlation (0.0617409)*/,
7, -12, 12, -13/*mean (5.62303e-05), correlation (0.0636977)*/,
2, -13, 2, 12/*mean (0.000134953), correlation (0.085099)*/,
1, -7, 1, 6/*mean (0.000528565), correlation (0.0857175)*/,
-2, -10, -2, -4/*mean (0.0188821), correlation (0.0985774)*/,
-13, -13, -11, -8/*mean (0.0363135), correlation (0.0899616)*/,
-13, -3, -12, -9/*mean (0.121806), correlation (0.099849)*/,
10, 4, 11, 9/*mean (0.122065), correlation (0.093285)*/,
-13, -8, -8, -9/*mean (0.162787), correlation (0.0942748)*/,
-11, 7, -9, 12/*mean (0.21561), correlation (0.0974438)*/,
7, 7, 12, 6/*mean (0.160583), correlation (0.130064)*/,
-4, -5, -3, 0/*mean (0.228171), correlation (0.132998)*/,
-13, 2, -12, -3/*mean (0.00997526), correlation (0.145926)*/,
-9, 0, -7, 5/*mean (0.198234), correlation (0.143636)*/,
12, -6, 12, -1/*mean (0.0676226), correlation (0.16689)*/,
-3, 6, -2, 12/*mean (0.166847), correlation (0.171682)*/,
-6, -13, -4, -8/*mean (0.101215), correlation (0.179716)*/,
11, -13, 12, -8/*mean (0.200641), correlation (0.192279)*/,
4, 7, 5, 1/*mean (0.205106), correlation (0.186848)*/,
5, -3, 10, -3/*mean (0.234908), correlation (0.192319)*/,
3, -7, 6, 12/*mean (0.0709964), correlation (0.210872)*/,
-8, -7, -6, -2/*mean (0.0939834), correlation (0.212589)*/,
-2, 11, -1, -10/*mean (0.127778), correlation (0.20866)*/,
-13, 12, -8, 10/*mean (0.14783), correlation (0.206356)*/,
-7, 3, -5, -3/*mean (0.182141), correlation (0.198942)*/,
-4, 2, -3, 7/*mean (0.188237), correlation (0.21384)*/,
-10, -12, -6, 11/*mean (0.14865), correlation (0.23571)*/,
5, -12, 6, -7/*mean (0.222312), correlation (0.23324)*/,
5, -6, 7, -1/*mean (0.229082), correlation (0.23389)*/,
1, 0, 4, -5/*mean (0.241577), correlation (0.215286)*/,
9, 11, 11, -13/*mean (0.00338507), correlation (0.251373)*/,
4, 7, 4, 12/*mean (0.131005), correlation (0.257622)*/,
2, -1, 4, 4/*mean (0.152755), correlation (0.255205)*/,
-4, -12, -2, 7/*mean (0.182771), correlation (0.244867)*/,
-8, -5, -7, -10/*mean (0.186898), correlation (0.23901)*/,
4, 11, 9, 12/*mean (0.226226), correlation (0.258255)*/,
0, -8, 1, -13/*mean (0.0897886), correlation (0.274827)*/,
-13, -2, -8, 2/*mean (0.148774), correlation (0.28065)*/,
-3, -2, -2, 3/*mean (0.153048), correlation (0.283063)*/,
-6, 9, -4, -9/*mean (0.169523), correlation (0.278248)*/,
8, 12, 10, 7/*mean (0.225337), correlation (0.282851)*/,
0, 9, 1, 3/*mean (0.226687), correlation (0.278734)*/,
7, -5, 11, -10/*mean (0.00693882), correlation (0.305161)*/,
-13, -6, -11, 0/*mean (0.0227283), correlation (0.300181)*/,
10, 7, 12, 1/*mean (0.125517), correlation (0.31089)*/,
-6, -3, -6, 12/*mean (0.131748), correlation (0.312779)*/,
10, -9, 12, -4/*mean (0.144827), correlation (0.292797)*/,
-13, 8, -8, -12/*mean (0.149202), correlation (0.308918)*/,
-13, 0, -8, -4/*mean (0.160909), correlation (0.310013)*/,
3, 3, 7, 8/*mean (0.177755), correlation (0.309394)*/,
5, 7, 10, -7/*mean (0.212337), correlation (0.310315)*/,
-1, 7, 1, -12/*mean (0.214429), correlation (0.311933)*/,
3, -10, 5, 6/*mean (0.235807), correlation (0.313104)*/,
2, -4, 3, -10/*mean (0.00494827), correlation (0.344948)*/,
-13, 0, -13, 5/*mean (0.0549145), correlation (0.344675)*/,
-13, -7, -12, 12/*mean (0.103385), correlation (0.342715)*/,
-13, 3, -11, 8/*mean (0.134222), correlation (0.322922)*/,
-7, 12, -4, 7/*mean (0.153284), correlation (0.337061)*/,
6, -10, 12, 8/*mean (0.154881), correlation (0.329257)*/,
-9, -1, -7, -6/*mean (0.200967), correlation (0.33312)*/,
-2, -5, 0, 12/*mean (0.201518), correlation (0.340635)*/,
-12, 5, -7, 5/*mean (0.207805), correlation (0.335631)*/,
3, -10, 8, -13/*mean (0.224438), correlation (0.34504)*/,
-7, -7, -4, 5/*mean (0.239361), correlation (0.338053)*/,
-3, -2, -1, -7/*mean (0.240744), correlation (0.344322)*/,
2, 9, 5, -11/*mean (0.242949), correlation (0.34145)*/,
-11, -13, -5, -13/*mean (0.244028), correlation (0.336861)*/,
-1, 6, 0, -1/*mean (0.247571), correlation (0.343684)*/,
5, -3, 5, 2/*mean (0.000697256), correlation (0.357265)*/,
-4, -13, -4, 12/*mean (0.00213675), correlation (0.373827)*/,
-9, -6, -9, 6/*mean (0.0126856), correlation (0.373938)*/,
-12, -10, -8, -4/*mean (0.0152497), correlation (0.364237)*/,
10, 2, 12, -3/*mean (0.0299933), correlation (0.345292)*/,
7, 12, 12, 12/*mean (0.0307242), correlation (0.366299)*/,
-7, -13, -6, 5/*mean (0.0534975), correlation (0.368357)*/,
-4, 9, -3, 4/*mean (0.099865), correlation (0.372276)*/,
7, -1, 12, 2/*mean (0.117083), correlation (0.364529)*/,
-7, 6, -5, 1/*mean (0.126125), correlation (0.369606)*/,
-13, 11, -12, 5/*mean (0.130364), correlation (0.358502)*/,
-3, 7, -2, -6/*mean (0.131691), correlation (0.375531)*/,
7, -8, 12, -7/*mean (0.160166), correlation (0.379508)*/,
-13, -7, -11, -12/*mean (0.167848), correlation (0.353343)*/,
1, -3, 12, 12/*mean (0.183378), correlation (0.371916)*/,
2, -6, 3, 0/*mean (0.228711), correlation (0.371761)*/,
-4, 3, -2, -13/*mean (0.247211), correlation (0.364063)*/,
-1, -13, 1, 9/*mean (0.249325), correlation (0.378139)*/,
7, 1, 8, -6/*mean (0.000652272), correlation (0.411682)*/,
1, -1, 3, 12/*mean (0.00248538), correlation (0.392988)*/,
9, 1, 12, 6/*mean (0.0206815), correlation (0.386106)*/,
-1, -9, -1, 3/*mean (0.0364485), correlation (0.410752)*/,
-13, -13, -10, 5/*mean (0.0376068), correlation (0.398374)*/,
7, 7, 10, 12/*mean (0.0424202), correlation (0.405663)*/,
12, -5, 12, 9/*mean (0.0942645), correlation (0.410422)*/,
6, 3, 7, 11/*mean (0.1074), correlation (0.413224)*/,
5, -13, 6, 10/*mean (0.109256), correlation (0.408646)*/,
2, -12, 2, 3/*mean (0.131691), correlation (0.416076)*/,
3, 8, 4, -6/*mean (0.165081), correlation (0.417569)*/,
2, 6, 12, -13/*mean (0.171874), correlation (0.408471)*/,
9, -12, 10, 3/*mean (0.175146), correlation (0.41296)*/,
-8, 4, -7, 9/*mean (0.183682), correlation (0.402956)*/,
-11, 12, -4, -6/*mean (0.184672), correlation (0.416125)*/,
1, 12, 2, -8/*mean (0.191487), correlation (0.386696)*/,
6, -9, 7, -4/*mean (0.192668), correlation (0.394771)*/,
2, 3, 3, -2/*mean (0.200157), correlation (0.408303)*/,
6, 3, 11, 0/*mean (0.204588), correlation (0.411762)*/,
3, -3, 8, -8/*mean (0.205904), correlation (0.416294)*/,
7, 8, 9, 3/*mean (0.213237), correlation (0.409306)*/,
-11, -5, -6, -4/*mean (0.243444), correlation (0.395069)*/,
-10, 11, -5, 10/*mean (0.247672), correlation (0.413392)*/,
-5, -8, -3, 12/*mean (0.24774), correlation (0.411416)*/,
-10, 5, -9, 0/*mean (0.00213675), correlation (0.454003)*/,
8, -1, 12, -6/*mean (0.0293635), correlation (0.455368)*/,
4, -6, 6, -11/*mean (0.0404971), correlation (0.457393)*/,
-10, 12, -8, 7/*mean (0.0481107), correlation (0.448364)*/,
4, -2, 6, 7/*mean (0.050641), correlation (0.455019)*/,
-2, 0, -2, 12/*mean (0.0525978), correlation (0.44338)*/,
-5, -8, -5, 2/*mean (0.0629667), correlation (0.457096)*/,
7, -6, 10, 12/*mean (0.0653846), correlation (0.445623)*/,
-9, -13, -8, -8/*mean (0.0858749), correlation (0.449789)*/,
-5, -13, -5, -2/*mean (0.122402), correlation (0.450201)*/,
8, -8, 9, -13/*mean (0.125416), correlation (0.453224)*/,
-9, -11, -9, 0/*mean (0.130128), correlation (0.458724)*/,
1, -8, 1, -2/*mean (0.132467), correlation (0.440133)*/,
7, -4, 9, 1/*mean (0.132692), correlation (0.454)*/,
-2, 1, -1, -4/*mean (0.135695), correlation (0.455739)*/,
11, -6, 12, -11/*mean (0.142904), correlation (0.446114)*/,
-12, -9, -6, 4/*mean (0.146165), correlation (0.451473)*/,
3, 7, 7, 12/*mean (0.147627), correlation (0.456643)*/,
5, 5, 10, 8/*mean (0.152901), correlation (0.455036)*/,
0, -4, 2, 8/*mean (0.167083), correlation (0.459315)*/,
-9, 12, -5, -13/*mean (0.173234), correlation (0.454706)*/,
0, 7, 2, 12/*mean (0.18312), correlation (0.433855)*/,
-1, 2, 1, 7/*mean (0.185504), correlation (0.443838)*/,
5, 11, 7, -9/*mean (0.185706), correlation (0.451123)*/,
3, 5, 6, -8/*mean (0.188968), correlation (0.455808)*/,
-13, -4, -8, 9/*mean (0.191667), correlation (0.459128)*/,
-5, 9, -3, -3/*mean (0.193196), correlation (0.458364)*/,
-4, -7, -3, -12/*mean (0.196536), correlation (0.455782)*/,
6, 5, 8, 0/*mean (0.1972), correlation (0.450481)*/,
-7, 6, -6, 12/*mean (0.199438), correlation (0.458156)*/,
-13, 6, -5, -2/*mean (0.211224), correlation (0.449548)*/,
1, -10, 3, 10/*mean (0.211718), correlation (0.440606)*/,
4, 1, 8, -4/*mean (0.213034), correlation (0.443177)*/,
-2, -2, 2, -13/*mean (0.234334), correlation (0.455304)*/,
2, -12, 12, 12/*mean (0.235684), correlation (0.443436)*/,
-2, -13, 0, -6/*mean (0.237674), correlation (0.452525)*/,
4, 1, 9, 3/*mean (0.23962), correlation (0.444824)*/,
-6, -10, -3, -5/*mean (0.248459), correlation (0.439621)*/,
-3, -13, -1, 1/*mean (0.249505), correlation (0.456666)*/,
7, 5, 12, -11/*mean (0.00119208), correlation (0.495466)*/,
4, -2, 5, -7/*mean (0.00372245), correlation (0.484214)*/,
-13, 9, -9, -5/*mean (0.00741116), correlation (0.499854)*/,
7, 1, 8, 6/*mean (0.0208952), correlation (0.499773)*/,
7, -8, 7, 6/*mean (0.0220085), correlation (0.501609)*/,
-7, -4, -7, 1/*mean (0.0233806), correlation (0.496568)*/,
-8, 11, -7, -8/*mean (0.0236505), correlation (0.489719)*/,
-13, 6, -12, -8/*mean (0.0268781), correlation (0.503487)*/,
2, 4, 3, 9/*mean (0.0323324), correlation (0.501938)*/,
10, -5, 12, 3/*mean (0.0399235), correlation (0.494029)*/,
-6, -5, -6, 7/*mean (0.0420153), correlation (0.486579)*/,
8, -3, 9, -8/*mean (0.0548021), correlation (0.484237)*/,
2, -12, 2, 8/*mean (0.0616622), correlation (0.496642)*/,
-11, -2, -10, 3/*mean (0.0627755), correlation (0.498563)*/,
-12, -13, -7, -9/*mean (0.0829622), correlation (0.495491)*/,
-11, 0, -10, -5/*mean (0.0843342), correlation (0.487146)*/,
5, -3, 11, 8/*mean (0.0929937), correlation (0.502315)*/,
-2, -13, -1, 12/*mean (0.113327), correlation (0.48941)*/,
-1, -8, 0, 9/*mean (0.132119), correlation (0.467268)*/,
-13, -11, -12, -5/*mean (0.136269), correlation (0.498771)*/,
-10, -2, -10, 11/*mean (0.142173), correlation (0.498714)*/,
-3, 9, -2, -13/*mean (0.144141), correlation (0.491973)*/,
2, -3, 3, 2/*mean (0.14892), correlation (0.500782)*/,
-9, -13, -4, 0/*mean (0.150371), correlation (0.498211)*/,
-4, 6, -3, -10/*mean (0.152159), correlation (0.495547)*/,
-4, 12, -2, -7/*mean (0.156152), correlation (0.496925)*/,
-6, -11, -4, 9/*mean (0.15749), correlation (0.499222)*/,
6, -3, 6, 11/*mean (0.159211), correlation (0.503821)*/,
-13, 11, -5, 5/*mean (0.162427), correlation (0.501907)*/,
11, 11, 12, 6/*mean (0.16652), correlation (0.497632)*/,
7, -5, 12, -2/*mean (0.169141), correlation (0.484474)*/,
-1, 12, 0, 7/*mean (0.169456), correlation (0.495339)*/,
-4, -8, -3, -2/*mean (0.171457), correlation (0.487251)*/,
-7, 1, -6, 7/*mean (0.175), correlation (0.500024)*/,
-13, -12, -8, -13/*mean (0.175866), correlation (0.497523)*/,
-7, -2, -6, -8/*mean (0.178273), correlation (0.501854)*/,
-8, 5, -6, -9/*mean (0.181107), correlation (0.494888)*/,
-5, -1, -4, 5/*mean (0.190227), correlation (0.482557)*/,
-13, 7, -8, 10/*mean (0.196739), correlation (0.496503)*/,
1, 5, 5, -13/*mean (0.19973), correlation (0.499759)*/,
1, 0, 10, -13/*mean (0.204465), correlation (0.49873)*/,
9, 12, 10, -1/*mean (0.209334), correlation (0.49063)*/,
5, -8, 10, -9/*mean (0.211134), correlation (0.503011)*/,
-1, 11, 1, -13/*mean (0.212), correlation (0.499414)*/,
-9, -3, -6, 2/*mean (0.212168), correlation (0.480739)*/,
-1, -10, 1, 12/*mean (0.212731), correlation (0.502523)*/,
-13, 1, -8, -10/*mean (0.21327), correlation (0.489786)*/,
8, -11, 10, -6/*mean (0.214159), correlation (0.488246)*/,
2, -13, 3, -6/*mean (0.216993), correlation (0.50287)*/,
7, -13, 12, -9/*mean (0.223639), correlation (0.470502)*/,
-10, -10, -5, -7/*mean (0.224089), correlation (0.500852)*/,
-10, -8, -8, -13/*mean (0.228666), correlation (0.502629)*/,
4, -6, 8, 5/*mean (0.22906), correlation (0.498305)*/,
3, 12, 8, -13/*mean (0.233378), correlation (0.503825)*/,
-4, 2, -3, -3/*mean (0.234323), correlation (0.476692)*/,
5, -13, 10, -12/*mean (0.236392), correlation (0.475462)*/,
4, -13, 5, -1/*mean (0.236842), correlation (0.504132)*/,
-9, 9, -4, 3/*mean (0.236977), correlation (0.497739)*/,
0, 3, 3, -9/*mean (0.24314), correlation (0.499398)*/,
-12, 1, -6, 1/*mean (0.243297), correlation (0.489447)*/,
3, 2, 4, -8/*mean (0.00155196), correlation (0.553496)*/,
-10, -10, -10, 9/*mean (0.00239541), correlation (0.54297)*/,
8, -13, 12, 12/*mean (0.0034413), correlation (0.544361)*/,
-8, -12, -6, -5/*mean (0.003565), correlation (0.551225)*/,
2, 2, 3, 7/*mean (0.00835583), correlation (0.55285)*/,
10, 6, 11, -8/*mean (0.00885065), correlation (0.540913)*/,
6, 8, 8, -12/*mean (0.0101552), correlation (0.551085)*/,
-7, 10, -6, 5/*mean (0.0102227), correlation (0.533635)*/,
-3, -9, -3, 9/*mean (0.0110211), correlation (0.543121)*/,
-1, -13, -1, 5/*mean (0.0113473), correlation (0.550173)*/,
-3, -7, -3, 4/*mean (0.0140913), correlation (0.554774)*/,
-8, -2, -8, 3/*mean (0.017049), correlation (0.55461)*/,
4, 2, 12, 12/*mean (0.01778), correlation (0.546921)*/,
2, -5, 3, 11/*mean (0.0224022), correlation (0.549667)*/,
6, -9, 11, -13/*mean (0.029161), correlation (0.546295)*/,
3, -1, 7, 12/*mean (0.0303081), correlation (0.548599)*/,
11, -1, 12, 4/*mean (0.0355151), correlation (0.523943)*/,
-3, 0, -3, 6/*mean (0.0417904), correlation (0.543395)*/,
4, -11, 4, 12/*mean (0.0487292), correlation (0.542818)*/,
2, -4, 2, 1/*mean (0.0575124), correlation (0.554888)*/,
-10, -6, -8, 1/*mean (0.0594242), correlation (0.544026)*/,
-13, 7, -11, 1/*mean (0.0597391), correlation (0.550524)*/,
-13, 12, -11, -13/*mean (0.0608974), correlation (0.55383)*/,
6, 0, 11, -13/*mean (0.065126), correlation (0.552006)*/,
0, -1, 1, 4/*mean (0.074224), correlation (0.546372)*/,
-13, 3, -9, -2/*mean (0.0808592), correlation (0.554875)*/,
-9, 8, -6, -3/*mean (0.0883378), correlation (0.551178)*/,
-13, -6, -8, -2/*mean (0.0901035), correlation (0.548446)*/,
5, -9, 8, 10/*mean (0.0949843), correlation (0.554694)*/,
2, 7, 3, -9/*mean (0.0994152), correlation (0.550979)*/,
-1, -6, -1, -1/*mean (0.10045), correlation (0.552714)*/,
9, 5, 11, -2/*mean (0.100686), correlation (0.552594)*/,
11, -3, 12, -8/*mean (0.101091), correlation (0.532394)*/,
3, 0, 3, 5/*mean (0.101147), correlation (0.525576)*/,
-1, 4, 0, 10/*mean (0.105263), correlation (0.531498)*/,
3, -6, 4, 5/*mean (0.110785), correlation (0.540491)*/,
-13, 0, -10, 5/*mean (0.112798), correlation (0.536582)*/,
5, 8, 12, 11/*mean (0.114181), correlation (0.555793)*/,
8, 9, 9, -6/*mean (0.117431), correlation (0.553763)*/,
7, -4, 8, -12/*mean (0.118522), correlation (0.553452)*/,
-10, 4, -10, 9/*mean (0.12094), correlation (0.554785)*/,
7, 3, 12, 4/*mean (0.122582), correlation (0.555825)*/,
9, -7, 10, -2/*mean (0.124978), correlation (0.549846)*/,
7, 0, 12, -2/*mean (0.127002), correlation (0.537452)*/,
-1, -6, 0, -11/*mean (0.127148), correlation (0.547401)*/
};
// 计算landmark标志点对应的BRIEF特征描述子
// half_patch_size = 8，代表以landmark标志点为中心，16个像素的正方形为patch，选定邻域
// 如果关键点超出边界（8像素），描述符将不会被计算，该关键点会被忽略舍弃
void ComputeORB(const cv::Mat &img, vector<cv::KeyPoint> &keypoints, vector<DescType> &descriptors) {
const int half_patch_size = 8;
const int half_boundary = 16;
int bad_points = 0;
for (auto &kp: keypoints) {
// 如果关键点超出边界（16像素），将该landmark标志点push出容器，舍弃
if (kp.pt.x < half_boundary || kp.pt.y < half_boundary ||
kp.pt.x >= img.cols - half_boundary || kp.pt.y >= img.rows - half_boundary) {
bad_points++;
descriptors.push_back({
});
continue;
}
// 通过矩找到图像块的质心，计算得到m10、m01，以周边8个像素为区域块
float m01 = 0, m10 = 0;
for (int dx = -half_patch_size; dx < half_patch_size; ++dx) {
for (int dy = -half_patch_size; dy < half_patch_size; ++dy) {
uchar pixel = img.at<uchar>(kp.pt.y + dy, kp.pt.x + dx);
m10 += dx * pixel;
m01 += dy * pixel;
}
}
// 连接图像块的几何中心O和质心C，计算得到特征点landmark的方向
// angle should be arc tan(m01/m10)
float m_sqrt = sqrt(m01 * m01 + m10 * m10) + 1e-18; // avoid divide by zero
float sin_theta = m01 / m_sqrt;
float cos_theta = m10 / m_sqrt;
// 利用BRIEF算法，提取landmark标志点对应的特征描述子，用8个十进制数值(0-255)存储，则可得到256维度的BRIEF特征
DescType desc(8, 0);
for (int i = 0; i < 8; i++) {
uint32_t d = 0;
for (int k = 0; k < 32; k++) {
// 从ORB_pattern数组中，选出需要进行比较的随机点对坐标
int idx_pq = i * 32 + k;
cv::Point2f p(ORB_pattern[idx_pq * 4], ORB_pattern[idx_pq * 4 + 1]);
cv::Point2f q(ORB_pattern[idx_pq * 4 + 2], ORB_pattern[idx_pq * 4 + 3]);
// 根据landmark标志点的方向，旋转点对坐标
cv::Point2f pp = cv::Point2f(cos_theta * p.x - sin_theta * p.y, sin_theta * p.x + cos_theta * p.y) + kp.pt;
cv::Point2f qq = cv::Point2f(cos_theta * q.x - sin_theta * q.y, sin_theta * q.x + cos_theta * q.y) + kp.pt;
// 比较pp和qq对应的像素值，赋值0或1，最后将8维二进制转化为十进制存储，存储8个十进制数值，则存储得到256维度的BRIEF特征
// c++中1<<k，表示1左移k位,即2^k，2的k次方
// | 表示按位或，是双目运算符，功能是参与运算的两数各对应的二进位（也就是最后一位）相或。只要对应的二个二进位有一个为1时，结果位就为1
if (img.at<uchar>(pp.y, pp.x) < img.at<uchar>(qq.y, qq.x)) {
d |= 1 << k;
}
}
desc[i] = d;
}
descriptors.push_back(desc);
}
cout << "被舍弃的landmark标志点/总landmark标志点: " << bad_points << "/" << keypoints.size() << endl;
}
// 根据不同landmark标志点计算出的BRIEF特征描述子，进行匹配
void BfMatch(const vector<DescType> &desc1, const vector<DescType> &desc2, vector<cv::DMatch> &matches) {
// 当描述子之间的距离大于40时，则认为匹配有误，舍弃
const int d_max = 40;
// 对desc1里的所有标志点，依次计算它与desc2中所有标志点的距离
for (size_t i1 = 0; i1 < desc1.size(); ++i1) {
if (desc1[i1].empty()) continue;
cv::DMatch m{
int(i1), 0, 256};
for (size_t i2 = 0; i2 < desc2.size(); ++i2) {
if (desc2[i2].empty()) continue;
// 利用SSE指令，计算汉明距离
// 对两个字符串进行异或运算，并统计结果为1的个数，那么这个数就是汉明距离
int distance = 0;
for (int k = 0; k < 8; k++) {
distance += _mm_popcnt_u32(desc1[i1][k] ^ desc2[i2][k]);
}
// 存储匹配信息
if (distance < d_max && distance < m.distance) {
m.distance = distance;
m.trainIdx = i2;
}
}
if (m.distance < d_max) {
matches.push_back(m);
}
}
}
5. Project link
If the code doesn't work, or you want to use the dataset I made directly, you can download the project link:
https://blog.csdn.net/Twilight737