Lao Liu is a second-year graduate student who is about to find a job. On the one hand, writing a blog is to review and summarize the knowledge points of big data development, on the other hand, he hopes to help partners who teach themselves programming like himself. Since Lao Liu is self-taught in big data development, there will definitely be some shortcomings in the blog, and I hope everyone can criticize and correct, let us make progress together!
What I bring to you today is the architecture design of HBase. Why is the architecture design of HBase so awesome? The content of this article will not be very long, it is all the essence of Lao Liu's summary, everyone should not miss it!
1 background
We need to know two questions in advance, and the solution of these two questions also happens to answer why the architecture design of HBase is so awesome!
The first question is HBase, as a distributed database, how does it ensure low-latency random read and write in massive data?
The second question is how does Hbase ensure data security in a distributed architecture?
2 Answer to the first question
To ensure low-latency random reads and writes in the massive data, HBase really took a lot of thought and made a lot of design. Let Lao Liu talk about it for everyone!
2.1 Memory + Disk
This design exists in many frameworks. The advantage of doing so is to make a trade-off between data security and data operation efficiency, which is the pursuit of data security and data operation efficiency.
2.2 Good data structure of memory data
HBase in order to ensure higher efficiency of data operations, and to ensure that the data in memory is flushed out to form an orderly file, what does this sentence mean? What is more efficient data manipulation?
That is, the memory file uses the data structure of ConcurentSkipListMap in order to ensure efficient operations such as inserting and deleting data, and the data is in order!
2.3 Range partition + sort
What method is generally used to query massive data?
The most rude way is violent scanning, but everyone generally doesn't use it, and everyone understands the truth! Therefore, we generally adopt the method of sorting first, then range partition + segment scanning. In order to keep the massive data in order, how can the data be ordered in the process of inserting data?
According to the picture above, Lao Liu will explain the reasoning. A comes first and then c, and finally b also comes. But how do we put b between a and c?
First put the data in the memory, a and c are already adjacent, now b is coming, to ensure the order of abc, it is actually very easy to do it, we can use an array or a linked list to move c back, and then insert b Come in, but the memory is limited, you need to write the data in the memory to the disk file every once in a while. The files in the disk are in order, which will facilitate quick positioning in the future!
2.4 Jump table Topo structure
What does it mean? In fact, it is the addressing mode of HBase.
HBase has two addressing modes, one is before version 0.96 and the other is after version 0.96.
Before HBase-0.96, HBase had two special tables, namely -ROOT- table and .META. table, where -ROOT- is stored in ZooKeeper, -ROOT- itself stores the RegionInfo of .META. Table information. The addressing process is as follows:
- The Client requests ZooKeeper to obtain the RegionServer address where -ROOT- is located.
- The Client requests the RS address of -ROOT- to obtain the address of the .META. table, and the Client will cache the relevant information of -ROOT- for the next quick access.
- The Client requests the RegionServer address of the .META. table to obtain the address of the RegionServer where the access data is located. The Client will cache the relevant information of the .META. for the next quick access.
- The Client requests access to the address of the RegionServer where the data is located, and obtains the corresponding data.
We can see that the user needs 3 requests to know the real location of the user table, which brings performance degradation to a certain extent. The main reason for using a 3-tier design before 0.96 is that the metadata may need to be large.
The addressing process after version 0.96 is as follows:
- The Client requests ZooKeeper to obtain the address of the RegionServer where .META. is located.
- The Client requests the RegionServer where .META. is located to obtain the address of the RegionServer where the access data is located, and the Client will
cache the relevant information of .META. for the next quick access. - The Client requests the RegionServer where the data is located to obtain the required data.
The main reason for removing -ROOT- is that after HBase-1.x, the default maximum size of each region is 10G, which was 128M before. Now it is found that the 2-layer structure has met the needs of the cluster, and the performance has been relatively improved.
After talking about the Region addressing method, go back to the topic jump table Topo structure. Take the previous version of 0.96. The jump table has three layers. The first layer is the user table, the second layer is the meta table, and the third layer is the root table. Why design such a jump table?
Assuming that the data of a table is extremely large and a server cannot save it, it is necessary to split the table into several parts (for example, 4 parts). If this table has been sorted in order when it is split again, then the split data should be segment by segment, and each segment contains all the data in this segment.
The client now queries the data and scans the data. Which server does it scan? We are not sure, maybe every server has to be scanned again, which leads to poor efficiency!
We can first go to an intermediary agency to confirm which of the 4 paragraphs the data we are looking for is in, create a metadata table meta to store the real data in the Region location, first go to the meta table. When the metadata table becomes very large, it will be divided into multiple segments and placed in the RegionServer. At this time, a ROOT table will be provided to determine the location of the meta table.
This forms a hierarchical relationship, jumping from the ROOT table to the meta table to the specific RegionServer.
Forming a skip table Topo structure reduces the number of scans. Originally, n or 4 scans were required, but now it is 1 scan. The performance is improved and a lot of data can be managed.
How to understand that a lot of data can be managed?
Before hbase1.0, the default size of each Region was 128M, and each metadata was 1KB, so it could store more than 1,000 metadata, and the 3-tier structure would store tens of millions of records. After hbase1.0, the default size of each Region is 10G, and the data that can be stored in the two-tier structure is large enough to meet the needs of the cluster.
2.5 Read cache + write cache
The memory is divided into read cache and write cache, and frequently queried data is placed in the read cache, which can improve efficiency. How does the write cache scan files at the highest rate? It is to use the memory + disk method, first put the data in the memory to sort, and then write the data to the disk, so that the files in the disk are ordered, and then search for the disk files in a binary search, which increases the efficiency.
3 Answer to the second question
In order to ensure data security in a distributed architecture, what does HBase do?
3.1 Memory + Disk
HBase uses a memory + disk storage method. The advantage of doing so is to make a trade-off between data security and data operation efficiency. It pursues both data security and data operation efficiency.
3.2 WAL mechanism
WAL means Write Ahead Log, similar to binlog in MySQL, used for disaster recovery. In order to prevent data from being lost due to an exception in the RegionServer process after data is written to the cache, HBase first writes the data sequentially to HLog (WAL) before writing to the cache. In the unfortunate event of a RegionServer downtime or other abnormalities, this design can perform log playback from HLog to recover data to ensure that data is not lost. The biggest point of HBase failure recovery is how to remedy lost data through HLog playback.
4 summary
Alright, the architecture design of HBase is about to talk about it. Lao Liu mainly talked about why the architecture design of HBase is so awesome. Although the current level may not be as good as you guys, Lao Liu still hopes to become better and to help more self-taught partners.
If you have any related questions, please contact the official account: Lao Liu who works hard. If you think it helped you, you might as well like to follow and support a wave!
