We know that HDFS was first designed and implemented based on the GFS (Google File System) paper conceptual model.
Then, I went to find out the original paper of GFS and read it carefully. The overall architecture of GFS is as follows:
HDFS refers to it, so most of the architectural design concepts are similar. For example, HDFS NameNode is equivalent to GFS Master, and HDFS DataNode is equivalent to GFS chunkserver.
But there are some differences in details, so this article mainly analyzes the differences.
Write model
HDFS made a simplification when considering the write model, that is, only one writer or appender is allowed at a time.
Under this model, only one client can write or append to the same file at the same time.
GFS allows multiple clients to write or append the same file concurrently at the same time.
Allowing concurrent writes brings more complicated consistency issues.
When multiple clients write concurrently, the order between them cannot be guaranteed, and multiple records that have been successfully appended by the same client may be interrupted.
This means that when a client writes file data continuously, the final distribution of its data in the file may be discontinuous.
The so-called consistency is that for the same file, all the data seen by the client is consistent, no matter from which copy they are read.
If multiple clients are allowed to write a file at the same time, how to ensure that the written data is consistent among multiple copies?
When we talked about HDFS earlier, it only allowed one writer to write multiple copies in a pipelined manner. The writing order is consistent, and the data will remain consistent after the writing is completed.
For multiple clients, it is necessary to allow all clients that write at the same time to write in the same pipeline mode to ensure that the writing order is consistent.
We will analyze this writing process in detail in the next section.
Write process
GFS uses a lease mechanism to ensure sequential consistency in the writing of data across multiple copies.
The GFS Master issues the chunk lease to one of the copies. We call this copy the primary copy, and the other copies are called secondary copies.
The primary copy determines a write order for the chunk, and the secondary copy adheres to this order, thus ensuring global order consistency.
The design of the chunk lease mechanism is mainly to reduce the burden on the Master, and the chunk server where the master copy is located is responsible for the arrangement of the pipeline sequence.
As shown below, we describe this process in detail.
- The client asks the Master to ask which chunkserver holds the lease and the location of other copies.
If no chunkserver holds a lease, it indicates that the chunk has not recently written.
The Master chose to authorize the lease to one of the chunkservers. - Master returns the location information of the client's primary and secondary copies.
The client caches this information for future use.
The client no longer needs to contact the Master, unless the chunkserver where the master copy is located is unavailable or the return lease expires. - The client selects the optimal network sequence to push the data, and the chunkserver caches the data in the internal LRU cache first.
GFS adopts the method of separating the data flow and the control flow, so that the transmission of the data flow can be better scheduled based on the network topology. - Once all replicas confirm that they have received the data, the client will send a write request control command to the master replica.
The master copy assigns consecutive serial numbers to determine the final write order. - The primary copy forwards write requests to all secondary copies, and the secondary copies perform write operations in the order that the primary copy arranges.
- After writing the secondary copy, reply to the primary copy to confirm the completion of the operation.
- Finally, the master copy answers the client. If any error occurs during the writing of any copy, it will be reported to the client, and the client will initiate a retry.
The writing process of GFS and HDFS adopts pipeline method, but HDFS does not separate data flow and control flow.
The transmission order of HDFS data pipeline writing on the network is consistent with the order of final writing of files.
The transmission order of GFS data on the network may not be the same as the order in which the files are finally written.
GFS makes the best compromise in supporting concurrent writing and optimizing network data transmission.
First of all, one thing to confirm is that as one of the most important implementations of GFS, HDFS design goals and GFS are highly consistent. In terms of architecture, block size, metadata, etc., HDFS and GFS are roughly the same. However, in some places, HDFS is different from GFS. Such as: 1. Snapshot: The snapshot function in GFS is very powerful, which can copy files or directories very quickly, and does not affect the current operation (read / write / copy). The method of generating snapshots in GFS is called copy-on-write. In other words, at some point, the backup of the file just points the snapshot file to the original chunk and increases the reference count of the chunk. When the write operation is performed on the chunk, the Chunk Server will copy the chunk block, and the subsequent modification operation On the newly generated chunk. HDFS does not support the snapshot function temporarily, but uses the most basic copy to complete. Imagine that when the data on HBase is being repartitioned (the process is similar to hash balancing), HDFS needs to copy and migrate all the data (P / T level) in it, while GFS only needs snapshots, which is inconvenient! 2. Record appending (append): In terms of data consistency, GFS is theoretically more complete than HDFS. a) GFS provides a relatively loose consistency model. GFS supports both write and record append operations. The write operation allows us to write files randomly. Record append operations make parallel operations safer and more reliable. b) HDFS has the same function as GFS for data flow of write operation. However, HDFS does not support record appending and parallel write operations. The NameNode uses the INodeFileUnderConstruction attribute to mark the file block being operated on, regardless of whether it is read or written. DataNode doesn't even see the lease! Once a file is created, written, and closed, there is no need to modify it. Such a simple model is suitable for Map / Reduce programming. 3. Garbage recycling(GC): a) GFS garbage collection adopts inert recycling strategy, that is, the master will not immediately recycle the file resources deleted by the program. GFS chooses to mark deleted files in a specific form (usually changing the file name to a hidden name that contains time information) so that such files are no longer accessed by ordinary users. The Master will periodically check the file's namespace and delete hidden files some time ago (default 3 days). b) HDFS does not use such a garbage collection mechanism, but adopts a simpler but easier to implement direct deletion method. c) It should be said that delayed recovery and direct deletion have their own advantages. Delayed recovery leaves the way for those "accidentally" delete operations. At the same time, the specific operation of resource recovery is completed when the Master node is idle, which improves the performance of GFS. But delayed collection will take up a lot of storage space, what if some nasty users are bored and have been creating and deleting files? Try to analyze this difference. Some people say that GFS is very complete in function and very powerful, while HDFS is relatively simple in strategy, mainly to facilitate implementation. But in fact, GFS as a storage platform has long been widely deployed within Google, storing data generated or processed by Google services, and is also used for research and development of large-scale data sets. Therefore, GFS is not just a theoretical study, but a concrete realization. As the descendants of GFS and open source implementation, HDFS should be more mature in technology, and it is impossible to simplify functions for "laziness". Therefore, simplification should not be established. I personally believe that the difference between GFS and HDFS is due to the difference between "special" and "common". As we all know, Hadoop is an open source software / framework. At the beginning of the design, the differences in needs of users (for all individuals and enterprises in the world) were considered, such as data-intensive (such as Taobao ’s data storage), Computationally intensive (Baidu's PR algorithm), hybrid, etc. At the beginning of the design, GFS was relatively clear about the goals, all of which were Google's. Therefore, GFS can optimize the performance of its main functions.