We all know that Kafka very fast, other messaging middleware faster than the vast majority of the market. Here to study under so why Kafka so fast (of course not because it took Scala).
Kafka message is stored or cached on disk, generally read and write data on the disk that will degrade performance because the address would be more time consuming.
But in fact, Kafka One of the features is a high throughput, even an ordinary server, Kafka can easily support for millions of write requests per second level, more than most of the messaging middleware. This characteristic makes Kafka widely used in log processing massive data in the scene. So why Kafka speed so fast, can read and write data from both data analysis.
Kafka data write (producer)
Producers (Producer) is responsible for submitting data to Kafka, Kafka will receive the message are written to disk, it can be considered that it will not lose data.
In order to optimize the write speed, Kafka using two techniques, one is sequentially written, one is MMFile.
Sequential Write
Disk read and write speed depends on how you use it, it can generally be divided into reading and writing sequential or random read and write.
Because the hard disk is a mechanical structure, each goes through a read address [->] of the writing process, the addressing of which is a time-consuming mechanical action, so the hard disk hate random I / O, I favorite order / O. In order to improve read and write speed of the hard disk, Kafka is to use sequential I / O. For Linux and optimize disk read and write will be more, including read-ahead, write-behind, and disk caching. More on Java memory management and garbage collection will be optimized because if the time to do these operations in memory, can cause a lot of memory overhead Java objects, and the other is with the increase of heap memory data, the Java GC time becomes very long.
Therefore it can be concluded using the disk operation has the following benefits:
1. The sequential read and write disk exceeds random access memory.
GC 2.JVM of low efficiency, large memory footprint, use the disk to avoid this problem.
3. After cold start, it is still available on the disk cache (memory data will be cleared once the shutdown, persisted to disk is not).
The figure shows how Kafka to write data, each Partition in fact a file, after receiving the message Kafka will insert data into the end of the file (the dotted rectangle section).
However, this method has a drawback: There is no way to delete the data. Kafka once the data is not deleted, all the data will only be retained, each consumer (Consumer) for each Topic has an offset is used to indicate to read the first few data.
Figure above two consumers, Consumer1 two offset corresponding Partition0 and Partition1 (assuming each is a Topic Partition); Consumer2 has an offset corresponding to the offset Partition2 client is responsible for keeping the SKD, Kafka's Broker. completely ignoring the existence of this thing; under normal circumstances SDK will save it Zookeeper inside. (It is necessary to provide the address Zookeeper to Consumer).
If the data is completely deleted, will certainly lead to a hard disk full, so Kafka provides two strategies to delete data, one is based on the time, the second is based on Partition file size. Specific configuration can be found in its configuration file.
MMFiles（Memory Mapped Files）
Even if the order is written to disk, disk access speed is still impossible to catch memory. So Kafka is not a real-time data is written to disk, it makes full use of modern paging storage operating system to use the memory in order to improve I / O efficiency. Memory Mapped Files (later referred to as an MMAP) is also translated into the memory-mapped files, the operating system 64 may generally be a data file representing the 20G. It works to achieve the files directly to the direct mapping of physical memory using the operating system's Page. After completion of the mapping of the physical memory of your operation will be synchronized to your hard drive (operating system when appropriate).
By MMAP, the process can be read like a hard disk read and write the same memory (of course, the virtual machine memory), do not have the relationship as memory size, because there are virtual memory for us reveal all the details. This way you can get a lot of I / O upgrade, eliminating the overhead of user space to kernel space to copy (read calls the file will have the first data into the kernel memory space, and then copied to the user space memory). But this also has an obvious flaw: not reliable because the data written in the MMAP was not actually written to the hard disk, the operating system before the data is actually written to when the program automatically call flush command hard disk. Kafka provides a parameter to control is not active prducer.type flush, if Kafka written after the MMAP and then return immediately flush Producer, called the synchronization (sync); return immediately after the Producer if Kafka writes to MMAP not call flush , called asynchronous (async).
MMAP is actually a function in Linux is used to implement the memory map. NIO provides the Java class that implements a memory mapping MappedByteBuffer (Kafka thus soiled Java light, instead Scala).
Kafka read data (consumer)
Why Kafka can use disk files so fast - with the use of a memory faster than hard disk, this is definitely a violation of common sense, because Kafka cheating, whether written order or MMAP, in fact, all the preparatory work before Kafka cheating.
Zero Copy
Kafka uses sendfile of Zero Copy-based Web Server to improve the speed of static files.
Under the traditional model, a file is read from the hard disk:
1. The read function call, the file data is to copy the kernel buffer (read system call, put the DMA, so that a kernel space).
2.read function returns, the file copy data from the kernel buffer to the user buffer.
3.write function call, the file copy data from the user buffer to the kernel buffer associated with the Socket.
4. Socket buffer to copy the data from the relevant protocol engine (NIC).
More detail is the traditional way of read / write mode for network transmission, we can see that in this process, the data file is actually a copy after four operations: HDD -> kernel buf-> User buf-> socket-related buffer -> protocol engine. The sendfile system call is to provide a method of reducing the above multiple copy, improve the performance of file transfer. Kafka in kernel version 2.1, cited sendfile system call, in order to simplify data transfer between the network and two local papers. Sendfile introduced not only reduces the replication of data, also reduces context switching: sendfile (socket, file, len).
Run process is as follows:
1.sendfile system calls, file data is to copy the kernel buffer.
2. And then related to the kernel socket buffers from the kernel buffer copy.
3. Finally socket buffer copy to the relevant protocol engine.
Compared to traditional read / write mode, the introduction of the 2.1 version of the kernel sendfile kernel buffer has been reduced to the user buffer, and then copy the file from the user to the socket buffer associated buffer, and after the kernel version 2.4, the file descriptor results It is changed, sendfile to achieve a more simple way, once again reducing the copy operation.
Among apache, nginx, lighttpd web server and so on, have a sendfile-related configuration, use sendfile can significantly improve file transfer performance.
Kafka all messages are stored in a a file, when the consumer needs to send data directly to the consumer Kafka file, as the file read and write with the MMAP, pass it directly sendfile.
Java's NIO provides FileChannle, it transferTo () method and transferFrom () method is Zero Copy.
Kafka batch compression
In many cases, the bottleneck is not CPU or disk, but the network IO, especially the need to send messages between the data center WAN data lines. Data compression consumes a small amount of CPU resources, but for kafka, network IO should be considered.
1. If each message is compressed, but the compression rate is relatively low, so Kafka uses a batch compression, compression coming together multiple messages rather than a single message compression.
2.Kafka allows recursive set of messages, and bulk messages may be transmitted in the log can be maintained by compressing the compression format form, until the consumer is decompressed.
3.Kafka supports multiple compression protocols, including Gzip and Snappy compression protocol.
Fast speed of Kafka secret - Cheating
Kafka put all the messages are stored in a file in a, when consumers need to send the data directly to the Kafka papers to consumers. This is the secret lies, such as: news combined together 10W amount of data is 10MB, and Kafka similar manner to send files directly thrown out, if the network between consumers and producers is very good (just a little normal network 10MB is not a little thing ... the Internet is home to 100Mbps a), 10MB may only need 1s. So the answer is TPS --10W of, Kafka 10W messages processed per second.
You might say: it is impossible to go sent the entire file? There are a number of unwanted messages inside it? Yes, Kafka as a senior [molecular] I naturally wanted to cheat cheat do have to force the grid. Zero Copy corresponds sendfile this function (to Linux for example), this function accepts:
1.out_fd as an output (typically timely handle socket).
2.in_fd as the input file handle.
3.off_t represents in_fd offset (where to start reading).
4.size_t represent the number of reads.
Yes, Kafka is MMAP as file reading and writing, which is a file handle, so a direct pass it on to sendfile; offset Ye Hao resolved, users can keep themselves this offset, each request will be sent to this offset. (Remember zookeeper put in?); The amount of data easier to solve, and if consumers want faster, throw on all consumers. If this is done under normal circumstances consumers will definitely direct was crushed to death ; so Kafka offers two ways --Push, I throw all of you, you're dead no matter what I do; Pull, Okay, you tell me how many you need, I'll give you the number.
to sum up
Kafka speed secret that it all the messages into a batch file, and reasonable bulk compression to reduce the loss of network IO, by MMAP improve I / O speed. When writing data, since the speed of the individual so Partition (partition) is added to the end of the optimal; sendfile read data when direct input with violence. Ali RocketMQ is this model, but is written in Java.
"You can be confused, but please do not waste."