Storage
Physically the topic into one or more patition (corresponding to server.properties in num.partitions = 3 configuration), each patition corresponds to a folder (the folder storing the physical patition all messages and index files), as follows :
[hadoop@masterlogs]$ ll
drwxrwxr-x. 2 hadoop hadoop 4096 8月 6 14:37 first-0
drwxrwxr-x. 2 hadoop hadoop 4096 8月 6 14:35 first-1
drwxrwxr-x. 2 hadoop hadoop 4096 8月 6 14:37 first-2
[hadoop@masterlogs]$ cd first-0
[hadoop@masterfirst-0]$ ll
-rw-rw-r--. 1 hadoop hadoop 10485760 8月 6 14:33 00000000000000000000.index
-rw-rw-r--. 1 hadoop hadoop 219 8月 6 15:07 00000000000000000000.log
-rw-rw-r--. 1 hadoop hadoop 10485756 8月 6 14:33 00000000000000000000.timeindex
-rw-rw-r--. 1 hadoop hadoop 8 8月 6 14:37 leader-epoch-checkpoint
3.2.2 Storage Policy
Regardless of whether the message is consumed, Kafka used to live will retain all messages. There are two strategies you can delete the old data:
1) Time-based: log.retention.hours = 168
2) based on the magnitude: log.retention.bytes = 1073741824
Note that because Kafka read a particular message, the time complexity is O (1) , that is, regardless of the file size, so here delete outdated files and improve Kafka independent of performance.
. 3 . 2 . . 3 the Zookeeper storage structure
Note : Producer not zk registered , consumers zk registration.
The above article content from "Monkey Wisdom Education": www.wukongone.com, "Wukong wisdom education" is a professional video teaching large data network, exclusive tailor your future.