Introduction and configuration of elk log analysis

Chapter 12 ELK Log Analysis System
1. ELK: an open source real-time log analysis platform, consisting of three open source tools: ElasticSearch, Logstash, and Kiabana.
ElasticSearch (ES for short): a distributed real-time analysis search engine with distributed, zero-configuration, Features include automatic discovery, automatic index sharding, index copy mechanism, RESTful interface, multiple data sources, automatic search load, etc.
Logstash: used for log collection, and can process and output data to ES.
Kibana: provides a graphical log analysis interface that can summarize, analyze and search important data logs.
2. Steps of log processing and analysis:
1. Centralize logs Management
2. Format the log (logstash) and output it to ES
3. Index and store the formatted data (ES)
4. Display of previous data
3. Core concepts of Elasticsearch:
1. Near real-time (NRT): The delay in searching for documents is very small (1s)
2. Cluster: It consists of one or more nodes, stores user data on all nodes, and provides indexing and search functions together, and elects the master node.
3. Node: Each node stores data and participates in the indexing and search of the cluster.
4. Index: A place where data is stored (similar to a database), where full-text indexing can be easily performed. The index contains the type of stored data (type, similar to a table), and type contains documents (similar to a record).
5. Sharding and copies: ES divides the index into several shards (the default is 5 shards). A copy is a backup of the index and is used for data redundancy and load sharing.
ES port 9200. You can install the Elasticsearch-head (access port 9100) plug-in to manage the cluster more conveniently. The cluster status is green (green) which represents health. 4.
Logstash: realizes data transmission, format processing, and formatted output. Use pipeline method to collect, process and output logs. Requires java environment. Logstash has three stages: input, processing (filter, optional), and output.
5. Kibana: An open source analysis and visualization platform for ES. Mainly designed to work with ES, it can search and view data stored in ES indexes, and perform advanced data analysis and display through various charts. Access port 5601
Kinaba's main functions:
1. Seamless integration with ES 2. Integrated data 3. Composite data analysis 4. Allow more teams to benefit 5. Flexible interface 6. Simple configuration 7. Visualize multiple data sources 8. Simple data export

---