Log server application development Organize log information
in the log server and use Log4j to output
Method 1:
The log server generates the log into a file with
flume to obtain the log information in the file
Advantage : The
data is first landed once, so even if there is an accident, it can still be recovered from the file The previous log
Disadvantage :
It takes time to scan data, which has a certain impact on real-time performance.
Pit :
log4j needs to keep writing data, and spooldirsource requires that the directory to be observed cannot have files with the same name and the files cannot be modified. There is a contradiction.
Solution 1:
Use ExecSource to call the tail command to continuously scan the specified file to get the latest log
Solution 2:
Configure Log4j to generate files according to time/size scrolling and configure the file suffix format before and after the scroll is different
, and Flume uses SpoolDirSource to pay attention This folder, and configure ignorePattern to ignore the file format in use
Mode 2:
The log server directly sends logs to the flume Agent through log4j
Advantages:
Direct data transmission ensures high real-time performance
Disadvantages :
No data landing and lack of reliability
Pit :
How to Let log4j send logs directly to the flumeAgent
solution:
Flume natively provides Log4jAppender. Log4j can directly send log information to Flume's AvroSource
Example:
log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
log4j.appender.flume.Hostname = example.com
log4j.appender .flume.Port = 41414
log4j.appender.flume.UnsafeMode = true
Our solution: accept the log submitted by the client in the log server application and write it to flume through Log4jAppender
in the log server and use Log4j to output
Method 1:
The log server generates the log into a file with
flume to obtain the log information in the file
Advantage : The
data is first landed once, so even if there is an accident, it can still be recovered from the file The previous log
Disadvantage :
It takes time to scan data, which has a certain impact on real-time performance.
Pit :
log4j needs to keep writing data, and spooldirsource requires that the directory to be observed cannot have files with the same name and the files cannot be modified. There is a contradiction.
Solution 1:
Use ExecSource to call the tail command to continuously scan the specified file to get the latest log
Solution 2:
Configure Log4j to generate files according to time/size scrolling and configure the file suffix format before and after the scroll is different
, and Flume uses SpoolDirSource to pay attention This folder, and configure ignorePattern to ignore the file format in use
Mode 2:
The log server directly sends logs to the flume Agent through log4j
Advantages:
Direct data transmission ensures high real-time performance
Disadvantages :
No data landing and lack of reliability
Pit :
How to Let log4j send logs directly to the flumeAgent
solution:
Flume natively provides Log4jAppender. Log4j can directly send log information to Flume's AvroSource
Example:
log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
log4j.appender.flume.Hostname = example.com
log4j.appender .flume.Port = 41414
log4j.appender.flume.UnsafeMode = true
Our solution: accept the log submitted by the client in the log server application and write it to flume through Log4jAppender