foreword
With the prevalence of microservices, many companies have split the system into many microservices according to business boundaries, when troubleshooting logs. Because the business link runs through many microservice nodes, it becomes difficult to locate the log of a request and the logs of upstream and downstream services.
At this time, many children's shoes will start to consider distributed tracking systems such as SkyWalking and Pinpoint to solve the problem. Based on the OpenTracing specification, they are usually non-invasive and have a relatively friendly management interface to query the link span.
However, it takes a certain period of time to build a distributed tracking system, familiarize with and promote the system to the whole company, and it involves the storage cost of link span nodes. Is full collection or partial collection? If the data is collected in full, taking SkyWalking's storage as an example, at least 5 nodes are required to build an ES cluster. This requires increased server costs. Moreover, if there are many microservice nodes, it is quite normal to generate tens of gigabytes or hundreds of gigabytes of data in one day. If you want to keep it for a longer time, you also need to increase the cost of the server disk.
Therefore, the open source project introduced this time is TLog. Its background originates from the desire to be a lightweight log tracking solution, covering the current popular log frameworks and RPC frameworks, allowing users to access at the least cost to solve micro The pain point of log tracking in the service system.
Of course the distributed tracing system is a final solution, if your company has already implemented the distributed tracing system, then TLog is not suitable.
TLog provides the easiest way to solve the log tracking problem. It does not collect logs and does not require additional storage space. It just automatically tags your business logs and automatically generates a TraceId that runs through your microservices. the entire link. And provide upstream and downstream node information. It is suitable for small and medium-sized enterprises and corporate projects that want to quickly solve log tracking problems.
To this end, I have adapted three major log frameworks for TLog, which supports automatic detection and adaptation. It supports the three RPC frameworks of dubbo, dubbox and spring cloud. More importantly, if your project is connected to TLog, you may not need it for ten minutes :)
TLog projects and features
Project address: https://gitee.com/bryan31/TLog
TLog has the following features:
- Complete lightweight microservice log tracking by tagging logs
- Non-intrusive design of business code, easy to use, can be accessed in 10 minutes
- Support the three common log4j, log4j2, logback three log frameworks, and provide automatic detection to complete the adaptation
- Support dubbo, dubbox, springcloud three RPC frameworks
- Configuration of custom templates that support log tags
- Virtually no performance loss
How to quickly access
TLog supports the automatic assembly of springboot. In the springboot environment, you only need the following two steps to access!
rely
<dependency>
<groupId>com.yomahub</groupId>
<artifactId>tlog-all-spring-boot-starter</artifactId>
<version>1.0.0</version>
</dependency>
At present, the jar package has been uploaded to the central warehouse and can be directly relied on to
Log framework adaptation
You only need to add a line of code to your startup class to automatically detect the Log framework used by your project and enhance it. Currently, it supports three log frameworks: log4j, log4j2, and logback.
@SpringBootApplication
public class Runner {
static {AspectLogEnhance.enhance();}//进行日志增强，自动判断日志框架
public static void main(String[] args) {
SpringApplication.run(Runner.class, args);
}
}
It should be noted that because this is implemented with javassit, bytecode enhancement needs to be performed before the jvm loads the classes corresponding to the logging framework. So use static block here. However, this one-line automatic adaptation may not take effect in the following two cases:
1. The Springboot/Spring startup class is added to the log definition, which will not take effect, because the classload has already loaded the log log framework before loading the static block.
2. You started it with an external container such as tomcat/jboss/jetty (it has no effect on the springboot built-in container)
For these two cases, TLog also provides independent adaptation of the three major log frameworks, as long as you replace the corresponding layout class in the log configuration file, which will also be introduced below
Adaptation of RPC framework
In the Springboot environment, TLog will automatically detect the RPC framework you use and adapt it automatically.
After completing the above 2 steps, it is ok, and the final effect is as follows (here takes dubbo+log4j as an example):
Consumer side code:
log print:
2020-09-16 18:12:56,748 [WARN] [TLOG]重新生成traceId[7161457983341056] >> com.yomahub.tlog.web.TLogWebInterceptor:39
2020-09-16 18:12:56,763 [INFO] <7161457983341056> logback-dubbox-consumer:invoke method sayHello,name=jack >> com.yomahub.tlog.example.dubbox.controller.DemoController:22
2020-09-16 18:12:56,763 [INFO] <7161457983341056> 测试日志aaaa >> com.yomahub.tlog.example.dubbox.controller.DemoController:23
2020-09-16 18:12:56,763 [INFO] <7161457983341056> 测试日志bbbb >> com.yomahub.tlog.example.dubbox.controller.DemoController:24
Provider code:
log print:
2020-09-16 18:12:56,854 [INFO] <7161457983341056> logback-dubbox-provider:invoke method sayHello,name=jack >> com.yomahub.tlog.example.dubbo.service.impl.DemoServiceImpl:15
2020-09-16 18:12:56,854 [INFO] <7161457983341056> 测试日志cccc >> com.yomahub.tlog.example.dubbo.service.impl.DemoServiceImpl:16
2020-09-16 18:12:56,854 [INFO] <7161457983341056> 测试日志dddd >> com.yomahub.tlog.example.dubbo.service.impl.DemoServiceImpl:17
It can be seen that after simple access, each request between each microservice has a globally unique traceId running through it, which is valid for all log outputs, which makes it easier to locate the log chain of a request. .
Change the format of log labels
TLog allows users to customize the format of log labels. TLog only prints traceId by default, and prints it with the template of <$traceId>. Of course, you can customize its template. Additional headers can also be added
You only need to define the following in springboot's application.properties, change the format of the label, and you can print according to the template you define
tlog.pattern=[$preApp][$preIp][$traceId]
$preApp: upstream microservice node name
$preIp: IP address of the upstream microservice
$traceId: Globally unique tracking ID
Individual adaptation of the Log framework
If your automated logging detection fails or you are using an external container, you need to modify the logging framework configuration in your project, and the modification method is also very simple.
Log4j configuration file enhancements
Just layoutreplace the implementation class
The templates of each company's Log4J are similar, only the xml example is given here
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration>
<appender name="stdout" class="org.apache.log4j.ConsoleAppender">
<!--替换成AspectLog4jPatternLayout-->
<layout class="com.yomahub.tlog.core.enhance.log4j.AspectLog4jPatternLayout">
<param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss,SSS} [%p] %m >> %c:%L%n"/>
</layout>
</appender>
<appender name="fileout" class="org.apache.log4j.DailyRollingFileAppender">
<param name="File" value="./logs/test.log"/>
<!--替换成AspectLog4jPatternLayout-->
<layout class="com.yomahub.tlog.core.enhance.log4j.AspectLog4jPatternLayout">
<param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss,SSS} [%p] %m >> %c:%L%n"/>
</layout>
</appender>
<root>
<priority value="info" />
<appender-ref ref="stdout"/>
<appender-ref ref="fileout"/>
</root>
</log4j:configuration>
Logback configuration file enhancements
The replaced encoderimplementation class or layoutthe replaced implementation class is fine.
The xml example is given below:
<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="false">
<property name="APP_NAME" value="logtest"/>
<property name="LOG_HOME" value="./logs" />
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<!--替换成AspectLogbackEncoder-->
<encoder class="com.yomahub.tlog.core.enhance.logback.AspectLogbackEncoder">
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n</pattern>
</encoder>
</appender>
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<File>${LOG_HOME}/${APP_NAME}.log</File>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<FileNamePattern>${LOG_HOME}/${APP_NAME}.log.%d{yyyy-MM-dd}.%i.log</FileNamePattern>
<MaxHistory>30</MaxHistory>
<maxFileSize>1000MB</maxFileSize>
</rollingPolicy>
<!--替换成AspectLogbackEncoder-->
<encoder class="com.yomahub.tlog.core.enhance.logback.AspectLogbackEncoder">
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n</pattern>
</encoder>
</appender>
<!-- 日志输出级别 -->
<root level="INFO">
<appender-ref ref="STDOUT" />
<appender-ref ref="FILE" />
</root>
</configuration>
Log4j2 configuration file enhancements
Since log4J2 is implemented in the form of plug-ins, log4J2 has the function of automatically scanning plug-ins. So it takes effect without making any changes to the config file.
afterword
TLog is essentially a log framework, but it has the log tracking function that supports the RPC framework. It took me about a week to write the framework, trying to be the most lightweight log tracking solution, allowing users to access at the least cost. TLog not only supports springboot, but also supports non-springboot projects. For specific configuration methods, please check the documentation on the project homepage.
I also wrote a detailed example project for TLog, covering almost all the scenarios involved, and the specific example project address is also on the project homepage.
If you also encounter difficulties in log tracking in your project, you might as well try TLog. I hope this open source framework can help you. Open source is not easy. If you like it, please help to star. Your support is the biggest thing for me to move forward. power. If you have any questions, please feel free to contact me.
Follow the author
"Yuanren Tribe" is a technology sharing account that insists on being original. I hope you can pay attention to me. I will publish a practical original technical article every week to walk with you and grow with you. Follow the official account to reply tlogto join the group chat, I will patiently answer every question you have in use, and will also maintain and iterate on this project for a long time.
