Spring's Road to God Chapter 52: Spring realizes database read-write separation

Spring's Road to God Chapter 52: Spring realizes database read-write separation

1. Background

Most systems read more and write less. In order to reduce the pressure on the database, multiple slave libraries can be created for the main library. The slave library automatically synchronizes data from the main library. The program sends the write operation to the main library, and the read operation Send it to the slave library for execution.

Today's main goal: to achieve read-write separation through spring .

Read-write separation needs to implement the following two functions:

1. The method of reading is controlled by the caller whether to read from the library or the main library

2. There is a transaction method, and all internal read and write operations go to the main library

2. Think about 3 questions

1. The method of reading is controlled by the caller, whether to read from the slave library or the main library. How to realize it?

You can add a parameter to all read methods to control whether to read from the library or the master library.

2. How to route the data source?

The spring-jdbc package provides an abstract class: AbstractRoutingDataSource, which implements the javax.sql.DataSource interface. We use this class as the data source class. The point is that this class can be used for data source routing and can be configured internally. There are multiple real data sources, and it is up to the developer to decide which data source to use in the end.

There is a map in AbstractRoutingDataSource to store multiple target data sources

private Map<Object, DataSource> resolvedDataSources;

For example, the master-slave library can be stored like this

resolvedDataSources.put("master",主库数据源);
resolvedDataSources.put("salave",从库数据源);

There is also an abstract method in AbstractRoutingDataSource determineCurrentLookupKey, use the return value of this method as a key to find the corresponding data source in the above resolvedDataSources, as the data source for the current operation db

protected abstract Object determineCurrentLookupKey();

3. Where is the separation of reading and writing controlled?

Read-write separation is a general function, which can be realized by spring aop . Add an interceptor, before intercepting the target method, before the target method is executed, get which library you need to go currently, and store this flag in ThreadLocal. Use this flag as the return value of the AbstractRoutingDataSource.determineCurrentLookupKey() method. After the target method is executed in the interceptor, clear this flag from ThreadLocal.

3. Code implementation

3.1. Project structure diagram

[External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-xzUJnT7h-1684759744195)(%E6%96%B0%E5%BB%BA%E6%96%87% E6%9C%AC%E6%96%87%E6%A1%A3/1369022-20211107221309265-2099055565.png)]

3.2、DsType

Indicates the type of data source, and has 2 values, which are used to distinguish whether it is the master library or the slave library.

package com.javacode2018.readwritesplit.base;

public enum DsType {
    
    
    MASTER, SLAVE;
}

3.3、DsTypeHolder

There is a ThreadLocal inside, which is used to record whether the main library or the slave library is currently used, and this flag is placed in dsTypeThreadLocal

package com.javacode2018.readwritesplit.base;

public class DsTypeHolder {
    
    
    private static ThreadLocal<DsType> dsTypeThreadLocal = new ThreadLocal<>();

    public static void master() {
    
    
        dsTypeThreadLocal.set(DsType.MASTER);
    }

    public static void slave() {
    
    
        dsTypeThreadLocal.set(DsType.SLAVE);
    }

    public static DsType getDsType() {
    
    
        return dsTypeThreadLocal.get();
    }

    public static void clearDsType() {
    
    
        dsTypeThreadLocal.remove();
    }
}

3.4. IService interface

This interface acts as a flag. When a class needs to enable read-write separation, it needs to implement this interface. The class that implements this interface will be intercepted by the read-write separation interceptor.

package com.javacode2018.readwritesplit.base;

//需要实现读写分离的service需要实现该接口
public interface IService {
    
    
}

3.5、ReadWriteDataSource

Read and write separate data sources, inherit ReadWriteDataSource, pay attention to its internal determineCurrentLookupKey method, and obtain the flag of whether to go to the main library or the slave library from the above ThreadLocal.

package com.javacode2018.readwritesplit.base;

import org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource;
import org.springframework.lang.Nullable;

public class ReadWriteDataSource extends AbstractRoutingDataSource {
    
    
    @Nullable
    @Override
    protected Object determineCurrentLookupKey() {
    
    
        return DsTypeHolder.getDsType();
    }
}

3.6、ReadWriteInterceptor

The read-write separation interceptor needs to be executed before the transaction interceptor. Through the @1 code, we set the order of this interceptor to Integer.MAX_VALUE - 2, and later we set the order of the transaction interceptor to Integer.MAX_VALUE - 1 , the execution order of the transaction interceptor is from small to small, so ReadWriteInterceptor will be executed before the transaction interceptor org.springframework.transaction.interceptor.TransactionInterceptor.

Since there are mutual calls in the business methods, for example, service2.m2 is called in service1.m1, and service2.m3 is called in service2.m2, we only need to obtain the specific data source to be used before the m1 method is executed. , so the following code will record whether to go to the main library or the slave library when entering the interceptor for the first time.

The following method will get the last parameter of the current target method. The last parameter can be of type DsType. Developers can use this parameter to control whether to use the main library or the slave library.

package com.javacode2018.readwritesplit.base;

import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Pointcut;
import org.springframework.core.annotation.Order;
import org.springframework.stereotype.Component;

import java.util.Objects;

@Aspect
@Order(Integer.MAX_VALUE - 2) //@1
@Component
public class ReadWriteInterceptor {
    
    

    @Pointcut("target(IService)")
    public void pointcut() {
    
    
    }

    //获取当前目标方法的最后一个参数
    private Object getLastArgs(final ProceedingJoinPoint pjp) {
    
    
        Object[] args = pjp.getArgs();
        if (Objects.nonNull(args) && args.length > 0) {
    
    
            return args[args.length - 1];
        } else {
    
    
            return null;
        }
    }

    @Around("pointcut()")
    public Object around(final ProceedingJoinPoint pjp) throws Throwable {
    
    
        //判断是否是第一次进来,用于处理事务嵌套
        boolean isFirst = false;
        try {
    
    
            if (DsTypeHolder.getDsType() == null) {
    
    
                isFirst = true;
            }
            if (isFirst) {
    
    
                Object lastArgs = getLastArgs(pjp);
                if (DsType.SLAVE.equals(lastArgs)) {
    
    
                    DsTypeHolder.slave();
                } else {
    
    
                    DsTypeHolder.master();
                }
            }
            return pjp.proceed();
        } finally {
    
    
            //退出的时候，清理
            if (isFirst) {
    
    
                DsTypeHolder.clearDsType();
            }
        }
    }
}

3.7、ReadWriteConfiguration

Spring configuration class, function

1. @3: Used to register some classes in the com.javacode2018.readwritesplit.base package into the spring container, such as the above interceptor ReadWriteInterceptor

2. @1: Enable the function of spring aop

3. @2: Enable the function of spring to automatically manage transactions. The order of @EnableTransactionManagement is used to specify the order of the transaction interceptor org.springframework.transaction.interceptor.TransactionInterceptor. Here we set the order to Integer.MAX_VALUE - 1, and the above ReadWriteInterceptor The order is Integer.MAX_VALUE - 2, so the ReadWriteInterceptor will be executed before the transaction interceptor.

package com.javacode2018.readwritesplit.base;

import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.EnableAspectJAutoProxy;
import org.springframework.transaction.annotation.EnableTransactionManagement;

@Configuration
@EnableAspectJAutoProxy //@1
@EnableTransactionManagement(proxyTargetClass = true, order = Integer.MAX_VALUE - 1) //@2
@ComponentScan(basePackageClasses = IService.class) //@3
public class ReadWriteConfiguration {
    
    
}

3.8、@EnableReadWrite

This annotation uses two to enable the function of read-write separation. @1 imports ReadWriteConfiguration into the spring container through @Import, which will automatically enable the function of read-write separation. If you need to use read-write separation in your business, you only need to add the @EnableReadWrite annotation to the spring configuration class.

package com.javacode2018.readwritesplit.base;

import org.springframework.context.annotation.Import;

import java.lang.annotation.*;

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Import(ReadWriteConfiguration.class) //@1
public @interface EnableReadWrite {
    
    
}

4. Case

The key code for reading and writing separation is finished, let's use a case to verify the effect.

4.1, execute sql script

Two databases are prepared below: javacode2018_master (main library), javacode2018_slave (slave library)

A t_user table is created in both databases, and a piece of data is inserted respectively. Later, this data is used to verify whether it is the master database or the slave database.

DROP DATABASE IF EXISTS javacode2018_master;
CREATE DATABASE IF NOT EXISTS javacode2018_master;

USE javacode2018_master;
DROP TABLE IF EXISTS t_user;
CREATE TABLE t_user (
  id   INT PRIMARY KEY       AUTO_INCREMENT,
  name VARCHAR(256) NOT NULL DEFAULT ''
  COMMENT '姓名'
);

INSERT INTO t_user (name) VALUE ('master库');

DROP DATABASE IF EXISTS javacode2018_slave;
CREATE DATABASE IF NOT EXISTS javacode2018_slave;

USE javacode2018_slave;
DROP TABLE IF EXISTS t_user;
CREATE TABLE t_user (
  id   INT PRIMARY KEY       AUTO_INCREMENT,
  name VARCHAR(256) NOT NULL DEFAULT ''
  COMMENT '姓名'
);
INSERT INTO t_user (name) VALUE ('slave库');

4.2, spring configuration class

@1: Enable read-write separation

masterDs() method: define the main database data source

slaveDs() method: define slave data source

dataSource(): Define the read-write separation routing data source

There are two more methods to define the JdbcTemplate and the transaction manager. In the method, @Qualifier("dataSource") is used to limit the name of the injected bean to dataSource: that is, the read-write separation routing data source returned by the above dataSource() is injected .

package com.javacode2018.readwritesplit.demo1;

import com.javacode2018.readwritesplit.base.DsType;
import com.javacode2018.readwritesplit.base.EnableReadWrite;
import com.javacode2018.readwritesplit.base.ReadWriteDataSource;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.springframework.transaction.PlatformTransactionManager;

import javax.sql.DataSource;
import java.util.HashMap;
import java.util.Map;

@EnableReadWrite //@1
@Configuration
@ComponentScan
public class MainConfig {
    
    
    //主库数据源
    @Bean
    public DataSource masterDs() {
    
    
        org.apache.tomcat.jdbc.pool.DataSource dataSource = new org.apache.tomcat.jdbc.pool.DataSource();
        dataSource.setDriverClassName("com.mysql.jdbc.Driver");
        dataSource.setUrl("jdbc:mysql://localhost:3306/javacode2018_master?characterEncoding=UTF-8");
        dataSource.setUsername("root");
        dataSource.setPassword("root123");
        dataSource.setInitialSize(5);
        return dataSource;
    }

    //从库数据源
    @Bean
    public DataSource slaveDs() {
    
    
        org.apache.tomcat.jdbc.pool.DataSource dataSource = new org.apache.tomcat.jdbc.pool.DataSource();
        dataSource.setDriverClassName("com.mysql.jdbc.Driver");
        dataSource.setUrl("jdbc:mysql://localhost:3306/javacode2018_slave?characterEncoding=UTF-8");
        dataSource.setUsername("root");
        dataSource.setPassword("root123");
        dataSource.setInitialSize(5);
        return dataSource;
    }

    //读写分离路由数据源
    @Bean
    public ReadWriteDataSource dataSource() {
    
    
        ReadWriteDataSource dataSource = new ReadWriteDataSource();
        //设置主库为默认的库，当路由的时候没有在datasource那个map中找到对应的数据源的时候，会使用这个默认的数据源
        dataSource.setDefaultTargetDataSource(this.masterDs());
        //设置多个目标库
        Map<Object, Object> targetDataSources = new HashMap<>();
        targetDataSources.put(DsType.MASTER, this.masterDs());
        targetDataSources.put(DsType.SLAVE, this.slaveDs());
        dataSource.setTargetDataSources(targetDataSources);
        return dataSource;
    }

    //JdbcTemplate，dataSource为上面定义的注入读写分离的数据源
    @Bean
    public JdbcTemplate jdbcTemplate(@Qualifier("dataSource") DataSource dataSource) {
    
    
        return new JdbcTemplate(dataSource);
    }

    //定义事务管理器，dataSource为上面定义的注入读写分离的数据源
    @Bean
    public PlatformTransactionManager transactionManager(@Qualifier("dataSource") DataSource dataSource) {
    
    
        return new DataSourceTransactionManager(dataSource);
    }
}

4.3、UserService

This class is equivalent to the service we usually write. I directly use JdbcTemplate to operate the database for the method, and the real project operation db will be placed in dao.

getUserNameById method: query name by id.

insert method: Insert data. All internal operations will go to the main database. In order to verify whether the query will also go to the main database. After inserting the data, we will call the this.userService.getUserNameById(id, DsType.SLAVE) method to execute the query Operation, the second parameter uses SLAVE intentionally. If the query has results, it means that the master library is used, otherwise the slave library is used. Why do you need to call getUserNameById through this.userService here?

this.userService is ultimately a proxy object, accessing its internal methods through the proxy object will be intercepted by the interceptor that separates reading and writing.

package com.javacode2018.readwritesplit.demo1;

import com.javacode2018.readwritesplit.base.DsType;
import com.javacode2018.readwritesplit.base.IService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Component;
import org.springframework.transaction.annotation.Propagation;
import org.springframework.transaction.annotation.Transactional;

import java.util.List;

@Component
public class UserService implements IService {
    
    

    @Autowired
    private JdbcTemplate jdbcTemplate;

    @Autowired
    private UserService userService;

    @Transactional(propagation = Propagation.SUPPORTS, readOnly = true)
    public String getUserNameById(long id, DsType dsType) {
    
    
        String sql = "select name from t_user where id=?";
        List<String> list = this.jdbcTemplate.queryForList(sql, String.class, id);
        return (list != null && list.size() > 0) ? list.get(0) : null;
    }

    //这个insert方法会走主库，内部的所有操作都会走主库
    @Transactional
    public void insert(long id, String name) {
    
    
        System.out.println(String.format("插入数据{id:%s, name:%s}", id, name));
        this.jdbcTemplate.update("insert into t_user (id,name) values (?,?)", id, name);
        String userName = this.userService.getUserNameById(id, DsType.SLAVE);
        System.out.println("查询结果：" + userName);
    }

}

4.4. Test cases

package com.javacode2018.readwritesplit.demo1;

import com.javacode2018.readwritesplit.base.DsType;
import org.junit.Before;
import org.junit.Test;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;

public class Demo1Test {
    
    

    UserService userService;

    @Before
    public void before() {
    
    
        AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext();
        context.register(MainConfig.class);
        context.refresh();
        this.userService = context.getBean(UserService.class);
    }

    @Test
    public void test1() {
    
    
        System.out.println(this.userService.getUserNameById(1, DsType.MASTER));
        System.out.println(this.userService.getUserNameById(1, DsType.SLAVE));
    }

    @Test
    public void test2() {
    
    
        long id = System.currentTimeMillis();
        System.out.println(id);
        this.userService.insert(id, "张三");
    }
}

The test1 method executes two queries, querying the main library and the slave library respectively, and outputs:

master库
slave库

Isn't it cool, the developers themselves control whether to use the main library or the slave library.

The execution results of test2 are as follows. It can be seen that the data just inserted has been queried, indicating that all operations in insert go through the main database.

1604905117467
插入数据{id:1604905117467, name:张三}
查询结果：张三

5. Case source code

git地址：
https://gitee.com/javacode2018/spring-series

本文案例对应源码：
    spring-series\lesson-004-readwritesplit

Everyone star it, all series of codes will be in this.

Source: https://mp.weixin.qq.com/s?__biz=MzA5MTkxMDQ4MQ==&mid=2648938118&idx=2&sn=baef96540a8936e49db0bfe62f909f24&scene=21#wechat_redirect

ster library
slave library

是不是很爽，由开发者自己控制具体走主库还是从库。

test2 执行结果如下，可以看出查询到了刚刚插入的数据，说明 insert 中所有操作都走的是主库。

1604905117467
Insert data {id: 1604905117467, name: Zhang San}
Query result: Zhang San

## 5、案例源码

Git address:
https://gitee.com/javacode2018/spring-series

The source code corresponding to the case in this article:
spring-series\lesson-004-readwritesplit

大家 star 一下，所有系列代码都会在这个里面。

来源：https://mp.weixin.qq.com/s?__biz=MzA5MTkxMDQ4MQ==&mid=2648938118&idx=2&sn=baef96540a8936e49db0bfe62f909f24&scene=21#wechat_redirect