说明
通过上篇博文《MyBatis多数据源配置》,学习了MyBatis多数据源的配置。在实际环境中,数据库一般配置为主从结构的形式,甚至是一主多从的形式,这时,我们希望读操作都在从数据库中进行,增删改操作都在主数据库中进行,希望mybatis能够动态地选择数据库。通过本篇博文,学习记录下通过注解+AOP和只使用AOP两种方式实现动态数据源配置。文中使用主机的mysql作为主数据库,虚机中的mysql作为从数据库。
正文
1.通过spirngboot构建项目
主要依赖 mybatis+mysql+aspect
<dependency>
<groupId>org.mybatis.spring.boot</groupId>
<artifactId>mybatis-spring-boot-starter</artifactId>
<version>1.3.2</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjweaver</artifactId>
<version>1.8.7</version>
</dependency>
2.在application.properties配置数据库
#主库
datasource.master.driverClassName=com.mysql.jdbc.Driver
datasource.master.url=jdbc:mysql://ip:3306/dbname?characterEncoding=UTF-8
datasource.master.username=**
datasource.master.password=**
#从库
datasource.slave.driverClassName=com.mysql.jdbc.Driver
datasource.slave.url=jdbc:mysql://ip:3306/dbname?characterEncoding=UTF-8
datasource.slave.username=**
datasource.slave.password=**
3.在启动类关闭自动配置数据源
这个很重要,如果没有关闭,在启动程序时会发生循环依赖的错误
@EnableAutoConfiguration(exclude = {DataSourceAutoConfiguration.class})
@SpringBootApplication
public class DynamicDatasourceAnnoApplication {
public static void main(String[] args) {
SpringApplication.run(DynamicDatasourceAnnoApplication.class, args);
}
}
4.配置数据源
要实现数据源的动态选择,AbstractRoutingDataSource这个抽象类很重要,通过阅读源码,发现数据源在该类中以key-value的形式存储在map中,其中在determineTargetDataSource()方法中,通过determineCurrentLookupKey()方法得到key,在map中寻找需要的数据源,而determineCurrentLookupKey()是抽象方法,我们可以通过继承这个抽象类实现该方法,配置我们期望的数据源。
public abstract class AbstractRoutingDataSource extends AbstractDataSource implements InitializingBean {
protected DataSource determineTargetDataSource() {
Assert.notNull(this.resolvedDataSources, "DataSource router not initialized");
Object lookupKey = this.determineCurrentLookupKey();
DataSource dataSource = (DataSource)this.resolvedDataSources.get(lookupKey);
if (dataSource == null && (this.lenientFallback || lookupKey == null)) {
dataSource = this.resolvedDefaultDataSource;
}
if (dataSource == null) {
throw new IllegalStateException("Cannot determine target DataSource for lookup key [" + lookupKey + "]");
} else {
return dataSource;
}
}
protected abstract Object determineCurrentLookupKey();
在了解了基本原理后,开始配置数据源,继承AbstractRoutingDataSource 类
DynamicDataSource
public class DynamicDataSource extends AbstractRoutingDataSource {
@Override
protected Object determineCurrentLookupKey() {
return DynamicDataSourceHolder.getType();
}
}
实现了该方法后,如何返回一个key,这个key从何而来?
每个请求都是一个线程,我们可以再创建一个类,其中使用ThreadLocal线程局部变量保存每个线程所请求方法对应的key,通过该类的静态方法返回该请求的key
DynamicDataSourceHolder
public class DynamicDataSourceHolder {
private static final ThreadLocal<DataSourceType> dataSourceHolder = new ThreadLocal<>();
private static final Set<DataSourceType> dataSourceTypes = new HashSet<>();
static {
dataSourceTypes.add(DataSourceType.MASTER);
dataSourceTypes.add(DataSourceType.SLAVE);
}
public static void setType(DataSourceType dataSourceType){
if(dataSourceType == null){
throw new NullPointerException();
}
dataSourceHolder.set(dataSourceType);
}
public static DataSourceType getType(){
return dataSourceHolder.get();
}
public static void clearType(){
dataSourceHolder.remove();
}
public static boolean containsType(DataSourceType dataSourceType){
return dataSourceTypes.contains(dataSourceType);
}
}
其中使用了枚举表示不同类型的数据源
DataSourceType
public enum DataSourceType {
MASTER,
SLAVE,
}
创建了关键类DynamicDataSource后,开始配置数据源
DynamicDataSourceConfig
@Configuration
public class DynamicDataSourceConfig {
@Bean(name = "masterDataSource")
@ConfigurationProperties(prefix = "datasource.master")
public DataSource masterDataSource(){
return DataSourceBuilder.create().build();
}
@Bean(name = "slaveDataSource")
@ConfigurationProperties(prefix = "datasource.slave")
public DataSource slaveDataSource(){
return DataSourceBuilder.create().build();
}
@Bean(name = "dynamicDataSource")
@Primary
public DataSource dynamicDataSource(@Qualifier(value = "masterDataSource") DataSource masterDataSource,
@Qualifier(value = "slaveDataSource") DataSource slaveDataSource){
DynamicDataSource dynamicDataSource = new DynamicDataSource();
dynamicDataSource.setDefaultTargetDataSource(masterDataSource);
Map<Object,Object> dataSourceMap = new HashMap<>();
dataSourceMap.put(DataSourceType.MASTER,masterDataSource);
dataSourceMap.put(DataSourceType.SLAVE,slaveDataSource);
dynamicDataSource.setTargetDataSources(dataSourceMap);
return dynamicDataSource;
}
}
配置好数据源后,开始配置mybatis
MyBatisConfig
@Configuration
@MapperScan(basePackages = "com.example.dynamic_datasource_anno.db.dao")
public class MyBatisConfig {
@Autowired
private DataSource dataSource;
@Bean(name = "sqlSessionFactory")
public SqlSessionFactory sqlSessionFactory() throws Exception {
SqlSessionFactoryBean sessionFactoryBean = new SqlSessionFactoryBean();
sessionFactoryBean.setDataSource(dataSource);
sessionFactoryBean.setTypeAliasesPackage("com.example.dynamic_datasource_anno.db.pojo");
Resource[] resources = new PathMatchingResourcePatternResolver()
.getResources("classpath:com/example/dynamic_datasource_anno/db/mapper/*Mapper.xml");
sessionFactoryBean.setMapperLocations(resources);
return sessionFactoryBean.getObject();
}
}
数据源和mybatis配置好后,开始配置切面,动态地选择都在此类中完成。
第一种 Anno+Aspect
创建注解DBType,默认为主数据库
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface DBType {
DataSourceType value() default DataSourceType.MASTER;
}
切面DynamicDataSourceAspect
@Aspect
@Component
public class DynamicDataSourceAspect {
private static final Logger logger = LoggerFactory.getLogger(DynamicDataSourceAspect.class);
@Before("@annotation(dbType)")
public void changeDataSourceType(JoinPoint joinPoint, DBType dbType){
DataSourceType curType = dbType.value();
if(!DynamicDataSourceHolder.containsType(curType)){
logger.info("指定数据源[{}]不存在,使用默认数据源-> {}",dbType.value(),joinPoint.getSignature());
}else{
logger.info("use datasource {} -> {}",dbType.value(),joinPoint.getSignature());
DynamicDataSourceHolder.setType(dbType.value());
}
}
@After("@annotation(dbType)")
public void restoreDataSource(JoinPoint joinPoint, DBType dbType){
logger.info("use datasource {} -> {}",dbType.value(),joinPoint.getSignature());
DynamicDataSourceHolder.clearType();
}
}
配置好切面后,可以在Mapper接口的方法上使用@DBType指定要使用的数据源,使用自动代码生成插件将表中数据生成对应的实体类,mapper文件,mapper接口,关于插件的使用,请看之前的博文《使用c3p0连接池集成mybatis及mybatis自动代码生成插件》。
此处,我在service层使用注解
@DBType(DataSourceType.SLAVE)
public void select(){
User user = userMapper.selectByPrimaryKey(1);
System.out.println(user.getId() + "--" + user.getName() + "==" + user.getGender());
}
@DBType(DataSourceType.MASTER)
public void insert(User user){
userMapper.insertSelective(user);
}
第二种 只使用Aspect,判断sql类型
在此方式中,没有再使用枚举、注解,数据源和mybatis的配置与之前相似,不再赘述,详细配置可以查看此方式的源码。主要不同在于切面的配置,此配置中通过切点判断请求的SqlCommandType来选择数据源
DynamicDataSourceAspect
@Aspect
@Component
public class DynamicDataSourceAspect {
private static final Logger logger = LoggerFactory.getLogger(DynamicDataSourceAspect.class);
@Autowired
@Qualifier("sqlSessionFactory")
private SqlSessionFactory sqlSessionFactory;
// first * any return value
// second * any class name of the dao package
// third * any method name of the class
// (..) any number of parameters
// @Pointcut("within(com.example.dynamic_datasource_method.db.dao..*)")
@Pointcut("execution(* com.example.dynamic_datasource_method.db.dao.*.*(..))")
public void declareJoinPoint(){
}
@Before("declareJoinPoint()")
public void matchDataSoruce(JoinPoint point){
//得到被代理对象
Object target = point.getTarget();
//目标方法名
String methodName = point.getSignature().getName();
Class<?>[] interfaces = target.getClass().getInterfaces();
Class<?>[] parametersTypes = ((MethodSignature)point.getSignature()).getMethod().getParameterTypes();
try {
Method method = interfaces[0].getMethod(methodName,parametersTypes);
String key = interfaces[0].getName() + "." + method.getName();
//sql 的类型
SqlCommandType type = sqlSessionFactory.getConfiguration().getMappedStatement(key).getSqlCommandType();
//查询从库 增删改主库
if(type == SqlCommandType.SELECT){
logger.info("use slaveDataSource");
DynamicDataSourceHolder.setType("slave");
}
if(type == SqlCommandType.DELETE || type == SqlCommandType.UPDATE || type == SqlCommandType.INSERT){
logger.info("use masterDataSource");
DynamicDataSourceHolder.setType("master");
}
} catch (NoSuchMethodException e) {
e.printStackTrace();
}
}
@After("declareJoinPoint()")
public void restoreDataSource(JoinPoint joinPoint){
logger.info(DynamicDataSourceHolder.getType() + " change to master" + joinPoint.getSignature());
DynamicDataSourceHolder.clearType();
}
}
通过此方式,可以直接调用mapper接口的方法而不需要注解直接动态地选择数据源
问题
当在启动类没有关闭数据源的自动配置时,启动项目会报以下错误:
Description:
The dependencies of some of the beans in the application context form a cycle:
dynamicDataSourceAspect (field private org.apache.ibatis.session.SqlSessionFactory com.example.dynamic_datasource_method.config.mybatis.DynamicDataSourceAspect.sqlSessionFactory)
↓
myBatisConfig (field private javax.sql.DataSource com.example.dynamic_datasource_method.config.mybatis.MyBatisConfig.dataSource)
┌─────┐
| dynamicDataSource defined in class path resource [com/example/dynamic_datasource_method/config/mybatis/DynamicDataSourceConfig.class]
↑ ↓
| masterDataSource defined in class path resource [com/example/dynamic_datasource_method/config/mybatis/DynamicDataSourceConfig.class]
↑ ↓
| dataSourceInitializer
└─────┘
2018-09-08 10:04:09.537 ERROR 20372 --- [ main] o.s.test.context.TestContextManager : Caught exception while allowing TestExecutionListener [org.springframework.test.context.web.ServletTestExecutionListener@68ceda24] to prepare test instance [com.example.dynamic_datasource_method.DynamicDatasourceMethodApplicationTests@65ae095c]
java.lang.IllegalStateException: Failed to load ApplicationContext
at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:124) ~[spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.test.context.support.DefaultTestContext.getApplicationContext(DefaultTestContext.java:83) ~[spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.test.context.web.ServletTestExecutionListener.setUpRequestContextIfNecessary(ServletTestExecutionListener.java:189) ~[spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.test.context.web.ServletTestExecutionListener.prepareTestInstance(ServletTestExecutionListener.java:131) ~[spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.test.context.TestContextManager.prepareTestInstance(TestContextManager.java:230) ~[spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.createTest(SpringJUnit4ClassRunner.java:228) [spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner$1.runReflectiveCall(SpringJUnit4ClassRunner.java:287) [spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12]
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.methodBlock(SpringJUnit4ClassRunner.java:289) [spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:247) [spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:94) [spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12]
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61) [spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70) [spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12]
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:191) [spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.junit.runner.JUnitCore.run(JUnitCore.java:137) [junit-4.12.jar:4.12]
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) [junit-rt.jar:na]
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47) [junit-rt.jar:na]
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) [junit-rt.jar:na]
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) [junit-rt.jar:na]
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'dynamicDataSourceAspect': Unsatisfied dependency expressed through field 'sqlSessionFactory'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'myBatisConfig': Unsatisfied dependency expressed through field 'dataSource'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'dynamicDataSource' defined in class path resource [com/example/dynamic_datasource_method/config/mybatis/DynamicDataSourceConfig.class]: Unsatisfied dependency expressed through method 'dynamicDataSource' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'masterDataSource' defined in class path resource [com/example/dynamic_datasource_method/config/mybatis/DynamicDataSourceConfig.class]: Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSourceInitializer': Invocation of init method failed; nested exception is org.springframework.beans.factory.BeanCurrentlyInCreationException: Error creating bean with name 'dynamicDataSource': Requested bean is currently in creation: Is there an unresolvable circular reference?
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:588) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1272) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867) ~[spring-context-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543) ~[spring-context-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:693) ~[spring-boot-1.5.16.BUILD-20180902.070551-14.jar:1.5.16.BUILD-SNAPSHOT]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:360) ~[spring-boot-1.5.16.BUILD-20180902.070551-14.jar:1.5.16.BUILD-SNAPSHOT]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:303) ~[spring-boot-1.5.16.BUILD-20180902.070551-14.jar:1.5.16.BUILD-SNAPSHOT]
at org.springframework.boot.test.context.SpringBootContextLoader.loadContext(SpringBootContextLoader.java:121) ~[spring-boot-test-1.5.16.BUILD-20180902.070551-14.jar:1.5.16.BUILD-SNAPSHOT]
at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContextInternal(DefaultCacheAwareContextLoaderDelegate.java:98) ~[spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:116) ~[spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE]
... 24 common frames omitted
思考
关于最后一种方式判断sql类型达到数据源的动态地选择,此时mysql为一主一从的结构,若为一主多从的结构时如何实现动态选择?以下是大体思路:
本项目实现了一主一从动态数据源的选择,使用切面判断sql方法的方式。
这里的多数据的动态选择,主要是一主一从的形式,若有一主多从的方式,怎么实现数据源的动态选择?
增删改可以使用一个主数据库,查则可以有多个从数据库可以选择,对于多个从数据库,如何选择?
当有多个从数据库时,初步设想使用负载均衡,使用轮询的方式将查询分布到多个从数据库中,那如何实现?
动态数据源的实现是继承了抽象类AbstractRoutingDataSource,实现了其中的抽象方法 determineCurrentLookupKey()
通过方法名可知,该方法返回一个key,通过阅读AbstractRoutingDataSource的源码,多个数据源的存储方式
是以key-value的形式存储在map中,当动态选择时,使用determineTargetDataSourc()根据key找到对应的DataSource。
现在要实现多个数据库,设想继承AbstractRoutingDataSource类,重写determineTargetDataSource()方法,将原来Map<Object, Object> targetDataSources的存储方式改为Map<Object,List<Object>> slaveDataSources 用来存储从库数据源,当动态选择时,在determineTargetDataSource()方法中 判断 determineCurrentLookupKey()方法的key,若为master,则直接返回resolvedDefaultDataSource,(前提是在配置数据源时,必须将主数据设置为defaultDataSource).若为slave,则得到slave的List<DataSource>,使用轮询的方式得到其中一个DataSource返回,实现多从的动态选择。
项目源码:
第一种:https://github.com/Edenwds/springboot_study/tree/master/dynamic_datasource_anno
第二种:https://github.com/Edenwds/springboot_study/tree/master/dynamic_datasource_method
参考资料:
https://blog.csdn.net/neosmith/article/details/61202084
https://www.codeblogbt.com/archives/153425
http://fedulov.website/2015/10/14/dynamic-datasource-routing-with-spring/
https://stackoverflow.com/questions/47970429/query-is-always-execute-before-than-aop-in-springboot-and-mybatis-application-fo