batch method
IService's updateBatchById method
default batchSize = 1000
com.baomidou.mybatisplus.extension.service.impl.ServiceImpl#updateBatchById
@Transactional(rollbackFor = Exception.class)
@Override
public boolean updateBatchById(Collection<T> entityList, int batchSize) {
String sqlStatement = getSqlStatement(SqlMethod.UPDATE_BY_ID);
return executeBatch(entityList, batchSize, (sqlSession, entity) -> {
MapperMethod.ParamMap<T> param = new MapperMethod.ParamMap<>();
param.put(Constants.ENTITY, entity);
sqlSession.update(sqlStatement, param);
});
}
Construct a callback and enter the executeBatch method
/**
* 执行批量操作
*
* @param entityClass 实体类
* @param log 日志对象
* @param list 数据集合
* @param batchSize 批次大小
* @param consumer consumer
* @param <E> T
* @return 操作结果
* @since 3.4.0
*/
public static <E> boolean executeBatch(Class<?> entityClass, Log log, Collection<E> list, int batchSize, BiConsumer<SqlSession, E> consumer) {
Assert.isFalse(batchSize < 1, "batchSize must not be less than one");
return !CollectionUtils.isEmpty(list) && executeBatch(entityClass, log, sqlSession -> {
int size = list.size();
int i = 1;
for (E element : list) {
consumer.accept(sqlSession, element);
if ((i % batchSize == 0) || i == size) {
sqlSession.flushStatements();
}
i++;
}
});
}
You can basically see this in this method. After executing the method 1000 times, flushStatements are executed once. In other words, in theory, 1000 update sql are accumulated before a database update is performed. BatchExcutor is used for batch execution
.
BatchExcutor
Before analyzing the BatchExecutor class, first understand the batch processing related knowledge of JDBC.
JDBC batch processing
Batch processing allows related SQL statements to be grouped into batches and submitted with a single call to the database, completing the interaction with the database in one execution. It should be noted that batch processing in JDBC only supports insert, update, delete and other types of SQL statements, and does not support select type SQL statements.
** When sending multiple SQL statements to the database at one time, communication overhead can be reduced, thereby improving performance. **
No JDBC driver is required to support this feature. The DatabaseMetaData.supportsBatchUpdates() method should be used to determine whether the target database supports batch update processing. This method will return true if the JDBC driver supports this feature.
The addBatch() method of Statement, PreparedStatement and CallableStatement is used to add a single statement to a batch. executeBatch() is used to execute all statements that make up the batch.
executeBatch() returns an integer array, each element of the array represents the update count of the corresponding update statement.
Just like batch statements are added to the process, they can be removed using the clearBatch() method. This method will remove all statements added using the addBatch() method. However, you cannot specify a statement to be deleted.
The process of batch processing using Statement objects
A Statement can execute multiple sql (provided that the sql is the same and the placeholder is used).
The following is a typical sequence of steps for batch processing using the Statement object.
Use the createStatement() method to create a Statement object.
Use setAutoCommit() to set autocommit to false.
Use the addBatch() method to add SQL statements to the batch on the created Statement object.
Use the executeBatch() method on the created Statement object to execute all SQL statements.
Finally, use the commit() method to commit all changes.
Using PrepareStatement objects for batch processing
A Statement can execute multiple sql (provided the sql is the same and the placeholder is used).
The following is the typical sequence of steps for batch processing using the PrepareStatement object -
using placeholders to create SQL statements.
Use the prepareStatement() method to create a PrepareStatement object.
Use setAutoCommit() to set autocommit to false.
Use the addBatch() method to add SQL statements to the batch on the created Statement object.
Use the executeBatch() method on the created Statement object to execute all SQL statements.
Finally, use the commit() method to commit all changes.
BatchExecutor class
BatchExecutor also inherits the BaseExecutor abstract class and implements the function of batch processing multiple SQL statements. Because JDBC does not support select type SQL statements and only supports insert, update, and delete type SQL statements, in the BatchExecutor class, batch processing mainly targets the update() method. The overall logic implemented by the BatchExecutor class: The doUpdate() method mainly adds the SQL statements that need to be batched to the batched Statement or PrepareStatement object through the statement.addBatch() method, and then executes the Statement through the doFlushStatements() method. The executeBatch() method executes batch processing. In the doQueryCursor() method and doQuery() method, the flushStatements() method will first be executed. The bottom layer of the flushStatements() method is actually the doFlushStatements() method, so it is equivalent to first adding to the Statement or The batch statement in the PrepareStatement object is executed, and then the query operation is executed.
doUpdate() method
This method mainly adds the SQL statements that require batch processing to the Statement or PrepareStatement object of the batch through the statement.addBatch() method, and waits for the execution of the batch. It is mainly based on judging whether the currently executed SQL mode is the same as the last executed SQL mode and the corresponding MappedStatement objects are the same to determine whether to use an existing Statement object or create a new Statement object to perform the addBatch() operation.
@Override
public int doUpdate(MappedStatement ms, Object parameterObject) throws SQLException {
final Configuration configuration = ms.getConfiguration();
final StatementHandler handler = configuration.newStatementHandler(this, ms, parameterObject, RowBounds.DEFAULT, null, null);
final BoundSql boundSql = handler.getBoundSql();
final String sql = boundSql.getSql();
final Statement stmt;
if (sql.equals(currentSql) && ms.equals(currentStatement)) {
int last = statementList.size() - 1;
stmt = statementList.get(last);
applyTransactionTimeout(stmt);
handler.parameterize(stmt);//fix Issues 322
BatchResult batchResult = batchResultList.get(last);
batchResult.addParameterObject(parameterObject);
} else {
Connection connection = getConnection(ms.getStatementLog());
stmt = handler.prepare(connection, transaction.getTimeout());
handler.parameterize(stmt); //fix Issues 322
currentSql = sql;
currentStatement = ms;
statementList.add(stmt);
batchResultList.add(new BatchResult(ms, sql, parameterObject));
}
// handler.parameterize(stmt);
handler.batch(stmt);
return BATCH_UPDATE_RETURN_VALUE;
}
Here I found a special situation
where batch updates are performed. Logically, the same statement will be reused, but because the parameters are empty
userService.updateBatchById(Arrays.asList(u, u1, u2, u3));
For example, the mapStatement defined in xml like this, the update method of MybatisPlus is to use if tag to judge empty.
<update id="updateByExampleSelective" parameterType="map" >
update user
<set >
<if test="record.age != null" >
age = #{record.age,jdbcType=int},
</if>
</set>
············
</update>
Then if the u1 parameter age is not empty and the u2 parameter age is empty, it will also lead to different SQL comparisons and will not be added to the same batch.
doFlushStatements() method
In the doFlushStatements() method, the underlying executeBatch() method of Statement is executed to submit the batch operation. The BatchResult object maintains the execution result of a Statement.executeBatch() method. Compared with JDBC batch processing, this is equivalent to encapsulating multiple executeBatch() methods.
@Override
public List<BatchResult> doFlushStatements(boolean isRollback) throws SQLException {
try {
List<BatchResult> results = new ArrayList<BatchResult>();
//如果明确指定了要回滚事务，则直接返回空集合，忽略 statementList集合中记录的 SQL语句
if (isRollback) {
return Collections.emptyList();
}
//遍历statementList集合
for (int i = 0, n = statementList.size(); i < n; i++) {
Statement stmt = statementList.get(i);
applyTransactionTimeout(stmt);
//获取对应BatchResult对象
BatchResult batchResult = batchResultList.get(i);
try {
//调用 Statement.executeBatch()方法批量执行其中记录的 SQL语句，并使用返回的int数组
//更新 BatchResult.updateCounts字段，其中每一个元素都表示一条 SQL语句影响的记录条数
batchResult.setUpdateCounts(stmt.executeBatch());
MappedStatement ms = batchResult.getMappedStatement();
List<Object> parameterObjects = batchResult.getParameterObjects();
//获取配置的KeyGenerator对象
KeyGenerator keyGenerator = ms.getKeyGenerator();
if (Jdbc3KeyGenerator.class.equals(keyGenerator.getClass())) {
//获取数据库生成的主键，并设置到parameterObjects中
Jdbc3KeyGenerator jdbc3KeyGenerator = (Jdbc3KeyGenerator) keyGenerator;
jdbc3KeyGenerator.processBatch(ms, stmt, parameterObjects);
} else if (!NoKeyGenerator.class.equals(keyGenerator.getClass())) {
//issue #141
for (Object parameter : parameterObjects) {
//对于其他类型的 keyGenerator，会调用其processAfter()方法
keyGenerator.processAfter(this, ms, stmt, parameter);
}
}
// Close statement to close cursor #1109
closeStatement(stmt);
} catch (BatchUpdateException e) {
//异常处理
StringBuilder message = new StringBuilder();
message.append(batchResult.getMappedStatement().getId())
.append(" (batch index #")
.append(i + 1)
.append(")")
.append(" failed.");
if (i > 0) {
message.append(" ")
.append(i)
.append(" prior sub executor(s) completed successfully, but will be rolled back.");
}
throw new BatchExecutorException(message.toString(), e, results, batchResult);
}
//添加batchResult到results集合中
results.add(batchResult);
}
return results;
} finally {
//关闭或清空对应对象
for (Statement stmt : statementList) {
closeStatement(stmt);
}
currentSql = null;
statementList.clear();
batchResultList.clear();
}
}
The key method is stmt.executeBatch(), which can execute all sql under the current statement in batches.
doQuery() method, doQueryCursor() method
In the doQuery() and doQueryCursor() methods, they are similar to those in the SimpleExecutro class. The only difference is that the flushStatements() method is executed first, and the bottom layer of the flushStatements() method is actually the doFlushStatements() method, so it is equivalent to first adding the Execute the batch statement in the Statement or PrepareStatement object, and then execute the query operation
@Override
public <E> List<E> doQuery(MappedStatement ms, Object parameterObject, RowBounds rowBounds, ResultHandler resultHandler, BoundSql boundSql)
throws SQLException {
Statement stmt = null;
try {
flushStatements();
Configuration configuration = ms.getConfiguration();
StatementHandler handler = configuration.newStatementHandler(wrapper, ms, parameterObject, rowBounds, resultHandler, boundSql);
Connection connection = getConnection(ms.getStatementLog());
stmt = handler.prepare(connection, transaction.getTimeout());
handler.parameterize(stmt);
return handler.<E>query(stmt, resultHandler);
} finally {
closeStatement(stmt);
}
}
@Override
protected <E> Cursor<E> doQueryCursor(MappedStatement ms, Object parameter, RowBounds rowBounds, BoundSql boundSql) throws SQLException {
flushStatements();
Configuration configuration = ms.getConfiguration();
StatementHandler handler = configuration.newStatementHandler(wrapper, ms, parameter, rowBounds, null, boundSql);
Connection connection = getConnection(ms.getStatementLog());
Statement stmt = handler.prepare(connection, transaction.getTimeout());
handler.parameterize(stmt);
return handler.<E>queryCursor(stmt);
}