目录
实现目标
- 队列缓冲业务数据
- 线程空闲忙碌识别
- 任务处理的进度控制
目标分析
进度控制：需要将队列里面总任务和线程消费的任务进行记录，实现一个completed/total 这样的控制。
线程池控制：线程池使用计数器，在完成任务和新增任务进行原子操作维护计数器数量。
线程池设计
线程池实际上就是一个线程的池化处理，一般会初始化几个线程，需要使用线程时从池子里面拿，池子里面的总容量占用多少可以用来标记线程繁忙和空闲。线程池的设计跟数据库JDBC的设计非常相似。比如获取连接多久超时等等，在线程池里面也是可以这样来实现，线程的示例是用来执行任务的，线程池大小的个数不宜太大，一般core*2或者core*2+1 ，再或者凑个十进制十位整数。
队列Queue：队列可以是一个链表，也可以是一个简单的集合，需要设计相应的队列排队策略（入队、出队、优先级），还需要设计相应的方法来便于外部操作，Java队列主要有以下操作方法：
线程池：线程池的实现也有很多，比如最常用的Excutors的多种类型线程池，比如：
- 自动伸缩类型线程池
- 固定大小类型线程池
- 单个线程池
- 调度式线程池
- 不可配置线程池...
package java.util.concurrent;
import java.util.*;
import java.util.concurrent.atomic.AtomicInteger;
import java.security.AccessControlContext;
import java.security.AccessController;
import java.security.PrivilegedAction;
import java.security.PrivilegedExceptionAction;
import java.security.PrivilegedActionException;
import java.security.AccessControlException;
import sun.security.util.SecurityConstants;
/**
* Factory and utility methods for {@link Executor}, {@link
* ExecutorService}, {@link ScheduledExecutorService}, {@link
* ThreadFactory}, and {@link Callable} classes defined in this
* package. This class supports the following kinds of methods:
*
* <ul>
* <li> Methods that create and return an {@link ExecutorService}
* set up with commonly useful configuration settings.
* <li> Methods that create and return a {@link ScheduledExecutorService}
* set up with commonly useful configuration settings.
* <li> Methods that create and return a "wrapped" ExecutorService, that
* disables reconfiguration by making implementation-specific methods
* inaccessible.
* <li> Methods that create and return a {@link ThreadFactory}
* that sets newly created threads to a known state.
* <li> Methods that create and return a {@link Callable}
* out of other closure-like forms, so they can be used
* in execution methods requiring {@code Callable}.
* </ul>
*
* @since 1.5
* @author Doug Lea
*/
public class Executors {
/**
* Creates a thread pool that reuses a fixed number of threads
* operating off a shared unbounded queue. At any point, at most
* {@code nThreads} threads will be active processing tasks.
* If additional tasks are submitted when all threads are active,
* they will wait in the queue until a thread is available.
* If any thread terminates due to a failure during execution
* prior to shutdown, a new one will take its place if needed to
* execute subsequent tasks. The threads in the pool will exist
* until it is explicitly {@link ExecutorService#shutdown shutdown}.
*
* @param nThreads the number of threads in the pool
* @return the newly created thread pool
* @throws IllegalArgumentException if {@code nThreads <= 0}
*/
public static ExecutorService newFixedThreadPool(int nThreads) {
return new ThreadPoolExecutor(nThreads, nThreads,
0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<Runnable>());
}
/**
* Creates a thread pool that maintains enough threads to support
* the given parallelism level, and may use multiple queues to
* reduce contention. The parallelism level corresponds to the
* maximum number of threads actively engaged in, or available to
* engage in, task processing. The actual number of threads may
* grow and shrink dynamically. A work-stealing pool makes no
* guarantees about the order in which submitted tasks are
* executed.
*
* @param parallelism the targeted parallelism level
* @return the newly created thread pool
* @throws IllegalArgumentException if {@code parallelism <= 0}
* @since 1.8
*/
public static ExecutorService newWorkStealingPool(int parallelism) {
return new ForkJoinPool
(parallelism,
ForkJoinPool.defaultForkJoinWorkerThreadFactory,
null, true);
}
/**
* Creates a work-stealing thread pool using all
* {@link Runtime#availableProcessors available processors}
* as its target parallelism level.
* @return the newly created thread pool
* @see #newWorkStealingPool(int)
* @since 1.8
*/
public static ExecutorService newWorkStealingPool() {
return new ForkJoinPool
(Runtime.getRuntime().availableProcessors(),
ForkJoinPool.defaultForkJoinWorkerThreadFactory,
null, true);
}
/**
* Creates a thread pool that reuses a fixed number of threads
* operating off a shared unbounded queue, using the provided
* ThreadFactory to create new threads when needed. At any point,
* at most {@code nThreads} threads will be active processing
* tasks. If additional tasks are submitted when all threads are
* active, they will wait in the queue until a thread is
* available. If any thread terminates due to a failure during
* execution prior to shutdown, a new one will take its place if
* needed to execute subsequent tasks. The threads in the pool will
* exist until it is explicitly {@link ExecutorService#shutdown
* shutdown}.
*
* @param nThreads the number of threads in the pool
* @param threadFactory the factory to use when creating new threads
* @return the newly created thread pool
* @throws NullPointerException if threadFactory is null
* @throws IllegalArgumentException if {@code nThreads <= 0}
*/
public static ExecutorService newFixedThreadPool(int nThreads, ThreadFactory threadFactory) {
return new ThreadPoolExecutor(nThreads, nThreads,
0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<Runnable>(),
threadFactory);
}
/**
* Creates an Executor that uses a single worker thread operating
* off an unbounded queue. (Note however that if this single
* thread terminates due to a failure during execution prior to
* shutdown, a new one will take its place if needed to execute
* subsequent tasks.) Tasks are guaranteed to execute
* sequentially, and no more than one task will be active at any
* given time. Unlike the otherwise equivalent
* {@code newFixedThreadPool(1)} the returned executor is
* guaranteed not to be reconfigurable to use additional threads.
*
* @return the newly created single-threaded Executor
*/
public static ExecutorService newSingleThreadExecutor() {
return new FinalizableDelegatedExecutorService
(new ThreadPoolExecutor(1, 1,
0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<Runnable>()));
}
/**
* Creates an Executor that uses a single worker thread operating
* off an unbounded queue, and uses the provided ThreadFactory to
* create a new thread when needed. Unlike the otherwise
* equivalent {@code newFixedThreadPool(1, threadFactory)} the
* returned executor is guaranteed not to be reconfigurable to use
* additional threads.
*
* @param threadFactory the factory to use when creating new
* threads
*
* @return the newly created single-threaded Executor
* @throws NullPointerException if threadFactory is null
*/
public static ExecutorService newSingleThreadExecutor(ThreadFactory threadFactory) {
return new FinalizableDelegatedExecutorService
(new ThreadPoolExecutor(1, 1,
0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<Runnable>(),
threadFactory));
}
/**
* Creates a thread pool that creates new threads as needed, but
* will reuse previously constructed threads when they are
* available. These pools will typically improve the performance
* of programs that execute many short-lived asynchronous tasks.
* Calls to {@code execute} will reuse previously constructed
* threads if available. If no existing thread is available, a new
* thread will be created and added to the pool. Threads that have
* not been used for sixty seconds are terminated and removed from
* the cache. Thus, a pool that remains idle for long enough will
* not consume any resources. Note that pools with similar
* properties but different details (for example, timeout parameters)
* may be created using {@link ThreadPoolExecutor} constructors.
*
* @return the newly created thread pool
*/
public static ExecutorService newCachedThreadPool() {
return new ThreadPoolExecutor(0, Integer.MAX_VALUE,
60L, TimeUnit.SECONDS,
new SynchronousQueue<Runnable>());
}
/**
* Creates a thread pool that creates new threads as needed, but
* will reuse previously constructed threads when they are
* available, and uses the provided
* ThreadFactory to create new threads when needed.
* @param threadFactory the factory to use when creating new threads
* @return the newly created thread pool
* @throws NullPointerException if threadFactory is null
*/
public static ExecutorService newCachedThreadPool(ThreadFactory threadFactory) {
return new ThreadPoolExecutor(0, Integer.MAX_VALUE,
60L, TimeUnit.SECONDS,
new SynchronousQueue<Runnable>(),
threadFactory);
}
/**
* Creates a single-threaded executor that can schedule commands
* to run after a given delay, or to execute periodically.
* (Note however that if this single
* thread terminates due to a failure during execution prior to
* shutdown, a new one will take its place if needed to execute
* subsequent tasks.) Tasks are guaranteed to execute
* sequentially, and no more than one task will be active at any
* given time. Unlike the otherwise equivalent
* {@code newScheduledThreadPool(1)} the returned executor is
* guaranteed not to be reconfigurable to use additional threads.
* @return the newly created scheduled executor
*/
public static ScheduledExecutorService newSingleThreadScheduledExecutor() {
return new DelegatedScheduledExecutorService
(new ScheduledThreadPoolExecutor(1));
}
/**
* Creates a single-threaded executor that can schedule commands
* to run after a given delay, or to execute periodically. (Note
* however that if this single thread terminates due to a failure
* during execution prior to shutdown, a new one will take its
* place if needed to execute subsequent tasks.) Tasks are
* guaranteed to execute sequentially, and no more than one task
* will be active at any given time. Unlike the otherwise
* equivalent {@code newScheduledThreadPool(1, threadFactory)}
* the returned executor is guaranteed not to be reconfigurable to
* use additional threads.
* @param threadFactory the factory to use when creating new
* threads
* @return a newly created scheduled executor
* @throws NullPointerException if threadFactory is null
*/
public static ScheduledExecutorService newSingleThreadScheduledExecutor(ThreadFactory threadFactory) {
return new DelegatedScheduledExecutorService
(new ScheduledThreadPoolExecutor(1, threadFactory));
}
/**
* Creates a thread pool that can schedule commands to run after a
* given delay, or to execute periodically.
* @param corePoolSize the number of threads to keep in the pool,
* even if they are idle
* @return a newly created scheduled thread pool
* @throws IllegalArgumentException if {@code corePoolSize < 0}
*/
public static ScheduledExecutorService newScheduledThreadPool(int corePoolSize) {
return new ScheduledThreadPoolExecutor(corePoolSize);
}
/**
* Creates a thread pool that can schedule commands to run after a
* given delay, or to execute periodically.
* @param corePoolSize the number of threads to keep in the pool,
* even if they are idle
* @param threadFactory the factory to use when the executor
* creates a new thread
* @return a newly created scheduled thread pool
* @throws IllegalArgumentException if {@code corePoolSize < 0}
* @throws NullPointerException if threadFactory is null
*/
public static ScheduledExecutorService newScheduledThreadPool(
int corePoolSize, ThreadFactory threadFactory) {
return new ScheduledThreadPoolExecutor(corePoolSize, threadFactory);
}
/**
* Returns an object that delegates all defined {@link
* ExecutorService} methods to the given executor, but not any
* other methods that might otherwise be accessible using
* casts. This provides a way to safely "freeze" configuration and
* disallow tuning of a given concrete implementation.
* @param executor the underlying implementation
* @return an {@code ExecutorService} instance
* @throws NullPointerException if executor null
*/
public static ExecutorService unconfigurableExecutorService(ExecutorService executor) {
if (executor == null)
throw new NullPointerException();
return new DelegatedExecutorService(executor);
}
/**
* Returns an object that delegates all defined {@link
* ScheduledExecutorService} methods to the given executor, but
* not any other methods that might otherwise be accessible using
* casts. This provides a way to safely "freeze" configuration and
* disallow tuning of a given concrete implementation.
* @param executor the underlying implementation
* @return a {@code ScheduledExecutorService} instance
* @throws NullPointerException if executor null
*/
public static ScheduledExecutorService unconfigurableScheduledExecutorService(ScheduledExecutorService executor) {
if (executor == null)
throw new NullPointerException();
return new DelegatedScheduledExecutorService(executor);
}
/**
* Returns a default thread factory used to create new threads.
* This factory creates all new threads used by an Executor in the
* same {@link ThreadGroup}. If there is a {@link
* java.lang.SecurityManager}, it uses the group of {@link
* System#getSecurityManager}, else the group of the thread
* invoking this {@code defaultThreadFactory} method. Each new
* thread is created as a non-daemon thread with priority set to
* the smaller of {@code Thread.NORM_PRIORITY} and the maximum
* priority permitted in the thread group. New threads have names
* accessible via {@link Thread#getName} of
* <em>pool-N-thread-M</em>, where <em>N</em> is the sequence
* number of this factory, and <em>M</em> is the sequence number
* of the thread created by this factory.
* @return a thread factory
*/
public static ThreadFactory defaultThreadFactory() {
return new DefaultThreadFactory();
}
/**
* Returns a thread factory used to create new threads that
* have the same permissions as the current thread.
* This factory creates threads with the same settings as {@link
* Executors#defaultThreadFactory}, additionally setting the
* AccessControlContext and contextClassLoader of new threads to
* be the same as the thread invoking this
* {@code privilegedThreadFactory} method. A new
* {@code privilegedThreadFactory} can be created within an
* {@link AccessController#doPrivileged AccessController.doPrivileged}
* action setting the current thread's access control context to
* create threads with the selected permission settings holding
* within that action.
*
* <p>Note that while tasks running within such threads will have
* the same access control and class loader settings as the
* current thread, they need not have the same {@link
* java.lang.ThreadLocal} or {@link
* java.lang.InheritableThreadLocal} values. If necessary,
* particular values of thread locals can be set or reset before
* any task runs in {@link ThreadPoolExecutor} subclasses using
* {@link ThreadPoolExecutor#beforeExecute(Thread, Runnable)}.
* Also, if it is necessary to initialize worker threads to have
* the same InheritableThreadLocal settings as some other
* designated thread, you can create a custom ThreadFactory in
* which that thread waits for and services requests to create
* others that will inherit its values.
*
* @return a thread factory
* @throws AccessControlException if the current access control
* context does not have permission to both get and set context
* class loader
*/
public static ThreadFactory privilegedThreadFactory() {
return new PrivilegedThreadFactory();
}
/**
* Returns a {@link Callable} object that, when
* called, runs the given task and returns the given result. This
* can be useful when applying methods requiring a
* {@code Callable} to an otherwise resultless action.
* @param task the task to run
* @param result the result to return
* @param <T> the type of the result
* @return a callable object
* @throws NullPointerException if task null
*/
public static <T> Callable<T> callable(Runnable task, T result) {
if (task == null)
throw new NullPointerException();
return new RunnableAdapter<T>(task, result);
}
/**
* Returns a {@link Callable} object that, when
* called, runs the given task and returns {@code null}.
* @param task the task to run
* @return a callable object
* @throws NullPointerException if task null
*/
public static Callable<Object> callable(Runnable task) {
if (task == null)
throw new NullPointerException();
return new RunnableAdapter<Object>(task, null);
}
/**
* Returns a {@link Callable} object that, when
* called, runs the given privileged action and returns its result.
* @param action the privileged action to run
* @return a callable object
* @throws NullPointerException if action null
*/
public static Callable<Object> callable(final PrivilegedAction<?> action) {
if (action == null)
throw new NullPointerException();
return new Callable<Object>() {
public Object call() { return action.run(); }};
}
/**
* Returns a {@link Callable} object that, when
* called, runs the given privileged exception action and returns
* its result.
* @param action the privileged exception action to run
* @return a callable object
* @throws NullPointerException if action null
*/
public static Callable<Object> callable(final PrivilegedExceptionAction<?> action) {
if (action == null)
throw new NullPointerException();
return new Callable<Object>() {
public Object call() throws Exception { return action.run(); }};
}
/**
* Returns a {@link Callable} object that will, when called,
* execute the given {@code callable} under the current access
* control context. This method should normally be invoked within
* an {@link AccessController#doPrivileged AccessController.doPrivileged}
* action to create callables that will, if possible, execute
* under the selected permission settings holding within that
* action; or if not possible, throw an associated {@link
* AccessControlException}.
* @param callable the underlying task
* @param <T> the type of the callable's result
* @return a callable object
* @throws NullPointerException if callable null
*/
public static <T> Callable<T> privilegedCallable(Callable<T> callable) {
if (callable == null)
throw new NullPointerException();
return new PrivilegedCallable<T>(callable);
}
/**
* Returns a {@link Callable} object that will, when called,
* execute the given {@code callable} under the current access
* control context, with the current context class loader as the
* context class loader. This method should normally be invoked
* within an
* {@link AccessController#doPrivileged AccessController.doPrivileged}
* action to create callables that will, if possible, execute
* under the selected permission settings holding within that
* action; or if not possible, throw an associated {@link
* AccessControlException}.
*
* @param callable the underlying task
* @param <T> the type of the callable's result
* @return a callable object
* @throws NullPointerException if callable null
* @throws AccessControlException if the current access control
* context does not have permission to both set and get context
* class loader
*/
public static <T> Callable<T> privilegedCallableUsingCurrentClassLoader(Callable<T> callable) {
if (callable == null)
throw new NullPointerException();
return new PrivilegedCallableUsingCurrentClassLoader<T>(callable);
}
// Non-public classes supporting the public methods
/**
* A callable that runs given task and returns given result
*/
static final class RunnableAdapter<T> implements Callable<T> {
final Runnable task;
final T result;
RunnableAdapter(Runnable task, T result) {
this.task = task;
this.result = result;
}
public T call() {
task.run();
return result;
}
}
/**
* A callable that runs under established access control settings
*/
static final class PrivilegedCallable<T> implements Callable<T> {
private final Callable<T> task;
private final AccessControlContext acc;
PrivilegedCallable(Callable<T> task) {
this.task = task;
this.acc = AccessController.getContext();
}
public T call() throws Exception {
try {
return AccessController.doPrivileged(
new PrivilegedExceptionAction<T>() {
public T run() throws Exception {
return task.call();
}
}, acc);
} catch (PrivilegedActionException e) {
throw e.getException();
}
}
}
/**
* A callable that runs under established access control settings and
* current ClassLoader
*/
static final class PrivilegedCallableUsingCurrentClassLoader<T> implements Callable<T> {
private final Callable<T> task;
private final AccessControlContext acc;
private final ClassLoader ccl;
PrivilegedCallableUsingCurrentClassLoader(Callable<T> task) {
SecurityManager sm = System.getSecurityManager();
if (sm != null) {
// Calls to getContextClassLoader from this class
// never trigger a security check, but we check
// whether our callers have this permission anyways.
sm.checkPermission(SecurityConstants.GET_CLASSLOADER_PERMISSION);
// Whether setContextClassLoader turns out to be necessary
// or not, we fail fast if permission is not available.
sm.checkPermission(new RuntimePermission("setContextClassLoader"));
}
this.task = task;
this.acc = AccessController.getContext();
this.ccl = Thread.currentThread().getContextClassLoader();
}
public T call() throws Exception {
try {
return AccessController.doPrivileged(
new PrivilegedExceptionAction<T>() {
public T run() throws Exception {
Thread t = Thread.currentThread();
ClassLoader cl = t.getContextClassLoader();
if (ccl == cl) {
return task.call();
} else {
t.setContextClassLoader(ccl);
try {
return task.call();
} finally {
t.setContextClassLoader(cl);
}
}
}
}, acc);
} catch (PrivilegedActionException e) {
throw e.getException();
}
}
}
/**
* The default thread factory
*/
static class DefaultThreadFactory implements ThreadFactory {
private static final AtomicInteger poolNumber = new AtomicInteger(1);
private final ThreadGroup group;
private final AtomicInteger threadNumber = new AtomicInteger(1);
private final String namePrefix;
DefaultThreadFactory() {
SecurityManager s = System.getSecurityManager();
group = (s != null) ? s.getThreadGroup() :
Thread.currentThread().getThreadGroup();
namePrefix = "pool-" +
poolNumber.getAndIncrement() +
"-thread-";
}
public Thread newThread(Runnable r) {
Thread t = new Thread(group, r,
namePrefix + threadNumber.getAndIncrement(),
0);
if (t.isDaemon())
t.setDaemon(false);
if (t.getPriority() != Thread.NORM_PRIORITY)
t.setPriority(Thread.NORM_PRIORITY);
return t;
}
}
/**
* Thread factory capturing access control context and class loader
*/
static class PrivilegedThreadFactory extends DefaultThreadFactory {
private final AccessControlContext acc;
private final ClassLoader ccl;
PrivilegedThreadFactory() {
super();
SecurityManager sm = System.getSecurityManager();
if (sm != null) {
// Calls to getContextClassLoader from this class
// never trigger a security check, but we check
// whether our callers have this permission anyways.
sm.checkPermission(SecurityConstants.GET_CLASSLOADER_PERMISSION);
// Fail fast
sm.checkPermission(new RuntimePermission("setContextClassLoader"));
}
this.acc = AccessController.getContext();
this.ccl = Thread.currentThread().getContextClassLoader();
}
public Thread newThread(final Runnable r) {
return super.newThread(new Runnable() {
public void run() {
AccessController.doPrivileged(new PrivilegedAction<Void>() {
public Void run() {
Thread.currentThread().setContextClassLoader(ccl);
r.run();
return null;
}
}, acc);
}
});
}
}
/**
* A wrapper class that exposes only the ExecutorService methods
* of an ExecutorService implementation.
*/
static class DelegatedExecutorService extends AbstractExecutorService {
private final ExecutorService e;
DelegatedExecutorService(ExecutorService executor) { e = executor; }
public void execute(Runnable command) { e.execute(command); }
public void shutdown() { e.shutdown(); }
public List<Runnable> shutdownNow() { return e.shutdownNow(); }
public boolean isShutdown() { return e.isShutdown(); }
public boolean isTerminated() { return e.isTerminated(); }
public boolean awaitTermination(long timeout, TimeUnit unit)
throws InterruptedException {
return e.awaitTermination(timeout, unit);
}
public Future<?> submit(Runnable task) {
return e.submit(task);
}
public <T> Future<T> submit(Callable<T> task) {
return e.submit(task);
}
public <T> Future<T> submit(Runnable task, T result) {
return e.submit(task, result);
}
public <T> List<Future<T>> invokeAll(Collection<? extends Callable<T>> tasks)
throws InterruptedException {
return e.invokeAll(tasks);
}
public <T> List<Future<T>> invokeAll(Collection<? extends Callable<T>> tasks,
long timeout, TimeUnit unit)
throws InterruptedException {
return e.invokeAll(tasks, timeout, unit);
}
public <T> T invokeAny(Collection<? extends Callable<T>> tasks)
throws InterruptedException, ExecutionException {
return e.invokeAny(tasks);
}
public <T> T invokeAny(Collection<? extends Callable<T>> tasks,
long timeout, TimeUnit unit)
throws InterruptedException, ExecutionException, TimeoutException {
return e.invokeAny(tasks, timeout, unit);
}
}
static class FinalizableDelegatedExecutorService
extends DelegatedExecutorService {
FinalizableDelegatedExecutorService(ExecutorService executor) {
super(executor);
}
protected void finalize() {
super.shutdown();
}
}
/**
* A wrapper class that exposes only the ScheduledExecutorService
* methods of a ScheduledExecutorService implementation.
*/
static class DelegatedScheduledExecutorService
extends DelegatedExecutorService
implements ScheduledExecutorService {
private final ScheduledExecutorService e;
DelegatedScheduledExecutorService(ScheduledExecutorService executor) {
super(executor);
e = executor;
}
public ScheduledFuture<?> schedule(Runnable command, long delay, TimeUnit unit) {
return e.schedule(command, delay, unit);
}
public <V> ScheduledFuture<V> schedule(Callable<V> callable, long delay, TimeUnit unit) {
return e.schedule(callable, delay, unit);
}
public ScheduledFuture<?> scheduleAtFixedRate(Runnable command, long initialDelay, long period, TimeUnit unit) {
return e.scheduleAtFixedRate(command, initialDelay, period, unit);
}
public ScheduledFuture<?> scheduleWithFixedDelay(Runnable command, long initialDelay, long delay, TimeUnit unit) {
return e.scheduleWithFixedDelay(command, initialDelay, delay, unit);
}
}
/** Cannot instantiate. */
private Executors() {}
}
队列与线程池结合
以阿里云OSS文件存储为例：
package com.forestar.aliyun.service.oss.queue;
import com.forestar.aliyun.service.oss.bean.FileInfo;
import com.forestar.aliyun.service.oss.tds.OssHttpService;
import lombok.extern.slf4j.Slf4j;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicLong;
/**
* @Copyright: 2019-2021
* @FileName: FileUploadQueue.java
* @Author: PJL
* @Date: 2020/12/3 10:35
* @Description: 文件上传队列
*/
@Slf4j
public class FileUploadQueue {
private ConcurrentLinkedQueue<FileInfo> queue = new ConcurrentLinkedQueue<>();
private AtomicLong total = new AtomicLong(0);
private AtomicLong completed = new AtomicLong(0);
private AtomicInteger exeCount = new AtomicInteger(0);
private ExecutorService executorService;
private OssHttpService ossHttpService;
private Boolean started = false;
private int poolSize;
/**
* 文件上传队列初始化
*
* @param ossHttpService
* @param executorPoolSize
* @param startConsumer
*/
public FileUploadQueue(OssHttpService ossHttpService, int executorPoolSize, boolean startConsumer) {
this.poolSize = executorPoolSize;
this.ossHttpService = ossHttpService;
this.executorService = Executors.newWorkStealingPool(this.poolSize);
if (startConsumer) {
this.start();
}
}
/**
* 文件入队列
*
* @param fileInfo
*/
public Long enqueue(FileInfo fileInfo) {
queue.add(fileInfo);
return total.incrementAndGet();
}
/**
* 空闲判断
*
* @return
*/
public Boolean isUnFull() {
return exeCount.get() < poolSize;
}
/**
* 空闲判断
*
* @return
*/
public Boolean isBusy() {
return exeCount.get() > 5 && isUnFull();
}
/**
* 空闲判断
*
* @return
*/
public Boolean isIdle() {
return exeCount.get() <= 5;
}
/**
* 开启消费线程
*/
public void start() {
if (!started) {
new Thread(() -> {
while (true) {
try {
int count = queue.size();
// 线程池消费
if (count > 0 && isUnFull()) {
consumer();
}
// 忙碌延长休眠
if (count > 0 && isBusy()) {
Thread.sleep(100);
}
// 空闲缩短休眠
else if (count > 0 && isIdle()) {
Thread.sleep(50);
}else{
Thread.sleep(1000);
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}).start();
started = true;
}
}
/**
* 执行业务处理
*/
private void consumer() {
FileInfo fileInfo = queue.poll();
if (null != fileInfo) {
// 增加线程占用数量
exeCount.incrementAndGet();
// 提交执行任务
executorService.submit(() -> {
// 处理业务数据
ossHttpService.syncToAliyunOss(fileInfo);
// 执行个数增加
completed.incrementAndGet();
// 线程池占用减少
if (exeCount.get() > 0) {
exeCount.decrementAndGet();
}
// 打印处理进度
log.info("===队列消费进度==={}/{}", completed.get(), total.get());
// 处理完成通知
if (completed.get() == total.get()) {
log.info("=====================所有文件上传完成!=======================");
}
});
}
}
}
最终我们得到类似下面的效果输出：
2020-12-03 12:38:08.802 INFO 9792 --- [pool-3-thread-1] c.f.a.service.oss.tds.OssHttpService : >>>开始解析....D:/TDSpath/list.txt
2020-12-03 12:38:08.804 INFO 9792 --- [pool-3-thread-1] c.f.a.service.oss.tds.OssHttpService : httpUrl = http://remote_host/upload/eventAttach/eventpic/original/202007/20200711/05290729-bf44-41c6-8da7-44251e131d15.jpg , objectName = /upload/eventAttach/eventpic/original/202007/20200711/05290729-bf44-41c6-8da7-44251e131d15.jpg
2020-12-03 12:38:08.804 INFO 9792 --- [pool-3-thread-1] c.f.a.service.oss.tds.OssHttpService : ==文件队列长度== size =1
2020-12-03 12:38:08.805 INFO 9792 --- [pool-3-thread-1] c.f.a.service.oss.tds.OssHttpService : httpUrl = http://remote_host/upload/eventAttach/eventpic/thumb/202007/20200711/05290729-bf44-41c6-8da7-44251e131d15.jpg , objectName = /upload/eventAttach/eventpic/thumb/202007/20200711/05290729-bf44-41c6-8da7-44251e131d15.jpg
2020-12-03 12:38:08.805 INFO 9792 --- [pool-3-thread-1] c.f.a.service.oss.tds.OssHttpService : ==文件队列长度== size =2
2020-12-03 12:38:08.805 INFO 9792 --- [pool-3-thread-1] c.f.a.service.oss.tds.OssHttpService : httpUrl = http://remote_host/upload/eventAttach/eventpic/original/202007/20200711/15d1724a-a969-41f7-abe8-171904a746da.jpg , objectName = /upload/eventAttach/eventpic/original/202007/20200711/15d1724a-a969-41f7-abe8-171904a746da.jpg
2020-12-03 12:38:08.805 INFO 9792 --- [pool-3-thread-1] c.f.a.service.oss.tds.OssHttpService : ==文件队列长度== size =3
2020-12-03 12:38:08.806 INFO 9792 --- [pool-3-thread-1] c.f.a.service.oss.tds.OssHttpService : httpUrl = http://remote_host/upload/eventAttach/eventpic/thumb/202007/20200711/15d1724a-a969-41f7-abe8-171904a746da.jpg , objectName = /upload/eventAttach/eventpic/thumb/202007/20200711/15d1724a-a969-41f7-abe8-171904a746da.jpg
2020-12-03 12:38:08.807 INFO 9792 --- [pool-3-thread-1] c.f.a.service.oss.tds.OssHttpService : ==文件队列长度== size =4
2020-12-03 12:38:09.805 INFO 9792 --- [Pool-3-worker-9] c.f.a.service.oss.queue.FileUploadQueue : ===队列消费进度===1/4
2020-12-03 12:38:09.856 INFO 9792 --- [Pool-3-worker-9] c.f.a.service.oss.queue.FileUploadQueue : ===队列消费进度===2/4
2020-12-03 12:38:09.907 INFO 9792 --- [Pool-3-worker-9] c.f.a.service.oss.queue.FileUploadQueue : ===队列消费进度===3/4
2020-12-03 12:38:09.957 INFO 9792 --- [Pool-3-worker-9] c.f.a.service.oss.queue.FileUploadQueue : ===队列消费进度===4/4
2020-12-03 12:38:09.957 INFO 9792 --- [Pool-3-worker-9] c.f.a.service.oss.queue.FileUploadQueue : =====================所有文件上传完成!=======================