学习目标:
熟悉 soul 的数据同步机制中的 http 同步
学习内容:
1.http数据同步原理
2.配置
3.源码分析
学习时间:
2020年1月25号
学习产出:
-
http数据同步原理
zookeeper、websocket 数据同步的机制比较简单,而 http 同步会相对复杂一些。Soul 借鉴了 Apollo、Nacos 的设计思想,取其精华,自己实现了 http 长轮询数据同步功能。注意,这里并非传统的 ajax 长轮询!
soul-web 网管请求admin的配置服务,读取超是时间为90s,意味着网关层请求配置服务最多会等待90s,这样便于admin配置服务的及时相应变更数据,从而实现准实时推送。
http请求到达soul-admin后,并非立马响应数据,而是利用Servlet3.0 的异步机制,异步响应数据。首先,将长轮询请求任务LongPollingClient 扔到 BlocingQueue中,并开启调度任务,60s后执行,这样做的目的是60s后将该长轮询请求移除队列,即便是这段时间内没有发生数据变更。因为即便是没有配置变更,也得让网关知道,总不能让其干等吧,而且网关请求配置服务时,也有 90s 的超时时间。 -
配置
admin:soul: sync: http: enabled: true
soul-bootstrap配置
soul: sync: http: url : http://localhost:9095
pom依赖:
<!--soul data sync start use http--> <dependency> <groupId>org.dromara</groupId> <artifactId>soul-spring-boot-starter-sync-data-http</artifactId> <version>${project.version}</version> </dependency>
-
源码解析
admin启动
DataSyncConfiguration初始化对象:HttpLongPollingDataChangedListener更新数据信息到本地缓存:
@Override public final void afterPropertiesSet() { updateAppAuthCache(); updatePluginCache(); updateRuleCache(); updateSelectorCache(); updateMetaDataCache(); afterInitialize(); } protected <T> void updateCache(final ConfigGroupEnum group, final List<T> data) { String json = GsonUtils.getInstance().toJson(data); ConfigDataCache newVal = new ConfigDataCache(group.name(), json, Md5Utils.md5(json), System.currentTimeMillis()); ConfigDataCache oldVal = CACHE.put(newVal.getGroup(), newVal); log.info("update config cache[{}], old: {}, updated: {}", group, oldVal, newVal); }
默认5分钟刷新一次本地缓存,可配置
@Override protected void afterInitialize() { long syncInterval = httpSyncProperties.getRefreshInterval().toMillis(); // Periodically check the data for changes and update the cache scheduler.scheduleWithFixedDelay(() -> { log.info("http sync strategy refresh config start."); try { this.refreshLocalCache(); log.info("http sync strategy refresh config success."); } catch (Exception e) { log.error("http sync strategy refresh config error!", e); } }, syncInterval, syncInterval, TimeUnit.MILLISECONDS); log.info("http sync strategy refresh interval: {}ms", syncInterval); }
DataChangedEventDispatcher 监听数据变更事件:
```
@Override
public void afterPropertiesSet() {
Collection<DataChangedListener> listenerBeans = applicationContext.getBeansOfType(DataChangedListener.class).values();
this.listeners = Collections.unmodifiableList(new ArrayList<>(listenerBeans));
}
```
soul-bootstrap 启动
HttpSyncDataConfiguration实例化HttpSyncDataService对象
public HttpSyncDataService(final HttpConfig httpConfig, final PluginDataSubscriber pluginDataSubscriber,
final List<MetaDataSubscriber> metaDataSubscribers, final List<AuthDataSubscriber> authDataSubscribers) {
this.factory = new DataRefreshFactory(pluginDataSubscriber, metaDataSubscribers, authDataSubscribers);
this.httpConfig = httpConfig;
this.serverList = Lists.newArrayList(Splitter.on(",").split(httpConfig.getUrl()));
this.httpClient = createRestTemplate();
this.start();
}
private void start() {
// It could be initialized multiple times, so you need to control that.
if (RUNNING.compareAndSet(false, true)) {
// fetch all group configs.
this.fetchGroupConfig(ConfigGroupEnum.values());
int threadSize = serverList.size();
this.executor = new ThreadPoolExecutor(threadSize, threadSize, 60L, TimeUnit.SECONDS,
new LinkedBlockingQueue<>(),
SoulThreadFactory.create("http-long-polling", true));
// start long polling, each server creates a thread to listen for changes.
this.serverList.forEach(server -> this.executor.execute(new HttpLongPollingTask(server)));
} else {
log.info("soul http long polling was started, executor=[{}]", executor);
}
}
启动http
修改rule数据
触发: DataChangeTask
```
@Override
protected void afterRuleChanged(final List<RuleData> changed, final DataEventTypeEnum eventType) {
scheduler.execute(new DataChangeTask(ConfigGroupEnum.RULE));
}
```
触发事件:
private void publishEvent(final RuleDO ruleDO, final List<RuleConditionDTO> ruleConditions) {
SelectorDO selectorDO = selectorMapper.selectById(ruleDO.getSelectorId());
PluginDO pluginDO = pluginMapper.selectById(selectorDO.getPluginId());
List<ConditionData> conditionDataList =
ruleConditions.stream().map(ConditionTransfer.INSTANCE::mapToRuleDTO).collect(Collectors.toList());
// publish change event.
eventPublisher.publishEvent(new DataChangedEvent(ConfigGroupEnum.RULE, DataEventTypeEnum.UPDATE,
Collections.singletonList(RuleDO.transFrom(ruleDO, pluginDO.getName(), conditionDataList))));
}
监听器:DataChangedEventDispatcher 监听到 RULE变化更新本地缓存
@Override
@SuppressWarnings("unchecked")
public void onApplicationEvent(final DataChangedEvent event) {
for (DataChangedListener listener : listeners) {
switch (event.getGroupKey()) {
case APP_AUTH:
listener.onAppAuthChanged((List<AppAuthData>) event.getSource(), event.getEventType());
break;
case PLUGIN:
listener.onPluginChanged((List<PluginData>) event.getSource(), event.getEventType());
break;
case RULE:
listener.onRuleChanged((List<RuleData>) event.getSource(), event.getEventType());
break;
case SELECTOR:
listener.onSelectorChanged((List<SelectorData>) event.getSource(), event.getEventType());
break;
case META_DATA:
listener.onMetaDataChanged((List<MetaData>) event.getSource(), event.getEventType());
break;
default:
throw new IllegalStateException("Unexpected value: " + event.getGroupKey());
}
}
}
更新本地缓存之后HttpLongPollingDataChangedListener 监听到rule变化
触发:
@Override
protected void afterRuleChanged(final List<RuleData> changed, final DataEventTypeEnum eventType) {
scheduler.execute(new DataChangeTask(ConfigGroupEnum.RULE));
}