Record the analysis process of a lossy RPC service online

1. Problem background

After an application started to provide JSF services, a large number of null pointer exceptions occurred in a short period of time.

After analyzing the log, it was found that the configuration data of Zangjing Pavilion that the service depends on was not loaded. That is the so-called lossy online or direct release . When the application starts, the service starts to provide external services before it is loaded, resulting in a failed call .

The key code is as follows

The initial loading of data is completed by implementing the CommandLineRunner interface.

@Component
public class LoadSystemArgsListener implements CommandLineRunner {

    @Resource
    private CacheLoader cjgConfigCacheLoader;

    @Override
    public void run(String... args) {
        // 加载藏经阁配置
        cjgConfigCacheLoader.refresh();

    }
}

The cjgConfigCacheLoader.refresh() method will load data into memory internally

/** 藏经阁配置数据 key:租户 value:配置数据 */
public static Map<String, CjgRuleConfig> cjgRuleConfigMap = new HashMap<>();

If the data has not been loaded at this time and cjgRuleConfigMap.get("301").getXX() is called, a null pointer exception will be reported.

To summarize the root cause: JSF Provider releases initial data loading earlier than service dependencies, resulting in failed calls.



2. Problem solving

Before solving this problem, we need to recall and become familiar with the startup process of Spring Boot and the release process of JSF services.

1) Spring Boot startup process (version 2.0.7.RELEASE)

run method, mainly focusing on refreshContext(context) refresh context

public ConfigurableApplicationContext run(String... args) {
    // 创建 StopWatch 实例：用于计算启动时间
    StopWatch stopWatch = new StopWatch();
    stopWatch.start();
    ConfigurableApplicationContext context = null;
    Collection<SpringBootExceptionReporter> exceptionReporters = new ArrayList<>();
    configureHeadlessProperty();

    // 获取SpringApplicationRunListeners：这些监听器会在启动过程的各个阶段发送对应的事件
    SpringApplicationRunListeners listeners = getRunListeners(args);
    listeners.starting();
    try {
        ApplicationArguments applicationArguments = new DefaultApplicationArguments(
                args);

        // 创建并配置Environment：包括准备好对应的`Environment`，以及将`application.properties`或`application.yml`中的配置项加载到`Environment`中
        ConfigurableEnvironment environment = prepareEnvironment(listeners,
                applicationArguments);
        configureIgnoreBeanInfo(environment);

        // 打印Banner：如果 spring.main.banner-mode 不为 off，则打印 banner
        Banner printedBanner = printBanner(environment);

        // 创建应用上下文：根据用户的配置和classpath下的配置，创建合适的`ApplicationContext`
        context = createApplicationContext();
        exceptionReporters = getSpringFactoriesInstances(
                SpringBootExceptionReporter.class,
                new Class[] { ConfigurableApplicationContext.class }, context);

        // 准备上下文：主要是将`Environment`、`ApplicationArguments`等关键属性设置到`ApplicationContext`中，以及加载`ApplicationListener`、`ApplicationRunner`、`CommandLineRunner`等。
        prepareContext(context, environment, listeners, applicationArguments,
                printedBanner);

        // 刷新上下文：这是Spring IoC容器启动的关键，包括Bean的创建、依赖注入、初始化，发布事件等
        refreshContext(context);
        afterRefresh(context, applicationArguments);
        stopWatch.stop();
        // 打印启动信息：如果 spring.main.log-startup-info 为 true，则打印启动信息
        if (this.logStartupInfo) {
            new StartupInfoLogger(this.mainApplicationClass)
                    .logStarted(getApplicationLog(), stopWatch);
        }
        // 发布 ApplicationStartedEvent：通知所有的 SpringApplicationRunListeners 应用已经启动
        listeners.started(context);
        
        // 调用 Runner：调用所有的ApplicationRunner和CommandLineRunner
        callRunners(context, applicationArguments);
    }
    catch (Throwable ex) {
        handleRunFailure(context, ex, exceptionReporters, listeners);
        throw new IllegalStateException(ex);
    }

    try {
        // 运行中：通知所有的 SpringApplicationRunListeners 应用正在运行
        listeners.running(context);
    }
    catch (Throwable ex) {
        handleRunFailure(context, ex, exceptionReporters, null);
        throw new IllegalStateException(ex);
    }
    return context;
}

The refresh () method is called internally in refreshContext(context) . This method mainly focuses on finishBeanFactoryInitialization(beanFactory) instantiation of the Bean, which occurs earlier than finishRefresh().

public void refresh() throws BeansException, IllegalStateException {
    synchronized (this.startupShutdownMonitor) {
        // 准备刷新的上下文环境：设置启动日期，激活上下文，清除原有的属性源
        prepareRefresh();

        // 告诉子类启动 'refreshBeanFactory()' 方法，创建一个新的bean工厂。
        ConfigurableListableBeanFactory beanFactory = obtainFreshBeanFactory();

        // 为 BeanFactory 设置上下文特定的后处理器：主要用于支持@Autowired和@Value注解
        prepareBeanFactory(beanFactory);

        try {
            // 为 BeanFactory 的处理提供在子类中的后处理器。
            postProcessBeanFactory(beanFactory);

            // 调用所有注册的 BeanFactoryPostProcessor Bean 的处理方法。
            invokeBeanFactoryPostProcessors(beanFactory);

            // 注册 BeanPostProcessor 的处理器，拦截 Bean 创建。
            registerBeanPostProcessors(beanFactory);

            // 为此上下文初始化消息源。
            initMessageSource();

            // 为此上下文初始化事件多播器。
            initApplicationEventMulticaster();

            // 在特定的上下文子类中刷新之前的进一步初始化。
            onRefresh();

            // 检查监听器 Bean 并注册它们：注册所有的ApplicationListenerbeans
            registerListeners();

            // 实例化所有剩余的（非延迟初始化）单例。
            finishBeanFactoryInitialization(beanFactory);

            // 完成刷新：发布ContextRefreshedEvent，启动所有Lifecyclebeans，初始化所有剩余的单例（lazy-init 单例和非延迟初始化的工厂 beans）。
            finishRefresh();
        }
        ...
    }

When instantiating a bean, you need to be familiar with the life cycle of the bean (important)





 

2) Release process of JSF Provider (version 1.7.5-HOTFIX-T6)

Class com.jd.jsf.gd.config.spring.ProviderBean calls the method com.jd.jsf.gd.config.ProviderConfig#export for publishing

JSF source code address: http://xingyun.jd.com/codingRoot/jsf/jsf-sdk

public class ProviderBean<T> extends ProviderConfig<T> implements InitializingBean, DisposableBean, ApplicationContextAware, ApplicationListener, BeanNameAware {
    
    // 此处代码省略...

    public void onApplicationEvent(ApplicationEvent event) {
        if (event instanceof ContextRefreshedEvent && this.isDelay() && !this.exported && !CommonUtils.isUnitTestMode()) {
            LOGGER.info("JSF export provider with beanName {} after spring context refreshed.", this.beanName);
            if (this.delay < -1) {
                Thread thread = new Thread(new Runnable() {
                    public void run() {
                        try {
                            Thread.sleep((long)(-ProviderBean.this.delay));
                        } catch (Throwable var2) {
                        }

                        ProviderBean.this.export();
                    }
                });
                thread.setDaemon(true);
                thread.setName("DelayExportThread");
                thread.start();
            } else {
                this.export();
            }
        }

    }

    private boolean isDelay() {
        return this.supportedApplicationListener && this.delay < 0;
    }

    public void afterPropertiesSet() throws Exception {
        // 此处代码省略...

        if (!this.isDelay() && !CommonUtils.isUnitTestMode()) {
            LOGGER.info("JSF export provider with beanName {} after properties set.", this.beanName);
            this.export();
        }

    }
}

public synchronized void export() throws InitErrorException {
    if (this.delay > 0) {
        Thread thread = new Thread(new Runnable() {
            public void run() {
                try {
                    Thread.sleep((long)ProviderConfig.this.delay);
                } catch (Throwable var2) {
                }

                ProviderConfig.this.doExport();
            }
        });
        thread.setDaemon(true);
        thread.setName("DelayExportThread");
        thread.start();
    } else {
        this.doExport();
    }

}



It can be seen that there are two places where Provider is released

Ⅰ. Bean initialization process (delay>=0)

Implement the InitializingBean interface and override the afterPropertiesSet method. Here it will be judged whether to delay the release. If it is greater than or equal to 0, it will be released here. Specifically, in the export method, when delay>0, the release will be delayed. For example, if 5000 is configured, it means the release will be delayed for 5 seconds; when delay=0, the release will be released immediately.



Ⅱ. Listen to the ContextRefreshedEvent event trigger (delay<0)

Implement the ApplicationListener interface and override the onApplicationEvent method. It belongs to the event ContextRefreshedEvent. When delay<-1, the release will be delayed. For example, if -5000 is configured, it means delayed release for 5 seconds; otherwise, it will be released immediately.



3) Solution



Scenario 1: Automatically publish Provider in XML mode (commonly used)

From the above introduction, we know the execution sequence : 1. Bean initialization > 2. ContextRefreshedEvent event trigger > 3. Call ApplicationRunner or CommandLineRunner;

It has been known above that provider publishing is in the 1 and 2 processes , and it is necessary to avoid using method 3 to initialize data.

Prerequisite suggestion: The default configuration of delay is -1, which can be left unconfigured or a negative number. Then JSF Provider publishing is in process 2, that is, listening to the ContextRefreshedEvent event to trigger



Method 1: During the initialization process of Bean

Solution: Use the @PostConstruct annotation, implement the InitializingBean interface, and configure the init-method method.

@Component
public class DataLoader {

    @PostConstruct
    @Scheduled(cron = "${cron.config}")
    public void loadData() {
        // 数据加载
        System.out.println("数据加载工作");
    }

}

Note: If the bean depends on other beans, you need to ensure that the dependent beans have been instantiated, otherwise a null pointer exception will be reported.



Method 2: ContextRefreshedEvent event triggered

How the ContextRefreshedEvent event is published

调用过程 AbstractApplicationContext#finishRefresh -> AbstractApplicationContext#publishEvent-> SimpleApplicationEventMulticaster#multicastEvent

public void multicastEvent(final ApplicationEvent event, @Nullable ResolvableType eventType) {
   ResolvableType type = (eventType != null ? eventType : resolveDefaultEventType(event));
   for (final ApplicationListener<?> listener : getApplicationListeners(event, type)) {
      Executor executor = getTaskExecutor();
      if (executor != null) {
         executor.execute(() -> invokeListener(listener, event));
      }
      else {
         invokeListener(listener, event);
      }
   }
}

Call invokeListener() in the multicastEvent method of SimpleApplicationEventMulticaster to publish events . The default value of getTaskExecutor() is null (except for custom-set Executor objects). All ApplicationListener implementation classes execute the onApplicationEvent method serially.

getApplicationListeners(event, type) obtains all implementation classes. If you continue to look down, AnnotationAwareOrderComparator.sort(allListeners) will be called to sort all ApplicationListeners . allListeners is a list of objects to be sorted. This method will determine the sort order based on the sort annotation or interface on the object and return a list of objects sorted in the specified order. Specifically, the sorting rules are as follows:

1. First, sort according to the value of the @Order annotation on the object . The smaller the value of the @Order annotation, the higher the sorting priority .

2. If there is no @Order annotation on the object, or the @Order annotation value of multiple objects is the same, the objects will be sorted according to whether they implement the Ordered interface. Objects that implement the Ordered interface can return a sorting value through the getOrder() method.

3. 如果对象既没有 @Order 注解，也没有实现 Ordered 接口，则使用默认的排序值 LOWEST_PRECEDENCE（Integer.MAX_VALUE）。特别的：如果BeanA和BeanB排序值都是默认值，则保持原顺序，即Bean的加载顺序



总结：默认情况所有ApplicationListener实现类串行执行onApplicationEvent方法，而顺序取决于AnnotationAwareOrderComparator.sort(allListeners)，@Order 注解的值越小，排序优先级越高

解决方法：使用@Order注解保证执行顺序早于ProviderBean

@Component
@Order(1)
public class DataLoader implements ApplicationListener<ContextRefreshedEvent> {
    @Override
    public void onApplicationEvent(ContextRefreshedEvent event) {
        // 数据准备
        System.out.println("初始化工作");
        
    }
}

此外带有@SpringBootApplication的启动类中实现也是可以的（在Spring Boot中默认使用基于注解的方式进行配置和管理Bean，所以注解定义的Bean会在XML定义的Bean之前被加载）

@SpringBootApplication
public class DemoApplication implements ApplicationListener<ContextRefreshedEvent> {

    public static void main(String[] args) {
        SpringApplication.run(DemoApplication.class, args);
    }

    @Override
    public void onApplicationEvent(ContextRefreshedEvent event) {
        System.out.println("初始化工作");
    }
}

场景2：API方式发布Provider（较少使用）

应用启动完成后，先做初始化动作，完成后再手动发布Provider。这种就可以通过实现接口ApplicationRunner或接口CommandLineRunner去执行初始化。

@Component
public class DataLoader implements ApplicationRunner {

    @Override
    public void run(ApplicationArguments args) throws Exception {
        // 数据准备
        System.out.println("初始化工作");

        // 发布provider
        // 参考：https://cf.jd.com/pages/viewpage.action?pageId=296129902
    }
}

场景3：XML方式手动发布（不常用）

provider的dynamic属性设置为false

标签	属性	类型	是否必填	默认值	描述
provider	dynamic	boolean	否	true	是否动态注册Provider，默认为true，配置为false代表不主动发布，需要到管理端进行上线操作



3. 总结

RPC服务（如JSF、Dubbo）进行优雅上线，常用的两种方式：1、延迟发布 2、手动发动

如果你的服务需要一些初始化操作后才能对外提供服务，如初始化缓存（不限与藏经阁、ducc、mysql、甚至调用其他jsf服务）、redis连接池等相关资源就位，可以参考本文中介绍的几种方式。

此文是笔者通过读取源码+本地验证得出的结论，如有错误遗漏或者更好的方案还烦请各位指出共同进步！



作者：京东零售 郭宏宇

来源：京东云开发者社区 转载请注明来源

博通宣布终止现有 VMware 合作伙伴计划 deepin-IDE 版本更新，旧貌换新颜 周鸿祎：鸿蒙原生必将成功 WAVE SUMMIT 迎来第十届，文心一言将有最新披露！ 养乐多公司确认 95 G 数据被泄露 2023 年各编程语言中最流行的许可证 《2023 中国开源开发者报告》正式发布 Julia 1.10 正式发布 Fedora 40 计划统一 /usr/bin 和 /usr/sbin Rust 1.75.0 发布

{{o.name}}

{{m.name}}