Author: JD Technology Feng Jianhua
1. Background
With the continuous iterative update of the business, the size of the App is also increasing rapidly. From 2019 to 2022, it once exceeded 117M. During this period, we also made some optimizations, as shown in the red part of Figure 1. However, we are facing new challenges while optimizing. Incremental code, the package size has continued to rise . Package size directly or indirectly affects important indicators such as download conversion rate, installation time, and disk space, so it is necessary to devote energy to exploring deeper optimization of installation package size. According to the internal data of Google Store, every 10M reduction in APK size can increase the download conversion rate by an average of ~1.5%, as shown in Figure 2:
Figure 1 The volume change process of JD Finance Android version 2019-2022 (the red part is part of the optimization done during the period, but it bounced back soon)
Figure 2 Google Store app conversion rate increase / 10M [1]
Therefore, starting from September 2022, we have carried out a special rectification for the slimming of the financial APP. Without considering the increase in the situation and without deleting the business code, we realized the slimming from 117M to 74M. In the process of slimming down the installation package this time, we encountered A lot of pitfalls, but also accumulated some experience, here to share with you.
2. APK Analysis
Next, we will briefly analyze the various components in the Apk, and the standard structure of the Apk as a ZIP, to provide data support for the goal setting and task disassembly of the package slimming.
2.1 APK content analysis
Figure 3 APK structure
•classes.dex APK may contain one or more classes.dex files, and the Java/Kotlin source code in the application will eventually exist in the classes.dex file in the form of bytecode.
•resources.arsc The aapt tool will package some resources or resource indexes into resources.arsc when compiling resources.
•res/ Resource files except values under the res directory in the source code project, and the paths of these files will be recorded in resources.arsc at the same time.
• lib/ nativeLibraries, that is, the so file under the jni directory of the source code project, and the secondary directory is the ABI supported by NDK.
•assets/ is different from the res/ resource directory. The resource files under assets/ will not generate query entries in resources.arsc, and the resource directory under assets/ can be completely customized and obtained through the AssetManager object in the program.
•META-INF/ This folder mainly contains CERT.SF and CERT.RSA signature files, and MANIFEST.MF manifest file .
•AndroidManifest.xml application manifest file, used to describe the basic information of the application, mainly including the application package name, application id, application components, required permissions, device compatibility, etc.
2.2 SDK size analysis
Through our self-developed energy efficiency improvement platform Pandora[7], we can intuitively see the size of the SDK, as shown in Figure 4:
Figure 4 SDK size sorting (including version number)
Figure 5 List and size of SO libraries included in the SDK
According to the analysis of the SDK and combined with the business, determine which business is suitable for plug-in, and then intuitively reduce the package size.
2.3 ZIP structure analysis
You can use the zipinfo command to output the detailed information log of each file in the compressed package, usage: zipinfo -l --t --h test.apk > test.txt
The output log file is opened as shown in Figure 6. There is one line of compression information for each file, including file name, original size, compressed size and other indicators:
Figure 6 The size of the file information in the APK
Analyze the above log information line by line, and classify and count according to the deobfuscated file name path and file type, and you can get the overview information of Apk, including the number of various types of files, total size, single file size and other indicators. And create a file size index.
3. Slimming practice
The overall implementation path is shown in Figure 7, which is mainly divided into:
1. Conventional technical solutions, through the Gradle plug-in (no code intrusion, automation) to complete APP slimming during compilation;
2. Advanced technical solutions, transform some business lines differently through plug-in or SO dynamic download. The more business transformations, the higher the benefits;
3. Business optimization plan, based on the data burying point of the business line, generate access UV for ranking, and feed back the business line with lower UV to the Architecture Committee to evaluate whether it can be offline or transformed through the advanced technical solution (2), and then Reduce package size.
Figure 7 overall implementation path
3-1 Conventional technical solutions
3-1-1 Image processing
After the above analysis of the APP, it is concluded that the image occupies the largest volume. Therefore, all the images in the APP including the SDK are automatically optimized through the slimming task during the compilation and packaging process. The overall optimization scheme is shown in Figure 8:
Figure 8 Picture optimization scheme
1. Multi-DPI optimization:
In order to adapt to devices with different resolutions or modes, Android has designed resource paths for developers with multiple configurations of the same resource. When the app obtains image resources through resources, it automatically loads the adapted resources according to the device configuration, but these configurations are accompanied by The obvious problem is that high resolution devices contain low resolution useless images or low resolution devices contain high resolution useless images.
Under normal circumstances, for the domestic application market, in order to reduce the package size, the App will choose a set of dpi with the highest market share (google recommends xxhdpi) to be compatible with all devices. Most of the apps targeting the overseas application market will be packaged and uploaded to Google Play through AppBundle, and can enjoy the function of dynamically distributing dpi. Mobile phones with different resolutions can download image resources of different dpi, so we need to provide multiple sets of dpi to satisfy all devices. . In the project, some of our pictures have only one set of dpi, and some have multiple sets of dpi. For the above two scenarios, we merged resources and copied resources when packaging, reducing the package size.
2. Convert to webp format:
WebP is an image file format provided by Google that supports lossy compression and lossless compression, and can provide better compression than JPEG or PNG. Supports lossy WebP images in Android 4.0 (API level 14), lossless and transparent WebP images in Android 4.3 (API level 18) and higher
Therefore: we use the plug-in to convert the format of the image through the shell program provided by Google during the compilation period, and the conversion successfully deletes the old image, thereby achieving the effect of APK slimming
3.png compression
Pngquant is an easy-to-use png compression tool, a command-line tool that can perform lossy image compression, so after the processing of 1 and 2, you can use Pngquant to perform secondary compression to achieve better image slimming.
3-1-2 R file inline optimization
DEX is the bytecode file compiled from Java/Kotlin source code. The optimization of DEX is actually how to optimize the bytecode file. DEX contains a large number of resource index R files. Here we mainly talk about how to inline through the resource ID. Delete the R file to achieve the purpose of APK slimming:
Feasibility analysis of R file slimming
In the daily development stage, reference resources in the main project through R.xx.xx. After compilation, the constants corresponding to the R class references will be compiled into the class.
setContentView(2131427356);This change is called inlining, which is a mechanism of java (if a constant is marked as static final, the constant will be inlined into the code during java compilation, reducing the memory addressing of variables once). In the non-main project, the R class resource ID is compiled into the class by reference, and no inlining will be generated.
setContentView(R.layout.activity_main);The reason for this phenomenon is caused by the AGP packaging tool. For details, you can check the processing process of the android gradle plugin on the R file. Conclusion: The program can run after the R class id is inlined, but not all projects will automatically generate inline phenomenon. We need to use technical means to inline the R class id into the program at the right time. If you rely on R files again, you can delete the R files to achieve the purpose of reducing the size of the package while the application is running normally. As shown in Figure 9, a large number of R files will be generated after the compilation is completed:
Figure 9 Schematic diagram of project R file generation
The overall scheme is shown in Figure 10:
Figure 10 R file optimization process
Note : In the replacement phase, a secondary check must be added to prevent the ResourceNotFind exception from appearing at runtime after the replacement is completed, as shown below:
try {
int value = RManager.checkInt(type, name);
}catch (Exception e){
String errorMsg = "resource is not found(I),className="+className+",fieldName="+owner+"."+name;
throw new ResourceNotFoundException(errorMsg);
}
try {
int[] value = RManager.checkIntArray(type, name);
}catch (Exception e){
String errorMsg = "resource is not found(I[]),className="+className+",fieldName="+owner+"."+name;
throw new ResourceNotFoundException(errorMsg);
}
3-1-3 AndResGuard for resource confusion
1. Analysis of resource loading process
During the development process, we use the constants in R.java generated by aapt to use resources, and the places where the constants are used after compilation will be replaced with the constant values, as shown below:
final View layout = inflater.inflate(2131165182, container, false);That is to say, we use an int value to find the resource through Resource. So how does Resource find specific resources through int values? When we unzip the apk, we can see that there is a resources.arsc file inside. This file is also generated by aapt. The mapping relationship between resource id and resource key is stored in the file. Resource finds resources according to this mapping relationship.
2.resources.arsc：
Figure 11 shows the mapping relationship stored in resources.arsc. resources.arsc can be understood as a resource mapping database, and the specific path and name are mapped according to the ID.
Figure 11 resources.arsc analysis
After decompressing the APK, convert the resource file name into a short chain, such as res/layout/hello.xml into r/l/a.xml, and then change the value corresponding to resources.arsc to achieve the overall slimming effect.
AndResGuard[5] is a resource optimization tool launched by WeChat. Its basic idea is similar to the obfuscation in ProGuard, which can realize the above solutions.
3-1-4 7zip compression
7zip command explanation:
-t: Specifies the compression type, supports 7z, xz, split, zip, gzip, bzip2, tar, ....
-m: Specifies the compression algorithm, the default is Deflate
The specific process is as follows:
Step 1: Use the 7z command to decompress the unsigned package to the specified directory: 7za x ${unsigned package} -o${7z decompressed directory}
Step 2: First, use the 7z command to compress all the decompressed directories: 7za a -tzip -mx9 ${target 7z file name} ${7z decompressed directory}
Step 3: Obtain storage type files, and obtain a list of files whose compression mode is Stored through the aapt command in the Android SDK: aapt l -v ${unsigned package}
Step 4: Update the storage type file, and use the 7z command to update the storage type file to the 7zip installation package generated in the second step: 7za a -tzip -mx0 ${target 7z file name} ${storage type file directory}
3-1-5 Configuring the CPU architecture
Build different types of installation packages according to different CPU architectures. At present, mainstream devices are 64-bit machines, so the Android market mainly releases installation packages compiled and constructed based on arm64-v8a
ndk {
abiFilters arm64-v8a
}3-1-6 arsc compression
resources.arsc has a high volume gain for compression, but compressing it affects startup speed and memory metrics. The reason is: when the system loads the arsc file, if the arsc file is not compressed, mmap can be used for memory mapping; if the arsc file is compressed, it needs to be decompressed and read into the RAM buffer, which will increase memory usage and also It will slow down the startup speed.
For the same consideration, the official cannot use this method to force resources.arsc after targetSdkVersion>=30, otherwise the installation will fail directly, so this article will not elaborate.
3-1-7 Internationalized language processing
The JD Financial App is currently only operating in the domestic market, but dozens of languages have been added to a large number of SDKs that are connected , resulting in a larger overall size. After evaluation, useless language resources can be removed by configuring resConfigs.
defaultConfig {
resConfigs "zh","en"
}3-1-8 shrinkResources
shrinkResources: Used to detect and delete useless resource files during compilation, that is, resources that are not referenced
minifyEnabled: used to enable the deletion of useless codes, such as codes that are not referenced, so if you need to know whether a resource is referenced, you must use it in conjunction with minifyEnabled. Only when both are true will it truly delete invalid codes and unreferenced resources the goal of.
Its function is to replace unreferenced resource files with a small format file (there is still a footprint, while retaining the resource entry, so the volume of resources.arsc will not be reduced) , which can be accessed through res/raw/ The keep.xml file configures shrinkMode and whitelist.
buildTypes {
release {
// 不显示Log
buildConfigField "boolean", "LOG_DEBUG", "false"
//混淆
minifyEnabled true
// 移除无用的resource文件
shrinkResources true
proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
signingConfig sign.release
}
}3-1-9 Coding Constraints
• Use enumeration types as little as possible, because enumeration will increase a lot of volume after being compiled into bytecode, as shown in Figure 12 (the bytecode after 22 lines of code is compiled is 86 lines)
Figure 12 Comparison of compiled bytecodes of enumerated types
• Remove unnecessary LOG output
3-2 Advanced technical solutions
The SO library dynamic download and plug-in technology are essentially a category of dynamic download. The two solutions can be used continuously in the business for a long time. How to choose in the specific use process is shown in Figure 13:
Figure 13 How businesses choose advanced solutions
3-2-1 SO library dynamic loading
Some businesses in the APP are not suitable for plug-in transformation. After dismantling, it is found that the SO library accounts for a large proportion. Therefore, dynamic downloading can be considered for transformation to reduce the size.
Two ways of SO library loading
In the first way, we can directly download the SO library and put it in the specified directory.
The second way is to load the SO library in the directory set by the environment variable, so we need to add the specified directory to the environment variable to load the SO library normally
System.load("{安全路径}/libxxx.so")
System.load("xxx") 1. How to set the environment variable location of the SO library in the APP (referring to Tinker):
final Field pathListField = ShareReflectUtil.findField(classLoader, "pathList");
final Object dexPathList = pathListField.get(classLoader);
final Field nativeLibraryDirectories = ShareReflectUtil.findField(dexPathList, "nativeLibraryDirectories");
List<File> origLibDirs = (List<File>) nativeLibraryDirectories.get(dexPathList);
if (origLibDirs == null) {
origLibDirs = new ArrayList<>(2);
}
final Iterator<File> libDirIt = origLibDirs.iterator();
while (libDirIt.hasNext()) {
final File libDir = libDirIt.next();
if (folder.equals(libDir)) {
libDirIt.remove();
break;
}
}
origLibDirs.add(0, folder);
final Field systemNativeLibraryDirectories = ShareReflectUtil.findField(dexPathList, "systemNativeLibraryDirectories");
List<File> origSystemLibDirs = (List<File>) systemNativeLibraryDirectories.get(dexPathList);
if (origSystemLibDirs == null) {
origSystemLibDirs = new ArrayList<>(2);
}
final List<File> newLibDirs = new ArrayList<>(origLibDirs.size() + origSystemLibDirs.size() + 1);
newLibDirs.addAll(origLibDirs);
newLibDirs.addAll(origSystemLibDirs);
final Method makeElements = ShareReflectUtil.findMethod(dexPathList, "makePathElements", List.class);
final Object[] elements = (Object[]) makeElements.invoke(dexPathList, newLibDirs);
final Field nativeLibraryPathElements = ShareReflectUtil.findField(dexPathList, "nativeLibraryPathElements");
nativeLibraryPathElements.set(dexPathList, elements);2. How to delete the specified SO library and the entire loading process, as shown in Figure 14:
Figure 14 SO library deletion and loading process
3-2-2 Plug-in
What is plug-in:
Plug-in is to split an Apk into different sub-Apks (that is, different plug-ins) according to business functions. Each sub-Apk can be compiled and packaged independently, and the integrated Apk is finally released and launched. When Apk is used, each plug-in is dynamically loaded, and plug-ins can also be hot-fixed and hot-updated.
•Host: The main App can be used to load plugins and also become a Host
• Plug-in: Plug-in App, the App loaded by the host, can be the same Apk file as the normal App
What form of business is suitable for plug-in transformation:
• The business is relatively independent and completely decoupled from the host App
• Low renovation cost and relatively high income
• Occupies a large volume
After a series of evaluations, the video business meets the above points, and the effect after transformation is shown in Figure 15:
Figure 15 The effect of the plug-in transformation of the video business hall
3-3 Business optimization plan
With more and more businesses, some old business UVs are getting lower and lower, so a set of business offline optimization process has been formulated, as shown in Figure 16:
Figure 16 Process of business optimization solution
4. Control
The implementation of the slimming plan is very important, and it is more important that the follow-up control does not rebound. While doing slimming governance, we are exploring a normalized control mechanism on the other hand, and finally precipitated a set of control norms and control mechanisms. The purpose of management and control is not to limit business iteration or new code, but how to realize its functions in limited code and improve engineers' awareness of slimming in daily coding.
4.1 SDK access specification
In order to prevent the disorderly expansion of the SDK, the SDK access specification has been formulated. On the premise of ensuring the function, the size of the SDK is strictly controlled, and the rebound of the APP volume is controlled to the greatest extent.
4.2 Control process
Figure 17 Control process
According to the changes of resource files such as adding content, deleting content, increasing content, reducing content, duplicate files, code management, etc., combined with governance control specifications, etc., the packaging and construction will be compared with the historical version to obtain the changed content. To evaluate whether there is room for optimization, and give the optimization goal, and rebuild the packaging integration after optimization.
5. Achievements and follow-up planning
5.1 Achievements
Through the above measures, the Android version of JD Finance has been iterated by five versions in two quarters, from 117M to the current 74M (Figure 18), and the overall has been maintained within a controllable range. At the same time, in the next version iterations, we will normalize APK slimming, and always keep the package size within a controllable range.
Figure 18 Slimming results of financial APP
5.2 Subsequent planning
Continuous optimization of technical means:
The continuous accumulation and iteration of business will always generate some useless resources, so these useless files and codes should be cleaned up regularly to slim down the installation package;
Do a good job of monitoring each version, compare the differences between versions, and find that it can be optimized using technical means without affecting the business.
Online management and control platform construction:
In the early stage, offline management and control were used, which was a bit time-consuming to implement. In the future, we will improve the construction of the online management and control platform, integrate it with the entire App release and construction platform, form an assembly line mechanism, and do a good job in management and control.
Summary : There is still a long way to go to explore the slimming of installation packages. This article only lists some commonly used slimming solutions. In addition to optimization for huge projects, there is also a need to do a good job in governance between projects and continue to optimize the size of APP , to improve user experience.
【Reference】
[1] Package size and installation conversion rate
https://medium.com/googleplaydev/shrinking-apks-growing-installs-5d3fcba23ce2
[2] ProGuard https://www.guardsquare.com/proguard
[3] R8https://r8.googlesource.com/r8
[4] Comparison of ProGuard and R8
https://www.guardsquare.com/blog/proguard-and-r8
[5] AndResGuardhttps://github.com/shwenzhang/AndResGuard
[6] AGPhttps://developer.android.com/studio/releases/gradle-plugin
[7] Pandora: A tool for improving energy efficiency in R&D and testing phases based on decentralized technology