Attempts to compile and integrate FFmpeg for Android and step on the pit

Preface and Environment Notes

With the continuous iteration of FFmpeg, NDK and Android Studio, this article may become invalid like the outdated articles I have referred to (unfortunately), but I hope that the troubleshooting and step-by-step instructions mentioned in this article can help you, if found You are also welcome to provide suggestions and corrections for the fallacies and deficiencies in the article, thank you very much.

The initial goal is to use FFmpeg to implement simple video clips, add background music, add subtitles and other functions in Android. Since I am a beginner in Android development, my ability is limited and my foundation is weak, so I cannot comprehensively understand the problems encountered in the process of learning. The article may Please forgive me if there are some parts that I know but don't know why, or some inappropriate and imprecise personal understandings.

Device: macOS Big Sur 11.6 (Apple Silicon M1)

FFmpeg version: 4.4

Development environment:

Android Studio Arctic Fox | 2020.3.1 Patch 3 arm64 preview

JavaVersion = 1.8

minSdk : 21

NDK Version: 23.1.7779620

At present, the NDK can only use Rosetta 2 translation under the Apple chip.

CMake Version : 3.22.0-rc2

The latest version that can be obtained from the SDK Manager in Android Studio is 3.18.1

CMake has provided support for Apple chips after version 3.19.3

Gradle Version：7.0.3

Pre-knowledge preparation

Before actually doing it, I read the official development documents of Android and Java and several excellent related articles. According to my own understanding and knowledge level, I compiled some basic and simple explanations of concepts to facilitate understanding of the next step.

Native layer

JNI

GDR

Cross compilation, build system and CMake

ABI and Dynamic Link Libraries

FFmpeg

Native layer of Android system

Although many APIs of the Android system are developed in Java, many core Android system components and services (such as ART and HAL, etc.) are written in C/C++ and require Native libraries written in C/C++. Therefore, in addition to providing the JDK (Java Development Kit) required for developing Java code, Android also provides the NDK (Native Development Kit) for developers to develop the Native layer.

Java runs on the Java virtual machine, so it realizes features such as easy portability and cross-platform operation, but this also makes Android need to rely on some "Native" codes to access the bottom layer of the system to complete some tasks that Java cannot achieve. Because of this, "native" languages such as C/C++ also make Android programs lose the cross-platform feature. When compiling C/C++ programs for Android, you need to consider the CPU architecture and operating system version used by the target machine. .

JNI

JNI, or Java Native Interface, is an interface provided by Java for communicating with programs written in other languages, which defines the way Java bytecode interacts with Native code. Here we use JNI through NDK to realize mutual calling between Java code and C/C++ code in Android program.

Here are some problems encountered and some quick answers that I think can be temporarily passed:

The difference between dynamic link library (.so) and static library (.a)
- The code in the static library directly enters the executable file after compilation, while the dynamic link library contains the code in the library file outside the program, which is called by the program at runtime and cannot be executed separately

What is a Toolchain?

A set of tools provided in the NDK for cross-compiling C/C++ code

What exactly is Cmake used for here?

CMake generates a standard construction file that guides the toolchain to compile according to the CMakeLists.txt configuration file, and then the toolchain can compile the source code into a dynamic link library according to the construction file

When we compile a .c file, we can directly throw it into gcc to compile; but when we need to compile a series of .c files of a project, throwing it in and compiling it will obviously mess up, so we need A build system to manage compilation of this project. We use the .sln file of Visual Studio under Windows, the .xcodeproj file of Xcode under macOS, and the Makefile file of Make under Linux. The build files for these build systems instruct the compiler or build toolchain to compile .c files.

What about Ninja ?

Ninja is a construction system that focuses on compilation speed, and everyone agrees with it

GDR

NDK is Native Development Kit, which allows us to use libraries written in C/C++ language in Android development.

In Android development, we should first write the Native method in the Java file, then implement the Native method in the C/C++ file, then use the NDK toolchain to compile the C/C++ code into a dynamic link library, and then use Android Studio's Gradle Package our compiled library into APK. Then when the program is run, the Java code can call the Native methods in the library through the Java Native Interface (JNI) framework.

Cross compilation, build system and CMake

Cross Compile (Cross Compile) refers to compiling a program suitable for the target machine architecture on a compiling machine with a different processor architecture from the target machine. If we want to compile an Android device running on an arm architecture on a PC with the x86_64 platform For C/C++ programs, you need to use the cross-compilation toolchain (Toolchain), which is a series of tools for cross-compilation. Here we use the default toolchain provided by NDK (after r19 version, NDK no longer supports independent toolchain).

When we compile a .c file, we can directly throw it into gcc for compilation; but when we need to compile a series of .c files of a project or integrate existing libraries, it will obviously be a big problem to throw it in and compile it. It's messed up, so we need a build system to manage the compilation of this project. For example, there are .sln files of Visual Studio under Windows, .xcodeproj files of Xcode under macOS, Makefile files of Make or .ninja files of Ninja under Unix, etc.

The build files of these build systems instruct the compiler or build toolchain to compile the entire project. For simpler construction system files like Makefile or .ninja, we can try to write a hand-written one for construction, but when our construction and compilation involve cross-platform cross-compilation, we need to write different files for different target platforms , so it is currently more common practice to use a higher-level build system like CMake to generate these build files.

CMake is short for Cross platform Make. CMake is an open source cross-platform compilation tool (also known as "meta-construction system"), which can generate a standard construction file that guides the toolchain to compile according to the CMakeLists.txt configuration file (different constructions can be selected under different platforms The system's construction file), and then the tool chain can compile the source code into a dynamic link library according to the construction file.

Android Studio recommends using CMake + Ninja + NDK built-in toolchain for Native library development.

ABI

ABI stands for Application Binary Interface. The ABI contains the following information

Available CPU instruction sets (and extensions).

Endianness of memory stores and loads at runtime. Android is always little-endian (little endian).

Specifications for passing data between applications and the system, including alignment restrictions, and how the system uses the stack and registers when calling functions.

The format of executable binaries, such as programs and shared libraries, and the content types they support.

How to mang C++ names.

When we write Java code, because Java runs on the Java virtual machine, we don't need to care about the specific hardware conditions, architecture or CPU of the device, but when we need to use Native code in Android programs, because different Android devices use different CPU, and different CPUs support different instruction sets, and each combination of CPU and instruction set has its own ABI. Therefore, we need to build and compile .so dynamic link libraries adapted to different ABIs for different Android ABIs.

When we package these libraries compiled for different ABIs into APKs, these APKs can only be installed and used by Android devices with specific ABIs. For example: the arm64-v8a image supported by the Apple chip cannot install the APK package specially compiled for armeabi-v7a. When compiling, we can control which ABI library to compile and package in the ndk.abiFilters parameter of Gradle.

FFmpeg

FFmpeg is an open source, cross-platform complete solution for audio and video recording, transcoding and stream processing developed in C language, which is used by many open source projects.

After combing the above text, I must have a certain understanding of the basic logic and behavior of using the Native library under Android. Let's sort it out through a few flowcharts:

Write CMakeLists.txt, add C/C++ code and imported FFmpeg library to the project, and link them together.

Write and call Native methods in Java classes

Implement the Native method in C/C++ code, and the Native method calls the FFmpeg library

Use CMake + Ninja and NDK toolchain to compile and link C/C++ code and imported FFmpeg library together

Gradle packages the dynamic link library into the APK

Compile FFmpeg

First download a copy of FFmpeg source code and compile it. You can choose FFmpeg build compiled by others or use a compiled script written by others to save a lot of trouble and skip this step. FFmpegKit is recommended here .

Android projects only support importing dynamic libraries ending in .so, such as: libavcodec-57.so . However, the default format of the dynamic library compiled by FFmpeg is xx.so. version number, such as: libavcodec.so.57, so it is necessary to modify the configure file in the root directory of FFmpeg to generate a dynamic library ending in .so format:

# 将 configure 文件中 build settings 下的：
SLIBNAME_WITH_MAJOR='$(SLIBNAME).$(LIBMAJOR)' 
LIB_INSTALL_EXTRA_CMD='$$(RANLIB) "$(LIBDIR)/$(LIBNAME)"' 
SLIB_INSTALL_NAME='$(SLIBNAME_WITH_VERSION)' 
SLIB_INSTALL_LINKS='$(SLIBNAME_WITH_MAJOR) $(SLIBNAME)'

#替换为：
SLIBNAME_WITH_MAJOR='$(SLIBPREF)$(FULLNAME)-$(LIBMAJOR)$(SLIBSUF)'
LIB_INSTALL_EXTRA_CMD='$$(RANLIB)"$(LIBDIR)/$(LIBNAME)"'
SLIB_INSTALL_NAME='$(SLIBNAME_WITH_MAJOR)'
SLIB_INSTALL_LINKS='$(SLIBNAME)'

FFmpeg has prepared a Makefile for us that can be directly used for construction, and also provides us with a configure program that can adjust compilation settings. configure provides many parameters to choose from, such as compilation modules, target platforms, compilation toolchains, etc., usually The method is to write a script for setup and construction, we create a new build.sh script file in the directory .

In the process of compiling, due to my low technical level, I took a lot of detours in the process of copying other people's strategy scripts.

Here is the script I used to compile FFmpeg for arm64-v8a under macOS. Be sure to modify it according to the instructions and your own toolchain. If you encounter problems during the compilation process, you must first check the log and read the official documents, the documents referred to here.

NDK_ROOT=						#NDK 根目录
TOOLCHAIN=$NDK_ROOT/toolchains/llvm/prebuilt/darwin-x86_64
#工具链目录 目前 NDK 在 M1 还只能在 Rosseta 转译下使用 x64 的工具链

export PATH=$PATH:$TOOLCHAIN/bin
target_arch=aarch64
target_host=aarch64-linux 								
#编译目标平台
toolchain_prefix=$target_host-android21		
#在 configure 中定义了新变量

#target_arch=arm
#target_host=armv7a-linux							
#toolchain_prefix=$target_host-androideabi21
#这里是编译armv7的选项

#这里的变量设置以及接下来对 configure 的编辑非常重要
#如果照抄之前的过期博文(或此文)设置脚本会导致编译失败
#详见下面的分析

PREFIX= #编译输出路径
ANDROID_API=21 #最小API

./configure \
    --prefix=$PREFIX \								
    #设定输出路径
    --enable-shared \								
    #生成动态链接库
    --disable-static \ 								
    #不生成静态库
    --enable-postproc \
    --enable-jni \
    --enable-cross-compile \ 						
    #启用交叉编译
    --extra-cflags="-D__ANDROID__API__=21 -U_FILE_OFFSET_BITS" \
    --cross-prefix=$target_host- \ 					
    #设定交叉编译目标前缀
    --cross_prefix_clang=$toolchain_prefix- \
    --arch=$target_arch \							
    #设定目标框架
    --target-os=android	\							
    #设定目标平台系统 iOS = darwin
    --sysroot=$TOOLCHAIN/sysroot 					
    #设定sysroot目录

make clean
make -j4
make install

illustrate:

Before writing the script, please cd go to the toolchain bin directory ls to check the file name format of the toolchain program. In the NDK 23.1.7779620 darwin toolchain I use, the situation is as follows:

……

aarch64-linux-android-as           
aarch64-linux-android21-clang      
aarch64-linux-android21-clang++    
aarch64-linux-android22-clang      
aarch64-linux-android22-clang++    
aarch64-linux-android23-clang      
aarch64-linux-android23-clang++    
aarch64-linux-android24-clang      
aarch64-linux-android24-clang++    
aarch64-linux-android26-clang      
aarch64-linux-android26-clang++    
aarch64-linux-android27-clang      
aarch64-linux-android27-clang++    
aarch64-linux-android28-clang   

………

llvm-ar
llvm-as
llvm-cfi-verify
llvm-config
llvm-cov
llvm-cxxfilt
llvm-dis
llvm-dwarfdump
llvm-dwp
llvm-lib
llvm-link
llvm-lipo
llvm-modextract
llvm-nm

……

It can be seen that the clang provided by NDK is prefixed with the Android version number. At this time, open the source code of the configure file, search for if test "$target_os" = android this line, and check the Android compilation settings, and you can find many problems:

The file names here are all set to be cross_prefix prefixed with our input, but after our inspection, our file name prefix is actually in aarch64-linux-android21 this ${cross_prefix}-android+版本号 format.

2. Here, cc_default is rewritten to clang, but cxx_default is not rewritten.

3. The strip, ar, pkg-config and nm tools here are also set to prefix with cross_prefix, but in fact, we can see that the prefix of several of our file names is actually llvm-, which can also be found in the Android official document Learned that binutils tools such as ar and strip do not require a prefix because they are not affected by minSdkVersion. And pkg-config is not built into the tool chain, we need to manually obtain it through the package manager. (Compilation did not fail even if I did not install it)

brew install pkg-config

It is important to note that the actual setup here is based on your own NDK toolchain.

In order to ensure correct compilation, the relevant code of configure is modified as follows:

set_default target_os
if test "$target_os" = android; then
    cc_default="clang" 
    cxx_default="clang++" 					
    
fi											
#将cxx_default重写
#注：ndk r17版本后已弃用gcc

ar_default="llvm-${ar_default}"						
#将前缀修改为llvm-
cc_default="${cross_prefix_clang}${cc_default}"		
#在CMDLINE_SET中定义一个新变量cross_prefix_clang并在脚本中输入
cxx_default="${cross_prefix_clang}${cxx_default}"	
#也可以直接修改成${cross_prefix}-android21-${cxx_default}
nm_default="llvm-${nm_default}"
pkg_config_default="${pkg_config_default}"			
#使用我们安装的pkg-config
ranlib_default="llvm-${ranlib_default} -D"
strip_default="llvm-${strip_default}"
windres_default="${cross_prefix}${windres_default}"

Run the script, if the compilation is successful, you can see that the folders include, bin, shareand have appeared in the output directory we set . Inside the folder is the compiled FFmpeg dynamic link library we need.liblib

Integrate FFmpeg in Android

After getting the dynamic link library of FFmpeg, we can't directly use it in Android applications. Because we haven't realized the mutual communication between Java code and C code: JNI. Many tutorials use ndk-build provided by NDK, but Android officials now recommend using CMake, we can directly call CMake with the help of the Gradle plug-in without the trouble of command line operation, please check whether Ninja is installed first.

You can simply follow these steps:

Create a new project, right click in the app directory, select Add C++ to Module, Android Studio will automatically generate ProjectName.cpp and CMakeLists.txt in the main directory, open CMakeLists.txt to observe the format, the generated comments are easy to read, here No longer.

……
add_library(
        ffmpegtest				#库名
        SHARED	
        ffmpegtest.cpp)		#实现JNI方法的cpp代码 自动生成文件名为项目名
……

Create a lib/arm64-v8a folder in the cpp directory, paste the dynamic link library in .so format under the lib directory of FFmpeg compiled in the previous step, and copy and paste the include folder into the cpp directory. (If other ABI libraries are compiled, create a new subdirectory named ABI under the lib directory)

Open CMakeLists.txt for editing and add the following:

add_library(avcodec 								
			#库名 注意无lib前缀
        SHARED										
        #SHARED	表示动态链接库
        IMPORTED)									
        #IMPORTED 表示外部导入库
        
set_target_properties(avcodec			
#设置avcodec库的导入路径
        PROPERTIES IMPORTED_LOCATION
        ${CMAKE_SOURCE_DIR}/lib/${CMAKE_ANDROID_ARCH_ABI}/libavcodec.so)
       	#CMAKE_SOURCE_DIR是CMakeLists.txt所在目录 请勿省略
        	#CMAKE_ANDROID_ARCH_ABI是下文在Gradle中设置的abifilter参数 
        	#此处是arm64-v8a	请勿省略
        													
……															#如上格式添加所有FFmpeg动态链接库

include_directories(${CMAKE_SOURCE_DIR}/include)
															#添加FFmpeg头文件
																	
target_link_libraries(
        ffmpegtest
        
        avfilter									
        avformat									
        #将所有动态链接库与ffmpegtest库（实现了JNI）链接
        #至此 可以在java代码内通过JNI调用FFmpeg库中函数
        #同时在ffmpegtest.cpp中的JNI函数中调用Java中的方法 
        #从而实现在Android中实现与FFmpeg库交互
        avdevice
        avcodec
        avutil
        swresample
        swscale
        postproc
        ${log-lib})

Initialize the library in a class or application, and write a native method without a function body in the class and call it. At this time, the method will report an error. ⌥(Alt) + Enter will let Android Studio ffmpegtest.cppgenerate

private native void run();
static{
    System.loadLibrary("ffmpegtest");
}

……

#include <jni.h>
#include <android/log.h>						//导入Android log头文件
extern "C"{										//FFmpeg由C写成 注意使用C关键字括起来
#include "libavcodec/avcodec.h"					//导入FFmpeg依赖库头文件
JNIEXPORT void JNICALL
Java_com_example_ffmpegtest_MainActivity_run(JNIEnv *env, jobject thiz) {
    __android_log_print(ANDROID_LOG_INFO,"FFmpegTag",
                        "avcodec_configuration():\n%s",avcodec_configuration()); 
  																			//输出avcodec配置到logcat
}
}

Open build.gradle at module level

Check ndkVersion and cmake.version

Add ndk{abiFilters "arm64-v8a"} in defaultConfig.externalNativeBuild. If you do not specify this parameter, Gradle will build all ABI applications. If you have compiled FFmpeg for other ABIs and want to build applications for other ABIs, here Add to.

Do not add sourceSets.main.jniLibs.srcDirs, this parameter is obsolete and will cause the build to fail

Build and run, you can see the information sent from the FFmpeg function in the cpp code in logcat. So far, we have initially compiled and integrated FFmpeg in Android.

Troubleshooting

Emphasis again: If you encounter problems during the compilation process, you must first check the log and read the official documents

Most of the problems are caused by incorrectly written CMakeLists.txt. Please check the CMake error report and build log first to confirm that there are no errors in the header file addition path, dynamic library addition path, and link library name of the library under CMakeLists.txt.

Check that CMake and NDK versions are explicitly defined in build.gradle, and that CMake and Ninja are configured correctly

Check if abiFilters are configured

APK analyze checks whether the dynamic link library is successfully packaged into the application

Next, you can take a look at using the FFmpeg command line tool for audio and video operations

References

【LianchuangのAlchemy Workshop】Hello World of Android NDK

GCC/Make/CMake of GCC bzdww

Android integrated FFmpeg (1) basic knowledge and simple call - yhao's blog - CSDN blog

Using the NDK with other build systems | Android NDK | Android Developers

Android ABI | Android NDK | Android Developers

Automatically package pre-built dependencies used by CMake | Android Developers | Android Developers

Compile clang of FFmpeg,

Original link: Attempts to compile and integrate FFmpeg for Android and step on the pit

★The business card at the end of the article can receive audio and video development learning materials for free, including (FFmpeg, webRTC, rtmp, hls, rtsp, ffplay, srs) and audio and video learning roadmaps, etc.

see below!