This column will share a series of technical articles on Alibaba Cloud Video Cloud MediaBox, provide an in-depth analysis of the technical architecture, technical performance, development energy efficiency and best practices of the audio and video development tool, and start the audio and video development journey together. This article is the technical architecture of MediaBox. It focuses on the technical architecture, optimized design, and architectural advantages of the audio and video terminal SDK to introduce how MediaBox SDKs can achieve technical architecture upgrades.
Good Teacher｜Author
In the era of streaming media, more and more companies are investing in the research and development of short videos, live broadcasts, audio and video calls and other related businesses. Especially the mobile terminal, as the main entrance of user traffic, has become increasingly fierce streaming media competition. battlefield.
Under this wave, many App manufacturers have rushed into the audio and video track, and often access more than one audio and video business, which may involve multiple audio and video services such as live broadcast, players, and audio and video calls.
In the past, when manufacturers accessed multiple audio and video services, they usually needed to access multiple audio and video SDKs. The result was that the package size increased rapidly, and the access efficiency was low, making it impossible to go online quickly, resulting in competition. Missed the opportunity.
MediaBox audio and video terminal SDK was born to solve this pain point of enterprises.
01 Concept Description
MediaBox audio and video terminal SDKs (MediaBox SDKs) use a unified technical base to form a series of SDKs with different capabilities to provide customers with a flexible, easy-to-use, and efficient access experience. Audio and video terminal SDK, audio and video low-code development AUI Kits, and audio and video scenario solutions built based on industry ecological cooperation, together form the overall picture of the MediaBox audio and video terminal integrated suite.
From a business perspective, MediaBox SDKs is a collection of basic terminal capabilities of Alibaba Cloud Video Cloud, including player, live broadcast, ultra-low latency live broadcast, real-time audio and video communication, short video creation and other audio and video service capabilities.
In terms of technical capabilities, the audio and video terminal SDK provides various technical capabilities such as audio and video collection, pre-processing, audio and video encoding, encapsulation/de-encapsulation, media network transmission, audio and video decoding, post-processing, and audio and video rendering.
02 Technical architecture
In order to build an SDK with high performance, high availability, high flexibility, pluggability and easy scalability, we divide the media engine of the SDK into three parts: microkernel, media component and terminal basic capability component.
Microkernel : As the core and control center of the business SDK, it is responsible for connecting and managing various media components to form a business pipeline, which mainly consists of message center, dispatch center, link management, queue management, component management, plug-in management and monitoring center. composition.
Media component : The basic capability unit of audio and video media processing, consisting of audio and video collection, pre- and post-processing, audio and video encoding and decoding, file encapsulation and decapsulation, media network output, and media rendering.
End-side basic capabilities : End-side high-availability supporting basic capabilities, providing multiple end-side capabilities such as license, configuration delivery, exception management, and hidden points.
03 Optimize design
Usually, audio and video end-side SDKs do not have a unified core and components. Each SDK is an independent business development, and there is no functional reuse. This makes it impossible to carry out unified iterative optimization of components. Instead, different SDKs need to be individually optimized for performance, Function optimization, or compatibility adaptation, resulting in the inability to quickly apply to business.
In this regard, Alibaba Cloud Video Cloud MediaBox has optimized the design of the kernel and components of the terminal SDK to achieve high-performance, high-stability, and high-availability architecture upgrades:
• Microkernel design, reconstructing all businesses based on a unified microkernel;
• Integrate all SDK components, extract and precipitate media components in a unified manner, and ensure that all components have consistent capabilities, consistent behavior, and stable performance;
• Unify all terminal-side basic capabilities, authorize SDK services in a unified manner, increase terminal-side troubleshooting and abnormal warning capabilities, and ensure high availability.
1. Microkernel design
After comprehensively considering the original engine layer design of each SDK, we designed a set of microkernels that can be used for all audio and video SDKs based on the original business model.
In order to achieve high performance and high stability, and ensure that the kernel is simple and easy to use without causing damage to the business, the microkernel design we adopt consists of the following parts:
• Message center : responsible for message distribution between various business components;
• Scheduling center : responsible for thread scheduling of running tasks of each business component;
• Link management : Responsible for managing the link relationships between various business components of the media engine;
• Queue management : Message queue management, cooperating with the dispatch center to manage the current task queue;
• Component management : Responsible for managing all components of the business, responsible for registering and deregistering components;
• Monitoring and management : Monitor the data processing of components and detect abnormalities in message processing or data processing in a timely manner;
• Plug-in management : Plug-in design is pre-embedded, making it easy to customize plug-ins and expand capabilities.
2. Component extraction and reuse
We conduct component-based management of the media components used in the video cloud terminal SDK, and implement unified extraction, reuse, and unified optimization and iteration. The distribution of components in each audio and video SDK is as shown in the figure:
Take the business pipeline in live broadcast scenarios and short video shooting scenarios as an example to show the series effect of each component:
Business Pipeline in Live Broadcasting Scenario
Business Pipeline in short video shooting scenarios
3. Component development and design
During the application development process, R&D students often divide a library with complex functions into several small components with single functions and responsibilities, and manage the components through Maven (Android) or Cocoapod (iOS). Then Gradle and Xcode will automatically sort out the dependencies between components, so that business students do not need to care about each component, only the business itself.
However, the target product of the video cloud terminal SDK is the SDK, not the App. Therefore, when we face a cross-platform terminal such as SDK, we need to manage multi-platform code. The functional component code is distributed as: C++ code (main code), object-c code (iOS), Java code (Android) and WebAssembly. So far, there is no good tool that has the ability to manage mixed code and meet the development needs of multiple platforms.
To this end, we referred to package management solutions such as CPM, Java Maven, Cocoapod and Npm, and customized a multi-platform component management and dependency solution for the MediaBox terminal SDK. Through debugging, efficient compilation, and dependency conflict warning, we Such capabilities enable components to be reused on a large scale and achieve unified version iteration.
* To implement the same component-based usage management for components on different platforms, there will be a special article introducing ACPM components in the future, so stay tuned.
4. Unified basic capacity building
In addition to the unification of the media processing part, basic capabilities have also been integrated and unified. For example, license management. In the past, each SDK had its own license management. When customers used it, they needed to activate authorization in different places. But after we added unified authorization, customers only need to apply for authorization through the unified console and download the corresponding type. The MediaBox audio and video terminal SDK is enough.
1) Unified authorization
2) Unified crash capture
Through the crash capture capability, we can capture the exceptions of our SDK (those not captured by our SDK will not be captured), and analyze and repair them as soon as possible to create a highly stable SDK.
In addition, we have also added online abnormality warning, configuration delivery and burying capabilities to troubleshoot and adapt to customer sudden abnormalities or online compatibility issues.
04 Architectural advantages
After optimizing the design of the audio and video terminal SDK architecture, we have implemented a MediaBox core, multiple business engines, unified media components and complete terminal-side basic capability construction. The main results are as follows:
• High reuse rate . All media components on the device side are reused, the code reuse rate is as high as 80%, and the overall SDK package size is reduced by 60%;
• High reliability . After several versions of iterations, the online stability of MediaBox SDK has been improved, and SDK crashes have dropped to one in 100,000;
• High flexibility . MediaBox SDK can meet customers' various business combination capabilities, such as the combination of live broadcast and player, the combination of live broadcast and real-time audio and video, to more flexibly meet customer needs;
• High performance . Through a unified microkernel and unified monitoring of performance indicators, memory and CPU usage have been reduced.
Based on this technical architecture, MediaBox combines various scattered SDKs into a full-featured unified package, and finally forms a large family of video cloud MediaBox SDKs products.
To learn more about audio and video terminal SDK product information, welcome to the official website to experience: Audio and video terminal SDK_Intelligent Media Service IMS_Video Service_Alibaba Cloud
Lei Jun announced the complete system architecture of Xiaomi's ThePaper OS, saying that the bottom layer has been completely restructured. Yuque announced the cause of the failure and repair process on October 23. Microsoft CEO Nadella: Abandoning Windows Phone and mobile business was a wrong decision. Both Java 11 and Java 17 usage rates exceeded Java 8 Hugging Face was restricted from accessing. Yuque network outage lasted for about 10 hours and has now returned to normal. The National Data Administration officially unveiled Oracle. Launched Java development extension for Visual Studio Code. Musk: Donate 1 billion if Wikipedia is renamed "Weiji Encyclopedia" USDMySQL 8.2.0 GA