Alibaba Cloud retrieval analysis service Elasticsearch version cloud evolution path
In 2017, Alibaba Cloud and Elastic launched an open source strategic cooperation and officially released the Elasticsearch version of Alibaba Cloud's search and analysis service. It is 100% compatible with the open source Elasticsearch and can be used out of the box, providing an open and compatible cloud search and analysis service.
In 2019, Alibaba Cloud fully hosted the Elastic Stack on the cloud. It was the first product service provider in China to fully host the entire ecological components on the cloud, providing end-to-end search and analysis solutions to help open source users quickly migrate to the cloud and scale. Supports 10,000 core cloud users.
In 2021, a version based on the Elasticsearch kernel engine optimization was launched to the market. The Alibaba kernel is deeply optimized, and the self-developed read-write separation, storage and calculation separation architecture helps enterprises reduce costs and increase efficiency, and continues to promote the kernel improvements behind the version to the Elasticsearch community. .
In 2023, Alibaba Cloud's Elasticsearch Serverless service will be officially launched. The new product form is based on cloud-native Serverless technology and is committed to creating a lower-cost, flexible, open and compatible, and out-of-the-box Elasticsearch experience for users on the cloud.
Business challenges facing open source Elasticsearch
Since 2017, we have served tens of thousands of enterprise-level customers, operated and maintained hundreds of thousands of clusters, and summarized the common pain points faced by open source and self-built users.
- With constant promotions, game releases, swarming requests, and sudden read and write traffic, the cluster encounters resource bottlenecks, making it difficult to expand capacity in a short period of time.
- A large amount of resources are redundant during low business peak periods, and because traffic suddenly increases at a certain moment, capacity planning has to be done based on peak traffic.
- The cost of server resources is high, and the budget for R&D, testing, and production is low. It must be stable while reducing costs and increasing efficiency.
- The project team does not have exclusive operation and maintenance support. Cluster stability operation and maintenance is difficult and requires continuous optimization with business development. Development also requires operation and maintenance experience.
Alibaba Cloud retrieval and analysis service Elasticsearch Serverless version officially released
Based on the above challenges, we launched the Alibaba Cloud Elasticsearch Serverless service. Alibaba Cloud Elasticsearch Serverless is a cloud-native serverless service-oriented product built around Elasticsearch. It achieves dynamic matching of real business loads and resources. Users do not need to manage clusters and resource configurations. They only need to pay on demand based on actual resource usage; it provides elastic scaling capabilities. It can also meet business needs during peak periods; users of the original ES ecosystem can smoothly switch and use the Serverless version, helping businesses quickly migrate to the cloud.
Interpretation of core capabilities of Elasticsearch Serverless service
Second-level elastic scaling
When building open source Elasticsearch on IDC or ECS, you need to plan the cluster scale in advance, including the number of data nodes, coordination nodes, and master nodes, as well as the data storage disks that need to be configured for each node. In addition, during production use, the cluster size also needs to be adjusted based on fluctuations and changes in business traffic, and you will worry about the stability and security of the cluster during the change process.
Alibaba Cloud's Elasticsearch Serverless service uses virtual multi-tenancy capabilities to split a very large physical cluster into N tenants and provide it to users. Users actually get logical-level units instead of holding a single physical cluster. When the traffic peak arrives and resource consumption is higher than the set threshold, more existing resources will be allocated to tenants. This does not involve expansion at the physical level and can flexibly and dynamically allocate the upper and lower limits of resources to customers to achieve true sense of elastic expansion. In addition, the expansion and contraction of logical level units is relatively smooth, and there will be no cluster instability, business jitter, or other negative impacts.
True pay-as-you-go
There are two parts of resource overhead that cannot be avoided during the use of Elasticsearch. One part is the computing resources brought by CPU and Memory; the other part is the storage resources that come with the Store. Alibaba Cloud's Elasticsearch Serverless service breaks the traditional payment model based on fixed specifications and uses CU as computing resource billing. The performance of 1 CU is approximately equal to 1 Core CPU (maximum) 2 GB Mem, and is billed hourly based on actual usage, realizing a true sense of pay-as-you-go.
For instantaneous computing resources such as CPU and Mem: Pay by the hour, "traffic fluctuations are accurate to every hour", and the average computing resource within an hour period will be calculated for billing. You no longer have to worry about glitches, let alone prevention. Water level redundancy due to glitches. For continuous occupying resources such as storage: similar means are used to measure "storage fluctuations to an hourly accuracy" and calculate the average value of storage resources within an hour for billing. Although storage is continuously accumulated, in this way, you can use the storage of Elasticsearch on the cloud more flexibly, without worrying about buying too much or not having enough.
For example, a user has a website building information retrieval business scenario. There is traffic during the day but no traffic at night.
- Corresponding to the business traffic, you need to build a self-built ES 2-core 4GiB single-node instance to complete the task. The self-built one-day cost = 9.6 yuan (2-core 4 GiB specification: 0.407698/hour).
- The total cost of using the Elasticsearch Serverless service for a day = computing resource cost + storage cost = 1.2 × 8 × 0.3975 + 0.2 × 16 × 0.3975 +1 × 0.0021 × 24 = 5.1384 yuan
Using Elasticsearch Serverless service costs can save up to 50%, and this is only a scenario of only 2CU during peak periods. In a real production environment, when the concurrent QPS reaches 1000 and the required RT is 100 milliseconds, how many redundant resources are needed? How many resources can be saved by using Alibaba Cloud Elasticsearch Serverless service?
Simple operation and maintenance-free
When using Elasticsearch, you may face some complex and unavoidable operation and maintenance work. The resource preparation stage requires capacity planning and configuration based on a large amount of actual production experience; the R&D and deployment stage has various environments, and thousands of parameters need to be configured; the operation and maintenance stage also needs to configure a large disk for monitoring and alarming, and when encountering traffic The disk needs to be monitored during peak times, and cluster recovery is required when cluster stability fluctuates.
Alibaba Cloud's Elasticsearch Serverless service has no concept of physical clusters and does not require any form or type of cluster planning. Based on the intelligent operation and maintenance platform, the engine that separates reading and writing, and separating storage and computing, and based on Alibaba Cloud's best practice applications, more than 95% of the operation and maintenance costs are shielded. Users do not need to be aware of physical resources and can quickly create it in 1 minute. It can be used out of the box without human intervention in resource level and configuration changes, which greatly improves operation and maintenance efficiency.
Open and compatible
Alibaba Cloud's Elasticsearch Serverless service supports 200+ APIs for common Elasticsearch scenarios, provides Kibana, and is compatible with various open source components. It retains original usage habits, achieves smooth migration, and helps businesses quickly migrate to the cloud.
Elasticsearch Serverless service technology architecture
The picture above is the technical architecture diagram of the Elasticsearch Serverless service. The bottom layer is the storage layer. What is written here is distributed storage, which is a storage service provided by a multi-storage media set specifically for search. Behind it are EBS fast storage, local disks, high-performance robust storage and object storage. There is also a data scheduling strategy behind this multi-level set connection. What kind of data with high query density will be placed on the local disk, and what kind of data with low query density will be placed in the object storage. There is a balance between cost and performance. It brings users the most cost-effective storage experience for search scenarios.
The next layer is the computing layer. We separate the cluster for writing data and the cluster for querying indexes in Elasticsearch. This allows resources to better and more accurately serve the business during the expansion and contraction process, and also achieves a true sense of volume-based services. Reasons for paid and elastic scaling.
The next layer up is the routing layer, which is a very core link of the Elasticsearch Serverless service and provides authentication, routing, rights control and other related capabilities.
The top layer is the application layer that combines open source APIs, open source components and customer business scenarios.
Based on such a four-layer architecture, the Elasticsearch Serverless service provides operation-free, elastic scalability, low cost, and open and compatible product capabilities.
The future evolution direction of Elasticsearch Serverless services
Continuously improve product capabilities
Providing higher elasticity, higher availability, and easier-to-use Elasticsearch cloud services:
- High elasticity
Increase the elasticity limit of production scenarios; decouple storage and computing; guarantee higher elasticity efficiency
- High availability
Self-developed engine feature support, providing enterprise-level stability guarantee; security reinforcement, strengthening data security protection
- Ease of use
Scenario-based product capabilities, providing multi-scenario best practices; providing more open source compatibility, and realizing Elastic Stack Serverless
Continuously provide solutions
Continue to provide services to cloud customers in multiple industries and scenarios:
We hope to embrace various vertical scenarios, focus on the Elasticsearch ecosystem, and use solution encapsulation to serve more scenario needs. Build more solutions on the cloud such as multi-modal search, observability, SIEM, database acceleration, ChatBot Q&A interaction, and more. At the same time, we will work with partners in various scenarios to create an application-level market for Chinese search. With the help of Alibaba Cloud's PaaS products and technology accumulation, as well as the accumulation of cloud customers and businesses, we will provide the Chinese market with better products, solutions, etc. service capabilities.
