For more technical exchanges and job opportunities, please follow the WeChat official account of ByteDance Data Platform and reply [1] to enter the official communication group
Recently,
China's well-known
content social platform integrated marketing company Qianchuan Network has reached a cooperation with the volcano engine
ByteHouse
. Qianchuan Network will upgrade the real-time query and analysis capabilities of the underlying data engine through the volcano engine ByteHouse to achieve high throughput, low latency, and high stability while reducing resource costs, and provide brands with a full range of public opinion monitoring services.
Qianchuan Network is an AI-driven social media big data solution provider
that aims to use unique big data and independent research and development system empowerment to provide enterprise-level customers with more valuable KOL and public/private domain traffic management solutions. , and provides comprehensive digital marketing and management services such as social media listening, industry insights, SCRM system management, etc., to build a three-dimensional content ecology covering communication, placement, monitoring, evaluation, etc. for advertisers to create more valuable marketing services.
Today, with the rapid spread of digitalization, public opinion monitoring has become an essential tool for every organization and brand. Effective public opinion monitoring can help brands quickly respond to potential negative news and reduce the risk of reputational damage in an era of rapidly changing information. It can also allow companies to adjust marketing strategies based on public opinion data, improve consumer satisfaction, and seize industry opportunities. Qianchuan Network's mobile Internet advertising monitoring platform, social media data analysis platform, Xiaoying short video public opinion monitoring platform and other products provide brands with cutting-edge, professional and scientific strategy and project service support through fast, comprehensive and accurate advantages. .
Public opinion monitoring has high requirements for real-time data. Among the existing technical solutions, Qianchuan Network provides public opinion similarity retrieval capabilities through self-built
Elasticsearch
. However, from the perspective of performance and cost, it cannot fully meet the needs of Qianchuan Network. In terms of performance, the business requires the system to support real-time retrieval of tens of thousands of QPS, and it is difficult to achieve such high concurrency indicators using the ElasticSearch solution. In terms of cost, using the ElasticSearch solution is accompanied by high machine resource costs. As the amount of data continues to increase, machine resource costs will become increasingly uncontrollable. In addition, the data writing and query processes are relatively complex, resulting in high application development costs.
In this context, Qianchuan Network further seeks solutions to reduce costs and increase efficiency, and sets its sights on the volcano engine
ByteHouse
. As a
cloud-native
data warehouse, Volcano Engine ByteHouse provides extremely fast data analysis services, which can support real-time data analysis and massive data offline analysis. It has been tested by ByteDance internally and has been widely used in external Internet, games, finance, automobiles and other fields. produce actual results.
The core problem faced by both parties in smoothly migrating from
Elasticsearch to
ByteHouse is that thousands of online public opinion data reach billions, and the peak QPS reaches tens of thousands. ByteHouse must not only have the ability to handle complex OLAP analysis scenarios, but also support Qianchuan network's ultra-high QPS online query service, and achieve high throughput, low latency, and high stability.
In order to better solve the problem of high concurrency enumeration in Qianchuan network,
ByteHouse
adopts a multi-copy mechanism to write all data to the master node, automatically synchronize to multiple copies in real time, and achieve automatic
load balancing
and one-write-multiple-read through gateway query. This linearly improves concurrency capabilities.
ByteHouse
high concurrency enumeration technology solution
In addition,
ByteHouse
also supports completing business logic through SQL writing and querying, further simplifying the data processing process and improving development efficiency. In terms of resource efficiency, ByteHouse adopts a flexible resource model that can support any change in the number of nodes and replicas. Users only need to purchase minimal resources at the beginning, and can be expanded immediately as business grows, making costs more controllable.
Tang Genping, head of big data at Qianchuan Network, said: "
ByteHouse
is very stable in querying and writing under billions of data. It not only helped us improve performance, but also further reduced costs. Currently, the total resource cost has been saved by 60%. %."
At present, videos, pictures, etc. have become the mainstream content carriers on the Internet. Qianchuan's Xiaoying short video public opinion monitoring product is
based on data authorization from the short video platform
, combined with large language model (
LLM
), natural language analysis (NLP) and other AI capabilities. It can help
governments and enterprises
quickly, comprehensively and accurately grasp the dynamics of public opinion in short videos.
ByteHouse
’s soon-to-be-released
vector search
and full-text search functions will also further enhance Xiaoying’s capabilities in short video public opinion monitoring business.
Click to jump to
Volcano Engine ByteHouse
to learn more
{{o.name}}
{{m.name}}