DolphinDB x Hengtai Securities | Integrated trading platform: break the shackles of programming and liberate business thinking

"Before using DolphinDB, our team's investment and research languages involved Python, C++, R, Matlab, etc. The investment and research results of multiple languages are usually difficult to unify, and high-frequency investment and research personnel often need to spend extra energy to improve code performance." When the team's structure evolved, Wang Meng mentioned some problems that were eager to be solved at that time.

At present, the quantitative team of Hengtai Securities Advisory Service Department uses a set of scientific quantitative IT architecture that has been explored by itself. The threshold of engineering technology breaks through the talent bottleneck of the team.

This time we had the honor to invite Mr. Wang Meng Wang, who is in charge of quantitative high-frequency business of Hengtai Securities, to share. His experience and insights will reveal a new path for us and help us take an important step in how to better utilize physical resources, improve investment and research efficiency, and lower the threshold for talents.

This article is compiled from an interview with Wang Meng from Hengtai Securities Advisory Service Department.

Architecture, Performance, and Cost: A Common Dilemma for Quantifying Teams

Python has a rich ecological environment and easy-to-use technical features, and has always been a popular engineering choice for quantification. However, in the research of high-frequency strategies that need to process large-scale data and perform complex concurrent calculations and I/O, Python's performance may become a limitation.

At the same time, the execution efficiency of Python cannot meet the delay-sensitive trading scenarios, so the quantification team usually introduces C++ to make up for this problem.

However, there are also some disadvantages in adopting the technical architecture of Python plus C++.

First of all, two different programming languages bring more complex development and maintenance work.

The team needs to manage different tool chains, libraries, and dependencies at the same time, and cross-language interface development and debugging may also increase the team's workload and risks. For example, in the process of entering the production environment from investment research, the conversion of two sets of codes and data matching will become problems.

"If you use Python to do investment research, and then connect it to C++, there will often be a difference in the calculation of data points of about 1% to 5/1000."

Second, this hybrid programming environment means higher hiring and training costs.

To achieve business goals, team members need to be proficient in multiple programming skills at a high level. For example, compile SO files using C++, and then call or glue through Python.

On the other hand, if you want to further increase the resource utilization rate of Python, researchers need to be proficient in complex writing methods such as threads, processes, and coroutines, and develop the code habit of parallel processing of calculation, reading and writing. This puts higher demands on researchers' code engineering capabilities.

That is to say, in the face of complex engineering requirements, we not only want to improve the efficiency of the entire investment research transaction process, but also want to ensure data consistency as much as possible, which means that both researchers and IT personnel must have profound technical skills. Especially for researchers, if they want to give full play to their financial and mathematical thinking, they must first cross the threshold of code writing.

The quantitative team of Hengtai Securities is deeply involved in the large cross-section strategy represented by Alpha and the large time series strategy represented by T+0 . If they want to make continuous breakthroughs in the market, they must continue to pursue better strategic mathematical performance and higher engineering calculations. performance and lower transaction latency. And for these, the engineering plan is the basis, and talent competition is the key.

The key to solving the problem: focus on business logic and reduce talent costs

The current market style is switching rapidly and effective information is becoming more and more high-dimensional. Without an agile and iterative IT architecture and an engineering solution that is easy to collaborate and integrate, it will be difficult to develop in the high-frequency field. And if you want to go further, you need the support of better talents.

Is there a solution that can solve these problems at the same time without too much talent recruitment and training costs?

Since using DolphinDB in 2020, Hengtai Quantitative Team has explored and tried, and summed up a set of quantitative engineering architecture suitable for itself, which not only improves the computing performance of investment research, but also scientifically and reasonably solves task coordination, resource sharing and talent reduction. This question.

"It integrates programming language and database reading and writing, and helps us connect IaaS, DaaS, and PaaS in a stream-batch manner. It has high performance, is easy to use, and is irreplaceable."

This is Wang Meng's evaluation after using DolphinDB. In the interview, Wang Meng shared the team's practical experience in achieving business efficiency and talent cost reduction. Next, I will introduce this set of integrated delivery platform solutions based on DolphinDB.

Scientific quantitative IT architecture

A complete investment research architecture first needs to aggregate IT basic resources such as network security and streaming data components to form an infrastructure layer (IaaS). For high-frequency quantitative services, high-performance storage and query are required to support the data service layer (DaaS). At the same time, the factor side, the model side and the evaluation side need to form a platform service layer (PaaS) that supports the investment research backtesting business, to provide functions such as factor calculation, model and strategy evaluation, and to encapsulate IT components such as various functions and model plug-ins .

The IT design structure of Hengtai Quantitative Team is shown in the following figure:

Among them, the data service layer (DaaS) is used to connect and store the whole market, including various data such as tick by tick, snapshot, minute line and fundamentals. For databases, effective compression and high throughput support can improve storage performance, and millisecond-level response can ensure query speed. At the same time, a set of appropriate programming statements is an important prerequisite for efficient docking, convenient reading and writing, and analysis and calculation. In this layer, since DolphinDB naturally integrates the database with Python and SQL-like scripting languages, it is well adapted to the architectural requirements.

In addition, in this integrated high-performance computing and storage framework that integrates IaaS, DaaS, and PaaS, DolphinDB replaces Python or C++ to perform various computing tasks. For example, through the Python API, the application can execute scripts and functions on the DolphinDB server through the session (Session), and transfer data between the two. As a local scheduling language, Python can bond and serve the entire team while sending the DolphinDB code to the remote server, and the remote server does not retain the core computing code.

DolphinDB's complete technical function points perfectly match the needs of the quantitative team's required architecture. All processes are managed in the background, enabling parallel coordination within and between sessions, while researchers are decoupled from the underlying IT infrastructure. In addition, DolphinDB's stream-batch integrated computing can seamlessly migrate investment research codes to production transactions; its plug-in management is also convenient for production transactions to use C++ integration

This solution simplifies the technology stack and reduces the cost of talents. It not only allows members to focus on business, but also ensures the injection of fresh blood.

"The integrated investment platform can maximize parallel tasks, and researchers are indifferent to this process, which is a more scientific investment research architecture design."

Break through the performance bottleneck, high-frequency quantization goes a step further

After building such a platform, the development efficiency of the team has been significantly improved. Among them, the most important thing is to use DolphinDB instead of Python as the investment research tool, breaking through the performance bottleneck of high-frequency factor calculation. According to statistics, the calculation speed of single factor multi-parameter is more than 50-100 times faster than that of Python .

But it's not just a simple increase in computing speed, either. For researchers, when calculating complex factors, they need to call DolphinDB's built-in 1500+ functions, which are specifically optimized for financial scenarios, thus saving a lot of time and effort in implementing algorithms or engineering logic.

In addition, the team adopted the optimal storage scheme of "stock/factor + time" combination partition, which can quickly realize the storage of market data and massive factor data in a modular and functional form in DolphinDB .

When the investment research is connected to the production environment, since the bottom layer of DolphinDB is compiled with C++, it is naturally adapted to most trading platforms, and the investment research results can be matched with the real offer faster and more conveniently in the form of plug-ins and built-in functions. On the premise of improving the calculation and execution speed, the team reduced the data difference rate to less than one ten-thousandth.

Maximize resource utilization and reduce investment research costs

While continuously optimizing the profitability of the strategy, Hengtai Quantitative Team realized the parallel invocation of memory and disk read and write tasks, maximizing the utilization of server resources. Compared with Python's native parallel capabilities,

"DolphinDB can use both CPU and disk IO to reach 100% through multi-threading."

When the team uses the native distributed DolphinDB, it can naturally and transparently maximize resource utilization. In addition, through the embedded distributed file system, members can easily manage partition data and copies, and realize load balancing and high fault tolerance under the distributed computing framework.

catastrophe!

"The most important thing for a quantitative team to maintain its ability to break through the market is to constantly absorb new blood and maintain the team's innovative vitality."

With the update and iteration of cutting-edge research such as mathematical algorithms and computer technology, the requirements for talents in the field of quantification are also increasing. They must not only understand mathematics, but also be able to write code. In the field of quantification, they must be able to write C++ well. Integrate innovative algorithms in the field into the financial system to pursue better returns... Only lower learning costs can lower the threshold for quantitative talents who originally need to be proficient in Python and C++.

The Quantitative Team of Hengtai Securities has been using DolphinDB for nearly 3 years. Talking about the hands-on experience, Wang Meng said that because the DolphinDB script supports standard SQL compatibility and uses interpretive language programming, it is very easy to get started-as long as a researcher with a certain code foundation, "One week You can learn from left to right."

In the past year and a half, the team has developed tens of thousands of T0 factors and Alpha factors through DolphinDB, and within the unified strategy system, an average of 2 investment research factors are included or replaced by production factors every day.

How can quantitative institutions establish their core competitive advantages in a rapidly developing industry? Perhaps from the use cases of Hengtai Securities’ quantitative team, we can get a glimpse of one or two—improving the integration efficiency of investment, research and trading, and controlling the cost of talents. What quantitative institutions really need is a set of solutions that can help researchers break the shackles of code writing, give full play to their advantages in mathematics and financial thinking, and devote more energy to business logic .