Databend Open Source Weekly Issue 136

[Live broadcast preview] Will large models replace programmers? "

Databend is a modern cloud data warehouse. Designed for flexibility and efficiency to support your large-scale analysis needs. Free and open source. Experience the cloud service now: https://app.databend.cn .

What's On In Databend

Explore Databend's new developments this week and encounter Databend that is closer to your heart.

Understand tasks and notifications in Databend

Databend now supports a complete task and notification mechanism.

The task executes the specified SQL statement according to the plan or DAG-based task graph. Notification integration lets you send notifications to external messaging services.

CREATE TASK IF NOT EXISTS mytask
 WAREHOUSE = 'mywh'
 SCHEDULE = 30 SECOND
 ERROR_INTEGRATION = 'myerror'
AS
BEGIN
    BEGIN;
    INSERT INTO mytable(ts) VALUES(CURRENT_TIMESTAMP);
    DELETE FROM mytable WHERE ts < DATEADD(MINUTE, -5, CURRENT_TIMESTAMP());
    COMMIT;
END;

The above example defines a mytasktask named . It will mywhrun every 30 seconds on the computing cluster. The task executes a multi-statement transaction containing INSERT statements and DELETE statements. When a task fails, it triggers myerroran error integration called .

Task and notification related mechanisms are available out of the box in Databend Cloud. If you want to know more information, you are welcome to contact the Databend team , or check out the resources listed below.

Code Corner

Let’s explore code snippets or projects in Databend and the surrounding ecosystem.

Databend vs. Snowflake: Data Ingestion Benchmark

Recently, Databend and Snowflake conducted a data ingestion benchmark test, covering aspects such as performance and cost, which mainly included the following four basic tests:

TPC-H SF100 data set loading, oriented to large-scale data, about 600 million rows.
ClickBench Hits data set loading, oriented to large wide table data, about 100 million rows.
1 second freshness: Compare the data ingestion capabilities of the platform under the freshness limit of 1 second.
5-second freshness: Compare the data ingestion capabilities of the platform under the 5-second freshness limit.

Dataset loading comparison

Data freshness comparison

Interested parties are welcome to read the following document to learn about Databend Cloud's low-cost, high-performance data ingestion.

Docs | Databend vs. Snowflake: Data Ingestion Benchmark

Highlights

Here are some noteworthy events, maybe you can find something of interest.

CROSS JOIN supports overflow.
Aggregate hash tables support overflow.
Support refreshing inverted index.
Add more function aliases for time and date related functions to support more data analysis tools.

What's Up Next

We are always open to cutting-edge technologies and innovative ideas, and welcome you to join the community and inject vitality into Databend.

Support TOP K syntax

The Databend plan supports SELECT TOPthe statement, which is used to select the top K items of the result set.

The following SELECT TOPstatement is used to select the first 4 items of the ordered result set:

select TOP 4 c1 from testable ORDER BY c1;

Equivalent to the following SELECT ... LIMITstatement.

select c1 from testable order by c1 limit 4;

This is a good first issue (novice-friendly issue), designed to guide students who are interested in Rust and Databend to participate.

Issue #14972 | Feature: top k syntax support

If you are interested in this topic, you can try to solve some of the problems or participate in discussions and PR reviews. Alternatively, you can click on https://link.databend.rs/im-feeling-lucky to pick a random question, good luck!

New Contributors

Meet new people in the community. Databend is a better place because of you.

@suimenno3002 implemented support for histogram aggregation functions, #14839 .

Changelog

Check out the changelog for Databend's daily builds to stay up to date on the latest developments.

Address: https://github.com/datafuselabs/databend/releases

Contributors

A big thank you to the contributors for their great work this week.

Connect With Us

Databend is an open source, flexible, low-cost, new data warehouse based on object storage that can also perform real-time analysis. We look forward to your attention and exploring cloud native data warehouse solutions together to create a new generation of open source Data Cloud.