Efficient data processing of 500K objects per second using the KeyOnly feature

In some real-time applications, such as real-time product sales, live video bullet screen messages, etc. do not want users to input too long content, and do not want the same user to publish the same content at the same time to refresh the screen, generally there is only a very short message, for A value data of more than ten bytes, if each entry is inserted using Key-Value, it is a bit wasteful to read Key-Value. There is a new feature KeyOnly in iBoxDB, which can directly push the Value into the Key, and only need one Key to get all the needed. The following describes how to use this KeyOnly feature in the iBoxDB database.

To use the KeyValue feature in iBoxDB, you only need to add a '/' symbol in front of the table name, and the engine will automatically process it as KeyOnly. Other related operations are basically the same as ordinary tables. The only difference is that non-Key content will not be saved, because There is no Value value, so the Update() function always returns False when it is KeyOnly. You can use Delete first and then Insert to simulate the update, that is, the KeyValue operation is similar to a dictionary, and the KeyOnly operation is similar to a collection.

Because the database engine directly supports ORM, KeyOnly can also be used as an object, and the Key is defined below

public static class ShortMSG {
  public long productId;
  public java.util.Date time;
  public long userId;
  public String msg;
}

productId is the product id, time is the publication time, userId is the user id, and msg is the short message. This is a pure Java class.

 

Then define a table associated with this class in the database. The table name starts with a '/' character. If it is defined as a KeyOnly table, fields that are not defined as Key will not be saved. 

db.getConfig()
  .ensureTable(ShortMSG.class, "/shortmsg", "productId", "time", "userId", "msg(30)");

 "msg(30)" means that the maximum length of the string is 30 characters. Generally, the length is detected in the application, reducing the processing of the database.

if ( mymsg.length() <= 30 ){ shortmsg.msg = mymsg; }else{ ... }

 
Other operations are the same as for normal data tables. In this test example, 100 products are used, 100 time points are used, and 100 users send messages, which is 1,000,000 (one million) messages.
data insertion code

try (Box box = auto.cube()) {
    for (long userid = 1; userid <= count; userid++) {
        ShortMSG smsg = new ShortMSG();
        smsg.productId = fproId;
        smsg.time = new Date(fakeTime + fft);
        smsg.userId = userid;
        smsg.msg = smsg.productId + "-" + (fakeTime + fft) + "-" + smsg.userId;
        box.d("/shortmsg").insert(smsg);
    }
    if (box.commit().equals(CommitResult.OK)) {
        total.addAndGet(count);
    }
}

 Data reading code:

try (Box box = auto.cube()) {
    for (ShortMSG smsg : box.select(ShortMSG.class, "from /shortmsg where productId==?", fproId)) {
        if (smsg.productId != fproId) {
            System.out.println("Unreachable");
        }
        String cs = smsg.productId + "-" + (smsg.time.getTime()) + "-" + smsg.userId;
        if (!smsg.msg.equals(cs)) {
            System.out.println("Unreachable");
        }
        total.incrementAndGet();
    }
}

 Test Results:

Insert TotalObjects: 1,000,000 FileSize: 158MB
Elapsed 10.314s, AVG 96,955 o/sec

Select TotalObjects: 2,000,000
Elapsed 2.493s, AVG 802,246 o/sec

The results show that using the KeyOnly feature, it only takes about 160MB of hard disk space to process 1,000,000 (one million) short messages, and only 2.5 seconds to read 2,000,000 (two million) objects, and the average reading exceeds 500K data objects per second. .

More information:
full code address  https://github.com/iboxdb/forjava/tree/master/demos
iBoxDB database address http://www.iboxdb.com/

If you find more than 1000K objects per second in your tests, which is normal, the virtual machine will optimize the code on the fly. The test result is obtained under Java8, 1G memory, and supports Java7. In the test under Java6, you only need to change try( box ){ } to try{ } finally { box.close(); }