Bifrost ---- Heterogeneous middleware that synchronizes MySQL to Redis, ClickHouse and other services for production environment
You can synchronize the data in your MySQL in full and in real time to:
- Redis
- MongoDB
- ClickHouse
- MySQL
- Memcahe
- RabbitMQ
- ActiveMQ
- Kafka
- Http custom service
- Hprose RPC custom service
v1.7.2-deving
=============================
=============================
1. MariaDB GTID supported
v1.7.1-release 2021-02-09
===========================
1. fix binlog parser ddl(rename,drop) error
2. fix binlog parser error by "/*notes*/" filter
3. plugin ck ddl update
4. plugin mysql ddl update ,supported tidb rename multi-table
5. mysql multi-gtid support
6. fix percona 8.0 char parser error
7. recovery binlog position calc update
8. beore and after ddl, callback a dummy begin and commit event
9. fix windows path error
10. db list show eventID
11. doc update
12. fix other bugs
v1.7.0-rc.01 2021-01-24
===========================
1. gtid supported
2. plugin ck ddl supported(add,change column)
3. plugin ck decimal64 tansfer decimal64, decimal128 and decimal256 tansfer to String
4. plugin mysql filter data auto_increment be first
5. fix other bugs
v1.6.6-release 2021-01-17
===========================
1. fix rename table binlog parser error bug
2. fix http basic auth bug
v1.6.5-release 2021-01-10
===========================
1. fixed mysql desc SchemaName error bug
2. fixed mysql ddl tansfer error bug
3. fixed history task may recovry bug
4. fixed history flow href error
5. mysql plugin error update
6. fixed may time out contine bug
7. other bug fixed
v1.6.4-release 2021-01-03
=============================
1. Fix a fatal problem caused by multi-thread synchronization in a full task setting bug
2. fixed TRANSACTION_CONTEXT_EVENT,VIEW_CHANGE_EVENT,XA_PREPARE_LOG_EVENT filter error
3. add mysql version show
4. add http api doc
5. web view update and other bugs fixed
v1.6.3-release 2020-12-27
=============================
1. Fix plugin mysql lose data when the primary key is not incremented 2. Fix the bug
of blocking caused by plugin ddl synchronization
3. Fix the bug of data loss caused by plugin synchronization error without reporting an error
4. Fix the default value of dateime(6) type is 0000-00-00 00:00:00.000000 parsing error 5.
Traffic statistics optimization reduces the operation of map, and interface traffic display is optimized . 6.
User login verification Host supports multiple Host configurations, separated by commas
.
Some known bug fixes and interface optimizations
v1.6.2-release 2020-12-20
=============================
1. Added Host authentication function for login users
2. Added Login log and automatic IP ban function
v1.6.1-release 2020-12-16
=============================
1. Added mysql time(1-6), Datetime(1-6), timestamp(1-6) support (binlog parsing was submitted and merged by @zhwei820)
2. Fix a bug that may cause data loss
3. The mysql plugin supports the function of asynchronous data, only supports SQL DDL
4. Fix the bug that binlog commit events are filtered
v1.6.x
===========================
1. Framework optimization and plug-in interface changes to facilitate developers to intervene in development
2. Fix the bug of clickhouse plugin {$bifrostDataVersion} tag is 0
3. Modify the target library address online, it will take effect online, no need to restart the process
4. In the target address library, the new MinConn parameter can be set to keep the minimum connection pool size
5. Other known bug fixes and interface display optimization
6. The site saves the record and uses the commit event site to solve the problem of automatically finding parseable sites after restarting when synchronizing large transaction data.
v1.5.x
===========================
1. Fix enum and other bugs that may parse errors
2. clickhouse plugin supports automatic creation of CK table structure synchronization
3. Added fuzzy matching table to support specifying filter part table not to perform fuzzy matching
4. Supports the specified arm architecture compilation method
5. Fix the invalid bug after modifying the data source connection address
6. Verify that the mailbox is legal and regular changes
7. The row event parsing is lazy to abandon the tableId, and the row parsing module is optimized
8. Fix the bug of synchronization interruption caused by full multi-thread synchronization concurrency
v1.4.x
===========================
1. Support the synchronization configuration of fuzzy matching horizontal split table, such as binlog_field_test_* matching binlog_field_test_1, binlog_field_test_2
v1.4.5-release 2020-09-26
===========================
1. Fix the bug that some data of decimal type is wrongly parsed
2. Fix the bug that the alarm configuration cannot be deleted
3. Fix the bug of creating data path error under Windows
4. Fix the bug that some status tables of all tasks have data but cannot query the data
5. When fixing the full task AllTables, the bug of null pointer error
6. Channel operation plus lock management to solve the bug caused by CPU cache
v1.4.4-release 2020-09-23
===========================
1. The same transaction map event and row event in binlog are split in different binlog files, and there is a bug that causes a null pointer parsing error
2. Code optimization, unified use of a lock, reduce logic complexity
3. Code optimization, parsing status code changes
4. bash syntax is compatible with ubuntu
5. Fix the bug that TableCount plugin interface cannot be used
v1.4.3-release 2020-09-12
===========================
1. Fix the bug that the data source site acquisition site may be inconsistent because it is not locked
2. The plugin clickhouse {$BifrostDataVersion} tag uses a counter to ensure that different data values in the same batch are different
3. When only non-fuzzy matching tables are synchronized and configured, the bug that the synchronization configuration is lost is fixed.
4. The synchronization configuration does not synchronize any data. When the location is 0, the data source location fails to recover. Bug fixed
5. Added prompt judgment for binlog_row_image parameter
v1.4.2-release 2020-09-05
===========================
1. New mysql binlog_checksum=none parsing error bug
2. Fix the bug of displaying wrong data in full task query data
3. The plugin clickhouse primary key supports the use of tags
4. Fix some logical problems of stmt, rows close successively
5. Add Timeout and RequiredAcks configuration parameters to plugin kafka
6. Fix the bug that the total number of non-optimized queries for full tasks does not match
v1.4.1-release 2020-08-31
===========================
1. Fix the bug that the stmt is not closed when the connection id is obtained
2. Fix the bug that binlog_field_test_* will match binlog_field_test, all_binlog_field_test, and automatically convert to ^binlog_field_test_(.*) will not match binlog_field_test
3. Fixed a bug that was no longer executed after the partial table was executed for all tasks
4. The full task list shows the number of tables being executed and the number of tables that have been pulled
5. In the process of site recovery, the first synchronization is synchronization exception, and the bug of recovery site error
6. Added pause and start functions for table synchronization
7. Change the push method of http plugin to json raw format
8. Added table search function in configuration interface
9. The linux environment compiles automatically yum installs golang without the go environment
10. Home page version judgment logic update
11. Department API interface changes
v1.4.0-release 2020-08-24
===========================
1. Support the synchronization configuration of fuzzy matching horizontal split table, such as binlog_field_test_* matching binlog_field_test_1, binlog_field_test_2
2. Added version update prompt and latest dynamic display
3. Added plugin_commit_timeout, plugin_sync_retry_time parameter configuration (may have an impact on synchronization alarm configuration)
4. Add bin file to the compilation directory
5. The configuration file path adds the function of taking the etc directory and the root directory of the Bifrost folder as the relative path function
6. Tables that are not configured for synchronization will no longer perform data recovery
v1.3.2-release 2020-08-07
===========================
1. Bug fix for http plugin synchronization failure and no retry
2. The primary key set by the ck plugin matches the wrong bug
3. Fix the bug that mysql and ck plugins may have data disorder due to pointers
v1.3.1-release 2020-07-26
===========================
1. Import recovery configuration, add user and alarm configuration recovery
2.ck append mode synchronization, traversal is modified to start writing from the first data
3. Randomly write data to mysql update (test module)
v1.3.0-release 2020-07-17
===========================
1. MySQL json type support
2. Added tags that support {$json["key1"][0]}
3. There are still problems with type parsing such as deciaml
v1.2.4-release 2020-07-17
===========================
1. Get full access to sql statement function lock changes
2. Fix the bug that the NUMERIC_PRECISION value is incorrectly obtained
2. Fix the bug that binlog parsing cannot be parsed in mysql8 version because the obtained value returns from uint64 to uint32
v1.2.3-release 2020-07-07
===========================
1. Fix bugs caused by similar changes returned by mysql8.x in the information_schema table
v1.2.2-release 2020-06-10
===========================
1. Fix the bug that mysql, ck lose a piece of data when the batch is full
v1.2.1-release 2020-05-24
===========================
1. Invalid bug fix for starting tls configuration in non-Bifrost current directory
2. Bug fixes for errors in obtaining table structure information for all tasks
v1.2.1-rc.02 2020-05-18
===========================
1. Multiple synchronization threads for full tasks differentiate and optimize plug-in instantiation data
2. The json type field no longer reports an error in the map event event, but it does not support synchronizing tables with json fields
3. Optimizations such as judging whether synchronization is complete for all tasks
4. Added interface operation guidance function
5. Fix the problem that the file queue read out the set type data conversion failure
6. Repair the site file may repair the wrong problem
7. Clickhouse client driver package version update
8. Provide an interface to force gc
9. Fix the bug that the mysql connection is not closed for full pull data
10. Fix the bug that the full task of the same table is overwritten
11. Partial memory variables of the full task are automatically cleared at the end
12. In plugin mysql, the fatal bug caused by multiple synchronization threads sharing stmt is fixed
13. plugin mysql When the target table is an auto-incrementing primary key and the field name of the source table is not filled in, the current field is filtered and not processed.
14. The mysql, clickhouse plugin synchronously fixes the bug that the source is bool value and the target table is non-bool value data conversion failure
15. Call the bifrost server interface and display the create table sql
v1.2.1-rc.01 2020-05-13
===========================
1. Full support for configuring the number of pull data connections and synchronization threads, and query optimization
2. Added the function of configuring and enabling file queues
3. Fix file queue ack error bug
4. Fix the bug that the clickhouse plugin cannot submit
v1.2.0-rc.01 2020-05-10
===========================
1. The mysql plugin adds normal mode, log mode-modify, log mode-append, three modes
2. The clickhouse plugin adds a synchronous mode that converts delete to update
3. Clickhouse adds the {$BifrostDataVersion} tag, the fields of this tag will be used when deleting to ensure data security
4. Added file queue function. When a synchronization is blocked, the file queue is started to prevent the possibility of blocking other synchronizations.
5. Paging query optimization for full tasks, self-incrementing primary key query optimization into BETWEEN x AND y
6. When adding a data source, verify the permission to determine whether there is enough permission to parse the binlog
7. The MySQL plugin does not bind the fields of the source table and labels, and the default value is Null
8. ClickHouse plug-in, the fields that are not bound to the source table fields and labels, are converted to the corresponding type default values
9. Binlog parsing adds new error content, and unnecessary table structure query will no longer be performed if synchronization tables are not required.
v1.1.x
==========================
v1.1.x is a major version of Cheng Cup,
1. All plug-in functions, countless new functions have been added
2. All sync configurations are run in parallel
3. Optimized a large number of binlog parsing errors
4. The interface display also provides a lot of updates
5. There are still problems with type parsing such as deciaml
v1.1.1-release 2020-04-18
===========================
1. Clickhouse plugin delete merges a sql execution
2. Full task query sql query paging statement is optimized by sub query performance
3. Get the correct site at startup and add some logs
4. Fix the bug that the mysql plugin reports a statement error when there are special symbols in the database
5. Fix the bug that mysql write field name is reserved field error
6. The char storage byte is greater than 255 parsing error bug
7. Mysql plugin adds NullTransferDefault configuration, which can force NULL to be converted to the corresponding default value
8. Xinfu crashes due to abnormal mysql connection
9. Fix the error of Incorrect arguments to mysqld_stmt_execute in case of multiple null values
v1.1.0-release 2019-12-08
===========================
1. The redis configuration store removes a duplicate bifrost string
2. Toserver_queue_size is modified to 5000 by default
v1.1.0-rc.01 2019-12-02
===========================
1. clickhouse plugin package update
2. The configuration saving failed, causing the configuration to be lost to fix the bug
3. Added a configuration to save the configuration every 5 minutes
4. The plugin adds a method setPluginParamDefault to specify default parameters
5. Traffic display error bug fix
6. pprof needs to be logged in to access
v1.1.0-beta.21 2019-11-03
===========================
1. ClickHouse plug-in type conversion between int types, forced to 0 after memory overflow
2. Added the TableCount plugin, which is used to count the insert, update, delete, and ddl of each table, which can be displayed in a line graph
3. Added the *.* method to filter matching data tables. The interface AllDatabases represents all databases, and AllTables represents all tables.
4. The redis plug-in, if how many ips are configured, the cluster connection is used, and the multiple ip ports are separated by commas
5. If batch add-ins are not allowed, if you click the batch add button or select AllTables, you will be prompted that batch configuration is not supported
v1.1.0-beta.20 2019-10-27
===========================
1. The Bristol mysql package is updated, and the return value is directly converted to the specified type instead of the []byte type.
2. After binlog is parsed, add the Pri (primary key) field name
3. Clickhouse plugin, new decimal type (but there will be loss of precision), it is not recommended to use
4. The clickhouse plugin fixes the bug of Nullable type data conversion failure
5. Added search by state to the table synchronization list
6. Compile script update
7. Documentation update
8. Convert mysql connection to utf8 encoding
9. Added support for redis storage
v1.1.0-beta.19 2019-10-19
===========================
1. Fixed a bug in the display of site delay calculation site display error
2. Added configuration export and import functions (data source, target library, data synchronization configuration, alarm configuration, user, etc.)
3. Added user management module, the user name and password in the ini configuration file are used for user initialization at the first startup
4. Added custom where query condition function for full data configuration
5. No traffic display bug fix when brushing full data
6. Other bug fixes
v1.1.0-beta.18 2019-10-16
===========================
1. Added the ability to view the latest site comparison and display the delay time
2. The mysql plugin adds tag support and interface optimization
3. Other known bug fixes
v1.1.0-beta.17 2019-10-10
===========================
1. The traffic statistics are based on the actual number of rows, and the query_event event is not within the statistical scope
2. Clickhouse plugin, in the case of string to Int32, Float and other types, first remove the spaces before and after before converting
3. Added search function by status to all tasks
4. Support make install prefix=./target compilation
5. Compile script and docker.sh packaging script update
6. Other known bug fixes
v1.1.0-beta.16 2019-09-30
===========================
1. Modification of the parsing library module to support fine-to-table filtering
2. The parsing library timestamp conversion requires mysql and the current machine to be in the same time zone
3. Analysis library test module and performance module modification
4. If binlog parsing is abnormal, it will not retry, but exit the current parsing directly.
5. Output the address of http or https access when starting
6. When the data is synchronized from the table to the target database, if there is no data for 5 consecutive seconds, it will automatically exit the guardian module, release the memory, and wake up again when there is data next time
7. Added QueueMsgCount to the table synchronization list to accumulate the number of messages to be synchronized
8. Data source monitoring alarm optimization
9. When modifying the site, add whether to force the modification of all table ToServer sites
10. Fix the bug that the alarm configuration does not automatically recover after restarting
11. When restarting, the data source startup is modified to be asynchronous
12. Obtain the normal site, subject to the QUERY_EVENT event
13. 新增{$Timestamp},{$BinlogTimestamp},{$EventType},{$BinlogFileNum},{$BinlogPosition}标签
15. Compile script update
16. Fix the bug that Miss button misses the error data is invalid
17. The startup script is not compatible with Ubuntu bug fix (@Beijing-Xinglong)
18. Replace traffic display with echartJs
19. Function optimization such as interface
20. Other known bug fixes
21. Clickhouse plug-in, add all to insert synchronization
22. Automatically generate mysql to ck sql, convert decimal type to String
23. ck plugin auto-fill field function optimization
24. Other plug-ins close and other functional bug fixes
v1.1.0-beta.15 2019-09-24
===========================
1. If the table list is a view, it will prompt the user so that the user can perceive
2. On the data table configuration synchronization details page, the first non-MySQL library is selected by default
3. channel prevents repeated startup
4. Added the function of modifying the data source location, only the close state can be modified
6. In the case of the data source binlog_format!=row format, prompt the user
7. After the data source is added successfully, the user will be prompted whether they need to jump to the setting table synchronization interface.
8. Change the Detail button to Setting
9. No data table synchronization is configured, users are not allowed to open data synchronization
10. Start, close, stop add lock mechanism to prevent repeated submission
11. The default default channel, when adding a data source, starts automatically
v1.1.0-beta.14 2019-09-19
===========================
1. The redis plugin, the redis client package is changed to go-redis, and the redis 3.x version is supported
3. Other known bug fixes
4. Parse binlog first filter the library and then parse the related row events
5. Every time a new library is added, it is updated to the Britol parsing library at the same time.
6. When synchronizing to clickhouse and other types to int and float types, the data conversion fails, no error is reported, and the strong conversion becomes 0, 0.00 for synchronization
7. Fix the bug that the table synchronization list displays the wrong bug when the library name contains the special symbol - . Now _- is used as the separator, and the library name and table name are not allowed to have 2 consecutive characters of _-
8. Fix the bug of UFloat when the clickhouse interface generates sql
9. Fix other known bugs
v1.1.0-beta.13 2019-08-02
===========================
1. ck plug-in type conversion optimization, support conversion between different ints
2. Add plug-in development, interface debugging sdk
v1.1.0-beta.12 2019-07-26
===========================
1. Randomly generate data to facilitate plug-in development and testing, including providing plug-in performance testing methods and data verification
2. The unit testing of ck, mysql, mongodb, http, rpc, redis, rabbitmq and other plug-ins has been improved and bugs have been fixed. Data verification passed the test
3. mysql plugin, if the source table is null, if the target table has a default value, it will be filled with the default value
4. In the http plugin, the key:data of the passed value is changed to Data
5. Fix the bug that an error is reported after the rabbitmq Expir parameter is set
6. Update of documents such as http rabbitmq mongodb
7. Some components or languages tested do not support values larger than int64 (mongodb, php have been tested and not supported), please pay attention when using
8. Fix the bugs that may exist after the ck plugin delete and insert, and the update and simpler filtering method has been solved, and has been tested by data verification
9. The mysql plugin fixes the hidden bug that stmt is not closed
v1.0.0
===========================
1. In the first version, all synchronizations are calculated by channel. If multiple tables are bound to a channel, all synchronization configurations of the bound tables are serial.
2. There are still many problems in binlog parsing. Only common types are supported, such as char, varchar, int, etc. There are still many bugs in the type support of deciaml and bit.
{{o.name}}
{{m.name}}