1 / 9

Overview of TenDB and TenDB Cluster Features Ported into MariaDB

Tencent Game DBA Team has successfully ported various features from TenDB and TenDB Cluster into MariaDB, enhancing its capabilities. Notable features include Instant Add Column, column compression, connection pool optimization, and bug fixes. The roadmap includes further collaboration with MariaDB developers, aiming to integrate more enhancements in future releases.

hedrosa
Download Presentation

Overview of TenDB and TenDB Cluster Features Ported into MariaDB

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Features of TMySQL/TSpider port into MariaDB Tencent Game DBA Team

  2. Agenda • Tencent Game DBA Team • TenDB and TenDB Cluster introduction • What features have been port into MariaDB • Roadmap of open source • What will we do next • Back to the open source

  3. Tencent Game DBA Team • Tencent Games • Hundreds of self-developed or delegated games，20000+ MySQL instances and 10000+ machines • 50% of the Tencent’s revenue comes from games • DBA Team Member • 4 people work on MySQL/Spider source code development – mysql kernel developer • 2 people work on DB automation management platform • 24 people work on DB management and maintenance jobs

  4. TenDB and TenDB Cluster Introduction • TMySQL -> TenDB • A MySQL branch maintained by Tencent Game DBA Team • Based on MySQL 5.5.24/Percona 5.6.24(thread_pool feature) – large-scale use in production environment • Based on MySQL 5.7 – in developing • TSpider -> TenDB Cluster Domain+Port • Base on Spider engine 3.1, started in late 2013 • Spider Spider Spider The largest Spider Cluster, 2000+instances • Auto sharding and auto-online scale out TenDB hash(id) =0 TenDB hash(id) =1 TenDB hash(id) =2 TenDB hash(id) =.. TenDB hash(id) =..

  5. Part1: What features have been ported into MariaDB • TenDB features have been ported into MariaDB Version Version Core Features Core Features Description Description Release Time Release Time MariaDB MariaDB Version Version Add column finished in less than one second, no need to copy table Use tcmalloc to solve the memory fragmentation problem 5 important mysql bugs fixed TenDB 1.1 InnoDB InnoDB Instant Add Column Instant Add Column 2012/12/10 MariaDB 10.3.2 TenDB 1.2 2013/3/20 Memory optimization core bugs fixed Support MySQL5.0 —> TenDB inplace upgrade Parallel data import（A5 60% Z3 90% data import time recude ） TenDB 1.3 2013/6/6 (In-Place) Upgrade enhance backup/import data Integrated grammar, semantic check to OSS change subsystem, improve business change efficiency Storage-engine independent column compression enables blob/text/varbinary columns to be compressed TenDB 1.4 2014/10/7 MariaDB 10.3.2 SqlParse InnoDB InnoDB Big Column Compression Big Column Compression Add new binlog events to store the compressed binlog and reduce the storage capacity restrict the speed at which the slave reads the binlog from the master Extended binary log events, support for parallel replication and recovery based on table, more than 3 times more efficient TenDB 1.5 2015/06/07 MariaDB 10.2.3 Binlog Binlog compression compression Read Read binlog binlog speed limit speed limit TenDB 2.1 Parallel replication 2016/1/1 relay log compression TenDB 2.1.4 relay log compression 2017/9/7

  6. Part2: What features have been ported into MariaDB • TenDB Cluster features have been ported into MariaDB(immediately) Production Production Item Type Item Type Name Name Description Description Status Status MariaDB Version MariaDB Version TenDB Cluster has been Merged into Spider 3.3(2017.4.1) branch: bb-10.2- spider Allocate connections on demand and recycle them regularly, which can enhance the cluster performance New Feature Connection Pool TenDB Cluster force index Push Down has been Merged into Spider 3.3(2017.4.1) branch: bb-10.2- spider New Feature force index push down to the remote DB instances in most case, limit x,y will cost lot of overhead, the optimization calculate which exact shard holds the needed data TenDB Cluster limit x,y optimization has been Merged into Spider 3.3(2017.4.1) branch: bb-10.2- spider Optimization TenDB Cluster pre_scan optimization has been Merged into Spider 3.3(2017.4.1) branch: bb-10.2- spider The prefetch function (pre_scan) in Spider causes the net read timeout when a single shard is processed too long Optimization TenDB Cluster has been Merged into Spider 3.3(2017.4.1) branch: bb-10.2- spider BUG Fixed Open table BUG solve a particular situation where cause infinite loops TenDB Cluster has been Merged into Spider 3.3(2017.4.1) branch: bb-10.2- spider BUG Fixed System Table BUG solve table’s name exceed 80 bytes cannot be created TenDB Cluster has been Merged into Spider 3.3(2017.4.1) branch: bb-10.2- spider BUG Fixed windows BUG solve the problem of debug on windows environment

  7. Roadmap of open source 2017.10.9 The second time to participate the MariaDB Developer Conference a) Discuss the Instant Add Column design and implementation, 45 days integrated into MariaDB 10.3 b) Discuss the TenDB Cluster connection pool and other features and bugs fixed integrated to MariaDB 10.3, and Spider engine will be the first stable version of the release MariaDB releases the alpha version of 10.3.2 Covering the two major features of TenDB: Instant Add Column and column compression 2017.04 Invite Monty to Tencent Exchange Technology.We shared kernel development and online practice for TenDB and TenDB Cluster 2016.07 2016.12 MariaDB release the Beta version of 10.2.3Covering the two features of TenDB Replication optimization:binlog compression and slave pull binlog speed limit 2016.10 Tencent Game DBA Team become bronze Participate in the MariaDB Developer Conference for the first time Identified the two features of TenDB Replication optimization integrated into MariaDB 10.2.3 version member of 2016.08 MariaDB Foundation

  8. What will we do next • Make Spider engine more stable in MariaDB – work together with Kentoku • Add distributed transaction support in TenDB Cluster • Solve the transaction consistency in fault scenarios • Push more features to MariaDB, we will benefit from this too • Candidates: Parallel replication(table level) or Relay log compression? • We are going to port the Flash Back feature(MariaDB) back to TenDB

  9. Back to the open source • Good people together will become more outstanding • Open source process allows us to constantly improve • Win-Win, Help us, help you • Spend less time to migrate the code from our branch to the newest MySQL version • More and more important features can be merged into MariaDB, which will influence many users, we can be proud to say that: this feature is from our team! • We can get back from MariaDB • Port improvements from MariaDB to TenDB • Port some useful feature into TenDB

More Related