Description
This article compares the efficiency and speed of the GenericSQLInsertThreadReceiver and the GenericSQLInsertV2Receiver in specific scenarios. It explains how the performance tests were conducted to evaluate the time and speed required for creating database tables as the number of columns increases. The results show that the efficiency of the GenericSQLInsertThreadReceiver increases with an increase in the number of table columns due to its ability to process data using multiple data collectors, buckets, and values across multiple threads. In contrast, the performance of the GenericSQLInsertV2Receiver decreases as the number of table columns increases since it processes values in a single-threaded manner.
Table Of Results From Performance Tests
The table below contains the results of the performance test.
Performance Test Criteria
The following criteria were defined to effectively test the veracity of these two Receivers.
- There will be three levels of data insertion: 10,000, 100,000, and 1,000,000 values, respectively.
- There will be 6 performance tests for each insertion level, and the number of table columns will increase from 25 to 150 in increments of 25.
- For the GenericSQLInsertV2Receiver, there will be a total of 15 performance tests.
- For the GenericSQLInsertThreadReceiver, there will be a total of 90 performance tests.
- There will be 105 tests, all in total.
- The parameters for the GenericSQLInsertV2Receiver will be identical for all tests.
- The parameters for the GenericSQLInsertThreadReceiver are the same for the number of buckets and the number of values per bucket. However, the number of data controllers increases by 1 as the number of table columns increases by 25.
Test Result Observations
- Having a higher number of data collectors does not necessarily lead to greater efficiency as the quantity of table columns decreases. In fact, the opposite may occur. The reason for this is that the data collectors process and prepare data at a much faster rate than the database can handle the batches, which leads to the JVM managing more objects in memory, performing more garbage collection, and slowing down the over-processing of data.
- When inserting 10,000 values into any table, the tests where there is more than one data collector are not applicable. This is because each data collector has 10 buckets that can hold 1,000 values each. Therefore, the first data collector will process all 10,000 values.
- At 50 table columns or more, the GenericSQLInsertThreadReceiver was more efficient in all test cases.
- At 25 table columns or less, the GenericSQLInsertV2Receiver was more efficient. In fact, with 25 table columns or less, the GenericSQLInsertThreadReceiver was much less efficient on all tests except when three data collectors were used, and even then, it was only on par.
- Most tests show that three data collectors are optimal, but when the number of data columns is 150 or more, five data collectors are better.
Test Parameters
The performance tests utilized the hardware, OS, database, and GenRocket domain parameters defined below.
Hardware & OS Parameters
- Macbook Pro M1 Max
- Main OS Mac Sonoma 14.3.1 64-bit
- 10 CPUs
- 64GB RAM
- 2TB SSD
- Running within a Parallels VM
- Linux Ubuntu 22.04.4 LTS 64-bit
- 4 CPUs
- 20 GB RAM
- 274.9 GB of the SSD
Database Parameters
- Database - Mysql Ver 15.1 Distrib 10.6.16-MariaDB, for debian-linux-gnu (aarch64)
- Database Engine - InnoDB
- Five Test Tables
- load_tester_25 - 1 Bigint Primary Id column, 25 Bigint columns
- load_tester_50 - 1 Bigint Primary Id column, 50 Bigint columns
- load_tester_75 - 1 Bigint Primary Id column, 75 Bigint columns
- load_tester_100 - 1 Bigint Primary Id column, 100 Bigint columns
- load_tester_125 - 1 Bigint Primary Id column, 125 Bigint columns
- load_tester_150 - 1 Bigint Primary Id column, 150 Bigint columns
GenRocket Domain Models
- Six Domains
- id Attribute - Uses the RangeGen Generator
- col* Attributes - Use the RandomGen Generator
Image of the Parallels VM Where Tests Where Run
MySQL / MariaDB DDLs
The following DDLs define the Innodb tables created for performance testing with the GenericSQLInsertV2Receiver and the GenericSQLInsertThreadReceiver.