Generating Data for a Remote Database
The GenRocket engine generates data at approximately 10,000 rows per second. It merely hands off the data to a respective Receiver. In this case, it is handing it off to a Receiver (e.g., GenericSQLInsertReceiver) which is communicating to your database via JDBC.
The URL and connection to your database is over a distance outside of your location where GenRocket is generating the data. Thus, throttling and packet loss may occur when large quantities of data are sent. This is why we do not recommend that the generation of said data be generated to a non-local database via JDBC, nor do we consider generating data from a local machine to a source outside the local environment secure.
We also do not recommend this paradigm because sending large volumes of data outside the local environment of the data is slow.
Additional JDBC Information
JDBC is best designed to connect to a database within a closed secure environment where the database is near to the application speaking to it. Even Amazon recommends that applications connecting to their Aurora database exist within the same availability zone and replication servers are no more than 100 kilometers/60 miles apart to avoid latency.
We have not personally worked with any application where components outside of the primary data center (test or production) directly connect to a database via JDBC. The outside component normally communicates indirectly to the database via an application, and the application securely connects to the database within the same local environment.
Our customers insist on a high level of security from GenRocket even though the data we are generating is 100% secure because it's synthetic data that GenRocket produces. Thus, we recommend and follow only the most secure testing practices, and having an outside component directly connecting to a database via JDBC is not a secure practice. We don't recommend it, and we don't support it.
Recommended Solutions
The data performance and connection challenges are not a GenRocket issue when generating data for a remote database. Here are the solutions we recommend for achieving a more optimized and secure solution, and we will support:
Recommendation 1: GenRocket Multi-User Server
Install a GenRocket Multi-User Server (GMUS) on a machine within the same environment as your test database. The GMUS does not have to be on the same machine, just within the same location where the distance to the database is much shorter, and the connection to the database is secure because it's connecting within the same environment.
- Testers can send commands to the GMUS via REST as to which Scenario to run (this can include a G-Case).
- The GMUS will use the GenRocket engine to load and execute the instructions of the Scenario within the local environment.
- The GenenericSQLInsertReceiver will securely connect to the database via JDBC within the local environment and should be able to send batches of data to the database optimally.
Note: This still depends on how well your database has been configured to receive the data optimally (Primary Key, Indexes, Foreign Keys, etc.).
Recommendation 2: SQLFileInsertReceiver
In absence of using the GMUS, you can use the SQLFileInsertReceiver, to write ANSII SQL inserts statements (single or batch) to a file.
You can add a second Receiver (FTPReceiver, SFTPReceiver, S3Receiver, etc.) that can take the resulting file to send and deposit the file on a machine within the same secure location as your test database.
This alternative would require you to have a solution within your testing environment to read the SQL Inserts statements from the file. Most databases have this capability built in; it's just a matter of calling the database with the proper command.
Recommendation 3: Implement a JDBC Driver Designed for Long Distances
Your team can implement a type of JDBC driver better designed to work securely over long distances with large batches of data.