Overview
When a bug is discovered in your production environment, you have to become a detective and immediately seek answers to the following questions:
- What production data became corrupted because of the bug?
- Where in the source code does the bug exist?
- What steps need to be taken to repair the source code?
- What steps need to be taken to repair the production data?
Two Important Rules for Ensuring Bug Fixes Work in Production
- Never push repaired code into production unless it has been thoroughly tested and fully validated.
- Never use corrupted production data to test your repaired code.
You may also need to copy your Production Database to a Test Database, not to test the repaired code, but rather to test the automated scripts you must write to repair the corrupted data within your Production Database.
Recommended Methodology for Avoiding Broken Code and Corrupted Data
The following methodology described below defines three phases that, when followed, will facilitate getting your production environment repaired and free of the broken code and corrupted data. The three phases are:
- Find the Bug
- Repair the Bug
- Repair the Corrupted Data
Phase 1: Find the Bug
- Identify the data that was corrupted in production (normally one or more rows of data, within one or more tables, within one or more databases).
- Identify the code that caused the corruption (this code is where the bug exists).
Phase 2: Repair the Code
- Determine what modifications need to be made to the code.
- Write one or more stories that describe how you want to test and validate the new code.
- Repair the code that contains the bug.
- Use GenRocket Domains, Attributes, Generators, Receivers & Scenarios to model the data you need to test the repaired code.
- Use GenRocket Test Data Cases combined with Test Data Rules, and Test Data Queries to create the exact test data necessary to test the repaired code in numerous ways to attain maximum validation of the repaired code.
- Implement Unit Tests, that will use GenRocket Test Data Cases to generate data in real-time, and then test and validate the repaired code.
- Run your Unit Tests to validate the repaired code.
- Once the Unit Test successfully validates the repaired code, promote the repaired code to production.
- Add the Unit Tests to your automated regression suite to ensure automated real-time testing of the code on a daily basis and before every code release.
Phase 3: Repair the Corrupted Data
- Write one or more stories that describe the steps necessary to repair the corrupted data in production.
- Write the necessary Repair scripts to repair the corrupted data in production.
- Write the necessary Validation scripts to check, validate, and report findings.
- Make a copy of the Production database (at least the subset of the data that is corrupted) to a Test database.
- Run the Repair scripts on the Test database.
- Run the Validation scripts on the Test database.
- Review the Validation report.
- Once Validation reports a finding of 100% validation, do the following:
- Run the Repair Scripts on Production
- Run the Validation Scripts on Production
- Run the Repair Scripts on Production
- Run the Validation Scripts on Production
In Summary
While the three phases and inner steps defined above ensure a sound methodology for fixing production code and repairing production data, it should be noted that when GenRocket is used, with good automated testing practices where up to 98% of the code is eventually being thoroughly tested, the following benefits are received:
- The event of bugs entering the production environment is drastically reduced.
- The ability to implement new code and get new features to production is radically increased.
- Overall, the cost for implementing new features, bug fixing, and repairing corrupted production data is drastically reduced as well, sometimes by as much as 50%.