Data transformation testing and challenges

I am going to be discussing my experience in testing data transformation in one of the projects.

Requirement

What we wanted to do is verify that certain rules are followed when data in Oracle database is transformed. e.g. For specific column C, scan all records and verify that value "abc" is changed to "xyz" It was decided to use python scripts to validate the data.

Challenges

The challenge was that destination database was not available for testing purpose immediately.

Solution

So here was the thing. Python script wasready but since destination database was not available, we fetched data from source database twice and one copy was treated as source and other as destination. Even though data was same in source and destination, it was useful in testing our scripts. Once data is fetched from the source copy, we edited some records to simulate real tests. That's how we were able to ensure that python scripts were properly developed and it saved a lot of time.