I have a question which is what I believe to be quite complicated and would like to see what experts like yourselves think about my solution.
Problem:
I am collecting data from a number of third parties into a central mySQL database. The third parties can have any database product (mySQL, Oracle, postgreSQL, etc). I will need to perform two tasks, which are the following:
- On first start, replicate the full database onto the central mySQL database
- Incrementally update the central mySQL database as changes are done to the third party database
Proposed solution:
- I intend on creating database crawlers in C# per third party database. So say for example one of the third parties has an oracle database. I intend on creating a class which will query an oracle database for its definition and data and programmatically create sql queries to be executed by the central mySQL database. ex: create table XYZ (id1 int, id2 int), Insert into XYZ values(1,2), etc.
- Here is the problem which is giving me the greatest grief. Id like to know if anyone has ever queried a database to effectively ask it "what are your latest changes", or what are your changes since dd-mm-yyyy hh:MM:ss. By changes I mean both data changes and data definition changes. ex: add new column, update a [row][column] value, insert new row, etc.