I have a requirement to replace the contents of a database table with the contents of a file. This happens every day at a set time. This is done using a Java application. I don't have access to the original database that produces the file.
The legacy application is creating a new table and inserting the file contents into the new table. It then drops the old table and renames the new table to the old name.
My lead says he has a better way and that is to load the contents of the file into memory. Next we load the contents of the old table into memory. We do some comparisons and come up with an insert list (entries only in the file), update list (entries with the same pk that are in the old table and the file and don't match) and a delete list (entries that are only in the old table).
My lead says the benefit of this is that reading every row from a file and inserting it takes a long time, supposedly due to file I/O. Well we are still reading every line in the file here. Maybe the benefit is that it is not inserting duplicate rows which the data may very well have mostly duplicate rows. Also this method doesn't need drop and create table permissions as the old method would which seems like a good thing from a security perspective.
My question is are there other concerns with this technique? It does seem to use alot of memory and we are basically loading the contents of the table into memory twice. Is there a better way to do this? One thought would be to load the file contents into a new table and do some joins. Although after the new table is loaded we might as well just swap the tables as the legacy application does.
I probably will end up comparing the two strategies in terms of memory usage and speed but I would like to know if either of these are a best practice and if not, what is?