I need to insert a 5K records every minute over an "unreliable" Internet connection to a cloud-based database (from different 10 systems all running simultaneously). The Internet connection is 10 MB and my app needs to tolerate an outage once a month for up to 1 day. My app is running in a factory in Asia with poor IT infrastructure/support which is the source of the unreliability and long outages.
Can you please recommend how to do this without loosing data?
Solutions I've thought of:
Keep an in-memory (or simple text file) queue of inserts and every time an insert fails put it in the queue to be tried again later. I don't like this because I'll have to redesign my app to have a parallel loop to handle this so as to not throw off the timing of the main loops.
Use a message broker/queue (like RabbitMQ). I don't like this because while getting the SQL statements into the broker is easy I'd rather not have to right an additional service that polls the broker, executes the sql statements, handles errors, etc. I'm aware of Mule ESB which does all that, but it looks complex to setup.
Database replication. I would actually need N:1 (merge) replication which I believe isn't supported on any major database other than MS SQL Server which I'd rather not have to deal with because of the complexity. I would also likely need to restart the database after such an outage and wait for it to catch up before performing any more writes both of which are undesirable.
Any other solutions? Thanks.