Quantcast
Channel: StackExchange Replication Questions
Viewing all articles
Browse latest Browse all 17268

application that writes to two databases... mongo for reporting... redis for frequently used CRUD data

$
0
0

I am about to design a database solution that has the following requirements:

  • lives on a WAN where certain parts of the network suffers from high latency.
  • network connectivity in some areas can be lost multiple times / day
  • i have to replicate the master database to slave locations which will have no hard drives.
  • in addition to provide CRUD functionality, the administrators will need to have several reports created for them.

I'm just new to redis - aka this is literally day one for me... but it seems like a good fit as far as being able to set up some slave databases so they run in memory only...

But ... as far as the various types of data I will need to produce reports... I'm not how easy it will be to query redis in the ways we might need. From the little that Ive read so far, you can only query based on keys... not on values. I know you can create hash values and secondary indexes... but... I was wondering if I should consider using both redis and mongo. Mongo is a breeze as far as changing "schema" of documents on the fly (which is good for us because requirements constantly change) and you can query documents any number of ways.

Proposed Physical Architecture

Master Database Server

This server will run mongodb as a "reporting" repository. It will also have a Redis database which will contain just the information that will be queried and written most often. So for example:

  1. Redis Master widget query database

    • will contain basic widgets key/values that 90% of database clients will querying. It will only be written to by a few clients and not very often.
    • properties may include items like: widget_id, widget_name, widget_price.
    • this database will be persisted to disk.
  2. MongoDB Widget Reporting and Management database

    • will include a document for each widget which have the same base information contained in the redis database... plus a bunch of other properties and historial information about the widget that will only be used by administrators (10% of end users...)

Redis Slave Servers

These servers will only contain an in-memory copy of the Redis widget database. These servers will be read only. No reporting will be run off them.

Comments:

As I debate this design / architecture, these are the questions and thoughts that come to mind:

  1. How will you keep the mongodb and the redis db on the "master' server in sync? I was thinking of writing to the Redis server first... and only if successful, populating mongo db. What's the impact if they get out of sync? How to resolve?

  2. Why not just use mongodb across the board? Mongodb can also do in memory databases ... although I haven't tested it yet. From what I've read it "feels like" more people have been using redis for in memory stuff compared to mongodb. But I could be wrong.

I'd like to get some feedback about this idea... Am I overcomplicating things? should I just figure out how to set up replication in mongodb with the master persisting data to disk.. and the slaves running only in memory? or should I continue down this path where I evaluate trying to marry best of breed tools?

Thanks.


Viewing all articles
Browse latest Browse all 17268

Trending Articles