We run a postgreSQL (9.4) server to store data for a web application. Our next feature will require a very data intensive processing on the server requiring a lot of db queries and computation over these data.
As we update our database daily, the web requests are valid across all users and throughout the day. So we obviously want to cache results. That works well for us (results stored in Redis) except that everyday, we have a cold start problem, and each possible request can take multiple seconds to complete the first time it is requested by a user.
The possible web requests are actually predictable and today we can list circa 30000 ones, but as the feature evolves, we could easily reach 100000 or 500000.
So we think about warming the cache up, processing every request and caching the result, so our users wouldn't experience delays. The processing code runs in nodeJS in Docker. We would be able to run multiple of these containers (on another machine) to parallelize the computation and expect it to divide the total caching time by the number of containers that would be able to run at full speed.
A quick test shows that the postgres server becomes the bottleneck in the process: half (and sometimes more) of nodejs execution time is spent retrieving data, and during this time the postgres process runs at ~ 100% cpu. To be able to divide by the number of containers, we need postgres to answer each one of them simultaneously at full speed. I read that postgres dedicate a single core to each session, so my guess is: if we provide postgres we the same number of core than the number of container that it has to answer to, then we would be back in line with our original expectations (linear reduction of computing time).
So first question: Is this reasoning OK in theory? (Any alternative suggestion ?)
Let's assume it is.
Our production server currently runs on 4 vcores. Our provider can start instances up to 32 vCore on-demand. It is not reasonable to think that our production setup will run 24/7 a 32-core machine, to handle a job that might not exceed 1 or 2 hours a days, but won't be using more than 1 or 2 cores the rest of the day.
A solution that we think of is to spin a 32-vcore machine at a given time of the day, with a postgres server on it, wich would synchronize with our main 4-vcore postgres. Up on sync completion, we could run 16 containers that would work against a 16-vcore postgres instance. When all caching is done, we would suspend our 32-vcore machine and get away with an advantageous hourly billing. That also seems to scale linearly, as if we start multiple 32-vcore instances, each can sync their own postgres server and spin up their own containers.
Second question: Is this 2nd reasoning OK in theory? (Any alternative suggestion ?)
Let's assume it is.
How would we accomplish sync between 32-vcore postgres and main 4-vcore postgres ? A naive way would be to do a single pg_dump on the 4-vcore postgres, and download that file on each 32-vcore machine to run them against the local postgres. Seems simple, maybe time consuming. Another way would be to setup 32-vcore postgres servers as hot stanby servers, and to expect them to catch up only with missing data when we restart their host. But we are not yet familiar with that technology and its capabilities, so any advices would be kind.
Any help/idea appreciated.