Cluster question

I'm running a Node app that fetches Redis data into another database. It´s using the cluster module, everything works fine but I need to ensure that workers will finish properly their jobs(they use BRPOP then allocate into a object then insert in batches cleaning the objects), how do I do this?

There are many ways to do this but we created strong-mq with this kind of use case in mind. What’s nice is you can use a common message-passing pattern between workers that runs natively over node-cluster without installing anything else, and if you end up scaling across servers you can not change your code but go to a different messaging backend.

Another way would be to implement a simple semaphore kind of setup using strong-store-cluster. Or do the same in Redis.          

Create APIs in Node in minutes: LoopBack

if I do something like this on a worker process, wouldn't be effective?

process.on("message", function(m) {

  if(m.cmd === "disconnect") {

     // cleanup logic

  }

})

Yes, that works pretty well actually, I've used it a lot to synchronize a small in-memory cache to each process and distribute jobs to workers. 

>> I'm running a Node app that fetches Redis data into another database. It´s
>> using the cluster module, everything works fine but I need to ensure that
>> workers will finish properly their jobs(they use BRPOP then allocate into a
>> object then insert in batches cleaning the objects), how do I do this?
> if I do something like this on a worker process, wouldn't be effective?
>
> process.on("message", function(m) {
>   if(m.cmd === "disconnect") {
>      // cleanup logic
>   }
> })If I understand correctly, you want workers to be given a chance to
finish what they are doing when disconnected using node cluster?

In principle this works, though that message doesn't exist (unless you
send it?). You don't say how/when you fork and/or disconnect workers.

Generally, this would happen in gracefull shutdown by calling
`cluster.disconnect()` in the node master. Unfortunately, while that
waits
for cluster worker ***servers*** to close, it doesn't wait for
clients, like redis clients. LIke yours, likely.

You can trap the disconnect message if you are willing to use node
internals: https://gist.github.com/sam-github/8185222

node cluster should really expose that event, IMO, but so it goes, for
the moment.

Most supervisors, strongloops's included, offer a way to inform the
worker that it should cleanup. `slc run` sends a shutdown message,
https://github.com/strongloop/strong-cluster-control/blob/master/api.md#controlcmdshutdown,
for example, when shutting down, and gives 60seconds grace to the
worker (configurable), before SIGTERMing it.

Interesting Sam, but what if I set a flag that is checked every process.nextTick and suspends redis updates, so what's left is finishing iterating over in-memory items?

> Interesting Sam, but what if I set a flag that is checked every
> process.nextTickYou will eat 100% CPU, rather than waiting on event from the master
saying that its time to go away, and being able to do the cleanup you
want:

> suspends redis updates, so what's left is finishing
> iterating over in-memory items?




Well I did it and the CPU is running at 0.2%

Can you share how you accomplished this? (in code, if possible.. JS is way better than English with the diversity of backgrounds and native languages seen on this list)

I'm not aware of a JS level API in node core that provides what you described, so like Sam, I assumed you meant a busy loop which generally would indeed use 100% CPU and block your event loop.

Did you mean an arbitrarily long process.nextTick() chain, instead? The behaviour of which is so different from "once per tick" that it's a shame the name hasn't been changed! Your check would be called up to 1000 times (by default) per tick and then be queued into the next tick.

Or did you mean sprinkling one-off process.nextTick() calls throughout your code so that a check is queued at least once per event loop tick?