Sequential Job Processing in NodeJS

In software development sometimes we come across scenarios or tasks which consume a lot of system resources such as CPU time, memory, network bandwidth, or IO. If we have a bundle of such tasks to perform, it becomes really challenging to manage them with a limited amount of system resources. Soon processing such tasks becomes heavy and error-prone. So to handle these tasks or jobs we need a mechanism to process such tasks smoothly and without breaking the system i.e. making economical use of system resources and keeping the system running.

Processing such tasks in NodeJS becomes even more challenging due to its single-threaded nature and if you try to run such resource-intensive tasks in parallel, the system might become laggy or even crash.

One possible solution to handle such tasks is to build a queue of tasks and process each task one by one so that each task is given good enough system resources to complete without breaking.

Kue is a handy NodeJS module that can help us develop such a job processing priority queue in no time. At one end of the priority queue (tail end), we can submit our tasks to be processed and they will be enqueued in the queue. At another end of the queue (head of the queue), we can pick enqueued jobs one by one and process them. So it will be a fairly simple traditional queue with a priority of first-in-first-out. Kue makes use of Redis as an in-memory database, to store and maintain the queue. Besides managing queues, it also keeps track of the jobs and their state.

Lets Code !!!

So now that you have a brief overview of sequential job processing via a priority queue, it is time to implement it and see how it works.

Let's setup a node project first and then install the following packages:

npm install kue sleep --save

Kue as mentioned earlier is that node module for queue management. ‘sleep’ package is gonna let us halt the execution of the process for a given interval of time. I’ll just use it to imitate that a certain task/job is taking some time to finish (Will become clear in the example).

Since Kue is Redis based and requires an instance of a Redis server to be running in the background. So you should install redis if haven’t already and go to a terminal window and type :

redis-server

Redis server will be up and running and you will get some output like this.

Redis server instance running locally

So our job processing queue is gonna have two ends :

• One where we can submit a job to be processed, we call this script ‘client’
• Other where we’ll pick a job from the head of the queue and start processing it, we’ll call this script ‘worker’.

We’re gonna write a node script for each one of them and run them separately.

touch client.js worker.js

Submit Jobs to Queue

We will implement our client first, which will be responsible to submit a single job to the queue.

Here, we’re importing in our ‘kue’ module first and initializing an instance of ‘kue’ by calling the createQueue() function. Now we’re ready to create our job queue. We’ll make a ‘download’ job queue by calling queue.create() function on line 11(shown in the figure above). There are two parameters provided to create() function, job name, and some data in JSON format, related to the job or required by the job. Here we’re passing a sample file path to be downloaded in the JSON object for sake of an example. At the end, we’re also calling the save() function which means that our job will be submitted and saved in a queue.

There are several events in the queue that we created and we can do when they are invoked. For example on line 5 queue.on() function is used to listen for the ‘job enqueue’ event. This event will be fired when a job will be added to the queue. We’re using this event here to notify the user that their job is submitted in the queue. Every time calls the client script a job will be submitted to the queue. Simple and straightforward.

Process Jobs at Queue Head

So we have set up a simple client to enqueue/submit jobs to the queue. Now we need to implement the other part of the queue aka head of the queue to process these submitted jobs. We’ll implement it in a ‘worker.js’ script separately.

Job Worker to process jobs from the head of the queue

In worker, we’ll instantiate the same instance of the ‘download’ queue that we did in the client script using queue.process() function. We’ll pass it a callback function with job data. That function is actually responsible for performing the required task. We’re calling a helper method downloadFile() in this scenario to imitate a file download. We’re purposely halting it for a few seconds using sleep() function to imitate a file download. When the task is complete we’ll call the done() function that will indicate to our queue that job is now finished and we can pick the next job from the queue.

Let’s give our simple queue a test run. In a terminal run the worker script :

node worker.js

In a separate terminal run the client script several times to submit multiple jobs in the queue.

node client.js

Job Processing using Kue

As soon as we submit jobs in the queue the worker will start processing the jobs one at a time. We can submit as many jobs as we want as long as the worker is running.

Improvising our Queue a bit more

We’re able to successfully build a very simple job processing queue. We can submit jobs at one end and they are processed one by one on the other end, fair enough. But still, there’s plenty that we can do with our queue to further improvise it. For example, what would happen if our job fails to be complete, let’s say due to some network failure or power outage? We should make our queue to be able to handle such issues and exceptions and keep on processing the jobs avoiding all such situations.

In our client script where while submitting the job we can make the job make several attempts in case it fails the first time and so on. Kue provides us with an attempt() function, we can pass the number of attempts made to process a job.

Sometimes it’s not advisable to make a re-attempt right away after a job fails. For example, if a job fails due to network failure we might run out of a number of attempts even before the network recovers. So it’s a good idea to wait for some time before the next attempt for job processing is made after a failure.

To make a job wait before re-attempt we can use the backoff() function of kue to make it wait before it makes the next attempt.

Listing Jobs

Let’s say we’ve submitted several jobs to the queue, some of them have been completed, some are being processed and some are still waiting in the queue. Kue provides us with the option to list all the jobs that we ever submitted to the queue. Let’s write code for listing all the jobs in a separate script called ‘listJobs.js’.

Kue provides us with a range() function that we can use to list all the jobs that we ever submitted to the queue. Here 0 and -1 mean that we need a record of all the jobs from beginning to end, however, the range is customizable. Next, we’re providing a range function with the order in which we want to retrieve those jobs, and here it’s ‘asc’ ascending order.

The range function will return us a list of jobs. Next, we can iterate over all the jobs and display their info, and the output looks something like this.

We can also retrieve jobs by certain state. A job can be possibly in the following states :

• Active (being processed at the head of the queue)
• In active (Waiting in the Queue)
• Delayed (took more time to process than it should)
• Failed (failed due to some errors/exceptions)
• Complete

Just like range() function Kue API provides us with a helper function for each particular job state job/jobs are currently in. For example, to retrieve a list of completed jobs we’d write :

Similarly, we can list active, inactive, delayed, and failed jobs by calling jobs.active(), jobs.inactive, jobs.delayed, jobs.failed() respectively.

Setting UI to visualize Jobs

So we’ve covered how can we interact with Kue programmatically and retrieve different information from the queue. At this stage it would be ideal to set up an interactive user interface or sort of an admin panel in a browser window, to interact with the queue.

Luckily Kue already provides us with a user interface so that we don’t have to write one from scratch, we can if we want to however (better not to re-invent the wheel unless our wheel is more special :) ).

All we have to do is import a node module ‘kue-ui’ in our project and install it. We’re also gonna need express and host an instance of kue-ui on our local server.

Kue UI hosted on Express Server

After setting up Kue UI I’ll go ahead and view it in my browser :

Kue UI running locally

As you can see in the screenshot above, Kue UI list all the jobs by state. Not only we can view complete job details but we can also delete or reschedule jobs in certain states.

Conclusion

So in this post, you’ve learned how can we set up a sequential job processing queue using NodeJS. We’ve used the Kue node module for setting up a very simple demo queue. We’ve also set up a user interface in order to interact with the queue as well.

I’ve provided the link to the sample code I used for this tutorial. Feel free to let me know about your thoughts, queries, or suggestions, Thanks for the read.

(Code: https://github.com/shabbir1993/job-queue).