The Problem: Long-Running Tasks in a Web Request

Imagine a user signs up on your website. After they submit the form, you need to:

  1. Save their info to the database.
  2. Send them a welcome email.
  3. Generate a custom PDF report for them.

Step 1 is fast. But steps 2 and 3 can be slow. Sending an email might take 2-3 seconds if the email service is slow. Generating a PDF could take 10 seconds. If you perform all these tasks sequentially inside your HTTP request handler, the user will be staring at a loading spinner for over 10 seconds. This is a terrible user experience, and the request will likely time out.

The Solution: Job Queues for Background Processing

The solution is to do the fast work immediately and offload the slow work to the background. This is a perfect use case for a job queue.

The workflow is as follows:

  1. The user submits the form.
  2. Your web server receives the request. It performs the quick task of saving the user to the database.
  3. Instead of sending the email and generating the PDF itself, it adds two jobs to a queue (e.g., an email queue and a pdf queue). A job is just a message containing the data needed to perform the task (e.g., { userId: 123, email: 'user@example.com' }).
  4. The server immediately sends a success response back to the user (200 OK or 202 Accepted). The user's browser stops loading instantly.
  5. In a separate process, a Worker is constantly listening to the queues. It picks up the jobs one by one and executes the long-running tasks (sending the email, generating the PDF) in the background, without affecting the web server's performance.

This pattern is a specific, high-level application of the message queue concept we discussed earlier. Libraries like BullMQ use Redis as the message broker to provide a robust and feature-rich job queue system.

A Practical Example with BullMQ

Let's build a simple system to offload sending a welcome email. You'll need express and bullmq.

Part 1: The Producer (Your Express Web Server)

This code receives the request, adds a job to the queue, and responds immediately.

server.js:

JavaScript


import express from 'express';
import { Queue } from 'bullmq';

// --- Setup ---
const app = express();
app.use(express.json());

// Connect to the queue. The queue name 'emailQueue' is important.
// It uses Redis for storage.
const emailQueue = new Queue('emailQueue', {
  connection: { host: '127.0.0.1', port: 6379 },
});

// --- The Route Handler ---
app.post('/users', async (req, res) => {
  const { email, name } = req.body;
  
  if (!email || !name) {
    return res.status(400).json({ message: 'Email and name are required.' });
  }

  // Add a job to the queue
  // 'sendWelcomeEmail' is the job's name, for organization
  // The second argument is the data the worker will need
  await emailQueue.add('sendWelcomeEmail', { email, name });

  // Respond to the user immediately
  res.status(202).json({ message: 'User created. Welcome email will be sent shortly.' });
});

app.listen(3000, () => console.log('Web server running on port 3000'));

Part 2: The Consumer (Your Background Worker)

This is a completely separate process that listens for and processes jobs from the queue.

worker.js:

JavaScript


import { Worker } from 'bullmq';

// --- A mock function to simulate sending an email ---
const sendEmail = async (email, name) => {
  console.log(`Sending welcome email to ${name} at ${email}...`);
  // Simulate a 5-second delay
  await new Promise(resolve => setTimeout(resolve, 5000));
  console.log(`Email sent to ${email}.`);
};

// --- The Worker ---
// It listens to the same queue name 'emailQueue'.
const worker = new Worker('emailQueue', async job => {
  // This function is the "processor"
  console.log(`Processing job ${job.id} of type ${job.name}`);
  const { email, name } = job.data;
  
  try {
    await sendEmail(email, name);
  } catch (error) {
    console.error(`Job ${job.id} failed with error: ${error.message}`);
    throw error; // Throwing error will cause BullMQ to retry the job if configured
  }
}, { connection: { host: '127.0.0.1', port: 6379 } });

// --- Event Listeners for logging ---
worker.on('completed', job => {
  console.log(`Job ${job.id} has completed!`);
});

worker.on('failed', (job, err) => {
  console.log(`Job ${job.id} has failed with ${err.message}`);
});

console.log('Worker is listening for jobs...');

To run this, you would open two terminals. In one, run node server.js. In the other, run node worker.js. Now, if you send a POST request to http://localhost:3000/users, you'll get an instant response, and you'll see the worker log that it's started the 5-second email sending process in the background.