Distributed Systems

What are background jobs and job queues?

How background jobs and job queues keep apps fast, what happens when a worker crashes mid-job, and how Temporal resumes the work from where it stopped.

  • 7 min read
  • 5 questions
  • +35 XP
Transcript

Background Jobs, explained by cats. Kitty uploads a PDF. Big file. Lots of pages. The app needs to: Validate it. Convert pages. Generate previews. Store results. Send Kitty an email. Too slow to do now. So Kitty uses background jobs.

The app sends the job to a queue. Later, a worker picks it up. Pages start converting. Previews start showing. Everything seems fine. Until a random cat uploads a huge file. The Memory spikes. Job fails. Then, the same job starts again.

Pages convert again. Previews regenerate. notifications get sent twice. Kitty's sad. Very messy. So Kitty writes a workflow: Same steps as before. But running with Temporal. Temporal tracks everything. If Pages one, two, And three are processed, but the job fails, A new worker cat appears.

Temporal tells the new worker: “Resume at page 4.” In the Temporal dashboard, Kitty can see what’s running, what finished, and what retried. User cats keep meowing. Worker cats keep working. Temporal makes sure that the work completes correctly