When and why do we need background jobs?

Nowadays, background job processing is indispensable in the world of web development. The need for background jobs stems from the fact that synchronous execution of time-consuming and resource-intensive tasks would heavily impact an application’s  performance and user experience. 

Even though Elixir is famous for its concurrency model and can easily handle multiple processes simultaneously, there are still several reasons to consider adding a third-party tool like Oban to manage and monitor their background processes.

What is Oban and its advantages?

What if a process fails or the server crashes? Oban provides elegant solutions for those and any similar problems, all with a convenient UI for managing and monitoring said processes and load.

Oban is a highly efficient Ecto-based background job processing system. It is famous for its intuitiveness and reliability. 

One of the advantages of Oban is that it uses a database (PostgreSQL or SQLite) to store and manage its background jobs. The aftereffect of that is the mentioned reliability. Because the jobs are persisted in the database, if something were to happen, like the server crashing or a job failing, the job data wouldn’t be lost. With that, a lot of the complexity regarding background job processing is already taken away by Oban. Other key features Oban offers are job scheduling, job prioritization, periodic and unique jobs, concurrency control, etc.

Oban Integration

With Oban being Elixir-based and built on top of PostgreSQL, and Elixir applications mostly using PostgreSQL in their stack anyway, the integration of Oban into your project should be very simple and straightforward.

This article will explore Oban integration on an imaginary Elixir Airline application. 

Essentially, when purchasing a ticket at the airline’s website, the airline sends an email confirmation to the user. The whole process is done consecutively, which is very unnecessary in this case, having that the email confirmation could easily be delegated to a background process, and thus, making the process of the purchase faster and giving the user a better experience at using the application.

Let’s investigate how we would achieve this in our Elixir app using Oban.

Getting started

Firstly, we will need to add the Oban dependency into our project. To do so, we need to locate our mix.exs file and add the following: 

# mix.exs
def deps do
  [
    {:oban, "~> 2.16"}
  ]
end

After adding the dependency, we will need to run mix deps.get.

Because Oban uses the database to store jobs, we need to add a database migration to add the needed oban_jobs table. With this table, Oban will track everything it needs to know regarding our jobs, such as the state of the job, when it was attempted to execute the job, how many times it was attempted, etc.

mix ecto.gen.migration add_oban_jobs_table
defmodule Airline.Repo.Migrations.AddObanJobsTable do
  use Ecto.Migration

  def up do
    Oban.Migration.up(version: 11)
  end

  def down do
    Oban.Migration.down(version: 1)
  end
end

Now, we need to run the migration using mix ecto.migrate.

Configurations

We have successfully added Oban as a dependency in our project. The next step is adjusting some configurations. In our config.exs file, the next configuration is needed:

# config/config.exs
config :airline, Oban,
  repo: Airline.Repo,
  plugins: [Oban.Plugins.Pruner, max_age: 300],
  queues: [purchase_ticket: 10]

The queues option marks the maximum number of concurrent jobs allowed in the queue (purchase_ticket queue in this case). 

The Pruner allows us to state for how much time we want to keep completed, cancelled or discarded jobs in our database. This prevents our database from growing indefinitely. By default, the Pruner will retain jobs only for 60 seconds, but we can change this by adding the max_age option (300 seconds in this case, or 5 minutes) or it can also be specified by the number of rows we want to keep, for instance: max_len: 3000 means we only store the most recent 3000 jobs. Also worth mentioning is that the Pruner will never delete a new job, a scheduled job or a job which failed and will be retried.

The only thing between us and setting up a new Oban job is adding Oban to our supervisor. In our application.ex we have to add it to the list of supervised children:

# lib/my_app/application.ex
def start(_type, _args) do
  children = [
    Airline.Repo,
    {Oban, Application.fetch_env!(:airline, Oban)}
  ]
end

Defining Workers

Intuitively, the Oban module which does the work of processing jobs is called a Worker. To use it, we need to define our own module, let’s say PurchaseTicketWorker and use the Oban.Worker module. In the module, we call which process we want to call in the background in the purchase_ticket queue. Each Worker must define a perform/1 function. It should accept an Oban.Job struct with a list of arguments. The arguments can be used in a variety of ways, for example, differentiating what should be done in the perform function and they will be stored in the database. We will not use them in this case.

defmodule Airline.PurchaseTicketWorker do
  use Oban.Worker, queue: :purchase_ticket

  @impl Oban.Worker
  def perform(%Oban.Job{args: _args} = job) do
    # dbg(job)
    with :ok <- Airline.PurchaseTicket.send_email() do
      :ok
    end
  end
end

To handle errors and job retries, we just have to add another option to the job, max_attempts

Now, if the job execution fails for whatever reason, and the number of attempts to execute the job is lower than max_attempts, the job will automatically be retried in the future and the error will be logged in the database.

use Oban.Worker, queue: :purchase_ticket, max_attempts: 3

After executing our job, if dbg(job) is uncommented, we should get the following output:

[lib/airline/purchase_ticket_worker.ex:6: Airline.PurchaseTicketWorker.perform/1]
job #=> %Oban.Job{
  __meta__: #Ecto.Schema.Metadata<:loaded, "public", "oban_jobs">,
  id: 4,
  state: "executing",
  queue: "purchase_ticket",
  worker: "Airline.PurchaseTicketWorker",
  args: %{},
  meta: %{},
  tags: [],
  errors: [],
  attempt: 1,
  attempted_by: ["DESKTOP-8JVA10V", "bd6da488-083b-484f-b688-996a63997048"],
  max_attempts: 3,
  priority: 0,
  attempted_at: ~U[2024-01-28 21:29:07.048000Z],
  cancelled_at: nil,
  completed_at: nil,
  discarded_at: nil,
  inserted_at: ~U[2024-01-28 21:29:06.991971Z],
  scheduled_at: ~U[2024-01-28 21:29:06.991971Z],
  conf: %Oban.Config{
    dispatch_cooldown: 5,
    engine: Oban.Engines.Basic,
    get_dynamic_repo: nil,
    insert_trigger: true,
    log: false,
    name: Oban,
    node: "DESKTOP-8JVA10V",
    notifier: {Oban.Notifiers.Postgres, []},
    peer: {Oban.Peers.Postgres, []},
    plugins: [{Oban.Plugins.Pruner, []}],
    prefix: "public",
    queues: [purchase_ticket: [limit: 10]],
    repo: Airline.Repo,
    shutdown_grace_period: 15000,
    stage_interval: 1000,
    testing: :disabled
  },
  conflict?: false,
  replace: nil,
  unique: nil,
  unsaved_error: nil
}

Enqueuing

Now, to start using our job, we need to enqueue it.

Job creation is rather simple. A Job is just an Ecto struct which gets enqueued by inserting it into the database. Oban workers provide a new/2 function which takes in a map of arguments (the same arguments that will be sent to the perform function) and converts it into an Oban.Job which can be inserted into the database using Oban.insert/1.

# purchasing ticket code
%{}
|> Airline.PurchaseTicketWorker.new()
|> Oban.insert()

Now, after adding this to our code which handles ticket purchasing, whenever someone purchases a ticket, the email confirmation task will be offloaded to a separate process and handled in the background. With this, the process of ticket purchasing will not be waiting on the email confirmation task to be finished, thus becoming faster to execute.

Another thing worth mentioning, when the need arises, as our application grows and we create multiple jobs, there will come a need to prioritize some jobs over the others. Usually, all jobs are executed in the order in which they were scheduled. But luckily, giving preference to one job over another is fairly simple, it can be defined while defining the queue in the Worker or we can give an individual job a priority when enqueueing it.

use Oban.Worker, queue: :purchase_ticket, priority: 3
Oban.insert(Airline.PurchaseTicketWorker.new(%{}, queue: :purchase_ticket, priority: 2))

Jobs with the highest priority have a value of 0, and the lowest priority comes to the jobs which have the value of 9.

Scheduling

But, what happens if we want to send an email to all users who purchased a ticket for a specific flight 24 hours before the flight, reminding them to check in?

Well, Oban also has a very useful feature which is called job scheduling. Again, a very intuitive name as it’s used to schedule jobs for the future. The syntax is very similar, we just need to specify when the job should execute, either with schedule_in which accepts seconds, or with scheduled_at which accepts a DateTime.

In our example, we would use scheduled_at and subtract 24 hours from the flight’s departure time.

Another important thing to mention is that job scheduling is always done in UTC. This means that we will have to shift our DateTime struct to use UTC.

%{}
|> Airline.PurchaseTicketWorker.new(scheduled_at:  DateTime.shift_zone!(day_before_departure_time, "Etc/UTC")

)
|> Oban.insert()

Periodic jobs

If we wanted to send a daily email digest to every subscriber of our airline newsletter, standard scheduling wouldn’t cut it. That’s where periodic jobs come in. Oban’s Cron plugin can schedule and enqueue jobs automatically. We declare periodic jobs either as {cron, worker} or {cron, worker, opts}.

So, to send out daily emails, the declaration would look something like this:

config :airline, Oban,
  repo: Airline.Repo,
  plugins: [
    {Oban.Plugins.Cron,
    crontab: [
      {"@daily", Airline.DailyWorker, max_attempts: 2}
    ]}
  ]

To set a more specific periodic job, for example, to have a job on every Monday or every hour, you can read about the Cron Syntax on the official Oban documentation.


“Background Jobs in Elixir – Oban” Tech Bite was brought to you by Semra Kermo, Junior Software Engineer at Atlantbh.

Tech Bites are tips, tricks, snippets or explanations about various programming technologies and paradigms, which can help engineers with their everyday job.

oban
Software DevelopmentTech Bites
February 23, 2024

Background Jobs in Elixir – Oban

When and why do we need background jobs? Nowadays, background job processing is indispensable in the world of web development. The need for background jobs stems from the fact that synchronous execution of time-consuming and resource-intensive tasks would heavily impact an application's  performance and user experience.  Even though Elixir is…
selenium
QA/Test AutomationTech Bites
December 22, 2023

Selenium Grid 4 with Docker

Introduction When talking about automation testing, one of the first things that comes to mind is Selenium. Selenium is a free, open-source automated testing framework used to validate web applications across different browsers and platforms. It is not just a single tool but a suite of software. Every component of…

Want to discuss this in relation to your project? Get in touch:

Leave a Reply