Concurrent background jobs in elixir

One of the most (if not the most) effective thing when it comes to increasing performance is to do less work. When it comes to regular applications there are loads of operations that doesn’t have to finish before the server answer a request from the user. Some of these things includes sending emails, tracking data, generating pdfs and much more.

In many platforms and environments this can get quite complex, you’ll have to first store your jobs somewhere. Normally a simple database, redis or rabbit is used for this. Once the job is stored there needs to be something trying to execute the job, that can easily be achieved with a cronjob or a service that imitates the behaviour. Normally when executing the job everything succeeds and the job is done. However in the case where a job fails, what then? When the job is performed in a transaction like behavior the most sane thing would be to just retry it for a couple of times just to make sure the job isn’t failing because of an issue in an external service. In the case the queue of jobs isn’t executed in paralell retrying jobs could lead to latencies and all of this is something that you’d have write code to handle. Scared yet?

Elixir & OTP to the rescue

With OTP’s focus on concurrency and fault tolerance we can easily implement a concurrent background job queue in minutes, without any external dependencies and just let the BEAM handle any issues that we’d otherwise had to take care of ourselves.

We’ll do basic example by creating supervised tasks with Task.Supervisor and Task.

First we need to start the task supervisor by adding it into our application with the name MyApp.BackgroundTaskSupervisor. The restart: :transient means that the job should be retried if it did not finish correctly.

children = [
    ...
    {Task.Supervisor, [name: MyApp.BackgroundTaskSupervisor, restart:
:transient]}
]

Now whenever we want to do create a job to send an welcome email with the EmailSender module we can simply do it by telling our supervisor to execute a function like this:

{:ok, _pid} = Task.Supervisor(MyApp.BackgroundTaskSupervisor, EmailSender,
:send_welcome_email, ["myemail@mycompany.com"])

…and we have a background job that will run concurrently and restart whenever it fails.

Closing words

As demonstrated the Task.Supervisor in elixir is very powerful and can in many cases save you from external dependencies or memory intensive services. However it would be recommended to have a look at one of the many great background job libraries before taking this example and putting it in production. In the case where your server providers newly divorced technician has a bad day and accidently pulls your servers powercable it would be good if the jobs were persisted and not lost in oblivion.