Laravel Logo

Work queues allow you to perform tasks that are slow or error-prone outside of the current user’s request to improve their experience with your site. Read on to learn how you can use work queues in Laravel for your projects.

Why You Should Be Using Work Queues

Two or three jobs ago (depending on how you look at it), I managed a help desk and the software we used to record tickets. We were using open source software that allowed us to email all of the participants of a ticket. The problem we continued to have is that every time we performed an action that sent an email, the application would open a connection to our email server, send one email, and then close the connection. Every save action emailed at least one person and sometimes it would email four or five. This caused the save action to take one to five seconds. Every so often the connection would fail and a portion of the emails would go out so we would have to save again and it would email out duplicates. This became such a problem that we eventually abandoned the software completely.

Two or one jobs ago (again depending on how you look at it), I helped write an application that had thousands of users per customer. At least once a year our customers would provide us with a CSV file with all their current employees in the file and we would add new users, remove old users, and make changes based on what the file contained. This process took an extremely long time and we kept having time out and memory usage issues with some of the larger customers.

Both of these examples illustrate times when processes take a long time or are error-prone and cause a poor user experience. Work queues allow us to tell our application that we need to do a unit of work but allows us to handle that unit of work outside the normal application process. This gives the users of our application a potentially huge speed improvement.

Queues, Jobs, and Workers

There are three concepts we need to quickly define so it’s easier to understand the rest of this tutorial.

The first concept is a work queue. A queue is simply a list of items where you put items into the queue and remove them in the same order (First In/First Out or FIFO). Work queues provide a central location for us to store the units of work we want to be done.

The second concept is a job. In Laravel jobs are the classes that are performing the unit for work we want to be done.

The third concept is a worker. A worker is a process generally on the same server as the main application (it’s not a requirement which is great for scaling) that performs the unit of work. Laravel provides a simple interface for how to get a worker to run our jobs.

Queue Drivers

For work queues, a queue driver is what we’re using to store the jobs we need done. Out of the box, Laravel provides support for two different types of queue drivers. The easiest solution is using your existing database as the queue (which is the solution we’ll be using in this article). The other option is using a Redis server to store the data. Redis is a NoSQL solution that can handle queue like data exceptionally well. We highly recommend that as your application’s usage goes up you look into using Redis as the database can quickly become a bottleneck.

Creating Your First Job

The first thing we need to do is create our first job. We’ve been developing an example application in this series that has Projects that have Tasks. The next feature we need to add is the ability to add a task to every project that is not completed.

To create our job we’re going to use “artisan’s” make:job command:

php artisan make:job AddTaskToNotCompletedProjects
Job created successfully.

This will create a new class in “app/Jobs/AddTaskToNotCompletedProjects.php” that will look like the following:

<?php

namespace App\Jobs;

use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;

class AddTaskToNotCompletedProjects implements ShouldQueue
{
    use Dispatchable;
    use InteractsWithQueue;
    use Queueable;
    use SerializesModels;

    /**
     * Create a new job instance.
     *
     * @return void
     */
    public function __construct()
    {
        //
    }

    /**
     * Execute the job.
     *
     * @return void
     */
    public function handle()
    {
        //
    }
}

The important part of this class is the handle() function. The handle() function is what does the actual work in our job. To get us started we’re going to create a simple function that loops through all the open Projects (based on our logic that a null “end_date” indicates not completed) and adds a new Task to the Project.

public function handle()
{
    Project::whereNull('end_date')->get()->each(function(Project $project) {
        Task::create([
            'name' => 'Scheduled Task',
            'project_id' => $project->id,
        ]);
    });
}

Then to test this, we’re going to short circuit the whole queue complexity and add the following in our “routes/web.php” file.

\App\Jobs\AddTaskToNotCompletedProjects::dispatchNow();

The call to dispatchNow() causes the job to run inside the current request. We don’t recommend doing this in a production environment but it’s great for testing.

Now if we access our application the job will run and we can check the results by looking into the “tasks” table.

mysql> select * from tasks order by id desc limit 10;
+-----+----------------+-----------+----------------+------------+---------------------+---------------------+------------+
| id  | name           | completed | completed_date | deleted_at | created_at          | updated_at          | project_id |
+-----+----------------+-----------+----------------+------------+---------------------+---------------------+------------+
| 404 | Scheduled Task |         0 | NULL           | NULL       | 2020-01-15 01:57:51 | 2020-01-15 01:57:51 |        300 |
| 403 | Scheduled Task |         0 | NULL           | NULL       | 2020-01-15 01:57:51 | 2020-01-15 01:57:51 |        299 |
| 402 | Scheduled Task |         0 | NULL           | NULL       | 2020-01-15 01:57:51 | 2020-01-15 01:57:51 |        298 |
| 401 | Scheduled Task |         0 | NULL           | NULL       | 2020-01-15 01:57:51 | 2020-01-15 01:57:51 |        297 |
| 400 | Scheduled Task |         0 | NULL           | NULL       | 2020-01-15 01:57:51 | 2020-01-15 01:57:51 |        296 |
| 399 | Scheduled Task |         0 | NULL           | NULL       | 2020-01-15 01:57:51 | 2020-01-15 01:57:51 |        295 |
| 398 | Scheduled Task |         0 | NULL           | NULL       | 2020-01-15 01:57:51 | 2020-01-15 01:57:51 |        294 |
| 397 | Scheduled Task |         0 | NULL           | NULL       | 2020-01-15 01:57:51 | 2020-01-15 01:57:51 |        293 |
| 396 | Scheduled Task |         0 | NULL           | NULL       | 2020-01-15 01:57:51 | 2020-01-15 01:57:51 |        292 |
| 395 | Scheduled Task |         0 | NULL           | NULL       | 2020-01-15 01:57:51 | 2020-01-15 01:57:51 |        291 |
+-----+----------------+-----------+----------------+------------+---------------------+---------------------+------------+
10 rows in set (0.00 sec)

See how easy that was?

Specifying the “Name”

The downside to this is that all of our tasks have a name of “Scheduled Task”. We could create one job for every type of task we want to add but that would be a maintenance nightmare. Thankfully, that empty __construct() function can be used to set parameters for our job.

We’re going to create a parameter named $name which will allow us to specify the task name at runtime.

private $name;
public function __construct(string $name)
{
    $this->name = $name;
}

public function handle()
{
    Project::whereNull('end_date')->get()->each(function(Project $project) {
        Task::create([
            'name' => $this->name,
            'project_id' => $project->id,
        ]);
    });
}

Now when we call dispatchNow() we can pass it a string which will be passed directly to __construct():

\App\Jobs\AddTaskToNotCompletedProjects::dispatchNow("Send Weekly Status Update to Team");

Now if we check our “tasks” table we can see that our name is being set correctly.

mysql> select * from tasks order by id desc limit 10;
+-----+-----------------------------------+-----------+----------------+------------+---------------------+---------------------+------------+
| id  | name                              | completed | completed_date | deleted_at | created_at          | updated_at          | project_id |
+-----+-----------------------------------+-----------+----------------+------------+---------------------+---------------------+------------+
| 704 | Send Weekly Status Update to Team |         0 | NULL           | NULL       | 2020-01-15 02:04:09 | 2020-01-15 02:04:09 |        300 |
| 703 | Send Weekly Status Update to Team |         0 | NULL           | NULL       | 2020-01-15 02:04:09 | 2020-01-15 02:04:09 |        299 |
| 702 | Send Weekly Status Update to Team |         0 | NULL           | NULL       | 2020-01-15 02:04:09 | 2020-01-15 02:04:09 |        298 |
| 701 | Send Weekly Status Update to Team |         0 | NULL           | NULL       | 2020-01-15 02:04:09 | 2020-01-15 02:04:09 |        297 |
| 700 | Send Weekly Status Update to Team |         0 | NULL           | NULL       | 2020-01-15 02:04:09 | 2020-01-15 02:04:09 |        296 |
| 699 | Send Weekly Status Update to Team |         0 | NULL           | NULL       | 2020-01-15 02:04:09 | 2020-01-15 02:04:09 |        295 |
| 698 | Send Weekly Status Update to Team |         0 | NULL           | NULL       | 2020-01-15 02:04:09 | 2020-01-15 02:04:09 |        294 |
| 697 | Send Weekly Status Update to Team |         0 | NULL           | NULL       | 2020-01-15 02:04:09 | 2020-01-15 02:04:09 |        293 |
| 696 | Send Weekly Status Update to Team |         0 | NULL           | NULL       | 2020-01-15 02:04:09 | 2020-01-15 02:04:09 |        292 |
| 695 | Send Weekly Status Update to Team |         0 | NULL           | NULL       | 2020-01-15 02:04:09 | 2020-01-15 02:04:09 |        291 |
+-----+-----------------------------------+-----------+----------------+------------+---------------------+---------------------+------------+
10 rows in set (0.00 sec)

Getting Started With A Database Queue

We have our job defined now but we still haven’t handled the whole problem of it running in the background.

The first thing we need to do is create the tables that Laravel will use to store the job information. This is done by running php artisan queue:table and php artisan queue:failed-table to generate the migrations that will be used.

php artisan queue:table
Migration created successfully!
php artisan queue:failed-table
Migration created successfully!

And then we’re going to run php artisan migrate to get the tables created.

$ php artisan migrate
Migrating: 2020_01_15_014020_create_jobs_table
Migrated:  2020_01_15_014020_create_jobs_table (0.03 seconds)
Migrating: 2020_01_15_014316_create_failed_jobs_table
Migrated:  2020_01_15_014316_create_failed_jobs_table (0.01 seconds)

Next, we need to open our “.env” file and set “QUEUE_CONNECTION” to “database” so Laravel will know to save the jobs to the database tables we just created.

To get our job into the queue we’re going to run the following command. Please note we changed dispatchNow() to dispatch().

\App\Jobs\AddTaskToNotCompletedProjects::dispatch("Send Weekly Status Update to Team");

Now we can look into the “jobs” table and see that our job is there (the “payload” column contains so much data we removed the contents of the column from this article).

mysql> select * from jobs;
+----+---------+---------+----------+-------------+--------------+------------+
| id | queue   | payload | attempts | reserved_at | available_at | created_at |
+----+---------+---------+----------+-------------+--------------+------------+
|  1 | default | <snip>  |        0 |        NULL |   1579054121 | 1579054121 |
+----+---------+---------+----------+-------------+--------------+------------+
1 row in set (0.00 sec)

Managing Your Workers

Workers are the final component in this process we need to discuss. There are two ways of running your workers.

Please note that we have "database" at the end of some of these commands. This tells Laravel which driver we're using.

The first way is to use php artisan queue:listen.

ubuntu@ubuntu-xenial:/var/www$ php artisan queue:listen database
[2020-01-15 02:10:32][1] Processing: App\Jobs\AddTaskToNotCompletedProjects
[2020-01-15 02:10:33][1] Processed:  App\Jobs\AddTaskToNotCompletedProjects

This method works well in cases where you’re changing the source code files between jobs a lot but don’t have a lot of jobs running. This is because in this mode the jobs are run so it looks for changes in files but the downside is that it’s less performant. We recommend running your workers in this mode when you’re developing your application.

The second way is to use php artisan queue:work.

ubuntu@ubuntu-xenial:/var/www$ php artisan queue:work database
[2020-01-15 02:12:02][2] Processing: App\Jobs\AddTaskToNotCompletedProjects
[2020-01-15 02:12:03][2] Processed:  App\Jobs\AddTaskToNotCompletedProjects

This has better performance in comparison to queue:listen but doesn’t notice changes to files when it’s running. To get around this you’ll need to use php artisan queue:restart to force the workers to reset themselves whenever we deploy a new version of our application.

ubuntu@ubuntu-xenial:/var/www$ php artisan queue:restart 
Broadcasting queue restart signal.

Manually Managing Your Failed Jobs

One of the benefits of all this extra work was that it was supposed to be easier for us to handle failures more gracefully. Let’s discuss what happens when a job fails and we need to intervene.

Let’s say we accidentally changed ‘name’ to ‘name2’ in our handle() function so it fails to write to the database.

public function handle()
{
    Project::whereNull('end_date')->get()->each(function(Project $project) {
        Task::create([
            'name2' => $this->name,
            'project_id' => $project->id,
        ]);
    });
}

Now when our worker tries to run through the jobs it has it’s going to fail.

ubuntu@ubuntu-xenial:/var/www$ php artisan queue:listen database
[2020-01-15 02:14:11][3] Processing: App\Jobs\AddTaskToNotCompletedProjects
[2020-01-15 02:14:11][3] Failed:     App\Jobs\AddTaskToNotCompletedProjects
[2020-01-15 02:14:12][4] Processing: App\Jobs\AddTaskToNotCompletedProjects
[2020-01-15 02:14:12][4] Failed:     App\Jobs\AddTaskToNotCompletedProjects
[2020-01-15 02:14:13][5] Processing: App\Jobs\AddTaskToNotCompletedProjects
[2020-01-15 02:14:13][5] Failed:     App\Jobs\AddTaskToNotCompletedProjects
[2020-01-15 02:14:14][6] Processing: App\Jobs\AddTaskToNotCompletedProjects
[2020-01-15 02:14:14][6] Failed:     App\Jobs\AddTaskToNotCompletedProjects
[2020-01-15 02:14:15][7] Processing: App\Jobs\AddTaskToNotCompletedProjects
[2020-01-15 02:14:15][7] Failed:     App\Jobs\AddTaskToNotCompletedProjects
[2020-01-15 02:14:16][8] Processing: App\Jobs\AddTaskToNotCompletedProjects
[2020-01-15 02:14:16][8] Failed:     App\Jobs\AddTaskToNotCompletedProjects
[2020-01-15 02:14:17][9] Processing: App\Jobs\AddTaskToNotCompletedProjects
[2020-01-15 02:14:17][9] Failed:     App\Jobs\AddTaskToNotCompletedProjects

All of these failed jobs get moved into the “failed_jobs” table. We recommend adding something to your application that surfaces these errors so you’re aware of them. We generally display them on a “systems” dashboard so we can intervene.

To manually check the failed jobs you could do a “select” query through your MySQL client of choice or you can use the queue:failed command to see them.

ubuntu@ubuntu-xenial:/var/www$ php artisan queue:failed
+----+------------+---------+----------------------------+---------------------+
| ID | Connection | Queue   | Class                      | Failed At           |
+----+------------+---------+----------------------------+---------------------+
| 7  | database   | default | App\Jobs\AddTaskToNotCompletedProjects | 2020-01-15 02:14:17 |
| 6  | database   | default | App\Jobs\AddTaskToNotCompletedProjects | 2020-01-15 02:14:16 |
| 5  | database   | default | App\Jobs\AddTaskToNotCompletedProjects | 2020-01-15 02:14:15 |
| 4  | database   | default | App\Jobs\AddTaskToNotCompletedProjects | 2020-01-15 02:14:14 |
| 3  | database   | default | App\Jobs\AddTaskToNotCompletedProjects | 2020-01-15 02:14:13 |
| 2  | database   | default | App\Jobs\AddTaskToNotCompletedProjects | 2020-01-15 02:14:12 |
| 1  | database   | default | App\Jobs\AddTaskToNotCompletedProjects | 2020-01-15 02:14:11 |
+----+------------+---------+----------------------------+---------------------+

The ID column is important here because it’s what we’re going to use to interact with these jobs. It’s also important to notice that the ID here is different than the number in brackets above when the job fails. That is because the first number is the id column from the “jobs” table and the second is the id column from the “failed_jobs” table.

Let’s say we determine job #1 isn’t important because job 2 did the same work so repeating it would be wasteful. We can tell Laravel to delete (or forget) the job by using the queue:forget command which removes the job from the “failed_jobs” table.

ubuntu@ubuntu-xenial:/var/www$ php artisan queue:forget 1
Failed job deleted successfully!
ubuntu@ubuntu-xenial:/var/www$ php artisan queue:failed
+----+------------+---------+----------------------------+---------------------+
| ID | Connection | Queue   | Class                      | Failed At           |
+----+------------+---------+----------------------------+---------------------+
| 7  | database   | default | App\Jobs\AddTaskToNotCompletedProjects | 2020-01-15 02:14:17 |
| 6  | database   | default | App\Jobs\AddTaskToNotCompletedProjects | 2020-01-15 02:14:16 |
| 5  | database   | default | App\Jobs\AddTaskToNotCompletedProjects | 2020-01-15 02:14:15 |
| 4  | database   | default | App\Jobs\AddTaskToNotCompletedProjects | 2020-01-15 02:14:14 |
| 3  | database   | default | App\Jobs\AddTaskToNotCompletedProjects | 2020-01-15 02:14:13 |
| 2  | database   | default | App\Jobs\AddTaskToNotCompletedProjects | 2020-01-15 02:14:12 |
+----+------------+---------+----------------------------+---------------------+
ubuntu@ubuntu-xenial:/var/www$

We could also determine we want to have the workers retry the jobs using the queue:retry command. In this case, we’ll retry #2 but we could have given it a range like 2-6 to retry more than 1 or queue:retry all to retry all of them.

ubuntu@ubuntu-xenial:/var/www$ php artisan queue:retry 2
The failed job [2] has been pushed back onto the queue!

Finally, if we determine none of the jobs are worth keeping we can queue:flush the jobs out.

ubuntu@ubuntu-xenial:/var/www$ php artisan queue:flush
All failed jobs deleted successfully!

Automatic Retry

The huge downside to the process above is that it’s a manual process. Laravel provides two levers that allow us to tweak how it automatically retries failed jobs. The first is the $tries variable which determines how many times it should attempt to do the job. The second is the $retryAfter variable which determines how much time between each of the tries. We recommend setting a $retryAfter on all of your jobs that attempt to do something with an external service (send an email, connect to an API, etc.) because usually if the service is having a problem it needs a few minutes to fix itself.

For example, let’s say we’ve determined we want to try our AddTaskToNotCompletedProjects task 3 times and wait 10 seconds between the attempts. We would set the following variables inside our AddTaskToNotCompletedProjects class.

public $tries = 3;
public $retryAfter = 10;

Then when the job fails (maybe when we still haven’t fixed that ‘name2’ issue) we can see that it tries three times with 10 seconds between the attempts before it fails.

vagrant@ubuntu-xenial:/var/www$ php artisan queue:work database
[2020-02-10 01:34:37][5] Processing: App\Jobs\AddTaskToNotCompletedProjects
[2020-02-10 01:34:47][6] Processing: App\Jobs\AddTaskToNotCompletedProjects
[2020-02-10 01:34:57][7] Processing: App\Jobs\AddTaskToNotCompletedProjects
[2020-02-10 01:34:57][7] Failed:     App\Jobs\AddTaskToNotCompletedProjects

More Queues

Another thing that we haven’t covered is that right now all of the jobs are being handled by a single queue. While this does work, there are benefits to defining multiple queues. We generally recommend that you define one queue per workload area so that you can more easily determine if you need to spin up extra workers for that type of job.

To start we generally create:

  • One queue for every external service
  • One queue for scheduled tasks
  • One queue to handle reports

The default queue is named “default” (shocker I know), to place a job on a different queue you need to call the onQueue() function after the dispatch() queue.

\App\Jobs\AddTaskToNotCompletedProjects::dispatch("This is a test2")->onQueue('tasks');

If we look at the “queue” column in the “jobs” table we can see that it’s now getting a different name than “default”.

mysql> select queue from jobs;
+---------+
| queue   |
+---------+
| tasks   |
| default |
| tasks   | 
+---------+
3 rows in set (0.00 sec)

Now if we run our worker as we have above the jobs on the “tasks” queue won’t be run. This is because by default the worker only looks for jobs in the “default” queue. To fix this we need to specify the name of the queue(s) we want to have it run jobs from using the --queue command line flag.

vagrant@ubuntu-xenial:/var/www$ php artisan queue:work --queue=tasks

Again, we recommend at least one worker per work queue but sometimes you can get away with running a single worker for your test environment. To get this to work you’ll need to specify all the queues it should be pulling from.

To make this example a little easier to follow we changed our handle() function to look like the following.

public function handle()
{
    echo "Using queue {$this->queue}", PHP_EOL;
    // work here
}

Now we’re going to specify the queues that we want this worker to process in a comma-separated list.

vagrant@ubuntu-xenial:/var/www$ php artisan queue:work --queue=tasks,default
[2020-02-11 00:47:15][17] Processing: App\Jobs\AddTaskToNotCompletedProjects
Using queue tasks
[2020-02-11 00:47:15][17] Processed:  App\Jobs\AddTaskToNotCompletedProjects
[2020-02-11 00:47:15][19] Processing: App\Jobs\AddTaskToNotCompletedProjects
Using queue tasks
[2020-02-11 00:47:15][19] Processed:  App\Jobs\AddTaskToNotCompletedProjects
[2020-02-11 00:47:15][18] Processing: App\Jobs\AddTaskToNotCompletedProjects
Using queue 
[2020-02-11 00:47:15][18] Processed:  App\Jobs\AddTaskToNotCompletedProjects

There are two things to notice in this output. The first is that the worker processed all the “tasks” queue jobs before it processed the “default” queue jobs. This is because of the order we passed the queue names to the artisan command determines in what order the worker checks each queue for jobs. The second is that the queue name is blank for the “default” queue.

In Closing

Work queues are excellent solutions to perform work that is error-prone or time-intensive. Laravel provides an excellent interface for creating queues, jobs, and workers for your application.

In our next post, we’ll discuss how to schedule our AddTaskToNotCompletedProjects so it runs every week.

Hopefully, this has been as helpful to you as it has to me. Check back soon for more artisan commands.