Articles Go HTTP Servers for Hackers Courses CloudCasts Chipper CI

Production-Ready Beanstalkd with Laravel 4 Queues

Note: TL;DR version at the bottom!

Queues are a great way to take some task out of the user-flow and put them in the background. Allowing a user to skip waiting for these tasks makes our applications appear faster, and gives us another opportunity to segment our application and business logic out further.

For example, sending emails, deleting accounts and processing images are all potentially long-running or memory-intensive tasks; They make great candidates for work which we can off-load to a queue.

Laravel can accomplish this with its Queue package. Specifically, I use the Beanstalkd work queue with Laravel.

Here's how I set that up to be just about production-ready.

Note: I use Ubuntu for development and often in production. The following is accomplishsed in Ubuntu 12.04 Server LTS. Some instructions may differ for you depending on your OS

Here's what we'll cover:

  1. Laravel (~4.2 - 4.3) and Queues
  2. Installing Beanstalkd
  3. Churning through the queue with Supervisor

Laravel and Queues

Laravel makes using queues very easy. Our application, the "producer", can simply run something like Queue::push('SendEmail', array('message' => $message)); too add a "job" to the queue.

On the other end of the queue is the code listening for new jobs and a script to process the job (collectively, the "workers"). This means that in addition to adding jobs to the queue, we need to set up a worker to pull from the queue of available jobs.

Here's how that looks in Laravel. In this example, we'll create an image-processing queue.

Install dependencies

As noted in the docs, Laravel requires the Pheanstalk package for using Beanstalkd. We can install this using Composer:

$ cd /path/to/laravel/application
$ composer require pda/pheanstalk ~2.0

Create a script to process it

Once our PHP dependency in installed, we can begin to write some code. In this example, we'll create a PhotoService class to handle the processing. If no method is specified, laravel assumes the class will have a fire() method. This is half of a worker - the code which does some processing.

<?php namespace Myapp\Queue;

class PhotoService {

    public function fire($job, $data)
    {
        // Minify, crop, shrink, apply filters or otherwise manipulate the image
    }

}

This repository contains working (last I checked :P) code for an image processing queue job.

Push a job to a Queue

When a user uploads an image, we'll add a job to the queue so our worker can process it.

In Laravel, we'll create a job by telling the Queue library what code will handle the job (in this case the fire() method inside of Myapp\Queue\PhotoService as defined above) and give it some data to work with. In our example, we simply pass it a path to an image file.

Queue::push('Myapp\Queue\PhotoService', array('image_path' => '/path/to/image/file.ext'));

Process the jobs

At this point, we have code to process an image (most of a worker), and we've added a job to the queue. The last step is to have code pull a job from the queue.

This is the other half of a worker. The worker needs to both pull a job from the queue and do the processing. In Laravel, that's split into 2 functionalities - Laravel's queue listener, and the code we write ourselves - in this case, the PhotoService.

Laravel has some CLI tools to help with queues:

// Fire the latest job in the queue
$ php artisan queue:work

// Listen for new jobs in the queue
// and fire them off one at a time
// as they are created
$ php artisan queue:listen

You'll note that the queue:work command only processes the one next job in the queue and quits. As of Laravel 4.3, we can add the --daemon flag to this, which tells Laravel to load in the code and then process jobs. This means the framework isn't re-loaded in between jobs, saving CPU and time.

We'll use the following command in production:

$ php artisan queue:work --daemon [..other flags..]

When not working with the "sync" driver, these tools are what you need to use in order to process the jobs in your queue. We run the queue:listen or queue:work --daemon command to have laravel listen to the queue and pull jobs as they become available.

Let's install Beanstalkd to see how that works.

By default, laravel will run queue jobs synchronously - that is, it runs the job at the time of creation. This means the image will be processed in the same request that the user created when uploading an image. That's useful for testing, but not for production. We'll make this asynchronous by introducing Beanstalkd.

Beanstalkd

Let's install Beanstalkd:

# Debian / Ubuntu:
$ sudo apt-get update
$ sudo apt-get install beanstalkd

Note: You may be able to get a newer version of Beanstalkd by adding this PPA. Ubuntu 12.04 installs an older version of Beanstalkd.

Next, some quick configuration. The first thing we need to do is tell Beanstalkd to start when the system starts up or reboots. Edit /etc/default/beanstalkd and set START to "yes".

$ sudo vim /etc/default/beanstalkd
> START yes     # uncomment

Then we can start Beanstalkd:

$ sudo service beanstalkd start

Now we can setup Laravel. In your app/config/queue.php file, set the default queue to 'beanstalkd':

'default' => 'beanstalkd',

Then edit any connection information you need to change. I left my configuration with the defaults as I installed it on the same server as the application.

'connections' => array(
    
    'beanstalkd' => array(
        'driver' => 'beanstalkd',
        'host'   => 'localhost', 
        'queue'  => 'default',
        'ttr'    => 60,
    ),
    
),

Now when we push a job to the queue in Laravel, we'll be pushing to Beanstalkd!

Installing Beanstalkd on a remote server

You may (read: should) want to consider installing Beanstalkd on another server, rather than your application server. Since Beantalkd is an in-memory service, it can eat up your servers resources under load.

To do this, you can install Beanstalkd on another server, and simply point your "host" to the proper server address, rather than localhost.

This leaves the final detail - what server runs the job? If you follow all other steps here, Supervisord will still be watching Laravel's listener on your application server. You may want to consider running your job script (or even a copy of your application which has a job script) on yet another server whose job is purely to churn through Beanstalkd queue jobs. This means having a listener and working listener/job code on yet another server.

In fact, in a basic distributed setup, we'd probably have an application server (or 2, plus a load-balancer), a database server, a queue server and one or more job servers!

Supervisord

Let's say you pushed a job to Beanstalkd:

Queue::push('Myapp\Queue\PhotoService', array('image_path' => '/path/to/image/file.ext'));

Now what? You might notice that it goes to Beanstalkd, but Myapp\Queue\PhotoService@fire() doesn't seem to be getting called. You've checked your error logs, you see if the image was edited, and found that the the job is just "sitting there" in your Beanstalkd queue.

Beanstalkd doesn't actually PUSH jobs to a script - instead, we need a worker to check if there are jobs available and ask for them.

This is what $ php artisan queue:work --daemon does - It listens for jobs and runs them as they become available.

If you run that command, you'll see your job being sent to code. If all goes well, your image will be properly manipulated.

The question then becomes: How do I make php listen at all times? We need to avoid having to "supervise" that process manually. This is where Supervisord comes in.

Supervisord will watch our queue:work --daemon command and restart it if it fails. Let's see how to set that up.

First, we'll install it:

# Debian / Ubuntu:
$ sudo apt-get install supervisor

Next, we'll configure it. We need to define a process to listen to.

$ sudo vim /etc/supervisor/conf.d/myqueue.conf

Add this to your new conf file, changing file paths and your environment as necessary:

[program:myqueue]
command=php artisan queue:work --daemon --env=your_environment
directory=/path/to/laravel
stdout_logfile=/path/to/laravel/app/storage/logs/myqueue_supervisord.log
redirect_stderr=true
autostart=true
autorestart=true

Setting your environment correctly here (via --env) is very important. Otherwise you'll likely not get the right configuration for your environment.

We now have a process called "myqueue" which we can tell Supervisord to start and monitor.

Let's do that:

$ sudo supervisorctl
> reread # Tell supervisord to check for new items in /etc/supervisor/conf.d/
> add myqueue       # Add this process to Supervisord
> start myqueue     # May say "already started"

Now the "myqueue" process is on and being monitored. If our queue listener fails, Supervisord will restart the php artisan queue:listen --env=your_environment process.

You can check that it is indeed running that process with this command:

$ ps aux | grep php

# You should see some output similarish this:
php artisan queue:work --daemon --env=your_environment
sh -c php artisan queue:work  --queue="default" --delay=0 --memory=128 --sleep --env=your_environment
php artisan queue:work --queue=default --delay=0 --memory=128 --sleep --env=your_environment

Wrapping up

Now we have a full end-to-end queue working and in place!

  1. We create a script to process a queued job
  2. We installed Beanstalkd to act as the work queue
  3. We use Laravel to push jobs to our queue
  4. We use Laravel queue:listen to act as a worker and pull jobs from the queue
  5. We wrote some code to process a job from the queue
  6. We use Supervisord to ensure queue:listen is always listening for new jobs

Troubleshooting

Common issues seen involve:

  1. Not setting the --env option, which may cause your queue workers to not use correct API keys or database credentials
  2. Ensuring Supervisord and Beanstalkd are running. Check with: sudo service supervisord status and sudo service beanstalkd status
  3. Not checking for log output for unexplainable issues. Check your laravel output log for exceptions, and check beanstalk/supervisor logs, likely found in the /var/log directory of your server. Perhaps /var/log/supervisor and /var/log/beanstalkd.

Notes

  1. You might want to consider setting up log rotation on the Laravel and Supervisord logs
  2. You can read here for more information on setting up Supervisord on Ubuntu.
  3. Read the Laravel docs on queues to learn how and when to release or delete jobs.

 

Lastly, DEFINITELY use Beanstalkd Console, which provides a web interface for seeing and interacting with jobs in your Beanstalkd queue.

More

I cover Supervisord and other portions of this article in a lot more detail in my eBook Servers for Hackers.

If you're interested in learning more about the server-side of application development, definitely check it out!

TL;DR

For reference, just copy and paste the whole process from here:

$ sudo apt-get update
$ sudo apt-get install -y beanstalkd supervisor
$ sudo vim /etc/default/beanstalkd
> START yes     # uncomment this line
$ sudo service beanstalkd start
$ sudo vim /etc/supervisor/conf.d/myqueue.conf

Enter this, changing as needed:

[program:myqueue]
command=php artisan queue:work --daemon --env=your_environment
directory=/path/to/laravel
stdout_logfile=/path/to/laravel/app/storage/logs/myqueue_supervisord.log
redirect_stderr=true
autostart=true
autorestart=true

Start Supervisord:

$ sudo supervisorctl
> reread                # Get available jobs
> add myqueue
> start myqueue

Read more on Supervisord here for info on supervisorctl.