Avoiding Memory Leaks When Running Laravel Queue Workers

There has been a debate—for a long time—on whether or not PHP is a good choice for long-running processes. Based on my experience working on large scale projects that utilize hundreds of workers, I'm on the side of the debate that believes PHP is very efficient for getting the right job done.

The job of our workers here is running a Laravel application—PHP application—and get it to process background jobs. I've seen how powerful Laravel workers are with crunching a very large number of jobs on different server capacities and doing the job pretty well.

With that being said, avoiding memory leaks can still be a bit challenging. Over time, some references will pile up in the server memory that won't be detected by PHP and will cause the server to crash at some point.

The solution is easy though, restart the workers more often.

With a process manager like Supervisor in place, you can restart your workers every hour to clean the memory knowing that Supervisor will start them back for you automatically.

To do this, you can configure a CRON job to run every hour and call the queue:restart command:

0 * * * * forge php /home/forge/laravel.com/artisan queue:restart

Adding this to your /etc/crontab file will run queue:restart every hour. Workers will receive that signal and exit after finishing any job in hand.

If you don't want to use a CRON task, you can use the --max-jobs and --max-time options to limit the number of jobs the worker may process or the time it should stay up:

php artisan queue:work --max-jobs=1000 --max-time=3600

This worker will automatically exit after processing 1000 jobs or after running for an hour. After processing each job, the worker will check if it exceeded max-jobs or max-time and then decide between exiting or picking up more jobs. The worker will not exit while in the middle of processing a job.

Finally, you may signal the worker to exit from within a job handle method:

public function handle()
{
    // Run the job logic.

    app('queue.worker')->shouldQuit  = 1;
}

This method can be useful if you know this specific type of job could be memory consuming and you want to make sure the worker restarts after processing it to free any reserved resources.

Using Horizon

When using Laravel Horizon, you can restart the Horizon process using a CRON job or you can configure maxJobs and maxTime in your Horizon supervisor configurations:

'environments' => [
    'production' => [
        'supervisor-1' => [
            // ...
            'maxTime' => 3600,
            'maxJobs' => 1000
        ],
    ],
],

Occasionally restarting your workers is a very efficient strategy for dealing with memory leaks.

If you want to learn more about Laravel's queue system, make sure to check Laravel Queues in Action! I've put everything I know about the queue system in an eBook format along with several real-life uses cases. Check it out for a crash course, a cookbook, a guide, and a reference.