Concurrent processing in PHP

I occasionally run in to procedures that could really benefit from some form of concurrent processing. The most recent example was a script that created thumbnail images from a folder of large images but there have been many many others. Sometimes we’re using a lower level language that is geared towards multi-threading, which is great. A lot of the time we’ve thrown together a quick script then later discovered it takes ages if one of the images is super high resolution.

Rewriting the whole thing in a new language can mean learning new APIs and be very time consuming. If only there was a way to bodge in some concurrent processing to a PHP script. A lot of places tell you this is not possible for various reasons but in fact is is, well sort of. You can’t really do threads so have to resort to using multiple processes which can be slow depending on the use case. There are a few more requirements; you need the process control extension and to be able to execute shell commands (almost certainly not possible with shared web hosting)

Instead of explaining this one I’ll just post the, heavily commented, helper function I wrote to process a list of tasks using a maximum number of processes until it’s all done.

/**
 * Processes a list of tasks using a specifid number of processes.
 *  
 * @param array $tasks The task list to run. Each element in the array should contain a callback and an array of parameters to pass to it.
 * @param int $max_processes The maximum number of processes allowed to exist at once.
 */
function process_task_queue($tasks, $max_processes = 10){
    $total_tasks = count($tasks);
   
    // Initially populate the process queue with the first few tasks on the list.
    for ($i = 0; $i < $total_tasks && $i < $max_processes; ++$i){
        // Pick the first task in the list.
        $task = array_shift($tasks);
        $pid = pcntl_fork();
       
        if (!$pid){
            // Here we are the child process.
            call_user_func_array($task[0], $task[1]);
            die(); // Otherwise the child process continues the for loop.
        }
    }
   
    while (($pid = pcntl_waitpid(0, $status)) != -1){
        // Here we know that one of the child processes just ended.
       
        // Add another task if there are some left.
        if (!empty($tasks)){
            // Pick the first task in the list.
            $task = array_shift($tasks);
            $pid = pcntl_fork();
           
            if (!$pid){
                // Here we are the child process.
                call_user_func_array($task[0], $task[1]);
                die(); // Otherwise the child process continues the while loop.
            }
        }
    }
}

and an example use

function test_task($input){
    usleep(mt_rand(0, 3000000)); // Interesting quirk: using rand() here generates the same random number in all processes.
    echo 'Input: ', $input, "\n";
    echo 'Time: ', time(), "\n";
}
 
for ($i = 0; $i < 100; ++$i){
    // Build a task list to call test_task(rand()) in each process.
    $task_list[] = array('test_task', array(rand()));
}
 
// Run the task list, allowing no more than N processes to exists at any one time.
process_task_queue($task_list, 8);

The process_task_queue() function also blocks until all tasks have been completed which can be handy.