Fork()s for Thanksgiving

A long time ago I inherited some Perl code to take care of.  It's good code, it doesn't stay out late, cleans up after itself most of the time, and does things pretty well without a lot of fanfare.  Unfortunatly, that was also a big drawback.

The code was originally written when a "big system" was one with 512MB RAM and two CPUs (note not "two cores" - we're talking two big CPU chips on the motherboard).  This code would process incoming data one at a time, storing the data back to a directory structure on the disk.  The incoming data was all contained in a single archive file, and the resulting data stored to disk was mostly self contained too.  As time went by, the code never had to speed up, thankfully Moore's Law kept increasing the speed of the CPUs as the size and volume of incoming data increased.  That was, until a customer upgraded to a new quad-core CPU with gobs of RAM and lots of fast hard drives.  Everything about the new system was easily eight times the old system; four cores running at 1.2GHz, up from 600MHz, 16 GB RAM, up from 2GB, and a fast SAN drive for the data and not the old SCSI internal disks.  Yes, it was a big upgrade in all the key areas except one: the old code.

Don't get me wrong, the old code ran perfect on the new system - but the rate it was processing the incoming data was barely twice the old system.  And with the projected growth of the data center, they were expecting to increase the incoming data by ten times over the next 18 months.  Someting had to be done - Moore had got us this far, but it was time for a change.

As I described earlier, the data is really a best case scenario for parallel processing.  In only one function does the data from one incoming set ever need to interact with pre-existing data.  I set about learning as much as I could about the fork() subroutine in Perl.

For those who haven't dealt with it, fork() allows your program to clone itself in memory and have two running copies.  It's really easy to create a "fork bomb" to bring down some systems if the programer isn't careful.  Just remember that right after the fork() subroutine is called, there are two identical copies of your code running in memory.  It's up to the programmer to add code right after the fork() to ensure that each copy knows what it's role is.

For anyone wanting some boilerplate fork() code to play with, here's a bit of code that I wrote to demonstrate this:

#!/usr/bin/perl
use strict;
my @array = qw(AA BB CC DD EE FF GG);
my $sleep = 10;
my %children;

for my $A (0..scalar(@array)-1) {
        my $pid = fork();
        if ($pid) {
                # parent
                $children{$pid}=@array[$A];
        } elsif ($pid == 0) {
                # child
                my $X = $sleep*rand();
                my $now = localtime();
                printf "$now Executing %s for %5.3f seconds.\n",@array[$A],$X;
                sleep $X;
                exit(0);
        } else {
                die "couldn't fork: $!\n";
        }
}

my $exited;
while (($exited = wait()) && ($exited > 0 )) {
        my $now = localtime();
        printf "$now EXITED: $exited(%s)", $children{$exited};
        delete $children{$exited};
        if (scalar %children > 0 ) {
                printf ", waiting for";
                foreach my $B (sort keys(%children)) {
                        printf ": %5i(%s) ",$B, $children{$B};
                }
        }
        printf "\n";
}

Line 8 is where the magic begins.  The call to fork() clones the program in memory making a parent and child.  The parent copy has the $pid set to the process ID of the child that was just forked, and the child has $pid set to zero.  In this example, the parent saves the childs PID in an array for later reference and goes through the for-loop (line 7) until it's forked a child for each element in the array named @array.

Each child does his own thing - he prints that it is going to execute (sleep) for a few seconds, sleeps, then he dies.

While the children are sleeping, the parent watches over them in the while loop at line 25.  It's kinda morbid, but the parent uses the wait() call to signal when a child exits (dies), then it prints what it knew about the child process, and also prints the list of children it's still waiting for.

When that code is executed, it will produce output something like this:

$ perl fork_example.pl
Mon Apr 20 22:36:25 2009 Executing AA for 0.356 seconds.
Mon Apr 20 22:36:25 2009 Executing BB for 9.797 seconds.
Mon Apr 20 22:36:25 2009 Executing CC for 4.411 seconds.
Mon Apr 20 22:36:25 2009 Executing DD for 7.816 seconds.
Mon Apr 20 22:36:25 2009 Executing EE for 5.170 seconds.
Mon Apr 20 22:36:25 2009 Executing FF for 8.632 seconds.
Mon Apr 20 22:36:25 2009 Executing GG for 6.502 seconds.
Mon Apr 20 22:36:25 2009 EXITED: 12343(AA), waiting for: 12344(BB) : 12345(CC) : 12346(DD) : 12347(EE) : 12348(FF) : 12349(GG)
Mon Apr 20 22:36:29 2009 EXITED: 12345(CC), waiting for: 12344(BB) : 12346(DD) : 12347(EE) : 12348(FF) : 12349(GG)
Mon Apr 20 22:36:30 2009 EXITED: 12347(EE), waiting for: 12344(BB) : 12346(DD) : 12348(FF) : 12349(GG)
Mon Apr 20 22:36:31 2009 EXITED: 12349(GG), waiting for: 12344(BB) : 12346(DD) : 12348(FF)
Mon Apr 20 22:36:32 2009 EXITED: 12346(DD), waiting for: 12344(BB) : 12348(FF)
Mon Apr 20 22:36:33 2009 EXITED: 12348(FF), waiting for: 12344(BB)
Mon Apr 20 22:36:34 2009 EXITED: 12344(BB)

 In my situation, the fork() loop (lines 7..22) were a bit more involved.  I added code to limit the number of child processes it would fork at a time - basically keeping track of the active number of children and using wait() when it reached the threshold but still had more to process.

The data also had some situations where new incoming data was added to existing data.  If two children are processing data that has to update the data on the disk, it's entirely possible for both children to read the data at the same time, add their own bit of data to the mix, and write the file back to disk.  The end result would be that one child would end up writing last and overwriting the other childs data.  That lead to the use of the flock() command - which has its own quirks and deserves its own space.

So, when you're sitting down to your Thanksgiving dinner remember how usefull the fork() is!