Perl

Perl is the Swiss Army chainsaw of scripting languages: powerful and adaptable. It was first developed by Larry Wall, a linguist working as a systems administrator for NASA in the late 1980s, as a way to make report processing easier. Since then, it has moved into a large number of roles: automating system administration, acting as glue between different computer systems; and, of course, being one of the most popular languages for CGI programming on the Web. Why did Perl become so popular when the Web came along? Two reasons: First, most of what is being done on the Web happens with text, and is best done with a language that's designed for text processing. More importantly, Perl was appreciably better than the alternatives at the time when people needed something to use. C is complex and can produce security problems (especially with untrusted data), Tcl can be awkward and Python didn't really have a foothold. It also didn't hurt that Perl is a friendly language. It plays well with your personal programming style. The Perl slogan is ``There's more than one way to do it,'' and that lends itself well to large and small problems alike. In this first part of our series, you'll learn a few basics about Perl and see a small sample program.

A Word About Operating Systems In this series, I'm going to assume that you're using a Unix system and that your Perl interpreter is located at /usr/local/bin/perl. It's OK if you're running Windows; most Perl code is platform-independent.

Your First Perl Program Take the following text and put it into a file called first.pl: #!/usr/local/bin/perl print "Hi there!\n";

(Traditionally, first programs are supposed to say Hello world!, but I'm an iconoclast.) Now, run it with your Perl interpreter. From a command line, go to the directory with this file and type perl first.pl. You should see: Hi there!

The \n indicates the ``newline'' character; without it, Perl doesn't skip to a new line of text on its own.

Functions and Statements Perl has a rich library of functions. They're the verbs of Perl, the commands that the interpreter runs. You can see a list of all the built-in functions on the perlfunc main page. Almost all functions can be given a list of parameters, which are separated by commas.

The print function is one of the most frequently used parts of Perl. You use it to display things on the screen or to send information to a file (which we'll discuss in the next article). It takes a list of things to output as its parameters. print "This is a single statement."; print "Look, ", "a ", "list!";

A Perl program consists of statements, each of which ends with a semicolon. Statements don't need to be on separate lines; there may be multiple statements on one line or a single statement can be split across multiple lines. print "This is "; print "two statements.\n"; print "But this ", "is only one statement.\n";

Numbers, Strings and Quotes There are two basic data types in Perl: numbers and strings. Numbers are easy; we've all dealt with them. The only thing you need to know is that you never insert commas or spaces into numbers in Perl. always write 10000, not 10,000 or 10 000. Strings are a bit more complex. A string is a collection of characters in either single or double quotes: 'This is a test.' "Hi there!\n"

The difference between single quotes and double quotes is that single quotes mean that their contents should be taken literally, while double quotes mean that their contents should be interpreted. For example, the character sequence \n is a newline character when it appears in a string with double quotes, but is literally the two characters, backslash and n, when it appears in single quotes. print "This string\n shows up on two lines."; print 'This string \n shows up on only one.';

(Two other useful backslash sequences are \t to insert a tab character, and \\ to insert a backslash into a double-quoted string.)

Variables If functions are Perl's verbs, then variables are its nouns. Perl has three types of variables: scalars, arrays and hashes. Think of them as ``things,'' ``lists,'' and ``dictionaries.'' In Perl, all variable names are a punctuation character, a letter or underscore, and one or more alphanumeric characters or underscores. Scalars are single things. This might be a number or a string. The name of a scalar begins with a dollar sign, such as $i or $abacus. You assign a value to a scalar by telling Perl what it equals, like so: $i = 5; $pie_flavor = 'apple'; $constitution1776 = "We the People, etc.";

You don't need to specify whether a scalar is a number or a string. It doesn't matter,

because when Perl needs to treat a scalar as a string, it does; when it needs to treat it as a number, it does. The conversion happens automatically. (This is different from many other languages, where strings and numbers are two separate data types.) If you use a double-quoted string, Perl will insert the value of any scalar variables you name in the string. This is often used to fill in strings on the fly: $apple_count = 5; $count_report = "There are $apple_count apples."; print "The report is: $count_report\n";

The final output from this code is The report is: There are 5 apples.. Numbers in Perl can be manipulated with the usual mathematical operations: addition, multiplication, division and subtraction. (Multiplication and division are indicated in Perl with the * and / symbols, by the way.) $a = 5; $b = $a + 10; $c = $b * 10; $a = $a - 1; cringing.

# $b is now equal to 15. # $c is now equal to 150. # $a is now 4, and algebra teachers are

You can also use special operators like ++, --, +=, -=, /= and *=. These manipulate a scalar's value without needing two elements in an equation. Some people like them, some don't. I like the fact that they can make code clearer. $a = 5; $a++; $a += 10; $a /= 2;

# $a is now 6; we added 1 to it. # Now it's 16; we added 10. # And divided it by 2, so it's 8.

Strings in Perl don't have quite as much flexibility. About the only basic operator that you can use on strings is concatenation, which is a $10 way of saying ``put together.'' The concatenation operator is the period. Concatenation and addition are two different things: $a = "8"; # Note the quotes. $a is a string. $b = $a + "1"; # "1" is a string too. $c = $a . "1"; # But $b and $c have different values!

Remember that Perl converts strings to numbers transparently whenever it's needed, so to get the value of $b, the Perl interpreter converted the two strings "8" and "1" to numbers, then added them. The value of $b is the number 9. However, $c used concatenation, so its value is the string "81". Just remember, the plus sign adds numbers and the period puts strings together.

Arrays are lists of scalars. Array names begin with @. You define arrays by listing their contents in parentheses( ), separated by commas , : @lotto_numbers = (1, 2, 3, 4, 5, 6); # Hey, it could happen. @months = ("July", "August", "September");

The contents of an array are indexed beginning with 0. (Why not 1? Because. It's a computer thing.) To retrieve the elements of an array, you replace the @ sign with a $ sign, and follow that with the index position of the element you want. (It begins with a dollar

sign because you're getting a scalar value.) You can also modify it in place, just like any other scalar. @months = ("July", "August", "September"); print $months[0]; # This prints "July". $months[2] = "Smarch"; # We just renamed September!

If an array doesn't exist, by the way, you'll create it when you try to assign a value to one of its elements. $winter_months[0] = "December"; @winter_months.

# This implicitly creates

Arrays always return their contents in the same order; if you go through @months from beginning to end, no matter how many times you do it, you'll get back July, August and September in that order. If you want to find the length of an array, use the value $#array_name. This is one less than the number of elements in the array. If the array just doesn't exist or is empty, $#array_name is -1. If you want to resize an array, just change the value of $#array_name. @months = ("July", "August", "September"); print $#months; # This prints 2. $a1 = $#autumn_months; # We don't have an @autumn_months, so this is -1. $#months = 0; # Now @months only contains "July".

Hashes are called ``dictionaries'' in some programming languages, and that's what they are: a term and a definition, or in more correct language a key and a value. Each key in a hash has one and only one corresponding value. The name of a hash begins with a percentage sign, like %parents. You define hashes by comma-separated pairs of key and value, like so: %days_in_month = ( "July" => 31, "August" => 31, "September" => 30 );

You can fetch any value from a hash by referring to $hashname{key}, or modify it in place just like any other scalar. print $days_in_month{"September"}; # 30, of course. $days_in_month{"February"} = 29; # It's a leap year.

If you want to see what keys are in a hash, you can use the keys function with the name of the hash. This returns a list containing all of the keys in the hash. The list isn't always in the same order, though; while we could count on @months to always return July, August, September in that order, keys %days_in_summer might return them in any order whatsoever. @month_list = keys %days_in_summer(should be days_in_month); # @month_list is now ('July', 'September', 'August') !

The three types of variables have three separate namespaces. That means that $abacus and @abacus are two different variables, and $abacus[0] (the first element of @abacus) is not the same as $abacus{0} (the value in abacus that has the key 0).

Comments Notice that in some of the code samples from the previous section, I've used code

comments. These are useful for explaining what a particular piece of code does, and vital for any piece of code you plan to modify, enhance, fix, or just look at again. (That is to say, comments are vital for all code.) Anything in a line of Perl code that follows a # sign is a comment. (Except, of course, if the # sign appears in a string.) print "Hello world!\n"; # That's more like it. # This entire line is a comment.

Loops Almost every time you write a program, you'll need to use a loop. Loops allow you run a particular piece of code over and over again. This is part of a general concept in programming called flow control. Perl has several different functions that are useful for flow control, the most basic of which is for. When you use the for function, you specify a variable that will be used for the loop index, and a list of values to loop over. Inside a pair of curly brackets, you put any code you want to run during the loop: for $i (1, 2, 3, 4, 5) { print "$i\n"; }

This loop prints the numbers 1 through 5, each on a separate line. A handy shortcut for defining loops is using .. to specify a range of numbers. You can write (1, 2, 3, 4, 5) as (1 .. 5). You can also use arrays and scalars in your loop list. Try this code and see what happens: @one_to_ten = (1 .. 10); $top_limit = 25; for $i (@one_to_ten, 15, 20 .. $top_limit) { print "$i\n"; }

The items in your loop list don't have to be numbers; you can use strings just as easily. If the hash %month_has contains names of months and the number of days in each month, you can use the keys function to step through them. for $i (keys %month_has) { print "$i has $month_has{$i} days.\n"; } for $marx ('Groucho', 'Harpo', 'Zeppo', 'Karl') { print "$marx is my favorite Marx brother.\n"; }

The Miracle of Compound Interest You now know enough about Perl - variables, print, and for() - to write a small, useful program. Everyone loves money, so the first sample program is a compound-interest calculator. It will print a (somewhat) nicely formatted table showing the value of an investment over a number of years. (You can see the program at compound_interest.pl)

The single most complex line in the program is this one: $interest = int (($apr / 100) * $nest_egg * 100) / 100;

is the interest rate, and ($apr / 100) * $nest_egg is the amount of interest earned in one year. This line uses the int() function, which returns the integer value of a scalar (its value after any fractional part has been stripped off). We use int() here because when you multiply, for example, 10925 by 9.25%, the result is 1010.5625, which we must round off to 1010.56. To do this, we multiply by 100, yielding 101056.25, use int() to throw away the leftover fraction, yielding 101056, and then divide by 100 again, so that the final result is 1010.56. Try stepping through this statement yourself to see just how we end up with the correct result, rounded to cents. $apr / 100

Play Around! At this point you have some basic knowledge of Perl syntax and a few simple toys to play with - print, for(), keys(), and int(). Try writing some simple programs with them. Here are two suggestions, one simple and the other a little more complex: •A word frequency counter. How often does each word show up in an array of words? Print out a report. (Hint: Use a hash to count of the number of appearances of each word.) •Given a month and the day of the week that's the first of that month, print a calendar for the month. (Remember, you need \n to go to a new line.)

2.

In our last article, we talked about the core elements of Perl: variables (scalars, arrays, and hashes), math operators and some basic flow control (the for statement). Now it's time to interact with the world. In this installment, we're going to discuss how to slice and dice strings, how to play with files and how to define your own functions. But first, we'll discuss one more core concept of the Perl language: conditions and comparisons.

Comparison operators There's one important element of Perl that we skipped in the last article: comparison operators. Like all good programming languages, Perl allows you ask questions such as ``Is this number greater than that number?'' or ``Are these two strings the same?'' and do different things depending on the answer. When you're dealing with numbers, Perl has four important operators: <, >, == and !=. These are the ``less than,'' ``greater than,'' ``equal to'' and ``not equal to'' operators. (You can also use <=, ``less than or equal to,'' and >=, ``greater than or equal to.)

You can use these operators along with one of Perl's conditional keywords, such as if and unless. Both of these keywords take a condition that Perl will test, and a block of code in curly brackets that Perl will run if the test works. These two words work just like their English equivalents - an if test succeeds if the condition turns out to be true, and an unless test succeeds if the condition turns out to be false: if ($year_according_to_computer == 1900) { print "Y2K has doomed us all! Everyone to the compound.\n"; } unless ($bank_account > 0) { print "I'm broke!\n"; }

Be careful of the difference between = and ==! One equals sign means ``assignment'', two means ``comparison for equality''. This is a common, evil bug: if ($a = 5) { print "This works - but doesn't do what you want!\n"; }

Instead of testing whether $a is equal to five, you've made $a equal to five and clobbered its old value. (In a later article, we'll discuss a way to make sure this bug won't occur in running code.) Both if and unless can be followed by an else statement and code block, which executes if your test failed. You can also use elsif to chain together a bunch of if statements: if ($a == 5) { print "It's five!\n"; } elsif ($a == 6) { print "It's six!\n"; } else { print "It's something else.\n"; } unless ($pie eq 'apple') { print "Ew, I don't like $pie flavored pie.\n"; } else { print "Apple! My favorite!\n"; } while

and until

Two slightly more complex keywords are while and until. They both take a condition and a block of code, just like if and unless, but they act like loops similar to for. Perl tests the condition, runs the block of code and runs it over and over again for as long as the condition is true (for a while loop) or false (for a until loop). Take a look at the following code and try to guess what it will do before reading further: $a = 0; while ($a != 3) { $a++;

print "Counting up to $a...\n";

}

until ($a == 0) { $a--; print "Counting down to $a...\n"; }

Here's what you see when you run this program: Counting Counting Counting Counting Counting Counting

up to 1... up to 2... up to 3... down to 2... down to 1... down to 0...

String comparisons So that's how you compare numbers. Now, what about strings? The most common string comparison operator is eq, which tests for string equality - that is, whether two strings have the same value. Remember the pain that is caused when you mix up = and ==? Well, you can also mix up == and eq. This is one of the few cases where it does matter whether Perl is treating a value as a string or a number. Try this code: $yes_no = "no"; if ($yes_no == "yes") { print "You said yes!\n"; }

Why does this code think you said yes? Remember that Perl automatically converts strings to numbers whenever it's necessary; the == operator implies that you're using numbers, so Perl converts the value of $yes_no (``no'') to the number 0, and ``yes'' to the number 0 as well. Since this equality test works (0 is equal to 0), the if block gets run. Change the condition to $yes_no eq "yes", and it'll do what it should. Things can work the other way, too. The number five is numerically equal to the string " 5 ", so comparing them to == works. But when you compare five and " 5 " with eq, Perl will convert the number to the string "5" first, and then ask whether the two strings have the same value. Since they don't, the eq comparison fails. This code fragment will print Numeric equality!, but not String equality!: $a = 5; if ($a == " 5 ") { print "Numeric equality!\n"; } if ($a eq " 5 ") { print "String equality!\n"; }

More fun with strings You'll often want to manipulate strings: Break them into smaller pieces, put them together and change their contents. Perl offers three functions that make string manipulation easy and fun: substr(), split() and join(). If you want to retrieve part of a string (say, the first four characters or a 10-character

chunk from the middle), use the substr() function. It takes either two or three parameters: the string you want to look at, the character position to start at (the first character is position 0) and the number of characters to retrieve. If you leave out the number of characters, you'll retrieve everything up to the end of the string. $a = "Welcome to Perl!\n"; print substr($a, 0, 7); print substr($a, 7);

# "Welcome" # " to Perl!\n"

A neat and often-overlooked thing about substr() is that you can use a negative character position. This will retrieve a substring that begins with many characters from the end of the string. $a = "Welcome to Perl!\n"; print substr($a, -6, 4);

# "Perl"

(Remember that inside double quotes, \n represents the single new-line character.) You can also manipulate the string by using substr() to assign a new value to part of it. One useful trick is using a length of zero to insert characters into a string: $a = "Welcome to Java!\n"; substr($a, 11, 4) = "Perl"; # $a is now "Welcome to Perl!\n"; substr($a, 7, 3) = ""; # ... "Welcome Perl!\n"; substr($a, 0, 0) = "Hello. "; # ... "Hello. Welcome Perl!\n";

Next, let's look at split(). This function breaks apart a string and returns a list of the pieces. split() generally takes two parameters: a regular expression to split the string with and the string you want to split. (We'll discuss regular expressions in more detail in the next article; for the moment, we're only going to use a space. Note the special syntax for a regular expression: / /.) The characters you split won't show up in any of the list elements. $a = "Hello. Welcome Perl!\n"; @a = split(/ /, $a); # Three items: "Hello.", "Welcome", Perl!\n"

You can also specify a third parameter: the maximum number of items to put in your list. The splitting will stop as soon as your list contains that many items: $a = "Hello. Welcome Perl!\n"; @a = split(/ /, $a, 2); # Two items: "Hello.", "Welcome Perl!\n";

Of course, what you can split, you can also join(). The join() function takes a list of strings and attaches them together with a specified string between each element, which may be an empty string: @a $a $b $c

= = = =

("Hello.", "Welcome", "Perl!\n"); join(' ', @a); # "Hello. Welcome Perl!\n"; join(' and ', @a); # "Hello. and Welcome and Perl!\n"; join('', @a); # "Hello.WelcomePerl!\n";

Filehandles Enough about strings. Let's look at files - after all, what good is string manipulation if you can't do it where it counts? To read from or write to a file, you have to open it. When you open a file, Perl asks the operating system if the file can be accessed - does the file exist if you're trying to read it

(or can it be created if you're trying to create a new file), and do you have the necessary file permissions to do what you want? If you're allowed to use the file, the operating system will prepare it for you, and Perl will give you a filehandle. You ask Perl to create a filehandle for you by using the open() function, which takes two arguments: the filehandle you want to create and the file you want to work with. First, we'll concentrate on reading files. The following statement opens the file log.txt using the filehandle LOGFILE: open (LOGFILE, "log.txt");

Opening a file involves several behind-the-scenes tasks that Perl and the operating system undertake together, such as checking that the file you want to open actually exists (or creating it if you're trying to create a new file) and making sure you're allowed to manipulate the file (do you have the necessary file permissions, for instance). Perl will do all of this for you, so in general you don't need to worry about it. Once you've opened a file to read, you can retrieve lines from it by using the <> construct. Inside the angle brackets, place the name of your filehandle. What is returned by this depends on what you want to get: in a scalar context (a more technical way of saying ``if you're assigning it to a scalar''), you retrieve the next line from the file, but if you're looking for a list, you get a list of all the remaining lines in the file. (One common trick is to use for $lines () to retrieve all the lines from a file - the for means you're asking a list.) You can, of course, close a filehandle that you've opened. You don't always have to do this, because Perl is clever enough to close a filehandle when your program ends or when you try to reuse an existing filehandle. It's a good idea, though, to use the close statement. Not only will it make your code more readable, but your operating system has built-in limits on the number of files that can be open at once, and each open filehandle will take up valuable memory. Here's a simple program that will display the contents of the file log.txt, and assumes that the first line of the file is its title: open (LOGFILE, "log.txt") or die "I couldn't get at log.txt"; # We'll discuss the "or die" in a moment. $title = ; print "Report Title: $title"; for $line () { print $line; } close LOGFILE;

Writing files You also use open() when you are writing to a file. There are two ways to open a file for writing: overwrite and append. When you open a file in overwrite mode, you erase whatever it previously contained. In append mode, you attach your new data to the end of the existing file without erasing anything that was already there.

To indicate that you want a filehandle for writing, you put a single > character before the filename you want to use. This opens the file in overwrite mode. To open it in append mode, use two > characters. open (OVERWRITE, ">overwrite.txt") or die "$! error trying to overwrite"; # The original contents are gone, wave goodbye. open (APPEND, ">>append.txt") or die "$! error trying to append"; # Original contents still there, we're adding to the end of the file

Once our filehandle is open, we can use the humble print statement to write to it. Specify the filehandle you want to write to and a list of values you want to write: print OVERWRITE "This is the new content.\n"; print APPEND "We're adding to the end here.\n", "And here too.\n";

Live free or die! You noticed that most of our open() statements are followed by or die "some sort of message". This is because we live in an imperfect world, where programs don't always behave exactly the way we want them to. It's always possible for an open() call to fail; maybe you're trying to write to a file that you're not allowed to write, or you're trying to read from a file that doesn't exist. In Perl, you can guard against these problems by using or and and. A series of statements separated by or will continue until you hit one that works, or returns a true value. This line of code will either succeed at opening OUTPUT in overwrite mode, or cause Perl to quit: open (OUTPUT, ">$outfile") or die "Can't write to $outfile: $!";

The die statement ends your program with an error message. The special variable $! contains Perl's explanation of the error. In this case, you might see something like this if you're not allowed to write to the file. Note that you get both the actual error message (``Permission denied'') and the line where it happened: Can't write to a2-die.txt: Permission denied at ./a2-die.pl line 1.

Defensive programming like this is useful for making your programs more error-resistant - you don't want to write to a file that you haven't successfully opened! Here's an example: As part of your job, you write a program that records its results in a file called vitalreport.txt. You use the following code: open VITAL, ">vitalreport.txt";

If this open() call fails (for instance, vitalreport.txt is owned by another user who hasn't given you write permission), you'll never know it until someone looks at the file afterward and wonders why the vital report wasn't written. (Just imagine the joy if that ``someone'' is your boss, the day before your annual performance review.) When you use or die, you avoid all this: open VITAL, ">vitalreport.txt" or die "Can't write vital report:

$!";

Instead of wondering whether your program wrote your vital report, you'll immediately have an error message that both tells you what went wrong and on what line of your program the error occurred. You can use or for more than just testing file operations: ($pie eq 'apple') or ($pie eq 'cherry') or ($pie eq 'blueberry') or print "But I wanted apple, cherry, or blueberry!\n";

In this sequence, if you have an appropriate pie, Perl skips the rest of the chain. Once one statement works, the rest are ignored. The and operator does the opposite: It evaluates your chain of statements, but stops when one of them doesn't work. open (LOG, "log.file") and print "Logfile is open!\n";

This statement will only show you the words Logfile is open! if the open() succeeds - do you see why?

Subs So far, our Perl programs have been a bunch of statements in series. This is OK if you're writing very small programs, but as your needs grow, you'll find it's limiting. This is why most modern programming languages allow you to define your own functions; in Perl, we call them subs. A sub is defined with the sub keyword, and adds a new function to your program's capabilities. When you want to use this new function, you call it by name. For instance, here's a short definition of a sub called boo: sub boo { print "Boo!\n"; } boo();

# Eek!

(Older versions of Perl required that you precede the name of a sub with the & character when you call it. You no longer have to do this, but if you see code that looks like &boo in other people's Perl, that's why.) Subs are useful because they allow you to break your program into small, reusable chunks. If you need to analyze a string in four different places in your program, it's much easier to write one &analyze_string sub and call it four times. This way, when you make an improvement to your string-analysis routine, you'll only need to do it in one place, instead of four. In the same way that Perl's built-in functions can take parameters and can return values, your subs can, too. Whenever you call a sub, any parameters you pass to it are placed in the special array @_. You can also return a single value or a list by using the return keyword. sub multiply { my (@ops) = @_; return $ops[0] * $ops[1]; }

for $i (1 .. 10) { print "$i squared is ", multiply($i, $i), "\n"; }

Why did we use the my keyword? That indicates that the variables are private to that sub, so that any existing value for the @ops array we're using elsewhere in our program won't get overwritten. This means that you'll evade a whole class of hard-to-trace bugs in your programs. You don't have to use my, but you also don't have to avoid smashing your thumb when you're hammering nails into a board. They're both just good ideas. You can also use my to set up local variables in a sub without assigning them values right away. This can be useful for loop indexes or temporary variables: sub annoy { my ($i, $j); for $i (1 .. 100) { $j .= "Is this annoying yet?\n"; } print $j; }

If you don't expressly use the return statement, the sub returns the result of the last statement. This implicit return value can sometimes be useful, but it does reduce your program's readability. Remember that you'll read your code many more times than you write it!

Putting it all together At the end of the first article we had a simple interest calculator. Now let's make it a bit more interesting by writing our interest table to a file instead of to the screen. We'll also break our code into subs to make it easier to read and maintain. [Download this program] #!/usr/local/bin/perl -w

# compound_interest_file.pl - the miracle of compound interest, part 2

report. table. rate.

# First, we'll set up the variables we want to use. $outfile = "interest.txt"; # This is the filename of our $nest_egg = 10000; $year = 2000;

# $nest_egg is our starting amount # This is the starting year for our

$duration = 10; $apr = 9.5;

# How many years are we saving up? # This is our annual percentage

&open_report;

&print_headers; &interest_report($nest_egg, $year, $duration, $apr); &report_footer; sub open_report { open (REPORT, ">$outfile") or die "Can't open report: $!"; }

"\t",

sub print_headers { # Print the headers for our report. print REPORT "Year", "\t", "Balance", "\t", "Interest", "New balance", "\n";

}

sub calculate_interest { # Given a nest egg and an APR, how much interest do we collect?

my ($nest_egg, $apr) = @_;

}

return int (($apr / 100) * $nest_egg * 100) / 100;

sub interest_report { # Get our parameters. Note that these variables won't clobber the # global variables with the same name. my ($nest_egg, $year, $duration, $apr) = @_; # We have two local variables, so we'll use my to declare them here.

my ($i, $line); # Calculate interest for each year. for $i (1 .. $duration) { $year++; $interest = &calculate_interest($nest_egg, $apr); $line = join("\t", $year, $nest_egg, $interest, $nest_egg + $interest) . "\n"; print REPORT $line; }

$nest_egg += $interest;

} sub report_footer {

print print print print

REPORT REPORT REPORT REPORT

"\n " " "

Our original assumptions:\n"; Nest egg: $nest_egg\n"; Number of years: $duration\n"; Interest rate: $apr\n";

close REPORT; }

Notice how much clearer the program logic becomes when you break it down into subs. One nice quality of a program written as small, well-named subs is that it almost becomes self-documenting. Take a look at these four lines from our program: open_report; print_headers; interest_report($nest_egg, $year, $duration, $apr); report_footer;

Code like this is invaluable when you come back to it six months later and need to figure out what it does - would you rather spend your time reading the entire program trying to figure it out or read four lines that tell you the program 1) opens a report file, 2) prints some headers, 3) generates an interest report, and 4) prints a report footer? You'll also notice we use my to set up local variables in the interest_report and calculate_interest subs. The value of $nest_egg in the main program never changes. This is useful at the end of the report, when we output a footer containing our original assumptions. Since we never specified a local $nest_egg in report_footer, we use the global value.

Play around! In this article, we've looked at files (filehandles, open(), close(), and <>), string manipulation (substr(), split() and join()) and subs. Here's a pair of exercises again, one simple and one complex: •You have a file called dictionary.txt that contains dictionary definitions, one per line, in the format ``word space definition''. (Here's a sample.) Write a program that will look up a word from the command line. (Hints: @ARGV is a special array that contains your command line arguments and you'll need to use the threeargument form of split().) Try to enhance it so that your dictionary can also contain words with multiple definitions in the format ``word space definition:alternate definition:alternate definition, etc...''. •Write an analyzer for your Apache logs. You can find a brief description of the common log format at http://www.w3.org/Daemon/User/Config/Logging.html. Your analyzer should count the total number of requests for each URL, the total number of results for each status code and the total number of bytes output.

3. We've covered flow control, math and string operations, and files in the first two articles in this series. Now we'll look at Perl's most powerful and interesting way of playing with strings, regular expressions, or regexes for short. (The rule is this: after the 50th time you type ``regular expression,'' you find you type ``regexp'' the next 50 times.) Regular expressions are complex enough that you could write a whole book on them (and, in fact, someone did - Mastering Regular Expressions by Jeffrey Friedl).

Simple matching The simplest regular expressions are matching expressions. They perform tests using keywords like if, while and unless. Or, if you want to be really clever, tests that you can use with and and or. A matching regexp will return a true value if whatever you try to match occurs inside a string. When you want to use a regular expression to match against a string, you use the special =~ operator: $user_location = "I see thirteen black cats under a ladder."; if ($user_location =~ /thirteen/) { print "Eek, bad luck!\n"; }

Notice the syntax of a regular expression: a string within a pair of slashes. The code $user_location =~ /thirteen/ asks whether the literal string thirteen occurs anywhere inside $user_location. If it does, then the test evaluates true; otherwise, it evaluates false.

Metacharacters A metacharacter is a character or sequence of characters that has special meaning. We've discussed metacharacters in the context of double-quoted strings, where the sequence \n mean the newline character, not a backslash, and the character n and \t means the tab character. Regular expressions have a rich vocabulary of metacharacters that let you ask interesting questions such as, ``Does this expression occur at the end of a string?'' or ``Does this string contain a series of numbers?'' The two simplest metacharacters are ^ and $ These indicate ``beginning of string'' and ``end of string,'' respectively. For example, the regexp /^Bob/ will match ``Bob was here,'' ``Bob'' and ``Bobby.'' It won't match ``It's Bob and David,'' because Bob doesn't show up at the beginning of the string. The $ character, on the other hand, means that you are matching the end of a string. The regexp /David$/ will match ``Bob and David,'' but not ``David and Bob.'' Here's a simple routine that will take lines from a file and only print URLs that seem to indicate HTML files: for $line () {

}

# "If the line starts with http: and ends with html...." if (($line =~ /^http:/) and ($line =~ /html$/)) { print $line; }

Another useful set of metacharacters is called wildcards. If you've ever used a Unix shell or the Windows DOS prompt, you're familiar with wildcards characters like * and ?. For example when you type ls a*.txt, you see all filenames that begin with the letter a and end with .txt. Perl is a bit more complex, but works on the same general principle. In Perl, the generic wildcard character is .. A period inside a regular expression will match any character, except a newline. For example, the regexp /a.b/ will match anything that contains a, another character that's not a newline, followed by b - ``aab,'' ``a3b,'' ``a b,'' and so forth. If you want to literally match a metacharacter, you must escape it with a backslash. The regex /Mr./ matches anything that contains ``Mr'' followed by another character. If you only want to match a string that actually contains ``Mr.,'' you must use /Mr\./. On its own, the . metacharacter isn't very useful, which is why Perl provides three wildcard quantifiers: +, ? and *. Each quantifier means something different. The + quantifier is the easiest to understand: It means to match the immediately preceding character or metacharacter one or more times. The regular expression /ab+c/ will match ``abc,'' ``abbc,'' ``abbbc'' and so on. The * quantifier matches the immediately preceding character or metacharacter zero or more times. This is different from the + quantifier! /ab*c/ will match ``abc,'' ``abbc,'' and so on, just like /ab+c/ did, but it'll also match ``ac,'' because there are zero occurences of b in that string. Finally, the ? quantifier will match the preceding character zero or one times. The regex /ab?c/ will match ``ac'' (zero occurences of b) and ``abc'' (one occurence of b). It won't match ``abbc,'' ``abbbc'' and so on. We can rewrite our URL-matching code to use these metacharacters. This'll make it more concise. Instead of using two separate regular expressions (/^http:/ and /html$/), we combine them into one regular expression: /^http:.+html$/. To understand what this does, read from left to right: This regex will match any string that starts with ``http:'' followed by one or more occurences of any character, and ends with ``html''. Now, our routine is: for $line () { if ($line =~ /^http:.+html$/) { print $line; } }

Remember the /^something$/ construction - it's very useful!

Character classes We've already discussed one special metacharacter, ., that matches any character except a newline. But you'll often want to match only specific types of characters. Perl provides several metacharacters for this. <\d> will match a single digit, \w will match any single ``word'' character (which, to Perl, means a letter, digit or underscore), and \s matches a whitespace character (space and tab, as well as the \n and \r characters). These metacharacters work like any other character: You can match against them, or you can use quantifiers like + and *. The regex /^\s+/ will match any string that begins with whitespace, and /\w+/ will match a string that contains at least one word. (But remember that Perl's definition of ``word'' characters includes digits and the underscore, so whether or not you think _ or 25 are words, Perl does!) One good use for \d is testing strings to see whether they contain numbers. For example, you might need to verify that a string contains an American-style phone number, which has the form 555-1212. You could use code like this: unless ($phone =~ /\d\d\d-\d\d\d\d/) { print "That's not a phone number!\n"; }

All those \d metacharacters make the regex hard to read. Fortunately, Perl allows us to improve on that. You can use numbers inside curly braces to indicate a quantity you want to match, like this: unless ($phone =~ /\d{3}-\d{4}/) { print "That's not a phone number!\n"; }

The string \d{3} means to match exactly three numbers, and \d{4} matches exactly four digits. If you want to use a range of numbers, you can separate them with a comma; leaving out the second number makes the range open-ended. \d{2,5} will match two to five digits, and <\w{3,}> will match a word that's at least three characters long. You can also invert the \d, \s and \w metacharacters to refer to anything but that type of character. \D matches nondigits; \W matches any character that isn't a letter, digit or underscore; and \S matches anything that isn't whitespace. If these metacharacters won't do what you want, you can define your own. You define a character class by enclosing a list of the allowable characters in square brackets. For example, a class containing only the lowercase vowels is [aeiou]. /b[aeiou]g/ will match any string that contains ``bag,'' ``beg,'' ``big,'' ``bog'' or ``bug''. You use dashes to indicate a range of characters, like [a-f]. (If Perl didn't give us the \d metacharacter, we could do the same thing with [0-9].) You can combine character classes with quantifiers: if ($string =~ /[aeiou]{2}/) { print "This string contains at least two vowels in a row.\n"; }

You can also invert character classes by beginning them with the ^ character. An inverted character class will match anything you don't list. [^aeiou] matches every character

except the lowercase vowels. (Yes, ^ can also mean ``beginning of string,'' so be careful.)

Flags By default, regular expression matches are case-sensitive (that is, /bob/ doesn't match ``Bob''). You can place flags after a regexp to modify their behaviour. The most commonly used flag is i, which makes a match case-insensitive: $greet = "Hey everybody, it's Bob and David!"; if ($greet =~ /bob/i) { print "Hi, Bob!\n"; }

We'll talk about more flags later.

Subexpressions You might want to check for more than one thing at a time. For example, you're writing a ``mood meter'' that you use to scan outgoing e-mail for potentially damaging phrases. You can use the pipe character | to separate different things you are looking for: # In reality, @email_lines would come from your email text, # but here we'll just provide some convenient filler. @email_lines = ("Dear idiot:", "I hate you, you twit. You're a dope.", "I bet you mistreat your llama.", "Signed, Doug"); for $check_line (@email_lines) { if ($check_line =~ /idiot|dope|twit|llama/) { print "Be careful! This line might contain something offensive:\n", $check_line, "\n"; } }

The matching expression /idiot|dope|twit|llama/ will be true if ``idiot,'' ``dope,'' ``twit'' or ``llama'' show up anywhere in the string. One of the more interesting things you can do with regular expressions is subexpression matching, or grouping. A subexpression is like another, smaller regex buried inside your larger regexp, and is placed inside parentheses. The string that caused the subexpression to match will be stored in the special variable $1. We can use this to make our mood meter more explicit about the problems with your e-mail: for $check_line (@email_lines) { if ($check_line =~ /(idiot|dope|twit|llama)/) { print "Be careful! This line contains the offensive word $1:\n", $check_line, "\n"; } }

Of course, you can put matching expressions in your subexpression. Your mood watch program can be extended to prevent you from sending e-mail that contains more than three exclamation points in a row. We'll use the special {3,} quantifier to make sure we

get all the exclamation points. for $check_line (@email_lines) { if ($check_line =~ /(!{3,})/) { print "Using punctuation like '$1' is the sign of a sick mind:\n", $check_line, "\n"; } }

If your regex contains more than one subexpression, the results will be stored in variables named $1, $2, $3 and so on. Here's some code that will change names in ``lastname, firstname'' format back to normal: $name = "Wall, Larry"; $name =~ /(\w+), (\w+)/; # $1 contains last name, $2 contains first name $name = "$2 $1"; # $name now contains "Larry Wall"

You can even nest subexpressions inside one another - they're ordered as they open, from left to right. Here's an example of how to retrieve the full time, hours, minutes and seconds separately from a string that contains a timestamp in hh:mm:ss format. (Notice that we're using the {1,2} quantifier so that a timestamp like ``9:30:50'' will be matched.) $string = "The time is 12:25:30 and I'm hungry."; $string =~ /((\d{1,2}):(\d{2}):(\d{2}))/; @time = ($1, $2, $3, $4);

Here's a hint that you might find useful: You can assign to a list of scalar values whenever you're assigning from a list. If you prefer to have readable variable names instead of an array, try using this line instead: ($time, $hours, $minutes, $seconds) = ($1, $2, $3, $4);

Assigning to a list of variables when you're using subexpressions happens often enough that Perl gives you a handy shortcut: ($time, $hours, $minutes, $seconds) = ($string =~ /((\d{1,2}):(\d{2}):(\d{2}))/);

Watch out! Regular expressions have two traps that generate bugs in your Perl programs: They always start at the beginning of the string, and quantifiers always match as much of the string as possible. Here's some simple code for counting all the numbers in a string and showing them to the user. We'll use while to loop over the string, matching over and over until we've counted all the numbers. $number = "Look, 200 5-sided, 4-colored pentagon maps."; while ($number =~ /(\d+)/) { print "I found the number $1.\n"; $number_count++; } print "There are $number_count numbers here.\n";

This code is actually so simple it doesn't work! When you run it, Perl will print I found the number 200 over and over again. Perl always begins matching at the beginning of the string, so it will always find the 200, and never get to the following numbers. You can avoid this by using the g flag with your regex. This flag will tell Perl to remember where it was in the string when it returns to it. When you insert the g flag, our code looks like this: $number = "Look, 200 5-sided, 4-colored pentagon maps."; while ($number =~ /(\d+)/g) { print "I found the number $1.\n"; $number_count++; } print "There are $number_count numbers here.\n";

Now we get the results we expected: I found the number 200. I found the number 5. I found the number 4. There are 3 numbers here.

The second trap is that a quantifier will always match as many characters as it can. Look at this example code, but don't run it yet: $book_pref = "The cat in the hat is where it's at.\n"; $book_pref =~ /(cat.*at)/; print $1, "\n";

Take a guess: What's in $1 right now? Now run the code. Does this seem counterintuitive? The matching expression (cat.*at) is greedy. It contains cat in the hat is where it's at because that's the largest string that matches. Remember, read left to right: ``cat,'' followed by any number of characters, followed by ``at.'' If you want to match the string cat in the hat, you have to rewrite your regexp so it isn't as greedy. There are two ways to do this: 1. Make the match more precise (try /(cat.*hat)/ instead). Of course, this still might not work - try using this regexp against The cat in the hat is who I hate. 2. Use a ? character after a quantifier to specify nongreedy matching. .*? instead of .* means that Perl will try to match the smallest string possible instead of the largest: # Now we get "cat in the hat" in $1. $book_pref =~ /(cat.*?at)/;

Search and replace Now that we've talked about matching, there's one other thing regular expressions can do for you: replacing. If you've ever used a text editor or word processor, you're familiar with the search-andreplace function. Perl's regexp facilities include something similar, the s/// operator, which has the following syntax: s/regex/replacement string/. If the string you're

testing matches regex, then whatever matched is replaced with the contents of replacement string. For instance, this code will change a cat into a dog: $pet = "I love my cat.\n"; $pet =~ s/cat/dog/; print $pet;

You can also use subexpressions in your matching expression, and use the variables $1, $2 and so on, that they create. The replacement string will substitute these, or any other variables, as if it were a double-quoted string. Remember our code for changing Wall, Larry into Larry Wall? We can rewrite it as a single s/// statement! $name = "Wall, Larry"; $name =~ s/(\w+), (\w+)/$2 $1/;

# "Larry Wall"

can take flags, just like matching expressions. The two most important flags are g (global) and i (case-insensitive). Normally, a substitution will only happen once, but specifying the g flag will make it happen as long as the regex matches the string. Try this code, and then remove the g flag and try it again: s///

$pet = "I love my cat Sylvester, and my other cat Bill.\n"; $pet =~ s/cat/dog/g; print $pet;

Notice that without the g flag, Bill doesn't turn into a dog. The i flag works just as it did when we were only using matching expressions: It forces your matching search to be case-insensitive.

Putting it all together Regular expressions have many practical uses. We'll look at a httpd log analyzer for an example. In our last article, one of the play-around items was to write a simple log analyzer. Now, let's make it a bit more interesting: a log analyzer that will break down your log results by file type and give you a list of total requests by hour. (Complete source code.) First, let's look at a sample line from a httpd log: 127.12.20.59 - - [01/Nov/2000:00:00:37 -0500] "GET /gfx2/page/home.gif HTTP/1.1" 200 2285

The first thing we want to do is split this into fields. Remember that the split() function takes a regular expression as its first argument. We'll use /\s/ to split the line at each whitespace character: @fields = split(/\s/, $line);

This gives us 10 fields. The ones we're concerned with are the fourth field (time and date of request), the seventh (the URL), and the ninth and 10th (HTTP status code and size in bytes of the server response). First, we'd like to make sure that we turn any request for a URL that ends in a slash (like /about/) into a request for the index page from that directory (/about/index.html). We'll need to escape out the slashes so that Perl doesn't mistake them for terminators in

our s/// statement. $fields[6] =~ s/\/$/\/index.html/;

This line is difficult to read, because anytime we come across a literal slash character we need to escape it out. This problem is so common, it has acquired a name: leaningtoothpick syndrome. Here's a useful trick for avoiding the leaning-toothpick syndrome: You can replace the slashes that mark regular expressions and s/// statements with any other matching pair of characters, like { and }. This allows us to write a more legible regex where we don't need to escape out the slashes: $fields[6] =~ s{/$}{/index.html};

(If you want to use this syntax with a matching expression, you'll need to put a m in front of it. /foo/ would be rewritten as m{foo}.) Now, we'll assume that any URL request that returns a status code of 200 (request OK) is a request for the file type of the URL's extension (a request for /gfx/page/home.gif returns a GIF image). Any URL request without an extension returns a plain-text file. Remember that the period is a metacharacter, so we need to escape it out! if ($fields[8] eq '200') { if ($fields[6] =~ /\.([a-z]+)$/i) { $type_requests{$1}++; } else { $type_requests{'txt'}++; } }

Next, we want to retrieve the hour each request took place. The hour is the first string in $fields[3] that will be two digits surrounded by colons, so all we need to do is look for that. Remember that Perl will stop when it finds the first match in a string: # Log the hour of this request $fields[3] =~ /:(\d{2}):/; $hour_requests{$1}++;

Finally, let's rewrite our original report() sub. We're doing the same thing over and over (printing a section header and the contents of that section), so we'll break that out into a new sub. We'll call the new sub report_section(): sub report { print ``Total bytes requested: '', $bytes, ``\n''; print "\n"; report_section("URL requests:", %url_requests); report_section("Status code results:", %status_requests); report_section("Requests by hour:", %hour_requests); report_section("Requests by file type:", %type_requests); }

The new report_section() sub is very simple: sub report_section { my ($header, %type) = @_; print $header, "\n"; for $i (sort keys %type) {

}

print $i, ": ", $type{$i}, "\n";

print "\n"; }

We use the keys function to return a list of the keys in the %type hash, and the sort function to put it in alphabetic order. We'll play with sort a bit more in the next article.

Play around! As usual, here are some sample exercises: 1. A rule of good writing is ``avoid the passive voice.'' Instead of The report was read by Carl, say Carl read the report. Write a program that reads a file of sentences (one per line), detects and eliminates the passive voice, and prints the result. (Don't worry about irregular verbs or capitalization, though.) Sample solution. Sample test sentences. 2. You have a list of phone numbers. The list is messy, and the only thing you know is that there are either seven or 10 digits in each number (the area code is optional), and if there's an extension, it will show up after an ``x'' somewhere on the line. ``416 5551212,'' ``5551300X40'' and ``(306) 555.5000 ext 40'' are all possible. Write a fix_phone() sub that will turn all of these numbers into the standard format ``(123) 5551234'' or ``(123) 555-1234 Ext 100,'' if there is an extension. Assume that the default area code is ``123.'' Sample solution.

4. What is CGI? The Web is based on a client-server model: your browser (the client) making requests to a Web server. Most of these are simple requests for documents or images, which the server delivers to the browser for display. Of course, sometimes you want the server to do more than just dump the contents of a file. You'd like to do something with a server-side program - whether that "something" is using Web-based e-mail, looking up a phone number in a database or ordering a copy of Evil Geniuses in a Nutshell for your favorite techie. This means the browser must be able to send information (an e-mail address, a name to look up, shipping information for a book) to the server, and the server must be able to use that information and return the results to the user. The standard for communication between a user's Web browser and a server-side program running on the Web server is called CGI, or Common Gateway Interface. It is supported by all popular Web server software. To get the most out of this article, you will need to

have a server that supports CGI. This may be a server running on your desktop machine or an account with your ISP (though probably not a free Web-page service). If you don't know whether you have CGI capabilities, ask your ISP or a local sysadmin how to set things up. Notice that I haven't described how CGI works; that's because you don't need to know. There's a standard Perl module called CGI.pm that will handle the CGI protocol for you. CGI.pm is part of the core Perl distribution, and any properly installed Perl should have it available. Telling your CGI program that you want to use the CGI module is as simple as this: use CGI ':standard';

The use CGI ':standard'; statement tells Perl that you want to use the CGI.pm module in your program. This will load the module and make a set of CGI functions available for your code.

A Real CGI Program Let's write our first real CGI program. Instead of doing something complex, we'll write something that will simply throw back whatever we throw at it. We'll call this script backatcha.cgi: #!/usr/local/bin/perl use CGI ':standard'; print header(); print start_html(); for $i (param()) { print "", $i, ": ", param($i), "
\n"; } print end_html();

If you've never used HTML, the pair of and tags mean "begin bold" and "end bold", respectively, and the
tag means "line break." (A good paper reference to HTML is O'Reilly's HTML & XHTML: The Definitive Guide, and online, I like the Web Design Group.) Install this program on your server and do a test run. (If you don't have a Web server of your own, we've put a copy online for you here.) Here's a short list of what you do to install a CGI program: Make sure the program is placed where your Web server will recognize it as a CGI script. This may be a special cgi-bin directory or making sure the program's filename ends in .pl or .cgi. If you don't know where to place the program, your ISP or sysadmin should. Make sure the program can be run by the server. If you are using a Unix system, you may have to give the Web-server user read and execute permission for the

program. It's easiest to give these permissions to everybody by using chmod filename 755. Make a note of the program's URL, which will probably be something like http://server name/cgi-bin/backatcha.cgi) and go to that URL in your browser. (Take a guess what you should do if you don't know what the URL of the program is. Hint: It involves the words "ask," "your" and "ISP.") If this works, you will see in your browser ... a blank page! Don't worry, this is what is supposed to happen. The backatcha.cgi script throws back what you throw at it, and we haven't thrown anything at it yet. We'll give it something to show us in a moment. If it didn't work, you probably saw either an error message or the source code of the script. We'll try to diagnose these problems in the next section.

Uh-Oh! If you saw an error message, your Web server had a problem running the CGI program. This may be a problem with the program or the file permissions. First, are you sure the program has the correct file permissions? Did you set the file permissions on your program to 755? If not, do it now. (Windows Web servers will have a different way of doing this.) Try it again; if you see a blank page now, you're good. Second, are you sure the program actually works? (Don't worry, it happens to the best of us.) Change the use CGI line in the program to read: use CGI ':standard', '-debug';

Now run the program from the command line. You should see the following: (offline mode: enter name=value pairs on standard input)

This message indicates that you're testing the script. You can now press Ctrl-D to tell the script to continue running without telling it any form items. If Perl reports any errors in the script, you can fix them now. (The -debug option is incredibly useful. Use it whenever you have problems with a CGI program, and ignore it at your peril.) The other common problem is that you're seeing the source code of your program, not the result of running your program. There are two simple problems that can cause this. First, are you sure you're going through your Web server? If you use your browser's "load local file" option (to look at something like /etc/httpd/cgi-bin/backatcha.cgi instead of something like http://localhost/cgi-bin/backatcha.cgi), you aren't even touching the Web server! Your browser is doing what you "wanted" to do: loading the contents of a local file and displaying them. Second, are you sure the Web server knows it's a CGI program? Most Web server software will have a special way of designating a file as a CGI program, whether it's a

special cgi-bin directory, the .cgi or .pl extension on a file, or something else. Unless you live up to these expectations, the Web server will think the program is a text file, and serve up your program's source code in plain-text form. Ask your ISP for help. CGI programs are unruly beasts at the best of times; don't worry if it takes a bit of work to make them run properly.

Making the Form Talk Back At this point, you should have a working copy of backatcha.cgi spitting out blank pages from a Web server. Let's make it actually tell us something. Take the following HTML code and put it in a file:

What is your favorite color?

Be sure to replace putyourURLhere with the actual URL of your copy of backatcha.cgi! If you want, you can use the copy installed here at Perl.com. This is a simple form. It will show a text box where you can enter your favorite color and a "submit" button that sends your information to the server. Load this form in your browser and submit a favorite color. You should see this returned from the server: favcolor: green

CGI functions The CGI.pm module loads several special CGI functions for you. What are these functions? The first one, header(), is used to output any necessary HTTP headers before the script can display HTML output. Try taking this line out; you'll get an error from the Web server when you try to run it. This is another common source of bugs! The start_html() function is there for convenience. It returns a simple HTML header for you. You can pass parameters to it by using a hash, like this: print $cgi->start_html( -title => "My document" );

(The end_html() method is similar, but outputs the footers for your page.) Finally, the most important CGI function is param(). Call it with the name of a form item, and a list of all the values of that form item will be returned. (If you ask for a scalar, you'll only get the first value, no matter how many there are in the list.) $yourname = param("firstname"); print "

Hi, $yourname!

\n";

If you call param() without giving it the name of a form item, it will return a list of all the form items that are available. This form of param() is the core of our backatcha script: for $i (param()) { print "$i: ", param($i), "
\n";

}

Remember, a single form item can have more than one value. You might encounter code like this on the Web site of a pizza place that takes orders over the Web:

Pick your toppings!
Pepperoni
Mushrooms
Ham

Someone who wants all three toppings would submit a form where the form item top has three values: "pepperoni," "mushrooms" and "ham." The server-side code might include this: print "

You asked for the following pizza toppings: "; @top = param("top"); for $i (@top) { print $i, ". "; } print "

";

Now, here's something to watch out for. Take another look at the pizza-topping HTML code. Try pasting that little fragment into the backatcha form, just above the tag. Enter a favorite color, and check all three toppings. You'll see this: favcolor: burnt sienna top: pepperonimushroomsham

Why did this happens? When you call param('name'), you get back a list of all of the values for that form item. This could be considered a bug in the backatcha.cgi script, but it's easily fixed - use join() to separate the item values: print "$i: ", join(', ', param($i)), "
\n";

or call C<param()> in a scalar context first to get only the first value: $j = param($i); print "$i: $j

\n";

Always keep in mind that form items can have more than one value!

Our Second Script So now we know how to build a CGI program, and we've seen a simple example. Let's write something useful. In the last article, we wrote a pretty good HTTP log analyzer. Why not Web-enable it? This will allow you to look at your usage figures from anywhere you can get to a browser. Download the source code for the HTTP log analyzer First, let's decide what we want to do with our analyzer. Instead of showing all of the reports we generate at once, we'll show only those the user selects. Second, we'll let the user choose whether each report shows the entire list of items, or the top 10, 20 or 50 sorted by access count. We'll use a form such as this for our user interface:

Select the reports you want to see:

TYPE=checkbox TYPE=checkbox TYPE=checkbox TYPE=checkbox

NAME=report NAME=report NAME=report NAME=report

VALUE=url>URLs requested
VALUE=status>Status codes
VALUE=hour>Requests by hour
VALUE=type>File types

<SELECT NAME="number"> Show all Show top 10 Show top 20 Show top 50

(Remember that you may need to change the URL!) We're sending two different types of form item in this HTML page. One is a series of checkbox widgets, which set values for the form item report. The other is a single dropdown list which will assign a single value to number: either ALL, 10, 20 or 50. Take a look at the original HTTP log analyzer. We'll start with two simple changes. First, the original program gets the filename of the usage log from a command-line argument: # We will use a command line argument to determine the log filename. $logfile = $ARGV[0];

We obviously can't do that now, since the Web server won't allow us to enter a command line for our CGI program! Instead, we'll hard-code the value of $logfile. I'll use "/var/log/httpd/access_log" as a sample value. $logfile = "/var/log/httpd/access_log";

Second, we must make sure that we output all the necessary headers to our Web server before we print anything else: print header(); print start_html( -title => "HTTP Log report" );

Now look at the report() sub from our original program. It has one problem, relative to our new goals: It outputs all the reports instead of only the ones we've selected. We'll rewrite report() so that it will cycle through all the values of the report form item and show the appropriate report for each. sub report { for $i (param('report')) { if ($i eq 'url') { report_section("URL requests", %url_requests); } elsif ($i eq 'status') { report_section("Status code requests", %status_requests); } elsif ($i eq 'hour') { report_section("Requests by hour", %hour_requests); } elsif ($i eq 'type') {

report_section("Requests by file type", %type_requests);

} }

}

Finally, we rewrite the report_section() sub to output HTML instead of plain text. (We'll discuss the new way we're using sort in a moment.) sub report_section { my ($header, %type) = @_; my (@type_keys); # Are we sorting by the KEY, or by the NUMBER of accesses? if (param('number') ne 'ALL') { @type_keys = sort { $type{$b} <=> $type{$a}; } keys %type; # Chop the list if we have too many results if ($#type_keys > param('number') - 1) { $#type_keys = param('number') - 1; } } else { @type_keys = sort keys %type; } # Begin a HTML table print "\n"; # Print a table row containing a header for the table print "\n"; # Print a table row containing each item and its value for $i (@type_keys) { print "\n"; } # Finish the table print "

", $header, "
", $i, "	", $type{$i}, "

\n"; }

Sorting Perl allows you to sort lists with the sort keyword. By default, the sort will happen alphanumerically: numbers before letters, uppercase before lowercase. This is sufficient 99 percent of the time. The other 1 percent of the time, you can write a custom sorting routine for Perl to use. This sorting routine is just like a small sub. In it, you compare two special variables, $a and $b, and return one of three values depending on how you want them to show up in the list. Returning -1 means "$a should come before $b in the sorted list," 1 means "$b should come before $a in the sorted list" and 0 means "they're equal, so I don't care which comes first." Perl will run this routine to compare each pair of items in your list and produce the sorted result. For example, if you have a hash called %type, here's how you might sort its keys in descending order of their values in the hash.

sort { if ($type{$b} > $type{$a}) { return 1; } if ($type{$b} < $type{$a}) { return -1; } return 0; } keys %type;

In fact, numeric sorting happens so often, Perl gives you a convenient shorthand for it: the <=> operator. This operator will perform the above comparison between two values for you and return the appropriate value. That means we can rewrite that test as: sort { $type{$b} <=> $type{$a}; } keys %type

(And this, in fact, is what we use in our log analyzer.) You can also compare strings with sort. The lt and gt operators are the string equivalents of and , and cmp will perform the same test as =. (Remember, string comparisons will sort numbers before letters and uppercase before lowercase.) For example, you've got a list of names and phone numbers in the format "John Doe 5551212." You want to sort this list by the person's last name, and sort by first name when the last names are the same. This is a job made for cmp! @sorted = sort { ($c) = ($a =~ / (\w+)/); ($d) = ($b =~ / (\w+)/); if ($c eq $d) { # Last names are the same, sort on first name ($c) = ($a =~ /^(\w+)/); ($d) = ($b =~ /^(\w+)/); return $c cmp $d; } else { return $c cmp $d; } } @phone_numbers; for $i (@sorted) { print $i, "\n"; }

Trust No One Now that we know how CGI programs can do what you want, let's make sure they won't do what you don't want. This is harder than it looks, because you can't trust anyone to do what you expect. Here's a simple example: You want to make sure the HTTP log analyzer will never show more than 50 items per report, because it takes too long to send larger reports to the user. The easy thing to do would be to eliminate the "ALL" line from our HTML form, so that the only remaining options are 10, 20 and 50. It would be very easy - and wrong. Download the source code for the HTTP analyzer with security enhancements. We saw that you can modify HTML forms when we pasted the pizza-topping sample code into our backatcha page. You can also use the URL to pass form items to a script try going to http://www.perl.com/2000/12/backatcha.cgi?itemsource=URL&typedby=you in your browser. Obviously, if someone can do this with the backatcha script, they can also

do it with your log analyzer and stick any value for number in that they want: "ALL" or "25000", or "four score and seven years ago." Your form doesn't allow this, you say. Who cares? People will write custom HTML forms to exploit weaknesses in your programs, or will just pass bad form items to your script directly. You cannot trust anything users or their browsers tell you. You eliminate these problems by knowing what you expect from the user, and disallowing everything else. Whatever you do not expressly permit is totally forbidden. Secure CGI programs consider everything guilty until it is made innocent. For example, we want to limit the size of reports from our HTTP log analyzer. We decide that means the number form item must have a value that is between 10 and 50. We'll verify it like this: # Make sure that the "number" form item has a reasonable value ($number) = (param('number') =~ /(\d+)/); if ($number < 10) { $number = 10; } elsif ($number > 50) { $number = 50; }

Of course, we also have to change the report_section() sub so it uses the $number variable. Now, whether your user tries to tell your log analyzer that the value of number is "10," "200," "432023," "ALL" or "redrum," your program will restrict it to a reasonable value. We don't need to do anything with report, because we only act when one of its values is something we expected. If the user tries to enter something other than our expressly permitted values ("url," "status," "hour" or "type"), we just ignore it. Use this sort of logic everywhere you know what the user should enter. You might use s/\D//g to remove non-numeric characters from items that should be numbers (and then test to make sure what's left is within your range of allowable numbers!), or /^\w+$/ to make sure that the user entered a single word. All of this has two significant benefits. First, you simplify your error-handling code, because you make sure as early in your program as possible that you're working with valid data. Second, you increase security by reducing the number of "impossible" values that might help an attacker compromise your system or mess with other users of your Web server. Don't just take my word for it, though. The CGI Security FAQ has more information about safe CGI programming in Perl than you ever thought could possibly exist, including a section listing some security holes in real CGI programs.

Play Around! You should now know enough about CGI programming to write a useful Web application. (Oh, and you learned a little bit more about sorting and comparison.)

Write the quintessential CGI program: a guestbook. Users enter their name, e-mail address and a short message, which is appended to an HTML file for all to see. Be careful! Never trust the user! A good beginning precaution is to disallow all HTML by either removing < and > characters from all of the user's information or replacing them with the < and > character entities. Use substr(), too, to cut anything the user enters down to a reasonable size. Asking for a "short" message will do nothing to prevent the user dumping a 500k file into the message field! Write a program that plays tic-tac-toe against the user. Be sure that the computer AI is in a sub so it can be easily upgraded. (You'll probably need to study HTML a bit to see how to output the tic-tac-toe board.) FPRIVATE "TYPE=PICT;ALT= " 5. So far, we've mostly stuck to writing everything for our programs ourselves. One of the big advantages of Perl is that you don't need to do this. More than 1,000 people worldwide have contributed more than 5,000 utility packages, or modules, for common tasks. In this installment, we'll learn how modules work by building one, and along the way we'll learn a bit about object-oriented programming in Perl.

What Is an Object? Think back to the first article in this series, when we discussed the two basic data types in Perl, strings and numbers. There's a third basic data type: the object. Objects are a convenient way of packaging information with the things you actually do with that information. The information an object contains is called its properties, and the things you can do with that information are called methods. For example, you might have an AddressEntry object for an address book program - this object would contain properties that store a person's name, mailing address, phone number and e-mail address; and methods that print a nicely formatted mailing label or allow you to change the person's phone number. During the course of this article, we'll build a small, but useful, class: a container for configuration file information.

Our Goal So far, we've put the code for setting various options in our programs directly in the program's source code. This isn't a good approach. You may want to install a program and

allow multiple users to run it, each with their own preferences, or you may want to store common sets of options for later. What you need is a configuration file to store these options. We'll use a simple plain-text format, where name and value pairs are grouped in sections, and sections are indicated by a header name in brackets. When we want to refer to the value of a specific key in our configuration file, we call the key section.name. For instance, the value of author.firstname in this simple file is ``Doug:'' [author] firstname=Doug lastname=Sheppard [site] name=Perl.com url=http://www.perl.com/

(If you used Windows in the ancient days when versions had numbers, not years, you'll recognize this as being similar to the format of INI files.) Now that we know the real-world purpose of our module, we need to think about what properties and methods it will have: What do TutorialConfig objects store, and what can we do with them? The first part is simple: We want the object's properties to be the values in our configuration file. The second part is a little more complex. Let's start by doing the two things we need to do: read a configuration file, and get a value from it. We'll call these two methods read and get. Finally, we'll add another method that will allow us to set or change a value from within our program, which we'll call set. These three methods will cover nearly everything we want to do.

Starting Off We'll use the name TutorialConfig for our configuration file class. (Class names are normally named in this InterCapitalized style.) Since Perl looks for a module by its filename, this means we'll call our module file TutorialConfig.pm. Put the following into a file called TutorialConfig.pm: package TutorialConfig; warn "TutorialConfig is successfully loaded!\n"; 1;

(I'll be sprinkling debugging statements throughout the code. You can take them out in practice. The warn keyword is useful for warnings - things that you want to bring to the user's attention without ending the program the way die would.) The package keyword tells Perl the name of the class you're defining. This is generally the same as the module name. (It doesn't have to be, but it's a good idea!) The 1; will return a true value to Perl, which indicates that the module was loaded successfully.

You now have a simple module called TutorialConfig, which you can use in your code with the use keyword. Put the following into a very simple, one-line program: use TutorialConfig;

When we run this program, we see the following: TutorialConfig is successfully loaded!

What Does Our Object Do? Before we can create an object, we need to know how to create it. That means we must write a method called new that will set up an object and return it to us. This is also where you put any special initialization code that you might need to run for each object when it is created. The new method for our TutorialConfig class looks like this, and goes into TutorialConfig.pm right after the package declaration: sub new { my ($class_name) = @_; my ($self) = {}; warn "We just created our new variable...\n "; bless ($self, $class_name); warn "and now it's a $class_name object!\n";

}

$self->{'_created'} = 1; return $self;

(Again, you won't need those warn statements in actual practice.) Let's break this down line by line. First, notice that we define methods by using sub. (All methods are really just a special sort of sub.) When we call new, we pass it one parameter: the type of object we want to create. We store this in a private variable called $class_name. (You can also pass extra parameters to new if you want. Some modules use this for special initialization routines.) Next, we tell Perl that $self is a hash. The syntax my ($self) = {}; is a special idiom that's used mostly in Perl object programming, and we'll see how it works in some of our methods. (The technical term is that $self is an anonymous hash, if you want to read more about it elsewhere.) Third, we use the bless function. You give this function two parameters: a variable that you want to make into an object, and the type of object you want it to be. This is the line that makes the magic happen! Fourth, we'll set a property called ``_created''. This property isn't really that useful, but it does show the syntax for accessing the contents of an object: $object_name>{property_name}.

Finally, now that we've made $self into a new TutorialConfig object, we return it. Our program to create a TutorialConfig object looks like this: use TutorialConfig; $tut = new TutorialConfig;

(You don't need to use parentheses here, unless your object's new method takes any extra parameters. But if you feel more comfortable writing $tut = new TutorialConfig();, it'll work just as well.) When you run this code, you'll see: TutorialConfig is successfully loaded! We just created the variable ... and now it's a TutorialConfig object!

Now that we have a class and we can create objects with it, let's make our class do something!

Our Goal, Part 2 Look at our goals again. We need to write three methods for our TutorialConfig module: read, get and set. The first method, read, obviously requires that we tell it what file we want to read. Notice that when we write the source code for this method, we must give it two parameters. The first parameter is the object we're using, and the second is the filename we want to use. We'll use return to indicate whether the file was successfully read. sub read { my ($self, $file) = @_; my ($line, $section); open (CONFIGFILE, $file) or return 0; # We'll set a special property # that tells what filename we just read. $self->{'_filename'} = $file;

while ($line = ) { # Are we entering a new section? if ($line =~ /^\[(.*)\]/) { $section = $1; } elsif ($line =~ /^([^=]+)=(.*)/) { my ($config_name, $config_val) = ($1, $2); if ($section) { $self->{"$section.$config_name"} = $config_val; } else { $self->{$config_name} = $config_val; } } }

close CONFIGFILE; return 1;

}

Now that we've read a configuration file, we need to look at the values we just read. We'll call this method get, and it doesn't have to be complex: sub get { my ($self, $key) = @_; }

return $self->{$key};

These two methods are really all we need to begin experimenting with our TutorialConfig object. Take the module and sample configuration file from above (or download the configuration file here and the module here), put it in a file called tutc.txt, and then run this simple program: use TutorialConfig; $tut = new TutorialConfig; $tut->read('tutc.txt') or die "Couldn't read config file: $!"; print "The author's first name is ", $tut->get('author.firstname'), ".\n";

(Notice the syntax for calling an object's methods: $object->method(parameters).) When you run this program, you'll see something like this: TutorialConfig has been successfully loaded! We just created the variable... and now it's a TutorialConfig object! The author's first name is Doug.

We now have an object that will read configuration files and show us values inside those files. This is good enough, but we've decided to make it better by writing a set method that allows us to add or change configuration values from within our program: sub set { my ($self, $key, $value) = @_; $self->{$key} = $value; }

Now let's test it out: use TutorialConfig; $tut = new TutorialConfig; $tut->read('tutc.txt') or die "Can't read config file: $!"; $tut->set('author.country', 'Canada'); print $tut->get('author.firstname'), " lives in ", $tut->get('author.country'), ".\n";

These three methods (read, get and set) are everything we'll need for our TutorialConfig.pm module. More complex modules might have dozens of methods!

Encapsulation You may be wondering why we have get and set methods at all. Why are we using $tut->set('author.country', 'Doug') when we could use $tut>{'author.country'} = 'Doug' instead? There are two reasons to use methods instead of playing directly with an object's properties. First, you can generally trust that a module won't change its methods, no matter how much their implementation changes. Someday, we might want to switch from using text files to hold configuration information to using a database like MySQL or Postgres. Our new TutorialConfig.pm module might have new, read, get and set methods that look like this: sub new { my ($class) = @_; my ($self) = {}; bless $self, $class; return $self; } sub read { my ($self, $file) = @_; my ($db) = database_connect($file); if ($db) { $self->{_db} = $db; return $db; } return 0; } sub get { my ($self, $key) = @_; my ($db) = $self->{_db};

}

my ($value) = database_lookup($db, $key); return $value;

sub set { my ($self, $key, $value) = @_; my ($db) = $self->{_db};

}

my ($status) = database_set($db, $key, $value); return $status;

(Our module would define the database_connect, database_lookup and database_set routines elsewhere.) Even though the entire module's source code has changed, all of our methods still have the same names and syntax. Code that uses these methods will continue working just fine,

but code that directly manipulates properties will break! For instance, let's say you have some code that contains this line to set a configuration value: $tut->{'author.country'} = 'Canada';

This works fine with the original TutorialConfig.pm module, because when you call $tut->get('author.country'), it looks in the object's properties and returns ``Canada'' just like you expected. So far, so good. However, when you upgrade to the new version that uses databases, the code will no longer return the correct result. Instead of get() looking in the object's properties, it'll go to the database, which won't contain the correct value for ``author.country''! If you'd used $tut->set('author.country', 'Canada') all along, things would work fine. As a module author, writing methods will let you make changes (bug fixes, enhancements, or even complete rewrites) without requiring your module's users to rewrite any of their code. Second, using methods lets you avoid impossible values. You might have an object that takes a person's age as a property. A person's age must be a positive number (you can't be -2 years old!), so the age() method for this object will reject negative numbers. If you bypass the method and directly manipulate $obj->{'age'}, you may cause problems elsewhere in the code (a routine to calculate the person's birth year, for example, might fail or produce an odd result). As a module author, you can use methods to help programmers who use your module write better software. You can write a good error-checking routine once, and it will be used many times. (Some languages, by the way, enforce encapsulation, by giving you the ability to make certain properties private. Perl doesn't do this. In Perl, encapsulation isn't the law, it's just a very good idea.)

Play Around! 1. Our TutorialConfig.pm module could use a method that will write a new configuration file to any filename you desire. Write your own write() method (use keys %$self to get the keys of the object's properties). Be sure to use or to warn if the file couldn't be opened! 2. Write a BankAccount.pm module. Your BankAccount object should have deposit, withdraw, and balance methods. Make the withdraw method fail if you try to withdraw more money than you have, or deposit or withdraw a negative amount of money. 3. CGI.pm also lets you use objects if you want. (Each object represents a single CGI query.) The method names are the same as the CGI functions we used in the last article: use CGI; $cgi = new CGI;

print $cgi->header(), $cgi->start_html(); print "The 'name' parameter is ", $cgi->param('name'), ".\n"; print $cgi->end_html();

Try rewriting one of your CGI programs to use CGI objects instead of the CGI functions. 4. A big advantage of using CGI objects is that you can store and retrieve queries on disk. Take a look in the CGI.pm documentation to learn how to use the save() method to store queries, and how to pass a filehandle to new to read them from disk. Try writing a CGI program that saves recently used queries for easy retrieval.