Today's Lecture

• Pattern Matching
  • Shell Metacharacters (aka wildcards)
  • Regular Expressions

Don't Forget!!

• Quiz 2

Open today from 11 AM to 11:55 PM. Go to the class moodle to take the quiz.

Pattern Matching

• Software engineers are often confronted with the task of having to search a large information space quickly.
• For instance, large software systems can often consist of thousands to tens of thousands of individual files and these files can easily combine to millions of lines of code
• As a result, developers have created tools that allow:
  • file systems to be searched rapidly
  • large files to be searched rapidly
  • single commands to be applied to a large number of files at once
• A key component of these technologies is the mechanism used to specify queries
  • Shell metacharacters, aka wildcards, provide the ability to work with the names of files in a file system
  • Regular expressions provide a powerful mechanism for searching and modifying the contents of files
• It is important to remember that wildcards and regular expressions simply SPECIFY queries
  • In order to be effective, they need a tool that processes these specifications and actually performs the search they enable
    • For instance, wildcards are powerful because the Unix shell (in our case bash) and/or other tools processes those characters and give them their special meaning
    • Similarly, a regular expression engine is the entity that processes the meaning of a regular expression and uses it to search the contents of files in an efficient manner
• Why is this important?
  • Because more and more tools are being deployed that contain pattern matching engines and, if used consistently, can greatly reduce the time it takes a software engineer to locate, manipulate, or edit information in a software development project.
  • We will see examples of some of these tools later in the lecture...

Wildcards

• The bash shell provides the following metacharacters for file name substitution, aka globbing:
  • (I don't know where the term globbing comes from... a search on the Web turned up nothing.)

Simple Examples

• echo *
  • find all file names in the current directory
• echo a*
  • find all file names in the current directory that begin with the letter a
• echo a*b
  • find all file names in the current directory that begin with the letter a and end with the letter b
• echo *abc*
  • find all file names in the current directory that contain the substring abc, such as abc.txt, kensabcs.txt, abc, …
• echo ?
  • find all files with one letter file names
• echo ??
  • find all files with two letter file names
• echo [abc]
  • find all files named a, b, or c
• echo file[0-9][0-9]
  • find all files of the form file00, file01, file02, …, file98, file99
• echo [a-z][!a-z][0-9]
  • find all files with three letter file names that begin with a lowercase letter, followed by any character that is not a lowercase letter and that ends with a digit.

More on Ranges

• You can combine multiple ranges into a single character class
  • echo [a-zA-Z]
    • all single letter file names consisting of either a lower case letter or an upper case letter
• If you need to search for a hyphen make sure it appears first in the character class
  • echo ken[-0-9][0-9]
    • file names that begin with the string ken and are then followed by either two numbers or a dash and a number, such as ken-1 or ken09
• If you need to search for an exclamation point, then make sure that you have escaped it with a backslash
  • echo ken[\!]

Brace Expansion

• bash supports a feature known as brace expansion that is used to generate strings based on a supplied list a comma-separated substrings. (Note: there can be no unquoted whitespace within the braces.) Brace expansion is typically used to match file names, but in some cases the strings that are created do not have to refer to an existing file.
  • echo {foo,bar,baz,boz}
    • foo bar baz boz
• Characters that appear before the opening curly brace are known as the preamble and characters that appear after the curly brace are known as the postamble. Both the preamble and the postamble are optional.
  • echo file{A,B,C,D}.txt
    • fileA.txt fileB.txt fileC.txt fileD.txt
• Strings are expanded in order from left to right. There is no attempt to sort the strings that are being generated.
  • echo a{x,d,b}c
    • axc adc abc
• Some versions of bash (such as the one on csel.cs.colorado.edu) support sequence expressions, of the form [x..y] in which x and y can be single numbers or single characters. bash will then generate all strings implied by the sequence...
  • echo file{0..1}{0..9}
    • file00 file01 file02 file03 file04 file05 file06 file07 file08 file09 file10 file11 file12 file13 file14 file15 file16 file17 file18 file19
• This can be very useful since it can allow you to create or manipulate a large number of files at once.
  • touch file{0..1}{0..9}.txt
    • This would create the following files in the current directory: file00.txt file01.txt file02.txt file03.txt file04.txt file05.txt file06.txt file07.txt file08.txt file09.txt file10.txt file11.txt file12.txt file13.txt file14.txt file15.txt file16.txt file17.txt file18.txt file19.txt

Revisiting Lab 1

• In lab 1, you encountered the following wildcard expression

  
  case $1 in
    *[!0-9]*) echo "Error: test-exit requires an integer argument."
              exit 2
              ;;
          "") echo "Error: test-exit requires a non-empty argument."
              exit 3
              ;;
           *) ;;
  esac

• Do you now understand how the first branch of this case statement works?

Generating wildcard expressions

• How would you list all files that contain at least one lowercase letter in their name?
• How would you list all files that contain letters in their names before digits?
• How would you list all files whose names are five characters long?
• How would you list all files whose names begin with either an a, b, or c and end with a z?
• How would you list all files whose names begin with the string "file" and end with the string ".txt" and may then have three digits in between?

Regular Expressions

• Wildcard expressions let us match characters (in a number of ways); Regular Expressions differ in that they allow us to match patterns
  • This is a key difference that you must be sure to understand!
• This is important because some tools use wildcard syntax and others use regular expressions; you need to understand both
• Problem: wildcards and regular expressions often use the same metacharacters but they mean different things!

Regular Expression Syntax

• This table shows regular expression syntax and also attempts to highlight differences between regular expressions and wildcards

  Metacharacter	Name	Meaning	Wildcard Meaning
  .	period	match a single instance of any character except newline	question mark (?) had this meaning in wildcards, with the exception that wildcards don't generally worry about newlines
  [abc]	square brackets	matches ONE character in the set a, b, or c	ditto
  [^abc]	square brackets	matches ONE character NOT in the set a, b, or c	ditto, however syntax for bash is [!abc]
  [a-z]	square brackets	matches ONE character in the range a-z	ditto
  [^a-z]	square brackets	matches ONE character NOT in the range a-z	ditto, however syntax for bash is [!abc]
  pattern*	asterisk	matches zero or more instances of the previous pattern	matches zero or more characters
  pattern+	plus sign	matches one or more instances of the previous pattern	no special meaning
  pattern?	question mark	matches zero or one instance of the previous pattern	matches a single character
  pattern1 | pattern2	pipe	matches pattern1 or pattern2	brace expansion is closest match (definitely NOT an exact match)
  (pattern1pattern2...)	parens	combines one or more patterns in a logical group; also captures the characters that match the included patterns during a match for special processing	no corresponding construct
  ^pattern	hat	matches subsequent pattern at the beginning of a line	no special meaning
  pattern$	dollar sign	matches previous pattern at the end of a line	no special meaning
  \	backslash	interpret the next character literally	interpret the next character literally
  any other character		matches itself, e.g. an “a” character will match an “a” in the target string	ditto

Examples of regular expressions

• a*: zero or more “a” characters
  • a, aa, aaa, aaaa, aaaaa, …
• a.+b: matches a string that begins with “a”, contains at least one more character (but possibly more), and ends with a “b”.
  • acb, acdefgghkijbjjhkjhb, a898989789879879879b, …
• ([A-Z][a-zA-Z]+) ([A-Z]\. )?([A-Z][a-zA-Z]+)
  • Matches strings of the form: “Kenneth M. Anderson” or “Charlie Brown” or “Samuel L. Jackson” and captures the first, middle, and last name for later processing (we'll see an example of this in a little bit)
• [0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9][0-9][0-9]
  • What does this match?
• [0-9]+-[0-9]+-[0-9]+
  • How is this different than the previous one?

Differences between wildcards and regular expressions

• In regular expressions “*” means something very different than it does in wildcards
  • In wildcards, “a*” means any string starting with an “a”
  • In regular expressions, “a*” means strings consisting of only “a” characters, including the empty string
    • a, aa, aaa, aaaa, …
• This also applies to “+” and “?”. They do not stand for characters themselves but rather modify the previous pattern

Captures

• We mentioned that a regular expression that uses parentheses captures text for later processing. Lets now look at an example.
• ([A-Z][a-zA-Z]+) ([A-Z]\. )?([A-Z][a-zA-Z]+)
  • This expression matches names like “Kenneth M. Anderson”
  • The first group captures the name “Kenneth”
  • The second group captures “M. ”; note that the space after the middle initial is captured because it is part of the group
  • The third group captures “Anderson”
• Many programs that use regular expressions to search for regular expressions also allow you to do a replace operation and reference the groups in the replacement string
  • Typically, you reference the groups like this \1 \2 \3… but I've also seen $1 $2 $3…
• So, imagine the above regular expression matches the string “Kenneth M. Anderson” and you perform a replacement using the replacement string “$3, $1”, then after the replacement operation, you would be left with the string “Anderson, Kenneth”.

Regular Expression Strategies

• When using a regular expression to find a desired search string, be aware of the following three quantities
  • Hits: Strings you wanted to match
  • Omissions: Strings you didn't match but wanted to match
  • False Alarms: Strings you matched but didn't want to match

Examples

• Lets take a look at how you might use regular expressions in a text editor
  • I'll demonstrate the use of regular expressions in two different text editors on MacOS X: BBEdit and TextMate, including:
    • how to wrap lines at 80 columns using regular expressions
    • demonstrate the name regular expression from the previous slide
    • use BBEdit's text factory to process an electronic roster

Coming Up Next

• Lecture 7: Software Tools (find and grep)
• Lab 2 Assigned
• Homework 3 Due