CSE2031 2011-12S Lab 11

Lab 11

As with previous labs, set up an appropriate directory. Get feedback on your work from the TA during the lab. There is no need for a submission for this lab.

The Exercises

In the time remaining in this lab period, you will start with on a few shell scripting exercises given here, and then use any remaining time to complete - perhaps with the help of the TA - any of the lab exercises you have not successfully completed so far.

Lab 8 q1
Write, and test a bash shell script called script1 that prints out its command line arguments (not the command name), one per line. Test runs of the program should look like this:
```
$
$ script1
$ script1 b hello    a
b
hello
a
$
```
Some students confused commandline arguments with standard input and tried to use read to answer this question.

Some students remembered that the positional parameters $1 $2 $3 ... referred to the commandline arguments, with $0 being the command name and $1 being the first argument to the command, and so on. They wrote their script using these.

But that's the hard way and generally appropriate only if you want to deal with specific arguments, as opposed to all arguments. Bash has built-in variables that stand for "all arguments". You access them as
```
$*
```
and
```
$@
```
And the best choice for iteration here is a for-loop. Here are 3 ways to write this:
```
    for x in $*
    do
      # do something with $x, in this case just print it out alone on a line:
      echo $x
    done
```
or
```
    for x in $@
    ... same as above
```
or use the shortcut:
```
    for x  <-- by default, x takes on values of the commandline args
    do
        ...
    done
```
Do these scripts print out the command name (accessible as $0)? Try them and see.

...

At this point, if you haven't tried these scripts yet, do it now.

What's the difference between $* and $@? You'll see it only when you put quotes around them, as in
```
    for x in "$*"
    ...
```
and
```
    for x in "$@"
    ...
```
When the script is called this way:
```
    script1 b hello    a
```
the first handles the arguments as
```
    "b hello    a"
```
the second, like
```
    "b"  "hello"   "a"
```
Try it.

Lab 8 q2
This script should work just like the one above, except that the arguments should be printed out in sorted order.

Consider any command cmd that writes lines to standard output. You can get the sorted output of cmd by capturing the (unsorted) output in a file temp and sorting temp like this:
```
    $ cmd > temp
    $ sort temp <-- leaves temp unchanged, but writes sorted version to standard out
```
But recall that the filename passed as an argument to sort is optional. We can have either sort filename or just sort. In the latter case, where no file name is given, sort just reads lines from standard input and sorts them. Try it. You won't see sort do anything till you signal end of input using <ctrl>d, as in this example:
```
$ sort
line 1 is here   <--- entered at keyboard
here is line 2   <--- entered at keyboard
and line 3       <--- entered at keyboard
finally line 4   <--- entered at keyboard
^D               <--- <ctrl>d entered
and line 3       <--- output from sort command
finally line 4   <--- output from sort command
here is line 2   <--- output from sort command
line 1 is here   <--- output from sort command
$
```
And recall what a pipe does: in cmd1 | cmd2 it makes the standard output of cmd1 the standard input of cmd2. So in the above example, we can eliminate using the file called temp and just get the sorted output of cmd by using a pipe, like this:
```
    $ cmd | sort
    ... output of sort comes here ...
    $
```
All of this explanation just to say that we get the script for question 2 from the script from question 1, just by piping the output to sort, as in
```
    for x in $*
    do
      # do something with $x, in this case just print it out alone on a line:
      echo $x
    done | sort
```
Try it!

Lab 8 q3
For this exercise, we expect output like this
```
$ script3 Hello Goodbye c b a
arg number 1 is Hello
arg number 2 is Goodbye
arg number 3 is c
arg number 4 is b
arg number 5 is a
```
This is like the first question, but also prints out some text and the argument number on each line, along with the number of the argument.

So start with the solution to the first question, but use a variable to hold the number of the argument, and print out some extra text.

And put in a comment or two. Shell scripts are easy to write, but reading and understanding someone else's script is often quite difficult. Help the reader/user.
```
#!/bin/bash

# argument number
n=0

for x
do
  # increment the argument number
  let "n = $n + 1"  # or: n=`expr $n + 1`

  # print out argument along with its number in a message
  echo "argument number $n is $x"
done
```

Lab 8 q4
To begin with, this question specifies that the script should be invoked with a single commandline argument. If it is not, there should be a message of a certain form. To start, forget about what else the script should do, and concentrate on this part.

Start with a script that just prints how many commandline arguments it has. How does it know? The number of commandline arguments is accessible via a built-in variable: $#. Do you think it includes argument 0, i.e. $0, the command name itself? Try it and find out. By the way, in C programming argc in int main(int argc, char **argv) plays the same role as $#. In C, does it count argv[0], the command name itself? (Try that, too, sometime.)

Here's the starter script4:
```
#!/bin/bash

echo $#
```
Run it like this:
```
$ script4
0
$ script4 a
1
$ script4 a b c d
4
$
```
(So, $# doesn't seem to count $0.)

Now try changing the script so it outputs the usage message described in lab 8 if there is not exactly 1 commandline argument. A first try might look like this:
```
#!/bin/bash

# usage

if [ $# -ne 1 ]
then
    echo -e "Usage: script4 <logname or logname prefix>"
fi
```
Try running this:
```
$
$ script4
Usage: script4 <logname or logname prefix>
$ script4 a b
Usage: script4 <logname or logname prefix>
$ script4 xxx
$
```
This seems to work, but there are a couple of things to fix. First, we've hardcoded the name of the script: script4. If we rename the script, the usage message is then inappropriate, for example
```
$
$ mv script4 myScript
$ myScript a b
Usage: script4 <logname or logname prefix>
$
```
The message talks about "script4" although there is no longer such a thing.

We can fix this problem by using $0 in the script to replace the hardcoded script name: ... Usage: $0 <logname .... That will work in some cases, but will often give us the full pathname when we don't want it. Use $0 as just explained and we get
```
$
$ myScript a b
Usage: ./myScript <logname or logname prefix>
$ mv myScript script4
$ script4 a b
Usage: ./script4 <logname or logname prefix>
$ ./script4 a b
Usage: ./script4 <logname or logname prefix>
$ pwd
/cs/dept/www/course/2031/Labs/11
$ /cs/dept/www/course/2031/Labs/11/script4 a b
Usage: /cs/dept/www/course/2031/Labs/11/script4 <logname or logname prefix>
$
```
So we see that $0 is not always printed out the way specified in the question. We can fix this by using the command basename, which outputs only the last part of a pathname. (Try man basename.) For example,
```
$ basename /a/b/c/d/xxx
xxx
$
```
In the script, replace $0 with the output of basename $0. How do you get the output of a command? - use backquotes. So replace the $0 with `basename $0`. Then try the same examples.
```
$
$ script4 a b
Usage: script4 <logname or logname prefix>
$ ./script4 a b
Usage: script4 <logname or logname prefix>
$ pwd
/cs/dept/www/course/2031/Labs/11
$ /cs/dept/www/course/2031/Labs/11/script4 a b
Usage: script4 <logname or logname prefix>
$
```
Almost there: one problem left is that this error/usage message is written to standard output (the echo command writes to standard output) instead of to standard error. To fix this, redirect the output of echo to standard error. For this, use >&2, which says to redirect standard output to wherever standard error is going. The line we are working on now looks like this:
```
    echo  "Usage: `basename $0` <logname or logname prefix>" >&2
```
It remains only to add a statement that exits with non-zero exit status after the usage message is printed. That gives us
```
#!/bin/sh

# usage message
if [ $# -ne 1 ]
then
    echo -e "Usage: `basename $0` <logname or logname prefix>" >&2
    exit 1
fi

# program continues here is if there was one command line argument
# ...
```
If echo -e is used instead of just echo, then echo understands special characters (like \t, \n, \c) inside the quoted string; otherwise it treats them literally. This script should take a single commandline argument. That argument will be treated as the login id (or beginning of the login id) of some users on the system, and it will print out the real names of such users, if any exist.

For this exercise, we will access /etc/passwd, which contains one line for each user on the system. Each line consists of 7 colon-delimited fields, with the login id in the first field and the real name in the 5th field. Have a look at the file. Or just look at the line in the file that represents you:
```
$ grep "^cse12345:" /etc/passwd
```
Of course, replace cse12345 with your own login id.

Actually, a main part of the script will look roughly like this command; i.e. it will grep a pattern in /etc/passwd. The grep'd pattern will start with "^...", i.e. it will be anchored at the beginning of the line (that's what the ^ means), but it won't have the ending colon (:), since it can be just the prefix of a login id and not the whole login id.

So part of the script will be
```
$ grep "^$1" /etc/passwd
```
Try putting just this in the script and running it. Your script would look like this:
```
#!/bin/sh
...
#usage message
...
grep "^$1" /etc/passwd
```
Here is a sample run (try it):
```
$
$ script4 cse12043
cse12043:x:13557:10000:Ben Blanc:/cs/home/cse12043:/bin/false
$
$ script4 cse1204
cse12040:x:13551:10000:Golriz Abbaspour:/cs/home/cse12040:/bin/false
cse12041:x:13554:10000:Mark Hervias:/cs/home/cse12041:/bin/false
cse12043:x:13557:10000:Ben Blanc:/cs/home/cse12043:/bin/false
cse12044:x:13560:10000:Justin Wong:/cs/home/cse12044:/bin/false
cse12045:x:13561:10000:Daniel A Franklin:/cs/home/cse12045:/bin/false
cse12046:x:13562:10000:Mahbubul Quddus:/cs/home/cse12046:/bin/false
cse12047:x:13568:10000:Allan Lo:/cs/home/cse12047:/bin/false
cse12048:x:13569:10000:Dong Joon Lee:/cs/home/cse12048:/bin/false
cse12049:x:13571:10000:TImothy John Manas:/cs/home/cse12049:/bin/false
$
$ script4 ali
ali:x:3008:2000:Ali Mahmoodi:/cs/home/ali:/cs/local/bin/tcsh
alireza:x:12719:3000:alireza moghaddam:/cs/home/alireza:/cs/local/bin/tcsh
$
$ script4 alir
alireza:x:12719:3000:alireza moghaddam:/cs/home/alireza:/cs/local/bin/tcsh
$
```
Of course, we don't want all this output; we just want the 5th field in each line. The command we use to get fields is cut. cut is like cat or sort in that it can either take a filename and cut out columns from that file, or it can read standard input and cut out columns from that.

We pipe the output of the grep command to cut. We want to get the 5th field, so we use cut -f5. And since the field delimiter is not the default (tab), we have to specify the field delimiter with -d: or -d':'. The script then becomes
```
#!/bin/sh
...
#usage message
...
grep "^$1" /etc/passwd  | cut -f5 -d:
```
And running the same tests gives us:
```
$
$ script4 ali alir
Usage: script4 
$
$ script4 cse12043
Ben Blanc
$
$ script4 cse1204
Golriz Abbaspour
Mark Hervias
Ben Blanc
Justin Wong
Daniel A Franklin
Mahbubul Quddus
Allan Lo
Dong Joon Lee
TImothy John Manas
$
$ script4 ali
Ali Mahmoodi
alireza moghaddam
$
$ script4 alir
alireza moghaddam
$
```

Lab 9 q1

Here you were to write a Bash shell script to test an executable file that reads from standard input and writes to standard output. Have a look at that exercise in lab 9 before going any further here ...

The idea here is that you test an executable by redirecting standard input to come from a known input file, say called in.17. You have the expected/correct output for this input in another file called out.17. Capture the output of the tested program in a file called testedOut, or whatever, and then compare this output with the expected output using diff. Doing this one test at the command line might look like this for an executable called cmdToTest:

$ cmdToTest < in.17 > testedOut
$ diff testedOut out.17 >/dev/null 2>&1 <--- compare files. suppress output
$ if [ $? -eq 0 ]                       <--- if the exit status of the last command is 0, i.e. if the files were identical
> then                                  <--- the '>' is the secondary prompt here
> echo success
> else
> echo failure
> fi
success
$

Put this into a script where the name of the command to be tested is given as the first commandline argument $1. To start, assume there are 3 test cases: in.10/out.10, in.12/out.12 and in.30/out.30, and that these are all in a directory passed as the second commandline argument $2. Keep a variable to count the number of successes. It could look like this:

#!/bin/sh

# first argument is command to test
cmd=$1

# second argument is directory holding test cases
Test_Cases=$2

# number of tests passed
n_successes=0

# first test with input from $Test_Cases/in.10 and expected output $Test_Cases/out.10
$cmd < $Test_Cases/in.10 > testedOut
diff testedOut $Test_Cases/out.10 >/dev/null 2>&1
if [ $? -eq 0 ]
then
    let "n_successes = $n_successes + 1"  # or: n_successes=`expr $n_successes + 1`
fi

# second test with input from $Test_Cases/in.12 and expected output $Test_Cases/out.12
$cmd < $Test_Cases/in.12 > testedOut
diff testedOut $Test_Cases/out.12 >/dev/null 2>&1
if [ $? -eq 0 ]
then
    let "n_successes = $n_successes + 1"  # or: n_successes=`expr $n_successes + 1`
fi

# third test with input from $Test_Cases/in.30 and expected output $Test_Cases/out.13
$cmd < $Test_Cases/in.30 > testedOut
diff testedOut $Test_Cases/out.30 >/dev/null 2>&1
if [ $? -eq 0 ]
then
    let "n_successes = $n_successes + 1"  # or: n_successes=`expr $n_successes + 1`
fi

# output message with number of successful tests
echo -e "$n_successes tests passed out of 3"

Naturally, it is silly to repeat the testing code 3 times and we should just use a loop like this:

#!/bin/sh

# first argument is command to test
cmd=$1

# second argument is directory holding test cases
Test_Cases=$2

# number of tests passed
n_successes=0

for x in 10 12 30
do
    # test with input from $Test_Cases/in.$x and expected output $Test_Cases/out.$x and capture the output in testedOut
    $cmd < $Test_Cases/in.$x > testedOut

    # if the test produced the correct/expected output, increment count of successful tests
    diff testedOut $Test_Cases/out.$x >/dev/null 2>&1
    if [ $? -eq 0 ]
    then
        let "n_successes = $n_successes + 1"  # or: n_successes=`expr $n_successes + 1`
    fi
done


# output message with number of successful tests
echo -e "$n_successes tests passed out of 3"

Now, instead of having 3 tests, we want a test for each file with the name in.something found in the directory of test cases that is passed as the second argument to the command. We'll need to count the tests as well, since there may be any number, not just 3.

So how do we find all files in.something in that directory? Or, rather, how do we find the xx for all files in.xx in the passed directory?

First, finding all files in.something: when the output of ls is piped to another command, each filename output by ls is on a different line, i.e. the filenames are separated by newline characters. So we can pipe all the filenames to grep <pattern> and grep will output each line (i.e. filename) that matches pattern.

Like this: ls directoryName | grep <pattern>

But what should the pattern be? "in."? That would get in.10 and the like, but also satin.abc. We need to anchor the "in" at the beginning of the line/filename. So the pattern could be "^in.". That would match in.10, etc. but not satin.abc. However, it would also match inX35, since the dot '.' represents any single character in a regular expression. We need to escape the dot, so it has no special meaning and is just a dot. That leaves us with the pattern "^in\."

We can find all the files named

in.something with the 
command
ls directoryName | grep "^in\."


Try that with the directory given in lab 9:

$ ls /cs/course/2031/Lab9/A1_Test_Cases | grep "^in\."


Try this.  Then try it with "in\." and "^in." to see why you shouldn't 
use them




Now you can use the above command to output a list like
in.00
in.01
in.02
in.03
in.04
in.05
in.06
...

How do you then grab the 00, 01, 02, etc., so you can use them to refer
to the corresponding expected output file out.00, out.01, etc.?
Just pipe this output to the cut command and use the dot '.' as
the field separator.  You want the second field.  The final command would be
$ ls /cs/course/2031/Lab9/A1_Test_Cases | grep "^in\." | cut -f2 -d.


Try it.  You'll get output like
00
01
02
03
04
05
06
...


Finally, you can use this output to give the values the for-loop variable
will take on in your script.  Replace
for x in 10 12 30
do
    ....

with
for x in `ls $Test_Cases | grep "^in\." | cut -f2 -d.`
do
    ....

Don't forget the back quotes.




You'll also need to add a variable to keep track of the total number of tests.
Finally, you'll get this:
#!/bin/sh

# first argument is command to test
cmd=$1

# second argument is directory holding test cases
Test_Cases=$2

# number of tests run
n_tests=0

# number of tests passed
n_successes=0

for x in `ls $Test_Cases | grep "^in\." | cut -f2 -d.`
do
    # increment the number of tests
    let "n_tests = $n_tests + 1"

    # test with input from $Test_Cases/in.$x and expected output $Test_Cases/out.$x and capture the output in testedOut
    $cmd < $Test_Cases/in.$x > testedOut

    # if the test produced the correct/expected output, increment count of successful tests
    diff testedOut $Test_Cases/out.$x >/dev/null 2>&1
    if [ $? -eq 0 ]
    then
        let "n_successes = $n_successes + 1"  # or: n_successes=`expr $n_successes + 1`
    fi
done


# output message with number of successful tests
echo -e "$n_successes tests passed out of $n_tests"







end of lab 11

Lab 11

The Exercises

Lab 8 q1

Lab 8 q2

Lab 8 q3

Lab 8 q4

Lab 9 q1

end of lab 11