Intro to Scientific Computing PHYS 27/193 Physics Department University of the Pacific

Controlling Data Flow

 

Redirection

Most processes initiated by Unix commands write to the standard output (that is, they write to the terminal screen), and many take their input from the standard input (that is, they read it from the keyboard). There is also the standard error, where processes write their error messages, by default, to the terminal screen.

Here is one rather abstract example.
We have already seen one use of the cat command to write the contents of a file to the screen.

This time, type cat without specifing a file to read

cat

Now type a few words on the keyboard and press the [Return] key.
You should see your words as you type, then each time you hit [Return], the word(s) you typed should be printed again.
Finally, hold the [Ctrl] key down and press [d] (written ^D for short) to end the input.

What happened?

If you run the cat command without specifing a file to read, then instead of reading from a file, it reads from standard input (i.e. the keyboard), then writes to standard output (the screen), until receiving the 'end of file' signal (^D).

In Unix, we can redirect both the input and the output of commands, which can be VERY powerful.

 

Redirecting the Output:   >

We use the > symbol to redirect the output of a command to a file.

In your unixplay/ directory is a little program called squares which simply prints out integers and their squares.

If this program is not there, you can get it by downloading this file: squares.exe Save this file in your unixplay/ directory.

If called with no arguments (i.e. just by itself):

squares

it prints the first 10 integers and their squares.

If called with a min and max, such as:

squares 20 30

It prints the integers and squares between the min and max numbers, as shown above (these are called arguments of the command).
The output is shown below.

20      400
21      441
22      484
23      529
24      576
25      625
26      676
27      729
28      784
29      841
30      900

cd to your unixplay/ directory and type

squares

You should see the output on the screen.

Now redirect the output to a file:

squares > squares.out

Verify that the output file is there with ls and then have a look at it with less.

When doing scientific computing we will often use this method to catch the output of a program in a file.

Now type the squares command again, but this time give it arguments to change the range of the integers.

squares 20 30

Does it work correctly as described above?

Now catch the output in the same file as above, again:

squares > 20 30 squares.out

View this file again with less. Notice that the previous squares.out file has been overwritten. It is important to remember that when using > by itself, the redirection file is first cleared before the output is collected. Your previous file "squares.out" has been lost and replaced with the new output.

Append to file:   >>

If instead of using > for redirection, we can use the double redirect sign: >> to append to an existing file, as opposed to overwritting it. This is very handy if we do a computation over and over and want to add the result to the end of the output file after each iteration.

Look at your squares.out file with less. It should contain the squares of the integers from 20 to 30.

Now, do

squares 31 40 >> squares.out

Verify that the file squares.out now contains the integers and squares from 20 to 40.
The last command appended the results from 31 to 40 to the existing file "squares.out" which already had 20 through 30.

Exercise 3a

Using the above output redirection method, create a file (called sq0_50.out)

with the squares of the integers from 0 to 50.

Redirecting the Input:   <

We use the < symbol to redirect the input of a command from somewhere else, usually a file.

Download this small program called "cubes.exe", and save it in your unixplay/ directory: cubes.exe

Now, run the program by typing

cubes

The "cubes" program will prompt you for input. When it asks

enter min:

type 0 and hit [Return].

It next askes you to

enter max:

Type 20 and hit [Return]

The program outputs the cube (n³) of the integers from 0 to 20.

0       0
1       1
2       8
3       27
...

This program is a little different than squares. Squares used "command line arguments": items given on the command line after the name of the program that are passed to the program to modify its behavior. (Creating a program to use command line arguments is a little more complicated than the simpler approach used in cubes.)

cubes prompts you for input, by asking you questions to which you input data.

We can put this input in a file, then redirect the input to cubes from that file.

First, let's put the input numbers in a file. We can use cat as we did above. Without specifying an file, cat will read from standard input (the keybord), and we can use > to redirect the output to a file.

We simply want to have a file that contains the two numbers that we would input to cubes if we were entering the inputs by hand from the keyboad (standard input). In our example above we used 0 and 20. So, type cat with no input file specified, and redirect the output to a file called in.cubes which we will use momentarily.

cat > in.cubes

cat is waiting for you to give it something from standard input. So, type

0
20
^D (Control + D)

Now, do an ls to see that your new file, in.cubes is there. Have a look with less. It shoudl contain two lines: 0 and 20.

Now use this file as the input to cubes by redirecting input from in.cubes:

cubes < in.cubes

This should produce the integers from 0 to 20 and their cubes. Does it?

Exercise 3b

Use input redirection in this way to produce the cubes of integers from 0 to 50.

< and >

You can use both input and output redirection at the same time!

command < in.file > out.file

will make command read input from the file in.file and catch the output in the file out.file.

Exercise 3c

Use input redirection in this way to save the cubes of integers from 0 to 50

to a file called cubes_0_50.out.

The Pipe: |

Previously we learned a few really useful commands: grep, sort, and wc. Be assured, there are many more.

It turns out that we can string these together so that the output of one command serves as the input to the next, using the "pipe". The pipe is the vertical bar "|" on your keyboad, above the "\" backslash.

Recall our file, scifi_list.txt, which contains the year of publication, rank, author, and title of 100 of the best science fiction works.
Let's ask some questions:

Suppose you want to know the rank in popularity (column 2 in scifi_list.txt) of the books by Robert Heinlein. Have a look at the file to remind yourself what's there. Heinlein's books are mixed in with the rest--randomly.

grep comes to the rescue:

grep Heinlein scifi_list.txt

produces

1961    6       Heinlein, Robert A      Stranger in a Strange Land
1973    41      Heinlein, Robert A      Time Enough For Love
1958    93      Heinlein, Robert A      Have Space-Suit - Will Travel
1957    84      Heinlein, Robert A      Citizen Of the Galaxy
1951    88      Heinlein, Robert A      The Puppet Masters
1956    80      Heinlein, Robert A      The Door Into Summer
1966    17      Heinlein, Robert A      The Moon is a Harsh Mistress
1959    12      Heinlein, Robert A      Starship Troopers

To get these ordered by popularity rank, you could redirect the output of the grep command to a file, then sort the file.

grep Heinlein scifi_list.txt > heinlein.out
sort -k 2 -n heinlein.out

However, the pipe allows you to do this in one step:

grep Heinlein scifi_list.txt | sort -k 2 -n

yields

1961    6       Heinlein, Robert A      Stranger in a Strange Land
1959    12      Heinlein, Robert A      Starship Troopers
1966    17      Heinlein, Robert A      The Moon is a Harsh Mistress
1973    41      Heinlein, Robert A      Time Enough For Love
1956    80      Heinlein, Robert A      The Door Into Summer
1957    84      Heinlein, Robert A      Citizen Of the Galaxy
1951    88      Heinlein, Robert A      The Puppet Masters
1958    93      Heinlein, Robert A      Have Space-Suit - Will Travel

We "piped" the output of the grep command to the sort command, (where we used "-k 2 -n" in order to sort on the second "k"olumn in numerical order).

We can even chain pipes together:

grep Heinlein scifi_list.txt | sort -k 2 -n | wc

produces simply

      8      73     402

The output of the sorted grep contains
8 lines, 73 words, 402 bytes.

Exercise 3d

Using the pipe and one line of code (as above), produce a list of the

books of Arthur C. Clarke, ordered by year of publication.