NAME
UNIX_GUIDE - A self directed guide to improving your use of the UNIX
system
DESCRIPTION
This manual page is designed to build on the basic concepts and opera-
tions covered in introductory tutorials and is focused on techniques
that will help you make more effective use of UNIX systems.
OBJECTIVE
The material covered in this manual is intended to focus on a useful
subset of the available features of UNIX shells and a number of UNIX
commands. While not comprehensive, this subset probably represents more
functionality than many people use on a regular basis.
The manual pages for individual UNIX commands cover all the details, but
they are dense reference materials with few examples. There are a
comprehensive shell guidebooks but, as a rule, they do not differentiate
the important either from the trivial or the arcane.
Like traditional UNIX manuals, this one covers a lot of material and in
many cases simply references utilities which warrant considerably more
coverage. Unlike most manuals, this one does include examples and exer-
cises. Nonetheless, you should not expect to absorb all of the material
in a single pass.
OUTLINE
- The different UNIX shells
- Interactive use of the shell
- Defining a default login environment
- Writing and debugging shell scripts
- Managing disk space
- Text processing
- Using UNIX Manuals
NOTE
In the examples that follow, exercises are identified by an EXERCISE
header. There are a number of example commands other than the exercises
which you may or may not wish to try. Executable commands are proceeded
with "% "; Multi-line commands start with "%% " and go through the next
"%<cr>". If a command does not make sense, take a break and read the
UNIX man page for that command!
The SHELL[s]
The UNIX system user interface is a separate, command interface program
called a shell. Since the shell is not an integral part of the operat-
ing system, it is possible to have more than one. Over the years,
several shells have been developed. They include:
Bourne shell (sh) - The original
C Shell (csh) - University of California at Berkeley's contri-
bution
Korn Shell (ksh) - The latest from Bell Labs
Tcsh - A recent extension to the C Shell
Bash - Free Software Foundation's sh replacement
It is possible to modify your login shell by using the change shell com-
mand "chsh" or ("ypchsh" on YP workstation clients) available on many
UNIX systems; these commands accept any of the shells listed in the file
"/etc/shells".
In addition to their user interface function, shells function as a high
level programming language. You can write programs, called scripts,
which are interpreted by the shell.
The original shell (sh) is typically used for system maintenance
scripts. The major features sh offers is that it provides shell pro-
cedures and can trap signals and errors and then execute clean up code.
The later feature can be particularly critical when modifying system log
and data files in scripts that are run automatically by the system and
may need to handle unusual conditions.
The Korn shell is frequently the choice for people who've spent consid-
erable time working with Bourne shell scripts and have come to prefer
its syntax to that of the C shell (csh). This shell combines Bourne
shell syntax with command line editing, command history, job control,
and command aliases.
The C Shell (csh) is almost certainly the most commonly used shell. It
offers command history, aliasing, job control, and control flow syntax
similar to that of the C programming language.
The tcsh, a recent development from Ohio State, builds a large number of
additional features into csh. The most significant of these is the pro-
vision of a line-editor that can access the command history buffer and
edit the command line.
It is worth noting that the Bourne and C shells are supported by virtu-
ally every UNIX vendor and thus are part of the software available when
a new system is installed. The other shells, korn, tcsh, and bash are
available separately as "C" source code and must be built and installed
on each system. Contact your system administrator if your system does
not have a shell that you would like to use.
Sh is primarily used for scripts and is rarely used as a command inter-
preter. An example script (template.sh) has been included at the end of
your handout.
C Shell (csh)
As stated earlier, csh is the most commonly used shell. While sh is
frequently required for system shell scripts, csh also works well as a
programming language. Note that the information supplied here also
applies to tcsh since it is a superset of csh.
Interactive Shell Use
Filename substitution
There are several ways to specify filenames to the shell on a command
line. The obvious way is to type the filename explicitly. However, it
is frequently desirable to identify files that match a pattern. Pat-
terns include:
* Match any (zero or more) characters
? Match any single character
[...] Match any single character in the enclosed list or
range. A range is any two characters in the
ascii(7) character set.
{s1,s2,...} Expand to each string in the list. Note
that these strings are not matched against existing
filenames; thus this mechanism is also useful for
generating multiple arguments on a command line.
EXERCISE - See note on first
Filename Substitution
%% ls /usr/lib
ls /usr/lib/*.a
ls /usr/lib/lib?.a
ls /usr/lib/[begv-y]*
ls /usr/lib/{lib?.a,[beg]*}
%
Note use of list expansion to specify a file
template.sh that doesn't exist until cp creates it.
% cp {../bin/,}template.sh
C Shell History
earlier commands in building new commands. Without history, users are
often forced to type in many repetitive commands to accomplish a given
task. In short, shell history is a way to reduce duplicate typing
effort without building aliases or scripts for a single session.
EXERCISE
% history
Re-run previous command
% !!
Show, but do not execute last ls command. This is typically used to
redisplay a command that you wish to modify slightly for re-execution.
% !ls:p
Replace a string (usr/lib) in last command
with another (lib)
% ^usr/lib^lib
There is a lot more to csh history. If this is the shell that you use
it is worth reading the csh manual and learning more suffix commands
(:r, :x, :h, :s, etc.). However, another shell (tcsh) supports a
totally different way to access command history.
Tcsh History
Tcsh supports command line editing as well as the ability to scroll
through the command history buffer as if it were a line editor. Many
people find this to be a more natural history mechanism.
In the following exercise, we save history to ~/.history, then run tcsh.
It reads in this file on startup; this feature is normally used to
recover history from previous login sessions.
Once tcsh is started it accepts emacs control characters to edit the
command buffer and command line. It is also possible to setup a tcsh
startup file (.tcshcf) so that vi line mode editing is used instead.
EXERCISE
% history | awk '{print $3}' > ~/.history
% /usr/local/bin/tcsh
Use M-p, M-n to search the history list for patterns, i.e.:
% ls<M-p>
For additional commands see the tcsh manual (This is included at the end
of the tutorial handout).
Modifying default login environment
Many users invest a moderate amount of time customizing their login
environment. This effort can result in a significant time savings.
However, when customizing your environment, it is best to avoid redefin-
ing existing commands in your execution path. Even if you don't use the
original command at present, doing this often creates confusion later on
and may make it difficult for a consultant, a terminal assistant, or
another user to help you work out a problem using your login session.
In addition to developing personalized commands (aliases and scripts),
customization includes choosing a login shell, organizing your home
directory, and defining environment and shell variables that affect the
shell or application programs.
When changing your login environment *always* test the new setup using
rlogin or telnet to verify that your changes have not made it impossible
to login before logging out!
Changing login shell
As mentioned before, it is possible to change your default login shell
to the shell of your choice with ypchsh or chsh. If the command
"domainname" exists and returns anything but a null line, your system is
probably using Network Information Services (originally called Yellow
Pages or YP). In this case use ypchsh, otherwise use chsh.
% chsh tcsh
Aliases
It is possible to assign a name (called an alias) to any command
sequence that you can put on a line. These are typically defined in
".cshrc". A few common alias abbreviations are listed below. Note that
aliases can be included in other aliases; this is the reason that ls is
explicitly referenced as /bin/ls in these examples.
%% which cx h ls ll
alias cx 'chmod +x'
alias h history
alias ls '/bin/ls -F'
alias ll '/bin/ls -l'
%
Organizing home directories
If you build your own scripts and programs, they are typically put in a
home directory. This presents an organizational problem because binary
programs only run on one architecture, they must be compiled separately
for each separate architecture, and the compiled binary programs for a
given architecture only need to be visible when the user is logged into
a system of that type.
The recommended solution is to use the "arch" command to identify the
hardware platform and build a separate bin directory for each architec-
ture that you use. Compiled programs go in these directories while shell
scripts can go in the top level bin directory. For example, if you use
sun 3's, sun 4's(or Sparc stations), and Decstation's your bin directory
would look like this:
% arch
% mkdir ~/bin
% mkdir ~/bin/sun3
% mkdir ~/bin/sun4
% mkdir ~/bin/dec3100
Then your path can be modified to select the correct directory when it
starts up. For example:
% arch
% set path = (~/bin ~/`arch`/bin /usr/ucb /usr/bin /usr/local/bin .)
If your system does not have an "arch" command, it almost certainly will
have a "uname" command which can be used as follows:
% echo `uname -s`.`uname -m`
C Shell variables
Shell variables have names and values. As a special case they may be
defined with an empty or null value; such variables are typically used
as boolean toggles inside shell scripts. Variables can be created or
destroyed with the set and unset commands. Numerical values may be
operated on as numbers with the builtin '@' command which provides
expression evaluation. Variable values may also be defined as token
strings and as token lists. There are some predefined variables that
have special meanings either to the shell or to other application pro-
grams. Several of these are described below.
C Shell variables with special meanings
Variables typically set in .cshrc
cdpath List of directory "roots" in addition to "." for cd
history 100 Number of commands (100) to keep in command buffer
filec When set enables filename completion
ignoreof Disables logout on spurious EOF
path path-list Build hash table of available commands when
set. Sets PATH environment variable, so
path persists to sub-shells. Best defined in
.login, as opposed to .cshrc
savehist # When defined, csh saves history to file on exit
umask mode Used to permanently modify system default file
permissions
The difference between .login and .cshrc is that the former is loaded
only once when you login. The ".login variables" indicated above either
modify the environment which is passed to sub-shells or are only mean-
ingful in the login session. Shell commands in .cshrc are executed each
time a sub-shell is invoked(e.g. when you execute a script, or use rsh
to execute a command on a remote system.) It is worth noting that
.chsrc is loaded before .login.
Writing shell scripts
If you plan to spend a lot of time building scripts, it is probably best
to learn to use perl; a scripting language which incorporates a number
of system(2) and library(3) calls that would normally require the
development of C programs. While the following introduction also
applies to perl, it is based on csh and sh scripts.
Specify the execution shell
The first thing to specify when writing a shell script is which shell
program should execute the script. By default, the bourne shell is used
to execute any text file unless the first character in the file is a
'#'. If the second character (after a '#' is *not* a '!', then the csh
is used. If the second character *is* a '!', then the next string iden-
tifies the program that should be used to execute the script. Some
example first lines:
#* This is a csh script
#!/bin/sh This is a sh script
#!/usr/local/bin/tcshThis is a tcsh script
#!/usr/local/bin/perlThis is a perl script
The second thing to remember is that the script file must be executable.
This is done by changing the file mode.
%chmod +x template.sh
Shell variables in scripts
Shell variables are heavily used in scripts. The following exercises
demonstrate some features of accessing different kinds of variables
(null, string, and lists). There is much more to shell variables than
what can be covered here. If you develop many scripts it will be neces-
sary to study the variable substitution section of the shell that you
are using.
EXERCISE
Using null variables
%% if ($?debug) echo debugging
set debug
%% if ($?debug) echo debugging
%
echo $v
set v = "This is a token string with spaces"
echo $v
%
Using variable lists. Note elements of a variable
wordlist can be accessed by array subscripts and subscript ranges.
%%
set list = (A token list with 6 elements)
echo $v[2]
echo $v[5-]
%
Control Flow
Control flow includes conditional expressions (if, then, else, endif),
loop constructs, case, and switch statements. Again, these topics are
covered in the shell manuals. The example scripts (template.sh and
template.csh demonstrate some common usages.
I/O Redirection
You should be familiar with the commonly used I/O redirection commands:
|, >, >>, <
However, it is occasionally useful to directly include several lines of
text in stdin inside a shell script. In addition to the following exam-
ple see the sample bourne shell script provided.
%% cat <<Finished
This message could define a mail message
or
A warning message.
Note: Variable expansion works, so we can include $user, etc..
Finished
%
FILTERS
A filter is a program that reads standard input, modifies it as
sort, tee, tr, uniq, and wc. You should familiarize yourself with the
manual pages for these commands.
The following exercises are intended to show use of filters and to give
some familiarity with the concepts of stringing several commands
together to obtain useful information from raw data. The synopses for
each filter is included below just before the command is used in an
exercise for the first time.
NAME
tee - replicate the standard output
SYNOPSIS
tee [ -ai ] [ filename ] ...
NAME
sort - sort and collate lines
SYNOPSIS
sort [ -bdfiMnr ] [ -tc ] [ sort-field ...] [ -cmu ]
[ -o[ ]output-file ] [ -T directory ]
[ -y kmem ] [ -z recsz ] filename...
sort-field ::= +sw -ew
Sort rwho output by idletime
EXERCISE
% rwho | sort +5 | tee rwhof
Resort "rwhof" by user
Note: Never pipe sort output to input file
% sort +0 -o rwhof{,}
Note: equivalent to "sort +0 -o rwhof rwhof
Unique sort on user (first field)
Note: Deletes login on multiple hosts
% sort +0 -1 -u rwhof
same as
% rwho | sort +0 -1 -u
NAME
sed - stream editor
SYNOPSIS
sed [ -n ] [ -e script ] [ -f sfilename ] [ filename ]...
Strip tty info from field 2
EXERCISE
% sed -e "s/:[^ ]* //" rwhof
Sorted (by login) list of people logged into each system
Note: The login date and idletime are meaningless
% !! | sort +0 -2 -u
or
% rwho | sed -e "s/:[^ ]* / /" | sort +0 -2 -u
NAME
awk - pattern scanning and processing language
SYNOPSIS
awk [ -f program-file ] [ -Fc ] [ program ] [ variable=value
... ] [ filename...]
Sorted (by login) list of people logged into each system
EXERCISE
% !! | awk '{print $1, $2}'
or
% rwho | sed -e "s/:[^ ]* / /" | sort +0 -2 -u | awk '{print $1, $2}'
Now if this is information that you want to gather regularly, this can
be made an alias in "~/.cshrc"!
% alias ru 'rwho | sed -e "s/:[^ ]* / /" | sort +0 -2 -u '
Debugging Shell scripts
There are three options to sh and csh which help with debugging shell
scripts. These are: -v, -x, and -n.
-n No-execute mode. This mode is very useful
for syntax checking.
-v Verbose mode. Display command before execution.
-x Expand mode. Display command after filename and
variable expansion.
EXERCISE
Test and execute the scripts template.sh and template.csh.
% csh -vn template.csh foo
% sh -vn template.sh foo
Try csh syntax on sh script to see error output.
% csh -vn template.sh foo
Run scripts and try other arguments
% template.sh -h
% template.sh foo
% template.sh -h
% template.sh foo
Managing Files and Disk Space
Revision control is an important file management facility. There are
two systems available, RCS and SCCS. Sccs is available as part of
most vendor's OS release. However, RCS is freely available and is
more widely used.
Revision control systems were originally developed for checkpointing
and saving multiple versions programs during their life cycle. However,
it is a good idea to use revision control for any important document
(e.g. papers and other long term project work). The benefit of revision
control is that it reduces the potential damage from loss of any single
file. In addition, there are mechanisms to restore and examine earlr, RCS is freely available and is
more widely used.
Revision control systems were originally developed for checkpointing
and saving multiple versions programs during their life cycle. However,
it is a good idea to use revision control for any important document
(e.g. papers and other long term project work). The benefit of revision
control is that it reduces the potential damage from loss of any single
file. In addition, there are mechanisms to restore and examine earl