[TriLUG] essential Linux skills?

[Aaron S. Joyner]

Douglass Davis wrote:

> I am teaching Linux this semester.   If some one told you that they
> knew about Linux, what would be some essential skills that you would
> expect them to have, or things you would expect them to know.  I don't
> mean expert level Linux knowledge but just intermediate. 
>
> This message is especially for those of you who have ever hired some
> one for a position involving Linux.
>
> Thanks for your help.
>
>
> --
> http://www.douglassdavis.com/
>
Some basic things I would suggest:
- Man pages are perhaps the most important thing you can teach.  Show
them how to read a man page, and explain that every word in a man page
is important, and meant to be meditated on.  They're usually extremely
dense, but that's because every word caries some meaning.  If you can
get them to read man pages to solve they're problem, I'd bet they'll
learn everything twice as fast, and 4 times as well.
- File system permissions, how they work, both modal (755) and verbose
(o+rwx), how to read the output of ls, how to set permissions with chmod
(late in the class, touch on sticky bit, setuid, setgid, and how they
affect both files *and* directories)
- Basic UNIX filesystem concepts (everything is a file, hard drives are
files (/dev/hda), serial ports are files (/dev/ttyS0), etc, maybe block
device vs char device)
- Basic and important directories found in /, such as /etc, /home, /var,
/usr, /proc, /tmp - and what's usually stored under each, and why
- Cover some important and useful sub-dirs, such as /usr/share,
/usr/share/doc/, /usr/src, /var/log, /var/run...
- Cover the standard unix user setup, /etc/passwd and /etc/group,
/etc/shadow, and perhaps how /etc/skel is used in account setup. 
Possibly mention that in a modern distribution, through pam and
nsswitch, these concepts are extensible and can be centralized to allow
centralized control and management of users, much like active directory
does in windows.  You won't have time to teach ldap and kerberos
thoroughly, but at least make the point that it's possible.
- Cover man pages again
- Basic familiarity with a shell.  Just pick one (I'd suggest bash), and
explain to them some of the basics of commands, while being sure to
explain the difference between shell built-ins, vs system commands, i.e.
cd, export and set vs. ls, cat and grep.  Explain how echo can be
tricky, because traditionally they're commands but bash also has a
built-in version.  Explain shell variables vs environment variables, at
least at a cursory level, and also touch on inheritance of environment
variables.  Be sure to mention "special" variables, with some thoroughly
explained, such as $PATH, $CWD, etc.  Teach them some neat tricks with
the shell that save them time, like a few good aliases, or one-liner
simple scripts (while read line; do foo; done; # this is often a
favorite).  Explain output redirection, STDIN vs STDOUT vs STDERR, and
how to chain these together in the shell to take input from one place
(cat) and pipe it through another place (sed) and dump it out stdout
into a file (/tmp/foo).  Once you've shown them the basics of using the
shell as an interactive tool, then show them how to build on that
briefly to build very simple scripts to automate repetitive tasks.  Be
sure you at least gloss the primary programming constructs for
conditionals, loops, etc (if, for, while, etc - depending on how far
towards programming you want to go).  A really good combo exercise might
have them do something moderately tedious in the process of learning an
editor (vi), and teach them how to improve on that task in the editor
(more on that in a moment), then show them how to do the entire same
task in shell as an automated command, and call it from their ~/bin
directory.  If you can teach them to understand the shell, and show them
that it is good, you'll make life-long *NIX users out of them.
- Cover man pages again
- Teach them an editor, preferably one that is universally available. 
That pretty much means you teach vi, or you teach ed.  Most people
consider that an easy decision.  :)  Then again, there is merit in
teaching the full feature set of vim, or perhaps even in teaching
emacs.  Generally though, as far as basic skill set goes, if you can't
at least *use* vi, you're kinda in trouble on a minimal unix box. 
Teaching basic vi also goes very well with the rest of the unix
philosophy, in that it's very-much designed for the super-efficiency of
lazy typists.  Bonus points for being, or bringing in someone, who can
edit things *ridiculously* fast in vim, and wowing the students with the
tricks that can be done in the hands of an experienced vim user. 
Consult the list when searching for this person, if you can not fill the
role.
- Be sure to cover the basic suite of *NIX command line tools.  Those
classics we all take for granted, that will allow them to construct the
shell scripts in the previous example.  Some must-haves, cat, grep, cut,
wc, sort, uniq, tr, sed, mktmp, (this list could go on forever) and some
more advanced bits such as awk, or even perl one-liners, depending on
time and quality of students.
- Teach them about dot-files in their home directory, and how some files
share intelligent defaults from a common dotfile (.netrc, for example)
and how most daemons have their own dotfile or directory (.ssh is a good
one)
- Ensure they understand the myriad of remote access and file copy
options.  Touch on telnet, ftp, rcp, rlogin, .rlogin (that's dot
rlogin), and /etc/hosts.equiv - explain why they are bad.  Then
introduce ssh, scp, and ssh host keys.  Be sure to explain the rsa key
functionality of ssh, at least in terms of what it can allow you to do
with un-prompted logins, and how this can be used to do things securely
while unattended, and allow for secure authentication of hosts, not just
users.
- Be sure they understand the linux init process.  Try to give them at
least a cursory understanding of how the system goes from ground zero
(the bootloader), through the kernel, which starts init, which starts a
single shell script, which (in most distributions, at least) fans out
into a group of shell scripts and config files, eventually launching all
of the programs started at bootup, including things like the login
gettys.  Explain how all processes are children of init, and that they
all inherit certain things set early in the boot process (such as path,
umask, etc).  Show them that if they get lost, if they can't understand
where something is coming from, starting at, etc it's only a matter of
walking through the shell scripts, from beginning to end (you probably
can't do it in it's entirety in the time alloted for something as large
as RedHat, but the principal should at least be understood of how the
system works).  This lays the ground work for being able to troubleshoot
_all_manner_ of system start up problems, which anyone tinkering with
Linux anywhere beyond the cursory user level will inevitably run across
one day.
- Explain daemons.  Show their basic config files, explain a few of the
common linux config file formats (key=value, apache configs, etc) and go
into some more depth for some of the more common daemons they might deal
with (at least a cursory understanding of Apache, Samba, and maybe BIND
would be helpful).  Explain cron, probably in some detail, and how it's
commonly used.
- Explain the TCP/IP stack.  You don't have to teach a networking class,
but expose them to ifconfig and route, explain how to do the common
things such as set an IP on an interface, set a default route, list
those back out from ifconfig and route -- maybe even teach the ip tool
in addition to (not instead of) ifconfig and route.  Explain network
interfaces, how they're named, and the fact that they break the usual
convention of being a file (there is no /dev/eth0...).
- Here you should make sure you've touched on man pages
- Explain the "special" directory structures, specifically /proc, maybe
devfs, but only in the context of broader dev devices and "how it used
to be" / is on most unices vs. how it is on a very modern linux.
- Maybe explain the resolver structures, and everything keyed out of
nsswitch.conf.  Specifically, how is a dns lookup handled (the machine
looks in nsswitch.conf, which probably says "hosts dns", then looks at
/etc/hosts, then looks at /etc/resolv.conf, then talks to your
nameserver based on resolv.conf's config.  Explain how the same is done
for /etc/passwd lookups, by looking at nsswitch, and then possibly to an
external source such as ldap, nis, etc.
- Maybe explain PAM, not in intricate detail, but gloss it and provide
references.
- Explain ps, and how to look at running processes.  This is good to do
in conjunction with basic shell scripting, ps ax | grep.  Explain
signals, and the kill command.  Explain how signals are handled, and at
least SIG TERM, KILL, HUP, and why all of them but KILL and STOP are
handled by the process, and the other two are handled by the kernel.
- Maybe gloss over the concepts of selinux, and role-based access
controls.  I wouldn't go too deep into this, but explain how "root is
god" doesn't really go as far in an selinux system.

After you've hit the above topics, with sufficient depth, any thing else
is gravy.  You can take the class in one of several directions, towards
more programming (show more shell scripts, do more advanced shell
tricks, tackle perl or python), or towards more sysadmin directions
(setup daemons, configure them, make them do your bidding) or towards
more userland eye candy (setup gnome or kde, change the window manager
to something exotic, enlightenment perhaps, setup configs for the window
manager, etc).  But don't get too lulled into this part of the class too
fast.  Do note that I leave userland eye candy off my list of essential
unix features, because being able to configure gnome or open OpenOffice
and type up a document are no-brainer activities that any windows user
can manage.  Exploring the GUI tools that come with <your distro of
choice> is an easy, one afternoon process, and isn't likely to provide
you any career opportunities, or help you later in life.  These aspects
of the Linux GUI are still highly in flux, and likely to be out-dated by
the time your students finish the class.  Most of the points from
earlier in my message haven't changed in any fundamental way in the last
25 years.

As a professor, you should probably prepare by reading a few key books,
I'd specifically recommend The Art of UNIX Programming.  This will
provide you with the background (if you don't already possess it), to be
able to give insights into *why* the unix systems you're teaching work
the way they do, as well as a whole host of interesting historical
anecdotes that can make the class interesting while putting things in
proper context.  Note that this book is freely available under the
creative commons license, and can be pulled by a simple google search,
or picked up at your local Barnes and Nobles for under $40 conveniently
printed on dead trees.

Let me wrap all this up by saying that I've recently put a lot of
thought into all of this, as I've been doing phone screens of system
administrators for Google.  I'd say what I've described above
constitutes the basics of a person who would make a good Linux user,
who's got a good foundation laid to work towards being a sys admin or a
software engineer.

Best of luck with your class, I hope this has been worth the time spent
reading it,
Aaron S. Joyner
System Administrator
Google, Inc.

PS - If anyone has time to throw this knowledge into the wiki, please
do!  Shoot an email reminder when you've got it up there and I'll try to
pick a few sub-sections to write up details for.