Cluster: C3 Tools INSTALL

         C3 version 4.0:   Cluster Command & Control Suite
           Oak Ridge National Laboratory, Oak Ridge, TN,
     Authors: M.Brim, R.Flanery, G.A.Geist, B.Luethke, S.L.Scott
                 (C) 2001 All Rights Reserved

                             NOTICE

 Permission to use, copy, modify, and distribute this software and
 its documentation for any purpose and without fee is hereby granted
 provided that the above copyright notice appear in all copies and
 that both the copyright notice and this permission notice appear in
 supporting documentation.

 Neither the Oak Ridge National Laboratory nor the Authors make any
 representations about the suitability of this software for any
 purpose.  This software is provided "as is" without express or
 implied warranty.

 The C3 tools were funded by the U.S. Department of Energy.


I. REQUIRED SOFTWARE
--------------------

Before C3 can be installed on a system, you must ensure that the following
software is installed on your system. The following seven software packages
are required:  the C3 tools suite, Rsync, SSH (or OpenSSH), Python, and Perl.
You must also configure that system to support host name resolution of the
machines listed in the configuration file (either through DNS or /etc/hosts).
 Finally, if you wish to use the C3 pushimage command, which pushes system
images across a cluster, you must install SystemImager.

Instructions for obtaining each of these software packages are given below.

  C3 tools may be obtained from http://www.csm.ornl.gov/torc/C3

  Rsync, Perl, SSH, and Python should be included with your distribution.
  If they are not then download the source or binaries from their respected
  web sites.

  Perl may be obtained from http://www.perl.com
        C3 requires 5.005 or greater

  Python may be obtained from
  http://www.python.org/
        C3 version 3 requires Python 2.0 or greater

        additionally C3 requires either the binary or a link to the python
        interpreter to  be in your path (and that it be named python2). To
        check it type "python2 -V" and make sure you get output (the current
        version of python being run). If you do not get any output then you must
        find where the python library is on your machine and create a link to
        the binary. Such as "ln -s /usr/bin/python /usr/bin/python2" if
        /usr/bin/python is where your python binary is located and /usr/bin
        is in your path.

  SystemImager may be obtained from
  http://www.systemimager.org/


II. C3 INSTALLATION
----------------
NOTE: if you are wanting to use the scalable model of the C3 tolls then follow
steps A and B, read C as it still pertains to the scalable model, then see the
README.scale file for the scalable instructions.

A. pre-install
        Begin by making sure that Rsync, OpenSSL, OpenSSH, PERL, and
        Python are installed.  Install Systemimager, if needed.  Install DNS
        or /etc/hosts as needed, and make sure that hostname resolution is
        supported.

        Directions for downloading each of these packages are given in
        Section I above.  Perl, Python, Rsync, OpenSSH, and OpenSSL are included
        with most distributions

        You will need root access to install these packages on your system.
        Follow the instruction in each package if you need to install them.

B. C3 install
        After you complete the pre-install (step A), install the Cluster
        Command & Control (C3) tools. Begin by untar'ring the C3 package
        and running the install script.  The install script places the C3
        scripts in /opt/c3-4 and the man pages in the appropriate directory.

        The C3 install script installs the C3 command suite, but does not
        configure the commands or any local clusters for operation.
        Directions for the remaining tasks are given below.

C. C3 configuration
        Specific instances of C3 commands identify their compute nodes with
        the help of **cluster configuration files**:  files that name a set
        of accessible clusters, and that list and describe the set of
        machines in each accessible cluster.  Cluster configuration files
        are accessed in one of two ways:

        -.  explicitly:  an instance of a C3 command names a specific
        configuration file, using a command-line switch.

        -.  implicitly:  an instance of a C3 command fails to name a specific
        configuration file, and the command defaults to the list of cluster
        descriptions given in /etc/c3.conf.

        When you install C3, you should create a default configuration file
        that is appropriate to the site.  This file, which should be named
        /etc/c3.conf, should consist of a list of **cluster descriptor
        blocks**:  syntactic objects that name and describe a single cluster
        that is accessible to that system's users.

        The following is an example of a default configuration file that
        contains exactly one cluster descriptor block:  a block that
        describes a cluster of 64 nodes:

                cluster local {
                        htorc-00:node0  #head node
                        node[1-64]      #compute nodes
                }

        Cluster description blocks consist of the following basic elements:

        -.  a **cluster tag**:  the word "cluster", followed by a label,
            which assigns a name to the cluster.  This name--here, "local"--
            can be supplied to C3 commands as a way of specifying the cluster
            on which a command should execute.

        -.  an open curly brace, which signals the start of the cluster's
            declaration proper.

        -.  a **head node descriptor**:  a line that names the interfaces
            on the cluster's head node.  The head node descriptor shown here
            has two parts:

            -.  The string to the left of the colon identifies the head
                node's **external** interface: a network card that links
                the head node to computers outside the cluster.  This string
                can be the interface's IP address or DNS-style hostname.
            -.  The string to the right of the colon identifies the head
                node's **internal** interface: a network card that links the
                head node to nodes inside the cluster.  This string can be
                the interface's IP address or DNS-style hostname.

            Here, the head node descriptor names a head node with an external
            interface named htorc-00, and an internal interface named node0.

            A cluster that has no external interface--i.e., a cluster that is
            on a closed system--can be specified by either

            -.  making the internal and external name the same, or
            -.  dropping the colon, and using one name in the specifier.

        -.  a list of **compute node descriptors**:  a series of individual
            descriptors that name the cluster's compute nodes.

            The example given here contains exactly one compute node
            descriptor.  This descriptor uses a **range qualifier** to
            specify a cluster that contains 64 compute nodes, named node1,
            node2, etc., up through node64.  A range qualifier consists of
            -.  a first, nonnegative integer, followed by
            -.  a dash, followed by
            -.  a second integer that is at least as large as the first.

            In the current version of the C3 tools et, these range values are
            treated as numbers, with no leading zeroes.  A declaration like

                    cluster local {
                        htorc-00:node0  #head node
                        node[01-64]     #compute nodes
                    }

            expands to the same 64 nodes as the declaration shown above.  To
            specify a set of nodes with names like node01, node09, node10, ...
            node64, use declarations like

                    cluster local {
                        htorc-00:node0  #head node
                        node0[1-9]      #compute nodes node01..node09
                        node[10-64]     #compute nodes node10..node64
                    }

        -.  a final, closing curly brace.

        Configuration files that specify multiple clusters are constituted as
        a list of cluster descriptor blocks--one per accessible cluster.
        The following example of a cluster configuration file contains three
        blocks that specify configurations for clusters named local, torc,
        and my-cluster, respectively:

                cluster local {
                        htorc-00:node0  #head node
                        node[1-64]      #compute nodes
                        exclude 2
                        exclude [55-60]
                }

                cluster torc {
                        :orc-00b
                }

                cluster my-cluster {
                        osiris:192.192.192.2
                        woody
                        dead riggs
                }

        The first cluster in the file has a special significance that is
        analogous to the special significance accorded to the first
        declaration in a make file.  Any instance of a C3 command that fails
        to name the cluster on which it should run executes, by default, on
        the first cluster in the configuration file.  Here, for example, any
        command that fails to name its target cluster would default to local.

        The cluster configuration file shown above illustrates three final
        features of the cluster definition language:  **exclude qualifiers**,
        **dead qualifiers**, and **indirect cluster** descriptors.

        **Exclude qualifiers** allow nodes to be excluded from a cluster's
        configuration: i.e., to be identified as offline for the purpose of
        a command execution.  Exclude qualifiers may only be applied to
        range declarations, and must follow immediately after a range
        declaration to which they are being applied.  A series of exclude
        declarations is ended by a non-exclude declaration, or the final "}"
        in a cluster declaration block.

        An exclude qualifier can be written in one of three ways:
        -.   "exclude n", where n is the number of a node to exclude from the
             cluster;
        -.   "exclude[m-n]", where m, m+1, m+2, ..., n-1, n is the range of
             nodes to exclude; or as
        -.   "exclude [m-n], which has the same effect as "exclude[m-n]".
        Note that a string like "exclude5" is parsed as a node name, rather
        than as an exclude qualifier.

        In the above example, the two exclude qualifiers have the effect of
        causing node2, node55, node56 node57, node58, node59, and node60 to
        be treated as offline for the purpose of computation.

        **Dead qualifiers** are similar to exclude qualifiers, but apply to
        individual machines.  In the example given above, the machine named
        "riggs" in the cluster named "my-cluster" is excluded from all
        computations.

        "Dead", like "exclude", is not a reserved word in the current version
        of the C3 suite.  A specification block like

                cluster my-cluster {
                        alive:alive
                        dead
                }

        for example, declares a two-machine cluster with a head node named
        "alive" and a compute node named "dead".

        An **indirect cluster descriptor** is treated as a reference to
        another cluster, rather than as a characterization of a cluster per
        se.  In the example shown above, the descriptor

                cluster torc {
                        :orc-00b
                }

        is an indirect cluster descriptor.  An indirect descriptor consists
        of

        -.  a cluster tag, followed by,
        -.  an **indirect head head node descriptor**, followed by
        -.  an empty list of compute node descriptors.

        An indirect head node descriptor consists of an initial colon,
        followed by a string that names a **remote** system.  This name,
        which can either be an IP address or a DNS-style hostname, is checked
        whenever a C3 command executes to verify that that the machine being
        referenced is **not** the machine on which that command is currently
        executing.

        A command that is destined for an indirect cluster is executed by

        -.  first forwarding that command to the remote cluster's head node
        -.  next, executing that command, relative to the remote machine's
            default configuration file.

        For this feature to work properly, the remote machine must also
        support a fully operational C3 suite (version 4.0) placed in the
        /opt/c3-4 directory.

        The indirect cluster descriptors can be used to construct **chains**
        of remote references:  that is, multi-node configurations where an
        indirect cluster descriptor on a machine A references an indirect
        cluster descriptor on a machine B.  Here, it is the system
        administrator's responsibility to avoid circular references.

D. Post-install
        For the C3 ckill command to work properly, ckillnode must be copied
        to a directory on each compute node on every supported cluster.  The
        easy way to install ckillnode is to use cexec and cpush.  After
        installing and configuring C3 (cf. steps A-C above), use the
        following two commands to push ckillnode to each node in the default
        cluster.

        cexec mkdir /opt/c3-4
        cpush /opt/c3-4/ckillnode

        For the scalable version a full C3 install is needed on each node.
        This can be accomplished by either installing the RPM on each node
        or pushing the tarball out and using cexec (non-scalable at this point)
        to run the install script on each node.

This completes the installation of the C3 tools.

E. Notes
        The relative positions of nodes in c3.conf files can be significant
        for C3 command execution.  Version 3 of the C3 suite allows the use
        of node ranges on the command line.  The command line parameters used
        to specify the indices of compute nodes refer to relative node
        positions in c3.conf.

        Consider, for example, the semantics of node range parameters,
        relative to the following c3.conf file:

        cluster local {
                htorc-00:node0  #head node
                node[1-64]      #compute nodes
                exclude 60
                node[129-256]
        }

        This cluster is made up of 192 nodes.  Here,

        -.  the 64 nodes named node1 through node64 correspond to slots 0-63
        -.  the 128 nodes named node129 through node256 correspond to slots
            64-191--and **not**, for example, to slots 129-256.

        Note also that the excluded node--node60--acts as a place holder in
        the range of indices: node60 is a relative index of 59, which allows
        nodes node61, node62, node63, and node64 to correspond to 60, 61, 62,
        and 63, respectively.  This "place holder" effect is an important
        reason for explicitly specifying that a node is dead or excluded--as
        opposed to simply dropping that line from the specification.

        Two new tools in version 3.1 of the C3 tools suite support the
        management of node numbers.  The first, cname, inputs a node name,
        and outputs that node's relative position (slot number).  The second,
        cnum, inputs a range of slot numbers, and outputs the names of the
        corresponding compute nodes.


III  C3 SUITE DOCUMENTATION
---------------------------

C3 command documentation may be found in two locations.
  1. Quick Usage Info - enter "<command> --help" at the command line
  2. Full Man Page - enter "man <command>" at the command line