|
|
| Line 1: |
Line 1: |
| | <pre><nowiki> | | <pre><nowiki> |
| − | C3 version 4.0: Cluster Command & Control Suite
| |
| − |
| |
| − | Oak Ridge National Laboratory, Oak Ridge, TN,
| |
| − |
| |
| − | Authors: M.Brim, R.Flanery, G.A.Geist, B.Luethke, S.L.Scott
| |
| − |
| |
| − | (C) 2001 All Rights Reserved
| |
| − |
| |
| − |
| |
| − |
| |
| − | NOTICE
| |
| − |
| |
| − |
| |
| − |
| |
| − | Permission to use, copy, modify, and distribute this software and
| |
| − |
| |
| − | its documentation for any purpose and without fee is hereby granted
| |
| − |
| |
| − | provided that the above copyright notice appear in all copies and
| |
| − |
| |
| − | that both the copyright notice and this permission notice appear in
| |
| − |
| |
| − | supporting documentation.
| |
| − |
| |
| − |
| |
| − |
| |
| − | Neither the Oak Ridge National Laboratory nor the Authors make any
| |
| − |
| |
| − | representations about the suitability of this software for any
| |
| − |
| |
| − | purpose. This software is provided "as is" without express or
| |
| − |
| |
| − | implied warranty.
| |
| − |
| |
| − |
| |
| − |
| |
| − | The C3 tools were funded by the U.S. Department of Energy.
| |
| − |
| |
| − |
| |
| − |
| |
| − |
| |
| − |
| |
| − | I. REQUIRED SOFTWARE
| |
| − |
| |
| − | --------------------
| |
| − |
| |
| − |
| |
| − |
| |
| − | Before C3 can be installed on a system, you must ensure that the following
| |
| − |
| |
| − | software is installed on your system. The following seven software packages
| |
| − |
| |
| − | are required: the C3 tools suite, Rsync, SSH (or OpenSSH), Python, and Perl.
| |
| − |
| |
| − | You must also configure that system to support host name resolution of the
| |
| − |
| |
| − | machines listed in the configuration file (either through DNS or /etc/hosts).
| |
| − |
| |
| − | Finally, if you wish to use the C3 pushimage command, which pushes system
| |
| − |
| |
| − | images across a cluster, you must install SystemImager.
| |
| − |
| |
| − |
| |
| − |
| |
| − | Instructions for obtaining each of these software packages are given below.
| |
| − |
| |
| − |
| |
| − |
| |
| − | C3 tools may be obtained from http://www.csm.ornl.gov/torc/C3
| |
| − |
| |
| − |
| |
| − |
| |
| − | Rsync, Perl, SSH, and Python should be included with your distribution.
| |
| − |
| |
| − | If they are not then download the source or binaries from their respected
| |
| − |
| |
| − | web sites.
| |
| − |
| |
| − |
| |
| − |
| |
| − | Perl may be obtained from http://www.perl.com
| |
| − |
| |
| − | C3 requires 5.005 or greater
| |
| − |
| |
| − |
| |
| − |
| |
| − | Python may be obtained from
| |
| − |
| |
| − | http://www.python.org/
| |
| − |
| |
| − | C3 version 3 requires Python 2.0 or greater
| |
| − |
| |
| − |
| |
| − |
| |
| − | additionally C3 requires either the binary or a link to the python
| |
| − |
| |
| − | interpreter to be in your path (and that it be named python2). To
| |
| − |
| |
| − | check it type "python2 -V" and make sure you get output (the current
| |
| − |
| |
| − | version of python being run). If you do not get any output then you must
| |
| − |
| |
| − | find where the python library is on your machine and create a link to
| |
| − |
| |
| − | the binary. Such as "ln -s /usr/bin/python /usr/bin/python2" if
| |
| − |
| |
| − | /usr/bin/python is where your python binary is located and /usr/bin
| |
| − |
| |
| − | is in your path.
| |
| − |
| |
| − |
| |
| − |
| |
| − | SystemImager may be obtained from
| |
| − |
| |
| − | http://www.systemimager.org/
| |
| − |
| |
| − |
| |
| − |
| |
| − |
| |
| − |
| |
| − | II. C3 INSTALLATION
| |
| − |
| |
| − | ----------------
| |
| − |
| |
| − | NOTE: if you are wanting to use the scalable model of the C3 tolls then follow
| |
| − |
| |
| − | steps A and B, read C as it still pertains to the scalable model, then see the
| |
| − |
| |
| − | README.scale file for the scalable instructions.
| |
| − |
| |
| − |
| |
| − |
| |
| − | A. pre-install
| |
| − |
| |
| − | Begin by making sure that Rsync, OpenSSL, OpenSSH, PERL, and
| |
| − |
| |
| − | Python are installed. Install Systemimager, if needed. Install DNS
| |
| − |
| |
| − | or /etc/hosts as needed, and make sure that hostname resolution is
| |
| − |
| |
| − | supported.
| |
| − |
| |
| − |
| |
| − |
| |
| − | Directions for downloading each of these packages are given in
| |
| − |
| |
| − | Section I above. Perl, Python, Rsync, OpenSSH, and OpenSSL are included
| |
| − |
| |
| − | with most distributions
| |
| − |
| |
| − |
| |
| − |
| |
| − | You will need root access to install these packages on your system.
| |
| − |
| |
| − | Follow the instruction in each package if you need to install them.
| |
| − |
| |
| − |
| |
| − |
| |
| − | B. C3 install
| |
| − |
| |
| − | After you complete the pre-install (step A), install the Cluster
| |
| − |
| |
| − | Command & Control (C3) tools. Begin by untar'ring the C3 package
| |
| − |
| |
| − | and running the install script. The install script places the C3
| |
| − |
| |
| − | scripts in /opt/c3-4 and the man pages in the appropriate directory.
| |
| − |
| |
| − |
| |
| − |
| |
| − | The C3 install script installs the C3 command suite, but does not
| |
| − |
| |
| − | configure the commands or any local clusters for operation.
| |
| − |
| |
| − | Directions for the remaining tasks are given below.
| |
| − |
| |
| − |
| |
| − |
| |
| − | C. C3 configuration
| |
| − |
| |
| − | Specific instances of C3 commands identify their compute nodes with
| |
| − |
| |
| − | the help of **cluster configuration files**: files that name a set
| |
| − |
| |
| − | of accessible clusters, and that list and describe the set of
| |
| − |
| |
| − | machines in each accessible cluster. Cluster configuration files
| |
| − |
| |
| − | are accessed in one of two ways:
| |
| − |
| |
| − |
| |
| − |
| |
| − | -. explicitly: an instance of a C3 command names a specific
| |
| − |
| |
| − | configuration file, using a command-line switch.
| |
| − |
| |
| − |
| |
| − |
| |
| − | -. implicitly: an instance of a C3 command fails to name a specific
| |
| − |
| |
| − | configuration file, and the command defaults to the list of cluster
| |
| − |
| |
| − | descriptions given in /etc/c3.conf.
| |
| − |
| |
| − |
| |
| − |
| |
| − | When you install C3, you should create a default configuration file
| |
| − |
| |
| − | that is appropriate to the site. This file, which should be named
| |
| − |
| |
| − | /etc/c3.conf, should consist of a list of **cluster descriptor
| |
| − |
| |
| − | blocks**: syntactic objects that name and describe a single cluster
| |
| − |
| |
| − | that is accessible to that system's users.
| |
| − |
| |
| − |
| |
| − |
| |
| − | The following is an example of a default configuration file that
| |
| − |
| |
| − | contains exactly one cluster descriptor block: a block that
| |
| − |
| |
| − | describes a cluster of 64 nodes:
| |
| − |
| |
| − |
| |
| − |
| |
| − | cluster local {
| |
| − |
| |
| − | htorc-00:node0 #head node
| |
| − |
| |
| − | node[1-64] #compute nodes
| |
| − |
| |
| − | }
| |
| − |
| |
| − |
| |
| − |
| |
| − | Cluster description blocks consist of the following basic elements:
| |
| − |
| |
| − |
| |
| − |
| |
| − | -. a **cluster tag**: the word "cluster", followed by a label,
| |
| − |
| |
| − | which assigns a name to the cluster. This name--here, "local"--
| |
| − |
| |
| − | can be supplied to C3 commands as a way of specifying the cluster
| |
| − |
| |
| − | on which a command should execute.
| |
| − |
| |
| − |
| |
| − |
| |
| − | -. an open curly brace, which signals the start of the cluster's
| |
| − |
| |
| − | declaration proper.
| |
| − |
| |
| − |
| |
| − |
| |
| − | -. a **head node descriptor**: a line that names the interfaces
| |
| − |
| |
| − | on the cluster's head node. The head node descriptor shown here
| |
| − |
| |
| − | has two parts:
| |
| − |
| |
| − |
| |
| − |
| |
| − | -. The string to the left of the colon identifies the head
| |
| − |
| |
| − | node's **external** interface: a network card that links
| |
| − |
| |
| − | the head node to computers outside the cluster. This string
| |
| − |
| |
| − | can be the interface's IP address or DNS-style hostname.
| |
| − |
| |
| − | -. The string to the right of the colon identifies the head
| |
| − |
| |
| − | node's **internal** interface: a network card that links the
| |
| − |
| |
| − | head node to nodes inside the cluster. This string can be
| |
| − |
| |
| − | the interface's IP address or DNS-style hostname.
| |
| − |
| |
| − |
| |
| − |
| |
| − | Here, the head node descriptor names a head node with an external
| |
| − |
| |
| − | interface named htorc-00, and an internal interface named node0.
| |
| − |
| |
| − |
| |
| − |
| |
| − | A cluster that has no external interface--i.e., a cluster that is
| |
| − |
| |
| − | on a closed system--can be specified by either
| |
| − |
| |
| − |
| |
| − |
| |
| − | -. making the internal and external name the same, or
| |
| − |
| |
| − | -. dropping the colon, and using one name in the specifier.
| |
| − |
| |
| − |
| |
| − |
| |
| − | -. a list of **compute node descriptors**: a series of individual
| |
| − |
| |
| − | descriptors that name the cluster's compute nodes.
| |
| − |
| |
| − |
| |
| − |
| |
| − | The example given here contains exactly one compute node
| |
| − |
| |
| − | descriptor. This descriptor uses a **range qualifier** to
| |
| − |
| |
| − | specify a cluster that contains 64 compute nodes, named node1,
| |
| − |
| |
| − | node2, etc., up through node64. A range qualifier consists of
| |
| − |
| |
| − | -. a first, nonnegative integer, followed by
| |
| − |
| |
| − | -. a dash, followed by
| |
| − |
| |
| − | -. a second integer that is at least as large as the first.
| |
| − |
| |
| − |
| |
| − |
| |
| − | In the current version of the C3 tools et, these range values are
| |
| − |
| |
| − | treated as numbers, with no leading zeroes. A declaration like
| |
| − |
| |
| − |
| |
| − |
| |
| − | cluster local {
| |
| − |
| |
| − | htorc-00:node0 #head node
| |
| − |
| |
| − | node[01-64] #compute nodes
| |
| − |
| |
| − | }
| |
| − |
| |
| − |
| |
| − |
| |
| − | expands to the same 64 nodes as the declaration shown above. To
| |
| − |
| |
| − | specify a set of nodes with names like node01, node09, node10, ...
| |
| − |
| |
| − | node64, use declarations like
| |
| − |
| |
| − |
| |
| − |
| |
| − | cluster local {
| |
| − |
| |
| − | htorc-00:node0 #head node
| |
| − |
| |
| − | node0[1-9] #compute nodes node01..node09
| |
| − |
| |
| − | node[10-64] #compute nodes node10..node64
| |
| − |
| |
| − | }
| |
| − |
| |
| − |
| |
| − |
| |
| − | -. a final, closing curly brace.
| |
| − |
| |
| − |
| |
| − |
| |
| − | Configuration files that specify multiple clusters are constituted as
| |
| − |
| |
| − | a list of cluster descriptor blocks--one per accessible cluster.
| |
| − |
| |
| − | The following example of a cluster configuration file contains three
| |
| − |
| |
| − | blocks that specify configurations for clusters named local, torc,
| |
| − |
| |
| − | and my-cluster, respectively:
| |
| − |
| |
| − |
| |
| − |
| |
| − | cluster local {
| |
| − |
| |
| − | htorc-00:node0 #head node
| |
| − |
| |
| − | node[1-64] #compute nodes
| |
| − |
| |
| − | exclude 2
| |
| − |
| |
| − | exclude [55-60]
| |
| − |
| |
| − | }
| |
| − |
| |
| − |
| |
| − |
| |
| − | cluster torc {
| |
| − |
| |
| − | :orc-00b
| |
| − |
| |
| − | }
| |
| − |
| |
| − |
| |
| − |
| |
| − | cluster my-cluster {
| |
| − |
| |
| − | osiris:192.192.192.2
| |
| − |
| |
| − | woody
| |
| − |
| |
| − | dead riggs
| |
| − |
| |
| − | }
| |
| − |
| |
| − |
| |
| − |
| |
| − | The first cluster in the file has a special significance that is
| |
| − |
| |
| − | analogous to the special significance accorded to the first
| |
| − |
| |
| − | declaration in a make file. Any instance of a C3 command that fails
| |
| − |
| |
| − | to name the cluster on which it should run executes, by default, on
| |
| − |
| |
| − | the first cluster in the configuration file. Here, for example, any
| |
| − |
| |
| − | command that fails to name its target cluster would default to local.
| |
| − |
| |
| − |
| |
| − |
| |
| − | The cluster configuration file shown above illustrates three final
| |
| − |
| |
| − | features of the cluster definition language: **exclude qualifiers**,
| |
| − |
| |
| − | **dead qualifiers**, and **indirect cluster** descriptors.
| |
| − |
| |
| − |
| |
| − |
| |
| − | **Exclude qualifiers** allow nodes to be excluded from a cluster's
| |
| − |
| |
| − | configuration: i.e., to be identified as offline for the purpose of
| |
| − |
| |
| − | a command execution. Exclude qualifiers may only be applied to
| |
| − |
| |
| − | range declarations, and must follow immediately after a range
| |
| − |
| |
| − | declaration to which they are being applied. A series of exclude
| |
| − |
| |
| − | declarations is ended by a non-exclude declaration, or the final "}"
| |
| − |
| |
| − | in a cluster declaration block.
| |
| − |
| |
| − |
| |
| − |
| |
| − | An exclude qualifier can be written in one of three ways:
| |
| − |
| |
| − | -. "exclude n", where n is the number of a node to exclude from the
| |
| − |
| |
| − | cluster;
| |
| − |
| |
| − | -. "exclude[m-n]", where m, m+1, m+2, ..., n-1, n is the range of
| |
| − |
| |
| − | nodes to exclude; or as
| |
| − |
| |
| − | -. "exclude [m-n], which has the same effect as "exclude[m-n]".
| |
| − |
| |
| − | Note that a string like "exclude5" is parsed as a node name, rather
| |
| − |
| |
| − | than as an exclude qualifier.
| |
| − |
| |
| − |
| |
| − |
| |
| − | In the above example, the two exclude qualifiers have the effect of
| |
| − |
| |
| − | causing node2, node55, node56 node57, node58, node59, and node60 to
| |
| − |
| |
| − | be treated as offline for the purpose of computation.
| |
| − |
| |
| − |
| |
| − |
| |
| − | **Dead qualifiers** are similar to exclude qualifiers, but apply to
| |
| − |
| |
| − | individual machines. In the example given above, the machine named
| |
| − |
| |
| − | "riggs" in the cluster named "my-cluster" is excluded from all
| |
| − |
| |
| − | computations.
| |
| − |
| |
| − |
| |
| − |
| |
| − | "Dead", like "exclude", is not a reserved word in the current version
| |
| − |
| |
| − | of the C3 suite. A specification block like
| |
| − |
| |
| − |
| |
| − |
| |
| − | cluster my-cluster {
| |
| − |
| |
| − | alive:alive
| |
| − |
| |
| − | dead
| |
| − |
| |
| − | }
| |
| − |
| |
| − |
| |
| − |
| |
| − | for example, declares a two-machine cluster with a head node named
| |
| − |
| |
| − | "alive" and a compute node named "dead".
| |
| − |
| |
| − |
| |
| − |
| |
| − | An **indirect cluster descriptor** is treated as a reference to
| |
| − |
| |
| − | another cluster, rather than as a characterization of a cluster per
| |
| − |
| |
| − | se. In the example shown above, the descriptor
| |
| − |
| |
| − |
| |
| − |
| |
| − | cluster torc {
| |
| − |
| |
| − | :orc-00b
| |
| − |
| |
| − | }
| |
| − |
| |
| − |
| |
| − |
| |
| − | is an indirect cluster descriptor. An indirect descriptor consists
| |
| − |
| |
| − | of
| |
| − |
| |
| − |
| |
| − |
| |
| − | -. a cluster tag, followed by,
| |
| − |
| |
| − | -. an **indirect head head node descriptor**, followed by
| |
| − |
| |
| − | -. an empty list of compute node descriptors.
| |
| − |
| |
| − |
| |
| − |
| |
| − | An indirect head node descriptor consists of an initial colon,
| |
| − |
| |
| − | followed by a string that names a **remote** system. This name,
| |
| − |
| |
| − | which can either be an IP address or a DNS-style hostname, is checked
| |
| − |
| |
| − | whenever a C3 command executes to verify that that the machine being
| |
| − |
| |
| − | referenced is **not** the machine on which that command is currently
| |
| − |
| |
| − | executing.
| |
| − |
| |
| − |
| |
| − |
| |
| − | A command that is destined for an indirect cluster is executed by
| |
| − |
| |
| − |
| |
| − |
| |
| − | -. first forwarding that command to the remote cluster's head node
| |
| − |
| |
| − | -. next, executing that command, relative to the remote machine's
| |
| − |
| |
| − | default configuration file.
| |
| − |
| |
| − |
| |
| − |
| |
| − | For this feature to work properly, the remote machine must also
| |
| − |
| |
| − | support a fully operational C3 suite (version 4.0) placed in the
| |
| − |
| |
| − | /opt/c3-4 directory.
| |
| − |
| |
| − |
| |
| − |
| |
| − | The indirect cluster descriptors can be used to construct **chains**
| |
| − |
| |
| − | of remote references: that is, multi-node configurations where an
| |
| − |
| |
| − | indirect cluster descriptor on a machine A references an indirect
| |
| − |
| |
| − | cluster descriptor on a machine B. Here, it is the system
| |
| − |
| |
| − | administrator's responsibility to avoid circular references.
| |
| − |
| |
| − |
| |
| − |
| |
| − | D. Post-install
| |
| − |
| |
| − | For the C3 ckill command to work properly, ckillnode must be copied
| |
| − |
| |
| − | to a directory on each compute node on every supported cluster. The
| |
| − |
| |
| − | easy way to install ckillnode is to use cexec and cpush. After
| |
| − |
| |
| − | installing and configuring C3 (cf. steps A-C above), use the
| |
| − |
| |
| − | following two commands to push ckillnode to each node in the default
| |
| − |
| |
| − | cluster.
| |
| − |
| |
| − |
| |
| − |
| |
| − | cexec mkdir /opt/c3-4
| |
| − |
| |
| − | cpush /opt/c3-4/ckillnode
| |
| − |
| |
| − |
| |
| − |
| |
| − | For the scalable version a full C3 install is needed on each node.
| |
| − |
| |
| − | This can be accomplished by either installing the RPM on each node
| |
| − |
| |
| − | or pushing the tarball out and using cexec (non-scalable at this point)
| |
| − |
| |
| − | to run the install script on each node.
| |
| − |
| |
| − |
| |
| − |
| |
| − | This completes the installation of the C3 tools.
| |
| − |
| |
| − |
| |
| − |
| |
| − | E. Notes
| |
| − |
| |
| − | The relative positions of nodes in c3.conf files can be significant
| |
| − |
| |
| − | for C3 command execution. Version 3 of the C3 suite allows the use
| |
| − |
| |
| − | of node ranges on the command line. The command line parameters used
| |
| − |
| |
| − | to specify the indices of compute nodes refer to relative node
| |
| − |
| |
| − | positions in c3.conf.
| |
| − |
| |
| − |
| |
| − |
| |
| − | Consider, for example, the semantics of node range parameters,
| |
| − |
| |
| − | relative to the following c3.conf file:
| |
| − |
| |
| − |
| |
| − |
| |
| − | cluster local {
| |
| − |
| |
| − | htorc-00:node0 #head node
| |
| − |
| |
| − | node[1-64] #compute nodes
| |
| − |
| |
| − | exclude 60
| |
| − |
| |
| − | node[129-256]
| |
| − |
| |
| − | }
| |
| − |
| |
| − |
| |
| − |
| |
| − | This cluster is made up of 192 nodes. Here,
| |
| − |
| |
| − |
| |
| − |
| |
| − | -. the 64 nodes named node1 through node64 correspond to slots 0-63
| |
| − |
| |
| − | -. the 128 nodes named node129 through node256 correspond to slots
| |
| − |
| |
| − | 64-191--and **not**, for example, to slots 129-256.
| |
| − |
| |
| − |
| |
| − |
| |
| − | Note also that the excluded node--node60--acts as a place holder in
| |
| − |
| |
| − | the range of indices: node60 is a relative index of 59, which allows
| |
| − |
| |
| − | nodes node61, node62, node63, and node64 to correspond to 60, 61, 62,
| |
| − |
| |
| − | and 63, respectively. This "place holder" effect is an important
| |
| − |
| |
| − | reason for explicitly specifying that a node is dead or excluded--as
| |
| − |
| |
| − | opposed to simply dropping that line from the specification.
| |
| − |
| |
| − |
| |
| − |
| |
| − | Two new tools in version 3.1 of the C3 tools suite support the
| |
| − |
| |
| − | management of node numbers. The first, cname, inputs a node name,
| |
| − |
| |
| − | and outputs that node's relative position (slot number). The second,
| |
| − |
| |
| − | cnum, inputs a range of slot numbers, and outputs the names of the
| |
| − |
| |
| − | corresponding compute nodes.
| |
| − |
| |
| − |
| |
| − |
| |
| − |
| |
| − |
| |
| − | III C3 SUITE DOCUMENTATION
| |
| − |
| |
| − | ---------------------------
| |
| − |
| |
| − |
| |
| − |
| |
| − | C3 command documentation may be found in two locations.
| |
| − |
| |
| − | 1. Quick Usage Info - enter "<command> --help" at the command line
| |
| − |
| |
| − | 2. Full Man Page - enter "man <command>" at the command line
| |
| | | | |
| | | | |
| | </nowiki></pre> | | </nowiki></pre> |
