Difference between revisions of "Mosix"

Latest revision as of 12:31, 22 February 2009

MOSIX(M7)                      MOSIX Description                     MOSIX(M7)
   
NAME
     MOSIX - sharing the power of clusters and multi-clusters
     
INTRODUCTION
    MOSIX is a generic solution for dynamic management of resources in a
    cluster or in a multi-cluster organizational grid. MOSIX allows users to
    draw the most out of all the connected computers, including utilization
    of idle computers.
   
    At the core of MOSIX are adaptive resource sharing algorithms, applying
    preemptive process migration based on processor loads, memory and I/O
    demands of the processes, thus causing the cluster or the multi-cluster
    to work cooperatively similar to a single computer with many processors.
     
    Unlike earlier versions of MOSIX, only programs that are started by the
    mosrun(1) utility are affected and can be considered "migratable" - other
    programs are considered as "standard Linux programs" and are not affected
    by MOSIX.
     
    MOSIX maintains a high level of compatiblity with standard Linux, so that
    binaries of almost every application that runs under Linux can run com-
    pletely unmodified under the MOSIX "migratable" category.  The exceptions
    are usually system-administration or graphic utilities that would not
    benefit from process-migration anyway.  If a "migratable" program that
    was started by mosrun(1) attempts to use unsupported features, it will
    either be killed with an appropriate error message, or if a ``do not
    kill option is selected, an error is returned to the program: such pro-
    grams should probably run as standard Linux programs.
       
    In order to improve the overall resource usage, processes of "migratable"
    programs may be moved automatically and transparently to other nodes
    within the cluster or even the multi-cluster grid.  As the demands for
    resources change, processes may move again, as many times as necessary,
    to continue optimizing the overall resource utilization, subject to the
    inter-cluster priorities and policies.  Manual-control over process
    migration is also supported.
     
    MOSIX is particularly suitable for running CPU-intensive computational
    programs with unpredictable resource usage and run times, and programs
    with moderate amounts of I/O.  Programs that perform large amounts of I/O
    should better be run as standard Linux programs.
          
    Apart from process-migration, MOSIX can provide both "migratable" and
    "standard Linux" programs with the benefits of optimal initial assignment
    and live-queuing.  The unique feature of live-queuing means that although
    a job is queued to run later, when resources are available, once it
    starts, it remains attached to its original Unix/Linux environment (stan-
    dard-input/output/error, signals, etc.). 
         
REQUIREMENTS     
    1.   All nodes must run Linux (any distribution - mixing allowed).
    
    2.   All participating nodes must be connected to a network that supports
         TCP/IP and UDP/IP, where each node has a unique IP address in the
         range 0.1.0.0 to 255.254.254.255 that is accessible to all the other
         nodes.
    
    3.   TCP/IP ports 249-254 and UDP/IP ports 249-250 must be available for
         MOSIX (not used by other applications or blocked by a firewall).
     
    4.   The architecture of all nodes can be either i386 (32-bit) or x86_64
         (64-bit).  Processes that are started on a 32-bit node can migrate
         to a 64-bit node, but not the opposite.
     
    5.   In multiprocessor nodes (SMP), all the processors must be of the
         same speed.
     
    6.   The system-administrators of all the connected nodes must be able to
         trust each other (see more on SECURITY below).
           
CLUSTER, MULTI-CLUSTER, PARTITION
    The MOSIX concept of a "cluster" is a collection of computers that are
    owned and managed by the same entity (a person, a group of people or a
    project) - this can at times be quite different than a hardware cluster,
    as each MOSIX cluster may range from a single workstation to a large com-
    bination of computers - workstations, servers, blades, multi-core comput-
    ers, etc. possibly of different speeds and number of processors and pos-
    sibly in different locations.
    
    A MOSIX multi-cluster is a collection of clusters that belong to differ-
    ent entities (owners) who wish to share their resources subject to cer-
    tain administrative conditions.  In particular, when an owner needs its
    computers - these computers must be returned immediately to the exclusive
    use of their owner.  An owner can also assign priorities to guest pro-
    cesses of other owners, defining who can use their computers and when.
    Typically, an owner is an individual user, a group of users or a depart-
    ment that own the computers.  The multi-cluster is usually restricted,
    due to trust and security reasons, to a single organization, possibly in
    various sites/branches, even across the world.
     
    MOSIX supports dynamic multi-cluster configurations, where clusters can
    join and leave at any time.  When there are plenty of resources in the
    multi-cluster, the MOSIX queuing system allows more processes to start.
    When resources become scarce (because other clusters leave or claim their
    resources and processes must migrate back to their home-clusters), MOSIX
    has a freezing feature that can automatically freeze excess processes to
    prevent memory-overload on the home-nodes.
   
    Clusters may also be sub-divided into "partitions".  Nodes that are
    assigned to different cluster-partitions are halfway between being part
    of the cluster and belonging to a different cluster.
   
    Just as within the cluster:
    1. All cluster-partitions seem to other clusters as one cluster (elimi-
       nating the need to inform and update system-administrators of other 
       clusters about internal changes to one's cluster).
    2. Processes that migrate to another partition share the same top-prior-
       ity over processes from other clusters.
    3. Processes that migrate to another partition share the "Cluster" cate-
       gory disk-space allocation rather than the "Grid" category for Private
       Temporary Files (see below).
        
    However, just as other clusters:
    1. Only processes that were allowed to migrate to other clusters are
       allowed to migrate to other partitions.
    2. Batch jobs cannot be assigned to nodes in other partitions.
    3. Each partition maintains its own job-queue.
  
    When you have both 32-bit and 64-bit computers in the same cluster, it is
    highly recommended (but not mandatory) to set them up as different clus-
    ter partitions.
         
CONFIGURATION
       To configure MOSIX interactively, simply run mosconf: it will lead you
    step-by-step through the various configuration items.
   
    Mosconf can be used in two ways:
   
    1.   To configure the local node (press <Enter> at the first question).
   
    2.   To configure MOSIX for other nodes: this is typically done on a
         server that stores an image of the root-partition for some or all of
         the cluster-nodes.  This image can, for example, be NFS-mounted by
         the cluster-nodes, or otherwise copied or reflected to them by any
         other method: at the first question, enter the path to the stored
         root-image.
    
    There is no need to stop MOSIX in order to modify the configuration -
    most changes will take effect within a minute.  However, after modifying
    the list of nodes in the cluster (/etc/mosix/mosix.map) or
    /etc/mosix/mosip or /etc/mosix/myfeatures, you should run the command
    "setpe" (but when you are using mosconf to configure your local node,
    this is not necessary).
   
    Below is a detailed description of the MOSIX configuration files (if you
    prefer to edit them manually).
  
    The directory /etc/mosix should include at least the subdirectories
    /etc/mosix/partners, /etc/mosix/var, /etc/mosix/var/grid and the follow-
    ing files:
  
    /etc/mosix/mosix.map
           This file defines which computers participate in your MOSIX clus-
           ter.  The file contains up to 256 data lines and/or alias lines
           that can be in any order.  It may also include any number of com-
           ment lines beginning with a '#', as well as empty lines.
  
           Data lines have 2 or 3 fields:
  
           1.   The IP ("a.b.c.d" or host-name) of the first node in a range
                of nodes with consecutive IPs.
  
           2.   The number of nodes in that range.
   
           3.   Optional combination of letter-flags and/or an integer:
                p[roximate]  do not use compression on migration, e.g., over
                             fast networks or slow CPUs.
                o[utsider]   inaccessible to local-class processes.
                {partition}  a positive integer indicating the partition num-
                             ber for that range.
   
           Alias lines are of the form:
               a.b.c.d=e.f.g.h
           or
               a.b.c.d=host-name
   
           They indicate that the IP address on the left-hand-side refers to
           the same node as the right-hand-side.
   
           NOTES:
   
           1.   It is an error to attempt to declare the local node an "out-
                sider".
   
           2.   When using host names, the first result of gethostbyname(3)
                must return their IP address that is to be used by MOSIX: if
                in doubt - specify the IP address.
  
           3.   The right-hand-side in alias lines must appear within the
                data lines.
   
           4.   IP addresses 0.0.x.x and 255.255.255.x are not allowed in
                MOSIX.
    
           5.   If you change /etc/mosix/mosix.map while MOSIX is running,
                you need to run setpe to notify MOSIX of the changes.
   
    /etc/mosix/secret
           This is a security file that is used to prevent ordinary users
           from interfering and/or compromizing security by connecting to the
           internal MOSIX TCP ports.  The file should contain just a single
           line with a password that must be identical on all the nodes of
           the cluster/multi-cluster.  This file must be accessible to ROOT
           only (chmod 600!)
  
    /etc/mosix/ecsecret
           Like /etc/mosix/secret, but used for running batch jobs as a
           client (see mosrun(1)).  If you do not wish to allow this node to
           send batch-jobs, do not create this file.
   
    /etc/mosix/essecret
           Like /etc/mosix/secret, but used for running batch jobs as a
           server (see mosrun(1)).  The password must match the client's
           /etc/mosix/ecsecret.  If you do not wish to allow this node to be
           a batch-server, do not create this file.
  
    The following files are optional:
  
    /etc/mosix/mosip
           This file contains our IP address, to be used for MOSIX purposes,
           in the regular format - a.b.c.d.  This file is only necessary when
           the node's IP address is ambiguous: it can be safely omitted if
           the output of ifconfig(8) ("inet addr:") matches exactly one of
           the IP addresses listed in the data lines of /etc/mosix/mosix.map.
  
    /etc/mosix/myfeatures
           This file contains one line of comma-separated topological fea-
           tures for this node (if any).  For example: yellow,wood,chicken.
  
           The list of all 32 features (one line per feature) can be found in
           /etc/mosix/features.
  
           If this file is missing, this node is assumed to have no topologi-
           cal features.  (see topology(7))
  
    /etc/mosix/freeze.conf
           This file sets the automatic freezing policies on a per-class
           basis for MOSIX processes originating in this node.  Each line
           describes the policy for one class of processes.  The lines can be
           in any order and classes that are not mentioned are not touched by
           the automatic freezing mechanisms.
  
           The space-separated constants in each line are as follows:
           1. class-number
                  A positive integer identifying a class of processes
           2. load-units:
                  Used in fields #3-#6 below: 0=processes; 1=standard-load
           3. RED-MARK (floating point)
                  Freeze when load is higher
           4. BLUE-MARK (floating point)
                  Unfreeze when load is lower
           5. minautofreeze (floating point)
                  Freeze processes that are evacuated back home on arrival if
                  load gets equal or above this
           6. minclustfreeze (floating point)
                  Freeze processes that are evacuated back to this cluster on
                  arrival if load gets equal or above this
           7. min-keep
                  Keep running at least this number of processes - even if
                  load is above RED-MARK.
           8. max-procs
                  Freeze excess processes above this number - even if load is
                  below BLUE-MARK.
           9. slice
                  Time (in minutes) that a process of this class is allowed
                  to run while there are automatically-frozen process(es) of
                  this class. After this period, the running process will be
                  frozen and a frozen process will start to run.
  
           NOTES:
   
           1.   The load-units in fields #3-#6 depend on field #2.  If 0,
                each unit represents the load created by a CPU-bound process
                on this computer.  If 1, each unit represents the load cre-
                ated by a CPU-bound process on a "standard" MOSIX computer
                (e.g.  a 3GHz Pentium-IV).  The difference is that the faster
                the computer and the more processors it has, the load created
                by each CPU process decreases proportionally.
   
           2.   Fields #3,#4,#5,#6 are floating-point, the rest are integers.
    
           3.   A value of "-1" in fields #3,#5,#6,#8 means ignoring that
                feature.
  
           4.   The first 4 fields are mandatory: omitted fields beyond them
                have the following values: minautofreeze=-1,mincluster-
                freeze=-1,min-keep=0, max-procs=-1,slice=20.
     
           5.   The RED-MARK must be significantly higher than BLUE-MARK:
                otherwise a perpetual cycle of freezing and unfreezing could
                occur.  You should allow at least 1.1 processes difference
                between them.
    
           6.   Frozen processes do not respond to anything, except an
                unfreeze request or a signal that kills them.
    
           7.   Processes that were frozen manually are not unfrozen automat-
                ically.
  
           This file may also contain lines starting with '/' to indicate
           freezing-directory names.  A "Freezing directory" is an existing
           directory (often a mount-point) where the memory contents of
           frozen process is saved.  For successful freezing, the disk-parti-
           tion of freezing-directories should have sufficient free disk-
           space to contain the memory image of all the frozen processes.
  
           If more than one freezing directory is listed, the freezing direc-
           tory is chosen at random by each freezing process.  It is also
           possible to assign selection probabilities by adding a numeric
           weight after the directory-name, for example:
  
                /tmp       2
                /var/tmp   0.5
                /mnt/tmp   2.5
   
                In this example, the total weight is 2+0.5+2.5=5, so out of
                every 10 frozen processes, an average of 4 (10*2/5) will be
                frozen to /tmp, an average of 1 (10*0.5/5) to /var/tmp and an
                average of 5 (10*2.5/5) to /mnt/tmp.
  
           When the weight is missing, it defaults to 1.  A weight of 0 means
           that this directory should be used only if all others cannot be
           accessed.
   
           If no freezing directories are specified, all freezing will be to
           the /freeze directory (or symbolic-link).
   
           Freezing files are usually created with "root" (Super-User) per-
           missions, but if /etc/mosix/freeze.conf contains a line of the
           form:
                 U {UID}
           then they are created with permissions of the given numeric UID
           (this is sometimes needed when freezing to NFS directories that do
           not allow "root" access).
   
    /etc/mosix/partners/*
           If your cluster is part of a multi-cluster, then each file in
           /etc/mosix/partners describes another cluster that you want this
           cluster to cooperate with.
   
           The file-names should indicate the corresponding cluster-names
           (maximum 128 characters), for example: "geography", "chemistry",
           "management", "development", "sales", "students-lab-A", etc.  The
           format of each file is a follows:
   
           Line #1:
                  A verbal human-readable description of the cluster.
           Line #2:
                  Four space-separated integers as follows:
   
                  1. Priority:
                           0-65535, the lower the better.
                           The priority of the local cluster is always 0.
                           MOSIX gives precedence to processes with higher
                           priority - if they arrive, guests with lower pri-
                           ority will be expelled.
                  2. Cango:
                           0=never send local processes to that cluster.
                           1=local processes may go to that cluster.
                  3. Cantake:
                           0=do not accept guest-processes from that cluster.
                           1=accept guest-processes from that cluster.
                  4. Canexpand:
                           0=no:   Only nodes listed in the lines below may
                                   be recognized as part of that cluster: if
                                   a core node from that cluster tells us
                                   about other nodes in their cluster -
                                   ignore those unlisted nodes.
                           1=yes:  Core-nodes of that cluster may specify
                                   other nodes that are in that cluster, and
                                   this node should believe them even if they
                                   are not listed in the lines below.
                           -1=do not ask the other cluster:
                                   do not consult the other cluster to find
                                   out which nodes are in that cluster:
                                   instead just rely on and use the lines
                                   below.
           Following lines:
                  Each line describes a range of consecutive IP addresses
                  that are believed to be part of the other cluster, contain-
                  ing 5 space-separated items as follows:
     
                  1. IP1 (or host-name):
                       First node in range.
                  2. n:
                       Number of nodes in this range.
                  3. Core:
                       0=no:   This range of nodes may not inform us about
                               who else is in that cluster.
                       1=yes:  This range of nodes could inform us of who
                               else is in that cluster.
                  4. Participate:
                       0=no    This range is (as far as this node is con-
                               cerned) not part of that cluster.
                       1=yes   This range is probably a part of that cluster.
                  5. Proximate:
                       0=no    Use compression on migration to/from that
                               cluster.
                       1=yes   Do not use compression when migrating to/from
                               that cluster (network is very fast and CPU is
                               slow).
           NOTES:
     
           1.   From time-to-time, MOSIX will consult one or more of the
                "core" nodes to find the actual map of their cluster.  It is
                recommended to list such core nodes. The alternative is to
                set canexpand to -1, causing the map of that cluster to be
                determined solely by this file.
 
           2.   Nodes that do not "participate" are excluded even if listed
                as part of their cluster by the core-nodes (but they could
                possibly still be used as "core-nodes" to list other nodes)
   
           3.   All core-nodes must have the same value for "proximate",
                because the "proximate" field of unlisted nodes is copied
                from that of the core-node from which we happened to find
                about them and this cannot be ambiguous.
   
           4.   When using host names rather than IP addresses, the first
                result of gethostbyname(3) must return their IP address that
                is used by MOSIX: if in doubt - specify the IP address
                instead.
   
           5.   IP addresses 0.0.x.x and 255.255.255.x cannot be used in
                MOSIX.
   
    /etc/mosix/userview.map
           Although it is possible to use only IP numbers and/or host-names
           to specify nodes in your cluster (and multi-cluster), it is more
           convenient to use small integers as node numbers: this file allows
           you to map integers to IP addresses.  Each line in this file con-
           tains 3 elements:
   
           1.   A node number (1-65535)
           2.   IP1 (or host-name, clearly identifiable by gethostbyname(3))
           3.   Number of nodes in range (the number of the last one must not
                exceed 65535)
   
           It is up to the cluster administrator to map as few or as many
           nodes as they wish out of their cluster and multi-cluster - the
           most common practice is to map all the nodes in one's cluster, but
           not in other clusters.
  
    /etc/mosix/queue.conf
           This file configures the queueing system (see mosrun(1), mosq(1)).
           All lines in this file are optional and may appear in any order.
   
           Usually, one node in each cluster is elected by the system-admin-
           istrator to manage the queue, while the remaining nodes point to
           that manager.  As an exception, in a mixed cluster that has both
           32-bit and 64-bit computers, a separate 32-bit node should be
           elected to exclusively manage the queue for all 32-bit nodes and a
           64-bit node elected to exclusively manage the queue for all 64-bit
           nodes.
   
           Defining the queue manager:
   
           The line:
                C {hostname}
           assigns a specific node from the cluster (hostname) to manage the
           job queue.  In the absence of this line, each node manages its own
           queue (which is usually undesirable).  It is possible to have sev-
           eral 'C' lines - one for each cluster-partition.
   
           Defining the default priority:
   
           The line:
                P {priority}
           assigns a default job-priority to all the jobs from this node.
           The lower this value - the higher the priority.  In the absence of
           this line, the default priority is 50.
   
           Commonly, user-ID's are identical on all the nodes in the cluster.
           The line (with a single letter):
                S
           indicates that this is not the case, so users on other nodes
           (except the Super-User) will be prevented from sending requests to
           modify the status of queued jobs from this node.
   
           Configuring the queue manager:
  
           The following lines are relevant only in the queue manager node
           and are ignored on all other nodes:
   
           The MOSIX queueing system determines dynamically how many pro-
           cesses to run.  The line:
                M {maxproc}
           if present, imposes a maximal number of processes that are allowed
           to run from the queue simultaneously on top of the regular queue-
           ing policy.  For example,
                M 20
           sets the upper limit to 20 processes, even when more resources are
           available.
  
           The line:
                X {1 <= x <= 8}
           defines the maximal number of queued processes that may run simul-
           taneously per CPU.  This option applies only to processors within
           the cluster and is not available for other clusters in a multi-
           cluster (where the queueing system assigns at most one process per
           CPU). In the absence of this line the default is
                X 1
   
           The line:
                Z {n}
           causes the first n jobs of priority 0 to start immediately (out of
           order), without checking whether resources are available, leaving that
           responsibility to the system administrator.
   
           Example: the cluster has 10 dual-CPU nodes, so the queueing system
           normally allows 20 jobs to run.  In order to allow urgent jobs to
           run immediately (without waiting for regular jobs to complete),
           the system administrator configures a line: Z 10, thus allowing
           each node to run a maximum of 3 jobs.
   
           The line:
                N {n} [{mb}]
           causes the first n jobs of jobs of each user to start immediately 
           (out of order), without checking whether resources are available.
           Only jobs above that number, per user, will be queued and whenever
           the number of a user's running jobs drops below this number, a new 
           job of that user (if there is any waiting) will start to run.
       
           When the mb parameter is given, only jobs that do not exceed this
           amount of memory in MegaBytes will be started this way.
     
           The system-administrator should weigh carefully, based on knowledge
           about the patterns of jobs that users typically run, the benefits of
           this option against its risks, such as having at times more jobs in 
           their cluster(s) than available memory to run them efficiently.  If
           this option is selected with a memory-limitation (mb), then the 
           system-administrator should request that users always specify the
           maximum memory-requiremnts for all their queued jobs (using mosrun -m").
   
           Fair-share policy:
                The fairness policy determine the order in which jobs are
                initially placed in the queue.  Note that fairness should not
                be confused with priority (as defined by the P {priority}
                line or by mosrun -q{pri} and possibly modified by mosq(1)):
                priorities always take precedence, so here we only discuss
                the initial placement in the queue of jobs with the same pri-
                ority.
    
                The default queueing policy is "first-come-first-served".
                Alternatively, jobs of different users can be placed in the
                queue in an interleaved manner.
   
                The line (with a single letter):
                     F
                switches the queueing policy to the interleaved policy.
   
                The advantage of the interleaved approach is that a user
                wishing to run a relatively small number of processes, does
                not need to wait for all the jobs that were already placed in
                the queue.  The disadvantage is that older jobs need to wait
                longer.
   
                Normally, the interleaving ratio is equal among all users.
                For example, with two users (A and B) the queue may look like
                A-B-A-B-A-B-A-B.
   
                Each user is assigned an interleave ratio which determines
                (proportionally) how well their jobs will be placed in the
                queue relative to other users: the smaller that ratio - the
                better placement they will get (and vice versa).  Normally
                all users receive the same default interleave-ratio of 10 per
                process.  However, lines of the form:
                     U {UID} {1 <= interleave <= 100}
                can set a different interleave ratio for different users.
                UID can be either numeric or symbolic and there is no limit
                on the number of these 'U' lines.  Examples:
                1.   Two users (A & B):                      U userA 5
                     (userB is not listed, hence it gets the default of 10)
                     The queue looks like: A-A-B-A-A-B-A-A-B...
                2.   Two users (A & B):
                     U userA 20
                     U userB 15
                     The queue looks like: B-A-B-A-B-A-B-B-A-B-A-B-A-B-B-A...
                3.   Three users (A, B & C):
                     U userA 25
                     U userB 7
                     (userC is not listed, hence it gets the default of 10)
                     The queue looks like: B-C-B-C-B-A-B-C-B-C-B-A-B-C-B-C...
   
                Note that since the interleave ratio is determined per pro-
                cess (and not per job), different (more complex) results will
                occur when multi-process jobs are submitted to the queue.
   
    /etc/mosix/private.conf
           This file specifies where Private Temporary Files (PTFs) are
           stored: PTFs are an important feature of mosrun(1) and may consume
           a significant amount of disk-space.  It is important to ensure
           that sufficient disk-space is reserved for PTFs, but without
           allowing them to disturb other jobs by filling up disk-partitions.
           Guest processes can also demand unpredictable amounts of disk-
           space for their PTFs, so we must make sure that they do not dis-
           turb local operations.
  
           Up to 3 different directories can be specified: for local pro-
           cesses; guest-processes from the local cluster (including other
           partitions); and guest-processes from other clusters in the multi-
           cluster grid. Accordingly, each line in this file has 3 fields:
   
           1.   A combination of the letters: 'O' (own node), 'C' (own clus-
                ter) and 'G' (other clusters in the grid).  For example, OC,
                C, CG or OCG.
           2.   A directory name (usually a mount-point) starting with '/',
                where PTFs for the above processes are to be stored.
           3.   An optional numeric limit, in Megabytes, of the total size of
                PTFs per-process.
   
           If /etc/mosix/private.conf does not exist, then all PTFs will be
           stored in "/private".  If the directory "/private" also does not
           exist, or if /etc/mosix/private.conf exists but does not contain a
           line with an appropriate letter in the first field ('O', 'C' or
           'G'), then no disk-space is allocated for PTFs of the affected
           processes, which usually means that processes requiring PTFs will
           not be able to run on this node.  Such guest processes that start
           using PTFs will migrate back to their home-nodes.
   
           When the third field is missing, it defaults to:
                   5 Gigabytes for local processes.
                   2 Gigabytes for processes from the same cluster.
                   1 Gigabyte for processes from other clusters.
           In any case, guest processes cannot exceed the size limit of their
           home-node even on nodes that allow them more space.
     
    /etc/mosix/target.conf
           This file contains the MRC (MOSIX Reach the Clouds) configuration,
           which determines who can launch MRC jobs that run on this node and
           what privileges and restrictions those launched jobs may have.
           Each line begins with a colon-terminated keyword, followed by spe-
           cific parameters for that keyword.  Most keywords can be repeated
           (except uids:, gids:, defuid:, defgid:).  The keywords are:
    
           accept:
                An IP address, or a range of consecutive IP addresses from
                where this node is willing to accept MRC jobs.  An example of
                a single IP address is:
      
                    accept: 101.102.103.104
            
                An example of a range of IP address is:
                
                    accept: 101.102.103.1 - 101.102.104.254
                 
                The address(es) may be followed by an alternative file-name
                (starting in '/'): in that case, the priviliges and restric-
                tions for jobs from the given address(es) are contained in
                the given file INSTEAD of /etc/mosix/target.conf.  For exam-
                ple:
              
                    accept: 1.2.3.1 - 1.2.3.254 /etc/mosix/special_case_1.2.3
    
                Alternative files have the same format as
                /etc/mosix/target.conf, except that they do not contain the
                keywords accept: and reject:.
     
           reject:
                IP addresses are specified as in accept: all MRC jobs will be
                rejected from those address(es).  This option is useful for
                excluding particular addresses in the middle of a larger
                range that is defined by accept:, for example:
          
                    accept: 10.20.30.1 - 10.20.31.254
                    reject: 10.20.30.255 - 10.20.31.0
                
     
           nodir:
                Prevent callers from overriding a given directory with a
                directory from their calling computer.  Note that overriding
                all ancesstor-directories is also prevented (since overriding
                them would override everything inside them as well, including
                the given directory).  For example:
           
                    nodir: /usr/share/X11
               
                prevents callers from overriding the directories
                "/usr/share/X11", "/usr/share" and "/usr" (it is anyway pro-
                hibited to override the root-directory).
           
           nodir_under:
                As nodir: but all subdirectories are also prevented from
                being overriden.
            
           allow-subdirs:
                If a caller asks to export a directory under a directory-name
                where:
                1.  No file or directory exists under that name.
                2.  The caller has no permission to create this directory.
                3.  Overriding that directory-name is not forbidden (eg. by
                    nodir: or nodir_under:)
                and the named-directory or any of its ancesstor-directories
                appears with the allow-subdirs: keyword, then the given
                directory will be specially created for the caller (it will
                be empty and with "root" ownership).  For example:
                
                    allow-subdirs: /tmp
                    allow-subdirs: /var/tmp
                
                  
           uids:
                A list of user-names and/or user-IDs that may be used by MRC
                callers.  A '*' denotes all users.  A '-' preceding a user-
                name or user-ID explicitly excludes that user.  The following
                example allows all user-ID's except "root":
                
                    uids: * -root               
                               
        
           gids:
                A list of group-names and/or group-IDs that may be used by
                MRC callers.  A '*' denotes all groups.  A '-' preceding a
                group-name or group-ID explicitly excludes that group.
               
           defuid:
                The default user-ID under which jobs from users that are not
                listed with the uids: keyword should run.  When this keyword
                is absent, the default is user-ID 65534 ("nobody").
                 
           defgid:
                The default group-ID under which jobs from user-groups that
                are not listed with the gids: keyword should run.  When this
                keyword is absent, the default is group-ID 65534 ("nobody").
                          
    /etc/mosix/retainpri
           This file contains an integer, specifying a delay in seconds: how
           long after all MOSIX processes of a certain priority (see above,
           /etc/mosix/priority) finish (or leave) to allow processes of lower
           priority (higher numbers) to start.  When this file is absent,
           there is no delay and processes with lower priority may arrive as
           soon as there are no processes with a higher priority.
  
    /etc/mosix/speed
           If this file exists, it should contain a positive integer
           (1-10,000,000), providing the relative speed of the processor: the
           bigger the faster, where 10,000 units of speed are equivalent to a
           3GHz Pentium-IV, and AMD (Athlon or Opteron) processors are, as a
           rule of thumb, 1.5 times faster than Intel processors of the same
           frequency.
   
           Normally this file is not necessary because the speed of the pro-
           cessor is automatically detected by the kernel when it boots.
           There are however two cases when you should consider using this
           option:
           1.   When you have a heterogeneous cluster and always use MOSIX to
                run a specific program (or programs) that perform better on
                certain processor-types than on others.
           2.   On Virtual-Machines that run over a hosting operating-system:
                in this case, the speed that the kernel detects is unreliable
                and can vary significantly depending on the load of the
                underlying operating-systems when it boots.
   
    /etc/mosix/maxguests
           If this file exists, it should contain an integer limit on the
           number of simultaneous guest-processes from other clusters. Oth-
           erwise, the maximum number of guest-processes from other clusters
           is set to the default of 8 times the number of processors.
   
    /etc/mosix/.log_mosrun
           When this file is present, information about invocations of
           mosrun(1) and process migrations will be recorded in the system-
           log (by default "/var/log/messages" on most Linux distributions).
   
    /etc/mosix/newtune
           Tuning constants optimizes the MOSIX performance by telling it
           about the costs of networked operations.  MOSIX has built-in tun-
           ing default constants.  This file is used to override them to suit
           your particular hardware and networks.
   
           For most users, This file is difficult to set up manually. Thus,
           MOSIX comes with a program to assemble it.  For more information,
           see topology(7).
        
KERNEL
    Sometimes a MOSIX release provides patches for more than one Linux kernel
    version.  Also, special kernel-patches are released from time to time to
    support particular Linux distributions (such as openSUSE): it is fine to
    mix different such kernels within the same cluster.  It is even OK to mix
    older or newer kernels from other MOSIX releases, so long as the first
    two numbers in their MOSIX version (run cat /proc/mosix/version to view
    the version) are identical to the first two numbers of your MOSIX
    release.
       
    The MOSIX kernel patch is required for fully operational MOSIX systems
    with process-migration.  A limited number of functions, such as batch
    jobs, queuing and viewing the loads, still works over any Linux kernel,
    even without the MOSIX kernel patch (or when the kernel is incompatible
    with the current MOSIX version).
       
    It is not recommended to have mixed clusters where some nodes have the
    MOSIX kernel-patch and others do not, but if you do so anyway, you should
    observe the following rules regarding job-queuing:
        
    On each "mixed" cluster, you may queue either migratable jobs or batch
    jobs, but not both.  If you choose to queue migratable jobs, then you
    should select a node with the MOSIX kernel-patch as the queue-manager. If
    you choose to queue batch jobs, then you should select a node without the
    MOSIX kernel-patch as the queue-manager (see above the section about con-
    figuring /etc/mosix/queue.conf).
         
INTERFACE FOR PROGRAMS
    The following interface is provided for programs running under mosrun(1)
    that wish to interface with their MOSIX run-time environment:
  
    All access to MOSIX is performed via the "open" system call, but the use
    of "open" is incidental and does not involve actual opening of files.  If
    the program were to run as a regular Linux program, those "open" calls
    would fail, returning -1, since the quoted files never exist, and
    errno(3) would be set to ENOENT.
   
    open("/proc/self/{special}", 0)
    reads a value from the MOSIX run-time environment.
   
    open("/proc/self/{special}", 1|O_CREAT, newval)
    writes a value to the MOSIX run-time environment.
      
    open("/proc/self/{special}", 2|O_CREAT, newval)
    both writes a new value and return the previous value.
      
    (the O_CREAT flag is only required when your program is compiled with the
    64-bit file-size option, but is harmless otherwise).
       
    Some "files" are read-only, some are write-only and some can do both
    (rw).  The "files" are as follows:
   
    /proc/self/migrate
           writing a 0 migrates back home; writing -1 causes a migration con-
           sideration; writing the unsigned value of an IP address or a logi-
           cal node number, attempts to migrate there.  Successful migration
           returns 0, failure returns -1 (write only)
   
    /proc/self/lock
           When locked (1), no automatic migration may occur (except when
           running on the current node is no longer allowed); when unlocked
           (0), automatic migration can occur. (rw)
    
    /proc/self/whereami
           reads where the program is running: 0 if at home, otherwise usu-
           ally an unsigned IP address, but if possible, its corresponding
           logical node number. (read only)
    
    /proc/self/nmigs
           reads the total number of migrations performed by this process and
           its MOSRUN ancesstors before it was born. (read only)
  
    /proc/self/sigmig
           Reads/sets a signal number (1-64 or 0 to cancel) to be received
           after each migration. (rw)
   
    /proc/self/glob
           Reads/modifies the process class.  Processes of class 0 are not
           allowed to migrate outside the local cluster or even outside the
           local partition.  Classes can also affect the automatic-freezing
           policy. (rw)
   
    /proc/self/needmem
           Reads/modifies the process's memory requirement in Megabytes, so
           it does not automatically migrate to nodes with less free memory.
           Acceptable values are 0-262143.  (rw)
     
    /proc/self/unsupportok
           when 0, unsupported system-calls cause the process to be killed;
           when 1 or 2, unsupported system-calls return -1 with errno set to
           ENOSYS; when 2, an appropriate error-message will also be written
           to stderr. (rw)
     
    /proc/self/clear
           clears process statistics. (write only)
    
    /proc/self/cpujob
           Normally when 0, system-calls and I/O are taken into account for
           migration considerations.  When set to 1, they are ignored. (rw)
    
    /proc/self/localtime
           When 0, gettimeofday(2) is always performed on the home node.
           When 1, the date/time is taken from where the process is running.
           (rw)
   
    /proc/self/decayrate
           Reads/modifies the decay-rate per second (0-10000): programs can
           alternate between periods of intensive CPU and periods of demand-
           ing I/O.  Decisions to migrate should be based neither on momen-
           tary program behaviour nor on extremely long term behaviour, so a
           balance must be struck, where old process statistics gradually
           decay in favour of newer statistics.  The lesser the decay rate,
           the more weight is given to new information.  The higher the decay
           rate, the more weight is given to older information.  This option
           is provided for users who know well the cyclic behavior of their
           program. (rw)
       
    /proc/self/checkpoint
           When writing (any value) - perform a checkpoint.  When only read-
           ing - return the version number of the next checkpoint to be made.
           When reading and writing - perform a checkpoint and return its
           version.  Returns -1 if the checkpoint fails, 0 if writing only
           and checkpoint is successful. (rw)
   
    /proc/self/checkpointfile
           The third argument (newval) is a pointer to a file-name to be used
           as the basis for future checkpoints (see mosrun(1)).  (write only)
    
    /proc/self/checkpointlimit
           Reads/modifies the maximal number of checkpoint files to create
           before recycling the checkpoint version number.  A value of 0
           unlimits the number of checkpoints files.  The maximal value
           allowed is 10000000.
    
    /proc/self/checkpointinterval
           When writing, sets the interval in minutes for automatic check-
           points (see mosrun(1)).  A value of 0 cancels automatic check-
           points.  The maximal value allowed is 10000000.  Note that writing
           has a side effect of reseting the time left to the next check-
           point. Thus, writing too frequently is not recommended.  (rw)
   
    open("/proc/self/in_cluster", O_CREAT, node); and
    open("/proc/self/in_partition", O_CREAT, node);
           return 1 if the given node is in the same cluster/partition, 0
           otherwise.  The node can be either an unsigned, host-order IP
           address, or a node-number (listed in /etc/mosix/userview.map).
   
    More functions are available through the direct_communication(7) feature.
  
    The following information is available via the /proc file system for
    everyone to read (not just within the MOSIX run-time environment):
  
    /proc/{pid}/from
           The IP address (a.b.c.d) of the process' home-node ("0" if a local
           process).
   
    /proc/{pid}/where
           The IP address (a.b.c.d) where the process is runing ("0" if run-
           ning here).
   
    /proc/{pid}/class
           The class of the process.
   
    /proc/{pid}/origipid
           The original PID of the process on its home-node ("0" if a local
           process).
    
    /proc/{pid}/freezer
           Whether and why the process was frozen:
    
           0      Not frozen
   
           1      Frozen automatically due to high load.
   
           2      Frozen by the evacuation policy, to prevent flooding by
                  arriving processes when clusters are disconnected.
   
           3      Frozen due to manual request.
   
           -66    This is a guest process from another home-mode (freezing is
                  always on the home-node, hence not applicable here).
  
    Attempting to read the above for non-MOSIX processes returns the string
    "-3".
     
STARTING MOSIX
    To start MOSIX, run /etc/init.d/mosix start.  Alternately, run mosd. 
 
SECURITY
    All nodes within a MOSIX cluster and multi-cluster must trust each
    other's super-user(s) - otherwise the security of the whole cluster or
    multi-cluster is compromized.
  
    Hostile computers must not be allowed physical access to the internal
    MOSIX network where they could masquerade as having IP addresses of
    trusted nodes.
   
SEE ALSO
    mosrun(1), mosctl(1), migrate(1), setpe(1), mon(1), mosps(1), mosps(1),
    timeof(1), moskillall(1), mosq(1), bestnode(1), mospipe(1), mrc(1),
    direct_communication(7), topology(7).

HISTORY
    This is the 10-th version of MOSIX.  The MOSIX wiki has more information
    about the previous releases.

MOSIX                           February 2009                              MOSIX