2.1   Requirements

There a three kinds of nodes, each requiring a specific software configuration.

These are :

• the server node, which will hold all of OAR "smartness" ;
• the login nodes, on which you will be allowed to login, then reserve some computational nodes ;
• the computational nodes (a.k.a. the nodes), on which the jobs will run.

On every node, the "sentinelle" binary can be installed (it is not necessary). This tool is used to launch commands on several computers, at a same time. It ships with the current OAR package (but it is also available separately in Taktuk in an Open Source License).

On every nodes (server, login, computational), the following packages must be installed :

• sudo
• Perl
• Perl-base
• openssh (server and client)

On the OAR server and on the login nodes, the following packages must be installed:

• Perl-Mysql
• Perl-DBI
• MySQL
• MySQL-shared
• libmysql

From now on, we will suppose all the Perl/Mysql packages are correctly installed and configured and mysql database is started.

2.2   Configuration of the cluster

The following steps have to be done, prior to installing OAR:

• add a user named "oar" in the group "oar" on every node

• let the user "oar" connect through ssh from any node to any node WITHOUT password. To achieve this, here is some standard procedure for OpenSSH:

  • create a set of ssh keys for the user "oar" with ssh-keygen (for instance 'id_dsa.pub' and 'id_dsa')

  • copy these keys on each node of the cluster in the ".ssh" folder of the user "oar"

  • append the contents of 'id_dsa.pub' to the file "~/.ssh/authorized_keys"

  • in "~/.ssh/config" add the lines:

    Host *
        ForwardX11 no
        StrictHostKeyChecking no
        PasswordAuthentication no
        AddressFamily inet
    

  • test the ssh connection between (every) two nodes : there should not be any prompt.

• grant the user "oar" the permission to execute commands with root privileges. To achieve that, OAR makes use of sudo. As a consequence, /etc/sudoers must be configured. Please use visudo to add the following lines:

  Defaults>oar    env_reset,env_keep = "OARLIB OARUSER OARDIR PWD
  PERL5LIB DISPLAY OARCONFFILE"
  
  Cmnd_Alias OARCMD = /usr/lib/oar/oarnodes, /usr/lib/oar/oarstat,\
  /usr/lib/oar/oarsub, /usr/lib/oar/oardel, /usr/lib/oar/oarhold,\
  /usr/lib/oar/oarnotify, /usr/lib/oar/oarresume
  %oar ALL=(oar) NOPASSWD: OARCMD
  
  oar ALL=(ALL) NOPASSWD:ALL
  
  Defaults:www-data   env_keep += "SCRIPT_NAME SERVER_NAME
  SERVER_ADMIN HTTP_CONNECTION REQUEST_METHOD CONTENT_LENGTH
  SCRIPT_FILENAME SERVER_SOFTWARE HTTP_TE QUERY_STRING REMOTE_PORT
  HTTP_USER_AGENT SERVER_PORT SERVER_SIGNATURE REMOTE_ADDR
  CONTENT_TYPE SERVER_PROTOCOL PATH REQUEST_URI GATEWAY_INTERFACE
  SERVER_ADDR DOCUMENT_ROOT HTTP_HOST"
  
  www-data ALL=(oar) NOPASSWD: /usr/lib/oar/oar-cgi
  

There are a three different flavors of installation :

• server: install the daemon which must be running on the server
• user: install all the tools needed to submit and manage jobs for the users (oarsub, oarstat, oarnodes, ...)
• node: install the tools for a computing node

The installation is straightforward:

• become root

• go to OAR source repository

• You can set Makefile variables in the command line to suit your configuration (change "OARHOMEDIR" to the home of your user oar and "PREFIX" where you want to copy all OAR files).

• run make <module> [module] ...

  where module := { server-install | user-install | node-install | doc-install | debian-package }

  OPTIONS := { OARHOMEDIR | OARCONFDIR | OARUSER | PREFIX | MANDIR | OARDIR | BINDIR | SBINDIR | DOCDIR }

• Edit /etc/oar.conf file to match your cluster configuration.

• Make sure that the PATH environnement variable contains $PREFIX/$BINDIR of your installation (default is /usr/local/bin).

Initialization of OAR database (MySQL) is achieved using oar_mysql_db_init script provided with the server module installation and located in $PREFIX/sbin (/usr/local/sbin in default Makefile).

If you want to use a postgres SQL server then there is currently no automatic installation script. You have to add a new user which can connect on a new oar database(use the commands createdb and createuser). After that, you have to authorize network connections on the postgresql server in the postgresql.conf (uncomment tcpip_socket = true). Moreover a line like

host    oar         oar            X.X.X.X/Y    md5

to enable oar user to connect to the database.

Then you can import the database scheme stored in oar_postgres.sql (use the SQL command "\i").

For more informations about postgresql, go to http://www.postgresql.org/.

Note: The same machine may host several or even all modules.

Note about X11: The easiest and scalable way to use X11 application on cluster nodes is to open X11 ports and set the right DISPLAY environment variable by hand. Otherwise users can use X11 forwarding via ssh to access cluster frontal. After that you must configure ssh server on this frontal with

X11Forwarding yes
X11UseLocalhost no

With this configuration, users can launch X11 applications after a 'oarsub -I' on the given node.

Cpusets provide a mechanism for assigning a set of CPUs and Memory
Nodes to a set of tasks.

Cpusets constrain the CPU and Memory placement of tasks to only
the resources within a tasks current cpuset.  They form a nested
hierarchy visible in a virtual file system.  These are the essential
hooks, beyond what is already present, required to manage dynamic
job placement on large systems.

Each task has a pointer to a cpuset.  Multiple tasks may reference
the same cpuset.  Requests by a task, using the sched_setaffinity(2)
system call to include CPUs in its CPU affinity mask, and using the
mbind(2) and set_mempolicy(2) system calls to include Memory Nodes
in its memory policy, are both filtered through that tasks cpuset,
filtering out any CPUs or Memory Nodes not in that cpuset.  The
scheduler will not schedule a task on a CPU that is not allowed in
its cpus_allowed vector, and the kernel page allocator will not
allocate a page on a node that is not allowed in the requesting tasks
mems_allowed vector.

If a cpuset is cpu or mem exclusive, no other cpuset, other than a direct
ancestor or descendent, may share any of the same CPUs or Memory Nodes.
A cpuset that is cpu exclusive has a sched domain associated with it.
The sched domain consists of all cpus in the current cpuset that are not
part of any exclusive child cpusets.
This ensures that the scheduler load balacing code only balances
against the cpus that are in the sched domain as defined above and not
all of the cpus in the system. This removes any overhead due to
load balancing code trying to pull tasks outside of the cpu exclusive
cpuset only to be prevented by the tasks' cpus_allowed mask.

A cpuset that is mem_exclusive restricts kernel allocations for
page, buffer and other data commonly shared by the kernel across
multiple users.  All cpusets, whether mem_exclusive or not, restrict
allocations of memory for user space.  This enables configuring a
system so that several independent jobs can share common kernel
data, such as file system pages, while isolating each jobs user
allocation in its own cpuset.  To do this, construct a large
mem_exclusive cpuset to hold all the jobs, and construct child,
non-mem_exclusive cpusets for each individual job.  Only a small
amount of typical kernel memory, such as requests from interrupt
handlers, is allowed to be taken outside even a mem_exclusive cpuset.

User level code may create and destroy cpusets by name in the cpuset
virtual file system, manage the attributes and permissions of these
cpusets and which CPUs and Memory Nodes are assigned to each cpuset,
specify and query to which cpuset a task is assigned, and list the
task pids assigned to a cpuset.

2.3   Visualization tools installation

There are two different tools. One, named Monika, displays the current cluster state with all active and waiting jobs. The other, named drawgantt, diplays node occupation in a lapse of time. These tools are CGI scripts and generate HTML pages.

You can install these in this way:

drawgantt:

• Make sure you installed "ruby", "libdbd-mysql-ruby" or "libdbd-pg-ruby" and "libgd-ruby1.8" packages.
• Copy "drawgantt.cgi" and "drawgantt.conf" in the CGI folder of your web server (ex: /usr/lib/cgi-bin/ for Debian).
• Copy all icons and javascript files in a folder that web server can find them (ex: /var/www/oar/Icons and /var/www/oar/Icons).
• Make sure that these files can be read by the web server user.
• Edit "drawgantt.conf" and change tags to fit your configuration.

Monika:

• The package "perl-AppConfig" is required.
• Read INSTALL file in the monika repository.

2.4   Debian packages

OAR is also released under Debian packages (or Ubuntu). You can find them at http://oar.imag.fr/download.html.

If you want to add it as a new source in your /etc/apt/sources.list then add the line:

deb http://oar.imag.fr/download ./

The installation will ask you if you want to initialize the nodes. It will copy the oar SSH key on each specified nodes. You can skeep this step by you will have to do this manually.

After installing packages, you have to edit the configuration file on the server, submition nodes and computing nodes to fite your needs.

2.5   Starting

First, you must start OAR daemon on the server (its name is "Almighty").

• if you have installed OAR from sources, become oar user and launch command "Almighty" (it stands in $PREFIX/sbin).
• if you have installed OAR from Debian packages, use the script "/etc/init.d/oar-server" to start the daemon.

Then you have to insert new resources in the database via the command oarnodesetting. If you want to have an idea how does it work then launch $PREFIX/oar/detect_new_resources.sh. It will print right commands execute with an appropriate value for the memory and the cpuset properties.

If you want to initialize your whole cluster in 1 command you can use this one (tune it to fite your cluster). You must be oar to run this command because oarnodesetting will called and the sentinelle.pl will log onto all nodes stored in "node_list.txt" file without password:

export PREFIX=/var/lib
$PREFIX/oar/sentinelle.pl -f node_list.txt \
-p "$PREFIX/oar/detect_new_resources.sh" | sh

Then you can launch the oarnodes command and see all new resources inserted.

2.6   Further informations

For further information, please check http://oar.imag.fr/.

3.1   Description of the different commands

All user commands are installed on cluster login nodes. So you must connect to one of these computers first.

-f                    : prints each job in full details
-j job_id             : prints the specified job_id informations (even if it is finished)
--sql "sql where"     : Restricts display with the SQL where clause on the table jobs
-g "d1,d2"            : prints history of jobs and state of resources between two dates.
-D                    : formats outputs in Perl Dumper
-X                    : formats outputs in XML
-Y                    : formats outputs in YAML

# oarstat
# oarstat -j 42 -f
# oarstat --sql "project = 'p1'"

-a                : shows all resources with their properties
-r                : show only properties of a resource
-s                : shows only resource states
-l                : shows only resource list
--sql "sql where" : Display resources which matches this sql where clause
-D                : formats outputs in Perl Dumper
-X                : formats outputs in XML
-Y                : formats outputs in YAML

# oarnodes
# oarnodes -s
# oarnodes --sql "state = 'Suspected'"

3.1.3   oarsub

The user can submit a job with this command. So, what is a job in our context?

A job is defined by needed resources and a script/program to run. So, the user must specify how many resources and what kind of them are needed by his application. Thus, OAR system will give him or not what he wants and will control the execution. When a job is launched, OAR executes user program only on the first reservation node. So this program can access some environment variables to know its environment:

$OAR_NODEFILE                 contains the name of a file which lists
                              all reserved nodes for this job
$OAR_JOB_ID                   contains the OAR job identificator
$OAR_RESOURCE_PROPERTIES_FILE contains the name of a file which lists
                              all resources and their properties
$OAR_JOB_NAME                 name of the job given by the "-n" option
$OAR_RESOURCE_PROPERTIES_FILE contains the detailed resources used by the
                              job
$OAR_PROJECT_NAME             job project name

Options:

-q "queuename" : specify the queue for this job
-I : turn on INTERACTIVE mode (OAR gives you a shell instead of executing a
     script)
-l "resource description" : defines resource list requested for this job;
                            the different parameters are resource properties
                            registered in OAR database; see examples below.
                            (walltime : Request maximun time. Format is
                            [hour:mn:sec|hour:mn|hour]; after this elapsed
                            time, the job will be killed)
-p "properties" : adds constraints for the job
                  (format is a WHERE clause from the SQL syntax)
-S, --Scanscript : in batch mode, asks oarsub to scan the given script for
                   OAR directives (#OAR -l ...)
-r "2007-05-11 23:32:03" : asks for a reservation job to begin at the
                           date in argument
-C job_id : connects to a reservation in Running state
-k "duration" : asks OAR to send the checkpoint signal to the first processus
                of the job "number_of_seconds" before the walltime
--signal "signal name" : specify the signal to use when checkpointing
-t "type name" : specify a specific type (deploy, besteffort, cosystem,
                 checkpoint)
-d "directory path" : specify the directory where to launch the command
                      (default is current directory)
-n "job name" :  specify an arbitrary name for the job
-a job_id : anterior job that must be terminated to start this new one
--project : Specify the name of the project corresponding to the job.
--notify "method" : specify a notification method(mail or command); ex:
                    --notify "mail:name@domain.com"
                    --notify "exec:/path/to/script args"
--stdout "file name" : specify the name of the standard output file
--stderr "file name" : specify the name of the error output file
--resubmit job_id : resubmit the given job to a new one
--force_cpuset_name "cpuset name" : Instead of using job_id for the cpuset
                                    name you can specify one (WARNING: if
                                    several jobs have the same cpuset name
                                    then processes of a job could be killed
                                    when another finished on the same
                                    computer)
--hold : Set the job state into Hold instead of Waiting; so it is not scheduled (you must run "oarresume" to turn it into the Waiting state).
-D : Print result in DUMPER format.
-Y : Print result in XML format.
-X : Print result in YAML format.

Wanted resources have to be described in a hierarchical manner (this is the "-l" syntax option).

Moreover it is possible to give a property that they must match.

So the long and complete syntax is of the form:

"{ sql1 }/prop1=1/prop2=3+{sql2}/prop3=2/prop4=1/prop5=1+...,walltime=1:00:00"

where:

• sql1 : WHERE sql clause on the table resources that filters resource names used in the hierarchical description
• prop1 : first type of resources
• prop2 : second type of resources
• + : add another resource hierarchy to the previous one
• sql2 : WHERE sql clause to apply on the second hierarchy request
• ...

So we want to reserve 3 resources with the same value of the type prop2 and with the same property prop1 and these resources must fit sql1. To that possible resources we want to add 2 others which fit sql2 and the hierarchy /prop3=2/prop4=1/prop5=1.

Example of a resource hierarchy and 2 different oarsub commands

hierarchical_resources.svg

Examples

# oarsub -l /node=4 test.sh

(the "test.sh" script will be run on 4 entire nodes in the default queue with the default walltime)

# oarsub -q default -l walltime=50:30:00,/node=10/cpu=3,walltime=2:15:00 \
  -p "switch = 'sw1'" /home/users/toto/prog

(the "/home/users/toto/prog" script will be run on 10 nodes with 3 cpus (so a total of 30 cpus) in the default queue with a walltime of 2:15:00. Moreover "-p" option restricts resources only on the switch 'sw1')

# oarsub -r "2004-04-27 11:00:00" -l /node=12/cpu=2

(a reservation will begin at "2004-04-27 11:00:00" on 12 nodes with 2 cpus on each one)

#  oarsub -C 42

(connects to the job 42 on the first node and set all OAR environment variables)

# oarsub -I

(gives a shell on a resource)

--sql     : delete/checkpoint jobs which respond to the SQL where clause
            on the table jobs (ex: "project = 'p1'")
-c job_id : send checkpoint signal to the job (signal was
            definedwith "--signal" option in oarsub)

# oardel 14 42

# oardel -c 42

--sql : hold jobs which respond to the SQL where clause on the table
        jobs (ex: "project = 'p1'")
-r    : Manage not only Waiting jobs but also Running one
        (can suspend the job)

--sql : resume jobs which respond to the SQL where clause on the table
        jobs (ex: "project = 'p1'")

3.2   Visualisation tools

4.1   Administrator commands

-l : list properties
-a NAME : add a property
  -c : sql new field of type VARCHAR(255) (default is integer)
-d NAME : delete a property
-r "OLD_NAME,NEW_NAME" : rename property OLD_NAME into NEW_NAME

# oarproperty -a cpu_freq
# oarproperty -a type
# oarproperty -r "cpu_freq,freq"

-a    : add a new resource
-s    : state to assign to the node:
        * "Alive" : a job can be run on the node.
        * "Absent" : administrator wants to remove the node from the pool
           for a moment.
        * "Dead" : the node will not be used and will be deleted.
-h    : specify the node name (override hostname).
-r    : specify the resource number
--sql : get resource identifiers which respond to the
        SQL where clause on the table jobs
        (ex: "type = 'default'")
-p    : change the value of a property specified resources.
-n    : specify this option if you do not want to wait the end of jobs running
        on this node when you change its state into "Absent" or "Dead".

    Almighty_tag    send this tag to the Almighty (default is TERM)
-e                  active an existing queue
-d                  inactive an existing queue
-E                  active all queues
-D                  inactive all queues
--add_queue         add a new queue; syntax is name,priority,scheduler
                    (ex: "name,3,"oar_sched_gantt_with_timesharing"
--remove_queue      remove an existing queue
-l                  list all queues and there status
-h                  show this help screen
-v                  print OAR version number

4.2   Database scheme

Database scheme (red lines seem PRIMARY KEY, blue lines seem INDEX)

db_scheme.svg

Note : all dates and duration are stored in an integer manner (number of seconds since the EPOCH).

6.1   Almighty

This module is the OAR server. It decides what actions must be performed. It is divided into 2 processes:

• One listens to a TCP/IP socket. It waits informations or commands from OAR user program or from the other modules.
• Another one deals with commands thanks to an automaton and launch right modules one after one.

6.2   Sarko

This module is executed periodically by the Almighty (default is every 30 seconds).

The jobs of Sarko are :

• Look at running job walltimes and ask to frag them if they had expired.
• Detect if fragged jobs are really fragged otherwise asks to exterminate them.
• In "Desktop Computing" mode, it detects if a node date has expired and asks to change its state into "Suspected".
• Can change "Suspected" resources into "Dead" after DEAD_SWITCH_TIME seconds.

6.3   Judas

This is the module dedicated to print and log every debugging, warning and error messages.

6.4   Leon

This module is in charge to delete the jobs. Other OAR modules or commands can ask to kill a job and this is Leon which performs that.

There are 2 frag types :

• normal : Leon tries to connect to the first node allocated for the job and terminates the job. oarexec to end itself.
• exterminate : after a timeout if the normal method did not succeed then Leon notifies this case and clean up the database for these jobs. So OAR doesn't know what occured on the node and Suspects it.

6.5   NodeChangeState

This module is in charge of changing resource states and checking if there are jobs on these.

It also checks all pending events in the table event_logs.

6.6   Scheduler

This module checks for each reservation jobs if it is valid and launches them at the right time.

Scheduler launches all gantt scheduler in the order of the priority specified in the database and update all visualization tables (gantt_jobs_predictions_visu and gantt_jobs_resources_visu).

6.7   Runner

This module launches OAR effective jobs. These processes are run asynchronously with all modules.

For each job, the Runner uses OPENSSH_CMD to connect to the first node of the reservation and propagate a Perl script which handles the execution of the user command.

7.1   How does an interactive oarsub work?

Interactive oarsub decomposition

interactive_oarsub_scheme.svg

7.2   Job launch

For PASSIVE jobs, the mechanism is similar to the INTERACTIVE one, except for the shell launched from the frontal node.

The job is finished when the user command ends. Then oarexec return its exit value (what errors occured) on the Almighty via the SERVER_PORT if DETACH_JOB_FROM_SERVER was set to 1 otherwise it returns directly.

7.3   CPUSET

If the "--force_cpuset_name" option of the oarsub command is not defined then OAR will use job identifier. The CPUSET name is effectively created on each nodes and is composed as "user_cpusetname".

So if a user specifies "--force_cpuset_name" option, he will not be able to disturb other users.

OAR system steps:

1. Before each job, the Runner initialize the CPUSET (see CPUSET definition) with OPENSSH_CMD and an efficient launching tool : Taktuk. If it is not installed and configured (TAKTUK_CMD) then OAR uses an internal launching tool less optimized.
2. After each job, OAR deletes all processes stored in the associated CPUSET. Thus all nodes are clean after a OAR job.

If you don't want to use this feature, you can, but nothing will warranty that every user processes will be killed after the end of a job.

If you want you can implement your own cpuset management. This is done by editing 3 files (see also CPUSET installation):

• cpuset_manager.pl : this script creates the cpuset on each nodes and also delete it at the end of the job. For more informations, you have to look at this script (there are several comments).
• oarsh : (OARSH) this script is used to replace the standard "ssh" command. It gets the cpuset name where it is running and transfer this information via "ssh" and the "SendEnv" option. In this file, you have to change the "get_current_cpuset" function.
• oarsh_shell : (OARSH_SHELL) this script is the shell of the oar user on each nodes. It gets environment variables and look at if there is a cpuset name. So if there is one it assigns the current process and its father to this cpusetname. So all further user processes will remind in the cpuset. In this file you just have to change the "add_process_to_cpuset" function.

7.4   Suspend/resume

Jobs can be suspended with the command oarhold (send a "SIGSTOP" on every processes on every nodes) to allow other jobs to be executed.

"Suspended" jobs can be resumed with the command oarresume (send a "SIGSTOP" on every suspended processes on every nodes). They will pass into "Running" when assigned resources will be free.

IMPORTANT: This feature is available only if CPUSET is configured.

You can specify 2 scripts if you have to perform any actions just after (JUST_AFTER_SUSPEND_EXEC_FILE) suspend and just before resume (JUST_BEFORE_RESUME_EXEC_FILE).

Moreover you can perform other actions (than send signals to processes) if you want: just edit the "suspend_resume_manager.pl" file.

7.5   Job deletion

Leon tries to connect to OAR Perl script running on the first job node (find it thanks to the file /tmp/oar/pid_of_oarexec_for_jobId_id) and sends a "SIGTERM" signal. Then the script catch it and normally end the job (kill processes that it has launched).

If this method didn't succeed then Leon will flush the OAR database for the job and nodes will be "Suspected" by NodeChangeState.

If your job is check pointed and is of the type idempotent (oarsub "-t" option) and its exit code is equal to 0 then another job is automatically created and scheduled with same behaviours.

7.6   Checkpoint

The checkpoint is just a signal sent to the program specified with the oarsub command.

If the user uses "-k" option then Sarko will ask the OAR Perl script running on the first node to send the signal to the process (SIGUSR2 or the one specified with "--signal").

You can also use oardel command to send the signal.

7.7   Scheduling

General steps used to schedule a job:

1. All previous scheduled jobs are stored in a Gantt data structure.
2. All resources that match property constraints of the job("-p" option and indication in the "{...}" from the "-l" option of the oarsub) are stored in a tree data structure according to the hierarchy given with the "-l" option.
3. Then this tree is given to the Gantt library to find the first hole where the job can be launched.
4. The scheduler stores its decision into the database in the gantt_jobs_predictions and gantt_jobs_resources tables.

See User section from the FAQ for more examples and features.

7.8   User notification

This section explains how the "--notify" oarsub option is handled by OAR:

• The user wants to receive an email:

  The syntax is "mail:name@domain.com". Mail section in the Configuration file must be present otherwise the mail cannot be sent.

• The user wants to launch a script:

  The syntax is "exec:/path/to/script args". OAR server will connect (using OPENSSH_CMD) on the node where the oarsub command was invoked and then launches the script with in argument : job_id, job_name, tag, comments.

  (tag is a value in : "START", "END", "ERROR")

7.9   Accounting aggregator

In the Configuration file you can set the ACCOUNTING_WINDOW parameter. Thus the command oaraccounting will split the time with this amount and feed the table accounting.

So this is very easily and faster to get usage statistics of the cluster. We can see that like a "data warehousing" information extraction method.

7.10   Dynamic nodes coupling features

We are working with the Icatis company on clusters composed by Intranet computers. These nodes can be switch in computing mode only at specific times. So we have implemented a functionality that can request to power on some hardware if they can be in the cluster.

We are using the field cm_availability from the table resources to know when a node will be inaccessible in the cluster mode (easily settable with oarnodesetting command). So when the OAR scheduler wants some potential available computers to launch the jobs then it executes the command SCHEDULER_NODE_MANAGER_WAKE_UP_CMD.

Moreover if a node didn't execute a job for SCHEDULER_NODE_MANAGER_IDLE_TIME seconds and no job is scheduled on it before SCHEDULER_NODE_MANAGER_SLEEP_TIME seconds then OAR will launch the command SCHEDULER_NODE_MANAGER_SLEEP_CMD.

7.11   Timesharing

It is possible to share the slot time of a job with other ones. To perform this feature you have to specify the type timesharing when you use oarsub.

You have 4 different ways to share your slot:

1. timesharing=*,* : This is the default behavior if nothing but timesharing is specified. It indicates that the job can be shared with all users and every job names.
2. timesharing=user,* : This indicates that the job can be shared only with the same user and every job names.
3. timesharing=*,job_name* : This indicates that the job can be shared with all users but only one with the same name.
4. timesharing=user,job_name : This indicates that the job can be shared only with the same user and one with the same job name.

See User section from the FAQ for more examples and features.

7.12   Besteffort jobs

Besteffort jobs are scheduled in the besteffort queue. Their particularity is that they are deleted if another not besteffort job want resources where they are running.

For example you can use this feature to maximize the use of your cluster with multiparametric jobs. This what it is done by the CIGRI project.

When you submit a job you have to use "-t besteffort" option of oarsub to specify that this is a besteffort job.

Note : a besteffort job cannot be a reservation.

7.13   Cosystem jobs

This feature enables to reserve some resources without launching any program on corresponding nodes. Thus nothing is done by OAR when a job is starting (no prologue, no epilogue on the server nor on the nodes).

This is useful with an other launching system that will declare its time slot in OAR. So yo can have two different batch scheduler.

When you submit a job you have to use "-t cosystem" option of oarsub to specify that this is a besteffort job.

These jobs are stopped by the oardel command or when they reach their walltime.

7.14   Deploy jobs

This feature is useful when you want to enable the users to reinstall their reserved nodes. So the OAR jobs will not log on the first computer of the reservation but on the DEPLOY_HOSTNAME.

So prologue and epilogue scripts are executed on DEPLOY_HOSTNAME and if the user wants to launch a script it is also executed on DEPLOY_HOSTNAME.

OAR does nothing on computing nodes because they normally will be rebooted to install a new system image.

This feature is strongly used in the Grid5000 project with Kadeploy tools.

When you submit a job you have to use "-t deploy" option of oarsub to specify that this is a deploy job.

7.15   Desktop computing

If you cannot contact the computers via SSH you can install the "desktop computing" OAR mode. This kind of installation is based on two programs:

• oar-cgi : this is a web CGI used by the nodes to communicate with the OAR server.
• oar-agent.pl : This program asks periodically the server web CGI to know what it has to do.

This method replaces the SSH command. Computers which want to register them into OAR just has to be able to contact OAR HTTP server.

In this situation we don't have a NFS file system to share the same repertories over all nodes so we have to use a stagein/stageout solution. In this case you can use the oarsub option "stagein" to migrate your data.

8.1   User

oarsub -I -l '{switch = "sw1" or switch = "sw5"}/switch=1+/node=1'

oarsub -k 600 --signal 2 -t idempotent /path/to/prog

oarsub -t besteffort /path/to/prog

8.2   Administrator

oarsub -I -l "/switch=2/nodes=10+{type = 'mathlab'}/licence=20"

9.1   version 2.0.0++ enhanced:

• Now, with the ability to declare any type of resources like licences, VLAN, IP range, computing resources must have the type default and a network_address not null.

• Possibility to declare associated resources like licences, IP ranges, ... and to reserve them like others.

• Now you can connect to your jobs (not only for reservations).

• Add "cosystem" job type (execute and do nothing for these jobs).

• New scheduler : "oar_sched_gantt_with_timesharing". You can specify jobs with the type "timesharing" that indicates that this scheduler can launch more than 1 job on a resource at a time. It is possible to restrict this feature with words "user and name". For example, '-t timesharing=user,name' indicates that only a job from the same user with the same name can be launched in the same time than it.

• Add PostGresSQL support. So there is a choice to make between MySQL and PostgresSQL.

• New approach for the scheduling : administrators have to insert into the databases descriptions about resources and not nodes. Resources have a network address (physical node) and properties. For example, if you have dual-processor, then you can create 2 different resources with the same natwork address but with 2 different processor names.

• The scheduler can now handle resource properties in a hierarchical manner. Thus, for example, you can do "oarsub -l /switch=1/cpu=5" which submit a job on 5 processors on the same switch.

• Add a signal handler in oarexec and propagate this signal to the user process.

• Support '#OAR -p ...' options in user script.

• Add in oar.conf:

  • DB_BASE_PASSWD_RO : for security issues, it is possible to execute request with parts specified by users with a read only account (like "-p" option).
  • OARSUB_DEFAULT_RESOURCES : when nothing is specified with the oarsub command then OAR takes this default resource description.
  • OAREXEC_DEBUG_MODE : turn on or off debug mode in oarexec (create /tmp/oar/oar.log on nodes).
  • FINAUD_FREQUENCY : indicates the frequency when OAR launchs Finaud (search dead nodes).
  • SCHEDULER_TIMEOUT : indicates to the scheduler the amount of time after what it must end itself.
  • SCHEDULER_JOB_SECURITY_TIME : time between each job.
  • DEAD_SWITCH_TIME : after this time Absent and Suspected resources are turned on the Dead state.
  • PROLOGUE_EPILOGUE_TIMEOUT : the possibility to specify a different timeout for prologue and epilogue (PROLOGUE_EPILOGUE_TIMEOUT).
  • PROLOGUE_EXEC_FILE : you can specify the path of the prologue script executed on nodes.
  • EPILOGUE_EXEC_FILE : you can specify the path of the epilogue script executed on nodes.
  • GENERIC_COMMAND : a specific script may be used instead of ping to check aliveness of nodes. The script must return bad nodes on STDERR (1 line for a bad node and it must have exactly the same name that OAR has given in argument of the command).
  • JOBDEL_SOFTWALLTIME : time after a normal frag that the system waits to retry to frag the job.
  • JOBDEL_WALLTIME : time after a normal frag that the system waits before to delete the job arbitrary and suspects nodes.
  • LOG_FILE : specify the path of OAR log file (default : /var/log/oar.log).

• Add wait() in pingchecker to avoid zombies.

• Better code modularization.

• Remove node install part to launch jobs. So it is easier to upgrade from one version to an other (oarnodesetting must already be installed on each nodes if we want to use it).

• Users can specify a method to be notified (mail or script).

• Add cpuset support

• Add prologue and epilogue script to be executed on the OAR server before and after launching a job.

• Add dependancy support between jobs ("-a" option in oarsub).

• In oarsub you can specify the launching directory ("-d" option).

• In oarsub you can specify a job name ("-n" option).

• In oarsub you can specify stdout and stderr file names.

• User can resubmit a job (option "--resubmit" in oarsub).

• It is possible to specify a read only database account and it will be used to evaluate SQL properties given by the user with the oarsub command (more scecure).

• Add possibility to order assigned resources with their properties by the scheduler. So you can privilege some resources than others (SCHEDULER_RESOURCE_ORDER tag in oar.conf file)

• a command can be specified to switch off idle nodes (SCHEDULER_NODE_MANAGER_SLEEP_CMD, SCHEDULER_NODE_MANAGER_IDLE_TIME, SCHEDULER_NODE_MANAGER_SLEEP_TIME in oar.conf)

• a command can be specified to switch on nodes in the Absent state according to the resource property cm_availability in the table resources (SCHEDULER_NODE_MANAGER_WAKE_UP_CMD in oar.conf).

• if a job goes in Error state and this is not its fault then OAR will resubmit this one.

9.2   version 1.6.1:

• initialise the "ganttJobsPrediction" table with a right reference date (1970-01-01 00:00:01)

• oarsub has the "-k, --checkpoint" option. It specifies the number of seconds before job walltime to send a SIGUSR2 on it.

• You can see the list of events for jobs with -f option in oarstat

• oardel can now send SIGUSR2 to the oarexec of a job (-c option)

• oardel can now delete several jobs

• Add a signal handler in oarexec and propagate this signal to the user process

• Support '#OAR -p ...' options in user script

• Add in oar.conf the possibility to specify a different timeout for prologue and epilogue (PROLOGUE_EPILOGUE_TIMEOUT)

• when a connection to the DB fails, OAR retrie 5 times

• handle the EXTERMINATE_JOB event and suspect all nodes if possible

• add job state log table

• add node properties log table

• add possibility to use sentinelle.rb script in the ping_checker module

• add a GRID5000 specific scheduler which implements specific policy (oar_sched_gant_g5k)

• add -s option to oarnodes command --> show only state of nodes

• add -l option to oarnodes command --> show only the node list

• root can now delete any jobs in the same way the oar user could

• change a few oardel messages

• limit the commands (with arguments) specified by users:

  regular expression : [\w\s\/\.\-]*
  

• change oaremovenode command to oarremovenode

• enhance job error management

• enhance suspicious nodes detection

• fix bugs about accounting

9.3   version 1.6:

• debug reservation jobs (if the job has not the right number of requested nodes then it will wait for a delay. Once expired, the job will be launched on the nodes available).

• add a cache for visualization of the gantt scheme (add two gantt table for the visu)

• add an event log mechanism. It permits to know all decisions and events occuring in OAR with regards to jobs.

• detection of errors (they can be traced via event_log table):

  • job working directory does not exist on the node
  • output files cannot be created

• add deploy scheduler awareness (schedule on non deploy nodes firstly)

• possibility to change property value via oarnodesetting command (-p option)

• add the command "oaremovenode" that allows to delete a node of the database definitely

• bug fix : now oarsub can use the user's script even if the oar user cannot

• now, you can use special value "all" in oarsub ("nodes" resource). It gives all free nodes corresponding to the weight and properties specified.

• debug Gantt visualization

• add the possibility to test nodes via nmap

• add accounting features (accounting table)

• change nodeState_log table scheme to increase rapidity