NAME
qsub - Submit an NQS batch request.
SYNOPSIS
qsub [ flags ] [ script-file ]
SHORT DESCRIPTION OF PARAMETERS
The qsub flags are listed briefly below. An asterisk
indicates that the flag is not supported on the Paragon
system, but can be used when submitting NQS jobs from the
Paragon system to other systems that do support the flag. A
verbose description of all flags follows the Discussion.
-c establish per-request account
-e direct stderr output to stated destination
-eo direct stderr output to the stdout destination
-ke keep stderr output on the execution machine
-ko keep stdout output on the execution machine
-lc* establish per-process corefile size limit
-ld* establish per-process data-segment size limits
-lf * establish per-process permanent-file size limits
-lF* establish per-request permanent-file space limits
-lm* establish per-process memory size limits
-lM* establish per-request memory space limits
-ln establish per-process nice execution value limit
-lP establish per-request number of nodes
-ls establish per-process stack-segment size limits
-lt* establish per-process CPU time limits
-lT establish per-request CPU time limits
-lv* establish per-process temporary-file size limits
-lV* establish per-request temporary-file space limits
-lw* establish per-process working set limit
-mb send mail when the request begins execution
-me send mail when the request ends execution
-mu send mail for the request to the stated user
-nr declare that batch request is not restartable
-o direct stdout output to the stated destination
-q queue request in the stated queue
-r assign stated request name to the request
-re remotely access the stderr output file
-ro remotely access the stdout output file
-s specify shell to interpret the batch request script
-x export all environment variables with request
-z submit the request silently
The script-file is an optional batch file that contains
embedded default flags that set default characteristics for
the batch request.
DISCUSSION
The qsub command submits a batch request to the Network
Queueing System (NQS). The Paragon system can queue up to
500 requests at any given time.
If no script file is specified, then the set of commands to
be executed as a batch request is taken directly from the
standard input file (stdin). In all cases however, the
script file is spooled, so that later changes will not
affect previously queued batch requests.
All of the flags that can be specified on the command line
can also be specified within the first comment block inside
the batch request script file as embedded default flags.
Such flags appearing in the batch request script file set
default characteristics for the batch request. If the same
flag is specified on the command line, then the command line
flag (and any associated value) takes precedence over the
embedded flag.
The algorithm used to scan for such embedded default flags
within an NQS batch request script file is as follows:
1. Read the first line of the script file.
2. If the current line contains only whitespace characters,
or the first non-whitespace character of the line is :
(colon), then go to step 7.
3. If the first non-whitespace character of the current
line is not a # (pound) character, then go to step 8.
4. If the second non-whitespace character in the current
line is not the @ (at) character, or the character
immediately following the second non-whitespace
character in the current line is not a $ (dollar sign)
character, or if the second non-whitespace character is
not a Q character followed immediately by the string
``SUB'', then go to step 7.
5. If no - is present as the character immediately
following the ``@$'' sequence or the ``QSUB'' sequence,
then go to step 8.
6. Process the embedded flag, stopping the parsing process
upon reaching the end of the line, or upon reaching the
first unquoted # character.
7. Read the next script file line. Go to step 2.
8. End. No more embedded flags will be recognized.
Here is an example of the use of embedded flags within the
script file.
#
# Batch request script example:
#
# @$-a "11:30pm EDT" -lt "21:10, 20:00"
# # Run request after 11:30 EDT by default,
# # and set a maximum per-process CPU time
# # limit of 21 minutes and ten seconds.
# # Send a warning signal when any process
# # of the running batch request consumes
# # more than 20 minutes of CPU time.
# QSUB-lT 1:45:00
# # Set a maximum per-request CPU time limit
# # of one hour, and 45 minutes. (The
# # implementation of CPU time limits is
# # completely dependent upon the NQS
# # implementation at the execution
# # machine.)
# QSUB-mb -me
# # Send mail at beginning and end of
# # request execution.
# @$-q batch1 # Queue request to queue: batch1 by
# # default.
# @$ # No more embedded flags.
#
make all
QUEUE TYPES
NQS supports both batch queues and pipe queues. A batch
queue holds requests for scheduled or delayed processing. A
pipe queue is a queue that can pass queued requests on to
batch queues or other pipe queues.
Pipe queues are used to send NQS requests to other pipe
queues or to batch queues. In general, pipe queues act as
the mechanism that NQS uses to transport batch requests to
queues on other remote machines. Pipe queues can also
transport requests to queues on the same machine.
When a pipe queue is defined, it is given a destination set
which defines the set of possible destination queues for
requests entered in that pipe queue. In this manner, it is
possible for a batch request to pass through many pipe
queues on its way to its ultimate destination, which must
eventually be a batch queue.
Each pipe queue has an associated server. For each request
handled by a pipe queue, the associated server is spawned
which must select a queue destination for the request being
handled, based on the characteristics of the request, and
upon the characteristics of each queue in the destination
set defined for the pipe queue.
Since a different server can be configured for each pipe
queue, and batch queues can be endowed with the pipeonly
attribute that will only admit requests queued via another
pipe queue, it is possible for respective NQS installations
to use pipe queues as a request class mechanism, placing
requests that ask for different resource allocations in
different queues, each of which can have different
associated limits and priorities.
If a pipe queue server cannot handle a request, the request
is deleted and the user is notified by mail (see mail(1)).
QUEUE ACCESS
NQS supports queue access restrictions. For each queue,
access may be either unrestricted or restricted. If access
is unrestricted, any request may enter the queue. If access
is restricted, a request can only enter the queue if the
requester or the requester's login group has been given
access to that queue. Requests submitted by root are an
exception; they are always queued, even if root has not
explicitly been given access.
Use qstat to determine who has access to a particular queue.
LIMITS
NQS supports many batch request resource limit types that
can be applied to an NQS batch queue. The existence of
configurable resource limits allows an NQS user to set
resource limits within which his or her request must
execute. In many instances, smaller limit values can result
in a more favorable scheduling policy for a batch request.
For finite CPU time limits, the acceptable syntax is as
follows:
[[hours :] minutes : ] seconds[.milliseconds]
White space can appear anywhere between the principal
tokens, with the exception that no white space can appear
around the decimal point.
Example time limit-values are:
1234 : 58 : 21.29 - 1234 hrs 58 mins 21.290 secs
12345 - 12345 seconds
121.1 - 121.100 seconds
59:01 - 59 minutes and 1 second
For all other finite limits (with the exclusion of the
nice-value), the acceptable syntax is:
.fraction [units]
or
integer [.fraction] [units]
where the integer and fraction tokens represent strings of
up to eight decimal digits, denoting the obvious values. In
both cases, the units of allocation may also be specified as
one of the case insensitive strings:
b bytes
w words
kb kilobytes (2**()10 bytes)
kw kilowords (2**()10 words)
mb megabytes (2**()20 bytes)
mw megawords (2**()20 words)
gb gigabytes (2**()30 bytes)
gw gigawords (2**()30 words)
In the absence of any units specification, the units of
``bytes'' are assumed.
For all limit types with the exception of the nice-value, it
is possible to state that no limit should be applied. This
is done by specifying a limit of unlimited, or any initial
substring thereof. Whenever an infinite limit-value is
specified for a particular resource type, then the batch
request operates as though no explicit limits have been
placed upon the corresponding resource, other than by the
limitations of the physical hardware involved.
If a batch request specifies a limit that cannot be enforced
by the underlying operating system executing the job, then
the limit is simply ignored, and the batch request will
operate as though there were no limit (other than the
obvious physical maximums), placed upon that resource type.
For each remaining finite limit that can be supported by the
underlying operating system that is not a CPU time limit, or
nice value, the limit-value is internally converted to the
units of bytes or words, whichever is more appropriate for
the underlying machine architecture.
As an example, a per-process memory size limit value of 321
megabytes would be interpreted as 321 x 2**()20 bytes,
provided that the underlying machine architecture was
capable of directly addressing single bytes. Thus the
original limit coefficient of 321 would become 321 x
2**()20. On a machine that was only capable of addressing
words, the appropriate conversion of 321 x 2**()20 bytes /
#of-bytes-per-word would be performed.
If the result of such a conversion would cause overflow when
the coefficient was represented as a signed-long integer on
the supporting hardware, then the coefficient is replaced
with the coefficient of: of 2N-1 where N is equal to the
number of bits of precision in a signed long integer. For
typical
32-bit machines, this default extreme limit would therefore
be 231-1 bytes. For word addressable machines in the
supercomputer class supporting 64-bit long integers, the
default extreme limit would be 263-1 words.
Lastly, some operating systems reserve coefficients of the
form: 2**()N-1 as synonymous with infinity, meaning no limit
is to be applied. For such implementations, NQS further
decrements the default extreme limit so as to not imply
infinity.
The identical internal conversion process as described in
the preceding paragraphs is also performed for all finite
limit-values configured for a particular batch queue using
the qmgr program.
After all of the applicable limit-values have been converted
as described above, each such resulting limit-value is then
compared against the corresponding limit-value as configured
for the destination batch queue. If, for every type of
limit, the batch queue limit-value is greater than or equal
to the corresponding batch request limit-value, then the
request can be successfully queued, provided that no other
anomalous conditions occur. For request infinity
limit-values, the corresponding queue limit-value must also
be configured as infinity.
These resource limit checks are performed irrespective of
the batch request arrival mechanism, either by a direct use
of the qsub command, or by the indirect placement of a batch
request into a batch queue via a pipe queue. It is
impossible for a batch request to be queued in an NQS batch
queue if any of these resource limit checks fail.
Finally, if a request fails to specify a limit-value for a
resource limit type that is supported on the execution
machine, then the corresponding limit-value configured for
the destination queue becomes the limit-value for the
unspecified request limit.
Upon the successful queueing of a request in a batch queue,
the set of limits under which the request will execute is
frozen, and will not be modified by subsequent qmgr commands
that alter the limits of the containing batch queue.
BATCH REQUEST SEQUENCE
The following events take place in the following order when
an NQS batch request is spawned:
1. The process that will become the head of the process
group for all processes comprising the batch request is
created by NQS.
2. Resource limits are enforced.
3. The real and effective group-id of the process is set to
the group-id as defined in the local password file for
the request owner.
4. The real and effective user-id of the process is set to
the real user-id of the batch request owner.
5. The user file creation mask is set to the value that the
user had on the originating machine when the batch
request was first submitted.
6. If the user explicitly specified a shell by use of the
-s flag (discussed above), then that user-specified
shell is chosen as the shell that will be used to
execute the batch request script. Otherwise, a shell is
chosen based upon the shell strategy as configured for
the local NQS system (see the earlier discussion of the
-s flag for a description of the possible shell
strategies that can be configured for an NQS system).
7. The environment variables of HOME, SHELL, PATH, LOGNAME
(not all systems), USER (not all systems), and MAIL are
set from the user's password file entry, as though the
user had logged directly into the execution machine.
8. The environment variable ENVIRONMENT is set to BATCH.
This allows shell scripts and the user's .profile,
.cshrc, or .kshrc and .login files to test for batch
request execution when appropriate, and not set terminal
characteristics since a batch request is not connected
to an input terminal. For example, you could put
something similar to the following example in your
.cshrc file:
if ($?ENVIRONMENT == 0) then
# ENVIRONMENT is not set, must not be running under NQS
setenv ENVIRONMENT INTERACTIVE
endif
if ("$ENVIRONMENT" != "BATCH") then
# interactive shell
setenv HOME /tmp
else
# NQS shell
date
endif
9. The environment variables of QSUB_WORKDIR, QSUB_HOST,
QSUB_REQNAME, and QSUB_REQID are added to the
environment. These environment variables equate to the
obvious respective strings of the working directory at
the time that the request was submitted, the name of the
originating host, the name of the request, and the
request request-id.
10. All of the remaining environment variables saved for
recreation when the batch request is spawned are added
at this point to the environment. When a batch request
is initially submitted, the current values of the
environment variables HOME, SHELL, PATH, LOGNAME (not
all systems), USER (not all systems), MAIL, and TZ are
saved for later recreation when the batch request is
spawned. When recreated however, these variables are
added to the environment under the respective names
QSUB_HOME, QSUB_SHELL, QSUB_PATH, QSUB_LOGNAME,
QSUB_USER, QSUB_MAIL, and QSUB_TZ, to avoid the obvious
conflict with the local version of these environment
variables. Additionally, all environment variables
exported from the originating host by the -x option are
added to the environment at this time.
11. The current working directory is then set to the user's
home directory on the execution machine, and the chosen
shell is exec'd to execute the batch request script with
the environment as constructed in the steps outlined
above.
If the user did not specify a specific shell for the batch
request, then NQS chooses which shell should be used to
execute the shell script, based on the shell strategy as
configured by the system administrator (see the earlier
discussion of the -s flag). If the shell strategy is free,
NQS executes the login shell for the user (as configured in
the password file). The login shell is in turn instructed to
examine the shell script file, and fork another shell of the
appropriate type to interpret the shell script, behaving
exactly as an interactive invocation of the script.
Otherwise no additional shell is spawned, and the chosen
fixed or login shell sequentially executes the commands
contained in the shell script file until completion of the
batch request.
If the use_login configuration parameter is set to 1, the
chosen shell is exec'd as though it were the login shell.
(If the Bourne shell is chosen to execute the script, then
the .profile file is read. If the C shell is chosen, then
the .cshrc and .login files are read. If the Korn shell is
chosen, then both the .profile and .kshrc files are read.)
LONG DESCRIPTION OF PARAMETERS
-a date-time
Currently unused. Do not run the batch request
before the specified date and/or time. If a
date-time specification contains any whitespace
characters, then the date-time specification must
be placed within double quotes as in: -a "July, 4,
2026 12:31-EDT", or otherwise escaped such that
qsub and the shell will interpret the entire
date-time specification as a single character
string. This restriction also applies when an
embedded default -a flag with accompanying
date-time specification appears within the batch
request script file.
The syntax accepted for the date-time parameter is
relatively flexible. Unspecified date and time
values default to an appropriate value (e.g. if no
date is specified, then the current month, day,
and year are assumed).
A date may be specified as a month and day
(current year assumed), or the year can also be
explicitly specified. It is also possible to
specify the date as a weekday name (e.g. Tues), or
as one of the strings: today, or tomorrow. Weekday
names and month names can be abbreviated by any
three character (or longer) prefix of the actual
name. An optional period can follow an abbreviated
month or day name.
Time of day specifications can be given using a
twenty-four hour clock, or am and pm
specifications may be used. In the absence of a
meridian specification, a twenty-four hour clock
is assumed.
It should be noted that the time of day
specification is interpreted using the precise
meridian definitions whereby ``12am'' refers to
the twenty-four hour clock time of 0:00:00,
``12m'' refers to noon, and ``12-pm'' refers to
24:00:00. Alternatively, the phrases midnight and
noon are accepted as time of day specifications,
where midnight refers to the time of 24:00:00.
A time zone may also appear at any point in the
date-time specification. Thus, it is legal to say:
April 1, 1987 13:01-PDT. In the absence of a
timezone specification, the local timezone is
assumed, with daylight savings time being inferred
when appropriate, based on the date specified.
All alphabetic comparisons are performed in a case
insensitive fashion, such that both WeD and weD
refer to Wednesday.
Some valid date-time examples are:
01-Jan-1986 12am, PDT
Tuesday, 23:00:00
11pm tues.
tomorrow 23-MST
The -a switch is also used to make a reservation
for a dedicated job if the queue to which the
batch request is to be submitted is designated
"dedicated." A queue is designated "dedicated" if
the queue priority is greater than or equal to the
tsched_pri value defined in the
/usr/spool/nqs/conf/sched_param file. In addition,
the starting time specified by the -a switch and
the time limit (specified with the -1T switch)
must be specified in 30 minute increments for
dedicated job reservation because dedicated times
are allocated at 30 minute intervals.
-c account
An integer accounting ID or a character string
representing the MACS accounting group name may be
specified for account. If this option is not
specified, the account ID in the ACCOUNT
environment variable is used. If the ACCOUNT
environment variable is not set, the default
specified in /etc/nx/nx_dflt_accts file is used.
Accounting IDs and group names are defined in the
/etc/nxaccount file.
This flag is unique to the Paragon system, and as
such, is not supported by other NQS
implementations. For this reason, if a batch job
is submitted from a remote system (including
another Paragon system), this flag has no effect.
In these cases, the account ID should be specified
through the ACCOUNT environment variable and then
exported with the -x option.
-e [machine:][[/]path/]stderr-filename
Direct output generated by the batch request which
is sent to the stderr file to the named
[machine:][[/]path/]stderr-filename.
If no explicit machine destination is specified,
then the destination machine defaults to the
machine that originated the batch request, or to
the machine that will eventually run the request,
depending on the respective absence, or presence
of the -ke flag.
If no machine destination is specified, and the
path/filename does not begin with a /, then the
current working directory is prepended to create a
fully qualified path name, provided that the -ke
(keep stderr) flag is also absent. In all other
cases, any partial path/filename is interpreted
relative to the user's home directory on the
stderr destination machine.
This flag cannot be specified when the -eo flag
option is also present.
If the -eo and -e
[machine:][[/]path/]stderr-filename flag options
are not present, then all stderr output for the
batch request is sent to the file whose name
consists of the first seven characters of the
request-name followed by the characters .e,
followed by the request sequence number portion of
the request-id discussed below. In the absence of
the -ke flag, this default stderr output file is
placed on the machine that originated the batch
request in the current working directory, as
defined when the batch request was first
submitted. Otherwise, the file will be placed in
the user's home directory on the execution
machine.
-eo Direct all output that would normally be sent to
the stderr file to the stdout file for the batch
request. This flag cannot be specified when the -e
flag is present.
-ke In the absence of an explicit machine destination
for the stderr file produced by a batch request,
the machine destination chosen for the stderr
output file is the machine that originated the
batch request. In some cases however, this
behavior may be undesirable, and so the -ke flag
can be specified which instructs NQS to leave any
stderr output file produced by the request on the
machine that actually executed the batch request.
This flag is meaningless if the -eo flag is
specified, and cannot be specified if an explicit
machine destination is given for the stderr
parameter of the -e flag.
-ko In the absence of an explicit machine destination
for the stdout file produced by a batch request,
the machine destination chosen for the stdout
output file is the machine that originated the
batch request. In some cases however, this
behavior may be undesirable, and so the -ko flag
can be specified which instructs NQS to leave any
stdout output file produced by the request on the
machine that actually executed the batch request.
This flag cannot be specified if an explicit
machine destination is given for the stdout
parameter of the -o flag.
-lc per-process corefile size limit
Set a per-process maximum corefile size limit for
all processes that constitute the running batch
request. If any process comprising the running
request attempts to exit creating a core file
whose size would exceed the maximum per-process
corefile size limit for the request, then the core
file image of the aborting process will be reduced
to the necessary size by an algorithm dependent
upon the underlying operating system.
See the ``Limits'' discussion on page 6-68 for
more information on the implementation of batch
request limits, and for a description of the
precise syntax of a per-process corefile size
limit.
(This flag is not supported on the Paragon system,
but can be used when submitting NQS jobs from the
Paragon system to other systems that do support
the flag.)
-ld per-process data-segment size limit [, warn-limit]
Set a per-process maximum and an optional warning
data-segment size limit for all processes that
constitute the running batch request. If any
process comprising the running request exceeds the
maximum per-process data-segment size-limit for
the request, then that process may be terminated
by a signal chosen by the underlying operating
system. For OSF/1, memory allocation calls will
fail and no signal will be sent.
The ability to specify an optional warning limit
exists for those operating systems that support
per-process data-segment warning size limits. When
a warning limit is exceeded, a signal as
determined by the underlying operating system is
delivered to the offending process (OSF/1 does not
support a warning limit).
If a maximum limit (and optional warning limit)
specification is comprised of two or more tokens
separated by whitespace characters, then the
specification must be enclosed within double
quotes, or otherwise escaped such that qsub and
the shell will interpret the entire specification
as a single character string token. This caveat
also applies when an embedded default -ld flag
with its associated limit value(s) appears within
the batch request script file.
Not all NQS implementations support per-process
data-segment size limits. If a batch request
specifies this limit, and the machine upon which
the batch request is eventually run does not
support the enforcement of this limit, then the
limit is simply ignored.
See the ``Limits'' discussion on page 6-68 for
more information on the implementation of batch
request limits, and for a description of the
precise syntax of a per-process data-segment size
limit.
(This flag is not supported on the Paragon system,
but can be used when submitting NQS jobs from the
Paragon system to other systems that do support
the flag.)
-lf per-process permanent-file size limit [, warn-limit]
Set a per-process maximum and an optional warning
permanent-file size limit for all processes that
constitute the running batch request. If any
process comprising the running request attempts to
write to a permanent file such that the file size
would increase beyond the maximum per-process
permanent-file size limit for the request, then
that process is terminated by a signal chosen by
the underlying operating system. For OSF/1, this
is SIGXFSZ.
The ability to specify an optional warning limit
exists for those operating systems that support
per-process warning permanent-file size limits.
When a warning limit is exceeded, a signal, as
determined by the underlying operating system, is
delivered to the offending process. For OSF/1,
this is SIGXFSZ.
If a maximum limit (and optional warning limit)
specification is comprised of two or more tokens
separated by whitespace characters, then the
specification must be enclosed within double
quotes, or otherwise escaped such that qsub and
the shell will interpret the entire specification
as a single character string token. This caveat
also applies when an embedded default -lf flag
with its associated limit value(s) appears within
the batch request script file.
Not all NQS implementations support per-process
permanent-file size limits. If a batch request
specifies this limit, and the machine upon which
the batch request is eventually run does not
support the enforcement of this limit, then the
limit is simply ignored.
For all NQS implementations that do not support a
distinction between permanent, and temporary files
at the kernel level (such as OSF/1), this limit is
interpreted as a per-process file size limit, with
the word permanent removed from the definition.
See the ``Limits'' discussion on page 6-68 for
more information on the implementation of batch
request limits, and for a description of the
precise syntax of a per-process permanent-file
size limit.
(This flag is not supported on the Paragon system,
but can be used when submitting NQS jobs from the
Paragon system to other systems that do support
the flag.)
-lF per-request permanent-file space limit [, warn-limit]
Set a per-request maximum and an optional warning
cumulative permanent-file space limit for all
processes that constitute the running batch
request. If any process comprising the running
request attempts to write to a permanent file such
that the file space consumed by all permanent
files opened for writing by all of the processes
in the batch request would increase beyond the
maximum per-request permanent-file space limit for
the request, then all of the processes in the
request will be terminated by a signal chosen by
the underlying operating system implementation.
The ability to specify an optional warning limit
exists for those operating systems that support
per-request warning permanent-file space limits.
When such a warning limit is exceeded, a signal is
delivered to one or more of the processes
comprising the running request, depending upon the
underlying operating system implementation.
If a maximum limit (and optional warning limit)
specification is comprised of two or more tokens
separated by whitespace characters, then the
specification must be enclosed within double
quotes, or otherwise escaped such that qsub and
the shell will interpret the entire specification
as a single character string token. This caveat
also applies when an embedded default -lF flag
with its associated limit value(s) appears within
the batch request script file.
Not all NQS implementations support per-request
permanent-file space limits. If a batch request
specifies this limit, and the machine upon which
the batch request is eventually run does not
support the enforcement of this limit, then the
limit is simply ignored.
For all operating systems that do not support a
distinction between permanent and temporary files
at the kernel level, this limit is interpreted as
a per-request file space limit, with the word
permanent removed from the definition.
(This flag is not supported on the Paragon system,
but can be used when submitting NQS jobs from the
Paragon system to other systems that do support
the flag.)
See the ``Limits'' discussion on page 6-68 for
more information on the implementation of batch
request limits, and for a description of the
precise syntax of per-request permanent-file space
limit.
-lm per-process memory size limit [, warn-limit]
Set a per-process maximum and an optional warning
memory size limit for all processes that
constitute the running batch request. If any
process comprising the running request exceeds the
maximum per-process memory size limit for the
request, then that process is terminated by a
signal chosen by the underlying operating system.
The ability to specify an optional warning limit
exists for those operating systems that support
per-process warning memory size limits. When a
warning limit is exceeded, a signal, as determined
by the operating system, is delivered to the
offending process.
If a maximum limit (and optional warning limit)
specification is comprised of two or more tokens
separated by whitespace characters, then the
specification must be enclosed within double
quotes, or otherwise escaped such that qsub and
the shell will interpret the entire specification
as a single character string token. This caveat
also applies when an embedded default -lm flag
with its associated limit value(s) appears within
the batch request script file.
Not all NQS implementations support per-process
memory size limits. If a batch request specifies
this limit, and the machine upon which the batch
request is eventually run does not support the
enforcement of this limit, then the limit is
simply ignored.
See the ``Limits'' discussion on page 6-68 for
more information on the implementation of batch
request limits, and for a description of the
precise syntax of per-process memory size limit.
(This flag is not supported on the Paragon system,
but can be used when submitting NQS jobs from the
Paragon system to other systems that do support
the flag.)
-lM per-request memory space limit [, warn-limit]
Set a per-request maximum and an optional warning
cumulative memory space limit for all processes
that constitute the running batch request. If the
sum of all memory consumed by all of the processes
comprising the running request exceeds the maximum
per-request memory space limit for the request,
then all of the processes in the request will be
terminated by a signal chosen by the underlying
operating system.
The ability to specify an optional warning limit
exists for those operating systems that support
per-request warning memory size limits. When such
a warning limit is exceeded, a signal is delivered
to one or more of the processes comprising the
running request, depending upon the underlying
operating system.
If a maximum limit (and optional warning limit)
specification is comprised of two or more tokens
separated by whitespace characters, then the
specification must be enclosed within double
quotes, or otherwise escaped such that qsub and
the shell will interpret the entire specification
as a single character string token. This caveat
also applies when an embedded default -lM flag
with its associated limit value(s) appears within
the batch request script file.
Not all operating systems support per-request
memory space limits. If a batch request specifies
this limit, and the machine upon which the batch
request is eventually run does not support the
enforcement of this limit, then the limit is
simply ignored.
See the ``Limits'' discussion on page 6-68 for
more information on the implementation of batch
request limits, and for a description of the
precise syntax of per-request memory space limit.
(This flag is not supported on the Paragon system,
but can be used when submitting NQS jobs from the
Paragon system to other systems that do support
the flag.)
-ln per-request nice value limit
Set a per-process nice value for all processes
comprising the running batch request.
Most NQS implementations support the use of an
integer called the nice value, which determines
the execution-time priority of a process relative
to all other processes in the system. By letting
the user set a limit on the nice value for all
processes comprising the running request, a user
can cause a request to consume less (or more) of
the CPU resource presented by the execution
machine.
This is particularly useful when a user wishes to
execute a CPU intensive batch request on a machine
running interactive processes. By setting a low
execution-time priority, a user can make a long
running batch request give way to more interactive
processes during the daytime, while the coming of
the nighttime hours with typically smaller process
loads will allow such a request to gain more and
more of the CPU resource. In this way, long
running batch requests can be polite to their more
transient, interactive neighbor processes.
In general, increasingly negative nice values
cause the relative execution priority of a process
to increase, while increasingly positive nice
values causes the relative priority to decrease.
Thus, a nice value limit specification of -ln -10
is greedier than a nice value limit specification
of -ln 0.
Since varying operating systems often support a
different finite range of nice values, NQS allows
the specification of nice values that can
eventually turn out to be outside the limits for
the NQS implementation running at the execution
machine. In such cases, NQS will simply bind the
specified nice value limit to within the necessary
range as appropriate.
Any nice value specified by the use of this flag
must be acceptable to the batch queue in which the
request is ultimately placed (see the ``Limits''
discussion on page 6-68 for more information).
-lP node-number
Set a per-request limit on the number of nodes
used. This option is used to set a node limit for
each parallel application within an NQS job that
is less than or equal to the limit associated with
a batch queue. In the absence of this flag,
parallel applications of an NQS job will run in
the number of nodes specified as an attribute of
the NQS batch queue.
This flag is unique to the Paragon system, and as
such, is not supported by other NQS
implementations. For this reason, if a batch job
is submitted from a remote system (including
another Paragon system), this flag has no effect.
In these cases, a node limit can be specified
through the NCPUS environment variable and then
exported with the -x option.
-ls per-process stack-segment size limit [, warn-limit]
Set a per-process maximum and an optional warning
stack-segment size limit for all processes that
constitute the running batch request. If any
process comprising the running request exceeds the
maximum per-process stack-segment size limit for
the request, then that process is terminated by a
signal chosen by the underlying operating system.
The ability to specify an optional warning limit
exists for those operating systems that support
per-process warning stack-segment size limits.
When a warning limit is exceeded, a signal as
determined by the underlying operating system is
delivered to the offending process.
If a maximum limit (and optional warning limit)
specification is comprised of two or more tokens
separated by whitespace characters, then the
specification must be enclosed within double
quotes, or otherwise escaped such that qsub and
the shell will interpret the entire specification
as a single character string token. This caveat
also applies when an embedded default -ls flag
with its associated limit value(s) appears within
the batch request script file.
Not all operating systems support per-process
stack-segment size limits. If a batch request
specifies this limit, and the machine upon which
the batch request is eventually run does not
support the enforcement of this limit, then the
limit is simply ignored.
See the ``Limits'' discussion on page 6-68 for
more information on the implementation of batch
request limits, and for a description of the
precise syntax of per-process stack-segment size
limit.
(This flag is not supported on the Paragon system,
but can be used when submitting NQS jobs from the
Paragon system to other systems that do support
the flag.)
-lt per-process CPU time limit [, warn-limit]
Set a per-process CPU time limit and an optional
warning CPU time limit for all processes that
constitute the running batch request. If any
process comprising the running request exceeds the
maximum per-process node hour limit for the
request, then that process is terminated by a
signal chosen by the underlying operating system.
For OSF/1, this is SIGXCPU.
The ability to specify an optional warning limit
exists for those operating systems that support
per-process CPU warning time limits. When a
warning limit is exceeded, a signal, as determined
by the underlying operating system is delivered to
the offending process. For OSF/1, this is SIGXCPU.
If a maximum limit (and optional warning limit)
specification is comprised of two or more tokens
separated by whitespace characters, then the
specification must be enclosed within double
quotes, or otherwise escaped such that qsub and
the shell will interpret the entire specification
as a single character string token. This caveat
also applies when an embedded default -lt flag
with its associated limit value(s) appears within
the batch request script file.
Not all NQS implementations support per-process
CPU time limits. If a batch request specifies this
limit, and the machine upon which the batch
request is eventually run does not support the
enforcement of this limit, then the limit is
simply ignored.
See the ``Limits'' discussion on page 6-68 for
more information on the implementation of batch
request limits, and for a description of the
precise syntax of per-process CPU time limit.
(This flag is not supported on the Paragon system,
but can be used when submitting NQS jobs from the
Paragon system to other systems that do support
the flag.)
-lT per-request CPU time limit [, warn-limit]
Sets a per-request maximum node hour limit and an
optional warning limit for a batch request. If the
node hour limit is exceed, all of the processes in
the request are sent a SIGTERM signal, and then a
SIGKILL signal after the number of seconds
specified by the grace_time configuration
parameter in the sched_param file (10 seconds by
default). If the optional warning limit is
specified, the user is notified before the
processes are terminated.
If the node hour limit (and optional warning
limit) is comprised of two or more tokens
separated by whitespace characters, they must be
enclosed within double quotes, or otherwise
escaped such that qsub and the shell will
interpret the entire specification as a single
character string token. This caveat also applies
when an embedded default -lT flag with its
associated limit value(s) appears within the batch
request script file.
While the Paragon system supports this limit, not
all NQS implementations support per-request CPU
time limits. If a batch request specifies this
limit, and the machine upon which the batch
request is eventually run does not support the
enforcement of this limit, then the limit is
simply ignored.
See the ``Limits'' discussion on page 6-68 for
more information on the implementation of batch
request limits, and for a description of the
precise syntax of per-request CPU time limit.
-lv per-process temporary file size limit [, warn-limit]
Set a per-process maximum and an optional warning
temporary (volatile) file size limit for all
processes that constitute the running batch
request. If any process comprising the running
request attempts to write to a temporary file such
that the file size would increase beyond the
maximum per-process temporary-file size limit for
the request, then that process is terminated by a
signal chosen by the underlying operating system.
The ability to specify an optional warning limit
exists for those operating systems that support
per-process warning temporary-file size limits.
When a warning limit is exceeded, a signal as
determined by the underlying operating system
implementation is delivered to the offending
process.
If a maximum limit (and optional warning limit)
specification is comprised of two or more tokens
separated by whitespace characters, then the
specification must be enclosed within double
quotes, or otherwise escaped such that qsub and
the shell will interpret the entire specification
as a single character string token. This caveat
also applies when an embedded default -lv flag
with its associated limit value(s) appears within
the batch request script file.
See the ``Limits'' discussion on page 6-68 for
more information on the implementation of batch
request limits, and for a description of the
precise syntax of per-process temporary-file size
limit.
(This flag is not supported on the Paragon system,
but can be used when submitting NQS jobs from the
Paragon system to other systems that do support
the flag.)
-lV per-request temporary file space limit [, warn-limit]
Set a per-request maximum and an optional warning
cumulative temporary (volatile) file space limit
for all processes that constitute the running
batch request. If any process comprising the
running request attempts to write to a temporary
file such that the file space consumed by all
temporary files opened for writing by all of the
processes in the batch request would increase
beyond the maximum per-request temporary-file
space limit for the request, then all of the
processes in the request will be terminated by a
signal chosen by the underlying operating system
implementation.
The ability to specify an optional warning limit
exists for those operating systems that support
per-request warning temporary-file space limits.
When such a warning limit is exceeded, a signal is
delivered to one or more of the processes
comprising the running request, depending upon the
underlying operating system implementation.
If a maximum limit (and optional warning limit)
specification is comprised of two or more tokens
separated by whitespace characters, then the
specification must be enclosed within double
quotes, or otherwise escaped such that qsub and
the shell will interpret the entire specification
as a single character string token. This caveat
also applies when an embedded default -lV flag
with its associated limit value(s) appears within
the batch request script file.
See the ``Limits'' discussion on page 6-68 for
more information on the implementation of batch
request limits, and for a description of the
precise syntax of temporary-file space limit.
(This flag is not supported on the Paragon system,
but can be used when submitting NQS jobs from the
Paragon system to other systems that do support
the flag.)
-lw per-process working set size limit
Set a per-process maximum working set size limit
for all processes that constitute the running
batch request.
Not all operating systems support per-process
working set size limits, and such a limit only
makes sense in the context of a paged virtual
memory machine. If a batch request specifies this
limit, and the machine upon which the batch
request is eventually run does not support the
enforcement of this limit, then the limit is
simply ignored.
See the ``Limits'' discussion on page 6-68 for
more information on the implementation of batch
request limits, and for a description of the
precise syntax of per-process working set size
limit.
(This flag is not supported on the Paragon system,
but can be used when submitting NQS jobs from the
Paragon system to other systems that do support
the flag.)
-mb Send mail to the user on the originating machine
when the request begins execution. If the -mu flag
is also present, then mail is sent to the user
specified for the -mu flag instead of to the
invoking user.
-me Send mail to the user on the originating machine
when the request has ended execution. If the -mu
flag is also present, then mail is sent to the
user specified for the -mu flag instead of to the
invoking user.
-mu user-name
Specify that any mail concerning the request
should be delivered to the user user-name.
User-name may be formatted either as user
(containing no @ characters), or as user@machine.
In the absence of this flag, any mail concerning
the request will be sent to the invoker on the
originating machine.
-nr Declare that the request is non-restartable. If
this flag is specified, then the request will not
be restarted by NQS upon system boot if the
request was running at the time of an NQS shutdown
or system crash.
By default, NQS assumes that all requests are
restartable, with the caveat that it is the
responsibility of the user to ensure that the
request will execute correctly if restarted, by
the use of appropriate programming techniques.
Requests that are not running are always preserved
across host crashes and NQS shutdowns for later
requeueing, with or without this flag.
When NQS is shut down via an operator command to
the qmgr NQS control program, a SIGTERM signal is
sent to all processes comprising all running NQS
requests on the local host, and all queued NQS
requests are barred from beginning execution.
After a finite number of seconds have elapsed
(typically 60, but this value can be overridden by
the operator), all remaining processes comprising
all remaining running NQS requests are killed by
the SIGKILL signal.
For an NQS request to be properly restarted after
an NQS shutdown, the -nr flag must not be
specified, and the spawned batch request shell
must ignore SIGTERM signals. The spawned batch
request shell must also not exit before the final
SIGKILL arrives. Since the batch request shell is
simply spawning commands and programs, waiting for
their completion, this implies that the commands
and programs being executed by the batch request
shell must also be immune to SIGTERM signals,
saving state as appropriate before being killed by
the final SIGKILL signal.
See the ``Caveats'' discussion on page 6-92 for
more discussion concerning the restartability of
NQS batch requests.
-o [machine:][[/]path/]stdout-filename
Direct output generated by the batch request which
is sent to the stdout file of the named
[machine:][[/]path/]stdout-filename.
If no explicit machine destination is specified,
then the destination machine defaults to the
machine that originated the batch request, or to
the machine that will eventually run the request,
depending on the respective absence, or presence
of the -ko flag.
If no machine destination is specified, and the
path/filename does not begin with a /, then the
current working directory is prepended to create a
fully qualified path name, provided that the -ko
(keep stdout) flag is also absent. In all other
cases, any partial path/filename is interpreted
relative to the user's home directory on the
stdout destination machine.
If no -o [machine:][[/]path/]stdout-filename
flag is specified, then all stdout output for the
batch request is sent to the file whose name
consists of the first seven characters of the
request-name followed by the characters .o,
followed by the request sequence number portion of
the request-id discussed below. In the absence of
the -ko flag, this default stdout output file will
be placed on the machine that originated the batch
request in the current working directory, as
defined when the batch request was first
submitted. Otherwise, the file will be placed in
the user's home directory on the execution
machine.
-q queue-name
Specify the queue to which the batch request is to
be submitted. If no -q queue-name specification
is given, then the queue specified by the
environment variable QSUB_QUEUE is used. If the
QSUB_QUEUE environment variable is not found, then
the request will be submitted to the default batch
request queue, if defined by the local system
administrator. Otherwise, the request cannot be
queued, and an error message is displayed to this
effect.
-r request-name
Assign the specified request-name to the request.
In the absence of an explicit -r request-name
specification, the request-name defaults to the
name of the script file (leading path name
removed) given on the command line. If no script
file was given, then the default request-name
assigned to the request is stdin.
In all cases, if the request name is found to
begin with a digit, then the character R is
prepended to prevent a request-name from beginning
with a digit. All request names are truncated to a
maximum length of 15 characters.
-re By default, all output generated by a batch
request sent to the stderr file is temporarily
written to a file residing in a protected
directory on the machine that executes the
request. When the batch request completes
execution, this file is then spooled to its final
destination, possibly on a remote machine.
This default spooling of the stderr output file is
done to reduce the network traffic costs incurred
by the submitter (owner) of a batch request which
is to return its stderr output to a remote machine
upon completion. In some cases, this behavior is
not desired. If it is necessary to override this
behavior, then the -re flag can be specified which
says that stderr output produced by the request is
to be immediately written to the final destination
file, as output is generated, no matter what the
networking cost.
-ro By default, all output generated by a batch
request sent to the stdout file is temporarily
spooled into a file residing in a protected
directory on the machine that executes the
request. When the batch request completes
execution, this file is then spooled to its final
destination, possibly on a remote machine.
This default spooling of the stdout output file is
done to reduce the network traffic costs incurred
by the submitter (owner) of a batch request which
is to return its stdout output to a remote machine
upon completion. In some cases, this behavior is
not desired. If it is necessary to override this
behavior, then the -ro flag can be specified which
says that stdout output produced by the request is
to be immediately written to the final destination
file, as output is generated, no matter what the
networking cost.
-s shell-name
Specify the absolute path name of the shell which
will be used to interpret the batch request
script. This flag unconditionally overrides any
shell strategy configured on the execution machine
for selecting which shell to spawn in order to
interpret the batch request script.
In the absence of this flag, the NQS system at the
execution machine will use one of three distinct
shell choice strategies for the execution of the
batch request. Any one of the three strategies can
be configured by a system administrator for each
NQS machine.
The three shell strategies are called fixed, free,
and login. These shell strategies respectively
cause the configured fixed shell to be exec'd to
interpret all batch requests, cause the user's
login shell as defined in the password file to be
exec'd which in turn chooses and spawns the
appropriate shell for interpreting the batch
request script, or cause only the user's login
shell to be exec'd to interpret the script.
o A shell strategy of fixed means that the
same shell (as configured by the System
Administrator), will be used to execute all
batch requests.
o A shell strategy of free will run the
batch request script exactly as would an
interactive invocation of the script, and is
the default NQS shell strategy.
o A shell strategy of login means that the
user's login shell is used to execute the
batch request.
The strategies of fixed and login exist for host
systems that are short on available free
processes. In these two strategies, a single shell
is exec'd, and that same shell is the shell that
executes all of the commands in the batch request
script.
The shell strategy configured for a particular NQS
system can be determined by the qlimit command.
-x Export all environment variables. When a batch
request is submitted, the current values of the
environment variables HOME, SHELL, PATH, LOGNAME
(not all systems), USER (not all systems), MAIL,
and TZ are saved for later recreation when the
batch request is spawned, as the respective
environment variables QSUB_HOME, QSUB_SHELL,
QSUB_PATH, QSUB_LOGNAME, QSUB_USER, QSUB_MAIL, and
QSUB_TZ. Unless the -x flag is specified, no other
environment variables will be exported from the
originating host for the batch request. If the -x
flag option is specified, then all remaining
environment variables whose names do not conflict
with the automatically exported variables, are
also exported with the request. These additional
environment variables will be recreated under the
same name when the batch request is spawned.
-z Submit the batch request silently. If the request
is submitted successfully, then no messages are
displayed indicating this fact. Error messages
will, however, always be displayed.
If the batch request is successfully submitted and
the -z flag has not been specified, the request-id
of the request is displayed to the user. A
request-id is always of the form: seqno.hostname
where seqno refers to the sequence number assigned
to the request by NQS, and hostname refers to the
name of originating local machine. This identifier
is used throughout NQS to uniquely identify the
request, no matter where it is in the network.
CAVEATS
When an NQS batch request is spawned, a new process-group is
established such that all processes of the request exist in
the same process-group. If the qdel command is used to send
a signal to an NQS batch request, the signal is sent to all
processes of the request in the created process-group.
However, should one or more processes of the request choose
to successfully execute a setpgrp (2) system call, then such
processes will not receive any signals sent by the qdel
command. This can lead to rogue requests whose constituent
processes must be killed by other means such as the kill(1)
command. However, NQS takes advantage of any operating
systems that provide a mechanism of locking a process, and
all of its subsequent children in a particular
process-group. For such operating systems, this problem does
not occur.
It is extremely wise for all processes of an NQS request to
catch any SIGTERM signals. By default, the receipt of a
SIGTERM signal causes the receiving process to die. NQS
sends a SIGTERM signal to all processes in the established
process-group for a batch request as a notification that the
request should be prepared to be killed, either because of
an abort queue subcommand issued by an operator using the
qmgr program, or because it is necessary to shut down NQS
and all running requests as part of a general shutdown
procedure of the local host.
It must be understood that the spawned shell ignores SIGTERM
signals. If the current immediate child of the shell does
not ignore or catch SIGTERM signals, then it will be killed
by the receipt of such, and the shell will go on to execute
the next command from the script (if there is one). In any
case, the shell will not be killed by the SIGTERM signal,
though the executing command will have been killed.
After receiving a SIGTERM signal delivered from NQS, a
process of a batch request typically has sixty seconds to
get its house in order before receiving a SIGKILL signal
(though the sixty second duration can be changed by the
operator).
All batch requests terminated because of an operator NQS
shutdown request that did not specify the -nr flag are
considered restartable by NQS, and are requeued (provided
that the batch request shell process is still present at the
time of the SIGKILL signal broadcast as discussed above), so
that when NQS is rebooted, such batch requests will be
respawned to continue execution. It is however, up to the
user to make the request restartable by the appropriate
programming techniques. NQS simply spawns the request again
as though it were being spawned for the first time.
Upon completion of a batch request, a mail message can be
sent to the submitter (see the discussion of the -me flag
above). In many instances, the completion code of the
spawned Bourne shell, Korn shell, or C-shell is displayed.
This is merely the value returned by the shell through the
exit (2) system call.
If the qsub command returns the error:
Explicit request quota limits exceed maximums at local host.
The request could not be queued or delivered because the
following explicit request quota limit(s) exceeded the
corresponding limits of the target queue:
Per-request CPU quota limit;
This error message indicates that the request has either
exceeded the queue's CPU time limit (set by the qmgr
subcommand set per_request cpu_limit) or the queue's
number-of-CPUs limit (set by the qmgr subcommand set
per_request ncpus). These two errors result in the same
error message because they are represented internally by a
single error code; this is necessary in order to maintain
compatibility with other versions of NQS.
If the qsub command returns the error:
Permission denied
for a queue.
Check the setting of the qmgr set queue_request_limit
subcommand.
SEE ALSO
mail, qdel, qdev, qlimit, qmgr, qpr, qstat, kill, setpgrp(),
signal()
Acknowledgement and Disclaimer