qtop - Yet another tool to monitor the unmonitorable

The fast text mode way to present your cluster's utilization and status (currently PBS and SGE families are supported)

Five thousand words

Color code: ATLAS, CMS, LHCb, OPS, EGEE-VOs, OTHERS

Introduction

qtop (pronounced queue-top) is a tool written in order to summarize the state of an LRMS (Local Resource Management System),
along with some related information relevant on grid (and not grid) clusters. At present it supports the PBS family and SGE family is on the works.
The tool is not intended to be a Swiss-army knife, it rather adopts the unix philosophy: do one thing and do it well.

So, qtop tries to optimize on as little use of screen real estate as possible. It will report information in three sections:

Accounting summary

• Number of Nodes (available/total)
• Number of Cores (available/total)
• Number of Jobs (running+queued)
• Names of Queues, with running/queued jobs in each one; symbol * appears, when queue is blocked (ie. qstop'd OR qdisable'd)

Nodes occupancy

• Node ID (virtual or real nodename); Please read it vertically. This may be a REMAPed virtual node number.
• Node state, first letter of (job-exclusive, busy, offline, down, ...). Symbol '-' corresponds to the "free" node state.
• Job allocation table in a CPU-Node matrix layout; This is how your LRMS does the resource planning.

User account information

• "id" which is a symbol representing a single unix account
• User's jobs in Running state vs Total (this includes Completed state)
• unix account name
• Grid certificate DN: this is useful on grid clusters; it shows who owns the so-called pool account at the present moment.
  NOTE: This feature can only run under root privileges, due to dependency on command edg-mkgridpool --list . There can be workarounds, contact the developer for details on how to do this in your case. If you don't have an LHC/grid cluster, please prefer to run qtop as a non-root user, as normal.

Why qtop is useful

Here are some scenarios in which this tool has served faithfully:

• Overview of the LRMS state: nodes' status, queues' status, dynamic behavior of the system (eg. using watch -d)
• Test other schedulers or policies; eg. on maui/moab: CPULOAD FASTEST FIRSTAVAILABLE MAXBALANCE MINRESOURCE PRIORITY
• Detect black-holes and identify status of nodes with interesting behavior (eg. too much swapping with certain user's jobs)
• Security incidents, whereby a user account/DN is supplied and we must quickly find out where the relevant jobs run.
• To combine with ansi2html script and provide a front-end webpage with cluster occupancy details (an example is included in the package)

Download

Simplest, fastest option: rpm -Uvh http://fotis.web.cern.ch/fotis/QTOP/qtop-46-1.noarch.rpm

Alternatively, you may get the package in tarball, rpm or, via a repo file for multiple distros (tested: RHEL family) containing just 3 items:

• qtop
• qtop.colormap
• qtop.conf

The latter two are optional and supposed to be placed within your ~/.qtop directory or /etc and modified to suite your taste.

[coming soon: If you are on a RHEL clone and lucky enough with your EPEL repo, you may try: yum install qtop.]

How to use qtop

qtop runs fine in user space and should preferably be run in that way; the only reason to run it as root is on Computing Elements, for picking up DN-account mappings.

./qtop # try a single run of the utility and check its output

watch -d ./qtop # run it with watch; extremely useful for LRMS dynamic observation

./qtop|less -RS # this is for sites with too many nodes; use arrow keys for navigation

./qtop -c OFF # suppress color use in output (useful if colors appear corrupted); you can also fix this via ~/.qtop/qtop.conf

./qtop -a # try to renumber the names of WNs (useful if result is fishy, renumbering uses range wn[101-999])

./qtop -d # send this for debugging, if for whatever reason output seems strange; read paragraph on Reporting, too.

Command line options

# ./qtop -h

PBS report tool. Please try: watch -d ./qtop . All bugs added by fotisatcerndotch.

Usage: (simple vs extended format)

./qtop [-h] [-a] [-c OFF|ON|AUTO] [-f COLORMAPFILE ] [-o WN_COLON] [-p PBSPATH] [-s SOURCEDIR] [-x SUMMARY|NODES|ACCOUNTS]

./qtop [-h] [-a] [-b BDII] [-c OFF|ON|AUTO] [-d] [-f COLORMAPFILE ] [-i PBS] [-j JOBSUFFIX_OVERRIDE] [-l LRMS_OVERRIDE] [-s SOURCEDIR] \
            [-n PBSNODES_A] [-p PBSPATH] [-q QSTAT] [-r REPLACE_CHARS] [-s SOURCEDIR] [-u GETUSERS_COMMAND] [-x SUMMARY|NODES|ACCOUNTS] [-w]

-a      Try WN name remapping   This is used in situations where node names are not a pure arithmetic sequence (eg. rocks clusters)
-b      Local BDII hostname     Useful on cluster with grid m/w, to compare values among LRMS and Information System
-c      OFF|ON|AUTO colormode   Select use of ANSI-colored output. AUTO is the default and should work in most cases (sensitive upon input tty)
-d      Debug information       Use twice for extra output; mainly intended for usage by developer/debugging purposes
-f      Set COLORMAPFILE        Describe location of file with the definitions of color output, used for generating ANSI sequences
-i      Type of LRMS            One of TORQUE|PBS|GE|LL|LSF|SLURM|CONDOR (future-proof, currently only the first two are implemented)
-j      Set JOBSUFFIX_OVERRIDE  Use this if autodetection fails. eg. "jobsuffix.subdomain.example.com"
-l      Set LRMS_OVERRIDE       Use this if autodetection fails. eg. "mylrms"
-n      Set PBSNODES_A command  eg. '$PBSPATH/pbsnodes -a'
-q      Set QSTAT command       eg. '$PBSPATH/qstat'
-p      Set PBSPATH             Default is "/usr/bin". Some systems may have the respective pbs commands in PBSPATH='/usr/local/bin'
-s      Set SOURCEDIR           This is useful for offline testing. Directory should contain the stdout of pbsnodes -a , qstat -q , qstat
-o      Set vertical separator  Put vertical bar every WN_COLON nodes. Experimental option, output may be garbled in certain cases
-r      Character sequence      Define a sequence of replacement symbols for the individual user accounts. Experimental option
-t      Set alternative TMPDIR. qtop creates there a subdirectory and multiple temporary files underneath
-u      Cmd to get DN mappings  Useful on grid CEs. By default, '/opt/edg/sbin/edg-mkgridpool --list' should output the pool account mappings
-x      Exclude section         Exclude in output one or more of SUMMARY|NODES|ACCOUNTS sections
-w      Autotune, for first run Create a ~/.qtop directory and wget a default colormap file (will overwrite the existing one, so use with care)

Report issues, if any

Please report the results of any command listed under "How to use qtop", along with your expectations, in plain English language foremost.

Also, if possible, please do the following and attach the resulting files :

qstat -q > qstat_q.txt
qstat > qstat.txt
pbsnodes -a > pbsnodes_a.txt
tar -zcvf my_qtop_source_tarball.tar.gz pbsnodes_a.txt  qstat_q.txt  qstat.txt

If you place them in a single directory and run ./qtop -s mydir/ you should be able to reproduce the exact same effect as the one to be debugged.

Put all these in a tarball (tar.gz) or append them in an email to <fot...@cern.ch> (yes, eventually this procedure should get formalized with a bug tracker or such)

A very common cause for trouble is non-uniform worker node names. David O'Callaghan from Trinity College set an example on how to manage such cases:

# Because we have WNs with different domains (.cs.tcd.ie, .grid.cs.tcd.ie) I had to redefine PBSNODES_A in qtop.conf
function pbsnodesagrid()
{
  pbsnodes -a | grep -A7 wn[0-9]*.grid.cs.tcd.ie;
}
PBSNODES_A=pbsnodesagrid
# I had to make a function definition, as  the command did not work if I put it all on one line.

Epilogue

If you wish to modify the behavior of the script, there are a few variables possible to tune, which you could easily find inside /etc/qtop.conf, eg. PBSPATH=/usr/local/bin; (or, qtop -p /usr/local/bin) Once you decide what should be the defaults, save this file or tune the command line parameters.

NOTA BENE: The tool is really at a prototype state. It is not meant to be perfect for everybody (nor do I believe will ever that be possible, given the vast heterogeneity of cluster setups). Enjoy.

-- FotisGeorgatos - 2012-08-07