.. _detailed:

************************************
Running ChemShell: An Advanced Guide
************************************

This section outlines the procedures to run ChemShell on parallel computing 
platforms using MPI. It should be emphasised that due to the variations in 
operating systems between platform types (and indeed, between different 
installations running the same machines) that these instructions may need to 
be modified to reflect local conditions.

External codes
==============

It is possible in principle to launch parallel QM calculations without building 
or running ChemShell in parallel. Depending on the QM code, use specific
environment variables to invoke them in parallel. In some cases, parallel 
keywords should be specified as an argument to the QM codes. Consult the
manual page of the relevant code for more details.

This approach generally works well only on workstations or locally-managed
clusters. Other parallel environments, particularly HPC systems, may not allow
parallel binaries to be launched from another binary (e.g. if ``mpirun`` is not
available on the compute nodes). In this case, you must use a code that can be
directly linked into a parallel build of ChemShell (see below).

Parallel ChemShell
==================

ChemShell can be executed in parallel if it has been compiled using the 
``--mpi`` option (see the installation guide). The advantages of running 
ChemShell in parallel include execution of some internal modules such as DL-FIND 
in parallel, and the possibility to directly link external codes (see below).

The ``chemsh`` script contains the ``mpirun`` (or equivalent) command to 
launch the ChemShell binary (``chemsh.x``) in parallel. Do not pass the 
``chemsh`` script as an argument to ``mpirun`` as this will result in an error.
The ``-np`` argument is used to specify the number of processes required. 
E.g.::

  chemsh -np 4 input.py > output.log

will run ChemShell with 4 parallel processes.

As parallel execution is very platform-dependent, you may need to modify the
``chemsh`` script for your platform, particularly if ``mpirun`` is not used.

For some platforms (e.g. the UK national supercomputer service ARCHER), job 
submission functionality is included in the script, which can be specified
by e.g. ``--platform=archer`` and associated arguments.

Alternatively, there may be cases where it is more convenient to call the 
ChemShell binary directly. In this case, the environment variables that are 
normally set by the chemsh script must be set manually. 

Direct linking of external codes
================================

Some external codes (such as GAMESS-UK and GULP) can be linked in directly as 
libraries to ChemShell so that the package may be executed in parallel as a
single binary. This is more efficient than launching the external codes
separately, and is essential on platforms where it is not possible to launch
one parallel binary from another (e.g. most HPC facilities do not allow this).

The supported external codes must usually be compiled using special flags to
produce a library. Once this is done, direct linking must be specified when
ChemShell is compiled using arguments such as ``--gamess-uk``, etc. 
For further information, see the installation guide.

Once compiled, ChemShell is executed in parallel as described above. If the 
external code has been directly-lined, the directly-linked library will be
automatically used rather than a separate executable.

Task-farming parallelism
========================

MPI parallel builds of ChemShell can run in a task-farm parallel mode. The 
standard parallel mode is a master/slave configuration in which a Tcl 
interpreter is run on the master node and the slave nodes remain idle until a 
parallel input command is executed. All nodes are grouped together in the 
default MPI World communicator. In the task-farming parallel mode the MPI 
World communicator is split to form smaller sets of processors called 
workgroups. The number of workgroups is fixed for the lifetime of the 
ChemShell run and is specified using the command-line argument ``-nwg``, which 
is passed to ChemShell when it is executed. For example, the following 
command would run a 64-processor job using 4 workgroups of 16 processors 
each::

  chemsh -np 64 -nwg 4 input.py > output.log

In each workgroup one node acts as a master and the rest are slaves. Each 
master runs a copy of the Python interpreter and independently executes the 
same ChemShell input script.

.. _ga_memory:

Global Arrays (GA) memory
=========================

:py:class:`NWChem` is driven by the `GA <https://hpc.pnl.gov/globalarrays/index.shtml>`__ library. The maximum size of global memory per single process (MPI task) NWChem can allocate is defined as: total memory = heap memory + stack memory + global memory, where heap = stack = global/2. If :py:class:`NWChem` is to be called as a :ref:`directly linked library <install_nwchem>` the GA memory will be managed by ChemShell itself and can only be controlled via command line ``-gm`` or ``--global-memory``. For example, the following command allows a maximum global memory of 1024 MB (total 2048 MB)::

  chemsh -gm 1024 input.py > output.log



