3D MPI/OpenMP Particle-in-Cell (PIC) BEPS3 codes
by Viktor K. Decyk, UCLA
copyright 1994-2018, regents of the university of california

These codes are part of the UPIC 2.0.4 Framework.  The primary purpose
of this framework is to provide trusted components for building and
customizing parallel Particle-in-Cell codes for plasma simulation.
The framework provides libraries as well as 3 different reference
applications illustrating their use.  The libraries are designed to
work with up to 3 levels of parallelization (using MPI for distributed
memory, OpenMP for shared memory, local vectorization).  Currently only
the first two are supported.  The reference applications are designed to
provide basic functionality common to many PIC codes, and are intended
to be customized and specialized by users for specific applications.
Currently, only spectral field solvers are currently supported, but the
particle methods are designed to be general.  In addition, other test
codes will also be provided, illustrating the use of some specific
component.  This is primarily intended to users who wish to incorporate
some component into an already existing code. Currently, only two test
codes are provided.

A Software License for use of this code is in the document:
Documents/License(CommercialReservation)

A description of the design of these codes and its components is in the
document:
Documents/BEPS3Design.pdf

A description of the current capabilities of these codes and its
components is in the document:
Documents/BEPS3Overview.pdf
Documents/MPI-OpenMP-PIC.pdf

Details about the mathematical equations and units used in this code is
given in the document:
Documents/UPICModels.pdf.

A description of the input parameters to these codes and their default
values are in the document:
mbeps3.source/input3mod.f90

For details about compilation and running, see mpbeps3.source/README

There are 3 Fortran reference codes for electrostatic, electromagnetic,
and darwin models, called mpbeps2, mpbbeps2, and mpdbeps2, respectively.
They are designed for Linux but will also compile for Mac OS X.  OpenMP
and MPI are used for parallelization, but if MPI is not available, the
codes can be compiled for OpenMP only

There are also 3 Python scripts for electrostatic, electromagnetic, and
darwin models, called mbeps3.py, mbbeps3.py, and mdbeps3.py,
respectively.  They require 3 dynamic libraries, libmpush3.so,
libmbpush3.so, and libmdpush3.so, respectively, which are located in
mpbeps3.source.  These python scripts currently only work in single
precision and only with OpenMP.  They work with both Python2 and Python3

A number of Directories are being developed to provide test codes to
illustrate components of these codes.  These are primarily intended for
those who wish to incorporate only pieces of the code into their own
codes.  The first one of these is the directory Periodic which
shows how to solve for the potential, longitudinal electric field,
vector potential and magnetic field using periodic boundary conditions.
A third one is the directory ReadFiles which contains sample programs
that illustrate how to read periodic fields which have been written
using simple Fortran IO by the reference applications.

To compile all these codes in the current (mpbeps3) directory, execute:

make

To compile just one, execute:

make program_name

where program_name is either: mpbeps3, mpbbeps3, or mpdbeps23

To execute, one first needs to copy an input file.  There are some
examples in the Examples directory in the current directory.  Files for
the electrostatic code are in the Examples/ES directory, those for the
electromagnetic code are in the Examples/EM directory, and those for the
darwin code are in the Examples/Darwin directory. To copy the the plasma
wave example for electrostatic code, type:

cp Examples/ES/input3.plasma input3

The command to execute a program with both MPI and OpenMP varies from
one system to another.  One possible command is:

mpiexec -np nproc -perhost n ./program_name

where nproc is the number of processors to be used, and where n is the
number of MPI nodes per host.

By default, OpenMP will use the maximum number of processors it can find
on the MPI node.  If the user wants to control the number of threads, a
parameter nvpp can be set in the main program.  In addition, the
environment variable OMP_NUM_THREADS may need to be set to the maximum
number of threads per node expected.  Finally, if OpenMP 3.1 is
available, setting the environment variable OMP_PROC_BIND=true generally
gives better performance by preventing threads from moving between CPUs.

To compile all these codes without MPI, execute:

make nompi

To compile just one, execute:

make program_name

where program_name is either: mbeps3, mbbeps3, or mdbeps3

To execute without MPI, type the name of the executable:

./program_name

To delete compiled libraries (.o files) and the executables, execute:

make clobber

For more details about compilation, including use of double precision,
see mpbeps3.source/README.

To compile the Python libraries in the current (mpbeps3) directory,
execute:

make python

To execute the scripts, then type:

python program_name

where program_name is either: mbeps3.py, mbbeps3.py, or mdbeps3.py

If files are generated by the reference applications, sample programs
to read them is available in the Directory ReadFiles.  A Fortran program
called diag3 is available to read the data.  This program can be used to
read scalar, vector, fluid, velocity, test particle trajectory or phase
space data.  The program diag3 calls lower level procedures called
preadf3, pvreadf3, pflreadf3, pfvreadf3, ptreadf3 and psreadf3 located
in the ReadFiles directory, each of which reads one of six types of
data.  Currently these are just skeleton codes which assembles data
arrays which have been written in segments by one of the MPI programs
and creates a normal array available for further processing.

To compile this code in the current (mpbeps3) directory, execute:

make readf

To execute this program, type:

./diag3

This program will examine a data metafile created by the simulation
(consisting of namelists) called diag3.idrun, where idrun is the run id
for that run, will determine which diagnostics were created, and will
ask the user which diagnostic to examine.

To delete compiled libraries (.o files) and the executable, execute:

make readfclobber

It is also possible to call executable versions of the low level
procedures directly by first compiling them with:

make readfall

Then executing

./ReadFiles/program_name

where program_name is of the the following:
preadf3, pvreadf3, pflreadf3, pfvreadf3, ptreadf3 and psreadf3

There is also a Python script available to read the data, diag3.py,
located in the mpbeps3 directory.  This in turn calls lower level
scripts called preadf3.py, pvreadf3.py, pflreadf3.py, pfvreadf3.py, 
ptreadf3.py, and psreadf3.py located in the ReadFiles directory.  These
scripts require a dynamic library (libcmfield3.so).  The scripts work
with both Python2 and Python3

To compile the dynamic library, execute:

make rfpython

The execute the top level script, type:

python diag3.py

which will call the low level scripts as needed.  It is also possible to
call the low level scripts directly as follows:

python ./ReadFiles/scriptname

where scriptname is either: preadf3.py, pvreadf3.py, pflreadf3.py, 
pfvreadf3.py, ptreadf3.py, or psreadf3.py

To delete the dynamic library, execute:

make rfpyclobber
