This document concerns the parallel solution of the adaptive driven cavity problem. It is the first in a series of tutorials that discuss how to modify serial driver codes to distribute the
Problem object across multiple processors.
Most of the driver code is identical to its serial counterpart and we only discuss the changes required to distribute the problem. Please refer to another tutorial for a more detailed discussion of the problem and its (serial) implementation.
As described in the parallel processing document all parallel driver codes must initialise and shutdown oomph-lib's MPI routines by calling
MPI_Helpers::finalize(). The functions
MPI_Helpers::finalize() call their MPI counterparts,
MPI_Finalize(), which must not be called again.
In our demo driver codes, we surround all parallel sections of code with
#ifdefs to ensure that the code remains functional if compiled without parallel support (in which case the macro
OOMPH_HAS_MPI is undefined), for example.
In order to distribute the problem over multiple processors a single call to the function
Problem::distribute() is all that is required. Thus, a minimally-changed serial driver code would be
Finally, we must call
MPI_finalize() before the end of the main function
The actual driver code is slightly more complicated that the version shown above because it also acts as a self-test. The distribution of individual elements over the processors is determined by
METIS and we have found that
METIS occasionally gives slightly different results on different machines. To ensure reproducible results when acting as a self-test, the code (like all our parallel test codes) reads a predetermined element distribution from the disk; see below for more details.
A particular feature of the driven cavity problem is that the flow is completely enclosed and that a single pressure degree of freedom must be prescribed. In the serial driver code, we arbitrarily pinned the first pressure degree of freedom in the "first" element in the mesh. Once the problem is distributed this element will only be available to particular processors and the pressure degree of freedom must be pinned on each one. Consequently we re-write the
actions_after_adapt() function as follows:
This change ensures that every processor that holds the element containing the node at position (0,0) (i.e. the first element) fixes the pressure for that element. The floating-point comparison does not cause any problems in this case because the
Node's position is explicitly set to "exactly" (0.0,0.0) in the Mesh constructor and never changes.
It is not necessary to change the corresponding statements in the problem constructor because the problem distribution occurs after the problem has been constructed. In fact, the problem constructor is unchanged from the serial version.
doc_solution() routine requires a slight modification to ensure that the output from different processors can be distinguished; this is achieved by including the current processor number in the filename of the solution:
The figure below shows the mesh after the final solution of this problem, distributed across two processors, with the two colours indicating which processor the elements belong to.
The actual driver code demonstrates two of the different options available for the
Problem::distribute() function, selected by the presence or absence of command line arguments.
If no command line arguments are specified we determine the problem distribution using
METIS, and write the distribution to a file.
The distribution is represented by a vector of unsigneds whose values indicate the processor on which the corresponding element is stored.
We distribute the problem with a call to
Problem::distribute(...), using the boolean flag to request that the relevant statistics are displayed on screen. The distribution chosen for the elements is returned in the vector created earlier.
We document the distribution by writing the distribution vector to a file.
Finally, we perform an optional self-test of the halo-haloed lookup schemes.
If command line arguments are specified (typically when the code is run in validation mode) we read the distribution from disk, using a file that was written using the procedure shown above. (This is useful because in our experience
METIS may produce slightly different distribution on different machines. This would cause the self-tests to fail even though the computed results would be correct).
We start by creating a
DocInfo object that specifies the directory in which the problem distribution will be documented and by reading in the vector that represents the distribution
We pass the distribution vector to
Problem::distribute(...) and thus bypass the partitioning by
We note that it is possible to document a mesh's distribution at any point, using the
Mesh::doc_mesh_distribution(...) function, as indicated here
The driver code from which this example is taken also solves the same distributed problem using Crouzeix-Raviart elements. The fully modified parallel driver code can be found at
For further examples of using the
distribute() function for both two-dimensional and three-dimensional single-domain problems, see the directory
A pdf version of this document is available.