Skip to content

This updates the AMR and AMB2018 documentation for OpenFOAM-14 runtime #107

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
colemanjs wants to merge 1 commit into
base: gh-pages
Choose a base branch
from
Open

This updates the AMR and AMB2018 documentation for OpenFOAM-14 runtime #107

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 36 additions & 16 deletions reference/amr.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ usemathjax: true

# Adaptive Mesh Refinement

AdditiveFOAM generates `refinementField` from two criteria: temperature above `refinementTemperature` and intersection with buffered, positive-power portions of the future scan path. OpenFOAM's `refiner` uses this field to refine and coarsen the mesh; its `loadBalancer` can then redistribute the changed mesh between MPI processes.
AdditiveFOAM generates `refinementField` from two criteria: temperature above `refinementTemperature` and intersection with buffered, positive-power portions of the future scan path. OpenFOAM's `refiner` uses this field to refine and coarsen the mesh; its cell-count-based mesh distributor can then redistribute the changed mesh between MPI processes.

The selected refinement model determines the future scan-path interval represented by `refinementField`:

Expand All @@ -22,13 +22,13 @@ The selected refinement model determines the future scan-path interval represent
| **`targetCellLoad`** | A variable look-ahead interval chosen to approach a target number of cells per process |
| **`none`** | No AdditiveFOAM refinement marker |

The top-level `refinement` dictionary is optional and defaults to `model none`. Users select the model and source-specific buffer sizes in `constant/heatSourceDict`, then configure refinement limits and optional load balancing in `constant/dynamicMeshDict`.
The top-level `refinement` dictionary is optional and defaults to `model none`. Users select the model and source-specific buffer sizes in `constant/heatSourceDict`, then configure refinement limits and optional runtime redistribution in `constant/dynamicMeshDict`.

To enable AMR:

1. Select `timeStep`, `uniformTimeIntervals`, or `targetCellLoad` in `constant/heatSourceDict` and provide one buffer for each named heat source.
2. Configure OpenFOAM's `refiner` in `constant/dynamicMeshDict`, including `refineInterval 1`, `maxRefinement`, and `maxCells`.
3. Add the `loadBalancer` and a redistribution method in `system/decomposeParDict` when the refined mesh should be redistributed in parallel.
3. Add the mesh `distributor` and a parallel redistribution method in `system/decomposeParDict` when the refined mesh should be redistributed in parallel.
4. During the run, inspect the reported cell count and load imbalance. With `dumpLevel true`, reconstruct and visualize `cellLevel` and `refinementField` in ParaView to confirm where refinement is requested and applied.

<figure class="documentation-figure">
Expand Down Expand Up @@ -210,40 +210,60 @@ topoChanger

distributor
{
type loadBalancer;
type distributor;
libs ("libfvMeshDistributors.so");
redistributionInterval 10;
maxImbalance 0.10;
}
```

`refineInterval` must be `1` for the AdditiveFOAM refinement models. The OpenFOAM `refiner` therefore evaluates the updated `refinementField` every CFD time step. `maxRefinement` supplies $$L_{\max}$$ in the equations above. `maxCells` bounds the global mesh size, and `redistributionInterval` is measured in time steps.
`refineInterval` must be `1` for the AdditiveFOAM refinement models. The OpenFOAM `refiner` therefore evaluates the updated `refinementField` every CFD time step. `maxRefinement` supplies $$L_{\max}$$ in the equations above. `maxCells` bounds the global mesh size. The distributor checks the cell-count imbalance every `redistributionInterval` time steps and redistributes only when it exceeds `maxImbalance`.

The load balancer reads its distributor from `system/decomposeParDict`:
The recommended OpenFOAM-14 configuration uses the parallel PT-Scotch distributor. In `system/decomposeParDict`:

```foam
numberOfSubdomains 8;

method scotch;
decomposer scotch;

distributor zoltan;
libs ("libzoltanRenumber.so");
distributor ptscotch;
libs ("libptscotchDecomp.so");

zoltanCoeffs
constraints
{
lb_method graph;
lb_approach repartition;
refinementHistory
{
type refinementHistory;
}
}
```

For processor CPU-load estimates $$C_p$$, OpenFOAM defines the average $$\overline C=N_p^{-1}\sum_pC_p$$ and a conservative maximum $$C_{\max}$$ from the maximum base-CFD and registered load components. Redistribution occurs when
Let $$n_p$$ be the number of cells on process $$p$$, $$N$$ the global number of cells, and $$N_p$$ the number of MPI processes. The ideal cell count is $$\bar n=N/N_p$$. OpenFOAM redistributes when

$$I=\frac{C_{\max}-\overline C}{\overline C}
$$I=\max_p\left|1-\frac{n_p}{\bar n}\right|
>I_{\max},$$

where $$I_{\max}$$ is `maxImbalance`. Zoltan operates only in parallel and requires `libzoltanRenumber.so`; `lb_approach repartition` computes a new weighted decomposition from the current one.
where $$I_{\max}$$ is `maxImbalance`. Because AdditiveFOAM does not currently supply per-cell cost weights, this balances cell count rather than measured CPU work. The `refinementHistory` constraint keeps the children of a refined parent on the same process, preserving reliable unrefinement after redistribution.

`redistributionInterval 10` and `maxImbalance 0.10` are useful starting values, not universal optima. A shorter interval responds sooner but can spend more time partitioning and migrating the mesh; a longer interval reduces that overhead but permits a larger temporary imbalance. Tune both values for the production mesh and MPI count.

Zoltan remains available as an optional OpenFOAM-14 distributor when OpenFOAM and ThirdParty are built with Zoltan support:

```foam
distributor zoltan;
libs ("libzoltanDecomp.so");

zoltan
{
lb_method graph;
lb_approach repartition;
}
```

Here $$p$$ indexes MPI processes, $$C_p$$ is the estimated load on process $$p$$, $$\overline C$$ is the process-average load, $$C_{\max}$$ is the conservative maximum load, and $$I$$ is the relative load imbalance.
The [`zoltanRenumber` removal](https://github.com/OpenFOAM/OpenFOAM-14/commit/4b6b4924082f3499c99687fd45108410e59d1788) removed a mesh-renumbering method, not this parallel distributor. In OpenFOAM-14 the distributor library is `libzoltanDecomp.so`, and its settings dictionary is named `zoltan`, without a `Coeffs` suffix.

{: .warning }
Do not select `type loadBalancer` for AdditiveFOAM at present. That OpenFOAM component requires solver-registered `cpuLoad` objects; AdditiveFOAM does not allocate them and the run terminates with `No CPU loads have been allocated`. Use `type distributor` for cell-count-based redistribution.

## `timeStep`

Expand Down
8 changes: 5 additions & 3 deletions tutorials/amb2018.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ AMB2018-02-B is the baseline single-track case and the starting point for the qu
- One `superGaussian` source with `k 2`, `radius (85 85)` µm, `depth 30` µm, `definition secondMoment`, and absorptivity `0.33`.
- `tolerance 1e-3` and `nPoints (10 10 10)`.
- A 179.2 W, 0.8 m/s scan path.
- Optional refinement configuration, disabled by default.
- Optional refinement and runtime redistribution configuration; both are disabled by default.
- Fluid flow disabled by `nOuterCorrectors 0` in the tutorial case.
- Eight MPI ranks.

Expand All @@ -46,16 +46,18 @@ The solver log should report approximately 33% of the active incident power as a
|---|---|
| `constant/heatSourceDict` | Absorption, source distribution, sampling, and optional AMR |
| `constant/scanPath` | Beam position, power, and speed |
| `constant/dynamicMeshDict` | OpenFOAM topology changer |
| `constant/dynamicMeshDict` | OpenFOAM topology changer and cell-count-based mesh distributor, with its type set to `none` by default |
| `system/blockMeshDict` | Base mesh |
| `system/fvSchemes` | Defaults to Euler |
| `system/fvSolution` | Defaults to the explicit temperature path |
| `system/decomposeParDict` | Eight-rank domain decomposition |
| `system/decomposeParDict` | Eight-rank initial decomposition, optional PT-Scotch redistribution, and refinement-history constraint |

## Workflow

`Allrun` generates the mesh, decomposes the case, runs `additiveFoam` in parallel, reconstructs the OpenFOAM fields, and invokes the ExaCA and solidification-data reconstruction utilities. The latter utilities produce output only when their corresponding Function Objects are enabled in `system/controlDict`.

The case includes a PT-Scotch runtime-redistribution configuration with a 10-time-step check interval, a 10% cell-count imbalance threshold, and the required refinement-history constraint. Redistribution is disabled by `type none` in `constant/dynamicMeshDict`; change this to `type distributor`, then select an active refinement model as described in the [AMR guide]({{ '/docs/amr/' | relative_url }}) to exercise refinement and redistribution.

## Outputs

Open the reconstructed case in ParaView to visualize temperature, phase fraction, velocity, pressure, and heat-source fields. The same case supplies the quantitative outputs below.
Expand Down
Loading