A parallel, adaptive library for logically Cartesian, mapped, multiblock domains
What is ForestClaw?
ForestClaw is a parallel, multi-block adaptive finite volume library for solving PDEs on mapped, logically Cartesian meshes. For solving hyperbolic problems using explicit, single step algorithms, ForestClaw's block-structured adaptive algorithm, including multi-rate time stepping uses the Berger, Oliger and Colella AMR algorithms (JCP, 1984, 1989). The hyperbolic solvers are currently based on ClawPack (R. J. LeVeque). Future plans include support for general method-of-lines solvers in a multi-rate setting.
Where ForestClaw departs from the standard Berger-Oliger-Colella block-structured approach is that the multi-resolution grid hierarchy is not stored as overlapping, nested grids but rather as a composite structure of non-overlapping fixed sized grids, each of which is stored as a leaf in a forest of quad- or octrees. The particular code base we use for managing the tree is p4est (www.p4est.org), (Carsten Burstedde, Univ. of Bonn).
Why develop another AMR code?
Currently, there are several AMR codes for doing block-structured AMR. What are the advantages of the ForestClaw approach?
- By using an external library for mesh management, we completely separate issues related to grid connectivity, load balancing, and regridding, from the underlying numerics. With p4est, we can exploit state-of-the-art technology for managing tree-based data structures, including the use of space filling curves for maintaining data locality in parallel.
- Because of the composite quad- or octree layout, ghost cell exchanges occur in a regular pattern at patch boundaries, rather than between ghost cells on the fine grid and interior grid cells on the coarse grid. This eliminates the need to store the locations of where fine grids are positioned in the coarse grid. Also, only a limited number of neighbor configurations are allowed, again reducing the code complexity and simplifying the development of new numerical algorithms. This simplification is particularly advantageous in 3d.
- p4est has been built for distributed parallelism and designed to allow transparent and efficient parallelization of numerical codes such as ForestClaw. For example, the p4est algorithms for regridding and re-balancing (ensuring that all levels are properly nested) has been shown to scale well on up to 262,144 CPU cores.
- ForestClaw inherits the multi-block domain structure from p4est and so adaptive mesh refinement for non-rectangular domains, cubed-sphere domains, and other domains not easily handled by a single adaptive quadtree.
Progress on the ForestClaw project
- (Winter, 2017) GeoClaw extension of ForestClaw largely complete. Begin work on 2.5 dimensional version.
- (Fall 2016) Dynamic support for both Clawpack 4.x and Clawpack 5.0 libraries in all examples (advection, gas dynamics, acoustics, SWE, transport and so on).
- (Summer 2016) Extend ForestClaw with GeoClaw (www.geoclaw.org).
- (Spring 2016) Final progress on our paper; accuracy tests of ForestClaw.
- (Fall/Winter, 2016) Developing performance testing tools for running on a local linux cluster (64-nodes) and JUQUEEN (Blue Gene/Q) machines; obtain scaling performance for a variety of test problems.
- (Spring/Summer, 2015) Parallel multi-block code mostly working; parallel performance optimizations reduce regridding time significantly.
- (Winter, 2015) Parallel IO for package and user options.
- (Fall, 2014) Periodic domains
- (Summer, 2014) Mapped, multiblock capabilities for several useful domains, including cubed disk and sphere,
- (Spring, 2014) Build system using GNU Autotools
- (Winter, 2014) Parco paper appears.
- (Summer, 2013) Parallel (MPI) capabilities added; Parco paper submitted and accepted.
- (Fall, 2012) General mappings available
- (Fall, 2012) Multirate time stepping
- (Summer, 2012) Development on ForestClaw begins
- (March, 2012) D. Calhoun in charge of an AMR Working Group at HPC3, a regular workshop held at KAUST, Saudi Arabia.
Publications and Presentations
Recent talks and posters
- Clifford Lectures, April 5-7, 2017, (New Orleans, LA) (talk : .pdf)
- SIAM CSE, Feb. 27 - Mar. 3, 2017 (Atlanta, GA) (talk: .pdf, (poster: .pdf)
- SIAM Parallel Processing, April 2016 (Paris, France) (talk 1: .pdf, (talk 2 : .pdf)
- SIAM CSE, March, 2015 (Salt Lake City, Utah) (talk: .pdf, poster: .pdf)
- HPC3, November, 2014 (Kaust). (.pdf)
- D. Calhoun, C. Burstedde, ForestClaw : A parallel algorithm for patch-based adaptive mesh refinement on a forest of quadtrees, (submitted), 2017. (arXiv:1703.03116)
- C. Burstedde, D. Calhoun, K. Mandli, A. Terrel, Hybrid forest-of-octrees AMR for hyperbolic conservation laws, (2014). (arXiv:1308.1472, iospress)
- (April, 2017) R. J. LeVeque (developer of the original ClawPack-GeoClaw code) was the invited lecturer at the Tulane University Clifford Lecture Series. Several ClawPack/GeoClaw/ForestClaw developers were invited.
- (February, 2017) At SIAM CSE 2017 (Atlanta, GA), the Clawpack-GeoClaw Mini-posterium won a poster prize, which came with $250, which will be given to junior members of the Clawpack group. Good work! (ForestClaw poster)
- (November, 2016) First ForestClaw Hackathon. Kyle Mandli (Columbia Univ.) and Melody Shih (Columbia Univ.) spent four days in Boise, working on updates to ForestClaw. See the picture of our hike into the Boise hills here.
- (September, 2016) D. Calhoun, ForestClaw developer, will attend the Clawpack Hackathon Hack-a-thon, in Seattle, WA, August 27-29, 2016, to work with Melody Shih (Columbia Univ.) on a ForestClaw implementation of GeoClaw
- (April, 2016) Attend the three-part mini-symposium at the upcoming
SIAM Parallel Processing Meeting (Paris, France, April 13-16,
2016) to find out more about progress in parallel adaptive
methods, and to find out more about ForestClaw
(Wednesday 4/14 : MS18 - 10:25AM, MS26 - 2:40PM, MS34 - 4:50PM)
- (February, 2015) Attend our minisymposium at SIAM
CSE 2015 (Salt Lake City, March 14-18, 2015)
(Sunday 3/15 : MS102 - 9:10AM, MS127 - 1:30PM)
- (September, 2014) Developer Donna Calhoun was recently featured in a Boise State update Math vs. the Volcano : Mitigating an Eruption's Aftermath
- (July, 2014) We have recently received three years of National Science Foundation funding (Award #1419108, $194K) for development of ForestClaw. Please see the project_summary for a description of the award. Collaborators on this project include Carsten Burstedde (Univ. Bonn, Germany), D. Ketcheson (Kaust, Saudi Arabia) and D. George (USGS, Vancouver, WA).
Working on the ForestClaw project
There are funding opportunities for students interested in working on developing numerical solvers and applications using ForestClaw. We are recruiting for :
- A Master's level student to work volcanic ash cloud dispersion using ForestClaw. Potential candidates should have excellent programming skills, as well as an interest in geophysical applications. Funding is available for 1 year. Please contact Donna Calhoun at firstname.lastname@example.org.
Talin Mirzakhanian (BS, California State Polytechnic University) started work towards a Master's Degree in the Fall of 2015, working with D. Calhoun on developing multi-rate Runge-Kutta-Chebyschev methods for adaptive Cartesian grids.
Melody Shih (Master's student, Columbia Univ.) is working with D. Calhoun and Kyle Mandli (Columbia Univ.) to port GeoClaw, a widely used application for modeling tsunamis, storm surges and debris flows, to ForestClaw. Check back for progress!
Where can I get ForestClaw?
(March, 2017) We just submitted our paper on the parallel ghost-filling algorithm in ForestClaw, and are now cleaning up the code for GitHub. It really is coming soon!
- Melody Shih, (Columbia University)
- Kyle Mandli, (Columbia University
- Grady Lemoine, (CD-adapco, Bellevue WA)
Donna Calhoun would like to acknowledge the support of the Isaac Newton Institute and the program Adaptive Multiscale Numerics for the Ocean and Atmosphere, where much of the initial work in developing ForestClaw was done, and the National Science Foundation (NSF).
|* more to come, including gas dynamics, acoustics and shallow water wave equations|
Last modified: Thu Mar 16 06:55:34 MDT 2017