Collaborators
PERI collaborates both with application teams and with other
performance engineering researchers. This page lists the collaborators
organized by their role within a particular application engagement or
performance tuning effort.
At present, there are 3 main collaboration areas, two focusing on
direct engagement with computational science projects:
1) Tiger Teams
2) Application code team liaisons
and one on community standards and tool interoperability:
3) PERI-DB, performance data interoperability.
Applications engagement
Tiger Teams
Tiger Teams are focused efforts applied to high priority application
codes with well documented performance goals or needs. They are
composed of performance optimization experts who are typically
members of the PERI collaboration, although other performance
engineering experts also participate. While the life time of a
Tiger Team is indeterminate, they usually last from six to twelve months.
At present, PERI has two tiger teams, working with the following codes.
S3D
S3D solves the full compressible Navier-Stokes equations that describe
the conservation of mass, momentum, and energy, and laws of gas
behavior, while simultaneously tracking the evolution of reactive
species on a rectangular mesh. It uses the message passing interface
(MPI) to efficiently distribute the calculation among parallel
processors, and is built on a hierarchical, modular structure.
The S3D Tiger Team was formed to assist the S3D application team
in meeting its Joule requirements. Specific concerns of the application
team focused on single node performance as well as how the code
would scale on the combined Cray XT3/XT4 system at Oak Ridge National
Laboratory (ORNL). The Tiger Team has actively worked with the
application team to identify several optimization opportunities
and to provide mechanisms to exploit them. Specific changes include
development of a simple transformation tool to optimize certain
Fortran statements; the modified code is being added to the application
team's source base. We have also targeted study of potential
optimizations of the exp intrinsic function that will likely
be adopted by the application team. Scaling studies on the ORNL
system identified load balance issues arising from the XT3 nodes
having lower memory bandwidth than the XT4 nodes. The Tiger Team
designed a potential solution to this issue as well although the
application team has decided not to adopt it since the ORNL
machine will only have XT4 nodes in the near future. While the
application team has successfully met its Joule requirements,
the Tiger Team continues to work with them in order to prepare
for full scale runs on the anticipated petascale platform at ORNL.
* Tiger Team Lead: Bronis de Supinski (LLNL)
* Application Team Contact: Jackie Chen (SNL)
* DOE Liaison: Kenneth Roche (ORNL)
* PERI Team Members:
* David Bailey (LBNL)
* Bronis de Supinski (LLNL)
* Mike Fagan, John Mellor-Crummey (Rice)
* Allan Snavely, Nick Wright (SDSC)
* Affiliate Team Members:
* Sameer Shende, Alan Morris, Allen Maloney, Kevin Huck (UOregon/ParaTools)
* Jeff Larkin (Cray/ORNL)
* Application Team Participants
* Jackie Chen, David Lignell (SNL)
GTC: Gyrokinetic Toroidal Code
The Gyrokinetic
Toroidal Code (GTC) has been developed to study plasma
microturbulence in fusion devices. The GTC Tiger Team was formed to
assist with performance optimization and scalability issues related to
achieving the Joule goals for GTC of 1) running 50 percent faster on
the model problem, and 2) running twice as long in simulation time for
a fixed wallclock time.
GTC solves the gyro-averaged Vlasov-Poisson system of equations using
the particle-in-cell (PIC) approach. In GTC, the main bottleneck is
the charge deposition, or scatter operation, and this is also true for
most particle codes. The scatter algorithm in GTC is more complex
since one is dealing with fast gyrating particles for which motion is
described by charged rings being tracked by their guiding center.
Hand-tuning techniques such as common subexpression elimination, code
movement, loop unrolling, and cache blocking were used to improve
performance of the charge deposition routine by around 10 percent.
These changes have been incorporated into the production version of
the code.
New physics in the GTC-S version of the code allows the
input of real experimental profiles and magnetic equilibria,
introducing the important effects of shaped cross-section physics.
The implementation of this new capability required extensive changes
to the code, and the goal is now to bring GTC-S to the performance
level of the original GTC and beyond, so as to be able to do global
simulations of plasmas in the ITER
fusion reactor, which will be coming on-line in about five years.
Tiger Team participants have used cache modeling techniques to develop
further optimizations to the charge deposition and other key GTC-S
routines. These code transformations have been sent to the developers
and are awaiting their evaluation. Tiger Team participants are also
investigating scalability and load imbalance issues with the new GTC-P
version of the code that partitions the poloidal plane into radial
shells.
* Tiger Team Lead: Shirley Moore (UTK)
* Application Team Contact: Stephane Ethier (PPPL)
* DOE Liaison: Kenneth Roche (ORNL)
* PERI Team Members
* Haihang You (UTK)
* John Mellor-Crummey, Gabriel Marin, Guohua Jin (Rice)
* Hongzhang Shan (LBNL)
* Affiliate Team Members
* Kevin Huck (UOregon)
* Eduardo D'Azevedo (ORNL)
* Application Team Participants
* Stephane Ethier, Weixing Wang, Wei-li Lee (PPPL)
* Scott Klasky (ORNL)
Application Liaisons
In addition to the focused Tiger Team activity, PERI designates
members of the PERI collaboration as liaisons with SciDAC science
application teams that desire additional help in the performance
engineering of their codes. The liaison activity is further broken
down into "active" liaisons, in which it is clear how PERI can
contribute and in which there is close collaboration between PERI and
the application team, and "passive" liaisons, in which PERI provides
advice and tracks the progress of the application teams. The status of
a PERI liaison activity can change between active and passive at any
time, and such changes are expected as the needs of the application
projects evolve. As distinct from the Tiger Teams, the liaison
activity is a long term engagement with the application teams.
PERI has active liaison relationships with the following application
teams currently:
1) Advanced Methods for Electronic Structures (http://www.scidac.gov/matchem/petachem.html)
Principal Investigator: George Fann
Application Team Contact: Rebecca Hartman-Baker (ORNL)
PERI Liaison: Philip Roth (ORNL)
Other PERI Personnel:
Codes: MADNESS
2) Center for Plasma Edge Simulation (http://www.scidac.gov/FES/FES_CPES.html)
Principal Investigator: C-S Chang (NYU)
Application Team Contact: Julian Cummings (CalTech)
PERI Liaison: Patrick Worley (ORNL)
Other PERI Personnel: John Mellor-Crummey (Rice)
Other Affiliates: Kevin Huck (UOregon)
Codes: XGC-1, XGC-0, GTC, M3D, NIMROD, ELITE, DEGAS-2
3) Community Petascale Project for Accelerator Science and Simulation
(ComPASS) (http://www.scidac.gov/physics/COMPASS.html)
Principal Investigator: Panagiotis Spentzouris (Fermi National
Accelerator Laboratory)
Application Team Contact: John Cary (Tech-X)
PERI Liaison: Boyana Norris (ANL)
Other PERI Personnel: ???
Codes: VORPAL
4) Hierarchical Petascale Simulation Framework for Stress and Corrosion
Cracking (http://www.scidac.gov/matchem/crack.html)
Principal Investigator: Priya Vashishta (USC)
Application Team Contact: Aiichiro Nakano (USC)
PERI Liaison: Mary Hall (USC)
Other PERI Personnel: Jacqueline Chame (USC), Chun Chen (USC), John Mellor-Crummey (Rice)
Codes: Quantum chemistry
5) Linear Scale Electronic Structure Calculations for Nanostructures
(http://www.nccs.gov/leadership-science/materials/linear-scale-electronic-structure-calculations-for-nanostructures/)
Principal Investigator: Lin-Wang Wang (LBNL)
Application Team Contact: Lin-Wang Wang (LBNL)
PERI Liaison: David Bailey (LBNL)
Other PERI Personnel: Dan Gunter (LBNL), Hongzhang Shan (LBNL)
Codes: LS3DF
6) Modeling Multiscale-Multiphase-Multicomponent Subsurface Reactive
Flows using Advanced Computing
(http://www.scidac.gov/groundwater/gwflow.html)
Principal Investigator: Peter Lichtner (LANL)
Application Team Contact: Glenn Hammond (PNNL), Richard Mills (ORNL)
PERI Liaison: G. Mahinthakumar (NCSU)
Other PERI Personnel: Sarat Sreepathi (NCSU), Vamsi Sreepathi (NCSU)
Codes: PFLOTRAN
PERI has also assigned passive liaisons to track the performance needs
of the following projects:
1) Hybrid Numerical Methods for Multiscale Simulations of Subsurface
Biogeochemical Processes
(http://www.scidac.gov/groundwater/gwscaling.html)
PERI Liaison: G. Mahinthakumar (NCSU)
2) Framework Application for Core-Edge Transport Simulations (FACETS)
(http://www.scidac.gov/fusion/fullscale.html)
PERI Liaison: Patrick Worley (ORNL)
3) Multidimensional Simulations of Core-Collapse Supernovas
(http://www.nccs.gov/leadership-science/astrophysics/multi-dimensional-simulations-of-core-collapse-supernovae/)
PERI Liaison: Robert Fowler (RENCI)
4) National Computational Infrastructure for Lattice Gauge Theory
(http://www.scidac.gov/physics/quarks.html)
PERI Liaison: Ying Zhang (RENCI)
5) A Scalable and Extensible Earth System Model for Climate Change
Science (http://www.scidac.gov/climate/earth.html)
PERI Liaison: Patrick Worley (ORNL)
6) Simulations of Turbulent Flows with Strong Shocks and Density
Variations (http://www.scidac.gov/physics/shock.html)
PERI Liaison: Bronis de Supinski (LLNL)