Framework for Modernization and Componentization of Fusion Modules FMCFM: Status and Vision DE-FG-02-05ER84192

1. Framework for Modernization and Componentization of Fusion Modules (FMCFM): Status and Vision DE-FG-02-05ER84192 Ovsei Volberg (PI) Svetlana Shasharina and Johan Carlsson

2. Tech-X Corporation 2 Acknowledgements We would like to express our gratitude for the help, advice, comments and encouragement to the following people (in alphabetical order): Glenn Bateman, John R. Cary, Stephen Jardin, Jon Kinsey and Arnold Kritz All the people of the NTCC project. US DOE for the Phase 1 SBIR award. Finally I personally would like to thank all of you for inviting me to this seminar and for coming to listen to my talk.

3. Tech-X Corporation 3 Project Goal Assist Fusion Integrated Modeling projects by developing a framework that will contain a set of tools, standards, and techniques for modernization of established fusion codes with subsequent conversion of these modules to components with clear defined interfaces in accordance with modern software engineering practices. The project addresses the following problems relevant for the FSP: Modernization of Fortran modules to avoid their reimplementation, protect investment in software development, and increase performance and safety. Conversion of modernized modules components with well-defined interfaces to facilitate their coupling. Solve the language interoperability problem – use in the simulation of components written in different programming languages. Gain experience in components coupling.

4. Tech-X Corporation 4 DOE SBIR Program Contains 2 development phases Phase I:  100K, prototype development, project’s concretization Phase II:  $750, full development FMCFM is currently a Phase I project

5. Tech-X Corporation 5 Goals in Phase 1 Evaluate NTCC standards and procedures. Compare them with other community-wide efforts in other areas of science. Extend them based on this comparison. Use the new standards and Fortran 90 modernization techniques to modernize MMM95 and GLF23 transport modules. Define simplified interfaces between models and solvers Investigate the feasibility of using Common Component Architecture for converting fusion transport modules and solvers into components Evaluate CCA CCaffiene framework for component coupling and simulation. This will result in a prototype framework for fusion code modernization and componentization

6. Tech-X Corporation 6 Accomplishments to date Modernization We produced drafts of the additional (to NTCC Standards) development documents to be included in the FMCFM Developer’s Guide: Fortran Coding Rules Naming Convention Policy for Interoperability Document that incorporates modified NTCC Standards as requirements Provided a flexible and portable build system based on GNU Autotools (de facto standard for open source development) for the framework that could benefit other community projects as well. By using modern software engineering concepts available in Fortran 90/95 we created Fortran 90 wrappers to modernize two NTCC transport modules MMM95 - Multi-Mode transport model GLF23 - Gyro-Landau-Fluid drift wave model Incorporated a new transport equation solver implemented in C A combination of a relaxation scheme with a quasi-Newton scheme, well-suited for block-diagonal problems, like transport equations FMCFM is work in progress, it is not finished yet.

7. Tech-X Corporation 7 Outline for the rest of the talk Overview of Phase 1 modernization efforts What do we like in the NTCC ? Comparison of NTCC with other community wide efforts New development documentation Modernization of MMM95 and GLF23 Componentization and coupling by using CCA tools and techniques Simplified simulation Driver, Transport and Solver CCA Components Development of Ccaffeine coupled simulation Conclusions

8. Tech-X Corporation 8 Phase 1 FMCMF is a prototype: Phase I project is limited in scope: Centered around the NTCC Modules Library and NTCC Standards Focused on modernization of just 2 transport modules: MMM95 - Multi-Mode transport model GLF23 - Gyro-Landau-Fluid drift wave model Uses only CCA tools for componentization But the FMCFM will help us to answer the following questions important for fusion integrated modeling: Which tools and techniques could be used for modernization? What are the benefits and difficulties of using the CCA tools for componentization? How does one make a component out of existing code? What kind of interfaces are needed for coupling?

9. Tech-X Corporation 9 What do we like about the NTCC project? NTCC Standards - a good set of requirements Rob Jacob, one of the designers of the CCSM Flux Coupler: “I wish that the Climate community had such a set of requirements before we started” Thanks to the strict acceptance requirements, NTCC modules are modified already: Carries out a specific task (transport, heating and etc.) Isolated with an interface such as an argument list of a procedure Contains the driver program for testing, test cases, and documentation We like how the maintenance issues are solved: Peer Review Process Modern coding practices encouraged (for the new modules) Web-based access to module download and documentation Short answer : a lot Rob Jacobs: I wish the Climate Community would have such standards before starting CCSM Significant step towards the integrated modeling. Short answer : a lot Rob Jacobs: I wish the Climate Community would have such standards before starting CCSM Significant step towards the integrated modeling.

10. Tech-X Corporation 10 We evaluated several community-wide projects and their development standards Sun-Earth System Modelling and Space Weather Prediction Space Weather Modelling Framework (SWMF) http://csem.engin.umich.edu/SWMF Center for Integrated Space Weather Modelling (CISM) http://www.bu.edu/cism Earth Climate and Weather Modelling and Forecasting Earth System Modelling Framework (ESMF) http://www.esmf.ucar.edu European Program for Integrated Earth System Model (PRISM) http://prism.enes.org Community Climate System Model (CCSM) http://www.ccsm.ucar.edu Aerodynamics Analysis and Design NASA Report TM-2003-212421 Meteorology European Standards for Writing and Documenting Exchangeable Fortran 90 Code Common Components Architecture http://www.cca-forum.org

11. Tech-X Corporation 11 Results: Evaluation and comparison with the NTCC We elevated several “NTCC goals” to standards, for example: Avoid using hard-wired I/O unit numbers. Allow informational output to be switched on or off. Provide a method for rerouting warning or error message output to a user specified file or I/O unit number We enhanced the NTCC Standards by the “Parallelization Standards” to accommodate requirements of high-performance computing. We developed additional software engineering documents for the future FMCFM Developer’s Guide: Fortran 90 Coding Rules Naming Convention (both for C/C++ and Fortran) Interoperability Specification (the enhanced and modified NTCC Standards became a section on Requirements in this document) The documents have been published on the Tech-X website via Confluence software:( http://grid.txcorp.com and go to Fusion Framework Space) We will publish number of evaluated documents on this web site as well. We use the portable build system based on GNU Autotools similar to many community-wide projects, for example CCA and ESMF. We reinforced the build system by the Tech-X config module.

12. Tech-X Corporation 12 Results (cont.): Parallelization Standards Standard: The parallelization mechanism (if any) should employ the MPI standard for inter-module communications. Standard: If the MPI standard is used for intra-module communications, the physics module should be able to use an arbitrary MPI communicator. Standard: The module needs to be compiled as a library in addition to its standard mode when it is compiled and linked into a stand-alone executable. Standard: The module should read input from and write output to files. No hard-coded unit numbers are allowed. Goal: Module should be able to write its state to restart file(s) and continue its run after a stop of execution using restart file(s) with bit-to bit compatibility on the same platform-compiler combination.

13. Tech-X Corporation 13 Our Fortran 90 Modernization Approach: Abstraction, Encapsulation and Modularity MODULE: modules encapsulate data and routines; modules are containers for grouping type definitions, interface blocks, data and procedures. USE gives the access to module’s content. Allocatable arrays/pointers: support for dynamic data structures. TYPE: the user defined data types support abstraction. INTERFACE and MODULE PROCEDURE: support method overloading: the same method name , different argument lists. PRIVATE and PUBLIC: support data hiding and public access. Explicit interface blocks: enforce argument types verification in procedure calls. Renaming: prevent method name clashes between modules. Array syntax: whole array operations

14. Tech-X Corporation 14 Fortran features to be avoided (from our Fortran Coding Rules document) COMMON blocks – use the declaration of module instead. real*8 - use kind statement instead EQUIVALENCE - use pointer or derived data types instead to form data structures. Assigned and computed GOTO - use the case construct instead. Arithmetic IF statements - use the block if, else, else if , end if or select case construct instead. Labeled DO constructs - use unlabeled end do instead. PAUSE DATA and BLOCK DATA.

15. Tech-X Corporation 15 Modernization of the NTCC Fortran 77 Modules Each transport module main program is broken into 2 Fortran 90 (F90) modules: a data definition module and a module with procedures that operate on this data. Each COMMON block is substituted by a F90 module No hard-wired I/O unit numbers, all I/O operations to and from file; these files are associated with unit numbers at run-time via ModIoUtil F90 module. ModKind F90 module is used ModTransportInterfaceBlocks that includes explicit interface blocks for callglf2d and mmm95 routines Each transport module is built into a library: we link it with the test program if we need a stand alone version or we use it to create a CCA component with ports.

16. Tech-X Corporation 16 Example: two main F90 modules for each transport module Data F90 module module ModMMM95 use ModUnitNumbers implicit none save ….. real :: zdx, zxfact, zepslon ! the dimension of arrays below <= kr real, dimension(:),allocatable :: zxb , zrminor , zrmajor , zelong , zdense real, dimension(:),allocatable :: zdensi , zdensh , zdensz , zdensf , zdensfe … end module ModMMM95 Procedure F90 module module ModWrapMMM95 use ModMMM95 use ModTranspInterfaceBlocks implicit none ! private ! except !PUBLIC MEMBER FUNCTIONS public :: init_io public :: set_defaults public :: read_input_namelist public :: verify_data public :: setup public :: compute public :: write_output public :: finalize contains …… end ModWrapMMM95

17. Tech-X Corporation 17 Example: ModIoUnit Checks for I/O units that are free (unused). Associates files named input, temp and output with unused I/O units and opens them. Associates any named file with an unused I/O unit number and opens it. Closes opened files. An additional set of subroutines in a separate file allows to interface module procedures from Fortran 77 when USE statement cannot be used.

18. Tech-X Corporation 18 Example: ModTranspInterfaceBlocks ModTranspInterfaceBlocks contains the explicit interface blocks for subroutines callglf2d and mmm95 that perform calculations for the transport modules GLF23 and MMM95. If this module is accessed by a USE statement in any program unit that makes calls to any of those subroutines, then those calls would be checked by compiler for errors in the number or type of arguments. An explicit interface for a computational routine of any new transport module will be added to this module

19. Tech-X Corporation 19 Evaluation of the CCA is in progress Our current/future use of the CCA tools: The Ccaffeine execution environment for assembling coupled simulation from the models converted to components A peer-wise communication pattern via ports (common interfaces): Provide ports that are implemented by a component Uses ports, where c component makes calls to ports provided by another component Scientific Interface Definition Language (SIDL) that works with Babel language interoperability tool to address the language interoperability between Fortran transport components and the solver written in C. Our current work is concentrated on : We prototyped the provide/uses ports (CCA interfaces) for a Driver Component, Transport Component and Solver Component At the current step, we have started to work with the CCA language interoperability tools (SIDL/Babel) to create interoperable components

20. Tech-X Corporation 20 Simplified coupling as a test For the initial coupling we consider densities and electron temperature and its gradient constant We assume that only ion temperature Ti and its gradient grad Ti constitute a state vector at the moment t in the simplified simulation (not to produce good physics, but to learn coupling in Ccaffeine) The initial input parameters for a transport component are taken from its test cases downloaded from the NTCC web site. Set and Get methods for the state vector The new evaluate method that includes set methods and compute method.

21. Tech-X Corporation 21 Sketch for Ccaffeine Coupling of FMCFM Components

22. Tech-X Corporation 22 We are ready for CCA: Specifying CCA provide and use ports for the driver component, transport component and solver component Using SIDL/Babel to address the language interoperability. Using Ccaffeine framework as an environment to couple components “Receiver makes right” approach to insure that components get data in the form they need it.

23. Tech-X Corporation 23 Conclusions We gained valuable experience in NTCC module modernization, standards development and documentation. We created a portable build system that has been successfully used in our project. We developed Fortran modules and collected tools that can assist the community in the modernization efforts. We incorporated a new solver into the FMCFM We are working to formulate a coherent approach to how to build CCA components from Fortran legacy codes. We need to spend more time to determine common interfaces for coupling Driver with Transport Components (problems input/output transformations)

24. Tech-X Corporation 24 Questions to you What suggestions do you have for our work with standards and development documentation? What improvements could you recommend to our Fortran code modernization process? Suggestions for finishing the Phase 1 project and contents of the final report? Could you recommend what kind of role we could play in modernization and componentization which would help existing FSP efforts? Other modules?