eCLM performance: I/O operations

[s-poll]

General

This is an issue to collect information and discuss the I/O behavior of eCLM. I/O operations in standard configuration take ~ 20% and strong scaling shown that it can easily take up to 50% of the runtime. This should be a discussion board to discuss eCLM performance in respect to I/O operations.

eCLM uses the ParallelIO library for I/O. Some I/O operations can be managed via the namelist lnd_modelio.nml. Details are below as well as discussion points for I/O strategy in coupled system.

namlist parameter &pio_inparm

'pio_netcdf_format'

Netcdf format. Currently set to '64BIT_OFFSET'.

'pio_numiotasks'

Pio number of IO tasks

It is hardcoded to 64, if not set (e.g. negative values)!

It should propably be set to the number of mpi-tasks. Not done any investigation on runtime yet.

'pio_rearranger'

pio_rearranger= 1 --> Box rearrangement
pio_rearranger= 2 --> Subset rearrangement

https://ncar.github.io/ParallelIO/decomp.html

'pio_root'

I/O root. Set in code

'pio_stride'

Stride in the mpi rank between io tasks.

'pio_typename'

NetCDF type. e.g. netcdf 3 or 4. Currently 'netcdf4p' (netcdf 4 HDF5 - parrallel) is chosen. Other option: 'netcdf4c' (compressed)

Discussion point

Tuning of runtime in interplay between 'pio_rearranger', 'pio_root' and 'pio_stride'. This will be case-dependent, but has potential for improving runtime performance.

Asychnronsous IO: The PIO library seems to be able asychnronsously. However, it seems that this is not integrated in eCLM.

I/O strategy in coupled system

eCLM is run sequentially coupled in the TSMP2 coupled system. The exchange of to the OASIS-coupler is done before/after the CLM call (clm_drv) and the I/O operation are done within the clm_drv. Thus, the I/O time is added to the total runtime of the coupled system. One approach would be that the I/O operations are done after the oas_send (OASIS send), so that the consumed time for I/O operations .

An alternative would be to include asynchronous I/O, with the aim that I/O operations minimizes the impact on the total runtime of the eCLM itself.

[kvrigor]

'pio_numiotasks'

Pio number of IO tasks

It should propably be set to the number of mpi-tasks. Not done any investigation on runtime yet.

So far this works on the NRW_ParFlow-eCLM test case.

[kvrigor]

Adding to @s-poll's notes: The possible values for pio_typename can be found here

eCLM/src/csm_share/util/shr_pio_mod.F90

Lines 644 to 668 in 7853ec4

	subroutine shr_pio_getiotypefromname(typename, iotype, defaulttype)
	use shr_string_mod, only : shr_string_toupper
	character(len=*), intent(inout) :: typename
	integer, intent(out) :: iotype
	integer, intent(in) :: defaulttype
	typename = shr_string_toupper(typename)
	if ( typename .eq. 'NETCDF' ) then
	iotype = pio_iotype_netcdf
	else if ( typename .eq. 'PNETCDF') then
	iotype = pio_iotype_pnetcdf
	else if ( typename .eq. 'NETCDF4P') then
	iotype = pio_iotype_netcdf4p
	else if ( typename .eq. 'NETCDF4C') then
	iotype = pio_iotype_netcdf4c
	else if ( typename .eq. 'NOTHING') then
	iotype = defaulttype
	else if ( typename .eq. 'DEFAULT') then
	iotype = defaulttype
	else
	write(shr_log_unit,*) 'shr_pio_mod: WARNING Bad io_type argument - using iotype_netcdf'
	iotype=pio_iotype_netcdf
	end if
	end subroutine shr_pio_getiotypefromname

... along with their corresponding descriptions below. Note that there are more pio_typename values defined below, but based from the code above only the netcdf* types seem to be supported.

eCLM/src/externals/pio1/pio_types.F90

Lines 304 to 311 in 7853ec4

	PIO_iotype_pbinary = 1, &! use MPI-IO with data types to read/write C like binary files
	PIO_iotype_direct_pbinary = 2, & !use MPI-IO with data types to read/write direct access binary files
	PIO_iotype_binary = 4, & ! serial read/write of binary files using 'base_node'
	PIO_iotype_pnetcdf = 5, & ! parallel read/write of pNetCDF files
	PIO_iotype_netcdf = 6, & ! serial read/write of NetCDF file using 'base_node'
	PIO_iotype_netcdf4c = 7, & ! netcdf4 (hdf5 format) file opened for compression (serial write access only)
	PIO_iotype_netcdf4p = 8, & ! netcdf4 (hdf5 format) file opened in parallel (all netcdf4 files for read will be opened this way)
	PIO_iotype_vdc2 = 10 ! VDC2 format file opened for compressed parallel write

[kvrigor]

Re: I/O strategy in coupled system:

An alternative would be to include asynchronous I/O, with the aim that I/O operations minimizes the impact on the total runtime of the eCLM itself.

A good first step towards this would be to replace the NetCDF APIs for opening and creating files with their parallel counterparts, i.e. nc_open_par() and nc_create_par(), in src/externals/pio1/ionf_mod.F90. This requires NetCDF to be built with parallel I/O support. The advantage of this is we eliminate implementing async I/O routines on our own and instead leverage the native parallel I/O functionality that already exists in the NetCDF library.