software:wrf:caviness.new

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
software:wrf:caviness.new [2024-04-04 13:10] – freysoftware:wrf:caviness.new [2024-04-05 12:22] (current) – frey
Line 1: Line 1:
 +====== Installation of WRF/WPS ======
  
 +This guide is loosely based on the information presented in the [[https://www2.mmm.ucar.edu/wrf/OnLineTutorial/Compile/index.php|WRF-ARW Online Tutorial]].  WRF expects the MPICH library for distributed-memory parallelism, but IT RCI recommends Open MPI on Caviness, so some minor adjustments are necessary.  A major difference will be in how the software is organized and used.
 +
 +As outlined in the [[technical:recipes:software-managment|software building and management guide]], WRF (and accompanying WPS) will be installed in a versioned hierarchy.  The base path for the installation will vary by user and use case:
 +
 +  * Versions managed by a singular user for that user's private usage
 +    * Home directory, e.g. ''${HOME}/sw/wrf''
 +    * Workgroup per-user directory, e.g. ''${WORKDIR}/users/«user»/sw/wrf''
 +  * Versions managed for all members of a workgroup
 +    * Workgroup software directory, e.g. ''${WORKDIR}/sw/wrf''
 +
 +In this document the software will be installed for all members of a workgroup.  Though the procedure assumes WRF was not previously installed for the workgroup, it has been constructed to work regardless.  Procedural differences based on previous installations will be highlighted where appropriate.
 +
 +===== Prepare Installation Path =====
 +
 +To begin, the workgroup is entered and the installation directories are created.  In this document version 4.5.2 will be built and installed:
 +
 +<code bash>
 +$ workgroup -g «workgroup»
 +$ WRF_PREFIX="${WORKDIR}/sw/wrf"
 +$ WRF_VERSION="4.5.2"
 +$ mkdir -p "${WRF_PREFIX}/${WRF_VERSION}/src"
 +$ BUILD_DATE="$(date +%Y.%m.%d)"
 +</code>
 +
 +Note that depending on the choice of base path (as outlined above) the value of ''WRF_PREFIX'' will be different.  The ''BUILD_DATE'' will be used to differentiate some paths and filenames throughout this document.  The ''WRF_VERSION'' corresponds with WRF itself; it is up to the user to determine the correct version of WPS to accompany the chosen version of WRF.
 +
 +===== Build Environment =====
 +
 +WRF depends on several external libraries, including the NetCDF and NetCDF-Fortran libraries.  Most of the other libraries are available in the OS.
 +
 +<code bash>
 +$ vpkg_devrequire netcdf-fortran/4.5.4:intel-oneapi-2023,openmpi
 +</code>
 +
 +The PNG library must be at least 1.2.50:
 +
 +<code bash>
 +$ pkg-config libpng --modversion
 +1.5.13
 +</code>
 +
 +The Jasper library must be at least 1.900.1:
 +
 +<code bash>
 +$ pkg-config jasper --modversion
 +1.900.1
 +</code>
 +
 +The zip library (zlib) dependency isn't directly needed by WRF but by the NetCDF library to which it is linked.  Since NetCDF is **not** being purpose-built for this copy of WRF (a VALET package is being used), the zip library is not needed.
 +
 +For the sake of dependencies and build system tests, a directory will be created to the purpose:
 +
 +<code bash>
 +$ mkdir -p "${WRF_PREFIX}/${WRF_VERSION}/src/3rd_party"
 +</code>
 +
 +===== Build Dependencies =====
 +
 +Any dependencies would be downloaded and built in the ''3rd_party'' directory and be installed to the WRF version directory.
 +
 +<WRAP center round alert 60%>
 +The build of the zip library is shown here for illustrative purposes only -- it does **not** need to be built as part of this procedure.
 +</WRAP>
 +
 +<code bash>
 +$ pushd "${WRF_PREFIX}/${WRF_VERSION}/src/3rd_party"
 +$ wget 'https://www.zlib.net/zlib-1.3.1.tar.gz'
 +$ tar -xf zlib-1.3.1.tar.gz
 +$ pushd zlib-1.3.1
 +$ vpkg_require cmake/default
 +$ mkdir build-${BUILD_DATE}
 +$ cd build-${BUILD_DATE}
 +$ CC=icx \
 +    cmake \
 +        -DCMAKE_BUILD_TYPE=Release \
 +        -DCMAKE_INSTALL_PREFIX="${WRF_PREFIX}/${WRF_VERSION}" \
 +        ..
 +$ make
 +$ make install
 +$ popd
 +</code>
 +
 +===== Build System Tests =====
 +
 +WRF has two tests to ensure the build system is likely to succees.  In the ''3rd_party'' directory:
 +
 +<code bash>
 +$ cd "${WRF_PREFIX}/${WRF_VERSION}/src/3rd_party"
 +$ wget 'https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/Fortran_C_NETCDF_MPI_tests.tar'
 +$ tar -xf Fortran_C_NETCDF_MPI_tests.tar
 +</code>
 +
 +==== Test 1:  Serial ====
 +
 +<code bash>
 +$ ${FC} ${CPPFLAGS} ${FCFLAGS} -c 01_fortran+c+netcdf_f.f
 +$ ${CC} ${CPPFLAGS} ${CFLAGS} -c 01_fortran+c+netcdf_c.c
 +$ ${FC} ${CPPFLAGS} ${FCFLAGS} -o 01_fortran+c+netcdf \
 +            01_fortran+c+netcdf_f.o 01_fortran+c+netcdf_c.o \
 +            ${LDFLAGS} -lnetcdff -lnetcdf
 +$ ./01_fortran+c+netcdf
 +   C function called by Fortran
 +   Values are xx =  2.00 and ii = 1 
 + SUCCESS test 1 fortran + c + netcdf
 +</code>
 +
 +==== Test 2:  MPI Parallelism ====
 +
 +<code bash>
 +$ mpif90 ${CPPFLAGS} ${FCFLAGS} -c 02_fortran+c+netcdf+mpi_f.f
 +$ mpicc ${CPPFLAGS} ${CFLAGS} -c 02_fortran+c+netcdf+mpi_c.c
 +$ mpif90 ${CPPFLAGS} ${FCFLAGS} -o 02_fortran+c+netcdf+mpi \
 +            02_fortran+c+netcdf+mpi_f.o 02_fortran+c+netcdf+mpi_c.o \
 +            ${LDFLAGS} -lnetcdff -lnetcdf
 +$ mpirun -np 1 02_fortran+c+netcdf+mpi
 +   C function called by Fortran
 +   Values are xx =  2.00 and ii = 1 
 + status =            2
 + SUCCESS test 2 fortran + c + netcdf + mpi
 +</code>
 +
 +===== Download Main Sources =====
 +
 +The WRF and WPS source code will be cloned from Github in the versioned source directory and the desired version checked-out:
 +
 +<code bash>
 +$ cd "${WRF_PREFIX}/${WRF_VERSION}/src"
 +
 +$ git clone https://github.com/wrf-model/WRF.git WRF
 +$ pushd WRF
 +$ git checkout v4.5.2
 +$ git submodule update --init --recursive
 +$ popd
 +
 +$ git clone https://github.com/wrf-model/WPS.git WPS
 +$ pushd WPS
 +$ git checkout v4.5
 +$ popd
 +</code>
 +
 +The clone and checkout process will take some time -- these are large source code repositories.
 +
 +<WRAP center round tip 60%>
 +For other 4.5.x and newer releases of WRF, the value assigned to ''WRF_VERSION'' should be altered and the versions checked-out in the git repositories will be different.
 +</WRAP>
 +
 +===== Build WRF =====
 +
 +For the Jasper library to be included in the program, the WRF configuration system **must** be altered:
 +
 +<code bash>
 +$ cd "${WRF_PREFIX}/${WRF_VERSION}/src/WRF"
 +$ sed -i 's/I_really_want_to_output_grib2_from_WRF = "FALSE"/I_really_want_to_output_grib2_from_WRF = "TRUE"/' arch/Config.pl
 +</code>
 +
 +Two builds will be done:  distributed-memory parallelism (MPI) and serial.
 +
 +==== Distributed-Memory Parallel ====
 +
 +For the MPI build, the ''configure'' script must be run with the following selections:
 +
 +  * Option 78:  Intel (ifx/icx), "dm"
 +  * 1:  Basic nesting
 +
 +<code bash>
 +$ ./clean -aa
 +$ NETCDF="${NETCDF_DASH_FORTRAN_PREFIX}" \
 +  JASPERLIB=/usr/lib64 JASPERINC=/usr/include ZLIB="${WRF_PREFIX}/${WRF_VERSION}" \
 +    ./configure
 +</code>
 +
 +The ''configure'' script produces a ''configure.wrf'' file for the build system.  A few minor modifications are necessary to the compilers and for the sake of brevity (no need to ''time'' the Fortran compiles):
 +
 +<code bash>
 +$ sed -i -e 's/^\(DM_FC *= *\)mpi.*$/\1mpif90/' \
 +         -e 's/^\(DM_CC *= *\)mpi.*$/\1mpicc/' \
 +         -e 's/^\(FC *= *\)time /\1/' \
 +         configure.wrf
 +</code>
 +
 +Finally, the code can be compiled.  The ''-j 4'' flag allows up to four concurrent compiles to accelerate the process:
 +
 +<code bash>
 +$ NETCDF="${NETCDF_DASH_FORTRAN_PREFIX}" \
 +  JASPERLIB=/usr/lib64 JASPERINC=/usr/include \
 +    ./compile -j 4 em_real 2>&1 | tee compile-mpi-${BUILD_DATE}.log
 +</code>
 +
 +Again, a lot of information will be displayed to the terminal as the build proceeds; all of it will also be written to a file with the name ''compile-mpi-«YYYY.MM.DD».log''.  If the build is successful, the output will terminate with e.g.
 +
 +<code>
 +==========================================================================
 +build started:   Tue Apr  2 16:39:04 EDT 2024
 +build completed: Tue Apr 2 16:53:00 EDT 2024
 + 
 +--->                  Executables successfully built                  <---
 + 
 +-rwxr-xr-x 1 user workgroup 67864392 Apr  2 16:53 main/ndown.exe
 +-rwxr-xr-x 1 user workgroup 67779464 Apr  2 16:53 main/real.exe
 +-rwxr-xr-x 1 user workgroup 67344608 Apr  2 16:53 main/tc.exe
 +-rwxr-xr-x 1 user workgroup 71494672 Apr  2 16:52 main/wrf.exe
 + 
 +==========================================================================
 +</code>
 +
 +At this point those executables can be installed outside the source directory to ''${WRF_PREFIX}/${WRF_VERSION}/bin'' for future computational usage:
 +
 +<code bash>
 +$ mkdir -p "${WRF_PREFIX}/${WRF_VERSION}/bin"
 +$ for exe in main/*.exe; do
 +    install --mode=0775 "$exe" "${WRF_PREFIX}/${WRF_VERSION}/bin/mpi_$(basename "${exe}")"
 +  done
 +</code>
 +
 +In the future, the distributed-memory parallel variants of the program will be referenced as ''mpi_wrf.exe'', ''mpi_real.exe'', etc.
 +
 +=== Building WPS ===
 +
 +Since the WPS build makes use of the headers and intermediate libraries produced by the WRF build, at this point the user would need to skip to the section on building WPS and choose the same "dm" variant used to build WRF.  However, in this document **only the serial WPS** will be built, so that step is skipped.
 +
 +==== Serial ====
 +
 +For the serial build, the ''configure'' script must be run with the following selections:
 +
 +  * Option 76:  Intel (ifx/icx) "serial"
 +  * 0:  No nesting
 +
 +<code bash>
 +$ ./clean -aa
 +$ NETCDF="${NETCDF_DASH_FORTRAN_PREFIX}" \
 +  JASPERLIB=/usr/lib64 JASPERINC=/usr/include ZLIB="${WRF_PREFIX}/${WRF_VERSION}" \
 +    ./configure
 +</code>
 +
 +The ''configure'' script produces a ''configure.wrf'' file for the build system.  A few minor modifications are necessary to the compilers and for the sake of brevity (no need to ''time'' the Fortran compiles):
 +
 +<code bash>
 +$ sed -i -e 's/^\(DM_FC *= *\)mpi.*$/\1mpif90/' \
 +         -e 's/^\(DM_CC *= *\)mpi.*$/\1mpicc/' \
 +         -e 's/^\(FC *= *\)time /\1/' \
 +         configure.wrf
 +</code>
 +
 +Finally, the code can be compiled.  The ''-j 4'' flag allows up to four concurrent compiles to accelerate the process:
 +
 +<code bash>
 +$ NETCDF="${NETCDF_DASH_FORTRAN_PREFIX}" \
 +  JASPERLIB=/usr/lib64 JASPERINC=/usr/include \
 +    ./compile -j 4 em_real 2>&1 | tee compile-serial-${BUILD_DATE}.log
 +</code>
 +
 +A lot of information will be displayed to the terminal as the build proceeds; all of it will also be written to a file with the name ''compile-serial-«YYYY.MM.DD».log''.  If the build is successful, the output will terminate with e.g.
 +
 +<code>
 +==========================================================================
 +build started:   Tue Apr  2 16:19:28 EDT 2024
 +build completed: Tue Apr 2 16:33:37 EDT 2024
 + 
 +--->                  Executables successfully built                  <---
 + 
 +-rwxr-xr-x 1 user workgroup 67864392 Apr  2 16:33 main/ndown.exe
 +-rwxr-xr-x 1 user workgroup 67779464 Apr  2 16:33 main/real.exe
 +-rwxr-xr-x 1 user workgroup 67344608 Apr  2 16:33 main/tc.exe
 +-rwxr-xr-x 1 user workgroup 71494672 Apr  2 16:32 main/wrf.exe
 + 
 +==========================================================================
 +</code>
 +
 +At this point those executables can be installed outside the source directory to ''${WRF_PREFIX}/${WRF_VERSION}/bin'' for future computational usage:
 +
 +<code bash>
 +$ mkdir -p "${WRF_PREFIX}/${WRF_VERSION}/bin"
 +$ for exe in main/*.exe; do
 +    install --mode=0775 "$exe" "${WRF_PREFIX}/${WRF_VERSION}/bin"
 +  done
 +</code>
 +
 +In the future, the serial variants of the program will be referenced as ''wrf.exe'', ''real.exe'', etc.
 +
 +In the next section WPS will be built, which will make use of the headers and intermediate libraries produced in this step.  **Do not alter the WRF source directory!**  E.g. **do not** remove build artifacts with ''./clean''.
 +
 +===== Build WPS =====
 +
 +The WRF 4.5.2 code has been updated to include support for the Clang-based Intel oneAPI compiler suite -- the configure options selected in the previous section reflected that.  However, WPS 4.5 does not offer those options.  The build system must be patched to add that support.
 +
 +Download the {{:software:wrf:intel-oneapi.patch|patch file}} to the WPS source directory and apply the patch:
 +
 +<code bash>
 +$ cd "${WRF_PREFIX}/${WRF_VERSION}/src/WPS"
 +$ wget 'https://docs.hpc.udel.edu/_media/software/wrf/intel-oneapi.patch'
 +$ patch -d arch -p1 < intel-oneapi.patch 
 +patching file configure.defaults
 +</code>
 +
 +Only the serial variant of WPS will be built.  Run the ''configure'' script with the following selection:
 +
 +  * Option 41: Intel oneAPI "serial"
 +
 +<code bash>
 +$ NETCDF="${NETCDF_DASH_FORTRAN_PREFIX}" \
 +  JASPERLIB="/usr/lib64" JASPERINC="/usr/include" \
 +  WRF_DIR="${WRF_PREFIX}/${WRF_VERSION}/src/WRF" \
 +    ./configure
 +</code>
 +
 +The ''WRF_DIR'' provides the WPS build system with the path to the WRF headers and intermediate libraries produced in the previous step.
 +
 +WRF made use of and linked against the NetCDF-Fortran library, but the WPS configure system does not link against that library.  The ''configure.wrf'' file generated by ''./configure'' must be adjusted accordingly:
 +
 +<code bash>
 +$ sed -i 's/-lnetcdf/-lnetcdff -lnetcdf/g' configure.wps
 +</code>
 +
 +At this point WPS can be built:
 +
 +<code bash>
 +$ NETCDF="${NETCDF_DASH_FORTRAN_PREFIX}" \
 +  JASPERLIB=/usr/lib64 JASPERINC=/usr/include \
 +  WRF_DIR="${WRF_PREFIX}/${WRF_VERSION}/src/wrf" \
 +    ./compile 2>&1 | tee compile-serial-${BUILD_DATE}.log
 +</code>
 +
 +If successful, the executables are installed to the ''${WRF_PREFIX}/${WRF_VERSION}/bin'' directory for future computational usage:
 +
 +<code bash>
 +$ find . -name \*.exe -type f -exec install --mode=0775 \{\} "${WRF_PREFIX}/${WRF_VERSION}/bin" \; -print
 +./ungrib/src/g1print.exe
 +./ungrib/src/ungrib.exe
 +./ungrib/src/g2print.exe
 +./metgrid/src/metgrid.exe
 +./util/src/rd_intermediate.exe
 +./util/src/height_ukmo.exe
 +./util/src/avg_tsfc.exe
 +./util/src/int2nc.exe
 +./util/src/mod_levs.exe
 +./util/src/calc_ecmwf_p.exe
 +./geogrid/src/geogrid.exe
 +</code>
 +
 +The ''link_grib.csh'' script is referenced in documentation as always being run from the source directory.  But it functions just fine when installed to the same ''bin'' directory as the executables:
 +
 +<code bash>
 +$ install --mode=0775 link_grib.csh "${WRF_PREFIX}/${WRF_VERSION}/bin"
 +</code>
 +
 +==== Distributed-Memory Parallel ====
 +
 +As with the "dm" WRF executables, the executables associated with a "dm" build of WPS should be installed with the ''mpi_'' prefix.
 +
 +<WRAP center round alert 60%>
 +The following is **only to be substituted for a "dm" WPS build.**  Do not execute it in concert with the code above!
 +</WRAP>
 +
 +<code bash>
 +$ for exe in $(find . -name \*.exe -type f); do
 +    echo "$exe"
 +    install --mode=0775 "$exe" "${WRF_PREFIX}/${WRF_VERSION}/bin/mpi_$(basename "${exe}")"
 +  done
 +./ungrib/src/g1print.exe
 +./ungrib/src/ungrib.exe
 +./ungrib/src/g2print.exe
 +./metgrid/src/metgrid.exe
 +./util/src/rd_intermediate.exe
 +./util/src/height_ukmo.exe
 +./util/src/avg_tsfc.exe
 +./util/src/int2nc.exe
 +./util/src/mod_levs.exe
 +./util/src/calc_ecmwf_p.exe
 +./geogrid/src/geogrid.exe
 +</code>
 +
 +The ''link_grib.csh'' script only needs to be installed once.
 +
 +===== VALET Package Definition =====
 +
 +Creating a VALET package to describe the WRF software makes it easy to dynamically configure the runtime environment for computational jobs.  The location to which the package definition file is installed depends on the base installation path chosen for WRF:
 +
 +  * Versions managed by a singular user for that user's private usage
 +    * ''WRF_VALET_DIR="${HOME}/.valet"''
 +  * Versions managed for all members of a workgroup
 +    * ''WRF_VALET_DIR="${WORKDIR}/sw/valet"''
 +
 +Three variable quantities are needed when authoring the VALET package definition for the WRF software built above.  The first two are the base installation directory and the version id:
 +
 +<code bash>
 +$ printf "%s\n%s\n" "$WRF_PREFIX" "$WRF_VERSION"
 +/work/it_nss/sw/wrf
 +4.5.2
 +</code>
 +
 +The third variable is the VALET package that was added to facilitate the build:
 +
 +  * ''netcdf-fortran/4.5.4:intel-oneapi-2023,openmpi''
 +
 +If no VALET package definition file exists, create the file with the following template (filling-in the values indicated by double angle brackets):
 +
 +<file yaml wrf.vpkg_yaml>
 +wrf:
 +    prefix: «WRF_PREFIX»
 +    description: the Weather Research and Forecasting model
 +    url: "https://github.com/wrf-model/WRF"
 +
 +    versions:
 +        "«WRF_VERSION»":
 +            description: WRF 4.5.2 and WPS 4.5, Intel oneAPI, Open MPI
 +            dependencies:
 +                - netcdf-fortran/4.5.4:intel-oneapi-2023,openmpi
 +            actions:
 +                - variable: WRF_SRC_DIR
 +                  value: ${VALET_PATH_PREFIX}/src/WRF
 +                - variable: WPS_SRC_DIR
 +                  value: ${VALET_PATH_PREFIX}/src/WPS
 + 
 +</file>
 +
 +This template can be downloaded directly to the ''WRF_VALET_DIR'' directory chosen above and then edited:
 +
 +<code bash>
 +$ wget -O "${WRF_VALET_DIR}/wrf.vpkg_yaml" \
 +    'https://docs.hpc.udel.edu/_export/code/software/wrf/caviness.new?codeblock=29'
 +</code>
 +
 +If a ''wrf.vpkg_yaml'' package definition already exists, simply edit the file and add the version section alone:
 +
 +<code yaml>
 +        "«WRF_VERSION»":
 +            description: WRF 4.5.2 and WPS 4.5, Intel oneAPI, Open MPI
 +            dependencies:
 +                - netcdf-fortran/4.5.4:intel-oneapi-2023,openmpi
 +            actions:
 +                - variable: WRF_SRC_DIR
 +                  value: ${VALET_PATH_PREFIX}/src/WRF
 +                - variable: WPS_SRC_DIR
 +                  value: ${VALET_PATH_PREFIX}/src/WPS
 + 
 + 
 +</code>
 +
 +==== Using the Package ====
 +
 +The package is added to the environment:
 +
 +<code bash>
 +$ vpkg_require wrf/4.5.2
 +Adding dependency `szip/2.1.1` to your environment
 +Adding dependency `hdf4/4.2.16` to your environment
 +Adding dependency `binutils/2.35.1` to your environment
 +Adding dependency `gcc/12.2.0` to your environment
 +Adding dependency `intel-oneapi/2023.0.0.25537` to your environment
 +Adding dependency `ucx/1.13.1` to your environment
 +Adding dependency `openmpi/4.1.5:intel-oneapi-2023` to your environment
 +Adding dependency `hdf5/1.10.9:intel-oneapi-2023,openmpi` to your environment
 +Adding dependency `pnetcdf/1.12.3:intel-oneapi-2023,openmpi` to your environment
 +Adding dependency `netcdf/4.9.1:intel-oneapi-2023,openmpi` to your environment
 +Adding dependency `netcdf-fortran/4.5.4:intel-oneapi-2023,openmpi` to your environment
 +Adding package `wrf/4.5.2` to your environment
 +</code>
 +
 +The executables are now available on the shell's path and can be executed as bare commands, e.g. ''real.exe'':
 +
 +<code bash>
 +$ which real.exe
 +/work/it_nss/sw/wrf/4.5.2/bin/real.exe
 +
 +$ which mpi_real.exe
 +/work/it_nss/sw/wrf/4.5.2/bin/mpi_real.exe
 +
 +$ real.exe
 +forrtl: No such file or directory
 +forrtl: severe (29): file not found, unit 27, file /home/1001/namelist.input
 +</code>
 +
 +In addition, two variables are set pointing to the respective source directories of WRF and WPS:
 +
 +<code bash>
 +$ ls -ld $WRF_SRC_DIR
 +drwxr-sr-x 22 user workgroup 41472 Apr  4 11:31 /work/it_nss/sw/wrf/4.5.2/src/WRF
 +
 +$ ls -ld $WPS_SRC_DIR
 +drwxr-sr-x 9 user workgroup 41472 Apr  4 13:00 /work/it_nss/sw/wrf/4.5.2/src/WPS
 +</code>