CDO

Hi Laura,
You can change truncation with the operator sp2sp,<truncation>:

cdo sp2sp,<truncation> -gp2sp infile outfile

cdo sp2gp -sp2sp,<truncation> -gp2sp infile outfile

cdo sp2gp -sp2sp,<oldTruncation> -sp2sp,<newTruncation> -gp2sp infile outfile

Hi Uwe,
Thank you for your answer. So there is no way to do it in just one step to make it faster?
Cheers, Laura

Hi Laura,
You can make it faster user more threads. Here is an example with 8 threads:

cdo -P 8 sp2sp,<truncation> -gp2sp infile outfile

cdo -P 8 gp2sp,trunc=<truncation> infile outfile

Hi Uwe,
Cool! Thank you!
Cheers,
Laura

Hello Uwe & Laura,
I am trying out the new version on Levante as follows, where the goal is to return to the original resolution:

cdo=/home/m/m214003/local/bin/cdo
nlat=512
trunc=$(( ( (nlat-100)*4)-1 )/2 )) #(some desired low pass filter truncation)
trunc_res= $(( (nlat*4-1)/2 ))     #(original resolution linear truncation)
$cdo -P 12 gp2sp,trunc=${trunc} infile tmp
$cdo sp2sp,${trunc_res} tmp tmp2
$cdo -P 12 sp2gp,linear tmp2 outfile     
rm ${outdir}/tmp*.grb

cdo    sp2sp (Abort): Too many parameters

When trying the code below interactively, I get prompted to enter the truncation for `sp2sp` and once I do, it completes the task.

[b381952@levante5 scripts]$  $cdo -P 12 sp2gp,linear -sp2sp,${trunc_res} -gp2sp,trunc=${trunc} ${infile} ${outfile}
cdo(1) sp2sp: Process started
cdo(2) gp2sp: Process started
cdo(2) gp2sp: Allocate 1 array to hold the Legendre polynoms: size=4307558400 Bytes
cdo(1) sp2sp     : Enter truncation > 1023
cdo    sp2gp: Allocate 1 array to hold the Legendre polynoms: size=4299161600 Bytes
cdo(2) gp2sp: Processed 645922816 values from 1 variable over 4 timesteps
cdo(1) sp2sp: Processed 323908200 values from 1 variable over 4 timesteps
cdo    sp2gp: Processed 323276800 values from 1 variable over 4 timesteps [75.91s 8400MB]

Am I misunderstanding the syntax? Or is it a more grievous error?

I get a syntax error running your calls in bash:

err.sh: line 2: syntax error near unexpected token `)'
err.sh: line 2: `trunc=$(( ( (nlat-100)*4)-1 )/2 )) #(some desired low pass filter truncation)'

anyway I think there is something from in your scripting and I dont have your input file. if possible provide the location on levante.

i cleanup your script a bit, you might give it a try:

nlat=512  
((trunc = (((nlat-100)*4)-1)/2))                                                                                                                               
((trunc_res = ((nlat*4-1)/2) ))

echo "trunc = $trunc" 
echo "trunc_res = $trunc_res" 

echo "cdo -P 12 gp2sp,trunc=${trunc} infile tmp" 
cdo gp2sp,trunc=${trunc} infile tmp
cdo sp2sp,${trunc_res} tmp tmp2
echo "cdo sp2sp,${trunc_res} tmp tmp2" 
cdo sp2sp,${trunc_res} tmp tmp2

cdo sp2gp,linear tmp2 outfile

Ah I edited the code a bit to make it readable and probably introduced an error. Here is the location of the script

/home/b/b381952/icon_for_ml/scripts/profile_filter.sh

Is it possible to do the second option (the one liner) without being prompted for the truncation for `sp2sp`?

This code also fails with the same problem as before:(cdo sp2sp (Abort): Too many parameters)

module purge
cdo=/home/m/m214003/local/bin/cdo
# ARGUMENTS
nlat=512   # Choose from {powers of 2's}. This is the number of latitudes between a pole and the equator
ggrid=n${nlat}
remaptype="nn" # Choose from {"nn","dis"}
OGfile="datawave_nwp_20171027_r2b10_atm_3d_u_ML_20171031T000000Z.grb" 

indir="/work/bm1233/icon_for_ml/output_files/remapped/u/"${ggrid}
infile=${remaptype}_${OGfile}
outdir="/work/bm1233/icon_for_ml/output_files/filtered/u/"${ggrid}
$cdo -P 12 gp2sp,trunc=500 ${indir}/${infile} ${outdir}/tmp.grb
$cdo sp2sp,1023 ${outdir}/tmp.grb ${outdir}/tmp2.grb

However if I replace the last line with

$cdo sp2sp ${outdir}/tmp.grb ${outdir}/tmp2.grb

cdo    sp2sp     : Enter truncation > 

cdo    sp2sp: Processed 77462616 values from 1 variable over 4 timesteps [21.22s 86MB]

But there is a fix! I think it's the version of cdo I'm using that's the problem-- the version I'm using is that one shared by Uwe that includes the `gp2sp` update with truncation directly available. If I use the module cdo then it is successful.

[b381952@l10321 scripts]$ cdo=0
[b381952@l10321 scripts]$ module load cdo
c[b381952@l10321 scripts]$ cdo -V
Climate Data Operators version 2.0.6 (https://mpimet.mpg.de/cdo)
System: x86_64-pc-linux-gnu
CXX Compiler: /sw/spack-workplace/spack/lib/spack/env/gcc/g++ -g -O2 -fopenmp -pthread
CXX version : g++ (Spack GCC) 11.2.0
C Compiler: /sw/spack-workplace/spack/lib/spack/env/gcc/gcc -g -O2 -fopenmp -pthread -pthread
C version : gcc (Spack GCC) 11.2.0
F77 Compiler: /sw/spack-workplace/spack/lib/spack/env/gcc/gfortran -g -O2
F77 version : GNU Fortran (Spack GCC) 11.2.0
Features: 251GB 256threads c++17 OpenMP45 Fortran pthreads HDF5 NC4/HDF5/threadsafe OPeNDAP sz udunits2 proj xml2 magics fftw3 avx2
Libraries: yac/2.4.2 HDF5/1.12.1 proj/8.1.0 xml2/2.9.7 magics/4.9.3
CDI data types: SizeType=size_t
CDI file types: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5 nczarr 
     CDI library version : 2.0.6
 cgribex library version : 2.0.2
 ecCodes library version : 2.21.0
  NetCDF library version : 4.8.1 of Feb 17 2022 15:36:58 $
    HDF5 library version : 1.12.1 threadsafe
    exse library version : 1.4.2
    FILE library version : 1.9.1

[b381952@l10321 scripts]$ cdo sp2sp,${trunc_res} ${outdir}/tmp.grb ${outdir}/tmp2.grb
cdo    sp2sp: Processed 111064800 values from 1 variable over 4 timesteps [3.89s 87MB]

Unfortunately an error occurred, we will solve the problem in the next CDO version.
Here is a workaround:

$cdo sp2sp ${outdir}/tmp.grb ${outdir}/tmp2.grb << EOR
1023
EOR

By the way, would it be possible to do a similar thing to sp2gp? That is, given a spectral coeff file with truncation T_low, apply sp2gp with a desired truncation T_high that is larger than T_low.

This feature will be available in the next CDO release 2.2.0.

Thank you very much!