CDO

Without knowing the details, I suggest a combination of sellonlatbox and fldmean:

cdo -fldmean -sellonlatbox,0,90,-20,20 <ifile> <ofile>

Thanks for the help - in theory that should work fine, and there should be no difference between these two graphs.

The left shows the method above - chaining 'fldmean' and 'sellonlatbox' operators, which averages all values that fall within the lat/long coordinates supplied.

For the right, I just downloaded the data within the same coordinates, and used the 'fldmean' operator, which gave a more realistic set of values.

I have no idea why this is the case, just thought I'd bring it up as I'm not sure chaining the two worked on this data.

Thanks

Apologies - I'm an idiot

The above code did indeed work, it's just an old copy of the CDO guide (2009) was telling me to input the longitude the wrong way round - west,east,north,south worked for the lat/long coordinates of the 'sellonlatbox' operator.

Weirdly, the two ways of doing it are still different:

> all.equal(var.mat,var.mat1)
[1] "Component “SIC”: Mean relative difference: 0.02355555" 

I'll look into this further, it may be that the two methods of retrieving the lat/long box are different; hopefully chaining the operators has no effect on the results.

Thanks again

Hi,

I am trying to do spatial total for a specific region using the maskregion operator. But is doesnt seem to be working:

home$ cdo -fldsum -yearsum -maskregion,bndry_mountain.txt pr_NPL-44i_MPI-M-MPI-ESM-LR_rcp85_r1i1p1_MPI-CSC-REMO2009_v1_day_2006-2099.nc test_fldsum_yrsum.nc
cdo fldsum: Started child process "yearsum -maskregion,bndry_mountain.txt pr_NPL-44i_MPI-M-MPI-ESM-LR_rcp85_r1i1p1_MPI-CSC-REMO2009_v1_day_2006-2099.nc (pipe1.1)".
cdo(2) yearsum: Started child process "maskregion,bndry_mountain.txt pr_NPL-44i_MPI-M-MPI-ESM-LR_rcp85_r1i1p1_MPI-CSC-REMO2009_v1_day_2006-2099.nc (pipe2.1)".
Segmentation fault: 11


I am able to perform the same operations individually:


home$ cdo maskregion,bndry_mountain.txt pr_NPL-44i_MPI-M-MPI-ESM-LR_rcp85_r1i1p1_MPI-CSC-REMO2009_v1_day_2006-2099.nc pr_masked_tmp.nc
cdo maskregion: Processed 8239920 values from 1 variable over 34333 timesteps ( 11.62s )
home$ cdo yearsum pr_masked_tmp.nc pr_yrsum.nc
cdo yearsum: Processed 8239920 values from 1 variable over 34333 timesteps ( 5.54s )
home$ cdo fldsum pr_yrsum.nc pr_fldsum_yrsum.nc
cdo fldsum: Processed 22560 values from 1 variable over 94 timesteps ( 0.05s )
home$ cdo outputtab,date,value pr_fldsum_yrsum.nc>mountain_yrsum.txt
cdo outputtab: Processed 94 values from 1 variable over 94 timesteps ( 0.04s )


What I am doing wrong in the operator chaining?

Thanks for the help!

Hi Sanita,

the chaining is correct and I can't reproduce the segmentation fault with my daily test data and CDO version 1.9.0.
Which CDO version are you using? Please, send us the result of

cdo -V

-Karin

Hi Karin,
Below are details of my CDO version. I am running this on a Mac OS X 10.9.5. Attached is the masking coordinates and a subset of the previous file with only 11 years of data that shows the same error.

@
home$ cdo -V
Climate Data Operators version 1.9.0 (http://mpimet.mpg.de/cdo)
Compiled: by sdhauban on MacBook-Pro.local (x86_64-apple-darwin13.4.0) Oct 31 2017 11:34:32
CXX Compiler: clang++ -std=gnu++11 -g -O2 -D_THREAD_SAFE -pthread
CXX version : Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)
C Compiler: clang -g -O2 -D_THREAD_SAFE -pthread
C version : Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)
Features: DATA PTHREADS HDF5 NC4/HDF5 OPeNDAP SZ SSE4_1
Libraries: HDF5/1.10.1
Filetypes: srv ext ieg grb1 nc1 nc2 nc4 nc4c
CDI library version : 1.9.0 of Oct 31 2017 11:33:10
CGRIBEX library version : 1.8.1 of Jul 12 2017 18:11:37
NetCDF library version : 4.4.1.1 of Oct 30 2017 22:18:23 $
HDF5 library version : 1.10.1
SERVICE library version : 1.4.0 of Oct 31 2017 11:33:00
EXTRA library version : 1.4.0 of Oct 31 2017 11:32:55
IEG library version : 1.4.0 of Oct 31 2017 11:32:58
FILE library version : 1.8.3 of Oct 31 2017 11:32:56

home$ cdo -fldsum -yearsum -maskregion,bndry_mountain.txt pr_NPL-44i_MPI-M-MPI-ESM-LR_rcp85_r1i1p1_MPI-CSC-REMO2009_v1_day_2006-2016.nc test_fldsum_yrsum.nc
cdo fldsum: Started child process "yearsum -maskregion,bndry_mountain.txt pr_NPL-44i_MPI-M-MPI-ESM-LR_rcp85_r1i1p1_MPI-CSC-REMO2009_v1_day_2006-2016.nc (pipe1.1)".
cdo(2) yearsum: Started child process "maskregion,bndry_mountain.txt pr_NPL-44i_MPI-M-MPI-ESM-LR_rcp85_r1i1p1_MPI-CSC-REMO2009_v1_day_2006-2016.nc (pipe2.1)".
Segmentation fault: 11

@

Interestingly though, if I add outputtab to the same list of command, i can save the output into a text file as follows. Is this because the fldsum+yearsum generates a one-grid time series that cannot be saved as a .nc file?

Sanitas-MacBook-Pro:tmp_CDOmasking sdhauban$ cdo -outputtab,date,value -fldsum -yearsum -maskregion,bndry_mountain.txt pr_NPL-44i_MPI-M-MPI-ESM-LR_rcp85_r1i1p1_MPI-CSC-REMO2009_v1_day_2006-2016.nc>test_fldsum_yrsum.txt
cdo outputtab: Started child process "fldsum -yearsum -maskregion,bndry_mountain.txt pr_NPL-44i_MPI-M-MPI-ESM-LR_rcp85_r1i1p1_MPI-CSC-REMO2009_v1_day_2006-2016.nc (pipe1.1)".
cdo(2) fldsum: Started child process "yearsum -maskregion,bndry_mountain.txt pr_NPL-44i_MPI-M-MPI-ESM-LR_rcp85_r1i1p1_MPI-CSC-REMO2009_v1_day_2006-2016.nc (pipe2.1)".
cdo(3) yearsum: Started child process "maskregion,bndry_mountain.txt pr_NPL-44i_MPI-M-MPI-ESM-LR_rcp85_r1i1p1_MPI-CSC-REMO2009_v1_day_2006-2016.nc (pipe3.1)".
cdo(4) maskregion: Processed 964320 values from 1 variable over 4018 timesteps ( 0.95s )
cdo(3) yearsum: Processed 964320 values from 1 variable over 4018 timesteps ( 0.95s )
cdo(2) fldsum: Processed 2640 values from 1 variable over 11 timesteps ( 0.95s )
cdo outputtab: Processed 11 values from 1 variable over 11 timesteps ( 0.95s )


Hi Karin,

I have been testing out different combinations for the chaining and it seems that chaining maskregion with any of the yearstat or fldstat operators is not working for me. But for some reason it works if i use the same chaining with sinfo. See below:

@home$ cdo -sinfon -yearsum -maskregion,bndry_mountain.txt pr_NPL-44i_MPI-M-MPI-ESM-LR_rcp85_r1i1p1_MPI-CSC-REMO2009_v1_day_2006-2016.nc
cdo sinfon: Started child process "yearsum -maskregion,bndry_mountain.txt pr_NPL-44i_MPI-M-MPI-ESM-LR_rcp85_r1i1p1_MPI-CSC-REMO2009_v1_day_2006-2016.nc (pipe1.1)".
cdo(2) yearsum: Started child process "maskregion,bndry_mountain.txt pr_NPL-44i_MPI-M-MPI-ESM-LR_rcp85_r1i1p1_MPI-CSC-REMO2009_v1_day_2006-2016.nc (pipe2.1)".
File format : NetCDF4 classic
-1 : Institut Source Steptype Levels Num Points Num Dtype : Parameter name
1 : unknown REMO instant 1 1 240 1 F32 : pr
Grid coordinates :
1 : lonlat : points=240 (20x12)
lon : 79.25 to 88.75 by 0.5 degrees_east
lat : 25.25 to 30.75 by 0.5 degrees_north
Vertical coordinates :
1 : surface : levels=1
Time coordinate : 4018 steps
RefTime = 1949-12-01 00:00:00 Units = days Calendar = proleptic_gregorian Bounds = true
YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss
2006-07-02 12:00:00 2007-07-02 12:00:00 2008-07-02 00:00:00 2009-07-02 12:00:00
2010-07-02 12:00:00 2011-07-02 12:00:00 2012-07-02 00:00:00 2013-07-02 12:00:00
2014-07-02 12:00:00 2015-07-02 12:00:00 2016-07-02 00:00:00
cdo(3) maskregion: Processed 964320 values from 1 variable over 4018 timesteps ( 1.03s )
cdo(2) yearsum: Processed 964320 values from 1 variable over 4018 timesteps ( 1.03s )
cdo sinfon: Processed 1 variable over 11 timesteps ( 1.03s )

home$ cdo -yearsum -maskregion,bndry_mountain.txt pr_NPL-44i_MPI-M-MPI-ESM-LR_rcp85_r1i1p1_MPI-CSC-REMO2009_v1_day_2006-2016.nc textmaskyrsum.nc
cdo yearsum: Started child process "maskregion,bndry_mountain.txt pr_NPL-44i_MPI-M-MPI-ESM-LR_rcp85_r1i1p1_MPI-CSC-REMO2009_v1_day_2006-2016.nc (pipe1.1)".
Segmentation fault: 11
@

Thanks for the detailed information. It seems that your netCDF/HDF installation is not thread-safe. Using non-threadsafe libraries could cause unexpected errors in CDO!
For further information see: https://code.mpimet.mpg.de/projects/cdo/wiki#Errors-with-operator-chaining-and-netCDF4HDF5-files

Hi Uwe,

Thank you for the response. I installed CDO and supporting libraries using homebrew. So I am unable to run the make check as suggested in your link. Do you have any advice on how else I can fix my installation?

For now both my chaining issues above seem to be resolved when I use the -L option in my chaining as suggested in the link you provided.

Thanks so much for the help!
Sanita

Hi,

I am trying to manually compile netcdf from source. I have downloaded the latest version and am trying to follow instruction from: https://www.unidata.ucar.edu/software/thredds/current/netcdf-java/reference/netcdf4Clibrary.html

But when trying out the make install, I get the following warnings. So I think my netCDF download is still not thread safe.


Warning: netcdf: this formula has no --enable-threadsafe option so it will be ignored!
Warning: netcdf: this formula has no --enable-unsupported option so it will be ignored!
Warning: netcdf: this formula has no --prefix= option so it will be ignored!
Warning: netcdf: this formula has no --with-pthread= option so it will be ignored!
Warning: netcdf: this formula has no --with-zlib= option so it will be ignored!

How do I install thread-safe netCDF on my Mac? Previously i used homebrew.

For the site you mentioned: The configure call is for hdf5 - not netcdf

thread-safty of netcdf comes from thread-savfty of unterlying hdf5

Hi Ralf,

Thanks for the response. So the order of my manual installations should be as follows:
- Compile hdf with thread safety
- Compile netcdf
- Compile cdo

I will reinstall everything and try again. Meanwhile does using the "-L" option solve issues created by installations that are not thread safe?

Sanita Dhaubanjar wrote:

Hi Ralf,

Thanks for the response. So the order of my manual installations should be as follows:
- Compile hdf with thread safety
- Compile netcdf
- Compile cdo

looks good to me

I will reinstall everything and try again. Meanwhile does using the "-L" option solve issues created by installations that are not thread safe?

correct

Hi Ralf!

I tried the above order of installations and was able to enable thread safe option. See HDF5 version in cdo -V below. This has also solved the chaining problems I was facing Ralf, I think it would be good if the need for thread-safe HDF-5 installation could be highlighted in the documentation under the installation and operator chaining sections as well. As a new user of CDO and command line operation, I was not familiar with this before. Would have been nice to know so I could have correctly installed HDF5 in the first run itself.

$ cdo -V
Climate Data Operators version 1.9.2 (http://mpimet.mpg.de/cdo)
Compiled: by sdhauban on Sanitas-MacBook-Pro.local (x86_64-apple-darwin13.4.0) Jan 24 2018 21:59:08
CXX Compiler: g++ -std=gnu++11 -g -O2 -D_THREAD_SAFE -pthread
CXX version : Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)
C Compiler: gcc -g -O2 -D_THREAD_SAFE -pthread
C version : Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)
F77 Compiler: gfortran -g -O2
F77 version : GNU Fortran (Homebrew GCC 7.2.0_1) 7.2.0
Features: 8GB Fortran DATA PTHREADS HDF5 NC4/HDF5/threadsafe OPeNDAP SSE3
Libraries: HDF5/1.10.1
Filetypes: srv ext ieg grb1 nc1 nc2 nc4 nc4c nc5
CDI library version : 1.9.2 of Jan 24 2018 21:55:14
CGRIBEX library version : 1.9.0 of Sep 29 2017 10:16:02
NetCDF library version : 4.5.0 of Jan 24 2018 21:13:51 $
HDF5 library version : 1.10.1 threadsafe
SERVICE library version : 1.4.0 of Jan 24 2018 21:51:08
EXTRA library version : 1.4.0 of Jan 24 2018 21:51:02
IEG library version : 1.4.0 of Jan 24 2018 21:51:06
FILE library version : 1.8.3 of Jan 24 2018 21:51:03

Just a note to other new users of homebrew in mac, to run manual compilations and maintain connectivity with brew use the diy option. For example, my manual compilation of hdf5 with threadsafe enabled looked like:
$ ./configure
$ brew diy
--prefix=/usr/local/Cellar/hdf5/1.10.1
$ ./configure --prefix=/usr/local/Cellar/hdf5/1.10.1 --with-zlib=${ZDIR} --enable-threadsafe --with-pthread=${PDIR} --enable-unsupported
$ make
$ make check
$ make install
$ make check-install
$ brew link hdf5


Thanks a ton Ralf and your team for the prompt and helpful feedbacks.

Hi Sanita,

Thanks so much for your instructions! Worked fine for me after one alteration -- when I attempted to configure with both the "threadsafe" and MPI (pthread) options as you did, I would get random file read failures and CDO "Unknown Error"s. Compiling with both of these features is not supported by the HDF5 group (see this discussion) and in my case I guess things went badly.

Instead I used

./configure --prefix=/usr/local/Cellar/hdf5/1.10.4 --with-zlib=${ZDIR} --disable-hl --enable-threadsafe

1.10.4 being the latest version as of December 2018. Note --disable-hl is required without the --enable-unsupported option.

Another note: I did not need to directly compile the NetCDF-C libraries. It turned out I could just install those with

brew install netcdf

like normal, and I was good to go.