Project

General

Profile

Forums » Support »

Selecting a single variable from multiple netcdfs (that have differing numbers of variables) and merging via mergetime or cat...

Added by chris haak over 3 years ago

First off I'd like to apologize if this is a duplicate question however I have sifted through the forum for hours on end and still not found a way to do this (despite seeing many similar questions!)

I have a sequence of netcdfs with the same grid size but differing numbers of variables. They do, however, all share a single variable of interest in common.

I would simply like to do a mergetime on these files, selecting only the common variable and "ignoring" the rest.

Both "mergetime" and "cat" both give me the "input streams have differing numbers of variables" error, as does using "select";

This seems like it should be a relatively straightforward operation to do in one shot but I cannot for the life of me figure it out.

Thanks in advance for any help you can offer....


Replies (13)

RE: Selecting a single variable from multiple netcdfs (that have differing numbers of variables) and merging via mergetime or cat... - Added by Karin Meier-Fleischer over 3 years ago

Hi Chris,

the following command line extracts the variable variable-name from a set of input files and do the mergetime. (CDO 1.9.9)

cdo -O -mergetime -apply,-selname,variable-name [ infiles*.nc ]  outfile.nc

-Karin

RE: Selecting a single variable from multiple netcdfs (that have differing numbers of variables) and merging via mergetime or cat... - Added by chris haak over 3 years ago

Hi Karin,

Thanks very much for the quick reply.

forgive my idiocy, but running this line of code:

cdo -O -mergetime -apply,-selname,PAR_mean [ L3m*.nc ] PARmerged.nc

returns the error:

Operator >-selname< not found!
Similar operators are:
delname seldate selname selstdname seltime selzaxisname setname setrcaname splitname selgridname

If I substitute "selname" instead of "-selname", it appears to run but I receive the error:

cdo (Abort): Limit of 128 processes reached!

I have roughly 1000 files to merge. From a quick perusal of the forum it appears that this is a had-coded limitation? And I need to rebuild CDO in order to change this limit?

If I do need to rebuild, can you direct me to any guidance on that? Alternatively, might I be able to achieve it with a loop?

Yes I'm running V 1.9.9 I believe. Here is my version info:

Climate Data Operators version 1.9.9rc1 (https://mpimet.mpg.de/cdo)
System: x86_64-pc-linux-gnu
CXX Compiler: g++ -g -O2 -fdebug-prefix-map=/build/cdo-51aH2m/cdo-1.9.9~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fopenmp
CXX version : g++ (Ubuntu 9.2.1-28ubuntu1) 9.2.1 20200203
C Compiler: gcc -g -O2 -fdebug-prefix-map=/build/cdo-51aH2m/cdo-1.9.9~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -pedantic -fPIC -fopenmp
C version : gcc (Ubuntu 9.2.1-28ubuntu1) 9.2.1 20200203
F77 Compiler: f77 -g -O2 -fdebug-prefix-map=/build/cdo-51aH2m/cdo-1.9.9~rc1=. -fstack-protector-strong
F77 version : unknown
Features: 125GB 24threads C++14 Fortran DATA PTHREADS OpenMP HDF5 NC4/HDF5/threadsafe OPeNDAP SZ UDUNITS2 PROJ MAGICS CURL FFTW3 SSE2
Libraries: HDF5/1.10.4 proj/6.3.0 curl/7.68.0(h7.66.0)
Filetypes: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5
CDI library version : 1.9.9rc1
ecCodes library version : 2.16.0
NetCDF library version : 4.7.3 of Jan 23 2020 04:18:16 $
hdf5 library version : library undefined
exse library version : 1.4.1
FILE library version : 1.8.3

Thanks again and sorry for the trouble,
Chris

RE: Selecting a single variable from multiple netcdfs (that have differing numbers of variables) and merging via mergetime or cat... - Added by Karin Meier-Fleischer over 3 years ago

Thought it must be with a leading '-' but when I do it with version 1.9.8 it is without.

cdo -O -mergetime -apply,selname,PAR_mean [ L3m*.nc ] PARmerged.nc

RE: Selecting a single variable from multiple netcdfs (that have differing numbers of variables) and merging via mergetime or cat... - Added by Suvarna Tikle 11 months ago

Hi,
I would like to extract and merge the variable 'PBLH' from hourly multiple wrfout files and create new nc file.

The command "cdo -O -mergetime -apply,-selname,PBLH [wrfout_d01_2019*] trail_PBLH.nc" gives error 'SysError: pthread_create failed for 'selname'System error message: Resource temporarily unavailable.

Can you please help to resolve it?

Thanks
Suvarna

RE: Selecting a single variable from multiple netcdfs (that have differing numbers of variables) and merging via mergetime or cat... - Added by Suvarna Tikle 11 months ago

Suvarna Tikle wrote in RE: Selecting a single variable from multiple netcdfs (th...:

I am using version 2.0.5 .

Tried "cdo -O -L -mergetime -apply,-selname,PBLH [ wrfout_d01_2019* ] trail_PBLH.nc " but same error.

RE: Selecting a single variable from multiple netcdfs (that have differing numbers of variables) and merging via mergetime or cat... - Added by Ralf Mueller 11 months ago 

 'SysError: pthread_create failed for 'selname'System error message: Resource temporarily unavailable.
looks like the system does not allow thread creation. Maybe you have too many files in the wrfout_d01_2019* list?

RE: Selecting a single variable from multiple netcdfs (that have differing numbers of variables) and merging via mergetime or cat... - Added by Ralf Mueller 11 months ago

I'd change the wildcard wrfout_d01_2019* to that it catches files from a single day and check if this works. A complete moth might be too much already (720 hourly files)

In case you files contain proper timestamps, they might be in the correct temporal order. in that case you can use cat instead of mergetime.

RE: Selecting a single variable from multiple netcdfs (that have differing numbers of variables) and merging via mergetime or cat... - Added by Suvarna Tikle 11 months ago

Yes, it's working for month but showing warning. Shall I ignored the warning. Is it possible to cdo monmean in same command?
Warning (cdf_check_variables): Unsupported data type (char/string), skipped variable Times!
Warning (cdfInqContents): Coordinates variable XTIME can't be assigned!
Warning (cdf_set_var): Inconsistent variable definition for XLAT!
Warning (cdf_set_var): Inconsistent variable definition for XLONG!
Warning (cdf_set_var): Inconsistent variable definition for XLAT_U!
Warning (cdf_set_var): Inconsistent variable definition for XLONG_U!
Warning (cdf_set_var): Inconsistent variable definition for XLAT_V!
Warning (cdf_set_var): Inconsistent variable definition for XLONG_V!

RE: Selecting a single variable from multiple netcdfs (that have differing numbers of variables) and merging via mergetime or cat... - Added by Ralf Mueller 11 months ago

WRF does use wrong coordinate attributes. If this is crucial for you depends on the PBLH. I'd check output for the first month in more detail. If it's ok, you can ignore the warnings, I guess.

RE: Selecting a single variable from multiple netcdfs (that have differing numbers of variables) and merging via mergetime or cat... - Added by Suvarna Tikle 11 months ago

This analysis will be used for identifications of local emissions source contribution correlation, as client and government in their country are concern about the high pollution in particular months.

    (1-13/13)