Problem combining multiple operators

Added by David Gobbett about 13 years ago

Hi all,

I am having problems combining multiple cdo operators, and am not sure why.
This example works:

cdo setrtomiss,-99999,0 -divc,$c3 $VP_FILE $tmp1
cdo ln $tmp1 $tmp2
cdo mulc,$c1 $tmp2 $tmp3
cdo mulc,-1 $tmp2 $tmp4
cdo div $tmp3 -addc,$c2 $tmp4 $Td_VP_FILE1

and so does this (and is faster too):

cdo ln -setrtomiss,-99999,0 -divc,$c3 $VP_FILE $tmp2
cdo mulc,$c1 $tmp2 $tmp3
cdo div $tmp3 -addc,$c2 -mulc,-1 $tmp2 $Td_VP_FILE2

but this:

cdo ln -setrtomiss,-99999,0 -divc,$c3 $VP_FILE $tmp2
cp $tmp2 $tmp2_copy
cdo div -mulc,$c1 $tmp2_copy -addc,$c2 -mulc,-1 $tmp2 $Td_VP_FILE2

fails with the error message...
Error (pstreamOpenRead) : pthread_attr_init failed for 'addc,17.269 -mulc,-1 /cygdrive/l/ClimateData/tmp2_copy.nc (pipe1.2)'
System error message : Invalid argument

The main difference is that I have incorporated the first mulc into the final line.
(Note that to avoid two threads accessing the same tmp2 file I have first created a copy)

Any thoughts or ideas on why this is occurring?

thanks
David

Replies (7)

RE: Problem combining multiple operators - Added by Ralf Mueller about 13 years ago

Hi David!
I tested this (modulo input files and operator parameters) on Linux and cywin (on WindowsXP) without any problems.
Maybe, you could provide the input data and other variable setting for your calls.
What does your cdo -V look like?

regards
ralf

Screenshot.png (21.3 KB) Screenshot.png

RE: Problem combining multiple operators - Added by Ralf Mueller about 13 years ago

I recompiled it under cygwin (the above version is a precompiles on) with the same result

Screenshot-recompiled.png (27 KB) Screenshot-recompiled.png

RE: Problem combining multiple operators - Added by David Gobbett about 13 years ago

Hi Ralf,

Thanks for taking a look at this. I am running Cygwin on a Windows 7 32-Bit OS (running on a Citrix virtual desktop).

$ cdo -V
Climate Data Operators version 1.4.7 (http://code.zmaw.de/projects/cdo)
Compiler: gcc -std=gnu99 -g -O2
version: gcc (GCC) 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)
with: PTHREADS Z
Compiled: by gobx on gobx-yc (i686-pc-cygwin) Jan 28 2011 12:51:02
CDI library version : 1.4.7 of Jan 28 2011 12:49:48
CGRIBEX library version : 1.4.7 of Nov 12 2010 08:21:37
netCDF library version : "3.6.3" of Jan 7 2011 16:20:41 $
SERVICE library version : 1.3.0 of Jan 28 2011 12:49:08
EXTRA library version : 1.3.0 of Jan 28 2011 12:48:52
IEG library version : 1.3.0 of Jan 28 2011 12:49:02
FILE library version : 1.7.1 of Jan 28 2011 12:48:54

RE: Problem combining multiple operators - Added by Ralf Mueller about 13 years ago

If this GCC version is not realy required, I suggest you install the gcc4 package and recompile CDO. GCC 3.4.4 is from 2005. I might have problems with recent pthread. Another benefit is OpenMP support, see CDO's OpenMP support page.

RE: Problem combining multiple operators - Added by David Gobbett about 13 years ago

OK, I think this was just the gcc version that I installed with Cygwin, and as far as I know I don't have a need for that. I'll look into recompiling with a later version of GCC.

thanks for your suggestion, I'll let you know how it goes.
David

RE: Problem combining multiple operators - Added by David Gobbett about 13 years ago

I have now rebuilt NetCDF CDO with gcc4. (I have a surprising problem that 'make' of NetCDF 4, and CDO sometimes completely crashes the Windows 7 system, which is why I still use netCDF 3.6.3.)

$ cdo -V

Climate Data Operators version 1.4.7 (http://code.zmaw.de/projects/cdo)
Compiler: gcc -std=gnu99 -g -O2
version: gcc (GCC) 4.3.4 20090804 (release) 1
with: PTHREADS Z
Compiled: by gobx on gobx-yc (i686-pc-cygwin) Feb 18 2011 09:32:28
CDI library version : 1.4.7 of Jan 28 2011 12:49:48
CGRIBEX library version : 1.4.7 of Nov 12 2010 08:21:37
netCDF library version : "3.6.3" of Feb 17 2011 17:30:37 $
SERVICE library version : 1.3.0 of Jan 28 2011 12:49:08
EXTRA library version : 1.3.0 of Jan 28 2011 12:48:52
IEG library version : 1.3.0 of Jan 28 2011 12:49:02
FILE library version : 1.7.1 of Jan 28 2011 12:48:54

I have now successfully run the following commands that did not previously work:

cdo setrtomiss,-99999,0 -divc,$c3 $VP_FILE $tmp1
cp $tmp1 $tmp1_copy
cdo div -mulc,$c1 -ln $tmp1 -addc,$c2 -mulc,-1 -ln $tmp1_copy $Td_VP_FILE2

and also all now combined into a single command line...

cdo div -mulc,$c1 -ln -setrtomiss,-99999,0 -divc,$c3 $VP_FILE -addc,$c2 -mulc,-1 -ln -setrtomiss,-99999,0 -divc,$c3 $VP_FILE $Td_VP_FILE2

This is excellent as it is now using the full 8 CPU cores and reduces disk I/O substantially.

Thanks for your suggestion.
David

RE: Problem combining multiple operators - Added by Ralf Mueller about 13 years ago

You can still improved it though ;-)

CDO is still working with netcdf 3.6.3. Is this intended? You mentioned netcdf4 ...
You could build CDO with OpenMP support, which can be a significant speed up for some operators: see OpenMP_support. For gcc, just reconfigure with CFLAGS=-fopenmp, rebuild and install.

cheers
ralf

(1-7/7)

Project

General

Profile

CDO