Problem with pipeline command
Added by Mao-Lin Shen over 9 years ago
Hi,
I used cdo to post-process data.
And because I am working with 30 ensemble members as well as thousands output files, sometimes I got the following situation.
---------------------------------------
cdo remapcon,r144x96 -seldate,1997-02-01,1997-02-31 /work/earnest/temp/HadISST_sst_flipped.nc /work/earnest/temp/temp_mod_Obs_1997-02
cdo remapcon: Started child process "seldate,1997-02-01,1997-02-31 /work/earnest/temp/HadISST_sst_flipped.nc (pipe1.1)".
cdo remapcon (Abort): Couldn't read data from input stream (pipe1.1)!
---------------------------------------
Not so sure it is a general issue, since I could get it done by re-run the command.
But it is quiet annoying that the job is interrupted.
The version of CDO is as follows.
See if any other users have similar experiments and know how to deal with it.
Thanks.
Climate Data Operators version 1.5.6.1 (http://code.zmaw.de/projects/cdo)
Compiler: icc -std=gnu99 -g -O2 -pthread
version: Intel(R) C Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 12.0.5.220 Build 20110719
with: PTHREADS NC4 Z PROJ.4
Compiled: by vegarde on service0 (x86_64-unknown-linux-gnu) Oct 15 2012 09:56:39
CDI library version : 1.5.6 of Oct 15 2012 09:56:12
CGRIBEX library version : 1.5.3 of Jun 14 2012 13:22:45
GRIB_API library version : 1.9.16
netCDF library version : 4.1.3 of Nov 1 2011 12:54:16 $
HDF5 library version : 1.8.9
SERVICE library version : 1.3.0 of Oct 15 2012 09:55:45
EXTRA library version : 1.3.0 of Oct 15 2012 09:55:33
IEG library version : 1.3.0 of Oct 15 2012 09:55:40
FILE library version : 1.8.0 of Oct 15 2012 09:55:34
Replies (4)
RE: Problem with pipeline command - Added by Jaison-Thomas Ambadan over 9 years ago
It's may be due to the fact that your NetCDF library is not compiled with thread-safe enabled. You could try your command sequence with "-L" option. since your CDO version is old I'm not sure if the -L feature is availale in your case. Anyway, I recommend upgrading CDO to the latest version.
Cheers,
J.
RE: Problem with pipeline command - Added by Mao-Lin Shen over 9 years ago
Hi,
Thanks for your reply.
It actually happens on the other HPC with the following cdo version as well.
Anyway, I'll try using "-L" option and ask the HPC-support about the thread-safe issue.
Climate Data Operators version 1.6.7 (https://code.zmaw.de/projects/cdo)
Compiled: by aoltu on hexagon-3 (x86_64-unknown-linux-gnu) Feb 10 2015 10:13:53
Compiler: cc -std=gnu99 -g -O2 -fopenmp
version: unknown
Features: PTHREADS OpenMP SZ Z JASPER UDUNITS2 FFTW3 SSE3
Libraries:
Filetypes: srv ext ieg grb grb2 nc
CDI library version : 1.6.7 of Feb 10 2015 10:10:08
CGRIBEX library version : 1.7.0 of Dec 3 2014 08:52:12
GRIB_API library version : 1.12.1
netCDF library version : 4.3.2 of Aug 7 2014 13:25:25 $
HDF5 library version : 1.8.13
SERVICE library version : 1.3.2 of Feb 10 2015 10:09:42
EXTRA library version : 1.3.2 of Feb 10 2015 10:09:33
IEG library version : 1.3.3 of Feb 10 2015 10:09:39
FILE library version : 1.8.2 of Feb 10 2015 10:09:33
RE: Problem with pipeline command - Added by Uwe Schulzweida over 9 years ago
This should only happened if the requested dates are not available in the input file. At least with CDO version 1.6.7 the following warning message is printed?:
cdo(2) seldate (Warning): Date between 19970201.0000 and 19970231.9990 not found!Please check the contents of your input file with sinfo:
cdo sinfo /work/earnest/temp/HadISST_sst_flipped.nc /work/earnest/temp/temp_mod_Obs_1997-02
RE: Problem with pipeline command - Added by Mao-Lin Shen over 9 years ago
Hi,
I think the requested date is not an issue here.
As with CDO version 1.6.7 I never got the warning in this case.