Project

General

Profile

Forums » Support »

segmentation fault with chained commands and compressed netCDF files

Added by Don Murray about 11 years ago

Hi-

I looked around, but didn't see that anyone else had reported this. I have files that I generated with cdo using -f nc4c -z zip. When I try to chain commands together to extract a subset of the file, I get a segmentation fault:

cdo sellonlatbox,-10,10,40,60 -selmon,1 dew2_echam5_amip_vary_ghg_mean_1979.nc boundary.nc
cdo sellonlatbox: Started child process "selmon,1 dew2_echam5_amip_vary_ghg_mean_1979.nc (pipe1.1)".
Segmentation fault

If I run the commands separately, it works fine:

cdo selmon,1 dew2_echam5_amip_vary_ghg_mean_1979.nc tmp.nc
cdo selmon: Processed 3571200 values from 1 variable over 365 timesteps ( 0.17s )
cdo sellonlatbox,-10,10,40,60 tmp.nc tmp2.nc
cdo sellonlatbox: Processed 3571200 values from 1 variable over 31 timesteps ( 0.05s )

but I'm trying to avoid the step of creating the tmp.nc file.

If I run it with -f nc, it also works, but then I don't have a compressed file:

cdo -f nc sellonlatbox,-10,10,40,60 -selmon,1 dew2_echam5_amip_vary_ghg_mean_1979.nc boundary.nc
cdo sellonlatbox: Started child process "selmon,1 dew2_echam5_amip_vary_ghg_mean_1979.nc (pipe1.1)".
cdo(2) selmon: Processed 3571200 values from 1 variable over 365 timesteps ( 0.10s )
cdo sellonlatbox: Processed 3571200 values from 1 variable over 31 timesteps ( 0.10s )

Using -f nc4c -z zip gives an error.

Is this a bug, or do I need to specify some other option.

Thanks for any help you can provide.


Replies (2)

RE: segmentation fault with chained commands and compressed netCDF files - Added by Uwe Schulzweida about 11 years ago

Hi Don,

CDO is a multi-threaded application. Therefor all external libraries should be compiled thread safe. Using non-threadsafe libraries could cause unexpected errors. Especially netCDF4(HDF5) in combination with operator chaining cause problems, if the HDF5 library is not threadsafe.
A workaround is to change the output file format to standard netCDF:

cdo -f nc sellonlatbox,-10,10,40,60 -selmon,1 ifile.nc4 ofile.nc
Since CDO version 1.5.8 you can lock the I/O with the option -L. This will serialize all I/O accesses.
cdo -L sellonlatbox,-10,10,40,60 -selmon,1 ifile.nc4 ofile.nc4
Hope this will help you.

Cheers,
Uwe

RE: segmentation fault with chained commands and compressed netCDF files - Added by Don Murray about 11 years ago

Uwe Schulzweida wrote:

Hi Don,

CDO is a multi-threaded application. Therefor all external libraries should be compiled thread safe. Using non-threadsafe libraries could cause unexpected errors. Especially netCDF4(HDF5) in combination with operator chaining cause problems, if the HDF5 library is not threadsafe.
A workaround is to change the output file format to standard netCDF:
[...]Since CDO version 1.5.8 you can lock the I/O with the option -L. This will serialize all I/O accesses.
[...]Hope this will help you.

Thanks for the explanation, Uwe. I'll try out the lock option as well.

Don

    (1-2/2)