Recoding values of variable fails for very large grid
Added by Nikolai Huckle over 8 years ago
I am working with a very large grid (8'398'080'000 cells) and am currently trying to recode certain values.
This is the code I am using
time cdo -v -expr,'v=(v==0)?1.0:0.0;' in.nc out.nc
And this is the error that is thrown when I try to run the code on a server node with 36GB memory.
OpenMP: num_procs = 1 max_threads = 1 num_devices = 0
cdo expr: v=(v==0)?1.0:0.0;
Error (gridDefXsize) : xsize 129600 is greater then gridsize -191854592
HDF5-DIAG: Error detected in HDF5 (1.8.16) thread 0:
#000: H5T.c line 1716 in H5Tclose(): not a datatype
major: Invalid arguments to routine
minor: Inappropriate type
Error (cdf_close) : NetCDF: HDF error
real 2m34.235s
user 2m27.234s
sys 0m7.116s
Although there is a lot of memory, I still think that memory is the issue somehow. The code obviously works with smaller netCDF files.
My questions are if this error is known and if anyone knows a more memory efficient way to recode values. Thanks in advance for any help.
Kind regards
Here is also the ncdump of the file in question
netcdf ESACCI-LC-L4-LCCS-Map-300m-P1Y-2014-v2.0.7 {
dimensions:
lat = 64800 ;
lon = 129600 ;
variables:
byte lccs_class(lat, lon) ;
lccs_class:long_name = "Land cover class defined in LCCS" ;
lccs_class:standard_name = "land_cover_lccs" ;
lccs_class:flag_values = 0b, 10b, 11b, 12b, 20b, 30b, 40b, 50b, 60b, 61b, 62b, 70b, 71b, 72b, 80b, 81b, 82b, 90b, 100b, 110b, 120b, 121b, 122b, -126b, -116b, -106b, -104b, -103b, -96b, -86b, -76b, -66b, -56b, -55b, -54b, -4
6b, -36b ;
lccs_class:flag_meanings = "no_data cropland_rainfed cropland_rainfed_herbaceous_cover cropland_rainfed_tree_or_shrub_cover cropland_irrigated mosaic_cropland mosaic_natural_vegetation tree_broadleaved_evergreen_closed_to_o
pen tree_broadleaved_deciduous_closed_to_open tree_broadleaved_deciduous_closed tree_broadleaved_deciduous_open tree_needleleaved_evergreen_closed_to_open tree_needleleaved_evergreen_closed tree_needleleaved_evergreen_open tree_needleleave
d_deciduous_closed_to_open tree_needleleaved_deciduous_closed tree_needleleaved_deciduous_open tree_mixed mosaic_tree_and_shrub mosaic_herbaceous shrubland shrubland_evergreen shrubland_deciduous grassland lichens_and_mosses sparse_vegetat
ion sparse_shrub sparse_herbaceous tree_cover_flooded_fresh_or_brakish_water tree_cover_flooded_saline_water shrub_or_herbaceous_cover_flooded urban bare_areas bare_areas_consolidated bare_areas_unconsolidated water snow_and_ice" ;
lccs_class:valid_min = 1 ;
lccs_class:valid_max = 220 ;
lccs_class:_Unsigned = "true" ;
lccs_class:_FillValue = 0b ;
lccs_class:ancillary_variables = "processed_flag current_pixel_state observation_count change_count" ;
byte processed_flag(lat, lon) ;
processed_flag:long_name = "LC map processed area flag" ;
processed_flag:standard_name = "land_cover_lccs status_flag" ;
processed_flag:flag_values = 0b, 1b ;
processed_flag:flag_meanings = "not_processed processed" ;
processed_flag:valid_min = 0 ;
processed_flag:valid_max = 1 ;
processed_flag:_FillValue = -1b ;
byte current_pixel_state(lat, lon) ;
current_pixel_state:long_name = "LC pixel type mask" ;
current_pixel_state:standard_name = "land_cover_lccs status_flag" ;
current_pixel_state:flag_values = 0b, 1b, 2b, 3b, 4b, 5b ;
current_pixel_state:flag_meanings = "invalid clear_land clear_water clear_snow_ice cloud cloud_shadow" ;
current_pixel_state:valid_min = 0 ;
current_pixel_state:valid_max = 5 ;
current_pixel_state:_FillValue = -1b ;
short observation_count(lat, lon) ;
observation_count:long_name = "number of valid observations" ;
observation_count:standard_name = "land_cover_lccs number_of_observations" ;
observation_count:valid_min = 0 ;
observation_count:valid_max = 32767 ;
observation_count:_FillValue = -1s ;
byte change_count(lat, lon) ;
change_count:long_name = "number of class changes" ;
change_count:valid_min = 0 ;
change_count:valid_max = 100 ;
change_count:_FillValue = -1b ;
float lat(lat) ;
lat:units = "degrees_north" ;
lat:long_name = "latitude" ;
lat:standard_name = "latitude" ;
lat:valid_min = -89.99861f ;
lat:valid_max = 89.99861f ;
float lon(lon) ;
lon:units = "degrees_east" ;
lon:long_name = "longitude" ;
lon:standard_name = "longitude" ;
lon:valid_min = -179.9986f ;
lon:valid_max = 179.9986f ;
int crs ;
crs:wkt = "GEOGCS[\"WGS 84\", \n",
" DATUM[\"World Geodetic System 1984\", \n",
" SPHEROID[\"WGS 84\", 6378137.0, 298.257223563, AUTHORITY[\"EPSG\",\"7030\"]], \n",
" AUTHORITY[\"EPSG\",\"6326\"]], \n",
" PRIMEM[\"Greenwich\", 0.0, AUTHORITY[\"EPSG\",\"8901\"]], \n",
" UNIT[\"degree\", 0.017453292519943295], \n",
" AXIS[\"Geodetic longitude\", EAST], \n",
" AXIS[\"Geodetic latitude\", NORTH], \n",
" AUTHORITY[\"EPSG\",\"4326\"]]" ;
crs:i2m = "0.002777777777778,0.0,0.0,-0.002777777777778,-180.0,90.0" ;
// global attributes:
:title = "ESA CCI Land Cover Map" ;
:summary = "This dataset contains the global ESA CCI land cover classification map derived from satellite data of one epoch." ;
:type = "ESACCI-LC-L4-LCCS-Map-300m-P1Y" ;
:id = "ESACCI-LC-L4-LCCS-Map-300m-P1Y-2015-v2.0.7" ;
:project = "Climate Change Initiative - European Space Agency" ;
:references = "http://www.esa-landcover-cci.org/" ;
:institution = "Universite catholique de Louvain" ;
:contact = "landcover-cci@uclouvain.be" ;
:comment = "" ;
:Conventions = "CF-1.6" ;
:standard_name_vocabulary = "NetCDF Climate and Forecast (CF) Standard Names version 21" ;
:keywords = "land cover classification,satellite,observation" ;
:keywords_vocabulary = "NASA Global Change Master Directory (GCMD) Science Keywords" ;
:license = "ESA CCI Data Policy: free and open access" ;
:naming_authority = "org.esa-cci" ;
:cdm_data_type = "grid" ;
:TileSize = "2048:2048" ;
:tracking_id = "202be995-43d8-4e3a-9607-1bd3f02a925e" ;
:product_version = "2.0.7" ;
:date_created = "20170330T210326Z" ;
:creator_name = "University catholique de Louvain" ;
:creator_url = "http://www.uclouvain.be/" ;
:creator_email = "landcover-cci@uclouvain.be" ;
:source = "MERIS FR L1B version 5.05, MERIS RR L1B version 8.0, SPOT VGT P" ;
:history = "amorgos-4,0, lc-sdr-1.0, lc-sr-1.0, lc-classification-1.0,lc-user-tools-3.12" ;
:time_coverage_start = "20150101" ;
:time_coverage_end = "20151231" ;
:time_coverage_duration = "P1Y" ;
:time_coverage_resolution = "P1Y" ;
:geospatial_lat_min = "-90.0" ;
:geospatial_lat_max = "90.0" ;
:geospatial_lon_min = "-180.0" ;
:geospatial_lon_max = "180.0" ;
:spatial_resolution = "300m" ;
:geospatial_lat_units = "degrees_north" ;
:geospatial_lat_resolution = "0.002778" ;
:geospatial_lon_units = "degrees_east" ;
:geospatial_lon_resolution = "0.002778" ;
Replies (2)
RE: Recoding values of variable fails for very large grid - Added by Uwe Schulzweida over 8 years ago
The maximum number of grid cells supported in CDO is 2147483647 (INT_MAX). We already started to change this to 64-bit. This project will be finished in Q4 2017.
RE: Recoding values of variable fails for very large grid - Added by Nikolai Huckle over 8 years ago
Oh right, thanks for your quick answer.
So the solution would be to split the nc-File and the run the cdo-command on the subsets?