Recoding values of variable fails for very large grid
Added by Nikolai Huckle over 7 years ago
I am working with a very large grid (8'398'080'000 cells) and am currently trying to recode certain values.
This is the code I am using
time cdo -v -expr,'v=(v==0)?1.0:0.0;' in.nc out.nc
And this is the error that is thrown when I try to run the code on a server node with 36GB memory.
OpenMP: num_procs = 1 max_threads = 1 num_devices = 0 cdo expr: v=(v==0)?1.0:0.0; Error (gridDefXsize) : xsize 129600 is greater then gridsize -191854592 HDF5-DIAG: Error detected in HDF5 (1.8.16) thread 0: #000: H5T.c line 1716 in H5Tclose(): not a datatype major: Invalid arguments to routine minor: Inappropriate type Error (cdf_close) : NetCDF: HDF error real 2m34.235s user 2m27.234s sys 0m7.116s
Although there is a lot of memory, I still think that memory is the issue somehow. The code obviously works with smaller netCDF files.
My questions are if this error is known and if anyone knows a more memory efficient way to recode values. Thanks in advance for any help.
Kind regards
Here is also the ncdump of the file in question
netcdf ESACCI-LC-L4-LCCS-Map-300m-P1Y-2014-v2.0.7 { dimensions: lat = 64800 ; lon = 129600 ; variables: byte lccs_class(lat, lon) ; lccs_class:long_name = "Land cover class defined in LCCS" ; lccs_class:standard_name = "land_cover_lccs" ; lccs_class:flag_values = 0b, 10b, 11b, 12b, 20b, 30b, 40b, 50b, 60b, 61b, 62b, 70b, 71b, 72b, 80b, 81b, 82b, 90b, 100b, 110b, 120b, 121b, 122b, -126b, -116b, -106b, -104b, -103b, -96b, -86b, -76b, -66b, -56b, -55b, -54b, -4 6b, -36b ; lccs_class:flag_meanings = "no_data cropland_rainfed cropland_rainfed_herbaceous_cover cropland_rainfed_tree_or_shrub_cover cropland_irrigated mosaic_cropland mosaic_natural_vegetation tree_broadleaved_evergreen_closed_to_o pen tree_broadleaved_deciduous_closed_to_open tree_broadleaved_deciduous_closed tree_broadleaved_deciduous_open tree_needleleaved_evergreen_closed_to_open tree_needleleaved_evergreen_closed tree_needleleaved_evergreen_open tree_needleleave d_deciduous_closed_to_open tree_needleleaved_deciduous_closed tree_needleleaved_deciduous_open tree_mixed mosaic_tree_and_shrub mosaic_herbaceous shrubland shrubland_evergreen shrubland_deciduous grassland lichens_and_mosses sparse_vegetat ion sparse_shrub sparse_herbaceous tree_cover_flooded_fresh_or_brakish_water tree_cover_flooded_saline_water shrub_or_herbaceous_cover_flooded urban bare_areas bare_areas_consolidated bare_areas_unconsolidated water snow_and_ice" ; lccs_class:valid_min = 1 ; lccs_class:valid_max = 220 ; lccs_class:_Unsigned = "true" ; lccs_class:_FillValue = 0b ; lccs_class:ancillary_variables = "processed_flag current_pixel_state observation_count change_count" ; byte processed_flag(lat, lon) ; processed_flag:long_name = "LC map processed area flag" ; processed_flag:standard_name = "land_cover_lccs status_flag" ; processed_flag:flag_values = 0b, 1b ; processed_flag:flag_meanings = "not_processed processed" ; processed_flag:valid_min = 0 ; processed_flag:valid_max = 1 ; processed_flag:_FillValue = -1b ; byte current_pixel_state(lat, lon) ; current_pixel_state:long_name = "LC pixel type mask" ; current_pixel_state:standard_name = "land_cover_lccs status_flag" ; current_pixel_state:flag_values = 0b, 1b, 2b, 3b, 4b, 5b ; current_pixel_state:flag_meanings = "invalid clear_land clear_water clear_snow_ice cloud cloud_shadow" ; current_pixel_state:valid_min = 0 ; current_pixel_state:valid_max = 5 ; current_pixel_state:_FillValue = -1b ; short observation_count(lat, lon) ; observation_count:long_name = "number of valid observations" ; observation_count:standard_name = "land_cover_lccs number_of_observations" ; observation_count:valid_min = 0 ; observation_count:valid_max = 32767 ; observation_count:_FillValue = -1s ; byte change_count(lat, lon) ; change_count:long_name = "number of class changes" ; change_count:valid_min = 0 ; change_count:valid_max = 100 ; change_count:_FillValue = -1b ; float lat(lat) ; lat:units = "degrees_north" ; lat:long_name = "latitude" ; lat:standard_name = "latitude" ; lat:valid_min = -89.99861f ; lat:valid_max = 89.99861f ; float lon(lon) ; lon:units = "degrees_east" ; lon:long_name = "longitude" ; lon:standard_name = "longitude" ; lon:valid_min = -179.9986f ; lon:valid_max = 179.9986f ; int crs ; crs:wkt = "GEOGCS[\"WGS 84\", \n", " DATUM[\"World Geodetic System 1984\", \n", " SPHEROID[\"WGS 84\", 6378137.0, 298.257223563, AUTHORITY[\"EPSG\",\"7030\"]], \n", " AUTHORITY[\"EPSG\",\"6326\"]], \n", " PRIMEM[\"Greenwich\", 0.0, AUTHORITY[\"EPSG\",\"8901\"]], \n", " UNIT[\"degree\", 0.017453292519943295], \n", " AXIS[\"Geodetic longitude\", EAST], \n", " AXIS[\"Geodetic latitude\", NORTH], \n", " AUTHORITY[\"EPSG\",\"4326\"]]" ; crs:i2m = "0.002777777777778,0.0,0.0,-0.002777777777778,-180.0,90.0" ; // global attributes: :title = "ESA CCI Land Cover Map" ; :summary = "This dataset contains the global ESA CCI land cover classification map derived from satellite data of one epoch." ; :type = "ESACCI-LC-L4-LCCS-Map-300m-P1Y" ; :id = "ESACCI-LC-L4-LCCS-Map-300m-P1Y-2015-v2.0.7" ; :project = "Climate Change Initiative - European Space Agency" ; :references = "http://www.esa-landcover-cci.org/" ; :institution = "Universite catholique de Louvain" ; :contact = "landcover-cci@uclouvain.be" ; :comment = "" ; :Conventions = "CF-1.6" ; :standard_name_vocabulary = "NetCDF Climate and Forecast (CF) Standard Names version 21" ; :keywords = "land cover classification,satellite,observation" ; :keywords_vocabulary = "NASA Global Change Master Directory (GCMD) Science Keywords" ; :license = "ESA CCI Data Policy: free and open access" ; :naming_authority = "org.esa-cci" ; :cdm_data_type = "grid" ; :TileSize = "2048:2048" ; :tracking_id = "202be995-43d8-4e3a-9607-1bd3f02a925e" ; :product_version = "2.0.7" ; :date_created = "20170330T210326Z" ; :creator_name = "University catholique de Louvain" ; :creator_url = "http://www.uclouvain.be/" ; :creator_email = "landcover-cci@uclouvain.be" ; :source = "MERIS FR L1B version 5.05, MERIS RR L1B version 8.0, SPOT VGT P" ; :history = "amorgos-4,0, lc-sdr-1.0, lc-sr-1.0, lc-classification-1.0,lc-user-tools-3.12" ; :time_coverage_start = "20150101" ; :time_coverage_end = "20151231" ; :time_coverage_duration = "P1Y" ; :time_coverage_resolution = "P1Y" ; :geospatial_lat_min = "-90.0" ; :geospatial_lat_max = "90.0" ; :geospatial_lon_min = "-180.0" ; :geospatial_lon_max = "180.0" ; :spatial_resolution = "300m" ; :geospatial_lat_units = "degrees_north" ; :geospatial_lat_resolution = "0.002778" ; :geospatial_lon_units = "degrees_east" ; :geospatial_lon_resolution = "0.002778" ;
Replies (2)
RE: Recoding values of variable fails for very large grid - Added by Uwe Schulzweida over 7 years ago
The maximum number of grid cells supported in CDO is 2147483647 (INT_MAX). We already started to change this to 64-bit. This project will be finished in Q4 2017.
RE: Recoding values of variable fails for very large grid - Added by Nikolai Huckle over 7 years ago
Oh right, thanks for your quick answer.
So the solution would be to split the nc-File and the run the cdo-command on the subsets?