Project

General

Profile

Recoding values of variable fails for very large grid

Added by Nikolai Huckle almost 7 years ago

I am working with a very large grid (8'398'080'000 cells) and am currently trying to recode certain values.
This is the code I am using

time cdo -v -expr,'v=(v==0)?1.0:0.0;' in.nc out.nc

And this is the error that is thrown when I try to run the code on a server node with 36GB memory.
 OpenMP:  num_procs = 1  max_threads = 1  num_devices = 0
cdo expr: v=(v==0)?1.0:0.0;
Error (gridDefXsize) : xsize 129600 is greater then gridsize -191854592
HDF5-DIAG: Error detected in HDF5 (1.8.16) thread 0:
  #000: H5T.c line 1716 in H5Tclose(): not a datatype
    major: Invalid arguments to routine
    minor: Inappropriate type
Error (cdf_close) : NetCDF: HDF error

real    2m34.235s
user    2m27.234s
sys     0m7.116s

Although there is a lot of memory, I still think that memory is the issue somehow. The code obviously works with smaller netCDF files.
My questions are if this error is known and if anyone knows a more memory efficient way to recode values. Thanks in advance for any help.
Kind regards

Here is also the ncdump of the file in question

netcdf ESACCI-LC-L4-LCCS-Map-300m-P1Y-2014-v2.0.7 {
dimensions:
    lat = 64800 ;
    lon = 129600 ;
variables:
    byte lccs_class(lat, lon) ;
        lccs_class:long_name = "Land cover class defined in LCCS" ;
        lccs_class:standard_name = "land_cover_lccs" ;
        lccs_class:flag_values = 0b, 10b, 11b, 12b, 20b, 30b, 40b, 50b, 60b, 61b, 62b, 70b, 71b, 72b, 80b, 81b, 82b, 90b, 100b, 110b, 120b, 121b, 122b, -126b, -116b, -106b, -104b, -103b, -96b, -86b, -76b, -66b, -56b, -55b, -54b, -4
6b, -36b ;
        lccs_class:flag_meanings = "no_data cropland_rainfed cropland_rainfed_herbaceous_cover cropland_rainfed_tree_or_shrub_cover cropland_irrigated mosaic_cropland mosaic_natural_vegetation tree_broadleaved_evergreen_closed_to_o
pen tree_broadleaved_deciduous_closed_to_open tree_broadleaved_deciduous_closed tree_broadleaved_deciduous_open tree_needleleaved_evergreen_closed_to_open tree_needleleaved_evergreen_closed tree_needleleaved_evergreen_open tree_needleleave
d_deciduous_closed_to_open tree_needleleaved_deciduous_closed tree_needleleaved_deciduous_open tree_mixed mosaic_tree_and_shrub mosaic_herbaceous shrubland shrubland_evergreen shrubland_deciduous grassland lichens_and_mosses sparse_vegetat
ion sparse_shrub sparse_herbaceous tree_cover_flooded_fresh_or_brakish_water tree_cover_flooded_saline_water shrub_or_herbaceous_cover_flooded urban bare_areas bare_areas_consolidated bare_areas_unconsolidated water snow_and_ice" ;
        lccs_class:valid_min = 1 ;
        lccs_class:valid_max = 220 ;
        lccs_class:_Unsigned = "true" ;
        lccs_class:_FillValue = 0b ;
        lccs_class:ancillary_variables = "processed_flag current_pixel_state observation_count change_count" ;
    byte processed_flag(lat, lon) ;
        processed_flag:long_name = "LC map processed area flag" ;
        processed_flag:standard_name = "land_cover_lccs status_flag" ;
        processed_flag:flag_values = 0b, 1b ;
        processed_flag:flag_meanings = "not_processed processed" ;
        processed_flag:valid_min = 0 ;
        processed_flag:valid_max = 1 ;
        processed_flag:_FillValue = -1b ;
    byte current_pixel_state(lat, lon) ;
        current_pixel_state:long_name = "LC pixel type mask" ;
        current_pixel_state:standard_name = "land_cover_lccs status_flag" ;
        current_pixel_state:flag_values = 0b, 1b, 2b, 3b, 4b, 5b ;
        current_pixel_state:flag_meanings = "invalid clear_land clear_water clear_snow_ice cloud cloud_shadow" ;
        current_pixel_state:valid_min = 0 ;
        current_pixel_state:valid_max = 5 ;
        current_pixel_state:_FillValue = -1b ;
    short observation_count(lat, lon) ;
        observation_count:long_name = "number of valid observations" ;
        observation_count:standard_name = "land_cover_lccs number_of_observations" ;
        observation_count:valid_min = 0 ;
        observation_count:valid_max = 32767 ;
        observation_count:_FillValue = -1s ;
    byte change_count(lat, lon) ;
        change_count:long_name = "number of class changes" ;
        change_count:valid_min = 0 ;
        change_count:valid_max = 100 ;
        change_count:_FillValue = -1b ;
    float lat(lat) ;
        lat:units = "degrees_north" ;
        lat:long_name = "latitude" ;
        lat:standard_name = "latitude" ;
        lat:valid_min = -89.99861f ;
        lat:valid_max = 89.99861f ;
    float lon(lon) ;
        lon:units = "degrees_east" ;
        lon:long_name = "longitude" ;
        lon:standard_name = "longitude" ;
        lon:valid_min = -179.9986f ;
        lon:valid_max = 179.9986f ;
    int crs ;
        crs:wkt = "GEOGCS[\"WGS 84\", \n",
            "  DATUM[\"World Geodetic System 1984\", \n",
            "    SPHEROID[\"WGS 84\", 6378137.0, 298.257223563, AUTHORITY[\"EPSG\",\"7030\"]], \n",
            "    AUTHORITY[\"EPSG\",\"6326\"]], \n",
            "  PRIMEM[\"Greenwich\", 0.0, AUTHORITY[\"EPSG\",\"8901\"]], \n",
            "  UNIT[\"degree\", 0.017453292519943295], \n",
            "  AXIS[\"Geodetic longitude\", EAST], \n",
            "  AXIS[\"Geodetic latitude\", NORTH], \n",
            "  AUTHORITY[\"EPSG\",\"4326\"]]" ;
        crs:i2m = "0.002777777777778,0.0,0.0,-0.002777777777778,-180.0,90.0" ;

// global attributes:
        :title = "ESA CCI Land Cover Map" ;
        :summary = "This dataset contains the global ESA CCI land cover classification map derived from satellite data of one epoch." ;
        :type = "ESACCI-LC-L4-LCCS-Map-300m-P1Y" ;
        :id = "ESACCI-LC-L4-LCCS-Map-300m-P1Y-2015-v2.0.7" ;
        :project = "Climate Change Initiative - European Space Agency" ;
        :references = "http://www.esa-landcover-cci.org/" ;
        :institution = "Universite catholique de Louvain" ;
        :contact = "landcover-cci@uclouvain.be" ;
        :comment = "" ;
        :Conventions = "CF-1.6" ;
        :standard_name_vocabulary = "NetCDF Climate and Forecast (CF) Standard Names version 21" ;
        :keywords = "land cover classification,satellite,observation" ;
        :keywords_vocabulary = "NASA Global Change Master Directory (GCMD) Science Keywords" ;
        :license = "ESA CCI Data Policy: free and open access" ;
        :naming_authority = "org.esa-cci" ;
        :cdm_data_type = "grid" ;
        :TileSize = "2048:2048" ;
        :tracking_id = "202be995-43d8-4e3a-9607-1bd3f02a925e" ;
        :product_version = "2.0.7" ;
        :date_created = "20170330T210326Z" ;
        :creator_name = "University catholique de Louvain" ;
        :creator_url = "http://www.uclouvain.be/" ;
        :creator_email = "landcover-cci@uclouvain.be" ;
        :source = "MERIS FR L1B version 5.05, MERIS RR L1B version 8.0, SPOT VGT P" ;
        :history = "amorgos-4,0, lc-sdr-1.0, lc-sr-1.0, lc-classification-1.0,lc-user-tools-3.12" ;
        :time_coverage_start = "20150101" ;
        :time_coverage_end = "20151231" ;
        :time_coverage_duration = "P1Y" ;
        :time_coverage_resolution = "P1Y" ;
        :geospatial_lat_min = "-90.0" ;
        :geospatial_lat_max = "90.0" ;
        :geospatial_lon_min = "-180.0" ;
        :geospatial_lon_max = "180.0" ;
        :spatial_resolution = "300m" ;
        :geospatial_lat_units = "degrees_north" ;
        :geospatial_lat_resolution = "0.002778" ;
        :geospatial_lon_units = "degrees_east" ;
        :geospatial_lon_resolution = "0.002778" ;


Replies (2)

RE: Recoding values of variable fails for very large grid - Added by Uwe Schulzweida almost 7 years ago

The maximum number of grid cells supported in CDO is 2147483647 (INT_MAX). We already started to change this to 64-bit. This project will be finished in Q4 2017.

RE: Recoding values of variable fails for very large grid - Added by Nikolai Huckle almost 7 years ago

Oh right, thanks for your quick answer.
So the solution would be to split the nc-File and the run the cdo-command on the subsets?

    (1-2/2)