CDO

Hi Justin,

I've played a little bit with your data but can't solve the problem.

The data file doesn't contain the appropriate coordinate variables for ts(eqcell).
lat and lon can't be correct because

lon = 360
lat = 180

but

eqcell = 41252

I would expect that eqlon(eqcell) which is "Center longitude of equal-area cell" and eqlat(eqcell) which is "Center latitude of equal-area cell" should be used instead.

The next problem is that the coordinate variables are of type float but have to be of type double. With NCO's ncap2 you can change the type of eqlon and eqlat.

ncap2 -O -s 'eqlat=double(eqlat)' ISCCP.HGM.v01r00.GLOBAL.2012.12.99.9999.GPC.10KM.CS00.EQ1.00.nc tmp1.nc
ncap2 -O -s 'eqlon=double(eqlon)' tmp1.nc tmp2.nc

Than add the correct coordinates attribute to the variable ts.

ncatted -O -a coordinates,ts,c,c,"eqlon eqlat" tmp2.nc tmp3.nc

Select the variable and its related coordinate variables.

cdo -selvar,ts tmp3.nc ts.nc

Compute the weights - nearest neighbor method. Note: you can't use genbil because 'Bilinear/bicubic interpolation doesn't support unstructured source grids!'.

cdo gennn,r360x180 ts.nc weights.nc

Regrid the variable ts to the r360x180.

cdo remap,r360x180,weights.nc ts.nc ts_r360x180.nc

ncdump -h ts_r360x180.nc

netcdf ts_r360x180 {
dimensions:
    lon = 360 ;
    lat = 180 ;
variables:
    float lon(lon) ;
        lon:standard_name = "longitude" ;
        lon:long_name = "longitude" ;
        lon:units = "degrees_east" ;
        lon:axis = "X" ;
    float lat(lat) ;
        lat:standard_name = "latitude" ;
        lat:long_name = "latitude" ;
        lat:units = "degrees_north" ;
        lat:axis = "Y" ;
    ubyte ts(lat, lon) ;
        ts:long_name = "Mean surface skin temperature" ;
        ts:units = "count" ;
        ts:_FillValue = 255UB ;
        ts:missing_value = 255UB ;
        ts:conversion_table = "tmptab" ;
...

-Karin

Karin,

I was able to change the type of eqlon and eqlat to double using NCO. I also added the coordinates attribute using ncatted and then used the 'selvar' command to choose only the variable "ts." However, when I go to generate the weights, I have the same issue:

cdo gennn,r360x180 ts.nc weights.nc

cdo gennn (Abort): Variable ts has an unsupported generic grid!

Here's the output of ncdump -h ts.nc:

netcdf ts {
dimensions:
        eqcell = 41252 ;
variables:
        ubyte ts(eqcell) ;
                ts:long_name = "Mean surface skin temperature" ;
                ts:units = "count" ;
                ts:_FillValue = 255UB ;
                ts:missing_value = 255UB ;
                ts:conversion_table = "tmptab" ;
                ts:description = "Grid cell average surface skin temperature for clear pixels detected by both IR and VIS (IR at night) thresholds and using actual surface emissivity, more cloud-contaminated than ts_clrsky" ;

// global attributes:
                :CDI = "Climate Data Interface version 1.7.2 (http://mpimet.mpg.de/cdi)" ;
                :history = "Tue Jul 24 17:55:30 2018: cdo -selvar,ts,eqlon,eqlat tmp3.nc ts.nc\nTue Jul 24 17:37:11 2018: ncatted -O -a coordinates,ts,c,c,eqlon eqlat tmp2.nc tmp3.nc\nTue Jul 24 17:31:13 2018: ncap -O -s eqlon=double(eqlon) tmp1.nc tmp2.nc\nTue Jul 24 17:27:27 2018: ncap -O -s eqlat=double(eqlat) ISCCP.HGM.v01r00.GLOBAL.2012.12.99.9999.GPC.10KM.CS00.EQ1.00.nc tmp1.nc\nWed Feb 21 20:46:51 2018: ncatted -O -a isccp_input_files,global,o,c,d2/ISCCP.HGH.v01r00.GLOBAL.2012.12.99.0000.GPC.10KM.CS00.EQ1.00.nc, d2/ISCCP.HGH.v01r00.GLOBAL.2012.12.99.0300.GPC.10KM.CS00.EQ1.00.nc, d2/ISCCP.HGH.v01r00.GLOBAL.2012.12.99.0600.GPC.10KM.CS00.EQ1.00.nc, d2/ISCCP.HGH.v01r00.GLOBAL.2012.12.99.0900.GPC.10KM.CS00.EQ1.00.nc, d2/ISCCP.HGH.v01r00.GLOBAL.2012.12.99.1200.GPC.10KM.CS00.EQ1.00.nc, d2/ISCCP.HGH.v01r00.GLOBAL.2012.12.99.1500.GPC.10KM.CS00.EQ1.00.nc, d2/ISCCP.HGH.v01r00.GLOBAL.2012.12.99.1800.GPC.10KM.CS00.EQ1.00.nc, d2/ISCCP.HGH.v01r00.GLOBAL.2012.12.99.2100.GPC.10KM.CS00.EQ1.00.nc, ISCCP.HGM.v01r00.GLOBAL.2012.12.99.9999.GPC.10KM.CS00.EQ1.00.nc\nWed Feb 21 20:46:51 2018: ncatted -O -a id,global,o,c,ISCCP.HGM.v01r00.GLOBAL.2012.12.99.9999.GPC.10KM.CS00.EQ1.00.nc ISCCP.HGM.v01r00.GLOBAL.2012.12.99.9999.GPC.10KM.CS00.EQ1.00.nc\nWed Feb 21 20:46:51 2018: ncatted -O -a product_version,global,o,c,v01r00, ISCCP.HGM.v01r00.GLOBAL.2012.12.99.9999.GPC.10KM.CS00.EQ1.00.nc\nWed Feb 21 20:46:50 2018: ncatted -a date_issued,global,d,, ISCCP.HGM.v01r00.GLOBAL.2012.12.99.9999.GPC.10KM.CS00.EQ1.00.nc\n2018-02-06T21:21:03.000Z bhankins d2prodc /snfs2/isccp-p/prd/wrkdirs/2012_12 2012 12" ;
                :source = "Satellite Observations" ;
                :institution = "NOAA National Centers for Environmental Information (NCEI)" ;
                :Conventions = "CF-1.6, ACDD-1.3" ;
                :title = "International Satellite Cloud Climatology Project Gridded Monthly Mean data -- ISCCP HGM" ;
                :summary = "The HGM product represents the monthly mean of the eight hourly-monthly mean HGH files, reported on a 1. degree equal area grid. Data reported include scene identification, cloud amounts, cloud frequency, mean cloud properties, cloud type properties, surface properties and atmospheric profiles." ;
                :keywords = "EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC CHEMISTRY > OXYGEN COMPOUNDS > OZONE, EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC PRESSURE > SURFACE PRESSURE, EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC TEMPERATURE, EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC TEMPERATURE > SURFACE TEMPERATURE > AIR TEMPERATURE, EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC TEMPERATURE > SURFACE TEMPERATURE > SKIN TEMPERATURE, EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC TEMPERATURE > UPPER AIR TEMPERATURE > VERTICAL PROFILES, EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC WATER VAPOR, EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC WATER VAPOR > HUMIDITY, EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC WATER VAPOR > WATER VAPOR PROFILES, EARTH SCIENCE > ATMOSPHERE > CLOUDS, EARTH SCIENCE > ATMOSPHERE > CLOUDS > CLOUD MICROPHYSICS > CLOUD LIQUID WATER/ICE, EARTH SCIENCE > ATMOSPHERE > CLOUDS > CLOUD MICROPHYSICS > CLOUD OPTICAL DEPTH/THICKNESS, EARTH SCIENCE > ATMOSPHERE > CLOUDS > CLOUD PROPERTIES, EARTH SCIENCE > ATMOSPHERE > CLOUDS > CLOUD PROPERTIES > CLOUD FRACTION, EARTH SCIENCE > ATMOSPHERE > CLOUDS > CLOUD PROPERTIES > CLOUD FREQUENCY, EARTH SCIENCE > ATMOSPHERE > CLOUDS > CLOUD PROPERTIES > CLOUD TOP PRESSURE, EARTH SCIENCE > ATMOSPHERE > CLOUDS > CLOUD PROPERTIES > CLOUD TOP TEMPERATURE, EARTH SCIENCE > ATMOSPHERE > CLOUDS > CLOUD PROPERTIES > CLOUD VERTICAL DISTRIBUTION, EARTH SCIENCE > ATMOSPHERE > CLOUDS > CLOUD TYPES, EARTH SCIENCE > CRYOSPHERE > SNOW/ICE, EARTH SCIENCE > LAND SURFACE > SURFACE RADIATIVE PROPERTIES, EARTH SCIENCE > LAND SURFACE > SURFACE RADIATIVE PROPERTIES > REFLECTANCE, EARTH SCIENCE > LAND SURFACE > SURFACE THERMAL PROPERTIES, EARTH SCIENCE > LAND SURFACE > SURFACE THERMAL PROPERTIES > SKIN TEMPERATURE, EARTH SCIENCE > LAND SURFACE > TOPOGRAPHY > TERRAIN ELEVATION, EARTH SCIENCE > OCEANS > OCEAN TEMPERATURE > SEA SURFACE TEMPERATURE" ;
                :keywords_vocabulary = "NASA Global Change Master Directory (GCMD) Science Keyword Version 8.1" ;
                :platform_vocabulary = "NASA Global Change Master Directory (GCMD) Platforms Keyword Version 8.1" ;
                :instrument_vocabulary = "NASA Global Change Master Directory (GCMD) Instruments Keywords Version 8.1" ;
                :processing_level = "3" ;
                :cdm_data_type = "Grid" ;
                :metadata_link = "gov.noaa.ncdc.C00956" ;
                :naming_authority = "gov.noaa.ncdc" ;
                :project = "International Satellite Cloud Climatology Project (ISCCP)" ;
                :program = "NOAA Climate Data Record Program for satellites, FY 2016" ;
                :acknowledgement = "This project received funding support from NASA REASON PROGRAM, NASA MEASURES PROGRAM and NOAA CLIMATE DATA RECORD (CDR) PROGRAM" ;
                :license = "No constraints on data access or use." ;
                :contributor_name = "William B. Rossow, Alison Walker, Violeta Golea, NOAA, EUMETSAT, ESA, JP/JMA, CHINA/CMA, BR/INPE, NASA" ;
                :contributor_role = "principalInvestigator, processor, resourceProvider, resourceProvider, resourceProvider, resourceProvider, resourceProvider, resourceProvider, resourceProvider" ;
                :creator_name = "NOAA National Centers for Environmental Information (NCEI); Ken Knapp, Bill Hankins, Alisa Young, Anand Inamdar" ;
                :creator_email = "ncdc.isccp.team@noaa.gov" ;
                :creator_url = "http://www.ncei.noaa.gov" ;
                :creator_type = "institution" ;
                :creator_institution = "NOAA National Centers for Environmental Information (NCEI)" ;
                :publisher_name = "NOAA National Centers for Environmental Information (NCEI)" ;
                :publisher_email = "ncdc.isccp.team@noaa.gov" ;
                :publisher_url = "http://www.ncei.noaa.gov" ;
                :publisher_type = "institution" ;
                :publisher_institution = "NOAA National Centers for Environmental Information (NCEI)" ;
                :references = "\'Please include a citation for this paper in addition to the dataset citation when using the dataset: Rossow, W.B. and R.A. Schiffer, 1999: Advances in understanding clouds from ISCCP. Bulletin of the American Meteorological Society, 80, 2261-2287. doi: http://dx.doi.org/10.1175/1520-0477(1999)080<2261:AIUCFI>2.0.CO;2\',\'ISCCP CDR Climate Algorithm Theoretical Basis Document (C-ATBD)\'" ;
                :comment = "---------- TO RE-MAP EQUAL-AREA MAP TO EQUAL-ANGLE (SQUARE LON,LAT) MAP ---------- For display purposes, the ISCCP equal-area map may be converted to an equal-angle map using replication. The variables \'eqlat_index\', \'sqlon_beg\' and \'sqlon_end\'are provided for this purpose. Each equal-area cell is replicated into a specific range of longitude cells in the equal-angle map. For example, to remap an equal-area array eqvar[41252] to an equal-angle array sqmap[360,180], each eqvar[i] should be replicated into the range of cells indicated by sqlon_beg[i] and sqlon_end[i], and the lat index eqlat_index[i]. Using Fortran notation the assignment is: sqmap[sqlon_beg[i]:sqlon_end[i], eqlat_index[i]] = eqvar[i]. ---------- TO CONVERT COUNT UNITS TO PHYSICAL UNITS ---------- When attribute conversion_table is present for any variable, the reported values of count units may be converted to physical quantities by using the specified conversion table variable as a look-up table whose index is count value 0-255. For example, temperature = tmptab(count), temperature_variance = tmpvar(count), pressure = pretab(count), reflectance = rfltab(count), optical_depth = tautab(count), ozone = ozntab(count), humidity = humtab(count), water_path = wpatab(count). ---------- DEFINITION OF CLOUD TYPES ---------- VIS/IR cloud types are defined by a histogram of cloud top pressure and cloud optical depth, for both liquid and ice clouds. IR cloud types are defined by a histogram of cloud top pressure. Identification labels for the 18 VIS/IR cloud types and the 3 IR cloud types are given in the \'cloud_type_label\' and \'cloud_irtype_label\' variables, which correspond to the order of the cloud type variable arrays." ;
                :date_created = "2018-02-06T21:21:03.000Z" ;
                :date_modified = "2018-02-06T21:21:03.000Z" ;
                :date_metadata_modified = "2018-02-06T21:21:03.000Z" ;
                :time_coverage_start = "2012-12-01" ;
                :time_coverage_end = "2012-12-31" ;
                :time_coverage_duration = "P1M" ;
                :time_coverage_resolution = "P1M" ;
                :geospatial_bounds = "POLYGON((-90.0 0.0, -90.0 360.0, 90.0 360.0, 90.0 0.0, -90.0 0.0))" ;
                :geospatial_bounds_crs = "EPSG:4326" ;
                :geospatial_lat_min = "-90.0" ;
                :geospatial_lat_max = "+90.0" ;
                :geospatial_lat_units = "degrees_north" ;
                :geospatial_lat_resolution = "1.0" ;
                :geospatial_lon_min = "0.0" ;
                :geospatial_lon_max = "360.0" ;
                :geospatial_lon_units = "degrees_east" ;
                :geospatial_lon_resolution = "1.0 at equator, varies with latitude" ;
                :geospatial_vertical_min = "10" ;
                :geospatial_vertical_max = "1025" ;
                :geospatial_vertical_units = "hPa" ;
                :geospatial_vertical_positive = "down" ;
                :isccp_year = 12 ;
                :isccp_month = 12 ;
                :isccp_gmt = 9999 ;
                :isccp_percent_full_cells = 100 ;
                :isccp_percent_empty_cells = 0 ;
                :isccp_number_of_satellites_contributing = 7 ;
                :platform = "MTS-1, METEOSAT-9, GOES-15, GOES-13, METEOSAT-7, NOAA-18, METOP-A" ;
                :instrument = "JAMI,   SEVIRI,   GOES-15 Imager,   GOES-13 Imager,   MVIRI,   AVHRR-3,   AVHRR-3" ;
                :product_version = "v01r00," ;
                :id = "ISCCP.HGM.v01r00.GLOBAL.2012.12.99.9999.GPC.10KM.CS00.EQ1.00.nc" ;
                :isccp_input_files = "d2/ISCCP.HGH.v01r00.GLOBAL.2012.12.99.0000.GPC.10KM.CS00.EQ1.00.nc, d2/ISCCP.HGH.v01r00.GLOBAL.2012.12.99.0300.GPC.10KM.CS00.EQ1.00.nc, d2/ISCCP.HGH.v01r00.GLOBAL.2012.12.99.0600.GPC.10KM.CS00.EQ1.00.nc, d2/ISCCP.HGH.v01r00.GLOBAL.2012.12.99.0900.GPC.10KM.CS00.EQ1.00.nc, d2/ISCCP.HGH.v01r00.GLOBAL.2012.12.99.1200.GPC.10KM.CS00.EQ1.00.nc, d2/ISCCP.HGH.v01r00.GLOBAL.2012.12.99.1500.GPC.10KM.CS00.EQ1.00.nc, d2/ISCCP.HGH.v01r00.GLOBAL.2012.12.99.1800.GPC.10KM.CS00.EQ1.00.nc, d2/ISCCP.HGH.v01r00.GLOBAL.2012.12.99.2100.GPC.10KM.CS00.EQ1.00.nc," ;
                :NCO = "\"4.6.1\"" ;
                :CDO = "Climate Data Operators version 1.7.2 (http://mpimet.mpg.de/cdo)" ;

I noticed that the "coordinates" attribute is no longer present after I used the "selvar" function.

I did use the ncap command rather than the ncap2 command (couldn't get it working) when changing the type of eqlon and eqlat from float to double, but I don't think this would cause this issue.

-Justin

I suggest the problem appears because you use ncap instead of ncap2 (ncap is deprecated since 2008) and maybe an old version of CDO, too. Which version of CDO are you using (cdo -V)? I'm using CDO version 1.9.2.

ncap2 -O -s 'eqlat=double(eqlat)' ISCCP.HGM.v01r00.GLOBAL.2012.12.99.9999.GPC.10KM.CS00.EQ1.00.nc tmp1.nc
ncap2 -O -s 'eqlon=double(eqlon)' tmp1.nc tmp2.nc

ncdump -h tmp2.nc

netcdf tmp2 {
dimensions:
    eqcell = 41252 ;
    edge = 2 ;
    label_len = 80 ;
    satid_len = 4 ;
    satname_len = 20 ;
    satpos = 12 ;
    eqzone = 180 ;
    lon = 360 ;
    lat = 180 ;
    cldbin = 10 ;
    levtmp = 16 ;
    levrh = 16 ;
    cloud_irtype = 3 ;
    cloud_type = 18 ;
    count = 256 ;
variables:
    double eqlon(eqcell) ;
        eqlon:long_name = "Center longitude of equal-area cell" ;
        eqlon:units = "degrees_east" ;
        eqlon:valid_max = 360. ;
        eqlon:valid_min = 0. ;
    double eqlat(eqcell) ;
        eqlat:long_name = "Center latitude of equal-area cell" ;
        eqlat:units = "degrees_north" ;
        eqlat:valid_max = 90. ;
        eqlat:valid_min = -90. ;
...

ncatted -O -a coordinates,ts,c,c,"eqlon eqlat" tmp2.nc tmp3.nc

ncdump -h tmp3.nc

...
    ubyte ts(eqcell) ;
        ts:long_name = "Mean surface skin temperature" ;
        ts:units = "count" ;
        ts:valid_min = 0 ;
        ts:valid_max = 254 ;
        ts:_FillValue = 255UB ;
        ts:conversion_table = "tmptab" ;
        ts:description = "Grid cell average surface skin temperature for clear pixels detected by both IR and VIS (IR at night) thresholds and using actual surface emissivity, more cloud-contaminated than ts_clrsky" ;
        ts:coordinates = "eqlon eqlat" ;
...

cdo -selvar,ts tmp3.nc ts.nc

ncdump -h ts.nc

netcdf ts {
dimensions:
    eqcell = 41252 ;
variables:
    double eqlon(eqcell) ;
        eqlon:standard_name = "longitude" ;
        eqlon:long_name = "Center longitude of equal-area cell" ;
        eqlon:units = "degrees_east" ;
    double eqlat(eqcell) ;
        eqlat:standard_name = "latitude" ;
        eqlat:long_name = "Center latitude of equal-area cell" ;
        eqlat:units = "degrees_north" ;
    ubyte ts(eqcell) ;
        ts:long_name = "Mean surface skin temperature" ;
        ts:units = "count" ;
        ts:CDI_grid_type = "unstructured" ;
        ts:coordinates = "eqlat eqlon" ;
        ts:_FillValue = 255UB ;
        ts:missing_value = 255UB ;
        ts:conversion_table = "tmptab" ;
        ts:description = "Grid cell average surface skin temperature for clear pixels detected by both IR and VIS (IR at night) thresholds and using actual surface emissivity, more cloud-contaminated than ts_clrsky" ;
...

ncap2 -O -s 'eqlat=double(eqlat)' ISCCP.HGM.v01r00.GLOBAL.2012.12.99.9999.GPC.10KM.CS00.EQ1.00.nc tmp1.nc
ncap2 -O -s 'eqlon=double(eqlon)' tmp1.nc tmp2.nc
ncatted -O -a coordinates,ts,c,c,"eqlon eqlat" tmp2.nc tmp3.nc
cdo -selvar,ts tmp3.nc ts.nc
cdo gennn,r360x180 ts.nc weights.nc
cdo remap,r360x180,weights.nc ts.nc ts_r360x180.nc

-Karin

Hello,
I found this very useful, but I have one additional problem.
I am trying to remap portions of ISCCP.HGM.v01r00.GLOBAL.1983.07.99.9999.GPC.10KM.CS00.EQ1.00.nc
I want to look at cldamt_irtypes(cloud_irtype, eqcell)
This has the nonstandard vertical dimension cloud_irtype, which makes it fail with this variable.
I have tried a variety of nc commands to pick off one level of this variable, but each time it excludes the other dimensions, making remapping impossible.
Does anyone have any suggestions?
Bryan