coordinate and grid increment distortion when applying selcode and gridboxmean
Added by Lutz Bendlin almost 12 years ago
I have been approached by a skipper to assist with a problem caused by a new format for ocean current data. They are in the large shipping business and use that data to calculate the optimal routes for their ships based on fuel consumption etc.
Up until March they were using the NCOM files that had a precision of 1/8 degrees. The data files had 2880 longitude points and 1113 latitude points resulting in 3205440 grid coordinates, each with u and v values (east and north currents at sea level) as well as three other variables. Now the US Navy has changed their file format to HYCOM with a precision of 1/12 degree, and 4499 longitude points and 2001 latitude points resulting in 9002499 grid coordinates for these variables.
This causes problems for their GRIB viewer ( Ballgate) which runs as a Windows 32 bit application and seems to hit the 2GB limits. They cannot change their computing environment, and cannot move to Panoply as they use a heavily customised version of Ballgate.
Now they have tried to convert the HYCOM 1/12 degree files to 1/6 degree files using cdo gridboxmean,2,2 . However when using this process they noticed that the resulting coordinates no longer follow the expected grid interval. The same appears to happen when extracting the required variables (through seclude,-4,-5).
Is that because the new precision cannot be described as a decimal fraction? 1/8 degrees is always shown in CDO as 0.125, but the 1/12 degrees are shown as 0.08 through cdo sinfo (alternatively as 0.0800000128 or 0.08000018311, while the proper value would have been 0.0833333. Similar for 1/6 degrees.
As a result the grid coordinates are also no longer showing the expected values. for example xvals could appear as
0.0400000066 0.200000025 0.360000044 0.520000085 0.680000097 0.840000093
1.00000009 1.16000021 1.32000017 1.4800002 1.64000016 1.80000025 1.96000022
2.12000012 2.28000045 2.4400003 2.60000038 2.76000023 2.92000043 3.0800004
etc.
Is this caused by some rounding issues? Is there any way to prevent this from happening?
Replies (6)
RE: coordinate and grid increment distortion when applying selcode and gridboxmean - Added by Jaison-Thomas Ambadan almost 12 years ago
Hi,
It would be very helpful if you could upload:
1) a sample of your original file (must be <50MB, may be just one variable and one level)
2) and the output of "cdo -V"
Cheers,
J
RE: coordinate and grid increment distortion when applying selcode and gridboxmean - Added by Ralf Mueller almost 12 years ago
I talked to Lutz yesterday. Input is a global file from http://ftp.opc.ncep.noaa.gov/grids/operational/GLOBALHYCOM/Navy/ like
RE: coordinate and grid increment distortion when applying selcode and gridboxmean - Added by Ralf Mueller almost 12 years ago
part of ncdump output:
netcdf hycom_glb_sfc_2013042400_t168 {
dimensions:
time = UNLIMITED ; // (1 currently)
depth = 1 ;
lat = 2001 ;
lon = 4499 ;
variables:
double time(time) ;
time:long_name = "Valid Time" ;
time:units = "hour since 2000-01-01 00:00:00" ;
time:time_origin = "2000-01-01 00:00:00" ;
time:calendar = "gregorian" ;
time:axis = "T" ;
time:NAVO_code = 13 ;
double tau(time) ;
tau:long_name = "Tau" ;
tau:units = "hours since analysis" ;
tau:time_origin = "2013-04-24 00:00:00" ;
tau:NAVO_code = 56 ;
double depth(depth) ;
depth:long_name = "Depth" ;
depth:standard_name = "depth" ;
depth:units = "meter" ;
depth:positive = "down" ;
depth:axis = "Z" ;
depth:NAVO_code = 5 ;
double lat(lat) ;
lat:long_name = "Latitude" ;
lat:standard_name = "latitude" ;
lat:units = "degrees_north" ;
lat:point_spacing = "even" ;
lat:axis = "Y" ;
lat:NAVO_code = 1 ;
double lon(lon) ;
lon:long_name = "Longitude" ;
lon:standard_name = "longitude" ;
lon:units = "degrees_east" ;
lon:modulo = "360 degrees" ;
lon:axis = "X" ;
lon:NAVO_code = 2 ;
short water_temp(time, depth, lat, lon) ;
water_temp:long_name = "Water Temperature" ;
water_temp:standard_name = "sea_water_temperature" ;
water_temp:units = "degC" ;
water_temp:_FillValue = -30000s ;
water_temp:missing_value = -30000s ;
water_temp:scale_factor = 0.001f ;
water_temp:add_offset = 20.f ;
water_temp:NAVO_code = 15 ;
on the ships, they use the win32 version of cdo, no idea which version.
RE: coordinate and grid increment distortion when applying selcode and gridboxmean - Added by Ralf Mueller almost 12 years ago
[ram@thingol:~/data/cdo]cdo sinfov hycom_glb_sfc_2013042400_t168.nc [14:49:31|13-04-25]
File format: netCDF
-1 : Institut Source Ttype Levels Num Gridsize Num Dtype : Parameter name
1 : unknown HYCOM instant 1 1 9002499 1 I16 : water_temp
2 : unknown HYCOM instant 1 1 9002499 1 I16 : salinity
3 : unknown HYCOM instant 1 2 9002499 1 I16 : surf_el
4 : unknown HYCOM instant 1 1 9002499 1 I16 : water_u
5 : unknown HYCOM instant 1 1 9002499 1 I16 : water_v
Grid coordinates :
1 : lonlat > size : dim = 9002499 nx = 4499 ny = 2001
lon : first = 0 last = 359.840057 inc = 0.0800000128 degrees_east
lat : first = -80 last = 80 inc = 0.0800018311 degrees_north
Vertical coordinates :
1 : generic meter : 0
2 : surface : 0
Time coordinate : 1 step
RefTime = 2000-01-01 00:00:00 Units = hours Calendar = standard
YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss
2013-05-01 00:00:00
strange, that the data variables are identified as I16
RE: coordinate and grid increment distortion when applying selcode and gridboxmean - Added by Jaison-Thomas Ambadan almost 12 years ago
Hi,
I think the problem is, your data is not exactly in a regulat lon-lat grid. From ncdump -h: "model_type = "x-curvilinear lon, y-curvilinear lat, hybrid z". As far as I understood CDO recognisze your data as REGULAR lon-lat grid - so that may be the reason for the co-ordinate increment precision issues. Anyway, try agin with -setgridtype,curvilinear:
cdo -b F64 -gridboxmean,2,2 -setgridtype,curvilinear hycom_glb_sfc_2013042400_t168.nc ofile.nc
if there is still issues, then it may be better to "interpolate" or "remap" to regular lon-lat using remap operators (e.g. remapbil/remapnn), which should give exact lon-lat increments (the "gridboxmean" doesn't do that I suppose)
Cheers,
J.
RE: coordinate and grid increment distortion when applying selcode and gridboxmean - Added by Jaison-Thomas Ambadan almost 12 years ago
strange, that the data variables are identified as I16
Ralf, I think CDO interpreted the data variables correctly. If you look at the ncdump -h, all data variables are "short" -> for short integer I guess (?) and all the co-ordinate variables are "double".