Project

General

Profile

Importing lat lon value data from text before interpolating

Added by Peter Jacobs almost 4 years ago

Hello! First of all, thanks to all who work on CDO. It's a wonderful tool that I use regularly and am citing in my forthcoming works.

I have point values for sea surface temperatures, along with their lat and lon coordinates, in a CSV file (LGM_coords_SSTs.csv).

I assume there is no way to read this directly into CDO based on previous readings, but I am unsure about this.

I am hoping to get these into CDO, mask out the land areas using an existing netcdf file, and then interpolate these point estimates.

In theory, I know how to do the second and third of these steps already (masking and interpolating).

I have been working on this for ages and can't figure it out.

For ingesting the lat lon data, based on previous answers from this forum I have tried:

cdo -f nc input,my_gridfile ofile.nc < my_ascii_data

I am completely agnostic as to how to accomplish this in terms of importing the data. Any help would be incredibly appreciated as I am working on a crazy deadline.

Thank you!


Replies (7)

RE: Importing lat lon value data from text before interpolating - Added by Uwe Schulzweida almost 4 years ago

Your data file my_ascii_data contains 140 points and the grid description file contains 140 lon/lat coordinates for this point.
You have to replace the following lines in my_gridfile:

xsize = 11
ysize = 11
by
gridsize = 140

RE: Importing lat lon value data from text before interpolating - Added by Peter Jacobs almost 4 years ago

Thanks very much! This did allow me to create the nc file, but attempting to read the values resulted in the attached error from Panoply about non-unique lon values.

I have datapoints along the same lat and same lon points (but not both).

It also looks like I will be unable to regrid properly because the grid is unstructured?

So I assume I need to do this in two steps somehow- define a regular grid that would encompass all the data points, and only then pass the actual lat, lon, and variable values?

Thanks!

error.png (212 KB) error.png

RE: Importing lat lon value data from text before interpolating - Added by Peter Jacobs almost 4 years ago

I should add that I can manipulate these data in R prior to dealing with them in CDO.

So if it in any way helps, I can save the data as a raster first.

RE: Importing lat lon value data from text before interpolating - Added by Peter Jacobs almost 4 years ago

Sorry for the multiple posts-

I tried writing the points out as a netcdf from R, with the attached result (r001.nc). Checking the file with ncdump and opening in Panoply seem to confirm that this worked.

However, prior to writing the file I tried to set the missing values to -999 rather than NaNs to save the trouble with masking. My goal now is to mask out the land values and then interpolate over the SSTs only.

I assume then I need to do a two step process for dealing with the mask and missing values.

Per the tutorial: https://code.mpimet.mpg.de/projects/cdo/wiki/Tutorial#Missing-values

I should

1. set the current missing values to -9999 or similar
2. then mask out land values to NaNs using an existing netcdf
3. then set the -9999 back to missing
4. interpolate the SST values

and this should result in interpolated SSTs values in the ocean only?

r001.nc (170 KB) r001.nc

RE: Importing lat lon value data from text before interpolating - Added by Peter Jacobs almost 4 years ago

And I have the attached mask file if that helps, although here I am stuck because the grid sizes are apparently incompatible even though I originally created the r001.nc using the raster data I used to create the mask....

Seems like the grids are different, but if I attempt to use the point based nc file for the grid to regrid my mask, it just NaNs everything.

For clarity I am renaming the files:

  • LGM-SSTs_from-raster.nc: a netcdf of point estimate SSTs exported from R
  • mask_360x180.nc: the land sea mask I was hoping to be able to apply to the above before interpolating the point values (i.e. force land-only to NaNs/missing)

RE: Importing lat lon value data from text before interpolating - Added by Uwe Schulzweida almost 4 years ago

Yes, the grids are different. You can either use remapnn on the unstructured data

cdo -f nc input,my_gridfile LGM_SST.nc < my_ascii_data
REMAP_EXTRAPOLATE=off CDO_GRIDSEARCH_RADIUS=0.5 cdo remapnn,mask_360x180.nc LGM_SST.nc LGM_SST_360x180.nc
or on the raster data from R to remap the data to 360x180
REMAP_EXTRAPOLATE=off cdo remapnn,mask_360x180.nc  LGM-SSTs_from-raster.nc LGM_SST_360x180.nc
Then you can use setmisstonn or setmisstodis to fill the missing areas. Apply the masking of the land at the end:
cdo setmisstodis LGM_SST_360x180.nc tmpfile
cdo mul tmpfile mask_360x180.nc result
rm tmpfile
Or in one line:
cdo -mul -setmisstodis LGM_SST_360x180.nc mask_360x180.nc result

RE: Importing lat lon value data from text before interpolating - Added by Peter Jacobs almost 4 years ago

Amazing, thank you so much!

I am incredibly grateful!

    (1-7/7)