Project

General

Profile

-outputtab for multiple combinations of points

Added by Fabio Farinosi over 4 years ago

Hello,

I have a large netcdf providing global daily values with a high spatial resolution. I would like to extract the time series for a selection of points (some hundred thousands of them) using the -outputtab operator. Is there a way to do something like this?

cdo -outputtab,lon,lat,date,value -remapnn,lon_and_lat_combination_list.txt ifile.nc > ofile.txt

In case this is possible, what format should I give to the points coordinate list file? Should it be a .txt? a .csv?
(Please, note that the points I need to extract the data for are not in a regular grid)
I tried a couple of combinations, but I received the following error messages:

cdo(1) remapnn: Process started
Namelist error: Invalid contents in pts_coords_list.txt!
cdo(1) remapnn (Abort): Namelist error!
cdo(1) remapnn: Process started
Namelist error: Invalid contents in pts_coords_list.csv!
cdo(1) remapnn (Abort): Namelist error!

Thank you


Replies (11)

RE: -outputtab for multiple combinations of points - Added by Karin Meier-Fleischer over 4 years ago

Hi Fabio,

I would guess the answer is no. But you can write a short shell script using a loop to select all points.
Here is an example Korn-Shell script which selects 4 points (in your case you can read the x and y arrays
from file):

#!/bin/ksh

#-- delete existing ASCII output file
rm -rf outfile.txt

#-- name of the input file
infile=inputfile.nc

#-- arrays of the points lon lat locations
set -A lon 0.0 5.0 10.0 15.0
set -A lat 0.0 5.0 10.0 15.0

#-- loop to retrieve the data of the selected points 
for i in $(seq 0 3)
do
   cdo -O remapnn,lon=${lon[$i]}/lat=${lat[$i]} $infile tmp.nc
   cdo -outputtab,lon,lat,lev,date,value tmp.nc >> outfile.txt
done

#-- delete the multiple entries of the header line in output file
header="#" 
awk '!header[$0]++' outfile.txt > tmp2.txt
mv tmp2.txt outfile.txt

#-- remove temporary file
rm tmp.nc

exit

Take care whether your data file has levels or not.

-Karin

RE: -outputtab for multiple combinations of points - Added by Fabio Farinosi over 4 years ago

Hi Karin,

Thanks for your help.
Sorry, I'm not very familiar with the ksh and I'm having some troubles running this script.

I installed the mksh in the cygwin environment, put your script in a .ksh file and tried to run it but it gives me the following error indicating the line where the for loop starts:

file.ksh[15]: syntax error: unexpected 'do

In addition, I'd really appreciate if you could give me some guidance in how I should edit the script to go take the lon lat arrays from a file like the one attached.
Thank you very much

RE: -outputtab for multiple combinations of points - Added by Karin Meier-Fleischer over 4 years ago

The script is a Korn Shell script so you need to have ksh installed.
You can modify the script and run it, lets assume the script name is your_script.ksh:

ksh your_script.ksh

But you can rewrite it to use another shell scripting language as you like, it's on you.

-Karin

RE: -outputtab for multiple combinations of points - Added by Fabio Farinosi over 4 years ago

Hello Karin
Sorry, I've then realized that there was a problem of Windows/UNIX encoding in the line ending of the script, because I used a normal EditPad Lite 7 to write down a script inspired by yours instead of writing it down directly in a vi editor.
The editor I used was using the Windows encoding to end the lines in the code and that was causing the syntax error. The encoding is specifically:

^M

It took me a while to figure this out because the vi editor of CygWin was not highlighting this problem, while the UNIX one was.

Once this issue was fixed, the code was running smoothly. This methodology, however, is extremely costly in processing time. The domain in my .nc file has almost 4 million cells for more than 14 thousands time steps, I need to extract time series for about 400 thousands points. This script would do that sequentially for each of the lon/lat combinations. In each time step, the procedure would spend about 800/1000 seconds only in the remapnn step.
It would be nice to have a more efficient way to extract data for specific set of coordinates.

Thank you very much again.

Best,
F

RE: -outputtab for multiple combinations of points - Added by Max Ballhausen over 4 years ago

Hi there, thanks for this script!

I am trying to extract from negative longitudes or latitudes, and run into an error:

test110: set: -0: unknown option
test110: set: -.: unknown option
test110: set: -9: unknown option
test110: set: -6: unknown option
test110: set: -3: unknown option
test110: set: -2: unknown option
test110: set: -4: unknown option
test110: set: -6: unknown option

This is for example for longitude -0.963246

I tried to fix it with the conversion using -convertlon or -convertlat but this only gives another error.
In that case all my extracted values become 1e+20

Any help would be appreciated!

Thanks
Max

script and inputfile are attached!

test1 (592 Bytes) test1
euin.nc (5.24 MB) euin.nc

RE: -outputtab for multiple combinations of points - Added by Karin Meier-Fleischer over 4 years ago

Hi Max,

what did you do exactly? -- Sorry didn't noticed that test1 is a script. It is always a good style to name a script including a suffix, e.g. test1.ksh.

-Karin

RE: -outputtab for multiple combinations of points - Added by Karin Meier-Fleischer over 4 years ago

if you want to extract only one single point than you don't have to use arrays.

cdo -O remapnn,lon=-0.963246/lat=38.308579 $infile tmp.nc
cdo -outputtab,lon,lat,date,value tmp.nc >> outfile.txt

Or in a script

#!/bin/ksh

#-- delete existing ASCII output file
rm -rf outfile.txt

#-- name of the input file
infile=euin.nc

#-- lon lat location
lon=-0.963246
lat=38.308579

#-- retrieve the data of the selected points 
cdo -O remapnn,lon=${lon}/lat=${lat} $infile tmp.nc
cdo -outputtab,lon,lat,date,value tmp.nc >> outfile.txt

#-- remove temporary file
rm tmp.nc

exit

-Karin

RE: -outputtab for multiple combinations of points - Added by Max Ballhausen over 4 years ago

Thanks for the quick reply. I will have to extract more than one pair of points. I just put one pair in for the sake of testing it :-)
I think the problem that I have is related to korn shell and how it deals with negative values.. Could that be the case?

RE: -outputtab for multiple combinations of points - Added by Max Ballhausen over 4 years ago

I solved it!

Instead of

set -A lon -0.963246
set -A lat 38.308579

it has to be

set A lon - 0.963246
set -A lat -
38.308579

Got the hint from this thread:
https://www.unix.com/shell-programming-and-scripting/186579-solved-capturing-output-korn-variable.html

Best
Max

RE: -outputtab for multiple combinations of points - Added by Max Ballhausen over 4 years ago

I solved it!

Instead of

set -A lon -0.963246
set -A lat 38.308579

it has to be

set -A lon -- -0.963246
set -A lat -- 38.308579

Got the hint from this thread:
https://www.unix.com/shell-programming-and-scripting/186579-solved-capturing-output-korn-variable.html

Best
Max

RE: -outputtab for multiple combinations of points - Added by Karin Meier-Fleischer over 4 years ago

Yup, that's the way when you want to use arrays. I should have mentioned that, too.

-Karin

    (1-11/11)