Project

General

Profile

Forums » Support »

Issue using outputttab operator: converting netcdf to csv files

Added by Brian Yalle almost 3 years ago

Hi,
Not sure about the cause but I think my problem concerns with "outputtab" operator.
My main data resource is in netCDF format. Due I want to use it in STATA software, netCDF files are converted into CSV ones. Before knowing CDO, I used Python to convert.

When I convert netCDF to CSV format using Jupyter (Python): the data in the output is delimited with commas like in the following image

Converting netCDF to CSV format using CDO: this can be observed in the image
  • data is delimited with blank spaces
  • there is a "#" before variable names
  • latitude and longitude values are rounded

Coding employed to obtain this result is
cdo -outputtab,date,lat,lon,value era5land_total_precipitation_daily_feb-nov_2017_mm.nc > test1_tp.csv

I'd like to obtain CSV file delimited by commas using CDO. Also, latitude/longitude variables do not show such rounded values.
Thanks!

Download all files
Picture_2.png (4.23 KB) Picture_2.png
Picture_1.png (7.32 KB) Picture_1.png

Replies (4)

RE: Issue using outputttab operator: converting netcdf to csv files - Added by Karin Meier-Fleischer almost 3 years ago

Hi Brian,

you can change the output of the cdo call in two steps:

echo 'time,latitude,longitude,tp' > test1_tp.csv
cdo -outputtab,date,lat:6,lon:6,value:8 era5land_total_precipitation_daily_feb-nov_2017_mm.nc | grep -v '#' | sed -e 's/   */,/g' >> test1_tp.csv

-Karin

RE: Issue using outputttab operator: converting netcdf to csv files - Added by Brian Yalle almost 3 years ago

Karin,

I tested the code you suggested but it stills has some problems.

For example, in the image:
  1. Delimitation parameter is not uniform in all rows: all values in 11 and 12 rows are delimited by commas but last two values in rows from 13 to 19 are delimited by blank space. The next row, 20, is delimited correctly.
  2. The data doesn't have variable names in the top.
  3. Latitude and longitude values still show rounded values.

I tried with another netCDF file (with far less storage weight than original) to verify if it repeats, and it does. I upload this file if you want test it.

Download all files
Picture_3.png (8.94 KB) Picture_3.png
era5land_tp_daily_ene-dic_2017.nc (66.3 MB) era5land_tp_daily_ene-dic_2017.nc

RE: Issue using outputttab operator: converting netcdf to csv files - Added by Karin Meier-Fleischer almost 3 years ago

1. Some commas missing due to the different numbers of blanks per line between the fields. See modified commands below.
2. Important is to use the '>>' instead '>' to append the output of the second call to the file.
3. The output of 'ncdump -v longitude era5land_tp_daily_ene-dic_2017.nc' shows that the output of the longitude values, for example, are not rounded values.

 longitude = -81.4, -81.3, -81.2, -81.1, -81, -80.9, -80.8, -80.7, -80.6, 
    -80.5, -80.4, -80.3, -80.2, -80.1, -80, -79.9, -79.8, -79.7, -79.6, 
    -79.5, -79.4, -79.3, -79.2, -79.1, -79, -78.9, -78.8, -78.7, -78.6, 
    -78.5, -78.4, -78.3, -78.2, -78.1, -78, -77.9, -77.8, -77.7, -77.6, 
    -77.5, -77.4, -77.3, -77.2, -77.1, -77, -76.9, -76.8, -76.7, -76.6, 
    -76.5, -76.4, -76.3, -76.2, -76.1, -76, -75.9, -75.8, -75.7, -75.6, 
    -75.5, -75.4, -75.3, -75.2, -75.1, -75, -74.9, -74.8, -74.7, -74.6, 
    -74.5, -74.4, -74.3, -74.2, -74.1, -74, -73.9, -73.8, -73.7, -73.6, 
    -73.5, -73.4, -73.3, -73.2, -73.1, -73, -72.9, -72.8, -72.7, -72.6, 
    -72.5, -72.4, -72.3, -72.2, -72.1, -72, -71.9, -71.8, -71.7, -71.6, 
    -71.5, -71.4, -71.3, -71.2, -71.1, -71, -70.9, -70.8, -70.7, -70.6, 
    -70.5, -70.4, -70.3, -70.2, -70.1, -70, -69.9, -69.8, -69.7, -69.6, 
    -69.5, -69.4, -69.3, -69.2, -69.1, -69, -68.9, -68.8, -68.7, -68.6 ;

 latitude = -0.1, -0.2, -0.3, -0.4, -0.5, -0.6, -0.7, -0.8, -0.9, -1, -1.1, 
    -1.2, -1.3, -1.4, -1.5, -1.6, -1.7, -1.8, -1.9, -2, -2.1, -2.2, -2.3, 
    -2.4, -2.5, -2.6, -2.7, -2.8, -2.9, -3, -3.1, -3.2, -3.3, -3.4, -3.5, 
    -3.6, -3.7, -3.8, -3.9, -4, -4.1, -4.2, -4.3, -4.4, -4.5, -4.6, -4.7, 
    -4.8, -4.9, -5, -5.1, -5.2, -5.3, -5.4, -5.5, -5.6, -5.7, -5.8, -5.9, -6, 
    -6.1, -6.2, -6.3, -6.4, -6.5, -6.6, -6.7, -6.8, -6.9, -7, -7.1, -7.2, 
    -7.3, -7.4, -7.5, -7.6, -7.7, -7.8, -7.9, -8, -8.1, -8.2, -8.3, -8.4, 
    -8.5, -8.6, -8.7, -8.8, -8.9, -9, -9.1, -9.2, -9.3, -9.4, -9.5, -9.6, 
    -9.7, -9.8, -9.9, -10, -10.1, -10.2, -10.3, -10.4, -10.5, -10.6, -10.7, 
    -10.8, -10.9, -11, -11.1, -11.2, -11.3, -11.4, -11.5, -11.6, -11.7, 
    -11.8, -11.9, -12, -12.1, -12.2, -12.3, -12.4, -12.5, -12.6, -12.7, 
    -12.8, -12.9, -13, -13.1, -13.2, -13.3, -13.4, -13.5, -13.6, -13.7, 
    -13.8, -13.9, -14, -14.1, -14.2, -14.3, -14.4, -14.5, -14.6, -14.7, 
    -14.8, -14.9, -15, -15.1, -15.2, -15.3, -15.4, -15.5, -15.6, -15.7, 
    -15.8, -15.9, -16, -16.1, -16.2, -16.3, -16.4, -16.5, -16.6, -16.7, 
    -16.8, -16.9, -17, -17.1, -17.2, -17.3, -17.4, -17.5, -17.6, -17.7, 
    -17.8, -17.9, -18, -18.1, -18.2, -18.3, -18.4 ;

Do the following:

echo 'time,latitude,longitude,tp' > test1_tp.csv

cdo -outputtab,date,lat:6,lon:6,value:8 era5land_tp_daily_ene-dic_2017.nc | grep -v '#' | tr -s ' ' | sed -e 's/ /,/g;s/^.//;s/.$//' >> test1_tp.csv

Output of 'head -16 test1_tp.csv'

time,latitude,longitude,tp
2016-12-31,-0.1,-81.4,-32767
2016-12-31,-0.1,-81.3,-32767
2016-12-31,-0.1,-81.2,-32767
2016-12-31,-0.1,-81.1,-32767
2016-12-31,-0.1,-81,-32767
2016-12-31,-0.1,-80.9,-32767
2016-12-31,-0.1,-80.8,-32767
2016-12-31,-0.1,-80.7,-32767
2016-12-31,-0.1,-80.6,-32767
2016-12-31,-0.1,-80.5,-32767
2016-12-31,-0.1,-80.4,-32767
2016-12-31,-0.1,-80.3,-32767
2016-12-31,-0.1,-80.2,2.84229351738396
2016-12-31,-0.1,-80.1,2.91305601159267
2016-12-31,-0.1,-80,2.84229351738396
    (1-4/4)