Converting netCDF to CSV: handle of missing values

Hi everyone,
I've been looking for the source of unexpected value in a csv file which I think it's a missing value.

I reviewed every file in each modification made.
1. Downloaded raw data
cdo showattsvar era5land_total_precipitation_hourly_feb-dic_2017_raw.nc
tp: long_name = "Total precipitation" units = "m" missing_value = "-3.276700e+04"
One can see as missing value: -3.276700e+04

2. Modifications applied
cdo -f nc -b F64 -daysum -shifttime,-1hour -mulc,1000 era5land_total_precipitation_hourly_feb-dic_2017_raw.nc era5land_tp_daily_feb-dic_2017_cdo.nc

cdo showattsvar era5land_tp_daily_feb-dic_2017_cdo.nc
tp: long_name = "Total precipitation" units = "m" missing_value = "-3.276700e+04" cell_methods = "time: sum"

3. Converting netCDF to CSV file
cdo -outputtab,date,lat:6,lon:6,value:8 era5land_tp_daily_feb-dic_2017_cdo.nc | grep -v '#' | tr -s ' ' | sed -e 's/ /,/g;s/^.//;s/.$//' >> era5land_tp_daily_feb-dic_2017_cdo.csv

In the image, the value "-32767" appears many times in the column. This is similar to the "-3.276700e+04" value when the "showattsvar" is used.
First question: Is the "-32767" in CSV file indeed a missing value?
Second question: Is there another way in convertion process (netCDF > CSV) that assigns no-numerical label to missing value (to avoid confusion with other climate values)?

Thanks!

Picture1.png (14.2 KB) Picture1.png

Replies (2)

RE: Converting netCDF to CSV: handle of missing values - Added by Karin Meier-Fleischer over 3 years ago

1. yes, -32767 is the missing value
2. if you want to change the missing value for example to the string 'nan' in the CSV file append ';s/-32767/nan/' to the sed section:

Your call from above (append data to file each call you do!):

cdo -outputtab,date,lat:6,lon:6,value:8 era5land_tp_daily_feb-dic_2017_cdo.nc | grep -v '#' | tr -s ' ' | sed -e 's/ /,/g;s/^.//;s/.$//' >> era5land_tp_daily_feb-dic_2017_cdo.csv

Change missing value -32767 to string 'nan' and overwrite existing file era5land_tp_daily_feb-dic_2017_cdo.csv:

cdo -outputtab,date,lat:6,lon:6,value:8 era5land_tp_daily_feb-dic_2017_cdo.nc | grep -v '#' | tr -s ' ' | sed -e 's/ /,/g;s/^.//;s/.$//;s/-32767/nan/' > era5land_tp_daily_feb-dic_2017_cdo.csv

RE: Converting netCDF to CSV: handle of missing values - Added by Robby Purdy over 3 years ago

Karin Meier-Fleischer wrote in RE: Converting netCDF to CSV: handle of missing values:

1. yes, -32767 is the missing value
2. if you want to change the missing value for example to the string 'nan' in the CSV file append ';s/-32767/nan/' to the sed section:
https://pngtoico.io
Your call from above (append data to file each call you do!):

[...]

Change missing value -32767 to string 'nan' and overwrite existing file era5land_tp_daily_feb-dic_2017_cdo.csv:

[...]

Hi Karin,

thank you very much for the answer. I will try right away.

Cheers,
Robby

(1-2/2)

Project

General

Profile

CDO