Converting netCDF to CSV: handle of missing values
Added by Brian Yalle over 3 years ago
Hi everyone,
I've been looking for the source of unexpected value in a csv file which I think it's a missing value.
I reviewed every file in each modification made.
1. Downloaded raw datacdo showattsvar era5land_total_precipitation_hourly_feb-dic_2017_raw.nc
tp:
long_name = "Total precipitation"
units = "m"
missing_value = "-3.276700e+04"
One can see as missing value: -3.276700e+04
2. Modifications appliedcdo -f nc -b F64 -daysum -shifttime,-1hour -mulc,1000 era5land_total_precipitation_hourly_feb-dic_2017_raw.nc era5land_tp_daily_feb-dic_2017_cdo.nc
cdo showattsvar era5land_tp_daily_feb-dic_2017_cdo.nc
tp:
long_name = "Total precipitation"
units = "m"
missing_value = "-3.276700e+04"
cell_methods = "time: sum"
3. Converting netCDF to CSV filecdo -outputtab,date,lat:6,lon:6,value:8 era5land_tp_daily_feb-dic_2017_cdo.nc | grep -v '#' | tr -s ' ' | sed -e 's/ /,/g;s/^.//;s/.$//' >> era5land_tp_daily_feb-dic_2017_cdo.csv
In the image, the value "-32767" appears many times in the column. This is similar to the "-3.276700e+04" value when the "showattsvar" is used.
First question: Is the "-32767" in CSV file indeed a missing value?
Second question: Is there another way in convertion process (netCDF > CSV) that assigns no-numerical label to missing value (to avoid confusion with other climate values)?
Thanks!
Picture1.png (14.2 KB) Picture1.png |
Replies (2)
RE: Converting netCDF to CSV: handle of missing values - Added by Karin Meier-Fleischer over 3 years ago
1. yes, -32767 is the missing value
2. if you want to change the missing value for example to the string 'nan' in the CSV file append ';s/-32767/nan/' to the sed section:
Your call from above (append data to file each call you do!):
cdo -outputtab,date,lat:6,lon:6,value:8 era5land_tp_daily_feb-dic_2017_cdo.nc | grep -v '#' | tr -s ' ' | sed -e 's/ /,/g;s/^.//;s/.$//' >> era5land_tp_daily_feb-dic_2017_cdo.csv
Change missing value -32767 to string 'nan' and overwrite existing file era5land_tp_daily_feb-dic_2017_cdo.csv:
cdo -outputtab,date,lat:6,lon:6,value:8 era5land_tp_daily_feb-dic_2017_cdo.nc | grep -v '#' | tr -s ' ' | sed -e 's/ /,/g;s/^.//;s/.$//;s/-32767/nan/' > era5land_tp_daily_feb-dic_2017_cdo.csv
RE: Converting netCDF to CSV: handle of missing values - Added by Robby Purdy over 3 years ago
Karin Meier-Fleischer wrote in RE: Converting netCDF to CSV: handle of missing values:
1. yes, -32767 is the missing value
2. if you want to change the missing value for example to the string 'nan' in the CSV file append ';s/-32767/nan/' to the sed section:
https://pngtoico.io
Your call from above (append data to file each call you do!):[...]
Change missing value -32767 to string 'nan' and overwrite existing file era5land_tp_daily_feb-dic_2017_cdo.csv:
[...]
Hi Karin,
thank you very much for the answer. I will try right away.
Cheers,
Robby