A problem while converting csv file to netcdf
Added by nazanin tavakoli almost 4 years ago
Hello,
I have 12 monthly precipitation data in 2001 which are in CSV format, and I want to convert them to a NetCDF file with the code below:
cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day\
-settaxis,2001-01-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_1.nc < p_2001_1.csv
cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day \
-settaxis,2001-02-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_2.nc < p_2001_2.csv
cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day \
-settaxis,2001-03-0,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_3.nc < p_2001_3.csv
cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day\
-settaxis,2001-04-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_4.nc < p_2001_4.csv
cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day \
-settaxis,2001-05-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_5.nc < p_2001_5.csv
cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day \
-settaxis,2001-06-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_6.nc < p_2001_6.csv
cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1mon \
-settaxis,2001-07-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_7.nc < p_2001_7.csv
cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day \
-settaxis,2001-08-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_8.nc < p_2001_8.csv
cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day \
-settaxis,2001-09-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_9.nc < p_2001_9.csv
cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day \
-settaxis,2001-10-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_1.nc < p_2001_1.csv
cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day \
-settaxis,2001-01-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_10.nc < p_2001_10.csv
cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day \
-settaxis,2001-11-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_11.nc < p_2001_11.csv
cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day \
-settaxis,2001-12-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_12.nc < p_2001_12.csv
cdo -O -chname,var1,temp \
-setattribute,FILE=attributes.txt \
-mergetime \
p_2001_*.nc prec.mon.nc
but, the grid of all my cells are changing in the final NetCDF file. I mean that if you consider a grid cell with the special latitude and longitude, the value of this cell in the NetCDF file is not the same as the CSV file. Also, if you open p_2001_1.nc, you will see that the time of this NetCDF file is 2001-10-01 which is not the time I set in the code.
Could anybody help me?
Replies (8)
RE: A problem while converting csv file to netcdf - Added by Karin Meier-Fleischer almost 4 years ago
When you encounter problems using a shell script to call cdo, you have to check your shell script first.
Your shell script is wrong, e.g. it overwrites existing p_2001_*.nc files with wrong input data.
The correct script:
#!/usr/bin/env bash cdo -f nc -setmissval,32768 -setmissval,nan \ -setreftime,2000-12-31,00:00:00,1day\ -settaxis,2001-01-01,12:00:00,1mon \ -setcalendar,standard \ -input,gridfile.txt \ p_2001_1.nc < p_2001_1.csv cdo -f nc -setmissval,32768 -setmissval,nan \ -setreftime,2000-12-31,00:00:00,1day \ -settaxis,2001-02-01,12:00:00,1mon \ -setcalendar,standard \ -input,gridfile.txt \ p_2001_2.nc < p_2001_2.csv cdo -f nc -setmissval,32768 -setmissval,nan \ -setreftime,2000-12-31,00:00:00,1day \ -settaxis,2001-03-0,12:00:00,1mon \ -setcalendar,standard \ -input,gridfile.txt \ p_2001_3.nc < p_2001_3.csv cdo -f nc -setmissval,32768 -setmissval,nan \ -setreftime,2000-12-31,00:00:00,1day\ -settaxis,2001-04-01,12:00:00,1mon \ -setcalendar,standard \ -input,gridfile.txt \ p_2001_4.nc < p_2001_4.csv cdo -f nc -setmissval,32768 -setmissval,nan \ -setreftime,2000-12-31,00:00:00,1day \ -settaxis,2001-05-01,12:00:00,1mon \ -setcalendar,standard \ -input,gridfile.txt \ p_2001_5.nc < p_2001_5.csv cdo -f nc -setmissval,32768 -setmissval,nan \ -setreftime,2000-12-31,00:00:00,1day \ -settaxis,2001-06-01,12:00:00,1mon \ -setcalendar,standard \ -input,gridfile.txt \ p_2001_6.nc < p_2001_6.csv cdo -f nc -setmissval,32768 -setmissval,nan \ -setreftime,2000-12-31,00:00:00,1mon \ -settaxis,2001-07-01,12:00:00,1mon \ -setcalendar,standard \ -input,gridfile.txt \ p_2001_7.nc < p_2001_7.csv cdo -f nc -setmissval,32768 -setmissval,nan \ -setreftime,2000-12-31,00:00:00,1day \ -settaxis,2001-08-01,12:00:00,1mon \ -setcalendar,standard \ -input,gridfile.txt \ p_2001_8.nc < p_2001_8.csv cdo -f nc -setmissval,32768 -setmissval,nan \ -setreftime,2000-12-31,00:00:00,1day \ -settaxis,2001-09-01,12:00:00,1mon \ -setcalendar,standard \ -input,gridfile.txt \ p_2001_9.nc < p_2001_9.csv cdo -f nc -setmissval,32768 -setmissval,nan \ -setreftime,2000-12-31,00:00:00,1day \ -settaxis,2001-10-01,12:00:00,1mon \ -setcalendar,standard \ -input,gridfile.txt \ p_2001_10.nc < p_2001_10.csv cdo -f nc -setmissval,32768 -setmissval,nan \ -setreftime,2000-12-31,00:00:00,1day \ -settaxis,2001-11-01,12:00:00,1mon \ -setcalendar,standard \ -input,gridfile.txt \ p_2001_11.nc < p_2001_11.csv cdo -f nc -setmissval,32768 -setmissval,nan \ -setreftime,2000-12-31,00:00:00,1day \ -settaxis,2001-12-01,12:00:00,1mon \ -setcalendar,standard \ -input,gridfile.txt \ p_2001_12.nc < p_2001_12.csv cdo -O -chname,var1,temp \ -setattribute,FILE=attributes.txt \ -mergetime \ p_2001_*.nc prec.mon.nc
Compare your script with the correct one above.
RE: A problem while converting csv file to netcdf - Added by nazanin tavakoli almost 4 years ago
Yes, you're right, but actually, my main problem was something else. I should transpose my CSV data then use this code. I have already understood what is my problem.
Thank you for your response.
RE: A problem while converting csv file to netcdf - Added by nazanin tavakoli almost 4 years ago
Actually, I have another problem. When I used the code above, I have the following error:
cdo chname (Abort): Too many streams! Operator needs 1 input and 1 output streams.
RE: A problem while converting csv file to netcdf - Added by Karin Meier-Fleischer almost 4 years ago
Sorry, but I can't reproduce the problem.
RE: A problem while converting csv file to netcdf - Added by nazanin tavakoli over 3 years ago
Hi,
I have a question. I want to convert a CSV file to NetCDF by code below, but my final file doesn't have a variable. It contains the lat,lon, and time. Could you please help me?
#!/usr/bin/env bash
cdo -f nc -setmissval,-3.402823060737097e+38 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day\
-settaxis,2001-01-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p1.nc < sand0_5.csv
RE: A problem while converting csv file to netcdf - Added by Karin Meier-Fleischer over 3 years ago
What do you mean with there is no variable? That makes no sense because CDO creates a variable called var1 when
your input files are correct. You didn't use the chname operator to rename the var1 variable contrary to what
we did before (better see above).
Using the files of the zip archive and the command above leads into an error because the gridfile is not correct.
By the way the attributes file is wrong, too. Delete the last line which contains the string EOF.
xsize = 5735 #(not 5734) gridsize = 28353840 #(not 28348896)
gridfile.txt gridtype = lonlat gridsize = 28353840 #28348896 xname = longitude xlongname = longitude xunits = degrees_east yname = latitude ylongname = latitude yunits = degrees_north xsize = 5735 #5734 ysize = 4944 xfirst = 140.9612 xinc = 0.0023 yfirst = -39.1503 yinc = 0.0023
attributes.txt
var1@long_name='sand' var1@units='percent sand' title='percent sand' source='soilsgrid'
bash script
#!/usr/bin/env bash cdo -L -f nc \ -setmissval,-3.402823060737097e+38 -setmissval,nan \ -setreftime,2000-12-31,00:00:00,1day \ -settaxis,2001-01-01,12:00:00,1mon \ -setcalendar,standard \ -input,gridfile.txt \ tmp.nc < sand0_5.csv cdo -chname,var1,sand \ -setattribute,FILE=attributes.txt \ tmp.nc p1.nc rm tmp.nc
cdo sinfon p1.nc
File format : NetCDF2 -1 : Institut Source T Steptype Levels Num Points Num Dtype : Parameter name 1 : unknown soilsgrid v instant 1 1 28353840 1 F32 : sand Grid coordinates : 1 : lonlat : points=28353840 (5735x4944) longitude : 140.9612 to 154.1494 by 0.0023 degrees_east latitude : -39.1503 to -27.7814 by 0.0023 degrees_north Vertical coordinates : 1 : surface : levels=1 Time coordinate : 1 step RefTime = 2000-12-31 00:00:00 Units = days Calendar = standard YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss 2001-01-01 12:00:00 cdo sinfon: Processed 1 variable over 1 timestep [0.01s 7292KB].
cdo infon p1.nc
-1 : Date Time Level Gridsize Miss : Minimum Mean Maximum : Parameter name 1 : 2001-01-01 12:00:00 0 28353840 8590646 : 0.0000 58.246 91.900 : sand cdo infon: Processed 28353840 values from 1 variable over 1 timestep [0.16s 115MB].
RE: A problem while converting csv file to netcdf - Added by nazanin tavakoli over 3 years ago
yes, you're right. GDAL had made a mistake while it was converting tiff data to NetCDF. It should be 5734.
I have a question. I want to convert 6 CSV files to a NetCDF. These data are containing clay information, so all 6 files should have the same time, but they are different in layers. I have converted them to a NetCDF file, but I have a question. How can I set values for the variable layer(sfc)? I have 6 layers and I want to set constant value for layer1=10, layer2= 25, layer3=50, and so on. Could you please help me? I have sent my code, data, and a sample file which is called soita.sand.nc. If you check the layer in this sample, you will understand what I mean.
I would appreciate your help.
https://drive.google.com/file/d/1HzIE3NkJJ8WJydYg9kwiYUYavXoSRBL8/view?usp=sharing
RE: A problem while converting csv file to netcdf - Added by nazanin tavakoli over 3 years ago
I have already understood that my code (cdo merge p1.nc p2.nc p3.nc p4.nc p5.nc p6.nc out.nc) didn't merge NetCDF files. out.nc shows the first NetCDF files (p1.nc)