Project

General

Profile

A problem while converting csv file to netcdf

Added by nazanin tavakoli almost 4 years ago

Hello,
I have 12 monthly precipitation data in 2001 which are in CSV format, and I want to convert them to a NetCDF file with the code below:
cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day\
-settaxis,2001-01-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_1.nc < p_2001_1.csv

cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day \
-settaxis,2001-02-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_2.nc < p_2001_2.csv

cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day \
-settaxis,2001-03-0,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_3.nc < p_2001_3.csv

cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day\
-settaxis,2001-04-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_4.nc < p_2001_4.csv

cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day \
-settaxis,2001-05-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_5.nc < p_2001_5.csv

cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day \
-settaxis,2001-06-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_6.nc < p_2001_6.csv

cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1mon \
-settaxis,2001-07-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_7.nc < p_2001_7.csv

cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day \
-settaxis,2001-08-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_8.nc < p_2001_8.csv

cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day \
-settaxis,2001-09-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_9.nc < p_2001_9.csv

cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day \
-settaxis,2001-10-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_1.nc < p_2001_1.csv

cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day \
-settaxis,2001-01-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_10.nc < p_2001_10.csv

cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day \
-settaxis,2001-11-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_11.nc < p_2001_11.csv

cdo -f nc -setmissval,32768 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day \
-settaxis,2001-12-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p_2001_12.nc < p_2001_12.csv

cdo -O -chname,var1,temp \
-setattribute,FILE=attributes.txt \
-mergetime \
p_2001_*.nc prec.mon.nc

but, the grid of all my cells are changing in the final NetCDF file. I mean that if you consider a grid cell with the special latitude and longitude, the value of this cell in the NetCDF file is not the same as the CSV file. Also, if you open p_2001_1.nc, you will see that the time of this NetCDF file is 2001-10-01 which is not the time I set in the code.
Could anybody help me?

prcp.rar (1.02 MB) prcp.rar

Replies (8)

RE: A problem while converting csv file to netcdf - Added by Karin Meier-Fleischer almost 4 years ago

When you encounter problems using a shell script to call cdo, you have to check your shell script first.
Your shell script is wrong, e.g. it overwrites existing p_2001_*.nc files with wrong input data.

The correct script:

#!/usr/bin/env bash

cdo -f nc -setmissval,32768 -setmissval,nan \
          -setreftime,2000-12-31,00:00:00,1day\
          -settaxis,2001-01-01,12:00:00,1mon \
          -setcalendar,standard \
          -input,gridfile.txt \
          p_2001_1.nc < p_2001_1.csv

cdo -f nc -setmissval,32768 -setmissval,nan \
          -setreftime,2000-12-31,00:00:00,1day \
          -settaxis,2001-02-01,12:00:00,1mon \
          -setcalendar,standard \
          -input,gridfile.txt \
          p_2001_2.nc < p_2001_2.csv

cdo -f nc -setmissval,32768 -setmissval,nan \
          -setreftime,2000-12-31,00:00:00,1day \
          -settaxis,2001-03-0,12:00:00,1mon \
          -setcalendar,standard \
          -input,gridfile.txt \
          p_2001_3.nc < p_2001_3.csv

cdo -f nc -setmissval,32768 -setmissval,nan \
          -setreftime,2000-12-31,00:00:00,1day\
          -settaxis,2001-04-01,12:00:00,1mon \
          -setcalendar,standard \
          -input,gridfile.txt \
          p_2001_4.nc < p_2001_4.csv

cdo -f nc -setmissval,32768 -setmissval,nan \
          -setreftime,2000-12-31,00:00:00,1day \
          -settaxis,2001-05-01,12:00:00,1mon \
          -setcalendar,standard \
          -input,gridfile.txt \
          p_2001_5.nc < p_2001_5.csv

cdo -f nc -setmissval,32768 -setmissval,nan \
          -setreftime,2000-12-31,00:00:00,1day \
          -settaxis,2001-06-01,12:00:00,1mon \
          -setcalendar,standard \
          -input,gridfile.txt \
          p_2001_6.nc < p_2001_6.csv

cdo -f nc -setmissval,32768 -setmissval,nan \
          -setreftime,2000-12-31,00:00:00,1mon \
          -settaxis,2001-07-01,12:00:00,1mon \
          -setcalendar,standard \
          -input,gridfile.txt \
          p_2001_7.nc < p_2001_7.csv

cdo -f nc -setmissval,32768 -setmissval,nan \
          -setreftime,2000-12-31,00:00:00,1day \
          -settaxis,2001-08-01,12:00:00,1mon \
          -setcalendar,standard \
          -input,gridfile.txt \
          p_2001_8.nc < p_2001_8.csv

cdo -f nc -setmissval,32768 -setmissval,nan \
          -setreftime,2000-12-31,00:00:00,1day \
          -settaxis,2001-09-01,12:00:00,1mon \
          -setcalendar,standard \
          -input,gridfile.txt \
          p_2001_9.nc < p_2001_9.csv

cdo -f nc -setmissval,32768 -setmissval,nan \
          -setreftime,2000-12-31,00:00:00,1day \
          -settaxis,2001-10-01,12:00:00,1mon \
          -setcalendar,standard \
          -input,gridfile.txt \
          p_2001_10.nc < p_2001_10.csv

cdo -f nc -setmissval,32768 -setmissval,nan \
          -setreftime,2000-12-31,00:00:00,1day \
          -settaxis,2001-11-01,12:00:00,1mon \
          -setcalendar,standard \
          -input,gridfile.txt \
          p_2001_11.nc < p_2001_11.csv

cdo -f nc -setmissval,32768 -setmissval,nan \
          -setreftime,2000-12-31,00:00:00,1day \
          -settaxis,2001-12-01,12:00:00,1mon \
          -setcalendar,standard \
          -input,gridfile.txt \
          p_2001_12.nc < p_2001_12.csv

cdo -O -chname,var1,temp \
    -setattribute,FILE=attributes.txt \
    -mergetime \
     p_2001_*.nc prec.mon.nc

Compare your script with the correct one above.

RE: A problem while converting csv file to netcdf - Added by nazanin tavakoli almost 4 years ago

Yes, you're right, but actually, my main problem was something else. I should transpose my CSV data then use this code. I have already understood what is my problem.
Thank you for your response.

RE: A problem while converting csv file to netcdf - Added by nazanin tavakoli almost 4 years ago

Actually, I have another problem. When I used the code above, I have the following error:

cdo chname (Abort): Too many streams! Operator needs 1 input and 1 output streams.

test.zip (786 Bytes) test.zip

RE: A problem while converting csv file to netcdf - Added by Karin Meier-Fleischer almost 4 years ago

Sorry, but I can't reproduce the problem.

RE: A problem while converting csv file to netcdf - Added by nazanin tavakoli over 3 years ago

Hi,
I have a question. I want to convert a CSV file to NetCDF by code below, but my final file doesn't have a variable. It contains the lat,lon, and time. Could you please help me?

#!/usr/bin/env bash
cdo -f nc -setmissval,-3.402823060737097e+38 -setmissval,nan \
-setreftime,2000-12-31,00:00:00,1day\
-settaxis,2001-01-01,12:00:00,1mon \
-setcalendar,standard \
-input,gridfile.txt \
p1.nc < sand0_5.csv

sand1.zip (20.4 MB) sand1.zip

RE: A problem while converting csv file to netcdf - Added by Karin Meier-Fleischer over 3 years ago

What do you mean with there is no variable? That makes no sense because CDO creates a variable called var1 when
your input files are correct. You didn't use the chname operator to rename the var1 variable contrary to what
we did before (better see above).

Using the files of the zip archive and the command above leads into an error because the gridfile is not correct.
By the way the attributes file is wrong, too. Delete the last line which contains the string EOF.

xsize    = 5735  #(not 5734)
gridsize = 28353840  #(not 28348896)
gridfile.txt
gridtype  = lonlat
gridsize  = 28353840 #28348896
xname     = longitude
xlongname = longitude
xunits    = degrees_east
yname     = latitude
ylongname = latitude
yunits    = degrees_north
xsize     = 5735 #5734
ysize     = 4944
xfirst    = 140.9612
xinc      = 0.0023
yfirst    = -39.1503   
yinc      = 0.0023

attributes.txt

var1@long_name='sand'
var1@units='percent sand'
title='percent sand'
source='soilsgrid'

bash script

#!/usr/bin/env bash

cdo -L -f nc \
    -setmissval,-3.402823060737097e+38 -setmissval,nan \
    -setreftime,2000-12-31,00:00:00,1day \
    -settaxis,2001-01-01,12:00:00,1mon \
    -setcalendar,standard \
    -input,gridfile.txt \
    tmp.nc < sand0_5.csv

cdo -chname,var1,sand \
    -setattribute,FILE=attributes.txt \
    tmp.nc p1.nc

rm tmp.nc

cdo sinfon p1.nc

   File format : NetCDF2
    -1 : Institut Source   T Steptype Levels Num    Points Num Dtype : Parameter name
     1 : unknown  soilsgrid v instant       1   1  28353840   1  F32  : sand          
   Grid coordinates :
     1 : lonlat                   : points=28353840 (5735x4944)
                        longitude : 140.9612 to 154.1494 by 0.0023 degrees_east
                         latitude : -39.1503 to -27.7814 by 0.0023 degrees_north
   Vertical coordinates :
     1 : surface                  : levels=1
   Time coordinate :  1 step
     RefTime =  2000-12-31 00:00:00  Units = days  Calendar = standard
  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss
  2001-01-01 12:00:00
cdo    sinfon: Processed 1 variable over 1 timestep [0.01s 7292KB].

cdo infon p1.nc

    -1 :       Date     Time   Level Gridsize    Miss :     Minimum        Mean     Maximum : Parameter name
     1 : 2001-01-01 12:00:00       0 28353840 8590646 :      0.0000      58.246      91.900 : sand          
cdo    infon: Processed 28353840 values from 1 variable over 1 timestep [0.16s 115MB].

RE: A problem while converting csv file to netcdf - Added by nazanin tavakoli over 3 years ago

yes, you're right. GDAL had made a mistake while it was converting tiff data to NetCDF. It should be 5734.
I have a question. I want to convert 6 CSV files to a NetCDF. These data are containing clay information, so all 6 files should have the same time, but they are different in layers. I have converted them to a NetCDF file, but I have a question. How can I set values for the variable layer(sfc)? I have 6 layers and I want to set constant value for layer1=10, layer2= 25, layer3=50, and so on. Could you please help me? I have sent my code, data, and a sample file which is called soita.sand.nc. If you check the layer in this sample, you will understand what I mean.
I would appreciate your help.
https://drive.google.com/file/d/1HzIE3NkJJ8WJydYg9kwiYUYavXoSRBL8/view?usp=sharing

RE: A problem while converting csv file to netcdf - Added by nazanin tavakoli over 3 years ago

I have already understood that my code (cdo merge p1.nc p2.nc p3.nc p4.nc p5.nc p6.nc out.nc) didn't merge NetCDF files. out.nc shows the first NetCDF files (p1.nc)

    (1-8/8)