CDO

Hi Nazanin,

there are already some explainations in the forums about this conversion.
First, you have to write a grid descriptor file.

gridfile.txt:

gridtype = lonlat
gridsize = 259200
xname = lon
xlongname = longitude
xunits = degrees_east
yname = lat
ylongname = latitude
yunits = degrees_north
xsize = 720
ysize = 360
xfirst = -179.75
xinc = 0.5
yfirst = -89.75
yinc = 0.5

Then convert each single CSV file to netCDF.

cdo -f nc -setreftime,1900-01-01,00:00:00,1day \
          -settaxis,1901-01-01,12:00:00,1day \
          -setcalendar,standard \
          -input,gridfile.txt \
          1901_1.nc < CSV_TO_nc/1901_1.csv

Use mergetime operator to merge all netCDF files to one single file.

-Karin

Thank you for answering my question but I have a problem. When I convert a CSV file to NetCDF, then I open the NetCDF file it is completely different from the CSV file. I have sent them for you, and if you open them and test them for a specific point, you will understand that the NetCDF file is not as same as the CSV file. Could you please help me? I really need help.
Also, I have a question. If you open the NetCDF file, you will see a variable called var1. I want to change it to tmp which has the following properties:
tmp
Size: 720x360x6
Dimensions: longitude, latitude, time
Datatype: single
Attributes:
long_name = 'monthly mean temperature'
units = 'degrees Celsius'

The problem is that you want to reorder the dimensions of the variable from (time,lat,lon) to
(lon,lat,time) which is not CF conform. CDO can't do it.

You can use NCOs ncpdq to reorder the dimensions but the time dimension will be set to fixed
record instead of unlimited record.

In the following example for each CSV file a netCDF file is generated. All netCDF files are merged into
one file, the variable name is changed to tmp, and and the attributes long_name and units are set.

The last two steps reorder the dimensions and set the lon dimension to fixed record.

cdo -f nc -setreftime,1900-01-01,00:00:00,1day \
          -settaxis,1901-01-01,12:00:00,1day \
          -setcalendar,standard \
          -input,gridfile.txt \
          1901_1.nc < CSV_TO_nc/1901_1.csv

cdo -f nc -setreftime,1900-01-01,00:00:00,1day \
          -settaxis,1901-01-02,12:00:00,1day \
          -setcalendar,standard \
          -input,gridfile.txt \
          1901_2.nc < CSV_TO_nc/1901_2.csv

cdo -f nc -setreftime,1900-01-01,00:00:00,1day \
          -settaxis,1901-01-03,12:00:00,1day \
          -setcalendar,standard \
          -input,gridfile.txt \
          1901_3.nc < CSV_TO_nc/1901_3.csv
...

cdo -O -chname,var1,tmp \
       -setattribute,var1@long_name='monthly mean temperature',var1@units='degrees Celsius' \
       -mergetime \
       1901_*.nc 1901.nc

ncpdq -O -a lon,lat,time 1901.nc tmp.nc
ncks -O --fix_rec_dmn lon tmp.nc 1901reorder.nc

ncdump -h 1901reorder.nc

ncdump -h 1901reorder.nc
netcdf \1901reorder {
dimensions:
    lat = 360 ;
    lon = 720 ;
    time = 3 ;
variables:
    double lat(lat) ;
        lat:standard_name = "latitude" ;
        lat:long_name = "latitude" ;
        lat:units = "degrees_north" ;
        lat:axis = "Y" ;
    double lon(lon) ;
        lon:standard_name = "longitude" ;
        lon:long_name = "longitude" ;
        lon:units = "degrees_east" ;
        lon:axis = "X" ;
    double time(time) ;
        time:standard_name = "time" ;
        time:units = "days since 1900-1-1 00:00:00" ;
        time:calendar = "standard" ;
        time:axis = "T" ;
    float tmp(lon, lat, time) ;
        tmp:code = 1 ;
        tmp:_FillValue = -9.e+33f ;
        tmp:missing_value = -9.e+33f ;
        tmp:long_name = "monthly mean temperature" ;
        tmp:units = "degrees Celsius" ;

// global attributes:
        :CDI = "Climate Data Interface version 1.9.8 (https://mpimet.mpg.de/cdi)" ;
        :Conventions = "CF-1.6" ;
        :history = "Wed Dec 16 13:57:24 2020: ncks -O --fix_rec_dmn lon tmp.nc 1901reorder.nc\n",
            "Wed Dec 16 13:57:24 2020: ncpdq -O -a lon,lat,time 1901.nc tmp.nc\n",
            "Wed Dec 16 13:57:24 2020: cdo -O -chname,var1,tmp -setattribute,var1@long_name=monthly mean temperature,var1@units=degrees Celsius -mergetime 1901_1.nc 1901_2.nc 1901_3.nc 1901.nc\n",
            "Wed Dec 16 13:57:24 2020: cdo -f nc -setreftime,1900-01-01,00:00:00,1day -settaxis,1901-01-01,12:00:00,1day -setcalendar,standard -input,gridfile.txt 1901_1.nc" ;
        :CDO = "Climate Data Operators version 1.9.8 (https://mpimet.mpg.de/cdo)" ;
        :NCO = "netCDF Operators version 4.9.1 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)" ;
}

I have tried the code in Cygwin64 Terminal in windows 10 but unfortunately, when I write these 2 code (ncpdq -O -a lon,lat,time 1901.nc tmp.nc) and (ncks -O --fix_rec_dmn lon tmp.nc 1901reorde.nc), the terminal shows me this error:
bash: ncpdq: command not found
bash: ncks: command not found
Could you please help me? I am new to this.

You have to install NCO see http://nco.sourceforge.net/

yes, I have already installed it. I have a question. when I write this code (ncks -O --fix_rec_dmn lon tmp.nc 1901reorder.nc), I have the following error:
ncks: ERROR received 3 filenames; need no more than two

I have installed nco version 4.9.5, then I do not have this error anymore: ncks: ERROR received 3 filenames; need no more than two
But I still have a previous problem. I have sent 3 files for you. If you open a jpg file, you will understand that when I convert my CSV files to NetCDF, my points completely change. For example, for a specific point which has longitude=160.25 and latitude= -81.25 in CSV file named 1901_1, the value is NaN.(NaN means that this point does not have any value)
If you open 1901reorder.nc for this point in this time, it has the value of -23.2 which is totally wrong. It should be NaN like the main CSV file(1901_1).
Do you have any idea how can I fix this problem?

The CSV data has the wrong order itself! Each file contains 720 rows and 360 columns but it has to be 360 rows (lat) and 720 columns (lon).

Here is my Korn-Shell script to do everything for the three 1901_*.csv files:

#!/bin/ksh

# original CSV file
#
#  720 lines
#  360 columns
# 
# it has to be transposed to 360 lines and 720 colimns

# transposing rows and columns

awk -F, '{for (i=1;i<=NF; i++) a[i,NR]=$i; max=(max<NF?NF:max)} END \
         {for (i=1; i<=max; i++) \
            {for (j=1; j<=NR; j++) printf "%s%s", a[i,j], (j==NR?RS:FS) }}' \
          1901_1.csv > trans_1901_1.csv

awk -F, '{for (i=1;i<=NF; i++) a[i,NR]=$i; max=(max<NF?NF:max)} END \
         {for (i=1; i<=max; i++) \
            {for (j=1; j<=NR; j++) printf "%s%s", a[i,j], (j==NR?RS:FS) }}' \
          1901_2.csv > trans_1901_2.csv

awk -F, '{for (i=1;i<=NF; i++) a[i,NR]=$i; max=(max<NF?NF:max)} END \
         {for (i=1; i<=max; i++) \
            {for (j=1; j<=NR; j++) printf "%s%s", a[i,j], (j==NR?RS:FS) }}' \
          1901_3.csv > trans_1901_3.csv

# convert CSV data to netCDF

cdo -f nc -setmissval,1e20 -setmissval,nan \
          -setreftime,1900-01-01,00:00:00,1day \
          -settaxis,1901-01-01,12:00:00,1day \
          -setcalendar,standard \
          -input,gridfile.txt \
          1901_1.nc < trans_1901_1.csv

cdo -f nc -setmissval,1e20 -setmissval,nan \
          -setreftime,1900-01-01,00:00:00,1day \
          -settaxis,1901-01-02,12:00:00,1day \
          -setcalendar,standard \
          -input,gridfile.txt \
          1901_2.nc < trans_1901_2.csv

cdo -f nc -setmissval,1e20 -setmissval,nan \
          -setreftime,1900-01-01,00:00:00,1day \
          -settaxis,1901-01-03,12:00:00,1day \
          -setcalendar,standard \
          -input,gridfile.txt \
          1901_3.nc < trans_1901_3.csv

cdo -O -chname,var1,tmp \
       -setattribute,var1@long_name='monthly mean temperature',var1@units='degrees Celsius' \
       -mergetime \
       1901_*.nc 1901.nc

# reorder the coordinate dimensions to (lon,lat,time). (Why is it needed?)

ncpdq -O -a lon,lat,time 1901.nc tmp.nc
ncks -O --fix_rec_dmn lon tmp.nc 1901reorder.nc

The following plot displays the data of 1901.nc following the CF-convention, first timestep, created by NCL's ncl_quicklook command line tool!

ncl_quicklook -o png 1901.nc

-Karin

Thank you for your help. I have some questions. I want to add title, source, and calendar to my Global Attributes. Also, I want to change the units of time to be 'days since 1900-1-1' and the standard_name of the time variable to long_name like below. My data is monthly and I don't want to have a clock in my dataset(00:00:00). How can I fix them?

Global Attributes:
title = 'monthly temperature climatology, 1900-99'
source = 'CRU'
calendar = 'gregorian'

Variables:
time

standard_name = 'time' >>>>>>>>>>>>>>>>>>   (long_name='time')
                       units         = 'days since 1900-1-1 00:00:00' >>>>>>>>>>>>>>>>>units='days since 1900-1-1'

You can add or change attributes with the setattribute operator (see https://code.mpimet.mpg.de/projects/cdo/embedded/cdo.pdf#subsection.2.6.1).

The time unit should not be changed in any case! The time depends on hh:mm:ss (see https://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/build/ch04s04.html).

When I want to change the standard_name to long_name in the Attributes of time-variable with the following code, it shows me a warning, and it doesn't change.

cdo setattribute,time@long_name='time' 1901_fina.nc 1901_final.nc
cdo setattribute (Warning): Variable >time< not found!

Variables:
time
Size: 6x1
Dimensions: time
Datatype: double
Attributes:
standard_name = 'time'
units = 'days since 1900-1-1 00:00:00'
calendar = 'standard'
axis = 'T'

Sorry, I've forgotten to mention that time is a dimension variable which CDO can't change. You have to use NCO's ncatted tool.

Adding the long_name to the file:

ncatted -O -a long_name,time,c,c,"time" 1901.nc

I wanted to convert 12 other CSV files to NetCDF and I used the following code but the latitude changed from 89.75 to 269.25 (it should changes from 89.75 to -89.75) and the value of cells is totally wrong. I have sent my files and my code here. Could you please help me?

$ cdo -f nc -setmissval,9e20 -setmissval,NaN \

-setreftime,1900-01-01,00:00:00,1day \
-settaxis,1901-01-01,12:00:00,1day \
-setcalendar,standard \
-input,gridfile.txt \
ws_2009_1.nc < ws_2009_1.csv

....
cdo -O -chname,var1,temp \

-setattribute,var1@long_name='monthly mean temperature',var1@units='degress Celsius' \
-mergetime \
ws_2009_*.nc temp.mon.nc

cdo setattribute,title='monthly temperature climatology' temp.mon.nc temp.mon1.nc
cdo setattribute: Processed 1 variable [0.03s 9364KB]
Nazanin@DESKTOP-UQT97M1 /
$ cdo setattribute,source='CRU' temp.mon1.nc temp.mon12.nc
cdo setattribute: Processed 1 variable [0.03s 9344KB]
Nazanin@DESKTOP-UQT97M1 /
$ cdo setattribute,calendar='gregorian' temp.mon12.nc temp.monf.nc

temp.monf.nc is my final file.

1. The yinc value in gridfile.txt has to be -0.5 not 0.5.
2. The CSV data is not the same as 1901_1.csv etc. from the first issue, they seem to be transposed already.

#!/bin/ksh

# convert CSV data to netCDF

typeset -RZ2 MON

files=$(ls ws_2009_*.csv)

i=1
for f in $files
do
    out=${f%.csv}.nc
    MON=${i}

    cdo -f nc -setmissval,9e+20 -setmissval,nan \
          -setreftime,1900-01-01,00:00:00,1day \
          -settaxis,1901-${MON}-01,12:00:00,1day \
          -setcalendar,standard \
          -input,grid.txt \
          ${out} < ${f}
    i=$(expr $i + 1)
done

# set attributes

cat << EOF > attributes.txt
var1@long_name='monthly mean temperature'
var1@units='degress Celsius'
title='monthly temperature climatology'
source='CRU'
calendar='gregorian'
EOF

cdo -O -chname,var1,temp \
    -setattribute,FILE=attributes.txt \
    -mergetime \
     ws_2009_*.nc temp.mon.nc

I understand. I should change yinc to -0.5.
I have another question:D. I write my code in Cygwin (Windows batch file) and it is really hard to write each code every time because I have a lot of files to convert to NetCDF. So, Could you please help me or recommend me to install the shell script which I can write the previous code? I am using CDO in windows 10.

How can I add variables like level,time_weights, and date like what I have sent below to my NetCDF file?

Variables:
longitude
Size: 720x1
Dimensions: longitude
Datatype: single
Attributes:
long_name = 'longitude'
units = 'degrees_east'
latitude
Size: 360x1
Dimensions: latitude
Datatype: single
Attributes:
long_name = 'latitude'
units = 'degrees_north'
level
Size: 1x1
Dimensions: level
Datatype: single
Attributes:
long_name = '_'
units = 'level'
time
Size: 12x1
Dimensions: time
Datatype: single
Attributes:
long_name = 'time'
units = 'days since 1960-12-31'
time_weights
Size: 12x1
Dimensions: time
Datatype: single
Attributes:
long_name = 'number of days per time step'
units = 'days'
date
Size: 10x12
Dimensions: lengthd,time
Datatype: char
Attributes:
long_name = 'label for each time step'
units = ''
temp
Size: 720x360x1x12
Dimensions: longitude,latitude,level,time
Datatype: single
Attributes:
long_name = 'monthly mean temperature'
units = 'degC'
missing_value = 8.999999828524175e+20

nazanin tavakoli wrote:

I understand. I should change yinc to -0.5.
I have another question:D. I write my code in Cygwin (Windows batch file) and it is really hard to write each code every time because I have a lot of files to convert to NetCDF. So, I used bash script and I have the following error:

$ ./first.sh.txt

cdo (Abort): Too few streams specified! Operator -setmissval,nan needs 1 input and 1 output streams.
./first.sh.txt: line 3: -setreftime,1900-01-01,00:00:00,1day: command not found
./first.sh.txt: line 4: -settaxis,1901-01-01,12:00:00,1day: command not found
./first.sh.txt: line 5: -setcalendar,standard: command not found
./first.sh.txt: line 6: -input,gridfile.txt: command not found
: No such file or directory2009_1.csv

cdo (Abort): Too few streams specified! Operator -setmissval,nan needs 1 input and 1 output streams.
./first.sh.txt: line 9: -setreftime,1900-01-01,00:00:00,1day: command not found
./first.sh.txt: line 10: -settaxis,1901-02-01,12:00:00,1day: command not found
./first.sh.txt: line 11: -setcalendar,standard: command not found
./first.sh.txt: line 12: -input,gridfile.txt: command not found
: No such file or directory_2009_2.csv
./first.sh.txt: line 24: warning: here-document at line 14 delimited by end-of-file (wanted `EOF')

I have sent my files here.

Your bash script file is corrupted! I am not able to run it, too. I don't know what you have done but maybe it is a problem with Cygwin (I'm working on Mac OSX).

I was able to write a correct script with:

1. cat first.sh.txt      (the name does not make sense; is it or is it not a shell script file?)

2. copy the content CTRL-c

3. open a new file in an editor   (in your case an editor in Cygwin)

4. paste the content with CTRL-v

5. save the file  (name it first.sh)

6. run it in a terminal  (bash first.sh)

Please, next time open a new issue if you have another question which is not directly connected to your last question. It is easier to find.

You can set the level description with setzaxis.

Here is an example for a description of 7 pressure level in hPa:

zaxistype = pressure
size      = 7
name      = lev
longname  = pressure
units     = hPa
levels    = 850 500 200 100 50 20 10

Store the above lines in a file, e.g. myzaxis and pass it to setzaxis:

cdo -setzaxis,myzaxis  infile  outfile

Sorry, but I don't know how to add the time_weights.

Hello again,
I have a question about converting CSV files to NetCDF. I have six CSV files which are describing the percentage of clay in 6 layers. The resolution of my data is 250meter (2.3*10^-3 degrees), and the first point of my data has latitude=-39.1503 and longitude= 140.9612.Finally, I want to have netcdf file which has the following properties:
Global Attributes:
title = 'percent clay'
source = 'soilsGrid'
history = 'JNUARY2001'
calendar='none'
Dimensions:
longitude = 5734
latitude = 4944
layer = 6
time = 1
lengthd = 19
Variables:
longitude
Size: 5734x1
Dimensions: longitude
Datatype: double
Attributes:
longname='longitude'
units = 'degrees-east'
latitude
Size: 4944x1
Dimensions: latitude
Datatype: double
Attributes:
longname='latitude'
units = 'degress-north'
layer
Size: 6x1
Dimensions: layer
Datatype: double
Attributes:
long_name='depth of bottom of soil layer, top=0 cm'
units = 'cm'
time
Size: 1x1
Dimensions: time
Datatype: double
Attributes:
long_name='time'
units = 'days since 2000-12-31'
claypct
Size: 5734x4944x6x1
Dimensions: longitude,latitude,layer,time
Datatype: double
Attributes:
longname='percent clay'
units = ''
missing_value=
Here is my data:
https://drive.google.com/file/d/1DwYqC0V6kADWhBG3zQw3rg1HRZdQ0uxJ/view?usp=sharing

Could you please help me? I really need help.