Concatenate NC file

Added by Pierre LACOSTE almost 8 years ago

Hello,

I'm trying to concatenate multiple "nc" files which contain exactly the same variables with CDO but I have some problems with this.

I first use

cdo merge file1.nc file2.nc file3.nc fileout.nc

But at the end of the execution, I got the message

Warning (cdfDefVar) : Changed double entry of variable name 'ssh' to 'ssh_2'! Warning (cdfDefVar) : Changed multiple entry of variable name 'ssh' to 'ssh_3'! cdo merge: Processed 7841522 values from 3 variables over 3 timesteps ( 0.44s )

And I doesn't want to have the variable "ssh" renamed for the data in the files file2.nc and file2.nc

I found another command in the document which is "cat". I tried this command instead of merge and I got the message
cdo cat (Abort): Grid size of the input parameters do not match!

How can I use CDO to concatenate nc files?

Thank you for your help.

Pierre

Replies (12)

RE: Concatenate NC file - Added by Ralf Mueller almost 8 years ago

cdo cat ifile0 ... ifileN ofile

docu

RE: Concatenate NC file - Added by Pierre LACOSTE almost 8 years ago

Hello Ralf,

I tried your solution and now I got the following message:
cdo cat (Abort): Grid size of the input parameters do not match!

Do you have an idea where the problem comes?

Thank you,

Pierre

RE: Concatenate NC file - Added by Ralf Mueller almost 8 years ago

in that case I need the input files, or at least some of them to reproduce the error.

In general this message has to do with variables with the

the same name in different files
different grid size in different files

but without the data, it's hard to say. Please not, that in most cases, the first timestep of each input file is sufficient.

best
ralf

RE: Concatenate NC file - Added by Pierre LACOSTE almost 8 years ago

Hello Ralf,

Sorry for the delay from your last reply.
I provide you the two files that I try to concatenate.

Thank you for your help.

Pierre

NcFileToMerge.zip (6.94 MB) NcFileToMerge.zip

RE: Concatenate NC file - Added by Ralf Mueller almost 8 years ago

Here are the grids of your three input files

ram@luthien:~/Downloads/NcFileToMerge cdo griddes 0-HR_MOD_1467706576365.nc                                                                                                                                                [12:46:10|16-07-19]
#
# gridID 1
#
gridtype  = lonlat
gridsize  = 3183960
xname     = longitude
xlongname = longitude
xunits    = degrees_east
yname     = latitude
ylongname = latitude
yunits    = degrees_north
xsize     = 1560
ysize     = 2041
xfirst    = 50
xinc      = 0.08333335
yfirst    = -80
yinc      = 0.08333588
cdo griddes: Processed 1 variable ( 0.00s )
ram@luthien:~/Downloads/NcFileToMerge cdo griddes 1-HR_MOD_1467706576365.nc                                                                                                                                                [12:46:41|16-07-19]
#
# gridID 1
#
gridtype  = lonlat
gridsize  = 4410601
xname     = longitude
xlongname = longitude
xunits    = degrees_east
yname     = latitude
ylongname = latitude
yunits    = degrees_north
xsize     = 2161
ysize     = 2041
xfirst    = -180
xinc      = 0.08333333
yfirst    = -80
yinc      = 0.08333588
cdo griddes: Processed 1 variable ( 0.00s )
ram@luthien:~/Downloads/NcFileToMerge cdo griddes 2-HR_MOD_1467706576365.nc                                                                                                                                                [12:46:44|16-07-19]
#
# gridID 1
#
gridtype  = lonlat
gridsize  = 246961
xname     = longitude
xlongname = longitude
xunits    = degrees_east
yname     = latitude
ylongname = latitude
yunits    = degrees_north
xsize     = 121
ysize     = 2041
xfirst    = 0
xinc      = 0.08333333
yfirst    = -80
yinc      = 0.08333588
cdo griddes: Processed 1 variable ( 0.00s )

All of them have different size but the same variable name (ssh). Hence a concatenation does not work, because in netcdf the shornames have to be uniq (not the shortnames together with some dimension info). By renaming, you can still put then into a single file

ram@luthien:~/Downloads/NcFileToMerge cdo infov 0-HR_MOD_1467706576365.nc                                                                                                                                                  [12:48:46|16-07-19]
    -1 :       Date     Time   Level Gridsize    Miss :     Minimum        Mean     Maximum : Parameter name
     1 : 2016-06-12 12:00:00       0  3183960 1076869 :     -1.8320   0.0039747      1.6297 : ssh           
cdo infon: Processed 3183960 values from 1 variable over 1 timestep ( 0.01s )
ram@luthien:~/Downloads/NcFileToMerge cdo infov 1-HR_MOD_1467706576365.nc                                                                                                                                                  [12:49:40|16-07-19]
    -1 :       Date     Time   Level Gridsize    Miss :     Minimum        Mean     Maximum : Parameter name
     1 : 2016-06-12 12:00:00       0  4410601  993417 :     -1.9581    -0.18370     0.97781 : ssh           
cdo infon: Processed 4410601 values from 1 variable over 1 timestep ( 0.02s )
ram@luthien:~/Downloads/NcFileToMerge cdo infov 2-HR_MOD_1467706576365.nc                                                                                                                                                  [12:49:43|16-07-19]
    -1 :       Date     Time   Level Gridsize    Miss :     Minimum        Mean     Maximum : Parameter name
     1 : 2016-06-12 12:00:00       0   246961   79085 :     -1.9663    -0.54538     0.45228 : ssh           
cdo infon: Processed 246961 values from 1 variable over 1 timestep ( 0.00s )
ram@luthien:~/Downloads/NcFileToMerge cdo merge -chname,ssh,ssh_0 0-HR_MOD_1467706576365.nc -chname,ssh,ssh_1 1-HR_MOD_1467706576365.nc -chname,ssh,ssh_2 2-HR_MOD_1467706576365.nc out.nc                                 [12:49:50|16-07-19]
cdo merge: Started child process "chname,ssh,ssh_0 0-HR_MOD_1467706576365.nc (pipe1.1)".
cdo merge: Started child process "chname,ssh,ssh_1 1-HR_MOD_1467706576365.nc (pipe1.2)".
cdo merge: Started child process "chname,ssh,ssh_2 2-HR_MOD_1467706576365.nc (pipe1.3)".
cdo(2) chname: Processed 3183960 values from 1 variable over 1 timestep ( 0.26s )
cdo(3) chname: Processed 4410601 values from 1 variable over 1 timestep ( 0.26s )
cdo(4) chname: Processed 246961 values from 1 variable over 1 timestep ( 0.26s )
cdo merge: Processed 7841522 values from 3 variables over 3 timesteps ( 0.26s )
ram@luthien:~/Downloads/NcFileToMerge nnc out.nc                                                                                                                                                                           [12:50:35|16-07-19]
ram@luthien:~/Downloads/NcFileToMerge cdo infov out.nc                                                                                                                                                                     [12:50:47|16-07-19]
    -1 :       Date     Time   Level Gridsize    Miss :     Minimum        Mean     Maximum : Parameter name
     1 : 2016-06-12 12:00:00       0  3183960 1076869 :     -1.8320   0.0039747      1.6297 : ssh_0         
     2 : 2016-06-12 12:00:00       0  4410601  993417 :     -1.9581    -0.18370     0.97781 : ssh_1         
     3 : 2016-06-12 12:00:00       0   246961   79085 :     -1.9663    -0.54538     0.45228 : ssh_2         
cdo infon: Processed 7841522 values from 3 variables over 1 timestep ( 0.04s )

hth
ralf

RE: Concatenate NC file - Added by Pierre LACOSTE almost 8 years ago

Hello ralf,

Thank you for your response.
The problem with the rename of the variable is that create 3 differents variables on the result file.
And, when I use the tool panoply, to create a plot, it's not possible to have it at the same time for the 3 variables?

The 3 files that I provide have not crossing area. The file _0 is from lat0 to lat1, the file _1 is from lat1 to lat2 and the file _3 is from lat2 to lat0.
I hoped that it was possible to have a merge file with the values of the variable ssh put into the same file from lat0 to lat0.

Do you think what I need to do is possible?

Thank you,

Pierre.

RE: Concatenate NC file - Added by Ralf Mueller almost 8 years ago

Hi Pierre!

sorry for the miss-understanding - merge can so many different things. If I got it right, you want to merge the grids, i.e. put all three variables on the same grid. Here's how I did this

I checked the grids of all three input files

ram@luthien:~/Downloads/NcFileToMerge cdo griddes 2-HR_MOD_1467706576365.nc                                                                                                                                                 [8:45:45|16-07-20]
#
# gridID 1
#
gridtype  = lonlat
gridsize  = 246961
xname     = longitude
xlongname = longitude
xunits    = degrees_east
yname     = latitude
ylongname = latitude
yunits    = degrees_north
xsize     = 121
ysize     = 2041
xfirst    = 0
xinc      = 0.08333333
yfirst    = -80
yinc      = 0.08333588
cdo griddes: Processed 1 variable ( 0.00s )
ram@luthien:~/Downloads/NcFileToMerge cdo griddes 1-HR_MOD_1467706576365.nc                                                                                                                                                 [8:46:02|16-07-20]
#
# gridID 1
#
gridtype  = lonlat
gridsize  = 4410601
xname     = longitude
xlongname = longitude
xunits    = degrees_east
yname     = latitude
ylongname = latitude
yunits    = degrees_north
xsize     = 2161
ysize     = 2041
xfirst    = -180
xinc      = 0.08333333
yfirst    = -80
yinc      = 0.08333588
cdo griddes: Processed 1 variable ( 0.00s )
ram@luthien:~/Downloads/NcFileToMerge cdo griddes 0-HR_MOD_1467706576365.nc                                                                                                                                                 [8:46:08|16-07-20]
#
# gridID 1
#
gridtype  = lonlat
gridsize  = 3183960
xname     = longitude
xlongname = longitude
xunits    = degrees_east
yname     = latitude
ylongname = latitude
yunits    = degrees_north
xsize     = 1560
ysize     = 2041
xfirst    = 50
xinc      = 0.08333335
yfirst    = -80
yinc      = 0.08333588
cdo griddes: Processed 1 variable ( 0.00s )

I created a new one with is a superset of the existing ones

ram@luthien:~/Downloads/NcFileToMerge cat targetGrid                                                [9:01:17|16-07-20]
#
# gridID 1
#
gridtype  = lonlat
gridsize  = 8817120
xname     = longitude
xlongname = longitude
xunits    = degrees_east
yname     = latitude
ylongname = latitude
yunits    = degrees_north
xsize     = 4320
ysize     = 2041
xfirst    = -180
xinc      = 0.08333333
yfirst    = -80
yinc      = 0.08333588

now I remap all three inputs to that target grid. I the locations should be identical, so no real interpolation is performed. You could use nearest neighbour interpolation, but that would will missing values: [zsh syntax]
```
for f (0-HR_MOD_1467706576365.nc 1-HR_MOD_1467706576365.nc 2-HR_MOD_1467706576365.nc ) {echo $f; cdo remapbil,targetGrid $f global_$f}
```

finally I merged them pair-wise

cdo mergegrid global_1-HR_MOD_1467706576365.nc global_2-HR_MOD_1467706576365.nc 12.nc
cdo mergegrid global_0-HR_MOD_1467706576365.nc 12.nc 012.nc

result is uploaded - hope, that's what you're looking for

ralf

012.nc (16.9 MB) 012.nc

RE: Concatenate NC file - Added by Pierre LACOSTE almost 8 years ago

Hi Ralf,

Thank you for your help.
The result file is good.
I need to be able to replay your procedure to merge other files.
I understand very well the most of your procedure.
Just one thing that I doesn't understand is how you create your super "targetGrid" with the command "cat".
Could you tell me what command it is and if you create the "targetgrid" for all the world or if you target only the area corresponding of the tree files?

Thank you,

Pierre.

RE: Concatenate NC file - Added by Ralf Mueller almost 8 years ago

You start with

cdo griddes 1-HR_MOD_1467706576365.nc > targetGrid

This will write the standard output of

cdo griddes 1-HR_MOD_1467706576365.nc

to the text file targetGrid. Now you can edit this file with you favourite text editor (gedit,emacs,nano,...)
and change the variables xsize and gridsize. Now you can go on with remapping and merging.

RE: Concatenate NC file - Added by Pierre LACOSTE over 7 years ago

Hi,

I have implemented the solution and that works.
But the problem it that solution use a lot of memory.
If I have 2 files with a size of 400Mo to merge, cdo needs more than 10Go of RAM to realize the "remapbil".
Is there a solution to improve the performance of "remapbil" operation?

Thank you,

Pierre

RE: Concatenate NC file - Added by Audace ADANTODE almost 4 years ago

Hi Ralf

I am trying to replicate the methode but this code does not work for me:

for f (0-HR_MOD_1467706576365.nc 1-HR_MOD_1467706576365.nc 2-HR_MOD_1467706576365.nc ) {echo $f; cdo remapbil,targetGrid $f global_$f}

Here is the error message:

syntax error near unexpected token `('

May you help me please?

Thanks

RE: Concatenate NC file - Added by Ralf Mueller almost 4 years ago

hi!

This looks like a ZSH-for loop, but I guess you are not using ZSH. So you might try the standard version of a for loop:

for f in 0-HR_MOD_1467706576365.nc 1-HR_MOD_1467706576365.nc 2-HR_MOD_1467706576365.nc; do
  echo $f; cdo remapbil,targetGrid $f global_$f
done

hth
ralf

(1-12/12)

Project

General

Profile

CDO