Concatenate NC file
Added by Pierre LACOSTE over 8 years ago
Hello,
I'm trying to concatenate multiple "nc" files which contain exactly the same variables with CDO but I have some problems with this.
I first use
cdo merge file1.nc file2.nc file3.nc fileout.nc
But at the end of the execution, I got the message
Warning (cdfDefVar) : Changed double entry of variable name 'ssh' to 'ssh_2'!
Warning (cdfDefVar) : Changed multiple entry of variable name 'ssh' to 'ssh_3'!
cdo merge: Processed 7841522 values from 3 variables over 3 timesteps ( 0.44s )
And I doesn't want to have the variable "ssh" renamed for the data in the files file2.nc and file2.nc
I found another command in the document which is "cat". I tried this command instead of merge and I got the messagecdo cat (Abort): Grid size of the input parameters do not match!
How can I use CDO to concatenate nc files?
Thank you for your help.
Pierre
Replies (12)
RE: Concatenate NC file - Added by Pierre LACOSTE over 8 years ago
Hello Ralf,
I tried your solution and now I got the following message:cdo cat (Abort): Grid size of the input parameters do not match!
Do you have an idea where the problem comes?
Thank you,
Pierre
RE: Concatenate NC file - Added by Ralf Mueller over 8 years ago
in that case I need the input files, or at least some of them to reproduce the error.
In general this message has to do with variables with the
- the same name in different files
- different grid size in different files
but without the data, it's hard to say. Please not, that in most cases, the first timestep of each input file is sufficient.
best
ralf
RE: Concatenate NC file - Added by Pierre LACOSTE over 8 years ago
Hello Ralf,
Sorry for the delay from your last reply.
I provide you the two files that I try to concatenate.
Thank you for your help.
Pierre
| NcFileToMerge.zip (6.94 MB) NcFileToMerge.zip |
RE: Concatenate NC file - Added by Ralf Mueller over 8 years ago
Here are the grids of your three input files
ram@luthien:~/Downloads/NcFileToMerge cdo griddes 0-HR_MOD_1467706576365.nc [12:46:10|16-07-19] # # gridID 1 # gridtype = lonlat gridsize = 3183960 xname = longitude xlongname = longitude xunits = degrees_east yname = latitude ylongname = latitude yunits = degrees_north xsize = 1560 ysize = 2041 xfirst = 50 xinc = 0.08333335 yfirst = -80 yinc = 0.08333588 cdo griddes: Processed 1 variable ( 0.00s ) ram@luthien:~/Downloads/NcFileToMerge cdo griddes 1-HR_MOD_1467706576365.nc [12:46:41|16-07-19] # # gridID 1 # gridtype = lonlat gridsize = 4410601 xname = longitude xlongname = longitude xunits = degrees_east yname = latitude ylongname = latitude yunits = degrees_north xsize = 2161 ysize = 2041 xfirst = -180 xinc = 0.08333333 yfirst = -80 yinc = 0.08333588 cdo griddes: Processed 1 variable ( 0.00s ) ram@luthien:~/Downloads/NcFileToMerge cdo griddes 2-HR_MOD_1467706576365.nc [12:46:44|16-07-19] # # gridID 1 # gridtype = lonlat gridsize = 246961 xname = longitude xlongname = longitude xunits = degrees_east yname = latitude ylongname = latitude yunits = degrees_north xsize = 121 ysize = 2041 xfirst = 0 xinc = 0.08333333 yfirst = -80 yinc = 0.08333588 cdo griddes: Processed 1 variable ( 0.00s )
All of them have different size but the same variable name (ssh). Hence a concatenation does not work, because in netcdf the shornames have to be uniq (not the shortnames together with some dimension info). By renaming, you can still put then into a single file
ram@luthien:~/Downloads/NcFileToMerge cdo infov 0-HR_MOD_1467706576365.nc [12:48:46|16-07-19]
-1 : Date Time Level Gridsize Miss : Minimum Mean Maximum : Parameter name
1 : 2016-06-12 12:00:00 0 3183960 1076869 : -1.8320 0.0039747 1.6297 : ssh
cdo infon: Processed 3183960 values from 1 variable over 1 timestep ( 0.01s )
ram@luthien:~/Downloads/NcFileToMerge cdo infov 1-HR_MOD_1467706576365.nc [12:49:40|16-07-19]
-1 : Date Time Level Gridsize Miss : Minimum Mean Maximum : Parameter name
1 : 2016-06-12 12:00:00 0 4410601 993417 : -1.9581 -0.18370 0.97781 : ssh
cdo infon: Processed 4410601 values from 1 variable over 1 timestep ( 0.02s )
ram@luthien:~/Downloads/NcFileToMerge cdo infov 2-HR_MOD_1467706576365.nc [12:49:43|16-07-19]
-1 : Date Time Level Gridsize Miss : Minimum Mean Maximum : Parameter name
1 : 2016-06-12 12:00:00 0 246961 79085 : -1.9663 -0.54538 0.45228 : ssh
cdo infon: Processed 246961 values from 1 variable over 1 timestep ( 0.00s )
ram@luthien:~/Downloads/NcFileToMerge cdo merge -chname,ssh,ssh_0 0-HR_MOD_1467706576365.nc -chname,ssh,ssh_1 1-HR_MOD_1467706576365.nc -chname,ssh,ssh_2 2-HR_MOD_1467706576365.nc out.nc [12:49:50|16-07-19]
cdo merge: Started child process "chname,ssh,ssh_0 0-HR_MOD_1467706576365.nc (pipe1.1)".
cdo merge: Started child process "chname,ssh,ssh_1 1-HR_MOD_1467706576365.nc (pipe1.2)".
cdo merge: Started child process "chname,ssh,ssh_2 2-HR_MOD_1467706576365.nc (pipe1.3)".
cdo(2) chname: Processed 3183960 values from 1 variable over 1 timestep ( 0.26s )
cdo(3) chname: Processed 4410601 values from 1 variable over 1 timestep ( 0.26s )
cdo(4) chname: Processed 246961 values from 1 variable over 1 timestep ( 0.26s )
cdo merge: Processed 7841522 values from 3 variables over 3 timesteps ( 0.26s )
ram@luthien:~/Downloads/NcFileToMerge nnc out.nc [12:50:35|16-07-19]
ram@luthien:~/Downloads/NcFileToMerge cdo infov out.nc [12:50:47|16-07-19]
-1 : Date Time Level Gridsize Miss : Minimum Mean Maximum : Parameter name
1 : 2016-06-12 12:00:00 0 3183960 1076869 : -1.8320 0.0039747 1.6297 : ssh_0
2 : 2016-06-12 12:00:00 0 4410601 993417 : -1.9581 -0.18370 0.97781 : ssh_1
3 : 2016-06-12 12:00:00 0 246961 79085 : -1.9663 -0.54538 0.45228 : ssh_2
cdo infon: Processed 7841522 values from 3 variables over 1 timestep ( 0.04s )
hth
ralf
RE: Concatenate NC file - Added by Pierre LACOSTE over 8 years ago
Hello ralf,
Thank you for your response.
The problem with the rename of the variable is that create 3 differents variables on the result file.
And, when I use the tool panoply, to create a plot, it's not possible to have it at the same time for the 3 variables?
The 3 files that I provide have not crossing area. The file _0 is from lat0 to lat1, the file _1 is from lat1 to lat2 and the file _3 is from lat2 to lat0.
I hoped that it was possible to have a merge file with the values of the variable ssh put into the same file from lat0 to lat0.
Do you think what I need to do is possible?
Thank you,
Pierre.
RE: Concatenate NC file - Added by Ralf Mueller over 8 years ago
Hi Pierre!
sorry for the miss-understanding - merge can so many different things. If I got it right, you want to merge the grids, i.e. put all three variables on the same grid. Here's how I did this
- I checked the grids of all three input files
ram@luthien:~/Downloads/NcFileToMerge cdo griddes 2-HR_MOD_1467706576365.nc [8:45:45|16-07-20] # # gridID 1 # gridtype = lonlat gridsize = 246961 xname = longitude xlongname = longitude xunits = degrees_east yname = latitude ylongname = latitude yunits = degrees_north xsize = 121 ysize = 2041 xfirst = 0 xinc = 0.08333333 yfirst = -80 yinc = 0.08333588 cdo griddes: Processed 1 variable ( 0.00s ) ram@luthien:~/Downloads/NcFileToMerge cdo griddes 1-HR_MOD_1467706576365.nc [8:46:02|16-07-20] # # gridID 1 # gridtype = lonlat gridsize = 4410601 xname = longitude xlongname = longitude xunits = degrees_east yname = latitude ylongname = latitude yunits = degrees_north xsize = 2161 ysize = 2041 xfirst = -180 xinc = 0.08333333 yfirst = -80 yinc = 0.08333588 cdo griddes: Processed 1 variable ( 0.00s ) ram@luthien:~/Downloads/NcFileToMerge cdo griddes 0-HR_MOD_1467706576365.nc [8:46:08|16-07-20] # # gridID 1 # gridtype = lonlat gridsize = 3183960 xname = longitude xlongname = longitude xunits = degrees_east yname = latitude ylongname = latitude yunits = degrees_north xsize = 1560 ysize = 2041 xfirst = 50 xinc = 0.08333335 yfirst = -80 yinc = 0.08333588 cdo griddes: Processed 1 variable ( 0.00s )
- I created a new one with is a superset of the existing ones
ram@luthien:~/Downloads/NcFileToMerge cat targetGrid [9:01:17|16-07-20] # # gridID 1 # gridtype = lonlat gridsize = 8817120 xname = longitude xlongname = longitude xunits = degrees_east yname = latitude ylongname = latitude yunits = degrees_north xsize = 4320 ysize = 2041 xfirst = -180 xinc = 0.08333333 yfirst = -80 yinc = 0.08333588
- now I remap all three inputs to that target grid. I the locations should be identical, so no real interpolation is performed. You could use nearest neighbour interpolation, but that would will missing values: [zsh syntax]
for f (0-HR_MOD_1467706576365.nc 1-HR_MOD_1467706576365.nc 2-HR_MOD_1467706576365.nc ) {echo $f; cdo remapbil,targetGrid $f global_$f} - finally I merged them pair-wise
cdo mergegrid global_1-HR_MOD_1467706576365.nc global_2-HR_MOD_1467706576365.nc 12.nc cdo mergegrid global_0-HR_MOD_1467706576365.nc 12.nc 012.nc
result is uploaded - hope, that's what you're looking for
ralf
RE: Concatenate NC file - Added by Pierre LACOSTE over 8 years ago
Hi Ralf,
Thank you for your help.
The result file is good.
I need to be able to replay your procedure to merge other files.
I understand very well the most of your procedure.
Just one thing that I doesn't understand is how you create your super "targetGrid" with the command "cat".
Could you tell me what command it is and if you create the "targetgrid" for all the world or if you target only the area corresponding of the tree files?
Thank you,
Pierre.
RE: Concatenate NC file - Added by Ralf Mueller over 8 years ago
You start with
cdo griddes 1-HR_MOD_1467706576365.nc > targetGridThis will write the standard output of
cdo griddes 1-HR_MOD_1467706576365.ncto the text file
targetGrid. Now you can edit this file with you favourite text editor (gedit,emacs,nano,...)and change the variables xsize and gridsize. Now you can go on with remapping and merging.
RE: Concatenate NC file - Added by Pierre LACOSTE over 8 years ago
Hi,
I have implemented the solution and that works.
But the problem it that solution use a lot of memory.
If I have 2 files with a size of 400Mo to merge, cdo needs more than 10Go of RAM to realize the "remapbil".
Is there a solution to improve the performance of "remapbil" operation?
Thank you,
Pierre
RE: Concatenate NC file - Added by Audace ADANTODE almost 5 years ago
Hi Ralf
I am trying to replicate the methode but this code does not work for me:
for f (0-HR_MOD_1467706576365.nc 1-HR_MOD_1467706576365.nc 2-HR_MOD_1467706576365.nc ) {echo $f; cdo remapbil,targetGrid $f global_$f}
Here is the error message:
syntax error near unexpected token `('
May you help me please?
Thanks
RE: Concatenate NC file - Added by Ralf Mueller almost 5 years ago
hi!
This looks like a ZSH-for loop, but I guess you are not using ZSH. So you might try the standard version of a for loop:
for f in 0-HR_MOD_1467706576365.nc 1-HR_MOD_1467706576365.nc 2-HR_MOD_1467706576365.nc; do echo $f; cdo remapbil,targetGrid $f global_$f done
hth
ralf