Merge datasets sorted by spatial coordinates?
Added by Clement Tisseuil almost 15 years ago
Dear all,
Is there a possibility to merge several dataset along the spatial dimension (like mergetime but applied to the spatial coordinates). Typically, the dataset are identical in terms of variables and time dimensions but they represent different spatial area.
Example :
cdo sellonlatbox,-10,10,30,40 file.nc file1.nc
cdo sellonlatbox,-10,10,20,30 file.nc file2.nc
cdo sellonlatbox,-10,10,20,40 file.nc file3.nc
I would like to build file4.nc as the spatial merging of file1.nc and file2.nc, which should be identical to file3.nc
Thanks in advance.
Regards,
Clement
Replies (48)
RE: Merge datasets sorted by spatial coordinates? - Added by Uwe Schulzweida over 14 years ago
Hi Clement,
We have an experimental operator called mergegrid for this task:
cdo setrtomiss,-1e33,1e33 file3.nc file4.nc # create file4 with the grid of file3 and set all values to miss cdo mergegrid file4.nc file1.nc file5.nc # merge all grid points of file1 to file4 and store it to file5 cdo mergegrid file5.nc file2.nc file6.nc # merge all grid points of file2 to file5 and store it to file6
This operator is not documented, yet.
Regards,
Uwe
RE: Merge datasets sorted by spatial coordinates? - Added by Clement Tisseuil over 14 years ago
Excellent !
Thank you very much !
Regards,
Clement
RE: Merge datasets sorted by spatial coordinates? - Added by Clement Tisseuil over 14 years ago
Actually, there is seems to be a problem with mergegrid :
- Example:
cdo sellonlatbox,-11,10,30,40 file.nc file1.nc
cdo sellonlatbox,-11,10,20,30 file.nc file2.nc
cdo sellonlatbox,-11,10,20,40 file.nc file3.nc
- Merge file1.nc and file2.nc into file4.nc.
cdo mergegrid file1.nc file2.nc file.4.nc
- Check grid characteristic for each file. Normally, file4.nc should be equal to file3.nc but it is not...
clem@clem-laptop:/media/VERBATIM$ cdo griddes file1.nc #
- gridID 0
#
gridtype = lonlat
gridsize = 40
xname = lon
xlongname = longitude
xunits = degrees_east
yname = lat
ylongname = latitude
yunits = degrees_north
xsize = 8
ysize = 5
xfirst = -8.75
xinc = 2.5
xbounds = 350 352.5
352.5 355
355 357.5
357.5 360
0 2.5
2.5 5
5 7.5
7.5 10
yfirst = 31
yinc = 2
ybounds = 30 32
32 34
34 36
36 38
38 40
cdo griddes: Processed 1 variable. ( 0.00s )
clem@clem-laptop:/media/VERBATIM$ cdo griddes file2.nc # - gridID 0
#
gridtype = lonlat
gridsize = 40
xname = lon
xlongname = longitude
xunits = degrees_east
yname = lat
ylongname = latitude
yunits = degrees_north
xsize = 8
ysize = 5
xfirst = -8.75
xinc = 2.5
xbounds = 350 352.5
352.5 355
355 357.5
357.5 360
0 2.5
2.5 5
5 7.5
7.5 10
yfirst = 21
yinc = 2
ybounds = 20 22
22 24
24 26
26 28
28 30
cdo griddes: Processed 1 variable. ( 0.00s )
clem@clem-laptop:/media/VERBATIM$ cdo griddes file3.nc # - gridID 0
#
gridtype = lonlat
gridsize = 80
xname = lon
xlongname = longitude
xunits = degrees_east
yname = lat
ylongname = latitude
yunits = degrees_north
xsize = 8
ysize = 10
xfirst = -8.75
xinc = 2.5
xbounds = 350 352.5
352.5 355
355 357.5
357.5 360
0 2.5
2.5 5
5 7.5
7.5 10
yfirst = 21
yinc = 2
ybounds = 20 22
22 24
24 26
26 28
28 30
30 32
32 34
34 36
36 38
38 40
cdo griddes: Processed 1 variable. ( 0.00s )
clem@clem-laptop:/media/VERBATIM$ cdo griddes file4.nc
Warning (cdfInqContents) : time variable not found! # - gridID 0
#
gridtype = lonlat
gridsize = 40
xname = lon
xlongname = longitude
xunits = degrees_east
yname = lat
ylongname = latitude
yunits = degrees_north
xsize = 8
ysize = 5
xfirst = -8.75
xinc = 2.5
xbounds = 350 352.5
352.5 355
355 357.5
357.5 360
0 2.5
2.5 5
5 7.5
7.5 10
yfirst = 31
yinc = 2
ybounds = 30 32
32 34
34 36
36 38
38 40
cdo griddes: Processed 1 variable. ( 0.01s )
RE: Merge datasets sorted by spatial coordinates? - Added by Uwe Schulzweida over 14 years ago
It seems that mergegrid is not the optimal name for this operator, because it does not merge two different grids to a new one!
The first input file have to contain the target grid. So you can add only the data from the second input file to the first grid.
That means the resulting output file has the same grid as the first input file.
Here again the example from above:
cdo setrtomiss,-1e33,1e33 file3.nc file4.nc # create file4 with the grid of file3 and set all values to miss cdo mergegrid file4.nc file1.nc file5.nc # merge all grid points of file1 to file4 and store it to file5 cdo mergegrid file5.nc file2.nc file6.nc # merge all grid points of file2 to file5 and store it to file6
Regards,
Uwe
RE: Merge datasets sorted by spatial coordinates? - Added by Clement Tisseuil over 14 years ago
Ok thank you for the precisions.
So, to your knowledge there is no function in cdo to merge two files with different grids ? In my case study, the two files have exactly the same temporal structure and spatial resolution and they correspond to two spatially adjacent files.
Let me know if you have any suggestions...
Regards
Clem
RE: Merge datasets sorted by spatial coordinates? - Added by Clement Tisseuil over 14 years ago
Hello,
I have found a solution to my problem but I do not know if it is the most efficient way to do it. What do you think ?
## Create three files, as file3=file1+file2
cdo sellonlatbox,-11,10,30,40 ulwsfc.gdas.197901.grb2 file1.grb2
cdo sellonlatbox,-11,10,20,30 ulwsfc.gdas.197901.grb2 file2.grb2
cdo sellonlatbox,-11,10,20,40 ulwsfc.gdas.197901.grb2 file3.grb2
## Let us build a "No data" file6 which have the same structure than file3
cdo ifthenc,NA file3.grb2 file5.grb2
cdo ifnotthenc,NA, file3.grb2 file6.grb2
## Merging file1 and file2 into file8.
cdo mergegrid file6.grb2 file1.grb2 file7.grb2
cdo mergegrid file7.grb2 file2.grb2 file8.grb2
## Checking file3 and file8. After merging, file8 is identical to file3.
cdo info file8.grb2
cdo info file3.grb2
Cheers
RE: Merge datasets sorted by spatial coordinates? - Added by Uwe Schulzweida over 14 years ago
Yes, this is the most efficient way for this task!
Regards,
Uwe
RE: Merge datasets sorted by spatial coordinates? - Added by Tom Akkermans over 13 years ago
I have another solution for this, and it works very well.
Clement, in your example you still actually need the bigger "file3".
Here is what you do, without the file "file3" itself but only with its info.
- Make a file called "grid.txt" and put in it the right info about file3, e.g.:
gridtype = lonlat
gridsize = 420
xname = lon
xlongname = longitude
xunits = degrees east
yname = lat
ylongname = latitude
yunits = degrees north
xsize = 21
ysize = 20
xfirst = -11.0
xinc = 1
yfirst = -20.0
yinc = 1
- First make an enlarged file with the info of file3. Its contents are rubbish but that doesnt matter:
cdo enlarge,grid.txt file1.nc file4.nc
- Then merge the enlarged domain with the content of the first and consequently with the second file:
cdo mergegrid file4.nc file1.nc file5.nc
cdo mergegrid file5.nc file2.nc file6.nc
- Remove the temporary files:
rm file4.nc
rm file5.nc
- Now the spatially merged file is "file6.nc".
Good luck!
RE: Merge datasets sorted by spatial coordinates? - Added by Rodrigo Sousa almost 11 years ago
Hi Tom Akkermans,
Thanks very much , your procedure worked for my situation, but I have a question, how the Mergegrid.c make the calculation ?
My situation is as follows:
I have two datasets : file1.nc and file2.nc
file1: Latitude : 9N to 60N
Longitude: 100W to 19W
file2: Latitude : 9N to 60N
Longitude: 21W to 45E
My final file: file_final.nc have this coordinates:
Latitude : 9N to 60N
Longitude: 100W to 45E
The mergegrid works fine, but I have a doubt about the intersection, what the mergegrid command do in intersection? Average or another calculation?
Regards,
Rodrigo
RE: Merge datasets sorted by spatial coordinates? - Added by K A over 9 years ago
Hello all,
First thank you Tom Akkermans for you solution, it has worked for me in the past.
However, I now need to merge several dataset along the spatial dimension that are adjacent, but have intersecting boundaries.
Example:
file1: Latitude: 42N to 46N
Longitude 120W to 110W
file2: Latitude: 42N to 46N
Longitude: 110W to 100W
For the intersecting Longitude 110W:
file1 has values for latitudes 42N to 44N and empty cells (NaN) for 45N to 46N
file2 has values for latidues 45N to 46N and empty cells (NaN) for 42N to 44N
When I use mergegrid the values in this intersecting longitude are all read from file2, overwriting the data from file1. Is there a way to have the function only append values that are not NaN?
I have searched extensively for a way (CDO, NCO, others) to merge files with adjacent and intersecting boundaries without finding a solution. Any ideas or other people to contact would be greatly appreciated!
Thank you for your time,
Kristen
Clement Tisseuil wrote:
Dear all,
Is there a possibility to merge several dataset along the spatial dimension (like mergetime but applied to the spatial coordinates). Typically, the dataset are identical in terms of variables and time dimensions but they represent different spatial area.
Example :
cdo sellonlatbox,-10,10,30,40 file.nc file1.nc
cdo sellonlatbox,-10,10,20,30 file.nc file2.nc
cdo sellonlatbox,-10,10,20,40 file.nc file3.ncI would like to build file4.nc as the spatial merging of file1.nc and file2.nc, which should be identical to file3.nc
Thanks in advance.
Regards,
Clement
RE: Merge datasets sorted by spatial coordinates? - Added by Noam Chomsky almost 5 years ago
Is there a simpler way to merge nc files without first creating an empty nc file? It's not very efficient if the creation of the grid.txt needs to be done manually.
RE: Merge datasets sorted by spatial coordinates? - Added by Ralf Mueller almost 5 years ago
create an empty file first? I never did this ...
what exactly the problem, Noam?
cheers
ralf
RE: Merge datasets sorted by spatial coordinates? - Added by Noam Chomsky almost 5 years ago
I am trying to merge multiple nc files with the same variables and timesteps, but containing different sections of the world. Say I have africa.nc and india.nc
Can I merge them so I end up with a file africa-india.nc without manually creating the grid.txt file as suggested above? In R for example is possible to do merge(file1,file2,file3,etc.) and the function will figure out the extent of the combined file. Creating many empty files will require to write many grid.txt files, or am I missing something?
RE: Merge datasets sorted by spatial coordinates? - Added by Karin Meier-Fleischer almost 5 years ago
Hi Noam,
what about remapping the africa.nc and india.nc to a global grid?
Assume you want to have a 0.5x0.5 degrees global grid (720x360):
cdo -remapbil,r720x360 africa.nc africa_r720x360.nc cdo -remapbil,r720x360 india.nc india_r720x360.nc cdo -merge africa_r720x360.nc india_r720x360.nc africa_india_r720x360.nc
-Karin
RE: Merge datasets sorted by spatial coordinates? - Added by Noam Chomsky almost 5 years ago
Thank you Karin. Your suggestion works
It looks though, that if I want to merge multiple files I have to do pair by pair.
Because
cdo merge out-a.nc out-n.nc out-i.nc test.nc
Warning: Duplicate entry of parameter name myvar in out-n.nc!
cdo merge: Processed 3 variables over 3 timesteps [0.02s 9MB].
but the result only contains out-a.nc
and mergegrid only takes 2 input files. Maybe it's just not possible? Because I also tried
cdo merge out-* test.nc
but no luck.
RE: Merge datasets sorted by spatial coordinates? - Added by Noam Chomsky almost 5 years ago
Hello Karin,
I would like to know if when remapping a section of the globe into a global grid the actual data is interpolated or not, because remapbil does bilinear interpolation. Let's assume that the resolution is the same, say I have India at 0.5x0.5 and I remap to a global grid at 0.5x0.5, in theory there should be no interpolation, correct?
RE: Merge datasets sorted by spatial coordinates? - Added by Uwe Schulzweida almost 5 years ago
Thats correct, if the resolution and the coordinates are the same then no interpolation takes place.
If the environment variable REMAP_EXTRAPOLATE=off is set than remapbil and remapnn will give the same result in this case.
RE: Merge datasets sorted by spatial coordinates? - Added by Noam Chomsky almost 5 years ago
I am sorry to bring this up again but I'm getting inconsistent results with the remapbil function.
What I do is
cdo remapbil,mygrid popavg-africa.nc out-a.nc
mygrid is
gridtype = lonlat
xsize = 180
ysize = 90
xfirst = -179
xinc = 2
yfirst = -89
yinc = 2
But the resulting values in out-a.nc are wrong, it IS interpolating the values. I am uploading the starting file. If I do the merge with R than the result is correct. Do you know why?
popavg-africa.nc (14.1 KB) popavg-africa.nc |
RE: Merge datasets sorted by spatial coordinates? - Added by Uwe Schulzweida almost 5 years ago
You are right, remapbil can be used only if the data come without missing values. Your data contain missing values, in this case you have to use remapnn in combination with the environment variable REMAP_EXTRAPOLATE=off.
RE: Merge datasets sorted by spatial coordinates? - Added by Noam Chomsky almost 5 years ago
Thank you Uwe, I will try that.
So to clarify:
remapnn with REMAP_EXTRAPOLATE=off (with NAs) should give the same results as remapbil (when there are no NAs), assuming all other values are the same.
RE: Merge datasets sorted by spatial coordinates? - Added by Audace ADANTODE over 4 years ago
Hello,
i am also trying to merge multiple nc files with the same variables and timesteps, but containing different sections of the world. all of them are 0.5° resolution grid files.for exemple:
MLLO.1998.2019.cld.dat.nc
ASKS.1998.2019.cld.dat.nc
NY.1998.2019.cld.dat.nc
Then I would like to know if Noam find the best way or if someone got it.
Thanks
RE: Merge datasets sorted by spatial coordinates? - Added by Noam Chomsky over 4 years ago
The solution suggested by Uwe works for me.
You should use set REMAP_EXTRAPOLATE=off and then use cdo remapbil,mygrid region1, region2 global.nc
The only problem I really have with this is that I cannot use more than two regions as an input so I have to use cdo remapbil,mygrid multiple times until I had all the regions.
RE: Merge datasets sorted by spatial coordinates? - Added by Audace ADANTODE over 4 years ago
Thank you, Noam
RE: Merge datasets sorted by spatial coordinates? - Added by Audace ADANTODE over 4 years ago
Please Noam, I did this:
$ REMAP_EXTRAPOLATE=off
$ cdo remapnn,mygrid MLLO.1998.2019.cld.dat.nc MLLOout.1998.2019.cld.dat.nc
but I have got this
cdo remapnn (Abort):
Did I miss something?
Thanks
RE: Merge datasets sorted by spatial coordinates? - Added by Ralf Mueller over 4 years ago
Hi!
Can you upload the files, Audace? A single timestep should be sufficient.
cheers
ralf