Project

General

Profile

Forums » Open discussion »

mergegrid

Added by Fred Leclercq about 2 years ago

Hi all,

I am trying to merge (mergegrid) two files, but it seems like it has a problem with large amounts of data. The result is definitely a variable that contains more than 4Gb of data. Is this a known problem, or is something else causing it. The result in the new file is a variable (two-dim) starting with a lot of NaN's and towards the end it stores the data as expected. Single files are about ~32Gb (2 dims, 10 variables)

Thanks!
Fred.


Replies (3)

RE: mergegrid - Added by ep gupal about 2 years ago

Well, this is not an ideal solution but, if the files are really big, I would (1) decompose the each data for each time step, and (2) mergegrid, and finally (3) mergetime. Oh make sure to copy the data into netcdf4 format with "-f nc4" so that it becomes netcdf4 data format. My understanding is that netcdf4 can handle larger size data, while netcdf3 is somewhat limited.

So, the steps would be

(1) decomposing the data for each time steps using splitsel
cdo -f nc4 --timestat_date first splitsel,1,0,0 in.nc out.nc
where --timestat_date first option is so that it would not skip the first time step

(2) then mergegrid
cdo mergegrid in*.nc out.nc

(3) then mergetime
cdo mergetime in*.nc out.nc

RE: mergegrid - Added by Fred Leclercq about 2 years ago

Thanks, but there is no time dimension in the files.

Basically there are two files with lats/lon
[1 2] and [5 6]
[3 4] [7 8]

I'm trying to get these geospatially merged as:
[1 2 5 6]
[3 4 7 8]

Therefore I've created an empty file with the correct expected shape:
[_ _ _ ]
[ _ _ _]

1) create expected shape (Python script)
2) cdo -f nc4 -v mergegrid expectedshape.nc file1 expectedshape2.nc
3) cdo -f nc4 -v mergegrid expectedshape2.nc file2 expectedshape3.nc

That setup seems to work as long as the amount of data in one variable is not too(?) big?

Thanks

RE: mergegrid - Added by Fred Leclercq about 2 years ago

Fred Leclercq wrote in RE: mergegrid:

Thanks, but there is no time dimension in the files.

Basically there are two files with lats/lon
[1 2] and [5 6]
[3 4] [7 8]

I'm trying to get these geospatially merged as:
[1 2 5 6]
[3 4 7 8]

Therefore I've created an empty file with the correct expected shape:
[ _ _ _ _ ]
[ _ _ _ _ ]

1) create expected shape (Python script)
2) cdo -f nc4 -v mergegrid expectedshape.nc file1 expectedshape2.nc
3) cdo -f nc4 -v mergegrid expectedshape2.nc file2 expectedshape3.nc

That setup seems to work as long as the amount of data in one variable is not too(?) big?

Thanks

    (1-3/3)