Project

General

Profile

merge all netCDF files in a directory

Added by Alex Mazurkiewicz over 12 years ago

Hello,

So far I haven't found a simple way to do this in cdo: I have 365 NetCDF files in a directory and want to merge them all into one.

cdo mergetime file1 file2 file3 ... file 365 outfile.nc isn't practical.

cdo mergetime *.nc outfile would be practical, but doesn't work. Is there an efficient way to do this?

To get around the problem I wrote a loop and 'appended' one file at a time using mergetime. If there is a more efficient method then please let me know!

Many thanks,

alex


Replies (25)

RE: merge all netCDF files in a directory - Added by Uwe Schulzweida over 12 years ago

Hi Alex,

cdo mergetime *.nc outfile
should normally work without problems. There is one limitation, to sort the time steps mergetime has to open all input files at the same time. How many file can be opened depends on the operating system. On some system the limit is 256.
You can use cat if your input files are already sorted in time:
cdo cat *.cat outfile

Best regards,
Uwe

RE: merge all netCDF files in a directory - Added by Jinghua Chen over 12 years ago

Do you solve this problem?
I just have the same problem, i use 'cdo mergetime *.nc outfile' and 'cdo cat *.nc outfile', and they are all going wrong with the message like bellow:

Assertion failed: pxp->bf_offset <= offset && offset < pxp->bf_offset + (off_t) pxp->bf_extent, file posixio.c, line 325

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

thank you very much!!!

Jinghua

RE: merge all netCDF files in a directory - Added by Uwe Schulzweida over 12 years ago

This error message came directly from the netCDF library.
In most cases the resulting file size will be larger than 2GB.
To solve this problem you have to change the output file type to netCDF2 (nc2):

 cdo -f nc2 mergetime *.nc outfile

RE: merge all netCDF files in a directory - Added by lkpj ggjjhhj over 12 years ago

Hi, I'm encountering the same error, as my output file is greater than 2GB
I tried the followings:
cdo -f nc2 mergetime *.nc outfile
cdo -f nc2 mergetime *.nc outfile.nc2
cdo -f nc2 mergetime *.nc2 outfile.nc2
cdo mergetime *.nc2 outfile.nc
cdo mergetime *.nc2 outfile.nc2

All to no avail. Same error(Assertion failed: pxp->bf_offset <= offset && offset < pxp->bf_offset + (off_t) pxp->bf_extent, file posixio.c, line 325)
Mr Uwe, what can i do to resolve this?
Mr Chen Jinghua, did you solve the issue?

RE: merge all netCDF files in a directory - Added by Jinghua Chen over 12 years ago

the reason is the file is too large(>2GB).
you may try with "cdo -f nc2" or "cdo -f nc4"

RE: merge all netCDF files in a directory - Added by Theodoros Gkousarov almost 12 years ago

any progress? i have the same problem when trying to merge more than 2 gb files. couldnt solve it.

RE: merge all netCDF files in a directory - Added by Jaison-Thomas Ambadan almost 12 years ago

Hi,

May be your file system is 32-bit? Anyway, can you please post the output of "cdo -V"

Cheers,
J.

RE: merge all netCDF files in a directory - Added by Theodoros Gkousarov almost 12 years ago

SOLVED .....

the files couldnt be merged under windows because been more than 2gb.

I run virtual machine and tried to merge them under Ubuntu 10 lts and it worked like charm.

Thank you. try doing it under linux.

RE: merge all netCDF files in a directory - Added by Ralf Mueller almost 12 years ago

Have you tried cygwin? in the minimal configuration, it gives you a unix-like prompt with all the great command line tools.

RE: merge all netCDF files in a directory - Added by Sohrab Kolsoumi over 6 years ago

Hi
You can easily do this by Netcdf Extractor. It's a windows tool that in this tool, the user can extract data from many netcdf files without any knowledge about programming.
https://agrimetsoft.com/Netcdf-Extractor.aspx

Help of Netcdf Extractor.pdf (1.82 MB) Help of Netcdf Extractor.pdf Help of Netcdf Extractor

RE: merge all netCDF files in a directory - Added by Odeyemi Olawale over 6 years ago

It worked for me doing it this way
cdo -f nc .nc outfile.nc

RE: merge all netCDF files in a directory - Added by Puri Am over 3 years ago

hi,

I do the same ( cdo -f nc mergetime era5_*.nc era5_hrs.nc) but it seems to be that my timesteps reach the limit of CDO when I merge files in one. The text appears as:
" cdo mergetime (Warning): Some data values (min=-10233 max=33136) are outside the valid range (-32768 - 32767) of the used output precision! Use the CDO option -b 32 or -b 64 to increase the output precision. "
So, I want to ask, How to do as the machine suggests? Is it cdo -b 64 -f nc mergetime era5_*.nc era5_hrs.nc? or cdo -f nc -b 64 mergetime era5_*.nc era5_hrs.nc?
Thanks so much!

RE: merge all netCDF files in a directory - Added by Karin Meier-Fleischer over 3 years ago

Hi Puri,

the warning says that the values are outside the value range and tells you what you can do.

cdo -f nc -b F32 mergetime era5_*.nc era5_hrs.nc

should work.

-Karin

RE: merge all netCDF files in a directory - Added by Suvarna Tikle about 2 years ago

I have separate daily nc files of three parameters (temp,WD and WS). Would like to merge parameter wise example temp_2022

RE: merge all netCDF files in a directory - Added by Suvarna Tikle about 2 years ago

I have separate daily nc files of three parameters (temp,WD and WS) of ERA5. Would like to merge parameter wise per year, example temp_2022.

Is their any specific command in CDO.

RE: merge all netCDF files in a directory - Added by Ralf Mueller about 2 years ago

hi!

You can use mergetime or merge. in case you have each variable in separate files, you can also use cat on these files, only.

Something like

cdo -merge temp_* all_temp.nc
or
cdo -mergetime temp_* all_temp.nc
or
cdo -timesort -cat [ temp_* ] all_temp.nc

give it a try

cheers
ralf

RE: merge all netCDF files in a directory - Added by Suvarna Tikle about 2 years ago

Hi
Thanks for reply.
I have used cdo mergetime. More than 24 hours no output.
Will try other two commands suggested by you.
Suvarna

RE: merge all netCDF files in a directory - Added by Suvarna Tikle about 2 years ago

Hi
I have used commands but hard luck.

cdo -timesort -cat [ _130*.nc ] all_130.nc

Error msg:Operator not found

cdo -merge _130*.nc all_130.nc
Error msg:Open failed on >_130*<
No such file or directory

Could you please help further?
my file names are as bellow , want to merge all _130 file for year 2019.
temp: 'E5ml00_1H_2019-05-01_130.nc'
WD: 'E5ml00_1H_2019-08-31_131.nc'
WD: 'E5ml00_1H_2019-12-31_132.nc'

Thanks
Suvarna

RE: merge all netCDF files in a directory - Added by Karin Meier-Fleischer about 2 years ago

1. The operator name is timsort not timesort.
2. you need to specify the file names (there are no 130*.nc files - do you mean E5ml00_1H*_130.nc ?

Try:

cdo -timsort -cat [ E5ml00_1H_*_130.nc ] all_130.nc

RE: merge all netCDF files in a directory - Added by Suvarna Tikle about 2 years ago

Thanks a lot.
its started processing.

RE: merge all netCDF files in a directory - Added by Suvarna Tikle about 2 years ago

It get stopped with msg "cdo(1) cat: 3%Killed"

RE: merge all netCDF files in a directory - Added by Karin Meier-Fleischer about 2 years ago

How many files do you try to merge? Maybe it is a memory issue or you have to use a job for it.

RE: merge all netCDF files in a directory - Added by Suvarna Tikle about 2 years ago

I have files are ranging from 340 to 365.

    (1-25/25)