merge all netCDF files in a directory
Added by Alex Mazurkiewicz over 12 years ago
Hello,
So far I haven't found a simple way to do this in cdo: I have 365 NetCDF files in a directory and want to merge them all into one.
cdo mergetime file1 file2 file3 ... file 365 outfile.nc isn't practical.
cdo mergetime *.nc outfile would be practical, but doesn't work. Is there an efficient way to do this?
To get around the problem I wrote a loop and 'appended' one file at a time using mergetime. If there is a more efficient method then please let me know!
Many thanks,
alex
Replies (25)
RE: merge all netCDF files in a directory - Added by Uwe Schulzweida over 12 years ago
Hi Alex,
cdo mergetime *.nc outfileshould normally work without problems. There is one limitation, to sort the time steps mergetime has to open all input files at the same time. How many file can be opened depends on the operating system. On some system the limit is 256.
You can use cat if your input files are already sorted in time:
cdo cat *.cat outfile
Best regards,
Uwe
RE: merge all netCDF files in a directory - Added by Jinghua Chen over 12 years ago
Do you solve this problem?
I just have the same problem, i use 'cdo mergetime *.nc outfile' and 'cdo cat *.nc outfile', and they are all going wrong with the message like bellow:
Assertion failed: pxp->bf_offset <= offset && offset < pxp->bf_offset + (off_t) pxp->bf_extent, file posixio.c, line 325
This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.
thank you very much!!!
Jinghua
2012-04-04_160824.png (3.58 KB) 2012-04-04_160824.png | the error |
RE: merge all netCDF files in a directory - Added by Uwe Schulzweida over 12 years ago
This error message came directly from the netCDF library.
In most cases the resulting file size will be larger than 2GB.
To solve this problem you have to change the output file type to netCDF2 (nc2):
cdo -f nc2 mergetime *.nc outfile
RE: merge all netCDF files in a directory - Added by Jinghua Chen over 12 years ago
Thank you very much!!!
RE: merge all netCDF files in a directory - Added by lkpj ggjjhhj over 12 years ago
Hi, I'm encountering the same error, as my output file is greater than 2GB
I tried the followings:
cdo -f nc2 mergetime *.nc outfile
cdo -f nc2 mergetime *.nc outfile.nc2
cdo -f nc2 mergetime *.nc2 outfile.nc2
cdo mergetime *.nc2 outfile.nc
cdo mergetime *.nc2 outfile.nc2
All to no avail. Same error(Assertion failed: pxp->bf_offset <= offset && offset < pxp->bf_offset + (off_t) pxp->bf_extent, file posixio.c, line 325)
Mr Uwe, what can i do to resolve this?
Mr Chen Jinghua, did you solve the issue?
RE: merge all netCDF files in a directory - Added by Jinghua Chen over 12 years ago
the reason is the file is too large(>2GB).
you may try with "cdo -f nc2" or "cdo -f nc4"
RE: merge all netCDF files in a directory - Added by Theodoros Gkousarov almost 12 years ago
any progress? i have the same problem when trying to merge more than 2 gb files. couldnt solve it.
RE: merge all netCDF files in a directory - Added by Jaison-Thomas Ambadan almost 12 years ago
Hi,
May be your file system is 32-bit? Anyway, can you please post the output of "cdo -V"
Cheers,
J.
RE: merge all netCDF files in a directory - Added by Theodoros Gkousarov almost 12 years ago
SOLVED .....
the files couldnt be merged under windows because been more than 2gb.
I run virtual machine and tried to merge them under Ubuntu 10 lts and it worked like charm.
Thank you. try doing it under linux.
RE: merge all netCDF files in a directory - Added by Ralf Mueller almost 12 years ago
Have you tried cygwin? in the minimal configuration, it gives you a unix-like prompt with all the great command line tools.
RE: merge all netCDF files in a directory - Added by Sohrab Kolsoumi over 6 years ago
Hi
You can easily do this by Netcdf Extractor. It's a windows tool that in this tool, the user can extract data from many netcdf files without any knowledge about programming.
https://agrimetsoft.com/Netcdf-Extractor.aspx
Help of Netcdf Extractor.pdf (1.82 MB) Help of Netcdf Extractor.pdf | Help of Netcdf Extractor |
RE: merge all netCDF files in a directory - Added by Odeyemi Olawale over 6 years ago
It worked for me doing it this way
cdo -f nc .nc outfile.nc
RE: merge all netCDF files in a directory - Added by Puri Am over 3 years ago
hi,
I do the same ( cdo -f nc mergetime era5_*.nc era5_hrs.nc) but it seems to be that my timesteps reach the limit of CDO when I merge files in one. The text appears as:
" cdo mergetime (Warning): Some data values (min=-10233 max=33136) are outside the valid range (-32768 - 32767) of the used output precision! Use the CDO option -b 32 or -b 64 to increase the output precision. "
So, I want to ask, How to do as the machine suggests? Is it cdo -b 64 -f nc mergetime era5_*.nc era5_hrs.nc? or cdo -f nc -b 64 mergetime era5_*.nc era5_hrs.nc?
Thanks so much!
RE: merge all netCDF files in a directory - Added by Karin Meier-Fleischer over 3 years ago
Hi Puri,
the warning says that the values are outside the value range and tells you what you can do.
cdo -f nc -b F32 mergetime era5_*.nc era5_hrs.nc
should work.
-Karin
RE: merge all netCDF files in a directory - Added by Suvarna Tikle about 2 years ago
I have separate daily nc files of three parameters (temp,WD and WS). Would like to merge parameter wise example temp_2022
RE: merge all netCDF files in a directory - Added by Suvarna Tikle about 2 years ago
I have separate daily nc files of three parameters (temp,WD and WS) of ERA5. Would like to merge parameter wise per year, example temp_2022.
Is their any specific command in CDO.
RE: merge all netCDF files in a directory - Added by Ralf Mueller about 2 years ago
hi!
You can use mergetime
or merge
. in case you have each variable in separate files, you can also use cat
on these files, only.
Something like
cdo -merge temp_* all_temp.ncor
cdo -mergetime temp_* all_temp.ncor
cdo -timesort -cat [ temp_* ] all_temp.nc
give it a try
cheers
ralf
RE: merge all netCDF files in a directory - Added by Suvarna Tikle about 2 years ago
Hi
Thanks for reply.
I have used cdo mergetime. More than 24 hours no output.
Will try other two commands suggested by you.
Suvarna
RE: merge all netCDF files in a directory - Added by Suvarna Tikle about 2 years ago
Hi
I have used commands but hard luck.
cdo -timesort -cat [ _130*.nc ] all_130.nc
Error msg:Operator not found
cdo -merge _130*.nc all_130.nc
Error msg:Open failed on >_130*<
No such file or directory
Could you please help further?
my file names are as bellow , want to merge all _130 file for year 2019.
temp: 'E5ml00_1H_2019-05-01_130.nc'
WD: 'E5ml00_1H_2019-08-31_131.nc'
WD: 'E5ml00_1H_2019-12-31_132.nc'
Thanks
Suvarna
RE: merge all netCDF files in a directory - Added by Karin Meier-Fleischer about 2 years ago
1. The operator name is timsort not timesort.
2. you need to specify the file names (there are no 130*.nc files - do you mean E5ml00_1H*_130.nc ?
Try:
cdo -timsort -cat [ E5ml00_1H_*_130.nc ] all_130.nc
RE: merge all netCDF files in a directory - Added by Suvarna Tikle about 2 years ago
Thanks a lot.
its started processing.
RE: merge all netCDF files in a directory - Added by Suvarna Tikle about 2 years ago
It get stopped with msg "cdo(1) cat: 3%Killed"
RE: merge all netCDF files in a directory - Added by Karin Meier-Fleischer about 2 years ago
How many files do you try to merge? Maybe it is a memory issue or you have to use a job for it.
RE: merge all netCDF files in a directory - Added by Suvarna Tikle about 2 years ago
I have files are ranging from 340 to 365.