Robustness of mergetime and cat operators
Added by Kieran Lynch about 11 years ago
Hi,
I have been having some problems with the 'mergetime' and 'cat' operators missing timesteps when processing large amounts of data.
One example is using mergetime on 12 small grib files to generate one output file. I have had to do this ~27,000 times and found that 9 of the output files had one timestep missing. Is this likely to be some I/O error from the hard disk rather than an internal cdo code issue? Has anyone else had this problem? In the future, should I check after writing the output file that it has the required number of timesteps with the operator 'ntime'?
Regards,
Kieran
Replies (1)
RE: Robustness of mergetime and cat operators - Added by Jaison-Thomas Ambadan about 11 years ago
Hi,
From an earlier post by Uwe Schulzweida : "it is a good praxis to name/rename your files in a way that there are alphanumerically sorted. it is very important for the CDO operators copy, cat and mergetime."
are you sure if all of your input files are alphanumerically sorted?
Also, sometimes multi-thread operations may create problems so you could try the mergetime with "-L option". From an earlier post by Uwe Schulzweida: "Since CDO version 1.5.8 you can lock the I/O with the option -L. This will serialize all I/O accesses."
Hope this helps!
Cheers,
J.