cdo splitsel splits the files - but the size is unchanged
Added by Mark Payne over 8 years ago
Hi,
I have a lon-lat-time file that is approximatley 8 MB that I would like to split into a collection of individual lon-lat files. I have used "splitsel" to do this as follows:
cdo splitsel,1 foo bar
and this works as expected, giving the expected 120 or so time step files. However, the wierd thing is that they are all exactly that same size as the original file - shouldn't there be an (approximate) conservation of size here? I have checked and as far as I can see, there is nothing wierd going on with compression or the likes.
Have attached the file here. Am running cdo 1.7.1.
Mark
Replies (3)
RE: cdo splitsel splits the files - but the size is unchanged - Added by Uwe Schulzweida over 8 years ago
Hi Mark,
Your data is stored in netCDF4/HDF5. I don't know how exactly the data is stored in HDF5 but it seems that HDF5 uses large blocks for better performance. Use netCDF3 to get smaller files:
cdo -f nc splitsel,1 foo barCheers,
Uwe
RE: cdo splitsel splits the files - but the size is unchanged - Added by Mark Payne over 8 years ago
Perfect! Thanks for the tip.
I've played a lot with chunking in the past, but this is the first time I've seen it create a problem like this. Am I correct in assuming that cdo is just copying the storage format from foo to bar, so that if foo is inappropriately chunked, so too will bar be?
Mark
RE: cdo splitsel splits the files - but the size is unchanged - Added by Uwe Schulzweida over 8 years ago
Yes, CDO is just copying the storage from foo to bar. The chunk parameter are only used for netCDF4.