Merging packed files
Added by Georgios Fragkoulidis 25 days ago
Hi everyone! I have applied the "cdo pack ..." operator on 2 netCDF files of precipitation in years 1998 and 1999: "file1998.nc" and "file1999.nc" (i.e., the two files have identical structure and only differ in the time axis). The resulting files are named: "file1998_packed.nc" and "file1999_packed.nc".
I then merge the 2 packed files: "cdo mergetime file1998_packed.nc file1999_packed.nc file1998-1999_packed.nc" without any error or warning. However, the merged file inherits the add_offset and scale_factor of the 1st file (file1998_packed.nc) and its 1999 values differ slightly to those of file1999_packed.nc (when viewed with ncview). So, mergetime probably applies the wrong scaling factors and offset to all but the 1st files one wants to merge.
Is there a solution to that, or should we avoid merging packed files?
Thanks!
Replies (5)
RE: Merging packed files - Added by Karin Meier-Fleischer 21 days ago
You should first merge the data and then pack the file. The documentation of 'pack' says that the add_offset and the scale_factor is calculated for all values (in a file). But you can generate those factors for each variable, store it in a separate file, and use the file name as input parameter of pack.
See https://code.mpimet.mpg.de/projects/cdo/embedded/cdo.pdf#subsection.2.2.4
Packing reduces the data volume by reducing the precision of the stored numbers. It is implemented using the NetCDF attributes add_offset and scale_factor. The operator pack calculates the attributes add_offset and scale_factor for all variables. The default data type for all variables is automatically changed to 16-bit integer. Use the CDO option -b to change the data type to a different integer precision, if needed. Missing values are automatically transformed to the current data type. Alternatively, the pack parameters add_offset and scale_factor can be read from a file for each variable.
RE: Merging packed files - Added by Georgios Fragkoulidis 20 days ago
Thank you for this. So, we should not merge packed files of the same variable in time. A warning would be good in this event.
On the other thing you note, could you give me the command to pack a file with the factors of another file? I don't understand the instructions on that. I tried for example "cdo pack,file1999_factors.txt file1998.nc file1998_packed.nc" and it throws "cdo pack (Abort): Too many arguments! Need 0 found 1.". What should the file with the factors look like?
RE: Merging packed files - Added by Karin Meier-Fleischer 20 days ago
You need a CDO version >= 2.4.0 to be able to use a file as input for the pack operator (search 'pack' at https://code.mpimet.mpg.de/projects/cdo/news).
RE: Merging packed files - Added by Karin Meier-Fleischer 20 days ago
Here is an example to pack 3 variables of the input file. But keep in mind that all other variables in the input file keep their data types.
factors_file.txt:
name=t add_offset=270.5633 scale_factor=0.001555741 name=u add_offset=1.117298 scale_factor=0.0008176539 name=v add_offset=0.4027506 scale_factor=0.0008038064
CDO pack command line:
cdo pack,filename=factors_file.txt infile.nc outfile.nc
RE: Merging packed files - Added by Georgios Fragkoulidis 20 days ago
Thank you Karin! That was very helpful.