CDO

Hi,
Not sure I completely understand but cat with globbing would be a better first step. Assuming bash shell:

$ cdo cat dtr_m_ECEarth_PD_s01r00_20??.nc dtr_m_ECEarth_PD_s01r00_2025-2039.nc

for r_number in {0..24} ; do r_tag='r'$(printf '%02d' "$r_number") ; echo "$r_tag" ; cdo cat dtr_m_ECEarth_PD_s01"$r_tag"_20??.nc dtr_m_ECEarth_PD_s01"$r_tag"_2035-2039.nc ; done

On second thought, include timeshift in the pipeline. Maybe:

$ for r_number in {0..24} ; do r_tag='r'$(printf '%02d' "$r_number") ; echo "$r_tag" ; cdo cat -shifttime,$(( $r_number * 5 ))year dtr_m_ECEarth_PD_s01"$r_tag"_20??.nc dtr_m_ECEarth_PD_s01r00-24_2035-2159.nc ; done

Brendan DeTracey wrote:

On second thought, include timeshift in the pipeline. Maybe:
[...]
The cat operator, appends to the output file if it already exists. Make sure you delete it before running the above code more than once.

Hi Brendan,

first of all, a massive thank you for the help. This is much more efficient and subtle than what I had in mind. I do indeed need to learn my way around bash as the more I get into climate data, the more I see xarray and other python libraries are incapable of solving issues.

The only thing that seems to be off is that while the first year of each run updates as expected (2040,45,50...), the following years keep the original date(2040, 2036, 2037,....2050, 2036,...). I'm trying to see where the problem is, but apparently the shifttime changing according to the r_number should work for all years of each run. Would you know why the command is not being applied for the following years?

A noob question: Is

??

*

Thank you once again!

Best regards,

Henrique

Sorry mate. It should have been:

$ for r_number in {0..24} ; do r_tag='r'$(printf '%02d' "$r_number") ; echo "$r_tag" ; cdo cat -apply,shifttime,$(( $r_number * 5 ))year [ dtr_m_ECEarth_PD_s01"$r_tag"_20??.nc ] dtr_m_ECEarth_PD_s01r00-24_2035-2159.nc ; done

Since you know python(and xarray), you might want to consider the python cdo bindings (https://code.mpimet.mpg.de/projects/cdo/wiki/Cdo%7Brbpy%7D). bash is great for quick and dirty pattern matching, but the python globbing module does the same, although you might have to sort the list of files afterwards. I can't remember. Also, using the cdo python bindings allows direct xarray access. I have yet to use these bindings, but make sure you read https://github.com/Try2Code/cdo-bindings/blob/master/python/test/test_cdo.py as it shows both the old and new way to assemble a cdo pipeline.

Edit: by pipeline I meant cdo chaining.

One last possible bump. With cdo 1.9.9 you must now add a - before the first argument to apply.

$ for r_number in {0..24} ; do r_tag='r'$(printf '%02d' "$r_number") ; echo "$r_tag" ; cdo cat apply,shifttime,$(( $r_number * 5 ))year [ dtr_m_ECEarth_PD_s01"$r_tag"_20??.nc ] dtr_m_ECEarth_PD_s01r00-24_2035-2159.nc ; done

I was testing under cygwin cdo, which is currently 1.9.8, but just gotten bitten by this change using cdo 1.9.9 on linux!

And I forgot to add it to my last post:

$ for r_number in {0..24} ; do r_tag='r'$(printf '%02d' "$r_number") ; echo "$r_tag" ; cdo cat -apply,-shifttime,$(( $r_number * 5 ))year [ dtr_m_ECEarth_PD_s01"$r_tag"_20??.nc ] dtr_m_ECEarth_PD_s01r00-24_2035-2159.nc ; done

Hi Brendan,

Sorry for the delay, I wrote this reply 6 days ago but apparently it was not submitted.

Thank you again for the reply. I had no clue about -apply, but now it is working indeed, that's outstanding!

Also, great to know about the bindings for python, first time I ever hear about them. I'll try them out (but also will practice some shell just for the sake of it).

Thank you once more, Brendan!