Project

General

Profile

Changing file structure based on season definition with ‘splitmon’ and ‘mergetime’

Added by Jorrit van der Schot over 3 years ago

Hello everyone,

I am wondering if someone can help me with the following.

My data contains 244 ncdf files split into 4 files for each year in the time period 1958-2019. The files are split into the following seasons: JFM, AMJ, JAS, OND.

What I want is to change the file structure so that the files represent actual meteorological seasons: DJF, MAM, JJA, SON. (Example: 1960_DJF would contain 1959_Dec, 1960_Jan and 1960_Feb)

My approach is to combine the commands ‘splitmon’ and ‘mergetime’. I want to use a for-loop since I do not want to merge all files with mergetime, but rather only specific months. I have not been able to define a Linux bash.script that loops trough all my files in the correct way.

Does anyone have a suggestion for a way to formulate a bash.script to loop through all files with 'splitmon' and 'mergetime'? Or are there easier ways to do change ncdf file structure based on season definition with cdo?

Unfortunately I am not able to upload the ncdf files, but I have attached the ‘cdo sinfon’ output as a text document.

The files are named as follows:

precip_WJB_int.1958_AMJ.BN_1958_2013_1km.DD.nc precip_WJB_int.1958_JFM.BN_1958_2013_1km.DD.nc
precip_WJB_int.1958_JAS.BN_1958_2013_1km.DD.nc precip_WJB_int.1958_OND.BN_1958_2013_1km.DD.nc

The output of 'cdo splitmon' is as follows: (1 = Jan, 2 = Feb, etc.)

precip_WJB_int.1958_JFM.BN_1958_2013_1km.DD.nc01.nc precip_WJB_int.1958_JAS.BN_1958_2013_1km.DD.nc07.nc
precip_WJB_int.1958_JFM.BN_1958_2013_1km.DD.nc02.nc precip_WJB_int.1958_JAS.BN_1958_2013_1km.DD.nc08.nc
precip_WJB_int.1958_JFM.BN_1958_2013_1km.DD.nc03.nc precip_WJB_int.1958_JAS.BN_1958_2013_1km.DD.nc09.nc

precip_WJB_int.1958_AMJ.BN_1958_2013_1km.DD.nc04.nc precip_WJB_int.1958_OND.BN_1958_2013_1km.DD.nc10.nc
precip_WJB_int.1958_AMJ.BN_1958_2013_1km.DD.nc05.nc precip_WJB_int.1958_OND.BN_1958_2013_1km.DD.nc11.nc
precip_WJB_int.1958_AMJ.BN_1958_2013_1km.DD.nc06.nc precip_WJB_int.1958_OND.BN_1958_2013_1km.DD.nc12.nc

Please let me know if anything is unclear in my post or if more information is needed. Many thanks in advance.

Regards,

Jorrit


Replies (3)

RE: Changing file structure based on season definition with ‘splitmon’ and ‘mergetime’ - Added by Brendan DeTracey over 3 years ago

Why keep separate files for seasons/months when cdo has seasonal/monthly operators? Why not mergetime all your files into one large file?

$ cdo mergetime precip_WJB_int.????_???.BN_1958_2013_1km.DD.nc precip_WJB_int.BN_1958_2013_1km.DD.nc

Once you have done this, the trick to getting meteorological seasons is shifttime (https://code.mpimet.mpg.de/projects/cdo/embedded/index.html#x1-2410002.6.4). shifttime back one month before any seasonal calculations(i.e. seastat https://code.mpimet.mpg.de/projects/cdo/embedded/index.html#x1-5070002.8.27). Then shifttime forward one month afterwards to correct the date.
$ cdo -shifttime,1month -seasmax -shifttime,-1month precip_WJB_int.BN_1958_2013_1km.DD.nc precip_WJB_int.BN_1958_2013_1km.DD_seasmax.nc

timstat operators (https://code.mpimet.mpg.de/projects/cdo/embedded/index.html#x1-4460002.8.16) can process your data without manually chopping your files up. I am trying to wean myself from using unneeded intermediate files, when possible.

RE: Changing file structure based on season definition with ‘splitmon’ and ‘mergetime’ - Added by Jorrit van der Schot over 3 years ago

Hi Brendan,

I appreciate your quick answer.

Why keep separate files for seasons/months when cdo has seasonal/monthly operators? Why not mergetime all your files into one large file?

That's a valid point. The reason I aimed for multiple files has to do with the fact that I am conducting most of my data analysis with Python. I am using CDO for some pre-processing steps like re-structuring the files and clipping my region of interest (sellonlatbox). I have some scripts available that require the file structure to be as I envisioned it when I wrote my earlier post.

I agree with you however, that there might be easier ways to go about this. Another good option for me would be to create 4 files for the whole time series, one for each meteorological season.

If I want to achieve this however, my earlier question still stands. I am unsure how to use CDO in a for-loop to obtain these 4 files.

(...) the trick to getting meteorological seasons is shifttime (https://code.mpimet.mpg.de/projects/cdo/embedded/index.html#x1-2410002.6.4). shifttime back one month before any seasonal calculations(i.e. seastat https://code.mpimet.mpg.de/projects/cdo/embedded/index.html#x1-5070002.8.27). Then shifttime forward one month afterwards to correct the date.

Thank you for this suggestion, as this will be useful if I need to do any other pre-processing steps with CDO that requires this changed season definition.

RE: Changing file structure based on season definition with ‘splitmon’ and ‘mergetime’ - Added by Brendan DeTracey over 3 years ago

I am re-reading the selseason docs (https://code.mpimet.mpg.de/projects/cdo/embedded/index.html#x1-1670002.3.4) and although you can specify DJF it does not help with grouping by year; shifttime is still needed.
Okay. Well, it is still easier to start with one big file than to code a mix and match.
Create one big file as shown above:

cdo mergetime precip_WJB_int.????_???.BN_1958_2013_1km.DD.nc precip_WJB_int.BN_1958_2013_1km.DD.nc

Then you need a bash script to loop through seasons and years using the selseason and selyear. I will use bash's associative arrays.
declare -A season_map
cdo_seasons='JFM AMJ JAS OND'
season_map=( [JFM]=DJF [AMJ]=MAM [JAS]=JJA [OND]=SON )
for year in {1958..2013} ; do
    for season in $cdo_seasons ; do
        cdo shifttime,1month -selyear,"$year" -selseason,"$season" -shifttime,-1month precip_WJB_int.BN_1958_2013_1km.DD.nc precip_WJB_int."$year"_${season_map[$season]}.BN_1958_2013_1km.DD.nc
    done
done

Since you know python and not bash, you could also write this in python using cdo's python bindings (https://code.mpimet.mpg.de/projects/cdo/wiki/Cdo%7Brbpy%7D). I have yet to try these. If you do, make sure you read https://github.com/Try2Code/cdo-bindings/blob/master/python/test/test_cdo.py as it shows both the old and new way to chain cdo operators.

    (1-3/3)