Changing file structure based on season definition with ‘splitmon’ and ‘mergetime’
Added by Jorrit van der Schot almost 4 years ago
Hello everyone,
I am wondering if someone can help me with the following.
My data contains 244 ncdf files split into 4 files for each year in the time period 1958-2019. The files are split into the following seasons: JFM, AMJ, JAS, OND.
What I want is to change the file structure so that the files represent actual meteorological seasons: DJF, MAM, JJA, SON. (Example: 1960_DJF would contain 1959_Dec, 1960_Jan and 1960_Feb)
My approach is to combine the commands ‘splitmon’ and ‘mergetime’. I want to use a for-loop since I do not want to merge all files with mergetime, but rather only specific months. I have not been able to define a Linux bash.script that loops trough all my files in the correct way.
Does anyone have a suggestion for a way to formulate a bash.script to loop through all files with 'splitmon' and 'mergetime'? Or are there easier ways to do change ncdf file structure based on season definition with cdo?
Unfortunately I am not able to upload the ncdf files, but I have attached the ‘cdo sinfon’ output as a text document.
The files are named as follows:
precip_WJB_int.1958_AMJ.BN_1958_2013_1km.DD.nc precip_WJB_int.1958_JFM.BN_1958_2013_1km.DD.nc
precip_WJB_int.1958_JAS.BN_1958_2013_1km.DD.nc precip_WJB_int.1958_OND.BN_1958_2013_1km.DD.nc
The output of 'cdo splitmon' is as follows: (1 = Jan, 2 = Feb, etc.)
precip_WJB_int.1958_JFM.BN_1958_2013_1km.DD.nc01.nc precip_WJB_int.1958_JAS.BN_1958_2013_1km.DD.nc07.nc
precip_WJB_int.1958_JFM.BN_1958_2013_1km.DD.nc02.nc precip_WJB_int.1958_JAS.BN_1958_2013_1km.DD.nc08.nc
precip_WJB_int.1958_JFM.BN_1958_2013_1km.DD.nc03.nc precip_WJB_int.1958_JAS.BN_1958_2013_1km.DD.nc09.nc
precip_WJB_int.1958_AMJ.BN_1958_2013_1km.DD.nc04.nc precip_WJB_int.1958_OND.BN_1958_2013_1km.DD.nc10.nc
precip_WJB_int.1958_AMJ.BN_1958_2013_1km.DD.nc05.nc precip_WJB_int.1958_OND.BN_1958_2013_1km.DD.nc11.nc
precip_WJB_int.1958_AMJ.BN_1958_2013_1km.DD.nc06.nc precip_WJB_int.1958_OND.BN_1958_2013_1km.DD.nc12.nc
Please let me know if anything is unclear in my post or if more information is needed. Many thanks in advance.
Regards,
Jorrit
cdo_sinfon_output.txt (2.73 KB) cdo_sinfon_output.txt |
Replies (3)
RE: Changing file structure based on season definition with ‘splitmon’ and ‘mergetime’ - Added by Brendan DeTracey almost 4 years ago
Why keep separate files for seasons/months when cdo has seasonal/monthly operators? Why not mergetime
all your files into one large file?
$ cdo mergetime precip_WJB_int.????_???.BN_1958_2013_1km.DD.nc precip_WJB_int.BN_1958_2013_1km.DD.nc
Once you have done this, the trick to getting meteorological seasons is
shifttime
(https://code.mpimet.mpg.de/projects/cdo/embedded/index.html#x1-2410002.6.4). shifttime
back one month before any seasonal calculations(i.e. seastat
https://code.mpimet.mpg.de/projects/cdo/embedded/index.html#x1-5070002.8.27). Then shifttime
forward one month afterwards to correct the date.$ cdo -shifttime,1month -seasmax -shifttime,-1month precip_WJB_int.BN_1958_2013_1km.DD.nc precip_WJB_int.BN_1958_2013_1km.DD_seasmax.nc
timstat
operators (https://code.mpimet.mpg.de/projects/cdo/embedded/index.html#x1-4460002.8.16) can process your data without manually chopping your files up. I am trying to wean myself from using unneeded intermediate files, when possible.RE: Changing file structure based on season definition with ‘splitmon’ and ‘mergetime’ - Added by Jorrit van der Schot almost 4 years ago
Hi Brendan,
I appreciate your quick answer.
Why keep separate files for seasons/months when cdo has seasonal/monthly operators? Why not
mergetime
all your files into one large file?
That's a valid point. The reason I aimed for multiple files has to do with the fact that I am conducting most of my data analysis with Python. I am using CDO for some pre-processing steps like re-structuring the files and clipping my region of interest (sellonlatbox
). I have some scripts available that require the file structure to be as I envisioned it when I wrote my earlier post.
I agree with you however, that there might be easier ways to go about this. Another good option for me would be to create 4 files for the whole time series, one for each meteorological season.
If I want to achieve this however, my earlier question still stands. I am unsure how to use CDO in a for-loop to obtain these 4 files.
(...) the trick to getting meteorological seasons is
shifttime
(https://code.mpimet.mpg.de/projects/cdo/embedded/index.html#x1-2410002.6.4).shifttime
back one month before any seasonal calculations(i.e.seastat
https://code.mpimet.mpg.de/projects/cdo/embedded/index.html#x1-5070002.8.27). Thenshifttime
forward one month afterwards to correct the date.
Thank you for this suggestion, as this will be useful if I need to do any other pre-processing steps with CDO that requires this changed season definition.
RE: Changing file structure based on season definition with ‘splitmon’ and ‘mergetime’ - Added by Brendan DeTracey almost 4 years ago
I am re-reading the selseason
docs (https://code.mpimet.mpg.de/projects/cdo/embedded/index.html#x1-1670002.3.4) and although you can specify DJF
it does not help with grouping by year; shifttime
is still needed.
Okay. Well, it is still easier to start with one big file than to code a mix and match.
Create one big file as shown above:
cdo mergetime precip_WJB_int.????_???.BN_1958_2013_1km.DD.nc precip_WJB_int.BN_1958_2013_1km.DD.nc
Then you need a bash script to loop through seasons and years using the
selseason
and selyear
. I will use bash's associative arrays.declare -A season_map cdo_seasons='JFM AMJ JAS OND' season_map=( [JFM]=DJF [AMJ]=MAM [JAS]=JJA [OND]=SON ) for year in {1958..2013} ; do for season in $cdo_seasons ; do cdo shifttime,1month -selyear,"$year" -selseason,"$season" -shifttime,-1month precip_WJB_int.BN_1958_2013_1km.DD.nc precip_WJB_int."$year"_${season_map[$season]}.BN_1958_2013_1km.DD.nc done done
Since you know python and not bash, you could also write this in python using cdo's python bindings (https://code.mpimet.mpg.de/projects/cdo/wiki/Cdo%7Brbpy%7D). I have yet to try these. If you do, make sure you read https://github.com/Try2Code/cdo-bindings/blob/master/python/test/test_cdo.py as it shows both the old and new way to chain cdo operators.