Project

General

Profile

automated and efficient remapping + data selection

Added by Juri Backes over 3 years ago

Hello, I want to log daily 24h prediction data of the ICON-D2-EPS Model for different variables and coordinates. To be more specific, every day I have a list of coordinates [(lat1,lon1)...] (which are all located in northern germany), for each coordinate i want the 24h hour forecast of the variables [x,y,z...]. Since the available data (like this one: https://opendata.dwd.de/weather/nwp/icon-d2-eps/grib/03/v/) provides a single file for every forecast hour per variable, I am ending up with a lot of files.
To extract the data:
  • First of all I download every file I need.
  • Then I run a loop over each file executing cdo remap with a pre-computed weight file (using the resources from ICON_D2_002_EASY.tar.bz2: https://opendata.dwd.de/weather/lib/cdo/), in order to receive a lat/lon grid.
  • Now (again) I run a loop over each coordinate to create a single coordinate file from the previously remapped files, using the remapnn command
  • In the last step I'm extracting the ensemble variables of each single coordinate file via the cdo outputtab command.

I know this isn't surprising to you, but it takes a lot of time (for 20+ coordinates, and 5 variables), this because I am probably using cdo in the most inefficient way possible. I have tried to speed up the process in different ways, but I have come to a dead end reading blog post after blog post (and not understanding the documentation properly). Now my question is, if you have any suggestions on how to speed up the whole process.

Thank you in advance
Juri