Project

General

Profile

Forums » Open discussion »

Issue with "yhourmean" Command Taking Too Long on ERA5 Hourly Pressure Level Data

Added by Trushna Mallick 5 days ago

Hello,

I used the new API beta version to download ERA5 reanalysis hourly pressure level data. The NetCDF file contains data for March, April, and May from 1990 to 2021 at hourly intervals, specifically for the 850 hPa pressure level. The file size is quite large (282.9 MB), and due to size restrictions, I cannot upload it here.

I wanted to calculate the multi-year hourly mean and used the following command:

cdo yhourmean inputfile.nc output.nc

While executing this, I received three warnings:

Warning (cdfScanVarAttr): NetCDF: Variable not found - >isobaricInhPa<
Warning (cdf_check_variables): Unsupported data type (char/string), skipped variable expver!
Warning (cdfInqContents): Coordinates variable number can't be assigned!

To check whether the NetCDF file was valid, I ran a basic command:

cdo sinfo inputfile.nc

This also gave warnings but still displayed all the details of the file.

I'm worried because the "yhourmean" command has been running for over 10 hours(multiple times tried but failed), but the output file is still only 16.2 KBand still processing. Is it normal for this command to take so long when calculating the multi-year hourly mean for 31 years (1990–2021), or could there be a problem with the data or the way I'm using the command?

Any help will be highly appreciated.
Thank you


Replies (5)

RE: Issue with "yhourmean" Command Taking Too Long on ERA5 Hourly Pressure Level Data - Added by Karin Meier-Fleischer 5 days ago

Did you use zip when downloading the data in netcdf (experimental) format?
It would be good when you extract and upload 2-3 years?

RE: Issue with "yhourmean" Command Taking Too Long on ERA5 Hourly Pressure Level Data - Added by Trushna Mallick 4 days ago

Hello karin

No I didnt zip while donwloading the data in netcdf(experimental) format.
Here, I am uploading a NC file of 3 years.

u_march_3y.nc (73.7 MB) u_march_3y.nc

RE: Issue with "yhourmean" Command Taking Too Long on ERA5 Hourly Pressure Level Data - Added by Uwe Schulzweida 4 days ago

The data is chunked and compressed over 1116 time steps. This means that cdo must keep all the 1116 steps in the main memory. You need the latest CDO version 2.5.1 to handle this with best performance.
With older CDO versions, the required chunk cache can be set with the environment variable CDI_CHUNK_CACHE.
Try it with 8gb:

CDI_CHUNK_CACHE=8gb cdo yhourmean inputfile.nc output.nc
See also https://code.mpimet.mpg.de/boards/2/topics/16082.

RE: Issue with "yhourmean" Command Taking Too Long on ERA5 Hourly Pressure Level Data - Added by Trushna Mallick 4 days ago

Hi Uwe

I upgraded to the CDO version 2.5.1, it worked well, and now 'yhourmean' command runs in just seconds.

Thank you

    (1-5/5)