Project

General

Profile

CDO ยป

Process FDB data with CDO

CDO version 2.4.0 supports the reading and writing of FDB (Fields DataBase) data. FDB is a domain-specific object store developed at ECMWF for storing, indexing and retrieving GRIB data.
The CDO support for FDB is under construction, experimental, and not further documented! So your feedback is very welcome!

CDO with FDB support

The FDB support in CDO is activated with the configure option:

 --with-fdb5=<location of fdb5 library>
A CDO version with FDB support is available at levante:/home/m/m214003/local/bin/cdofdb.

FDB config file

A config file is required to access FDB. You will get this config file from your FDB data provider. Config files define a number of parameters for the FDB. This basically describes the location of the database schema and the data. The environment variable FDB5_CONFIG_FILE needs to be set to a file path where the config file lives. 

export FDB5_CONFIG_FILE=/path/to/fdb/config/config.yaml

FDB schema

The schema describes rules for accessing all data stored by the FDB. These rules are described by attributes (key/values). To access the FDB, some of these attributes must be specified. The more attributes are specified, the more precisely the required data is described. Each FDB schema can use different attributes!

Accessing FDB with CDO

In CDO, FDB is then accessed with fdb://<attributes> instead of a file name.

Examples

FDB data of a coupled IFS-FESOM model is used for the examples. The data is available on 3 different grids (native|healix|latlon). Each grid is stored in its own FDB. A config file is required for each FDB. The data on the healpix grid is used in the examples. The healpix grid is available in the resolutions high and standard. The resolution must be selected with the resolution attribute. The data from IFS and FESOM have a different temporal resolution. IFS data is stored with hourly frequency and FESOM data with daily frequency. The temporal resolution must be selected with the attribute levtype (level type). IFS data has the level types
  • pl: pressure levels
  • sfc: surface
and FESOM has the level types
  • o2d: ocean 2d
  • o3d: ocean 3d

Other important attributes are param and date/time for selecting the required parameters and time period. Have a look to the FDB schema file for all available attributes.
The following example is using the CDO function sinfon on the hourly pressure level data from 2020-02-01:

export ECCODES_DEFINITION_PATH=/work/bm1235/b382757/ifs-bundle/source/eccodes/definitions
export FDB5_CONFIG_FILE=/work/bm1235/b382776/cycle4/fdb/healpix/etc/fdb/config.yaml

CDO=/home/m/m214003/local/bin/cdofdb
EXPID="class=d1,dataset=climate-dt,activity=ScenarioMIP,experiment=SSP3-7.0,expver=hz9o,stream=clte" 

$CDO sinfon fdb://${EXPID},resolution=standard,levtype=pl,date=20200201
ECCODES_DEFINITION_PATH needs to be set for reading data on the healpix grid with the latest ecCodes tables.

The second example selects the surface pressure (name=sp;param=134) from the surface dataset:

export ECCODES_DEFINITION_PATH=/work/bm1235/b382757/ifs-bundle/source/eccodes/definitions
export FDB5_CONFIG_FILE=/work/bm1235/b382776/cycle4/fdb/healpix/etc/fdb/config.yaml

CDO=/home/m/m214003/local/bin/cdofdb
EXPID="class=d1,dataset=climate-dt,activity=ScenarioMIP,experiment=SSP3-7.0,expver=hz9o,stream=clte" 

$CDO copy fdb://${EXPID},resolution=standard,levtype=sfc,date=20200201,param=134  sp.grb
Of course, attributes such as date and param can also be selected after the database query with CDO. However, this should be avoided if possible, as this causes a significant increase in FDB traffic for the meta-data of the fields that are not required.
$CDO select,name=sp fdb://${EXPID},resolution=standard,levtype=sfc,date=20200201  sp.grb

Limitations

The CDO support for FDB has been implemented as a prototype for testing. This prototype has not been fully tested and the user interface is kept very simple. Here is a list of the limitations:

  • Expansion of FDB attributes is not yet implemented. Each attribute may only have one value, e.g. date=20200201.
  • The value of the param attribute is the unique parameter ID. Parameter names are not supported.
  • Incorrect attributes lead to the CDO error message: FDB request doesn't find any database entries!
    To solve such problems, the FDB tools are recommended.

FDB tools

FDB has a set of general purpose CLI tools for archiving and retrieving user data, some special purpose tools for administrative maintenance and for developer tools for debugging. The following tools provide an overview of the FDB:

  • General FDB information with fdb-info:
    export FDB_PATH=/work/bm1235/u233156/ifs-bundle/ifs-bundle_nextGEMS_prod/build-fesom/bin
export FDB5_CONFIG_FILE=/work/bm1235/b382776/cycle4/fdb/healpix/etc/fdb/config.yaml

${FDB_PATH}/fdb-info --all
Version: 5.11.99
Home: /work/bm1235/u233156/ifs-bundle/ifs-bundle_nextGEMS_prod/build-fesom/
Schema: /work/bm1235/b382776/cycle4/fdb/healpix/etc/fdb/schema
Config: /work/bm1235/b382776/cycle4/fdb/healpix/etc/fdb/config.yaml
Lustre-API: enabled
  • Print the schema with fdb-schema:
    export FDB_PATH=/work/bm1235/u233156/ifs-bundle/ifs-bundle_nextGEMS_prod/build-fesom/bin
export FDB5_CONFIG_FILE=/work/bm1235/b382776/cycle4/fdb/healpix/etc/fdb/config.yaml

${FDB_PATH}/fdb-schema:
...
[class=d1,dataset=climate-dt,activity,experiment,generation,model,realization,expver,stream=clte/wave,date[resolution,type,levtype[time,levelist?,param,frequency?,direction?]]]
...
  • Print the location of the data with fdb-status:
    export FDB_PATH=/work/bm1235/u233156/ifs-bundle/ifs-bundle_nextGEMS_prod/build-fesom/bin
export FDB5_CONFIG_FILE=/work/bm1235/b382776/cycle4/fdb/healpix/etc/fdb/config.yaml

EXPID="class=d1,dataset=climate-dt,activity=ScenarioMIP,experiment=SSP3-7.0,expver=hz9o,stream=clte" 

${FDB_PATH}/fdb-status ${EXPID},resolution=standard,levtype=pl,date=20200201
Database: {class=d1,dataset=climate-dt,activity=ScenarioMIP,experiment=SSP3-7.0,generation=1,model=IFS-FESOM,realization=1,expver=hz9o,stream=clte,date=20200201}
  location: /work/bm1235/b382776/cycle4/fdb/healpix/root/d1:climate-dt:ScenarioMIP:SSP3-7.0:1:IFS-FESOM:1:hz9o:clte:20200201