Project

General

Profile

News

On the New Machine

Added by Christopher Kadow about 3 years ago

The change to the new MiKlip Machine is done.

https://code.zmaw.de/attachments/download/11689/MiKlipServerUmzug_explanation_UPDATE_19-01-2016.pdf

Therefore we have new PATHES:

Main data dir of MiKlip: /work/bmx825
Main climate data dir: /work/bmx825/data4miklip
Main projectdata path: /work/bmx825/data4miklip/projectdata/user-$USER
Main work dir of miklip: /work/bmx825/miklip-work
MiKlip-CES dir /work/bmx828/miklip-ces
Standardization Scripts: /work/bmx828/miklip-ces/misc4freva/cmor4freva
Repository of Plugins dir: /work/bmx828/miklip-ces/repos4freva

Therefore we renamed the evaluation_system to miklip-ces:

module load miklip-ces

Therefore OLD Commands are offline now:

analyze
solr_search

Please use freva:
Freva 
Available commands:
  --plugin       : Applies some analysis to the given data.
  --history      : provides access to the configuration history (use --help for more help)
  --databrowser  : Find data in the system
  --esgf         : Browse ESGF data and create wget script
  --crawl_my_data: Use this command to update your projectdata.

New features of the Web Evaluation System

Added by Oliver Kunst over 4 years ago

Dear MiKlippers!

We are happy to announce several new features of the evaluation system.

The new features at a short glance:
  • Adding a title to your results
  • Writing and leaving notes and comments belonging to the results.
  • The tool's repository version is saved in the history.
  • Developers can easily issue error messages and warnings.

Adding a title to your results

We put our hands on the result section to improve your workflow.
When you love tabbed browsing you certainly know the confusion
looking for a specific result between several tabs showing the same tool name.

Now, you can add a caption to your results page acting as title. On the result
page you find the button "add caption" which allows you to set a title for your
results. When you mark the "default" option, this caption will be shown to
everyone. We recommend to set a default caption before you share your
results.

You can also set the caption when starting a tool. Therefore a new field "caption"
appears at the bottom of the web interface. An "analyze --caption 'my genius title' --tool ...."
does the same from command line.

Writing and leaving notes and comments belonging to the results

A new section appears in the results called "Notes". There you can leave comments,
remarks and short notes. You can choose whether the comment should be private and
only be seen by yourself or if you would like to share this comment with the other
MiKilp members.

For team work a strategy could be to share the results after setting a caption
and gather the feedback in the notes section. Feel free to play around with the new
features.

The tool's repository version is saved in the history

Since all the tools underlying a lively development process, it could be hard to
reproduce some old results with new tool versions. The version management system git
keeps track of all the versions. Recently, the system stores the repository and git version
of your tool in the history. If you would like to know which version you used for an experiment
take a look to the configuration section of the results. The button "Additional Information"
reveals the version.

Developers can easily issue error messages and warnings

Last but not least we implemented a feature for developers to issue warnings and
error messages. When discovering a bug you probably would like to inform users that there
is an issue you are working on or even worse you have to shut down a tool
while fixing a severe bug. Please, contact us if you would like to use the option to
issue error messages and warnings and see also:
https://code.zmaw.de/projects/miklip-d-integration/wiki/Warnings_and_Errors

Sincerely,
your INTEGRATION Team

Web Evaluation System ... next Steps

Added by Christopher Kadow almost 5 years ago

Dear Users,

since the first announcement of the web-page
we developed many new features to support you and
your work-flow.

First of all, our powerful solr_search tool to find data
has got an online representation. It's named "data-browser"
in the main menu and you can locate more than 4 million
NetCDF files through a comfortable interface.

Are you annoyed of some results in your history you would like
to get rid of? Now, the web-page offers a delete button to hide
these results. To guarantee the scientific reproducibility you
can recover these results whenever you like.

Now, the system also supports to share your results with others.
Via the share button on the results page you can easily choose
MiKlip members to receive an email containing a link to your
results.

Last, but not least, we established a guest account with
restricted permissions. With the user "guest" and
password "miklip" you will get a guided tour through
the web-page showing all important features.

Then we are working on several small parts to improve the system and
listen carefully to the users. E.g. the system has now the possibility
to add informations about "what you see on it" directly to the plots.
An Example you find already in the ETCCDI and Monsoon tool.

If you have suggestions how2improve the system, write us!

Web of the Evaluation System - www-miklip.dkrz.de

Added by Christopher Kadow about 5 years ago

News from 30.01.2014

We are happy to announce that the evaluation system is available on the world wide web, now.
Under www-miklip.dkrz.de [sic!] you find the very first (beta)
version of the web based interface to the evaluation system.

The web page offers you all the opportunities of the evaluation system. It
provides an interface to the tools and allows you to start the tools in
a scheduler and watch their progress.

Old runs of the evaluation system can be viewed in the history and you can
comfortably browse a preview of the results. It is worth to mention that
you can access all your runs started with the evaluation system. The system
makes no difference whether the runs have been started with the web interface
or by using the shell.

There is a first documentation of the MurCSS tool and direct access to this wiki.

To access the full functionality of www-miklip.dkrz.de you have to login
with your DKRZ credentials to the web page.

NEWS from INTEGRATION - using&developing

Added by Christopher Kadow almost 6 years ago

NEWS from 19.07.2013

Many projects inside of MiKlip joined the Evaluation System and the MiKlip Server by using it, brought in analyses (MesoTel, EnsDiVal) and data (VADY, MesoTel).

The Basic User Guide (BUG) by INTEGRATION with an overview of what has been developed or plugged in and how2use and load the evaluation system you can find here:

https://code.zmaw.de/projects/miklip-d-integration/wiki/

Always in mind, working on two aspects: using&developing , today we want to pick up two areas of your interest.

SOLR_SEARCH

SOLR_SEARCH

The MiKlip Server and the mounted DKRZ ESGF node combine a huge database with direct (!No need to download all data from the ESGF or from slow band archives!) access to over 50 model systems and 2 million files. That is why standardization is so important, just to find everything and stay comparable with the international community. INTEGRATION combined (linked) the different areas to one

/miklip/integration/data4miklip

and developed the search tool "find_files", both already presented at the MiKlip Status-Seminar 2013. But any normal search is too slow, to run through that amount of data.

Therefore INTEGRATION developed the new search system "solr_search" for the ongoing evaluation system and scientists working at the MiKlip server. It has the same syntax as "find_files".

"solr_search" is a very fast searching solution with a user friendly 'tab' usability.

Looking for the BASELINE1 runs?

[b324031@miklip04 ~]$ module load evaluation_system

[b324031@miklip04 ~]$ solr_search ->PRESS TABULATOR
cmor_table=      ensemble=        institute=       project=         time_frequency=  
data_type=       experiment=      model=           realm=           variable=

[b324031@miklip04 ~]$ solr_search data_type= ->PRESS TABULATOR
baseline0     baseline1     cmip5         observations  reanalysis

[b324031@miklip04 ~]$ solr_search data_type=baseline1 ->PRESS TABULATOR
cmor_table=      experiment=      realm=           variable=        
ensemble=        model=           time_frequency=

[b324031@miklip04 ~]$ solr_search data_type=baseline1 model=mpi ->PRESS TABULATOR
mpi-esm-lr  mpi-esm-mr  

[b324031@miklip04 ~]$ solr_search data_type=baseline1 model=mpi-esm-lr ->PRESS TABULATOR
cmor_table=      ensemble=        realm=           time_frequency=  variable=      

[b324031@miklip04 ~]$ solr_search data_type=baseline1 model=mpi-esm-lr variable= ->PRESS TABULATOR
Display all 100 possibilities? (y or n)
areacella      clivi          dissoc         fddtdisi       hfls           intpn2         no3
areacello      clt            dms            ffire          hfsithermds    intpp          npp
baresoilfrac   clw            dpco2          fgco2          hfss           lai            o2
basin          clwvi          dpo2           fgdms          hfx            landcoverfrac  o2min
bfe            co2mass        epc100         fgo2           hfxba          masso          omldamax
bsi            co3            epcalc100      fluc           hfxdiff        mfo            omlmax
burntarea      co3satcalc     epfe100        frc            hfy            mlotst         orog
c3pftfrac      cropfrac       epsi100        frfe           hfyba          mlotstsq       pasturefrac
c4pftfrac      csoil          evspsbl        frn            hfydiff        mrro           pbfe
calc           cveg           fco2nat        fsfe           hur            mrros          pbo...
cct            dcalc          fddtalk        fveglitter     hus            mrso           
chl            deptho         fddtdic        gpp            intdic         mrsofc         
cl             detoc          fddtdife       grassfrac      intpbfe        msftbarot      
cli            dfe            fddtdin        graz           intpbsi        msftmyz        
clitter        dissic         fddtdip        hfds           intpcalcite    nbp

etc etc etc

Looking for models to compare to your results in temperature?

solr_search variable=ta model= ->PRESS TABULATOR
access1-0        cesm1-bgc        ec-earth         giss-e2-h-cc     ipsl-cm5a-lr     mpi-esm-p
access1-3        cesm1-cam5       fio-esm          giss-e2-r        ipsl-cm5a-mr     mri-agcm3-2h
bcc-csm1-1       cesm1-fastchem   geos-5           giss-e2-r-cc     ipsl-cm5b-lr     mri-cgcm3
bnu-esm          cesm1-waccm      gfdl-cm3         hadcm3           miroc4h          noresm1-m
canam4           cmcc-cesm        gfdl-esm2g       hadgem2-a        miroc5           noresm1-me
cancm4           cmcc-cm          gfdl-esm2m       hadgem2-ao       miroc-esm        
canesm2          cmcc-cms         gfdl-hiram-c180  hadgem2-cc       miroc-esm-chem   
ccsm4            cnrm-cm5         gfdl-hiram-c360  hadgem2-es       mpi-esm-lr       
cdas             csiro-mk3-6-0    giss-e2-h        inmcm4           mpi-esm-mr       

etc etc etc

Do we have also observations for temperature?

[b324031@miklip04 ~]$ solr_search data_type=observations variable=ta ->PRESS ENTER
/miklip/integration/data4miklip/observations/atmos/ta/mon/grid/NASA-JPL/MLS/v20111025/ta_MLS_L3_v03-3x_200408-201012.nc
/miklip/integration/data4miklip/observations/atmos/ta/mon/grid/NASA-JPL/AIRS/v20110608/ta_AIRS_L3_RetStd-v5_200209-201105.nc

SOLR_SEARCH can be used for developing analyses getting direct access to datasets using their meta data and it makes the ensemble handling in CMOR/CMIP5 datasets much easier! It will be also the basement of the hybrid "shell/web/operational" solution of the evaluation system, to assure a fast and effective use of the evaluation system and the MiKlip server. The system is in the beta phase and ready to be filled up with the regional data as well as the next general MiKlip runs.

More and detailed informations:
https://code.zmaw.de/projects/miklip-d-integration/wiki/Solr_search

PROJECTDATA

Our project combined different areas of data in

/miklip/integration/data4miklip

This is about data not belonging to the general runs of MiKlip. Where could you put them to compare them to general MiKlip data?

We set up a projectdata directory:

/miklip/integration/data4miklip/projectdata

If you have your CMORized data e.g. in your scratch, send us an email, then we link it to the system.

What does CMORized structure mean?

Let's have a look at user data from MESOTEL Module C - b324034

/miklip/integration/data4miklip/projectdata/b324034

This is a link b324034 -> /scratch/b324034/archive/CMIP5/ You see the user decided to keep the standard CMIP5 CMOR settings, to test his data, therefore just use the switch in the MPI-ESM for postproduction.

When we go deeper into the structure we find e.g.

/miklip/integration/data4miklip/projectdata/b324034/output/MPI-M/MPI-ESM-LR/decadal2000/mon/atmos/tas/r1i1p2/tas_Amon_MPI-ESM-LR_decadal2000_r1i1p2_200101-201012.nc

You see we need a specific directory structure to put in. Of course you already had a look into CMOR :) If not, here is a short description what you need in your data structure for linking in and the example you see above.

EXAMPLE:

b324034/output/MPI-M/MPI-ESM-LR/decadal2000/mon/atmos/tas/r1i1p2/tas_Amon_MPI-ESM-LR_decadal2000_r1i1p2_200101-201012.nc

DIRECTORY:

project/product/institute/model/experiment/time_frequency/realm/variable/ensemble/

FILE:

"variable"_"cmor_table"_"model"_"experiment"_"ensemble"_"start_time"-"end_time".nc

And now the different stories coming together as one!

If you consider solr_search now and look for your data, just do:

solr_search project=b324034 experiment=decadal2000 variable= -> PRESS TAB
Display all 100 possibilities? (y or n) -> n
solr_search project=b324034 experiment=decadal2000 variable=tas time_frequency=mon -> PRESS ENTER
/miklip/integration/data4miklip/projectdata/b324034/output/MPI-M/MPI-ESM-LR/decadal2000/mon/atmos/tas/r1i1p2/tas_Amon_MPI-ESM-LR_decadal2000_r1i1p2_200101-201012.nc

And that means you can ask solr_search for data to compare to, here baseline1-LR!

solr_search project=b324034 project=baseline1 experiment=dec*2000 variable=tas time_frequency=mon model=mpi-esm-lr
/miklip/integration/data4miklip/model/baseline1/output/MPI-M/MPI-ESM-LR/decs4e2000/mon/atmos/tas/r9i1p1/tas_Amon_MPI-ESM-LR_decs4e2000_r9i1p1_200101-201012.nc
/miklip/integration/data4miklip/model/baseline1/output/MPI-M/MPI-ESM-LR/decs4e2000/mon/atmos/tas/r8i1p1/tas_Amon_MPI-ESM-LR_decs4e2000_r8i1p1_200101-201012.nc
/miklip/integration/data4miklip/model/baseline1/output/MPI-M/MPI-ESM-LR/decs4e2000/mon/atmos/tas/r7i1p1/tas_Amon_MPI-ESM-LR_decs4e2000_r7i1p1_200101-201012.nc
/miklip/integration/data4miklip/model/baseline1/output/MPI-M/MPI-ESM-LR/decs4e2000/mon/atmos/tas/r6i1p1/tas_Amon_MPI-ESM-LR_decs4e2000_r6i1p1_200101-201012.nc
/miklip/integration/data4miklip/model/baseline1/output/MPI-M/MPI-ESM-LR/decs4e2000/mon/atmos/tas/r5i1p1/tas_Amon_MPI-ESM-LR_decs4e2000_r5i1p1_200101-201012.nc
/miklip/integration/data4miklip/model/baseline1/output/MPI-M/MPI-ESM-LR/decs4e2000/mon/atmos/tas/r4i1p1/tas_Amon_MPI-ESM-LR_decs4e2000_r4i1p1_200101-201012.nc
/miklip/integration/data4miklip/model/baseline1/output/MPI-M/MPI-ESM-LR/decs4e2000/mon/atmos/tas/r3i1p1/tas_Amon_MPI-ESM-LR_decs4e2000_r3i1p1_200101-201012.nc
/miklip/integration/data4miklip/model/baseline1/output/MPI-M/MPI-ESM-LR/decs4e2000/mon/atmos/tas/r2i1p1/tas_Amon_MPI-ESM-LR_decs4e2000_r2i1p1_200101-201012.nc
/miklip/integration/data4miklip/model/baseline1/output/MPI-M/MPI-ESM-LR/decs4e2000/mon/atmos/tas/r1i1p1/tas_Amon_MPI-ESM-LR_decs4e2000_r1i1p1_200101-201012.nc
/miklip/integration/data4miklip/model/baseline1/output/MPI-M/MPI-ESM-LR/decs4e2000/mon/atmos/tas/r10i1p1/tas_Amon_MPI-ESM-LR_decs4e2000_r10i1p1_200101-201012.nc
/miklip/integration/data4miklip/projectdata/b324034/output/MPI-M/MPI-ESM-LR/decadal2000/mon/atmos/tas/r1i1p2/tas_Amon_MPI-ESM-LR_decadal2000_r1i1p2_200101-201012.nc

If you want to put in other data, like reanalysis to test something you can of course use this too. E.g.:

b324034/reanalysis/ECMWF/IFS/ERAINT/mon/atmos/tas/r1i1p1/tas_Amon_reanalysis_ERAINT_r1i1p1_197901-201212.nc

For help getting your specific data CMORized, please have a look into:

/miklip/integration/data_management/standardization_examples

There you find 1 example for decadal, historical, reanalysis and observation standardization.

This is the idea of bringing data together instead of changing the tools for different datasets. All ongoing projects developing analyses, will use this structure. If we can handle that in MiKlip, we will have a fast, effective, growing, and open evaluation system.

Please contact us, if you have any question or advice.

Christopher Kadow -
and
Sebastian Illing -

    (1-5/5)

    Also available in: Atom