Integration: Newshttps://code.mpimet.mpg.de/https://code.mpimet.mpg.de/favicon.ico?17097380842016-04-12T14:39:55ZProject Management Service
Redmine On the New Machinehttps://code.mpimet.mpg.de/news/3462016-04-12T14:39:55ZChristopher Kadowchristopher.kadow@met.fu-berlin.de
<p>The change to the new MiKlip Machine is done.</p>
<p><a class="external" href="https://code.zmaw.de/attachments/download/11689/MiKlipServerUmzug_explanation_UPDATE_19-01-2016.pdf">https://code.zmaw.de/attachments/download/11689/MiKlipServerUmzug_explanation_UPDATE_19-01-2016.pdf</a></p>
<p>Therefore we have new PATHES:<br /><pre>
Main data dir of MiKlip: /work/bmx825
Main climate data dir: /work/bmx825/data4miklip
Main projectdata path: /work/bmx825/data4miklip/projectdata/user-$USER
Main work dir of miklip: /work/bmx825/miklip-work
MiKlip-CES dir /work/bmx828/miklip-ces
Standardization Scripts: /work/bmx828/miklip-ces/misc4freva/cmor4freva
Repository of Plugins dir: /work/bmx828/miklip-ces/repos4freva
</pre></p>
<p>Therefore we renamed the evaluation_system to miklip-ces:<br /><pre>
module load miklip-ces
</pre></p>
<p>Therefore OLD Commands are offline now:<br /><pre>
analyze
solr_search
</pre><br />Please use <strong>freva</strong>:<br /><pre>
Freva
Available commands:
--plugin : Applies some analysis to the given data.
--history : provides access to the configuration history (use --help for more help)
--databrowser : Find data in the system
--esgf : Browse ESGF data and create wget script
--crawl_my_data: Use this command to update your projectdata.
</pre></p> New features of the Web Evaluation Systemhttps://code.mpimet.mpg.de/news/2532014-07-30T13:29:59ZOliver Kunstoliver.kunst@met.fu-berlin.de
<p>Dear MiKlippers!</p>
<p>We are happy to announce several new features of the evaluation system.</p>
The new features at a short glance:
<ul>
<li>Adding a title to your results</li>
<li>Writing and leaving notes and comments belonging to the results.</li>
<li>The tool's repository version is saved in the history.</li>
<li>Developers can easily issue error messages and warnings.</li>
</ul>
<a name="Adding-a-title-to-your-results"></a>
<h3 >Adding a title to your results<a href="#Adding-a-title-to-your-results" class="wiki-anchor">¶</a></h3>
<p>We put our hands on the result section to improve your workflow.<br />When you love tabbed browsing you certainly know the confusion<br />looking for a specific result between several tabs showing the same tool name.</p>
<p>Now, you can add a caption to your results page acting as title. On the result<br />page you find the button "add caption" which allows you to set a title for your<br />results. When you mark the "default" option, this caption will be shown to<br />everyone. We recommend to set a default caption before you share your<br />results.</p>
<p>You can also set the caption when starting a tool. Therefore a new field "caption" <br />appears at the bottom of the web interface. An "analyze --caption 'my genius title' --tool ...." <br />does the same from command line.</p>
<a name="Writing-and-leaving-notes-and-comments-belonging-to-the-results"></a>
<h3 >Writing and leaving notes and comments belonging to the results<a href="#Writing-and-leaving-notes-and-comments-belonging-to-the-results" class="wiki-anchor">¶</a></h3>
<p>A new section appears in the results called "Notes". There you can leave comments,<br />remarks and short notes. You can choose whether the comment should be private and<br />only be seen by yourself or if you would like to share this comment with the other<br />MiKilp members.</p>
<p>For team work a strategy could be to share the results after setting a caption<br />and gather the feedback in the notes section. Feel free to play around with the new<br />features.</p>
<a name="The-tools-repository-version-is-saved-in-the-history"></a>
<h3 >The tool's repository version is saved in the history<a href="#The-tools-repository-version-is-saved-in-the-history" class="wiki-anchor">¶</a></h3>
<p>Since all the tools underlying a lively development process, it could be hard to<br />reproduce some old results with new tool versions. The version management system git<br />keeps track of all the versions. Recently, the system stores the repository and git version<br />of your tool in the history. If you would like to know which version you used for an experiment<br />take a look to the configuration section of the results. The button "Additional Information" <br />reveals the version.</p>
<a name="Developers-can-easily-issue-error-messages-and-warnings"></a>
<h3 >Developers can easily issue error messages and warnings<a href="#Developers-can-easily-issue-error-messages-and-warnings" class="wiki-anchor">¶</a></h3>
<p>Last but not least we implemented a feature for developers to issue warnings and<br />error messages. When discovering a bug you probably would like to inform users that there<br />is an issue you are working on or even worse you have to shut down a tool<br />while fixing a severe bug. Please, contact us if you would like to use the option to<br />issue error messages and warnings and see also:<br /><a class="external" href="https://code.zmaw.de/projects/miklip-d-integration/wiki/Warnings_and_Errors">https://code.zmaw.de/projects/miklip-d-integration/wiki/Warnings_and_Errors</a></p>
<p>Sincerely,<br />your INTEGRATION Team</p> Web Evaluation System ... next Stepshttps://code.mpimet.mpg.de/news/2452014-05-14T10:09:26ZChristopher Kadowchristopher.kadow@met.fu-berlin.de
<p>Dear Users,</p>
<p>since the first announcement of the web-page<br />we developed many new features to support you and<br />your work-flow.</p>
<p>First of all, our powerful solr_search tool to find data<br />has got an online representation. It's named "data-browser" <br />in the main menu and you can locate more than 4 million<br />NetCDF files through a comfortable interface.</p>
<p>Are you annoyed of some results in your history you would like<br />to get rid of? Now, the web-page offers a delete button to hide<br />these results. To guarantee the scientific reproducibility you<br />can recover these results whenever you like.</p>
<p>Now, the system also supports to share your results with others.<br />Via the share button on the results page you can easily choose<br />MiKlip members to receive an email containing a link to your<br />results.</p>
<p>Last, but not least, we established a guest account with<br />restricted permissions. With the user "guest" and<br />password "miklip" you will get a guided tour through<br />the web-page showing all important features.</p>
<p>Then we are working on several small parts to improve the system and<br />listen carefully to the users. E.g. the system has now the possibility<br />to add informations about "what you see on it" directly to the plots.<br />An Example you find already in the ETCCDI and Monsoon tool.</p>
<p>If you have suggestions how2improve the system, write us!</p> Web of the Evaluation System - www-miklip.dkrz.dehttps://code.mpimet.mpg.de/news/2332014-01-30T13:07:13ZChristopher Kadowchristopher.kadow@met.fu-berlin.de
<a name="News-from-30012014"></a>
<h1 >News from 30.01.2014<a href="#News-from-30012014" class="wiki-anchor">¶</a></h1>
<p>We are happy to announce that the evaluation system is available on the world wide web, now. <br />Under <a href="https://www-miklip.dkrz.de" class="external">www-miklip.dkrz.de</a> [sic!] you find the very first (beta)<br />version of the web based interface to the evaluation system.</p>
<p>The web page offers you all the opportunities of the evaluation system. It<br />provides an interface to the <strong>tools</strong> and allows you to start the tools in<br />a scheduler and watch their progress.</p>
<p>Old runs of the evaluation system can be viewed in the <strong>history</strong> and you can<br />comfortably browse a preview of the results. It is worth to mention that<br />you can access all your runs started with the evaluation system. The system<br />makes no difference whether the runs have been started with the web interface<br />or by using the shell.</p>
<p>There is a first <strong>documentation</strong> of the MurCSS tool and direct access to this <strong>wiki</strong>.</p>
<p>To access the full functionality of www-miklip.dkrz.de you have to login<br />with your DKRZ credentials to the web page.</p> NEWS from INTEGRATION - using&developinghttps://code.mpimet.mpg.de/news/2072013-07-19T14:05:04ZChristopher Kadowchristopher.kadow@met.fu-berlin.de
<a name="NEWS-from-19072013"></a>
<h1 >NEWS from 19.07.2013<a href="#NEWS-from-19072013" class="wiki-anchor">¶</a></h1>
<p>Many projects inside of MiKlip joined the <strong>Evaluation System</strong> and the <strong>MiKlip Server</strong> by using it, brought in analyses (<a href="http://fona-miklip.de/en/316.php" class="external">MesoTel</a>, <a href="http://fona-miklip.de/en/358.php" class="external">EnsDiVal</a>) and data (<a href="http://fona-miklip.de/en/348.php" class="external">VADY</a>, <a href="http://fona-miklip.de/en/316.php" class="external">MesoTel</a>).</p>
<p>The Basic User Guide (BUG) by INTEGRATION with an overview of what has been developed or plugged in and how2use and load the evaluation system you can find here:</p>
<p><a class="external" href="https://code.zmaw.de/projects/miklip-d-integration/wiki/">https://code.zmaw.de/projects/miklip-d-integration/wiki/</a></p>
<p>Always in mind, working on two aspects: <strong><em>using&developing</em></strong> , today we want to pick up two areas of your interest.</p>
<a name="SOLR_SEARCH"></a>
<h1 >SOLR_SEARCH<a href="#SOLR_SEARCH" class="wiki-anchor">¶</a></h1>
<p><strong><ins><a class="wiki-page" href="https://code.mpimet.mpg.de/projects/miklip-d-integration/wiki/SOLR_SEARCH">SOLR_SEARCH</a></ins></strong></p>
<p>The MiKlip Server and the mounted DKRZ ESGF node combine a huge database with direct (!No need to download all data from the ESGF or from slow band archives!) access to over 50 model systems and 2 million files. That is why standardization is so important, just to find everything and stay comparable with the international community. INTEGRATION combined (linked) the different areas to one</p>
<blockquote>
<p><strong><em>/miklip/integration/data4miklip</em></strong></p>
</blockquote>
<p>and developed the search tool "find_files", both already presented at the MiKlip Status-Seminar 2013. But any normal search is too slow, to run through that amount of data.</p>
<p>Therefore INTEGRATION developed the new search system "solr_search" for the ongoing evaluation system and scientists working at the MiKlip server. It has the same syntax as "find_files".</p>
<p>"solr_search" is a very fast searching solution with a user friendly 'tab' usability.</p>
<a name="Looking-for-the-BASELINE1-runs"></a>
<h3 >Looking for the BASELINE1 runs?<a href="#Looking-for-the-BASELINE1-runs" class="wiki-anchor">¶</a></h3>
<pre>
[b324031@miklip04 ~]$ module load evaluation_system
[b324031@miklip04 ~]$ solr_search ->PRESS TABULATOR
cmor_table= ensemble= institute= project= time_frequency=
data_type= experiment= model= realm= variable=
[b324031@miklip04 ~]$ solr_search data_type= ->PRESS TABULATOR
baseline0 baseline1 cmip5 observations reanalysis
[b324031@miklip04 ~]$ solr_search data_type=baseline1 ->PRESS TABULATOR
cmor_table= experiment= realm= variable=
ensemble= model= time_frequency=
[b324031@miklip04 ~]$ solr_search data_type=baseline1 model=mpi ->PRESS TABULATOR
mpi-esm-lr mpi-esm-mr
[b324031@miklip04 ~]$ solr_search data_type=baseline1 model=mpi-esm-lr ->PRESS TABULATOR
cmor_table= ensemble= realm= time_frequency= variable=
[b324031@miklip04 ~]$ solr_search data_type=baseline1 model=mpi-esm-lr variable= ->PRESS TABULATOR
Display all 100 possibilities? (y or n)
areacella clivi dissoc fddtdisi hfls intpn2 no3
areacello clt dms ffire hfsithermds intpp npp
baresoilfrac clw dpco2 fgco2 hfss lai o2
basin clwvi dpo2 fgdms hfx landcoverfrac o2min
bfe co2mass epc100 fgo2 hfxba masso omldamax
bsi co3 epcalc100 fluc hfxdiff mfo omlmax
burntarea co3satcalc epfe100 frc hfy mlotst orog
c3pftfrac cropfrac epsi100 frfe hfyba mlotstsq pasturefrac
c4pftfrac csoil evspsbl frn hfydiff mrro pbfe
calc cveg fco2nat fsfe hur mrros pbo...
cct dcalc fddtalk fveglitter hus mrso
chl deptho fddtdic gpp intdic mrsofc
cl detoc fddtdife grassfrac intpbfe msftbarot
cli dfe fddtdin graz intpbsi msftmyz
clitter dissic fddtdip hfds intpcalcite nbp
etc etc etc
</pre>
<a name="Looking-for-models-to-compare-to-your-results-in-temperature"></a>
<h3 >Looking for models to compare to your results in temperature?<a href="#Looking-for-models-to-compare-to-your-results-in-temperature" class="wiki-anchor">¶</a></h3>
<pre>
solr_search variable=ta model= ->PRESS TABULATOR
access1-0 cesm1-bgc ec-earth giss-e2-h-cc ipsl-cm5a-lr mpi-esm-p
access1-3 cesm1-cam5 fio-esm giss-e2-r ipsl-cm5a-mr mri-agcm3-2h
bcc-csm1-1 cesm1-fastchem geos-5 giss-e2-r-cc ipsl-cm5b-lr mri-cgcm3
bnu-esm cesm1-waccm gfdl-cm3 hadcm3 miroc4h noresm1-m
canam4 cmcc-cesm gfdl-esm2g hadgem2-a miroc5 noresm1-me
cancm4 cmcc-cm gfdl-esm2m hadgem2-ao miroc-esm
canesm2 cmcc-cms gfdl-hiram-c180 hadgem2-cc miroc-esm-chem
ccsm4 cnrm-cm5 gfdl-hiram-c360 hadgem2-es mpi-esm-lr
cdas csiro-mk3-6-0 giss-e2-h inmcm4 mpi-esm-mr
etc etc etc
</pre>
<a name="Do-we-have-also-observations-for-temperature"></a>
<h3 >Do we have also observations for temperature?<a href="#Do-we-have-also-observations-for-temperature" class="wiki-anchor">¶</a></h3>
<pre>
[b324031@miklip04 ~]$ solr_search data_type=observations variable=ta ->PRESS ENTER
/miklip/integration/data4miklip/observations/atmos/ta/mon/grid/NASA-JPL/MLS/v20111025/ta_MLS_L3_v03-3x_200408-201012.nc
/miklip/integration/data4miklip/observations/atmos/ta/mon/grid/NASA-JPL/AIRS/v20110608/ta_AIRS_L3_RetStd-v5_200209-201105.nc
</pre>
<p>SOLR_SEARCH can be used for developing analyses getting direct access to datasets using their meta data and it makes the ensemble handling in CMOR/CMIP5 datasets much easier! It will be also the basement of the hybrid "shell/web/operational" solution of the evaluation system, to assure a fast and effective use of the evaluation system and the MiKlip server. The system is in the <em>beta phase</em> and ready to be filled up with the regional data as well as the next general MiKlip runs.</p>
<p>More and detailed informations:<br /><a class="external" href="https://code.zmaw.de/projects/miklip-d-integration/wiki/Solr_search">https://code.zmaw.de/projects/miklip-d-integration/wiki/Solr_search</a></p>
<a name="PROJECTDATA"></a>
<h1 >PROJECTDATA<a href="#PROJECTDATA" class="wiki-anchor">¶</a></h1>
<p>Our project combined different areas of data in</p>
<blockquote>
<p><strong><em>/miklip/integration/data4miklip</em></strong></p>
</blockquote>
<p>This is about data not belonging to the general runs of MiKlip. Where could you put them to compare them to general MiKlip data?</p>
<p>We set up a projectdata directory:</p>
<blockquote>
<p><strong><em>/miklip/integration/data4miklip/projectdata</em></strong></p>
</blockquote>
<p>If you have your CMORized data e.g. in your scratch, send us an email, then we link it to the system.</p>
<a name="What-does-CMORized-structure-mean"></a>
<h3 >What does CMORized structure mean?<a href="#What-does-CMORized-structure-mean" class="wiki-anchor">¶</a></h3>
<p>Let's have a look at user data from MESOTEL Module C - b324034</p>
<blockquote>
<p><strong><em>/miklip/integration/data4miklip/projectdata/b324034</em></strong></p>
</blockquote>
<p>This is a link b324034 -> /scratch/b324034/archive/CMIP5/ You see the user decided to keep the standard CMIP5 CMOR settings, to test his data, therefore just use the switch in the MPI-ESM for postproduction.</p>
<p>When we go deeper into the structure we find e.g.</p>
<blockquote>
<p><strong><em>/miklip/integration/data4miklip/projectdata/b324034/output/MPI-M/MPI-ESM-LR/decadal2000/mon/atmos/tas/r1i1p2/tas_Amon_MPI-ESM-LR_decadal2000_r1i1p2_200101-201012.nc</em></strong></p>
</blockquote>
<p>You see we need a specific directory structure to put in. Of course you already had a look into CMOR <img src="/plugin_assets/redmine_wiki_extensions/images/smile.png" alt=":)"> If not, here is a short description what you need in your data structure for linking in and the example you see above.</p>
<p>EXAMPLE:</p>
<blockquote>
<p><strong><em>b324034/output/MPI-M/MPI-ESM-LR/decadal2000/mon/atmos/tas/r1i1p2/tas_Amon_MPI-ESM-LR_decadal2000_r1i1p2_200101-201012.nc</em></strong></p>
</blockquote>
<p>DIRECTORY:</p>
<blockquote>
<p><strong><em>project/product/institute/model/experiment/time_frequency/realm/variable/ensemble/</em></strong></p>
</blockquote>
<p>FILE:</p>
<blockquote>
<p><strong>"variable"_"cmor_table"_"model"_"experiment"_"ensemble"_"start_time"-"end_time".nc</strong></p>
</blockquote>
<a name="And-now-the-different-stories-coming-together-as-one"></a>
<h1 >And now the different stories coming together as one!<a href="#And-now-the-different-stories-coming-together-as-one" class="wiki-anchor">¶</a></h1>
<p>If you consider solr_search now and look for your data, just do:</p>
<pre>
solr_search project=b324034 experiment=decadal2000 variable= -> PRESS TAB
Display all 100 possibilities? (y or n) -> n
solr_search project=b324034 experiment=decadal2000 variable=tas time_frequency=mon -> PRESS ENTER
/miklip/integration/data4miklip/projectdata/b324034/output/MPI-M/MPI-ESM-LR/decadal2000/mon/atmos/tas/r1i1p2/tas_Amon_MPI-ESM-LR_decadal2000_r1i1p2_200101-201012.nc
</pre>
<p>And that means you can ask solr_search for data to compare to, here baseline1-LR!</p>
<pre>
solr_search project=b324034 project=baseline1 experiment=dec*2000 variable=tas time_frequency=mon model=mpi-esm-lr
/miklip/integration/data4miklip/model/baseline1/output/MPI-M/MPI-ESM-LR/decs4e2000/mon/atmos/tas/r9i1p1/tas_Amon_MPI-ESM-LR_decs4e2000_r9i1p1_200101-201012.nc
/miklip/integration/data4miklip/model/baseline1/output/MPI-M/MPI-ESM-LR/decs4e2000/mon/atmos/tas/r8i1p1/tas_Amon_MPI-ESM-LR_decs4e2000_r8i1p1_200101-201012.nc
/miklip/integration/data4miklip/model/baseline1/output/MPI-M/MPI-ESM-LR/decs4e2000/mon/atmos/tas/r7i1p1/tas_Amon_MPI-ESM-LR_decs4e2000_r7i1p1_200101-201012.nc
/miklip/integration/data4miklip/model/baseline1/output/MPI-M/MPI-ESM-LR/decs4e2000/mon/atmos/tas/r6i1p1/tas_Amon_MPI-ESM-LR_decs4e2000_r6i1p1_200101-201012.nc
/miklip/integration/data4miklip/model/baseline1/output/MPI-M/MPI-ESM-LR/decs4e2000/mon/atmos/tas/r5i1p1/tas_Amon_MPI-ESM-LR_decs4e2000_r5i1p1_200101-201012.nc
/miklip/integration/data4miklip/model/baseline1/output/MPI-M/MPI-ESM-LR/decs4e2000/mon/atmos/tas/r4i1p1/tas_Amon_MPI-ESM-LR_decs4e2000_r4i1p1_200101-201012.nc
/miklip/integration/data4miklip/model/baseline1/output/MPI-M/MPI-ESM-LR/decs4e2000/mon/atmos/tas/r3i1p1/tas_Amon_MPI-ESM-LR_decs4e2000_r3i1p1_200101-201012.nc
/miklip/integration/data4miklip/model/baseline1/output/MPI-M/MPI-ESM-LR/decs4e2000/mon/atmos/tas/r2i1p1/tas_Amon_MPI-ESM-LR_decs4e2000_r2i1p1_200101-201012.nc
/miklip/integration/data4miklip/model/baseline1/output/MPI-M/MPI-ESM-LR/decs4e2000/mon/atmos/tas/r1i1p1/tas_Amon_MPI-ESM-LR_decs4e2000_r1i1p1_200101-201012.nc
/miklip/integration/data4miklip/model/baseline1/output/MPI-M/MPI-ESM-LR/decs4e2000/mon/atmos/tas/r10i1p1/tas_Amon_MPI-ESM-LR_decs4e2000_r10i1p1_200101-201012.nc
/miklip/integration/data4miklip/projectdata/b324034/output/MPI-M/MPI-ESM-LR/decadal2000/mon/atmos/tas/r1i1p2/tas_Amon_MPI-ESM-LR_decadal2000_r1i1p2_200101-201012.nc
</pre>
<p>If you want to put in other data, like reanalysis to test something you can of course use this too. E.g.:</p>
<p><strong><em>b324034/reanalysis/ECMWF/IFS/ERAINT/mon/atmos/tas/r1i1p1/tas_Amon_reanalysis_ERAINT_r1i1p1_197901-201212.nc</em></strong></p>
<a name="For-help-getting-your-specific-data-CMORized-please-have-a-look-into"></a>
<h2 >For help getting your specific data CMORized, please have a look into:<a href="#For-help-getting-your-specific-data-CMORized-please-have-a-look-into" class="wiki-anchor">¶</a></h2>
<p><strong><em>/miklip/integration/data_management/standardization_examples</em></strong></p>
<p>There you find 1 example for decadal, historical, reanalysis and observation standardization.</p>
<p>This is the idea of bringing data together instead of changing the tools for different datasets. All ongoing projects developing analyses, will use this structure. If we can handle that in MiKlip, we will have a fast, effective, growing, and open evaluation system.</p>
<p>Please contact us, if you have any question or advice.</p>
<p>Christopher Kadow - <a class="email" href="mailto:christopher.kadow@met.fu-berlin.de">christopher.kadow@met.fu-berlin.de</a><br />and<br />Sebastian Illing - <a class="email" href="mailto:sebastian.illing@met.fu-berlin.de">sebastian.illing@met.fu-berlin.de</a></p>