NCL simple plot¶

• Table of contents
• NCL simple plot
  • Target
  • Procedure
  • Requirements
  • Notes
  • Tutorial
    • Setting up the environment
    • NCL
    • Wrapping up the script
    • Making a plug-in
    • Runing the plugin
  • NCLPlot Usage

Welcome to the first introduction to the evaluation system (and NCL). We'll try to keep things simple enough while showing you around.
We will be glad to get your feed-back about this tutorial or the system (or anything else) at estanislao.gonzalez[at]met.fu-berlin.de.

Target¶

To create a plugin for the evaluation system that just creates a simple plot of a 2D variable.

Procedure¶

We'll go bottom-up starting from the result we want to achieve and building up the plugin out of it.

Requirements¶

This tutorial requires you to have a bash shell, so if you are using any other (or don't know what you are using) Just start bash by issuing this at the prompt:

$ bash

We'll try to be thorough and assume the user has little knowledge about programming. Though some basic shell knowledge is required.
If not, just copy and paste the commands in here, you don't need to understand everything for getting through the tutorial, though it will certainly help you to locate errors and typos you might have.

Notes¶

The $ symbol at the beginning of a line is used to mark a command that is given to the bash shell. You don't have to input that character when issuing the same command in your shell.
When depicting an interactive shell session it denotes the commands issued to the shell and those lines which does not start with the $ character denote the output from the shell.
We might skip the character when there's no expected output from the shell (or if it's not useful), so that you may Copy&Paste the whole block.

We sometimes number the lines when dissecting a program for didactic reasons. Sadly, this breaks the ability to simply Copy&Paste the code. We'll try only to do that on complete programs that you might directly download from this page.

Tutorial¶

Setting up the environment¶

We'll need a file for testing, grab any suitable for the task. We can use the evaluation system to find one fast:

$ module load evaluation_system
Evaluation System successfully loaded.

New available commands:
*  analyze
*  esgf
*  find_files

To get more help use: 
<command> --help
$ find_files --baseline 1  variable=tas | head -n1
/miklip/integration/data4miklip/model/baseline1/output/MPI-M/MPI-ESM-LR/asORAoERAa/day/atmos/tas/r1i1p1/tas_day_MPI-ESM-LR_asORAoERAa_r1i1p1_19600101-19691231.nc

Let's store that in a variable for later use:

$ file=$(find_files --baseline 1  variable=tas | head -n1)
$ echo $file
/miklip/integration/data4miklip/model/baseline1/output/MPI-M/MPI-ESM-LR/asORAoERAa/day/atmos/tas/r1i1p1/tas_day_MPI-ESM-LR_asORAoERAa_r1i1p1_19600101-19691231.nc
$ export file

The last command (export) tells bash to pass that variable to other programs started from this shell (this is one of many methods of passing info to NCL). Ok now let's see what the file has to offer:

$ ncdump -h $file | grep ') ;'
    double time(time) ;
    double time_bnds(time, bnds) ;
    double lat(lat) ;
    double lat_bnds(lat, bnds) ;
    double lon(lon) ;
    double lon_bnds(lon, bnds) ;
    float tas(time, lat, lon) ;

As expected we have one variable, tas, but check the order in which the values are stored (time, lat, lon) as we will need them later.
Now let's start with our bottom-up procedure. We'll start ncl and try to get a plot by using the NCAR tutorial:

$ ncl
-bash: ncl: command not found

That means we need to load the ncl module, this will also be required when creating the plug-in.

$ module load ncl
$ ncl
 Copyright (C) 1995-2012 - All Rights Reserved
 University Corporation for Atmospheric Research
 NCAR Command Language Version 6.1.0-beta
 The use of this software is governed by a License Agreement.
 See http://www.ncl.ucar.edu/ for more details.
ncl 0> 

NCL¶

So we'll use NCL to produce a simple plot of a file. Let's use the ncl tutorial here to generate a plot: http://www.ncl.ucar.edu/Document/Manuals/Getting_Started/Examples/gsun02n.shtml

Now here's the complete session to generate a simple plot out of the information in the tutorial (you may Copy&Paste it into the ncl shell):

load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl" 
cdf_file = addfile("$NCARG_ROOT/lib/ncarg/data/cdf/contour.cdf","r")
temp = cdf_file->T(0,0,:,:)               ; temperature
lat  = cdf_file->lat                      ; latitude
lon  = cdf_file->lon                      ; longitude
xwks = gsn_open_wks("x11","gsun02n")      ; Open an X11 workstation
plot = gsn_contour(xwks,temp,False)       ; Draw a contour plot.

(Remember to hit enter on the last line too! If not you'll see just an empty window...)

That should have displayed a simple contour plot over some sample data. The ncl shell blocks until you click on the plot (don't ask me why...)
That shows everything is setup as expected. Now let's display the file we selected:

load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl" 
file_path=getenv("file")               ; get the value stored in the exported variable "file" 
input_file = addfile(file_path,"r")    ; now use that value to load the file
temp = input_file->tas(0,:,:)             ; temperature (CMOR name tas - surface temparature)
lat  = input_file->lat                    ; latitude
lon  = input_file->lon                    ; longitude
xwks = gsn_open_wks("x11","gsun02n")      ; Open the window with the tile "gsun02n" (same as before)
plot = gsn_contour(xwks,temp,False)       ; Draw the contour plot again

So, the only changes are that in the second line we are retrieving the file name from the environment (remember the "export file" command?), and in the 4th we retrieve a variable named tas instead of T that has only 3 dimensions (2D + time) (remember the the "ncdump -h" command?). So "tas(0,:,:)" means selecting all lat/lon elements of the first time-step. That's what we have plotted.

From the information in the UCAR page we could build a better plot before finishing it up:

load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl" 
file_path=getenv("file")
input_file = addfile(file_path,"r")
var  = input_file->tas(0,:,:)               ; variable
lat  = input_file->lat                      
lon  = input_file->lon                      
resources                 = True            ; Indicate you want to set some
                                            ; resources.
resources@cnFillOn        = True            ; Turn on contour line fill.
resources@cnMonoLineColor = False           ; Turn off the drawing of
                                            ; contours lines in one color.
resources@tiMainString     = var@standard_name + " (" + var@units + ")" ; a title
resources@tiXAxisString    = lon@long_name
resources@tiYAxisString    = lat@long_name
resources@sfXArray         = lon
resources@sfYArray         = lat

xwks = gsn_open_wks("x11","gsun02n") 
plot = gsn_contour_map(xwks,var,resources)  ; Draw a map

So, we are almost ready, we need to store the result into a file instead of displaying it, pass some parameters (the output directory and file) and extract others, e.g. the variablename is always the first string in the name of the file before the "_" character (Because the files we have follow the DRS standard) so we can use this to our advantage.

Another approach for passing values to the NCL program is by using the command line. We'll assume we have a variable called plot_name with the path to the file we want the resulting plot to be stored is already given; the same applies for file_path (just to show a different procedure for passing values around). And last (and definitely least), we'll pack everything in a begin block so we have a proper script.

So this is how it looks like:

load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl" 
begin
    input_file = addfile(file_path,"r")
    file_name = systemfunc("basename " + file_path)
    tmp = str_split(file_path, "_")
    var_name = tmp(0)
    delete(tmp)

    var  = input_file->$var_name$(0,:,:)        ; first timestep of a 2D variable
    lat  = input_file->lat                      
    lon  = input_file->lon                      
    resources                 = True            ; Indicate you want to set some
                                                ; resources.
    resources@cnFillOn        = True            ; Turn on contour line fill.
    resources@cnMonoLineColor = False           ; Turn off the drawing of
                                                ; contours lines in one color.
    resources@tiMainString     = var@standard_name + " (" + var@units + ")" ; a title
    resources@tiXAxisString    = lon@long_name
    resources@tiYAxisString    = lat@long_name
    resources@sfXArray         = lon
    resources@sfYArray         = lat

    xwks = gsn_open_wks("eps",plot_name)        ; create an eps file
    plot = gsn_contour_map(xwks,var,resources)  ; store the map
end

Store that in a file called plot.ncl.

So how can we test it? Simply by calling ncl in the directory where plot.ncl is located in the following way:

ncl plot.ncl file_path=\"$file\" plot_name=\"output\" 

You should have now a file called output.eps in your current directory which you can view with evince or gs.

You might have notice the strange looking \" characters. Ncl requires string values to be always quoted. The shell does use the quotes for something different, so we have to escape them, which basically means telling the shell to pass the " character as is instead of interpreting it in any particular way.

If you want more info on NCL functions: http://www.ncl.ucar.edu/Document/Functions/

Wrapping up the script¶

One last check... the environment in which we started the previous ncl command had already been setup (with module load ncl). Next time we want to use it might not be the case, or a different ncl version might be loaded, etc. In summary, the program might stop working because the environment is somehow different.

We should assure the environment doesn't change. At least as much as we can.
What we need to do is to clean the environment, check the program doesn't work and set it up again so we are sure we have set it up properly.

Furthermore, we can do that without affecting our current shell by generating a sub-shel in bash by using the () characters.
By the way, module purge will remove all modules from the environment. So the test might look like this:

$ (module purge && ncl plot.ncl file_path=\"$file\" plot_name=\"output\")
bash: ncl: Kommando nicht gefunden.

(The && tells bash to issue the next command only if the previous one finished successfully.)

As expected it fails because of the missing ncl command. Now we can test if just adding the ncl module is enough:

$ (module purge && module load ncl && ncl plot.ncl file_path=\"$file\" plot_name=\"output\")
 Copyright (C) 1995-2012 - All Rights Reserved
 University Corporation for Atmospheric Research
 NCAR Command Language Version 6.1.0
 The use of this software is governed by a License Agreement.
 See http://www.ncl.ucar.edu/ for more details.

Now we are sure it works as intended.

Making a plug-in¶

Up to now we've seen an introduction to making a script that's usable. This has nothing to do with the system, but plain informatics.
Since making a plug-in is so easy, we had to write something to fill up the tutorial :-)

So now we just have to wrap it up in our plug-in. From the information in Developing a plugin, Reference and some python knowledge we (at least I) could infer a very simple plugin:

import os
from evaluation_system.api import plugin, parameters

class NCLPlot(plugin.PluginAbstract):
    tool_developer = {'name':'Christopher Kadow', 'email':'christopher.kadow@met.fu-berlin.de'
    __short_description__ = "Creates a simple 2D countour map." 
    __version__ = (0,0,1)
    #plugi.metadic is a special dictionary used for defining the plugin parameters
    __parameters__ =  parameters.ParameterDictionary(
                            parameters.File(name='file_path', mandatory=True, help='The NetCDF file to be plotted'),
                            parameters.Directory(name='plot_name', default='$USER_PLOTS_DIR/output', help='The absolute paht to the resulting plot'))

    def runTool(self, config_dict=None):
        file_path = config_dict['file_path']
        plot_name = config_dict['plot_name']

        result= self.call('module load ncl && ncl %s/plot.ncl file_path=\\"%s\\" plot_name=\\"%s\\"' % (self.getClassBaseDir(), file_path, plot_name))
        print result[0]
        #ncl adds the ".eps" suffix to our output file!
        plot_name += ".eps" 
        if os.path.isfile(plot_name):
            return self.prepareOutput(plot_name)

That's certainly looks more daunting than it is. (If you haven't done the 5 Minutes Hands on tutorial do it now)

The difference with the dummy plug-in is that:

• In 1 we import the os package that will be used in 20 to check if the file was properly created.
• In 8 we provide a ParametersDictionary with some information about the parameters we expect.
  (Check the Reference section for more information about parameters and how it's used in the system)
  (You may refer directly to the source code: source:/src/evaluation_system/api/parameters.py)
  Here we basically define two input variables:
  • file_path: a required file (just a string) with no default value.
  • plot_name: An optional string for saving the resulting plot. By default it will be stored in a system managed directory with the name output.eps
• In 13 and 14 we read the variables passed to the application.
• In 16 we call the script (which should is assumed to be located in the same location as the script).
  (Check the code for more information source:/src/evaluation_system/api/plugin.py)
  • self.call is a method from the inherited class that issues a shell command and return it's output as a tuple.
  • self.getClassBaseDir() returns the location of the script.
  • %s is used to format strings in Python, so "%s - %s" % ( 1, 2) == "1 - 2"
  • \\" that's a double scape sequence... \\ tells python not to interpret the \ as enything special. It calls then bash with \" and we already saw why.
• in 17 we display the output from the call. By default stderr (All rerrors) are redirected to stdout so everything appears in the first element of the returned tuple (hence the result[0]
• in 19 we extend the named of the output file as ncl automatically adds that suffix.
• 20 checks if the file exists and in such a case it will be returned using a special construct which checks the status of it and store some info for later use.

Runing the plugin¶

We are done, now we have to test it. We will create a directory, store all files there (plot.ncl and nclplot.py) and tell the evaluation system to use the plugin.

$ mkdir plugin
$ cp nclplot.py plot.ncl plugin
$ cd plugin
$ ls
nclplot.py  nclplot.pyc plot.ncl
$ pwd
/home/user/plugin
$ export EVALUATION_SYSTEM_PLUGINS=/tmp/plguin,nclplot

Recalling from [5 Minutes Hands on]] the EVALUATION_SYSTEM_PLUGINS show the evaluation system where to find the code. It's a colon separated list of comma separated path, packages pairs. Basically: EVALUATION_SYSTEM_PLUGINS=/some/path,mypackage:/oth/path,some.more.complex.package

We are done!

NCLPlot Usage¶

Now let's see what the evaluation system already can do with our simple plugin.
Let's start for displaying some help:

$ analyze --list
NCLPlot: Creates a simple 2D countour map.
PCA: Principal Component Analysis
$ analyze --tool nclplot --help
NCLPlot (v0.0.1): Creates a simple 2D countour map.
Options:
file_path (default: None) [mandatory]
          The NetCDF file to be plotted

plot_name (default: $USER_PLOTS_DIR/output)
          The absolute paht to the resulting plot

$ analyze --tool nclplot
ERROR:  Missing required configuration for: file_path

So there you see to what purpose the metadict is being used. Let's use the $file and create a plot.

$ analyze --tool nclplot file_path=$file plot_name=first_plot
 Copyright (C) 1995-2012 - All Rights Reserved
 University Corporation for Atmospheric Research
 NCAR Command Language Version 6.1.0
 The use of this software is governed by a License Agreement.
 See http://www.ncl.ucar.edu/ for more details.

$ evince first_plot.eps

And evince should display the plot.

The configuration here is pretty simple, but you might still want to play around with it:

$ analyze --tool nclplot --save-config --config-file myconfig.cfg --dry-run file_path=$file plot_name=first_plot
INFO:__main__:Configuration file saved in myconfig.cfg
$ cat myconfig.cfg 
[NCLPlot]
#: The absolute paht to the resulting plot
plot_name=first_plot

#: [mandatory] The NetCDF file to be plotted
file_path=tas_Amon_MPI-ESM-LR_decadal1990_r1i1p1_199101-200012.nc

$ rm first_plot.eps
$ analyze --tool nclplot --config-file myconfig.cfg
$ evince first_plot.eps

As we say, it makes no sense here, but if you have multiple parameters you might want to let some predefined to suit your requirements.
See also how to use the Evaluation System User Configuration.

Now the system also offers a history. There's a lot to be said about it, so I'll just leave a sample session here until I write a tutorial about it.

$ analyze --history
21) nclplot [2013-01-11 14:44:15.297102] first_plot.eps {u'plot_name': u'first_plot', u'file_path': u'tas_Am...
22) ...
...
$ analyze --history limit=1 full_text
21) nclplot v0.0.1 [2013-01-11 14:44:15.297102] 
Configuration:
      file_path=tas_Amon_MPI-ESM-LR_decadal1990_r1i1p1_199101-200012.nc
      plot_name=first_plot
Output:
  /scratch/users/estani/ncl_plot/plugin/first_plot.eps (available)

$ rm /scratch/users/estani/ncl_plot/plugin/first_plot.eps
$ analyze --history limit=1 full_text21) nclplot v0.0.1 [2013-01-11 14:44:15.297102] 
Configuration:
      file_path=tas_Amon_MPI-ESM-LR_decadal1990_r1i1p1_199101-200012.nc
      plot_name=first_plot
Output:
  /scratch/users/estani/ncl_plot/plugin/first_plot.eps (deleted)
$ analyze --history entry_ids=21 store_file=myconfig.cfg
Configuration stored in myconfig.cfg
$ cat myconfig.cfg [NCLPlot]
#: The absolute paht to the resulting plot
plot_name=first_plot

#: [mandatory] The NetCDF file to be plotted
file_path=tas_Amon_MPI-ESM-LR_decadal1990_r1i1p1_199101-200012.nc

$ analyze --tool NCLPlot --config-file myconfig.cfg
 Copyright (C) 1995-2012 - All Rights Reserved
 University Corporation for Atmospheric Research
 NCAR Command Language Version 6.1.0
 The use of this software is governed by a License Agreement.
 See http://www.ncl.ucar.edu/ for more details.

$ analyze --history limit=2 full_text
22) nclplot v0.0.1 [2013-01-11 14:51:40.910996] 
Configuration:
      file_path=tas_Amon_MPI-ESM-LR_decadal1990_r1i1p1_199101-200012.nc
      plot_name=first_plot
Output:
  /scratch/users/estani/ncl_plot/plugin/first_plot.eps (available)
21) nclplot v0.0.1 [2013-01-11 14:44:15.297102] 
Configuration:
      file_path=tas_Amon_MPI-ESM-LR_decadal1990_r1i1p1_199101-200012.nc
      plot_name=first_plot
Output:
  /scratch/users/estani/ncl_plot/plugin/first_plot.eps (modified)

$ analyze --history --help
...

Well, this is it. I'll be adding more tutorials doing some more advanced stuff but I think now you have the knowledge to start creating plugins for the whole community. As mentioned before, feel free to contact us.

Happy Hacking!