CDO

hi Robert!

I tested the installation and some stuff from the tutorial: Everything was nice and easy except the plot window didn't pop up.

One hint for the usage of temporary files: Users might have long lasting interactive sessions with lots of files (esp. in jupyterhub). Esp. if they do this on a shared resource the /tmp folder can get completely filled and knock out the host machine. Unfortunately I don't have a perfect solution for this while keeping the nice feature of automatically generated tempfiles under the hood.

cheers
ralf

Thanks Ralf

Right now the plotting feature is still somewhat experimental, and has only really been tested to work with Jupyter notebooks. Before putting it on PyPi I hope to get it working when Python is run from the terminal.

I agree that temp file handling is a bit of a challenge. At the minute, when the package is loaded it checks for other tempfiles on the system generated in other sessions, and suggests deleting them. That should probably be built into every method, not simply something that occurs at startup. There are some other thing going behind the scenes. For example, the package automatically switches from /tmp to /var/tmp for tempfiles when there is less than 0.5 GB free in /tmp. But I think what it really needs is a way to look through the tmp files generated and then suggest changes to make, e.g. zipping files or just listing large files, when /tmp is getting clogged up.

Robert

you could also offer the user to use a non-standard location for tempfiles. That's what I did in the python-bindings of CDO. But scanning /tmp for the free space is also a good idea indeed

thx fir the hint
ralf

Doing that has been on my do-list for a while. Though I've procastinated on how to implement it. I'm not sure if the user should be able to define a folder where all temp files go, or if it's one where they go once the /tmp folder is clogged up. The latter probably provides better performance.

Robert

After some hacking, the GitHub version now has plotting that should work on any system.

Robert

I get this:

import nctoolkit as nc
sst = nc.open_data('topo.nc')
sst.plot()
:Image   [lon,lat]   (topo)

That is odd. I have just tried this and it is fine on my machine. Screenshot is attached.

What might be happening is that I have a line of code that detects whether it is being called from a Jupyter notebook. This is working properly on my machine, but maybe it isn't working on all Python installations. Based on your output above, it looks like that is happening.

If you are bored, you can try running this:

import sys

def in_notebook():
"""
Returns ``True`` if the module is running in IPython kernel,
``False`` if in IPython shell or other Python shell.
"""
return 'ipykernel' in sys.modules
in_notebook()

Looks like I forgot to push the updated setup.py with new dependencies. Though I don't think that is causing your problems.

hi Robert!
now it works: it opens a html version with a mouse-over for lon/lat location and the corresponding value.

cheers + thx for your help
ralf

Hi Ralf

Good that it works. Now I just need to figure out a way to create automated tests for whether the plotting works, which might be impossible...

Robert

you at least can plot into a file and compare with a reverence image. possible a simple one like a line plot or so

Something like that would work. Right now the method saves an html file each time. So in theory I just need to store the html file for each test case, and then just check the html file doesn't change each time I run the tests. It might not work in practise if the html files are different using different versions of the dependencies. I'll see.

Robert

yes, you can even compare the md5sum of a compresses html to shrink its size

Thanks for the tip.

This has now been released on PyPI: https://pypi.org/project/nctoolkit/, so can be installed using

pip install nctoolkit

hi Robert!

checked the new installation with data loading and plotting: works out-of-box!

thx for effort
ralf

Thanks Ralf

My next step is getting it on Conda. Though I still don't know if it's a good idea to make CDO a dependency. Doing that might annoy some users, who just want to use the system version of CDO, not Conda....

Robert

I am not sure: if people use conda, they possibly will get cdo from conda, too.

you could add a check, if the cdo binary is there and print a warning otherwise.

BTW: there is a python-binding package for cdo. In case you want to use it instead of your cdo_command module:

• it is called 'cdo' in pipy
• and 'python-cdo' in conda-forge
• restrict the dependency to version 1.5.3

cheers
ralf

Thanks Ralf

Calling python-cdo in cdo_command sounds like a good idea. When do you expect version 2 to be available? I should maybe wait until that comes out, as it sounds like there are big changes happening.

Robert

2.0 will take a while. it will add a new interface, which is already in the master branch as a 2nd option. but I want to implement a ruby-version,too. so it will take a while.

I will have a look at version 2. I should probably avoid modifying existing features in the next release, so I'm likely in no rush to add it in. But right now cdo_command is definitely something that could be better, and using the cdo bindings is likely to improve it.