Project

General

Profile

Python syntax for cdo commands

Added by Romain LE LAMER over 4 years ago

Hi,
I am trying to include cdo commands in my Python script but I am having a problem ...

    [...]
    if c_tm == 6:
        c_tm006_cnv = join(path_c_tz_cnv, 'GFS1_20200809_12_006_cnv.grb2')
        c_tm006_10uv006 = join(path_b_tm003, 'GFS1_20200809_12_006_10uv006.nc')

        # Rename 10u & 10v
        cdo.setpartabn('10uv006',
                       option='-f nc',
                       input=c_tm006_cnv,
                       output=c_tm006_10uv006)
    [...]

File 10uv006 are in the same directory that the script and contains
&parameter
  name = 10u
  out_name = 10u006
/
&parameter
  name = 10v
  out_name = 10v006
/

The file is created and in the correct directory but the parameters are not renamed.
What I don't understand is that this structure works perfectly in this case (cnvUV is also in the same location as the file above)
    [...]
    gribfile = 'GFS1_20200809_12_006.grb2'
    gribfile_cnv = 'GFS1_20200809_12_006_cnv.grb2'

    cdo.exprf('cnvUV',
              input=join(path_c_tz, gribfile),
              output=join(path_c_tz_cnv, gribfile_cnv))
    print('Convert OK')
    [...]

File cnvUV
10u = ((10u > 0) ? 1 : -1) * sqr(nint(sqrt(abs(10u * 230.4)))) / 230.4;
10v = ((10v > 0) ? 1 : -1) * sqr(nint(sqrt(abs(10v * 230.4)))) / 230.4;

I also have the controls
cdo.merge / cdo.setreftime & cdo.mergetime to translate with "python syntax"
:(

PS: Is there documentation somewhere on the Python syntax for commanced cdo ?
If yes I did not find it.
Thanks


Replies (20)

RE: Python syntax for cdo commands - Added by Romain LE LAMER over 4 years ago

I understood my mistake ...
(thanks cdo.debug = True)
Do not forget this "s" to options= ...

RE: Python syntax for cdo commands - Added by Ralf Mueller over 4 years ago

hi Romain!
yeah, unfortunately there is no warning for unknown keywords. Maybe I should implement this.

On Cdo{rbpy} you can find some examples for python usage.

hth
ralf

RE: Python syntax for cdo commands - Added by Kyle Lesinger over 4 years ago

Romain:

May I ask how you were even able to get cdo working in Python? Are you using a specific IDE or was there a website that helped? I have been trying for days and I have not yet found a good source for even getting it started. I do have it installed as version 1.5.3.

Any help would be appreciated.

Thanks,

KL

RE: Python syntax for cdo commands - Added by Romain LE LAMER over 4 years ago

Hi Kyle,
I'm using PyCharm CE and my "bible" is [[https://code.mpimet.mpg.de/projects/cdo/embedded/index.html]].
I have installed cdo with

pip install cdo netcdf4 xarray

Your file.py must be contains

from cdo import *
# OR
# from cdo import Cdo

cdo = Cdo()
#cdo.debug = True """ can be useful to help for solve errors """ 

# Commands
cdo.operator('parameters',
             options=XXX, """ if need """ 
             input=infile,
             output=outfile)

I don't know much more for now ;)

RE: Python syntax for cdo commands - Added by Kyle Lesinger over 4 years ago

Thank you for your assistance. I use spyder and unfortunately that pip install cdo netcdf4 xarray did not solve my issue. For some reason I get this error:

signal.siginterrupt(signal.SIGINT, False)
AttributeError: module 'signal' has no attribute 'siginterrupt'

Thanks for your help though. I'll figure it out one day!

RE: Python syntax for cdo commands - Added by Ralf Mueller over 4 years ago

@Kyle: Which version of python do you use?

signal.siginterrup() was introduced in python 2.6 and is still there in 3.8

RE: Python syntax for cdo commands - Added by Ralf Mueller over 4 years ago

Romain LE LAMER wrote:

Hi Kyle,
I'm using PyCharm CE and my "bible" is [[https://code.mpimet.mpg.de/projects/cdo/embedded/index.html]].

Amen!! ;-)

RE: Python syntax for cdo commands - Added by Kyle Lesinger over 4 years ago

@Ralf Thehos: I am running Python 3.7.6.

RE: Python syntax for cdo commands - Added by Romain LE LAMER over 4 years ago

I am on macOS and a friend also has the same problem (with my script) as you Kyle, and he is on Windows ...
My question is therefore are you under Windows also ?

Me:
macOS v10.15.6
python v3.8.5
cdo v1.9.8

$ cdo --version
Climate Data Operators version 1.9.8 (https://mpimet.mpg.de/cdo)
System: x86_64-apple-darwin19.4.0
CXX Compiler: clang++ -std=gnu++11 -g -O2  -D_THREAD_SAFE -pthread
CXX version : Apple clang version 11.0.3 (clang-1103.0.32.59)
C Compiler: clang -g -O2  -D_THREAD_SAFE -pthread -D_THREAD_SAFE -D_THREAD_SAFE -pthread
C version : Apple clang version 11.0.3 (clang-1103.0.32.59)
F77 Compiler: gfortran -g -O2
F77 version : GNU Fortran (Homebrew GCC 9.3.0_1) 9.3.0
Features: 8GB 4threads C++11 Fortran DATA PTHREADS HDF5 NC4/HDF5 OPeNDAP SZ SSE4_2
Libraries: HDF5/1.12.0
Filetypes: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5 
     CDI library version : 1.9.8
 cgribex library version : 1.9.4
 ecCodes library version : 2.18.0
  NetCDF library version : 4.7.4 of Jul  2 2020 22:40:24 $
    hdf5 library version : 1.12.0
    exse library version : 1.4.1
    FILE library version : 1.8.3

A friend :
Windows 10
python v3.8.5

cdo --version not possible

RE: Python syntax for cdo commands - Added by Romain LE LAMER over 4 years ago

Oop's it is not good ...
You must have at least these packages

pip list
Package         Version
--------------- ---------
cdo             1.5.3
cftime          1.2.1
netCDF4         1.5.4
numpy           1.19.1
pandas          1.1.0
pip             20.1.1
python-dateutil 2.8.1
pytz            2020.1
six             1.15.0
wheel           0.34.2
xarray          0.16.0

RE: Python syntax for cdo commands - Added by Romain LE LAMER over 4 years ago

I come back to the error encountered by Kyle and a friend (who uses my script)
As I indicated above it is under Windows 10, at Python v3.8.5 and via pip it installed cdo netcdf4 and xarray

C:\BIN\Python\python.exe D:/Temp/main.py
Traceback (most recent call last):
  File "D:/Temp/main.py", line 112, in <module>
    cdo = Cdo()
  File "C:\BIN\Python\lib\site-packages\cdo.py", line 170, in _init_
    signal.siginterrupt(signal.SIGINT, False)
AttributeError: module 'signal' has no attribute 'siginterrupt'
Process finished with exit code 1

My conclusion (after testing) is that it is necessary to have the cdo binary installed locally.
• For UNIX systems, it must be compiled or installed via the system's package manager (me on macOS I used Homebrew (brew install cdo))
• For Windows systems, it is necessary to do this via Cygwin.

I think a cdo developer will confirm this conclusion.

RE: Python syntax for cdo commands - Added by Kyle Lesinger over 4 years ago

@Romain:

Yes I am also using windows. I've tried to get CDO to work with Spyder, Pycharm, and cygwin now but to no avail. I did update all my packages to be exactly the same way that you did Romain, but for some reason I still get the error even in Pycharm

Traceback (most recent call last):
File "C:\Users\kylel\Anaconda3\envs\cdo\lib\site-packages\IPython\core\interactiveshell.py", line 3417, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-3-237a1757623c>", line 1, in <module>
cdo = Cdo()
File "C:\Users\kylel\Anaconda3\envs\cdo\lib\site-packages\cdo.py", line 170, in init
signal.siginterrupt(signal.SIGINT, False)
AttributeError: module 'signal' has no attribute 'siginterrupt'.

I tried to install Ubuntu through the VM Virtual Machine, but unfortunately computers really dislike me and there are errors in the download :(.

RE: Python syntax for cdo commands - Added by Ralf Mueller over 4 years ago

hi!

There is not CDO binary for the native Windows system. Hence you cannot use python/anaconda/pip to call CDO through cdo.py, because there is no CDO on plain Windows.
For Windows there are two option

The windows version of python doesn't seem to have this part of the signal module - sorry for missing the fact, that you work under windows directly.

RE: Python syntax for cdo commands - Added by Romain LE LAMER over 4 years ago

Hi Ralf & Kyle
I did not know Windows Subsystem for Linux (WSL) at all but I find it to be THE solution to adopt.
It is extremely easy to implement (compared to Cygwin)
Thanks for that ;)

RE: Python syntax for cdo commands - Added by Ralf Mueller over 4 years ago

hi Roman!

yes, it's a lot easier IMO. the only disadvantage is, that you won't get the latest release out-of-the-box. But for most applications this will not be needed.
The other question is, whether Microsoft is willing to support this on the long run. Many years ago there was a Unix Subsystem, but totally outdated and a Microsoft employer told me, he would use cygwin instead.

Maybe this has changed now - debian/ubuntu is well maintained, Microsoft updates the ubuntu version.

In case of problems using WSL please let me know. I am curious how a real world setup works in there.

cheers
ralf

RE: Python syntax for cdo commands - Added by Romain LE LAMER over 4 years ago

I think you have to do it in a simple way, that is to say you should not try to make Windows and Linux communicate, you have to run everything from Linux.
If Windows gives the possibility of running a Linux so easily it is a bit of the death of Windows (and it's not me that will bother :) )

RE: Python syntax for cdo commands - Added by Kyle Lesinger over 4 years ago

Thank you all for your input. After many hours, I have finally been able to run CDO through the Ubuntu Desktop app!

Here were my steps for anyone else in the future who tries to install it:

After first opening and creating your username:
sudo apt-get update and sudo apt-get upgrade

Install CDO
sudo apt-get install cdo

Try cdo - v if you get this error:
cdo: error while loading shared libraries: libQt5Core.so.5: cannot open shared object file: No such file or directory

Then:
sudo strip --remove-section=.note.ABI-tag /usr/lib/x86_64-linux-gnu/libQt5Core.so.5

Then you have to move your file you want to edit into the actual Desktop folder. Then cdo operators will work.

RE: Python syntax for cdo commands - Added by Kyle Lesinger over 4 years ago

I maybe shouldn't have posted this here considering the title of the post is Python syntax for cdo commands, but this may be much quicker for others if they have a hard time figuring it out.

RE: Python syntax for cdo commands - Added by Ralf Mueller over 4 years ago

hi Kyle!

no worries - it's good that we have it here! thx for the info

best wishes
ralf

    (1-20/20)