CDO

Hi ?,

the example from the CDO User Guide https://code.mpimet.mpg.de/projects/cdo/embedded/cdo.pdf#subsection.2.8.6 should look like

cdo.ensbrs(5, input='rfile ensfile1 ensfile2 ensfile3 ensfile4 ensfile5', output='obase')

-Karin

Hi,

Thank you.

Best regards.

P.S. I am not a big fan of Google so I do not provide that much public information if possible. However, if you insists I will be happy to think it over.

hi!

which Julia binding do you use? Something like this: https://github.com/Try2Code/cdo-bindings/blob/master/julia/test/runtests.jl ?

or did you write your own?

cheers
ralf

hi ralf,

What is the purpose of your question? This will allow me to better address the answer. I am a private individual and I am not a professional developer, however, in general, I understand that in the example provided by you, there are not Julia but Python bindings. I do not code in Python. I try to use it only for CDO as a necessity as there seems to be no other option. Thus I do not use xarray nor numpy. As a source, I operate on grib2 and grib1 files. For computations I am converting them to NetCDF4. Later I use native Julia NetCDF and DiskArrays packages to manipulate data. It is as for now, as I have not yet finaly decided what package to use for manipulation of multidmensional arrays as there is a choice of them in Julia geo community. In general, I am leaning towards YAXArrays or GeoData. If you are asking about my general experience with CDO in Julia, I have to admit, that overall I like it, however, it was not a smooth ride to make it work. From my point of view, what you wrote in the announcement post about the power of Python integration with Julia might be correct in simple environments with sudo (i.e. sudo apt-get install cdo) or simple environments which use "private" Julia miniconda distro that comes with Julia natively (i.e. >julia Conda.add("python-cdo")). In case of virtual environments or generic system-wide Anaconda installations things can get tricky. I spent more than a week trying to understand the correct setups in my three development environments. Even though I made it work, I am still not sure if scheduled jobs will be supported. Also not sure if this can be used with PackageCompiler to prepare operating system agnostic relocatable app. In addition I encountered some problems trying to make multithreded and distributed operations working correctly with CDO and PyCall (as for now I put it away). So, from my perspective this is not the same as a native Julia package or Julia wrapper. Again, I am not a professional developer but I am sure that this part about PyCall could be confirmed even by some experienced ones. To sum up, maybe you could reconsider your position that the Julia package is not needed. I am sure that the Julia geo community would be thrilled to have such a tool as CDO in its arsenal. Especially, that the efforts of some other leading organizations from your field seems to concentrate on Python (i.e. climetlab) or seems to be unfinished (i.e cfgrib.jl). Should you have any comments please feel free and let me know. And hey, is there any particular research that is complementary to the paper of Mr. Hans Hersbach published in 2000 "Decomposition of the Continuous Ranked Probability Score for Ensemble Prediction Systems" that you may recommend? The problem is that the yacht is sailing from Cape Horn to Sopot with a stopover at Ellenbogen, List, Sylt and the skipper is equipped with 30 days long probabilistic forecast and on its way has to made tough choices as the distribution of perturbations is usually multi-modal.

regards,
CU ;-P

cdo user wrote in RE: Correct Python syntax for ensbrs?:

hi ralf,

What is the purpose of your question?

I want to learn how you us CDO in Julia. I wrote the stuff I posted before and hoped you might use something different or that you can share some experience with it.

This will allow me to better address the answer. I am a private individual and I am not a professional developer, however, in general, I understand that in the example provided by you, there are not Julia but Python bindings. I do not code in Python. I try to use it only for CDO as a necessity as there seems to be no other option. Thus I do not use xarray nor numpy.

The example I gave is pure Julia. I only use the python package under the hood. you don't have to write python in order to use the python-bindings in Julia. just saying

...
In case of virtual environments or generic system-wide Anaconda installations things can get tricky. I spent more than a week trying to understand the correct setups in my three development environments.

IMO even in a system wide conda installation you should always create a private environment.

Even though I made it work, I am still not sure if scheduled jobs will be supported. Also not sure if this can be used with PackageCompiler to prepare operating system agnostic relocatable app.

Good point, the service team/admins should be able to help you I guess. I am not too overly familiar with Julia packaging.

In addition I encountered some problems trying to make multithreded and distributed operations working correctly with CDO and PyCall (as for now I put it away). So, from my perspective this is not the same as a native Julia package or Julia wrapper.

I would like to see an example of multi threaded code using PyCall vs. native Julia.

Again, I am not a professional developer but I am sure that this part about PyCall could be confirmed even by some experienced ones. To sum up, maybe you could reconsider your position that the Julia package is not needed. I am sure that the Julia geo community would be thrilled to have such a tool as CDO in its arsenal. Especially, that the efforts of some other leading organizations from your field seems to concentrate on Python (i.e. climetlab) or seems to be unfinished (i.e cfgrib.jl). Should you have any comments please feel free and let me know.

I decided against a separate Julia package after I searched for Julia features regarding Object Oriented Programming and Meta Programming. Unfortunately both areas do not seem to be covered by Julia very well. But this is the way I want to write the CDO wrapper. Then I found out about the PyCall module and the problem seem to be solved. So If you have some example, why the PyCall module has some serious limitations compared to nativ Julia, please let me know.

And hey, is there any particular research that is complementary to the paper of Mr. Hans Hersbach published in 2000 "Decomposition of the Continuous Ranked Probability Score for Ensemble Prediction Systems" that you may recommend? The problem is that the yacht is sailing from Cape Horn to Sopot with a stopover at Ellenbogen, List, Sylt and the skipper is equipped with 30 days long probabilistic forecast and on its way has to made tough choices as the distribution of perturbations is usually multi-modal.

sry, no idea ;-) -- but this sailing route sounds promising

cheers
ralf

Dear Herr Ralf,

Thanks so much for a detailed and prompt reply.

I wrote the stuff I posted before and hoped you might use something different or that you can share some experience with it.

I am glad I was able to help or at least this is what I think.

The example I gave is pure Julia.

I do not understand. As I understand it, you are using PyCall to call Python and you are asking Python to call Fortran and from there you are asking Fortran to call MPI to make parallel operations.

I only use the python package under the hood.

I am sorry. I do not understand. In the provided example I see numpy and xarray.

IMO even in a system wide conda installation you should always create a private environment.

This is why I wrote that, then, things can get tricky as Julia comes with own Conda, there are differences between Python versions as well as VS Code (preferred Julia IDE) is also sensitive to Python interpreter, especially over ssh. And on top of all that there is PyCall.

Good point, the service team/admins should be able to help you I guess. I am not too overly familiar with Julia packaging.

The environment is top class. Despite the fact that on a daily basis those guys are into design of processors, they know software architectures very well. Due to some reasons, I got particularly good relations with them, however, I do not think that there is a problem with the environment, so I do not see any reasons to be asking for help. As for the packaging apps please see the video by Mr. Kristoffer Carlsson titled "JuliaCon 2020 | Shippable apps and custom sysimages with PackageCompiler.jl" at youtube. |

I would like to see an example of multi threaded code using PyCall vs. native Julia.

As far as I know, there are differences. For details, I refer you to the posts by Mr. Przemyslaw Szufel at stackoverflow and by Stefan Karpinski and Jeff Bezanson at julia discourse as well as to general documentation.

I decided against a separate Julia package after I searched for Julia features regarding Object Oriented Programming and Meta Programming. Unfortunately both areas do not seem to be covered by Julia very well.

I understand. There is multiple dispatch, so I would say: no regrets. Besides as far as I know there are constructors. However, I would prefer not to get into specific details as I personally do not see this field as my area of expertise.

Then I found out about the PyCall module and the problem seem to be solved. So If you have some example, why the PyCall module has some serious limitations compared to nativ Julia, please let me know.

I provided you with some arguments in my earlier writing here. You may also be interested in reading recent posts under the topic: "I just decided to migrate from Python+Fortran to Julia as Julia was faster in my test" at julialang discourse as well as accompanying blog posts. It is not very strictly a scientific writing, however, I think it is interesting and refreshing. Fun to read.

sry, no idea ;-)

No need to feel sorry.

-- but this sailing route sounds promising

Holiday time.

Regards.