Project

General

Profile

MADIS netCdf to csv or xml so I can import into Python

Added by Jim Bungener about 4 years ago

Hello,

I am trying to understand how to convert a netcdf file downloaded from the NOAA MADIS servers to either xml or csv so I can process its content in a python scripts.
Any help is appreciated.
I am specifically looking for the latest time observation in that file for the various stations.
I am attaching a sample netcdf file below.
Thank you


Replies (4)

RE: MADIS netCdf to csv or xml so I can import into Python - Added by Jim Bungener about 4 years ago

In case you are wondering,

The reason I am not using NETcdf4 or another python package is that I am not the administrator on the machine I am using, and they don't want to install any other software or anything if we can do with what we have.
cdo is already installed on the machines so they are telling me to use that.

I was hoping to use netcdf4 but now I am really hoping that cdo will work for me
It's just the way it is, I guess.

I first need to figure out what the structure of the data is inside that file, then export it.
Thanks

RE: MADIS netCdf to csv or xml so I can import into Python - Added by Ralf Mueller about 4 years ago

If your admins don't want system-wide installations, you should use conda (see Anaconda for more) instead: Everything is in your private directories, you have full control. Please don't waste your time with text processing - you will loose the meta-information of your files.

Really, IMO you will put yourself into trouble working with an improper format. Check conda, you won't harm anyone.

RE: MADIS netCdf to csv or xml so I can import into Python - Added by Jim Bungener about 4 years ago

Thank you for this.

The problem is that I am writing code that will get exported to the website hosting servers and those are the machines that the admin don't want to mess with.

What are the different formats I can copy my netcdf file to? Maybe some of those already have a python library installed on our servers.

I just want to explore every avenue before going back to the administrators and requesting netcdf4.

Thanks.

RE: MADIS netCdf to csv or xml so I can import into Python - Added by Ralf Mueller about 4 years ago

there is native support for GRIB1 build into CDO, but you won't be able to convert every netcdf file to grib since netcdf allows full freedom in terms of variable names and grib is very restrictive. But I think I have an argument for you:

If the CDO installation on the target server already supports netcdf, then there already IS an installed version of the netcdf library. CDO does not implement its own netcdf read/write routines, but used the standard library from NCAR/Unidata. So there is no need to install something besides the netCDF4 bindings of python (and maybe xarray on top of it).

I see 3 options:
  1. ask for those packages to be installed by the admins
  2. If you can login to that server, you can try to install those on your own with the python package manager pip
    pip install netCDF4 --user
    This will lead to an installation somewhere in your $HOME dir, but python will find it. if pip is not installed, ask for it. If you can take responsibility for the packages instead of the admins, they might be ok with it
  3. If you can login to the server, install conda and all the needed packages. This way you will be completely independent of the admins

hth
ralf

    (1-4/4)