Project

General

Profile

Split files merged in one file by a specfic id

Added by Solomon Hailu over 3 years ago

I have a NetCDF file with more than 500000 files merged by their id (vid). the data is a time series from >500000 points with no lat. and lon. information.

I am trying to split the file using splitname and other related functions but it's not working. it works with nco (ncks -d var vid,1 in out)but this is very slow and takes about a week. Is there any code/ method in CDO to split the files using the vid and all the corresponding information, particularly the prcp ?

dimensions:
time = UNLIMITED ; // (35 currently)
bnds = 2 ;
vid = 562321 ;
variables:
double time(time) ;
time:standard_name = "time" ;
time:bounds = "time_bnds" ;
time:units = "days since 1989-1-1 00:00:00" ;
time:calendar = "proleptic_gregorian" ;
time:axis = "T" ;
double time_bnds(time, bnds) ;
int vid(vid) ;
vid:standard_name = "projection_x_coordinate" ;
vid:long_name = "unique identifier" ;
vid:units = "1" ;
vid:axis = "X" ;
int crs ;
crs:grid_mapping_name = "latitude_longitude" ;
float prcp(time, vid) ;
prcp:long_name = "upstream water storage" ;
prcp:units = "mm" ;
prcp:CDI_grid_type = "unstructured" ;
prcp:grid_mapping = "crs" ;
prcp:coordinates = "lat lon" ;
prcp:cell_methods = "time: mean" ;
Best regards,

Solomon


Replies (1)

RE: Split files merged in one file by a specfic id - Added by Ralf Mueller over 3 years ago

hi Solomon!

Your file does not seem to be CF-conform. So I don't expect CDO to work with it correctly.

There are several options to speed things up:

  • work in /dev/shm if your data is small enough (at least try to put the input there to reduce the read access time)
  • call ncks in parallel using tools like GNU parallel - it is written in PERL and should work on any unix-like system

hth
ralf

    (1-1/1)