Combining ESA Worldcover Tiles

Added by Johanna Strebl almost 2 years ago

Dear all,

I downloaded some WorldCover tiles for different countries (Greece, USA, Chile, Brazil, and Australia) from an S3 bucket (how to is described here https://esa-worldcover.org/en/data-access).

They come in .tif format, but I already translated them to .nc using GDAL. This worked just fine.
Now I'm trying to combine the tiles into one large netcdf. For this, I think I need cdo collgrid

However, I am running into some issues. I tried different things

When I run cdo collgrid *.nc combined.nc
I get the following error:
cdo collgrid (Abort): Number of input files (454) and number of blocks (3x151) differ!

Then I thought maybe I have to specify into how many I want to combine it, which is 1. So similar to the example in distgrid (here under 2.2.13 https://code.mpimet.mpg.de/projects/cdo/embedded/index.html)
I thought, maybe 1,1 would be correct, as I want it to be 1 in horizontal and 1 in vertical direction. However:
cdo collgrid,1,1 *.nc combined.nc results in the following error:
cdo collgrid (Warning): Variable name 1 not found!
cdo collgrid (Abort): Could not find all requested variables: (0/1)

My last attempt was: cdo collgrid,2 *.nc combined.nc
Which runs successfully, but the output is nonsense. It is a long line of squished data (see screenshot from QGIS).

I am not super experienced with geo data processing, so any help would be greatly appreciated!!

Best
Johanna

clipboard-202301241524-ijph4.png (870 KB) clipboard-202301241524-ijph4.png

Replies (3)

RE: Combining ESA Worldcover Tiles - Added by Karin Meier-Fleischer almost 2 years ago

Hi Johanna,

if you have 3 files to combine you should use

cdo collgrid,3 infile*.nc combined.nc

RE: Combining ESA Worldcover Tiles - Added by Johanna Strebl almost 2 years ago

Hi Karin,

thank you so much for your quick reply!
Does that mean that the number corresponds to the number of files I am trying to merge?
It executes without a problem then, but the output looks very similar. However, this time it's a horizontal line.

Do you think it is too many files/too much data? I already downsampled the NetCDFs to a lower resolution. Right now my .ncs have a 9km resolution (0.1 degrees).
Native resolution for ESA WorldCover is 10m, which used up 70 GBs of RAM before crashing.

cdo collgrid,454 *.nc combined.nc
cdo    collgrid: Processed 408600 values from 454 variables over 454 timesteps [0.11s 86MB].

I also tried writing a loop and merging them on top of each other, 1 by 1, to reduce RAM consumption. That works for a while, but there's a different error after a couple iterations (see second screen shot).

for FILE in $DATA_ROOT/ESAWorldcover/Version2020/*9km.nc; 
do echo $FILE 
if test -f "$DATA_ROOT/ESAWorldcover/_temp.nc"; then
    cdo -O -P 4 collgrid,1,1 $FILE $DataRoot/ESAWorldcover/_temp.nc $DATA_ROOT/ESAWorldcover/worldcover_complete_9km.nc; cp $DATA_ROOT/ESAWorldcover/worldcover_complete_9km.nc $DATA_ROOT/ESAWorldcover/_temp.nc
    else cp $FILE $DATA_ROOT/ESAWorldcover/_temp.nc
fi
done

Download all files

clipboard-202301241810-9cozw.png (809 KB) clipboard-202301241810-9cozw.png
clipboard-202301241831-55msv.png (665 KB) clipboard-202301241831-55msv.png

RE: Combining ESA Worldcover Tiles - Added by Estanislao Gavilan almost 2 years ago

Hi Johanna,

Have you checked how big is your file before crashing? Also, you could try to merge your files into small regions, and then merge them into big regions. Maybe, the loop is creating a huge file because it is not being compressed.

Kind regards,

Estanislao

(1-3/3)

Project

General

Profile

CDO