Project

General

Profile

Piping arithmetic gives different results than working on files

Added by Elio Campitelli 2 months ago

I was working on removing the ENSO signal on variables and noticed a strange behaviour. Using the attached era_sst.nc file, if I try to remove the ENSO effect within a single piped expression, I get a different result than if I create a file with the ENSO effect and then subtract it.

# Compute a enso-like index
cdo -L -fldmean -sellonlatbox,190,240,-5,5 era_sst.nc enso_temp.nc
cdo -L -ymonsub enso_temp.nc -ymonmean enso_temp.nc ensno.nc

rm enso_temp.nc

# Create random slope and intercept terms
# in reality these would come from linear regression
cdo -L random,era_sst.nc,42 slope.nc
cdo -L random,era_sst.nc,45 intercept.nc

cdo -L enlarge,era_sst.nc enso.nc enso_field.nc

# Expression to compute the effect of ENSO 
enso_effect="-add -mul slope.nc enso_field.nc intercept.nc" 

# Compute residuals by piping the whole expression
cdo -L -sub era_sst.nc $enso_effect residual_pipe.nc

# note the message 
# cdo    sub: Filling up stream2 >(pipe1.7)< by copying the first timestep.
# era_sst.nc and the result of enso_effect should have the same number of 
# timesteps. Why the need to fill up the second stream?

# Compute residuals by saving the effect to a file and then subtracting 
cdo -L $enso_effect enso_effect.nc
cdo -L -sub era_sst.nc enso_effect.nc residual_seq.nc

# The differnece is not zero!
cdo -L info -sub residual_seq.nc residual_pipe.nc | head

Result of the last expression:

    -1 :       Date     Time   Level Gridsize    Miss :     Minimum        Mean     Maximum : Parameter ID
     1 : 1980-01-01 00:00:00       0    10368    3911 :  -0.0039062  2.4286e-05   0.0039062 : -1            
     2 : 1980-02-01 00:00:00       0    10368    3911 :  -0.0033875    0.029031    0.061271 : -1            
     3 : 1980-03-01 00:00:00       0    10368    3911 :    -0.24288    -0.12062   0.0037231 : -1            
     4 : 1980-04-01 00:00:00       0    10368    3911 :    -0.86571    -0.43523   0.0022888 : -1            
     5 : 1980-05-01 00:00:00       0    10368    3911 :    -0.78418    -0.39365   0.0022888 : -1            
     6 : 1980-06-01 00:00:00       0    10368    3911 :    -0.57350    -0.28754   0.0022888 : -1            
     7 : 1980-07-01 00:00:00       0    10368    3911 :  -0.0029907     0.12092     0.24309 : -1            
     8 : 1980-08-01 00:00:00       0    10368    3911 :  -0.0029907     0.38596     0.76908 : -1            
     9 : 1980-09-01 00:00:00       0    10368    3911 :  -0.0029907     0.29365     0.58588 : -1 

Am I misunderstanding something important about how cdo pipes work? Or is this a bug in cdo itself?


Replies (2)

RE: Piping arithmetic gives different results than working on files - Added by Uwe Schulzweida 2 months ago

The problem occurs in arithmetic operations such as mul/div/sub/add when the first file has only one time step and the second has more than one time step. In this case, the information that there is only one time step is erroneously forwarded to other operators. We will fix this problem in the next CDO version.
A workaround is to swap the two input files in the multiplication:

enso_effect="-add -mul enso_field.nc slope.nc intercept.nc" 

RE: Piping arithmetic gives different results than working on files - Added by Elio Campitelli 2 months ago

Thanks for the quick reply and the workaround!

    (1-2/2)