5discussion.tex

\chapter{Discussion}
\label{chapter:discussion}


\section{Critical Evaluation of Results}


BCNN ensemble means produced acceptable deterministic skill, with properties in line with the RainNet baseline \cite{ayzel_rainnet_nodate} and other CNN based models, with characteristic blurring, tendency to be biased or unstable at further leadtimes, difficulties in skillfully nowcasting high rainrate thresholds, but high success at lower thresholds and very good MAE. It should be kept in mind that achieving high predictive skill both with low threshold/MAE and high threshold is a difficult problem which is the focus of active research, and is related in the Deep Learning approach to the data imbalance problem of strong precipitation. This means that failure to predict strong precipitation is related to those events representing only a small part of the training data, while being of high importance. Trying to accomodate this by means of event importance weighting is a possibility, but this weighting might easily lead to the network performing worse on all non-extreme cases.

Interestingly, \texttt{lt30} training after \texttt{lt5} pre-training not only did not improve, but worsened the predictive skill with Gaussian NLL loss. In unrelated experiments to this thesis, \texttt{lt30} used as a standalone has shown much better skill than \texttt{lt5} training using the LogCosh loss function, owning to a more stable iterative process. One hypothesis why the scheme didn't work out in the present case is that the network should not have been pre-trained with \texttt{lt5} but rather directly trained with \texttt{lt30}. One related explanation could be that the non-improvement might be due to the constant Gaussian NLL $\sigma_\ell$ data uncertainty parameter. Here with \texttt{lt5} training the parameter could indeed be assumed constant for each input to some degree, but it can be easily seen why iterative re-plugging in of the outputs breaks that assumption, as more uncertainty is brought into the inputs. However, even the assumption of constancy with \texttt{lt5} training is not really valid as aleatoric uncertainty strongly dependent on pixel position and rainrate values. For example places far from rain front have low uncertainty while convective cells have big uncertainty.

From a probabilistic nowcasting point of view, BCNN did not live up to expectations. Only CRPS is relatively close to baselines, but all other metrics show that BCNN makes relatively unskillful predictions compared to STEPS and LINDA-P. Rank histograms show that predictive uncertainty is most likely severely under-estimated. Reliability diagrams demonstrate that their predicted exceedance probabilities are not very reliable, and thus can not be trusted. ROC curves on the other hand show that the discriminative power of BCNN is also considerably worse than baseline, keeping in mind that the ROC curves suffer from too little probability categories, a subject which will be covered below. These results highlight the fact that in its current form, BCNN is not yet a useful probabilistic nowcasting method.

Looking at ensembles and rank histograms it is noticeable that ensemble variability seems low. This might be related to hyperparameters or more generally to the stochastic model as will be expanded on below, or it might have something to do with the small-scale detail loss occurring, as quick disappearance of them leads to only slowly evolving large-scale features remaining, which might undermine the variability at further leadtimes. If details were preserved such as in generative model based nowcasting methods, it could help assuming that is indeed part of the problem.% Related to the ensemble member variability, it is possible that 


\section{Validity of Assumptions and Experiments}

The way that hyperparameter optimization was carried out in this work is a possible source of problems, potentially having led to sub-optimal solutions. This and other validity issues will be covered in this section.

\subsection{SVI Issues}

When it comes to the Variational Inference method employed, there might have been unguarded assumptions regarding the prior and posterior families chosen. In particular, the problem in question might have benefited from a more careful choice of those families based on physical constraints and known behavior of precipitation.   
Related to this matter, it is known that Variational approximations tend to underestimate uncertainty of learned distributions. \cite{bishop2006pattern, minka_family_nodate}. However, it is hard to say if this is the only factor at play here when observing the rank histograms and reliability diagrams of BCNN models. As their rank histogram shape is generally concave compared to models based on stochastic perturbations, the variability of ensembles is indeed less than that of observations most pronouncedly in variational models, as one might expect. 

Nevertheless, the magnitude of the effect leaves the possibility that some parts of the model might have been non-optimal for the task, even within the SVI framework. The reason for this is that in part of preliminary experiments performed, uncertainty estimates for BCNN precursors were way more reliable than in current experiments. After performing the experiments, it was realized that one hyperparameter that was different and left un-optimized here is the initialization value of parameter posterior scales, which was set to $1e^{-3}$ here and $1e^{-2}$ in these preliminary experiments. Setting the parameter too high led to difficulties for the model to converge and unstable predicted rainrates, but while it was also known that setting it lower leads to initially smaller uncertainty estimates, no attention was paid while performing experiments to whether this might affect the asymptotic behavior of the estimates. Looking at Figure \ref{fig:bcnn-prior-weights}, it is seen that scales only deviate from their initial values by a factor of two to three at maximum, hinting at the fact that asymptotic marginal uncertainty estimates may very well be connected to the initial scale of parameters. 

\subsection{Likelihood Cost and Uncertainty}

It is possible that the magnitude of the Gaussian NLL loss observational noise parameter $\sigma_\ell$ influenced the predictive uncertainty estimates.
%, but increasing it from $1e^{-4}$ to $2e^{-2}$ as in preliminary experiments had no effect on the estimates. 
With regards to this parameter, optimizing for it using conventional RainNet was questionable, because the interactions between the complexity and likelihood terms of the ELBO are not thus taken into account, and the choice is only based on deterministic skill, which is only a small part and sometimes in contradiction with other aspects of what an ensemble probabilistic model should aim for. $10^{-4}$ is also much smaller than the realistic average uncertainty in data in the value range of interest, i.e. heavy rainfall. This uncertainty is expressed in terms of (log-transformed) rainrate (mm/h) and the data has a resolution expressed in reflectivity of 0.5 dBZ, which has for effect that the resolution of the rainrate data is dependent on rain intensity. This calls for non-constant data dependent, e.g. heteroscedastic uncertainties. Another facet of the model making the homoscedasticity assumption questionable is the iterative prediction scheme. This is because the outputs re-fed into the network will inherently have higher aleatoric uncertainty than original inputs, as model (epistemic) uncertainty of previous steps increments data uncertainty of next steps. Also, smaller-scale features often including high-intensity precipitation, have worse predictability and will suffer from higher uncertainty than large-scale features.

\subsection{Input Data}

Concerning the input data, the downsampling by a factor of two performed for computational performance reasons might have been detrimental for the validity of the verification of small-scale and especially high-intensity precipitation. There are two factors at play here. One is that because of the scale-dependence of precipitation lifetime, NN based nowcasting model performance might not scale the same way when changing scale as extrapolation-based classical models, which might make interpreting the performance obtained at bigger scales more difficult. The other factor is that because average pooling is used for downsampling, higher rainrates get diluted, an effect exacerbated by the on average small spatial extent of intense precipitation. Hence, downsampled performance may not necessarily extrapolate to higher spatial resolutions. This is important because in the end, developed precipitation nowcasting models are usually to be used with input data having a grid resolution of 500 meters to one kilometer. 

In addition to the downsampling issue, performing the train/validation/test split using six hour blocks might not have been enough to discard all harmful temporal correlations between different time-series, as input images from the end of one block are still correlated to images at the start of the next block, which is part of another split. Thus, even though the correlations are reduced, they do not disappear and the validation and test split performance is still over-estimated for Deep Learning models. A more robust split would have been using non-successive days as units.

\subsection{Verification Experiments}

RAPSD results suffer from potentially spurious gain in small wavelength power at 120 minutes. This might have something to do with the OR-masking of precipitation reducing too much the area used, or it might be some other unknown effect or mistake in the calculations, which is why the credibility of RAPSD results here is doubtful. 

ROC curves of Figure \ref{fig:roc} in probabilistic verification suffer from too few probability categories, as the curves exhibit apparent discontinuity, while they should approximately follow a bi-normal distribution \cite{mason1982model}. This undermines their accuracy for determining forecast accuracy and resolution.

\section{Directions for Further Work}

First and foremost, further work should aim to produce more reliable uncertainty estimates in the current framework. This could be approached by first examining the effect on initial posterior scales on ensemble variance. Secondly, BCNN would have to be trained and tested using non-downscaled 1km grid resolution reflectivity composites. If these do not resolve most current issues and more time and resources are available, it could be possible to implement other methods for uncertainty estimation using NN and compare them to Bayesian NN with Variational Inference. Different priors, maybe more informed ones or some with varying degrees of strictness, perhaps even non identically distributed among parameters, could be tried to see if that improves predictive skill. 

Some other simple tricks that might improve performance of BCNN are pre-learning the parameter means as point estimates and using the Bayesian approach to only learn posterior scales while keeping the means fixed. Assuming a non-biased deterministic model, this could make the learning process much more efficient, raising the quality of uncertainty estimates. Another trick would be to take a pre-trained BCNN model underestimating uncertainty, and add post-processing to the predictions, aiming to calibrate the bias and add stochastic noise for increasing predicted uncertainty. Lastly, despite not improving model performance, weight pruning could be tried for reducing computational cost of making nowcasts. \citet{blundell_weight_2015} show that Bayesian Neural Networks with a high number of parameters are very sparse thus and amenable to extensive weight pruning without much loss in skill. Although that study concentrates on fully-connected Neural Networks, it is worth considering if skillful pruning could be efficiently implemented in CNN architectures too, as \citet{shridhar_comprehensive_2019} seems to have experienced success doing that. 
Looking at a completely different direction, it might be a good idea to try and implement BNN versions of other established functional architectures, such as ConvLSTM or TrajGRU. 

Despite being one very interesting research direction, decomposition of predictive uncertainty into aleatoric and epistemic uncertainties is not performed, as it was deemed out of scope for this work. Indeed, trying to model the uncertainty the radar image inputs is a separate problem which is not so much of interest in the domain of radar-based precipitation, as so much more can be done to improve performance by ameliorating radar scan fidelity, preprocessing, and choice. Despite of this, one could still view aleatoric uncertainty as a proxy to non-predictability of precipitation given starting conditions represented by the input frames. This could in the case of the iterative models presented at least serve as a metric to quantify the uncertainty created by feeding back in previous model outputs and could potentially allow the compound aleatoric and epistemic ensemble spread to better match the true distribution of the data. If modeling the uncertainty as data dependent and using heteroscedastic Gaussian likelihood, it would be interesting to take the same approach as \citet{kendall_what_2017} and learn heteroscedastic uncertainties as a separate output channel. This uncertainty learning would eliminate the hairy problem of choosing a homoscedastic data uncertainty value a priori, and possibly make for more skillful probabilistic nowcasts. How the separate channel aleatoric uncertainty would be integrated with the ensembles for predictions is still an unsolved problem.