Browsing by Author "Saleki, Maryam"

Now showing 1 - 4 of 4

Metadata only
Misclassification risk and uncertainty quantification in deep classifiers
(IEEE, 2021) Şensoy, Murat; Saleki, Maryam; Julier, S.; Aydoğan, Reyhan; Reid, J.; Computer Science; ŞENSOY, Murat; AYDOĞAN, Reyhan; Saleki, Maryam
In this paper, we propose risk-calibrated evidential deep classifiers to reduce the costs associated with classification errors. We use two main approaches. The first is to develop methods to quantify the uncertainty of a classifier’s predictions and reduce the likelihood of acting on erroneous predictions. The second is a novel way to train the classifier such that erroneous classifications are biased towards less risky categories. We combine these two approaches in a principled way. While doing this, we extend evidential deep learning with pignistic probabilities, which are used to quantify uncertainty of classification predictions and model rational decision making under uncertainty.We evaluate the performance of our approach on several image classification tasks. We demonstrate that our approach allows to (i) incorporate misclassification cost while training deep classifiers, (ii) accurately quantify the uncertainty of classification predictions, and (iii) simultaneously learn how to make classification decisions to minimize expected cost of classification errors.
Metadata only
Not all mistakes are equal
(The ACM Digital Library, 2020) Şensoy, M.; Saleki, Maryam; Julier, S.; Aydoğan, Reyhan; Reid, J.; Computer Science; AYDOĞAN, Reyhan; Saleki, Maryam
In many tasks, classifiers play a fundamental role in the way an agent behaves. Most rational agents collect sensor data from the environment, classify it, and act based on that classification. Recently, deep neural networks (DNNs) have become the dominant approach to develop classifiers due to their excellent performance. When training and evaluating the performance of DNNs, it is normally assumed that the cost of all misclassification errors are equal. However, this is unlikely to be true in practice. Incorrect classification predictions can cause an agent to take inappropriate actions. The costs of these actions can be asymmetric, vary from agent-to-agent, and depend on context. In this paper, we discuss the importance of considering risk and uncertainty quantification together to reduce agents' cost of making misclassifications using deep classifiers.
Metadata only
Risk-calibrated evidential classifiers
(2020-01-17) Saleki, Maryam; Aydoğan, Reyhan; Aydoğan, Reyhan; Şensoy, Murat; Tek, B.; Department of Computer Science; Saleki, Maryam
In some applications, intelligent agents rely on classifiers in order to make their decisions and accuracy of their predictions may play a significant role in performing their tasks successfully. Although deep neural networks perform very well in many classification tasks, they may sometimes fail in their predictions and the cost of all misclassification errors are usually considered as the same, which is not true in practice. For instance, classifying a pedestrian in a given image as a cyclist may cost significantly different from classifying it as a car for a self-driving car application. The costs of errors can be asymmetric, vary from agent-to-agent, and depend on context. Accordingly, this thesis proposes a novel approach for uncertainty quantification and risk-awareness in deep neural networks for classification. Our main intuition is that the predictive uncertainty can be quantified in a principled way; hence, classifiers can associate high uncertainty with their predictions when these predictions are more likely to be wrong. Furthermore, they incorporate the notion of misclassification risk during training, which allows them to avoid making wrong predictions leading to higher losses. To achieve this, the proposed risk-calibrated classifiers quantify the uncertainty in predictions based on the mean and variance of the Dirichlet distribution, and increase the uncertainty value for the predictions, which are more likely to be wrong. Furthermore, the model increases the uncertainty for the classifications, which are more risky. To validate the performance of our approach, we conducted experiments on a variety of well-known data sets. The results show that the proposed risk-calibrated classifiers associate high uncertainty with their misclassification. Furthermore, the risk minimization objective of our loss function allows neural networks to make less risky decisions for classification.
Metadata only
Uncertainty-aware deep classifiers using generative models
(Association for the Advancement of Artificial Intelligence, 2020) Şensoy, Murat; Kaplan, L.; Cerutti, F.; Saleki, Maryam; Computer Science; ŞENSOY, Murat; Saleki, Maryam
Deep neural networks are often ignorant about what they do not know and overconfident when they make uninformed predictions. Some recent approaches quantify classification uncertainty directly by training the model to output high uncertainty for the data samples close to class boundaries or from the outside of the training distribution. These approaches use an auxiliary data set during training to represent out-of-distribution samples. However, selection or creation of such an auxiliary data set is non-trivial, especially for high dimensional data such as images. In this work we develop a novel neural network model that is able to express both aleatoric and epistemic uncertainty to distinguish decision boundary and out-of-distribution regions of the feature space. To this end, variational autoencoders and generative adversarial networks are incorporated to automatically generate out-of-distribution exemplars for training. Through extensive analysis, we demonstrate that the proposed approach provides better estimates of uncertainty for in- and out-of-distribution samples, and adversarial examples on well-known data sets against state-of-the-art approaches including recent Bayesian approaches for neural networks and anomaly detection methods.