As part of the SURE-AI seminar series, Niklas Koenen from the Leibniz Institute BIPS in Bremen, will give a talk on explainable AI.

The talk and discussion take place in the Lounge at SimulaMet, Stensberggata 27, Oslo, on May 27 from 14:00-15:00.

If you can not join in person, you can follow the talk online.

Title: From deep neural network predictions toward understanding: Advances in explainable AI for complex black-box models. 

Bio: Niklas Koenen is working on explainable AI (XAI), especially feature-based explanations. He studied mathematics at the University of Bremen and recently completed his PhD in XAI. During his funded research visit in Oslo, he is working on methods for explaining predictive uncertainty in probabilistic and multivariate forecasting models.

Abstract: Machine learning models, particularly deep neural networks (DNNs), achieve impressive predictive performance on high-dimensional and multimodal data, yet their decision-making remains hidden inside the "black box." Explainable AI (XAI) addresses this challenge through feature-based methods that reveal which inputs are decisive for a model's output.

This talk presents recent advances in two parts, organized around the input and output sides of the attribution problem.

The first part concerns the input side: attributing scalar predictions back to features. While many attribution methods already exist, especially for DNNs, the work here is less about adding new ones and more about understanding them and why they often disagree, making them broadly accessible to applied users, and applying them to multimodal medical data, e.g., for the early detection of cognitive impairment. Going further, a conditional variant of feature importance based on generative models considers the dependencies between input features and thereby captures each feature's unique contribution given the others, rather than its marginal effect.

The second part turns to the output side, where the prediction itself has structure beyond a single scalar. Attribution is first extended to survival analysis, where the prediction takes the form of a survival function over time and time-dependent feature effects can be quantified efficiently. While this treats the multivariate outcome pointwise, the most recent work extends the output-side perspective from time-varying predictions to the full predictive distribution and explains the model's predictive uncertainty itself. A hierarchical entropy-based Shapley framework attributes the uncertainty of multi-step probabilistic forecasts to individual features, decomposed into marginal, sequentially conditional, and joint contributions across output components, plus a cross-component term that captures how features induce temporal dependence in the output.

Taken together, this talk works toward opening the black box from both sides, by explaining what modern ML models predict and how confident they are about it, while accounting for dependencies on both the input and output side.