Algorithms modeling a possibly `deep’ structure in data have gained considerable popularity in recent years. Indeed, data sitting on a high-dimensional space can often be described by a hidden structure of much smaller « effective dimension ». A popular class of methods in this context is that of deep neural networks, e.g. with ReLU activation function. Another possibility is to use so-called deep Gaussian processes as prior distributions within a Bayesian approach.

In this talk I will first review a few results on Gaussian processes (GPs) used as priors in nonparametric function estimation problems, showing that GPs (combined with an  extra rescaling random variable) yield optimal convergence rates for posterior distributions, rates that are moreover adaptive to the `smoothness’ of the unknown function.

To achieve rates that are also adaptive to `structure’,  one may consider deep Gaussian Processes, namely compositions of Gaussian processes. Recently, Finocchio and Schmidt-Hieber (preprint, 2021) showed that there exists a well-chosen prior distribution consisting of a deep Gaussian process coupled with priors modeling both the smoothness and a `variable selection’ step, that achieves a near-optimal posterior rate, adaptive to both regularity and structure.

In this work, we introduce deep horseshoe Gaussian processes: each Gaussian process in the composition features a `lengthscale’ parameter along each dimension, parameter which itself gets a horseshoe prior distribution. We show that this horseshoe prior enables simultaneous adaptation to both smoothness and structure at optimal rates, without the need of a variable selection prior. The overall prior is then fairly close to the versions of deep GP priors that have been deployed in practice.

This is joint work with Thibault Randrianarisoa (Bocconi University, Milano).