Sha Taole · 沙陶乐

Mixture-of-expert (MOE) models are popular methods in machine learning, since they can model heterogeneous behaviour across the space of the data using an ensemble collection of learners. These models are especially useful for modelling dynamic data as time-dependent data often exhibit non-stationarity and heavy-tailed errors, which may be inappropriate to model with a typical single expert model. We propose a mixture of Student-t processes with an adaptive structure for the covariance and noise behaviour for each mixture. Moreover, we use a sequential Monte Carlo (SMC) sampler to perform online inference as data arrive in real time. We demonstrate the superiority of our proposed approach over other models on synthetic and real-world datasets to prove the necessity of the novel method.

Conventional Bayesian inference faces challenges regarding model misspecification and computational efficiency, especially in modern settings where datasets are increasingly large and high-dimensional. Within the nonparametric learning framework, the posterior bootstrap has become a viable alternative to posterior sampling due to its robustness and scalability. However, most existing methods are built on the Dirichlet process and its variants, which are known to produce posterior samples of the underlying distribution which are discrete. This discrete nature prohibits the inference of parameters which require the underlying distribution to have a density, and furthermore impede the existence of the posterior predictive density. In this work, we propose the partition posterior bootstrap (PPB): a smooth nonparametric learning method which relies on a data-adaptive partition of the sample space parametrized by a binary tree. Conditioned on this partition, posterior sampling involves a scalable posterior bootstrap which does not require Markov chain Monte Carlo. A key advantage of our method is that the tree partition can be learnt in a greedy empirical Bayesian manner from the corresponding marginal likelihood, which is both highly computationally efficient and theoretically tractable. Our proposed method can thus be regarded as a default and efficient nonparametric learning method for posterior inference and prediction when smoothness of the underlying distribution is required, especially for high-dimensional datasets. We demonstrate its excellent performance in both simulated and real-world datasets.