In this work, we introduce a family of novel activation functions for deep neural networks that approximate n-ary, or n-argument, probabilistic logic. Logic has long been used to encode complex relationships between claims that are either true or false. Thus, these activation functions provide a step towards models that can efficiently encode information. Unfortunately, typical feedforward networks with elementwise activation functions cannot capture certain relationships succinctly, such as the exclusive disjunction (p xor q) and conditioned disjunction (if c then p else q). Our n-ary activation functions address this challenge by approximating belief functions (probabilistic Boolean logic) with logit representations of probability and experiments demonstrate the ability to learn arbitrary logical ground truths in a single layer. Further, by representing belief tables using a basis that associates the number of nonzero parameters with the effective arity of each belief function, we forge a concrete relationship between logical complexity and sparsity, thus opening new optimization approaches to suppress logical complexity during training. We provide a computationally efficient PyTorch implementation and test our activation functions against other logic-approximating activation functions on both traditional machine learning tasks as well as reproducing known logical relationships.
In this paper, we address the problem of convergence of sequential variational inference filter (VIF) through the application of a robust variational objective and H∞-norm based correction for a linear Gaussian system. As the dimension of state or parameter space grows, performing the full Kalman update with the dense covariance matrix for a large-scale system requires increased storage and computational complexity, making it impractical. The VIF approach, based on mean-field Gaussian variational inference, reduces this burden through the variational approximation to the covariance usually in the form of a diagonal covariance approximation. The challenge is to retain convergence and correct for biases introduced by the sequential VIF steps. We desire a frame-work that improves feasibility while still maintaining reasonable proximity to the optimal Kalman filter as data is assimilated. To accomplish this goal, a H∞-norm based optimization perturbs the VIF covariance matrix to improve robustness. This yields a novel VIF-H∞ recursion that employs consecutive variational inference and H∞ based optimization steps. We explore the development of this method and investigate a numerical example to illustrate the effectiveness of the proposed filter.