Publications Details

Unsupervised physics-informed disentanglement of multimodal data

Walker, Elise; Trask, Nathaniel; Martinez, Carianne; Lee, Kookjin; Actor, Jonas A.; Saha, Sourav; Shilt, Troy; Vizoso, Daniel; Dingreville, Remi; Boyce, Brad L.

We introduce physics-informed multimodal autoencoders (PIMA)-a variational inference framework for discovering shared information in multimodal datasets. Individual modalities are embedded into a shared latent space and fused through a product-of-experts formulation, enabling a Gaussian mixture prior to identify shared features. Sampling from clusters allows cross-modal generative modeling, with a mixture-of-experts decoder that imposes inductive biases from prior scientific knowledge and thereby imparts structured disentanglement of the latent space. This approach enables cross-modal inference and the discovery of features in high-dimensional heterogeneous datasets. Consequently, this approach provides a means to discover fingerprints in multimodal scientific datasets and to avoid traditional bottlenecks related to high-fidelity measurement and characterization of scientific datasets.