In remote sensing systems, the capabilities of the system are constrained by the complex interactions between size, weight, and power (SWAP) of potential designs. In electro-optical (EO) systems, examples of these critical parameters include the system’s sensitivity and resolution. Those parameters can be increased by ever larger optical apertures and focal planes but at the cost of more SWAP. Multi-image super resolution (MISR) techniques allow resolution to be enhanced via computation rather than more sophisticated optical hardware. These algorithms combine multiple images together into a single, higher resolution image, trading temporal resolution and computation for spatial resolution. Fielded MISR techniques, such as Drizzle, can require several hundred images to create a single super resolved image, implying reduced temporal resolution, increased data acquisition load, and limiting mission applications. Iterative techniques, such as model-based image reconstruction and compressive sensing, have been shown to create super resolved images using fewer images than Drizzle. They do this by posing an optimization problem that balances accuracy between a highly accurate physical model and an image model. In the case of super resolution, the physical model is defined by the relation between low resolution input images and the desired high resolution output image. The image model encodes some assumptions about the super resolved image. These assumptions are meant to suppress reconstruction artifacts that arise due to deterministic physical model error, stochastic measurement noise, and potential undersampling. In practice, the performance of iterative methods are limited by imaging models compatible with optimization. Deep learning-based methods can effectively learn image models of arbitrary complexity, but lack the theoretical explainability and robustness of iterative techniques. Consensus equilibrium (CE) generalizes the iterative techniques beyond optimization, enabling blackbox algorithms such as traditional and neural image denoisers to be used as the image model. CE-based approaches retain much of the explainability and robustness of iterative techniques while allowing the expressiveness of machine learning image models to be used. Additionally, by unrolling iterations of CE with an embedded image denoiser, the image denoiser can be further trained and specialized to the specific application with potentially higher quality reconstructions. Under this project, we demonstrated the feasibility of training an unrolled neural network based upon CE. While we didn’t train one, we showed that the CE process is differentiable and its gradient can be tractably computed. We also explored the usage of a variants of CE akin to generative neural works. Most importantly, we applied the CE framework to a number of problems including non-blind deconvolution, upsampling, single-image super resolution, MISR, event-based sensing, and saturated deconvolution. Our MISR prototype creates high quality reconstructions with an order of magnitude fewer images than previous approaches and, critically, produces these reconstructions fast enough for practical usage.
Point-source transient events (PSTEs) - optical events that are both extremely fast and extremely small - pose several challenges to an imaging system. Due to their speed, accurately characterizing such events often requires detectors with very high frame rates. Due to their size, accurately detecting such events requires maintaining coverage over an extended field-of-view, often through the use of imaging focal plane arrays (FPA) with a global shutter readout. Traditional imaging systems that meet these requirements are costly in terms of price, size, weight, power consumption, and data bandwidth, and there is a need for cheaper solutions with adequate temporal and spatial coverage. To address these issues, we develop a novel compressed sensing algorithm adapted to the rolling shutter readout of an imaging system. This approach enables reconstruction of a PSTE signature at the sampling rate of the rolling shutter, offering a 1-2 order of magnitude temporal speedup and a proportional reduction in data bandwidth. We present empirical results demonstrating accurate recovery of PSTEs using measurements that are spatially undersampled by a factor of 25, and our simulations show that, relative to other compressed sensing algorithms, our algorithm is both faster and yields higher quality reconstructions. We also present theoretical results characterizing our algorithm and corroborating simulations. The potential impact of our work includes the development of much faster, cheaper sensor solutions for PSTE detection and characterization.
Spectrally resolved signals in the short- to mid-wave infrared (SWIR/MWIR) bands at high-temporal resolution are critical for many national security remote sensing missions. Currently available off the shelf technology can achieve either high temporal resolution or high spectral resolution, but rugged instruments that can achieve both simultaneously remain mostly in the realm of one-off R&D projects. This report documents efforts to demonstrate a new technique for designing and building high resolution, high framerate multichannel FTIR (MC-FTIR) spectrometers that operate in the SWIR/MWIR bands. The core optical element in a MC-FTIR spectrometer is an array of statically-tuned lamellar grating interferometers (LGI). In the original MC-FTIR work these arrays were fabricated using a synchrotron x-ray lithography method. We proposed to instead fabricate these LGI arrays using multiphoton lithography (MPL), a 3D printing technique that can fabricate meso-scale structures with sub-micron precision. Although we were able to fabricate LGI arrays of sufficient size using MPL, the realized optical surfaces had unsuitably high optical form errors, precluding their use in a fieldable instrument. Further advancement in MPL technology may eventually enable fabrication of interferometer-grade LGI arrays.
Over the last 15 years, compressive sensing techniques have been developed which have the potential to greatly reduce the amount of data collected by systems while preserving the amount of information obtained. A cost of this efficiency is that a computationally-intensive optimization routine must be used to put the sensed data into a form that a person can interpret. At the same time, machine learning techniques have experienced tremendous growth as well. Machines have demonstrated the ability learn how to effectively perform tasks such as detection and classification at speeds much faster than humanly possible. Our goal in this project was to study the feasibility of using compressive sensing systems "at the edge." That is, how can compressive sensing sensors be deployed such that information is created at the remote sensor rather than sending raw data to a central processing location? Studies were performed to analyze whether machine learning could be done on the compressively sensed data in its raw form. If a machine is performing the task, is it possible to do so without putting the data into a human interpretable form? We show that this is possible for some systems, in particular a compressive sensing snapshot imaging spectrometer. Machine learning tasks were demonstrated to be more effective and more robust to noise when the machine learning algorithm worked on data in its raw form. This system is shown to outperform a traditional spectrometer. Techniques for reducing the complexity of the reconstruction routine were also analyzed. Techniques for such as data regularization, deep neural networks, and matrix completion were studied and shown to have benefits over traditional reconstruction techniques. In this project we showed that compressive sensing sensors are indeed feasible at the edge. As always, sensors and algorithms must be carefully tuned to work in the constrained environment. In this project we developed tools and techniques to enable those analyses.
A Compressive Sensing Snapshot Imaging Spectrometer (CSSIS) and its performance are described. The number of spectral bins recorded in a traditional tiled array spectrometer is limited to the number of filters. By properly designing the filters and leveraging compressive sensing techniques, more spectral bins can be reconstructed. Simulation results indicate that closely-spaced spectral sources that are not resolved with a traditional spectrometer can be resolved with the CSSIS. The nature of the filters used in the CSSIS enable higher signal-to-noise ratios in measured signals. The filters are spectrally broad relative to narrow-line filters used in traditional systems, and hence more light reaches the imaging sensor. This enables the CSSIS to outperform a traditional system in a classification task in the presence of noise. Simulation results on classifying in the compressive domain are shown. This obviates the need for the computationally-intensive spectral reconstruction algorithm.
We present that distinguishing whether a signal corresponds to a single source or a limited number of highly overlapping point spread functions (PSFs) is a ubiquitous problem across all imaging scales, whether detecting receptor-ligand interactions in cells or detecting binary stars. Super-resolution imaging based upon compressed sensing exploits the relative sparseness of the point sources to successfully resolve sources which may be separated by much less than the Rayleigh criterion. However, as a solution to an underdetermined system of linear equations, compressive sensing requires the imposition of constraints which may not always be valid. One typical constraint is that the PSF is known. However, the PSF of the actual optical system may reflect aberrations not present in the theoretical ideal optical system. Even when the optics are well characterized, the actual PSF may reflect factors such as non-uniform emission of the point source (e.g. fluorophore dipole emission). As such, the actual PSF may differ from the PSF used as a constraint. Similarly, multiple different regularization constraints have been suggested including the l1-norm, l0-norm, and generalized Gaussian Markov random fields (GGMRFs), each of which imposes a different constraint. Other important factors include the signal-to-noise ratio of the point sources and whether the point sources vary in intensity. In this work, we explore how these factors influence super-resolution image recovery robustness, determining the sensitivity and specificity. In conclusion, we determine an approach that is more robust to the types of PSF errors present in actual optical systems.
Compressive sensing shows promise for sensors that collect fewer samples than required by traditional Shannon-Nyquist sampling theory. Recent sensor designs for hyperspectral imaging encode light using spectral modulators such as spatial light modulators, liquid crystal phase retarders, and Fabry-Perot resonators. The hyperspectral imager consists of a filter array followed by a detector array. It encodes spectra with less measurements than the number of bands in the signal, making reconstruction an underdetermined problem. We propose a reconstruction algorithm for hyperspectral images encoded through spectral modulators. Our approach constrains pixels to be similar to their neighbors in space and wavelength, as natural images tend to vary smoothly, and it increases robustness to noise. It combines L1 minimization in the wavelet domain to enforce sparsity and total variation in the image domain for smoothness. The alternating direction method of multipliers (ADMM) simplifies the optimization procedure. Our algorithm constrains encoded, compressed hyperspectral images to be smooth in their reconstruction, and we present simulation results to illustrate our technique. This work improves the reconstruction of hyperspectral images from encoded, multiplexed, and sparse measurements.
Compressive sensing shows promise for sensors that collect fewer samples than required by traditional Shannon-Nyquist sampling theory. Recent sensor designs for hyperspectral imaging encode light using spectral modulators such as spatial light modulators, liquid crystal phase retarders, and Fabry-Perot resonators. The hyperspectral imager consists of a filter array followed by a detector array. It encodes spectra with less measurements than the number of bands in the signal, making reconstruction an underdetermined problem. We propose a reconstruction algorithm for hyperspectral images encoded through spectral modulators. Our approach constrains pixels to be similar to their neighbors in space and wavelength, as natural images tend to vary smoothly, and it increases robustness to noise. It combines L1 minimization in the wavelet domain to enforce sparsity and total variation in the image domain for smoothness. The alternating direction method of multipliers (ADMM) simplifies the optimization procedure. Our algorithm constrains encoded, compressed hyperspectral images to be smooth in their reconstruction, and we present simulation results to illustrate our technique. This work improves the reconstruction of hyperspectral images from encoded, multiplexed, and sparse measurements.
Distinguishing whether a signal corresponds to a single source or a limited number of highly overlapping point spread functions (PSFs) is a ubiquitous problem across all imaging scales, whether detecting receptor-ligand interactions in cells or detecting binary stars. Super-resolution imaging based upon compressed sensing exploits the relative sparseness of the point sources to successfully resolve sources which may be separated by much less than the Rayleigh criterion. However, as a solution to an underdetermined system of linear equations, compressive sensing requires the imposition of constraints which may not always be valid. One typical constraint is that the PSF is known. However, the PSF of the actual optical system may reflect aberrations not present in the theoretical ideal optical system. Even when the optics are well characterized, the actual PSF may reflect factors such as non-uniform emission of the point source (e.g. fluorophore dipole emission). As such, the actual PSF may differ from the PSF used as a constraint. Similarly, multiple different regularization constraints have been suggested including the l1-norm, l0-norm, and generalized Gaussian Markov random fields (GGMRFs), each of which imposes a different constraint. Other important factors include the signal-to-noise ratio of the point sources and whether the point sources vary in intensity. In this work, we explore how these factors influence super-resolution image recovery robustness, determining the sensitivity and specificity. As a result, we determine an approach that is more robust to the types of PSF errors present in actual optical systems.
