By codesigning a metaoptical front end in conjunction with an image-processing back end, we demonstrate noise sensitivity and compactness substantially superior to either an optics-only or a computation-only approach, illustrated by two examples: subwavelength imaging and reconstruction of the full polarization coherence matrices of multiple light sources. Our end-to-end inverse designs couple the solution of the full Maxwell equations—exploiting all aspects of wave physics arising in subwavelength scatterers—with inverse-scattering algorithms in a single large-scale optimization involving degrees of freedom. The resulting structures scatter light in a way that is radically different from either a conventional lens or a random microstructure, and suppress the noise sensitivity of the inverse-scattering computation by several orders of magnitude. Incorporating the full wave physics is especially crucial for detecting spectral and polarization information that is discarded by geometric optics and scalar diffraction theory.
Computational imaging and computer vision plays an increasingly important role in modern technology, ranging from simplest image de-noising routines to state-of-the-art object recognition, robotic vision and machine intelligence algorithms with widespread demand in defense, medical as well as emerging Internet-of-Things (IoT) industries. Traditional computer vision is exclusively driven by innovating the computational back end, and more recently, via deep learning and AI software. Little attention has been paid to the optical hardware at the front end beyond conventional lenses and diffraction gratings, in which light propagation is designed only by geometric optics , , . The full potential of wave physics has yet to be exploited for imaging device design in conjunction with computational reconstruction, especially for extracting spectral and polarization information that is mostly discarded by geometric optics. The last decade has seen explosive advances in understanding and manipulation of light waves and light–matter interactions at the most profound level of nanomaterials, abetted by the development of efficient numerical modeling/design techniques as well as the advent of sophisticated nanofabrication machinery. Those capabilities have been exploited for purely optical designs, such as metasurface lenses, that involve little or no computational post-processing , [, 5]. In this paper, we demonstrate the potential of 3D nanophotonics in the development of next-generation computer-vision technologies, in which conventional optics hardware is replaced by exquisitely designed nanophotonic structures; in particular, we propose to bring deeper and richer physics to computer vision by optimally tailoring a nanophotonic front end for a computational-imaging back end using a fully coupled inverse-design process, offering ultracompact form factors as well as unprecedented capabilities for physical data acquisition and manipulation.
A conventional all-optical imaging system (Figure 1a) maps each point in a “target” space onto a separate sensor pixel, directly producing a faithful spatial image but requires bulky optics, and also typically fails to capture detailed spectral or polarization content without additional filters. In another extreme, a compact lens-free system (Figure 1b) would directly detect a blurry image of the target and attempt to solve the subsequent “inverse scattering” problem (target reconstruction by, e.g., least square fitting), which is typically very ill-conditioned and hence sensitive to noise , , , , . In this paper, we introduce an end-to-end approach for inverse scattering (Figure 1c), in which a compact metaoptical structure is generated by large-scale inverse design of the full Maxwell equations coupled with signal processing for target recovery, both for conventional spatial imaging and for spectral polarimetry. First, we show that noise-tolerant subwavelength (0.2λ) far-field reconstruction of a collection of point sources is possible even with an ultracompact (2λ-thick) imaging device. Second, we demonstrate a “multidimensional” polarimeter that can resolve the full polarization states of multiple point sources at multiple frequencies. Specifically, we design metaoptical structures that generate well-conditioned (noise-robust) inverse-scattering problems, while exploiting a simple Tikhonov-regularization method (Section 3) to obtain subwavelength resolution without subwavelength focusing, or to enable multidimensional information extraction from a single-shot measurement. Accomplishing this requires that the optical “inverse” design problem, involving large-scale optimization over ≈104 degrees of freedom, be coupled with the reconstruction algorithms (Section 2). That is, we perform “end-to-end” design in which the error L(ε, p) of the reconstructed targets is jointly minimized as a function of both the microstructure (ε) and the reconstruction parameters (p). Applying this approach to a two-dimensional (2D) example problem (Section 3), we obtain 0.22λ spatial resolution with a robust condition number (noise sensitivity) of only ≈10, an improvement of 102–103 over the condition numbers for lens-free or random (diffusing ) scattering structures. Applying similar techniques to the polarimetry problem (Section 4), we obtain a full-3D inverse-designed probe with a robust condition number of ≈6 that can reconstruct nine-parameter polarization-coherence matrices of two point sources emitting at two frequencies.
Recent work in end-to-end computational imaging achieved improved image quality using regularized least-square image reconstruction in conjunction with scalar diffraction theory (rather than the full Maxwell equations) to design a phase plate (i.e., treated as locally uniform and neglecting multiple scattering) . Flat-optics metalenses , , , in contrast, have utilized more complete wave optics theory ranging from locally periodic , [, 13] or overlapping  domain approximations to full Maxwell calculations , [, 16] coupled with optimization-based inverse design , , , exploiting local resonances and multiple scattering to achieve diffraction-limited focusing , [, 21]. Four-parameter Stokes imaging has also been demonstrated using a metaoptics polarization sorter in combination with a refractive lens . However, these works specified the focal point and/or the desired wavefront a priori, even with more complex focal patterns chosen to facilitate subsequent computational processing , , , , , rather than performing a fully coupled end-to-end design. There is also a vast body of work on computational image reconstruction , [, 29], but decoupled from the lens design (taking the optics as an immutable input rather than as design parameters). In contrast, we couple the full Maxwell equations with the post-processing reconstruction during the design process (Section 2), so that an optimal wavefront is determined for each source to maximize reconstruction accuracy. Specifically, we demonstrate imaging with subwavelength resolution and multidimensional information extraction in ultracompact form factors, a feat not possible using previously reported end-to-end computational imaging. In order to perform this optimization, we employ standard adjoint techniques from photonic inverse design , ,  combined with automatic-differentiation tools  to obtain the sensitivity to changes in structural parameters ε and reconstruction parameters p.
2 End-to-end framework
Figure 2 shows a schematic of our proposed framework which can be applied to any wave-scattering problem including imaging, spectroscopy, polarimetry or any combination thereof. Here, the goal is to reconstruct a target u in a preselected region of interest by computationally analyzing the captured image v on a sensor. In between the sensor and the target region, we place a scattering structure, aka a photonic probe, ε(r) to be designed, at a “working” distance du from the target and an “image” distance dv from the sensor. The state of the target is specified by a vector containing spatial, spectral and/or polarization information, the details of which depend on the specific problem at hand. The sensor has m pixels with corresponding intensities (raw image) given by the forward scattering model where G represents the solution of the Maxwell equations and η is an additive noise vector. For simplicity, we will consider zero-mean half-Gaussian white noise with nonzero standard deviation (; note that intensity noise η is non-negative) , although our method can be easily adapted to other noise models (such as Poisson/shot noise) by calibrating the camera beforehand. We consider a planar sensor, which is the most common configuration in imaging, but our framework can readily be extended to arbitrary sensor topologies. The linear kernel G is a m × n matrix whose columns are essentially point spread functions (PSF) , [, 31] computed from the underlying Maxwell equations given a structure ε(r).
The raw image v is fed into a signal-processing algorithm to approximately reconstruct u , in our case by a regularized least-square fit. That is, we find such that . Here, R is a regularization operator which serves to condition a typically ill-posed inverse problem; essentially, R incorporates any prior assumptions about u (such as smoothness or sparsity) which ensure that the inverse problem has a stable unique solution . In particular, we choose a Tikhonov (L2) regularization where α > 0 is a regularization parameter to be determined, and has a closed-form solution . The noise sensitivity of the reconstructed is characterized by the condition number of the matrix G, which is a dimensionless quantity ≥1 that is roughly proportional to the ratio of the relative error to the input noise  ( can be computed as the ratio of the largest to smallest singular values of G). Many other variations are possible, such as L1 “sparse” reconstruction  or artificial neural networks , [, 35]. As we discuss in Section 5, our approach extends easily to such techniques, even if the reconstruction problem does not have a closed-form solution or it involves a vast number of free parameters to be determined. In general, a reconstruction algorithm is characterized by a vector p of P parameters; in this example, and P = 1.
The end-to-end inverse design seeks optimal choices of ε and p, which are tightly coupled by the end-to-end work-flow, in order to minimize the difference between the reconstructed against the ground truth u. Specifically, we define a loss function , here a mean-square error (MSE), such that where denotes averaging (expected value) over many realizations of u and η. The formulation can be now written as:
Here, the PSF matrix G is extracted from the numerical solution of the Maxwell equations by any method.
In this paper, we consider the frequency-domain Maxwell equations with time-harmonic sources :
solved by a finite-difference frequency-domain (FDFD) method . For each voxel in the target region, J is chosen as a point-source situated at the center of the voxel and the corresponding PSF is obtained by simulating the integrated electric field intensities over the sensor plane. The optimization over ε, p require their gradients , , which can be found by back-propagation through the signal-processing stage  and adjoint sensitivity analysis , [, 19] of the Maxwell equations. We numerically implement these gradients by coupling an open-source automatic-differentiation packages  with our own Maxwell adjoint solvers .
3 Imaging at subwavelength resolutions
To demonstrate the capability of our framework, we consider an imaging problem at subwavelength resolutions. We consider a 2D problem (Figure 3) with y-polarized electric fields, so that the Maxwell equations are reduced to a scalar 2D Helmholtz equation. Specifically, we set (compact, but in the far field) and (where near-field effects may be relevant) (Figure 3a) where λ is the operating wavelength. Also, we discretize a 1D 2λ-wide target region into n = 10 point sources on an equispaced grid. Here, we assume incoherent illumination of the target region (as is common for imaging) so that only intensities need to be considered . Therefore, an arbitrary target residing within the region is described by an intensity vector with a spatial resolution of (for targets at “infinity,” such as a photographic scene, the region of interest is an angular field of view and one can consider plane-wave sources instead of point sources). The probe and sensor have a width of 50λ and the sensor contains m = 50 pixels with a pixel size . Although we have chosen these parameters for ease of demonstration, we note that this scenario is realizable using selective illumination , slit apertures, a high-speed scanning mode, and line sensors  to produce 2D or even 3D images over a wide field of view. More importantly, this system illustrates the essential ingredients of many important applications as discussed in Section 5.
Although we have set m > n (a nominally “over-determined” inverse problem), it is important to note that not any ε(r) will lead to a well-conditioned (noise-robust) PSF matrix G. It is ill-advised to use a randomly chosen ε profile and directly invert G because not every probe can resolve two point sources separated by a distance of 0.22λ and project measurably-distinct noise-tolerant PSFs onto a coarse-resolution sensor one wavelength away from the probe (small dv leaves little room for conventional magnification). For example, we checked that a uniform ε leads to G with a condition number ; even a disordered ε with rapidly varying fine features yields . Both of these values represent orders of magnitude amplification of input noise in the output reconstruction, indicating that radical re-design of ε is required.
We show that our end-to-end framework can discover novel geometries ε(r) with greatly reduced , thereby rendering the inverse problem robust against noise. Here, the ε degrees of freedom are a set of freeform variable heights  at each pixel within a double-layer design region made up of a low-permittivity polymer material εpolymer ≈ 2.3 in air (see Figure 3a). We have chosen these material settings because of rapidly maturing nanoscale 3D-printing technologies , [, 43] that would allow for exploration of such complex 3D geometries and are particularly suited for taking advantage of the full power of freeform topology optimization , [, 19].
We employ stochastic gradient descent  for optimizing ε and α over iterations including random noise η; we found that α stays close to an initial choice of 0.5 while ε evolves considerably during the course of optimization. In practice, we found that it works just as well to fix α with zero noise (η = 0) as to vary α under many realizations of η (note that α is closely related to the noise variance σ2 ). Figure 3a exhibits a double-layer optimized design in a 3D-printable polymer-matrix (for example, Nanoscribe IP-DIP , [, 47]); each layer has thickness λ/2 and the minimum feature size is ≈0.04λ, which may be challenging to fabricate at visible wavelengths but is feasible at longer wavelengths such as mid-wave and far-wave infrared or even millimeter waves , [, 49]. Figure 3b demonstrates that optimization rapidly improves both MSE (≈10−6) and . Figure 3c shows two example ground truth targets being reconstructed under different noise levels. A Gaussian noise is added to the image: , where the standard deviation σ is chosen as a percentage relative to the average PSF intensity ; for example 5% noise indicates . The low condition number ensures that the reconstruction errors are not amplified, remaining at ≈5% and 10%, respectively, for different σ′s.
Like almost all computational imaging , this device is designed to reconstruct targets situated at a fixed grid (as in any camera with discrete pixels), but the accuracy degrades gracefully for sources deviating from this grid. As shown in Figure 3d, even for the worst case of a light source that lies halfway between two grid points, the reconstructed image mostly divides the intensity between the two closest points (the error intensity at further points could be reduced if an L1 “sparsifying” reconstruction algorithm  were used instead of L2 minimization). This degradation is known as “gridding error” in the computational-imaging community . A number of supplementary algorithms have been proposed to further improve the reconstruction for off-grid sources, including atomic-norm minimization  and coherence-inhibition schemes , which could be incorporated into end-to-end optical design if desired. Yet another way to improve the reconstruction scenario would be to minimize the error over a collection of offset grids (instead of a single fixed grid of training data).
Our results suggest that a low-index photonic microstructure with a highly complex geometry can faithfully reconstruct an image down to deeply subwavelength resolutions (albeit over a finite array of equispaced calibrated point sources), while maintaining a sufficiently high signal-to-noise ratio. From a fundamental-physics perspective, we note that even though the probe is close to the target, the former is clearly not in the near field of the latter (since ), which means evanescent fields from the target have negligible amplitude at the probe. Instead, the subwavelength resolution is made possible by the ability of the computational probe to distinguish the subtle differences in spatial frequency components coming from adjacent point sources , , . Therefore, our approach is unlike negative-index metamaterial superlenses , [, 58] or super-oscillatory lenses , which seek perfect point-to-point physical image formation via amplification of evanescent waves or subdiffraction-limit focal spots without the aid of computational reconstruction.
4 Spatial + spectral + polarization extraction
The ability to intimately manipulate the polarization states of light is a hallmark of vectorial Maxwell photonics , [, 61], which sets it apart from traditional geometric or diffractive optics. For example, in super-resolution microscopy with traditional lenses, the unresolved polarization state of a fluorescent molecule may affect localization accuracy and degrade the image recovery process, posing a nuisance in many imaging systems , [, 63]. Here, we show that end-to-end optimization can be used to design a nanophotonic polarimeter that can resolve the polarization coherence state of a fluorescent molecule, approaching theoretical upper bounds . In particular, the instantaneous polarization state of a point-dipole source (e.g., a fluorescent molecule or a solid-state quantum emitter, such as a quantum dot or color center) is specified by the complex-valued 3-element polarization vector . However, only the time-averaged intensities can be detected at optical frequencies, and the detectable polarization state of the dipole is described by a 3 × 3 coherence matrix  (equivalent to a matrix ):
Here, denotes a time average. These nine coherence parameters associated are a natural generalization of the familiar four St parameters  that characterize the polarization coherence state of a plane wave.
The electric-field response of a nanophotonic structure in the presence of an arbitrarily polarized point dipole can be completely specified by the three “basis” fields, , and , derived from x-, y- and z-polarized test sources at the same location as the dipole: . The integrated time-averaged electric-field intensity at the ith pixel on the sensor is then given by:
Hence, it should be possible to extract the full nine-element coherence state from a linear inverse-scattering problem with an appropriately PSF kernel:
In fact, an ultracompact single-piece nanophotonic structure should be able to resolve not only polarization states but also extract spatial and spectral information simultaneously from a single measurement. As a proof of principle, we present in Figure 4 an “all-in-one super-probe” which can extract polarization coherence information from up to two spatial points and up to two spectral lines, in which case the target vector u to be reconstructed consists of 36 entries (9 polarization × 2 spatial × 2 spectral). The nanophotonic probe has an ultracompact volume of , comprising four layers of variable-thickness polymer (refractive index ≈ 1.5) (Figure 4a). A sensor of 10 × 10 pixels is located one λ away on one side of the probe; on the other side, the two dipoles are positioned one λ away from the probe and diagonally separated by away from each other; the dipoles may emit at λ, 1.1λ, or both. The kernel G is a 100 × 36 matrix, whose condition number has been optimized to (Figure 4b). We note that, in general, more layers lead to better optimal results. For example, we obtained an optimal condition number of 22 when using a single layer, 12 when using two layers, and arriving at 6.5 when using the four layers presented here. Here, it is important to emphasize the pivotal role of optimization which greatly improves the noise sensitivity of reconstruction compared to any other nonoptimized structure. For example, we found that the kernel G of free space has and that of a correlated random mask (with the correlation length chosen to have similar λ − 2λ length-scales to the optimized design) has ; even modifying the optimized probe by simply discretizing the gray-scale thickness or erasing the thin morphological features may spoil κ by a factor anywhere between 1.3 and 10 (that is, any feature-size or binary-thickness constraints must be incorporated during the optimization process using standard techniques , not imposed after the fact). We also found that the power captured by the sensor in the presence of the optimized probe is, on average, ∼3× greater than for free space or a random mask, indicating that the probe also serves to focus light onto the sensor, further enhancing the noise tolerance. Figure 4c shows the reconstruction of two dipoles with two different polarization states and emitting at two wavelengths λ (blue) and 1.1λ (red); the dipoles are circularly polarized along different axes; their coherence matrices are reconstructed under different noise levels with reconstruction errors of ≈5%, 10% respectively. The inverse-design optimization took ∼3 days and was performed using a time-domain Maxwell solver  distributed over 384 processors. We expect that further acceleration can be achieved by symmetry considerations , overlapping domain decompositions , broadband simulations and GPU-accelerated time-domain software.
We emphasize that although our proof-of-concept nanophotonic probe is only in area, it can be used to resolve two points, two spectral lines and nine polarization coherence values (36 in total) with a robust condition number of 6.5. Increasing the number of points and/or spectral lines under the same probe area surely increase the optimal condition number. However, modern image sensors are known for relatively low noise statistics ( noise), suggesting that many more points and/or spectral lines can be accurately reconstructed under a relatively high condition number ∼1000 associated with a small device area. At the same time, a larger probe area (wider field of view) admits even more spatial/spectral data without worsening the condition number; such a large-area probe may be realized by a periodic array of small probes or a domain-decomposition design of a large non-periodic probe . On the other hand, in our paper, we have used the simplest reconstruction algorithm based on Tikhonov regularization; we expect that more advanced methods like compressed sensing or neural networks will allow us to extract even more information.
5 Summary and outlook
The key conclusion of our paper is that optical metastructures designed in conjunction with signal processing result in nonobvious light-scattering patterns that greatly ease the computational reconstruction. These results in devices far more compact compared to optics-only solutions while being robust to noise compared to computation-only designs. By solving the full (Maxwell) wave equations during the design process, our optimized structure can exploit all available wave physics (nonparaxial scattering, near-field interactions, resonances, dispersion, etc.). We illustrated this idea in the context of examples involving subwavelength imaging and for polarization-state reconstruction, but the same essential ideas can be readily applied to many other systems and computational processing techniques. In contrast to the many previous metasurface designs that have attempted to mimic and compete with traditional curved lenses , our scattered fields look nothing like a focal pattern and represent a functionality that is fundamentally distinct from that of conventional optics. Fullwave end-to-end optimization is particularly powerful for problems requiring spectral and polarization information that is discarded by geometric optics, such as polarimetry or hyperspectral imaging.
There are many other sensing/imaging problems that could benefit from this approach. Our designs in this paper closely resemble lab-on-a-chip microscopy. Related situations arise in ultracompact optofluidic medical sensors, where the probe and sensor must be tightly integrated, the sample is situated only a few wavelengths away from the sensor, and scanning is naturally provided by sample flow . Inverse design can easily be applied to broadband problems, and we are especially excited about using it for computational spectroscopy , hyper-spectral imaging , [, 72], and other broadband sensing applications. Our framework can straightforwardly scale to larger 3D freeform structures , accommodate complex high-dimensional objects such as plenoptic light-fields , facilitate nonlinear mechanisms such as high dynamic-range imaging , and generalize to other challenging problems in physics such as nonlinear pulse shaping  and quantum coherence engineering , [, 77]. Optimization can easily incorporate constraints arising from different fabrication processes .
In this paper, our computational-reconstruction stage consisted of Tikhonov-regularized least-squares fitting, but end-to-end optical design can be coupled with many other computational techniques. In under-determined systems (many more targets than sensor pixels), a common approach is compressed sensing  for sparse targets, and techniques for end-to-end optimization with compressed sensing may include differentiable unrolled approximations  or epigraph formulations of basis pursuit denoising . One could also employ deep learning (neural networks) for imaging and other cognitive tasks (e.g. passive ranging, object recognition); from the perspective of deep learning, the Maxwell solver is simply a specialized “network stage” that is differentiable (via adjoint methods) and hence composable with deep-learning software.
Apart from numerical and experimental endeavors, an important theoretical question is to identify the absolute limits to achievable dispersion (spatial or spectral) and condition numbers, given a desired resolution, a design volume V, and a dielectric contrast . Recent approaches for shape-independent bounds to light–matter interactions , ,  may be capable of answering these questions.
Funding source: U.S. Army Research Office through the Institute for Soldier Nanotechnologies
Award Identifier / Grant number: W911NF-18-2-0048
Funding source: MIT-IBM Watson AI Laboratory
Award Identifier / Grant number: 2415
Funding source: National Science Foundation
Award Identifier / Grant number: NSF-SNM-1825308
Author contributions: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.
Research funding: Z.L, C.R.C, R.P, M.S and S.G.J were supported in part by the U.S. Army Research Office through the Institute for Soldier Nanotechnologies under award number W911NF-18-2-0048. Z.L and R.P were partially supported by the MIT-IBM Watson AI Laboratory under Challenge 2415. A.M. was partially supported by a Sloan Fellowship and by the National Science Foundation under award NSF-SNM-1825308.
Conflict of interest statement: The authors declare no conflicts of interest regarding this article.
 N. Antipa, G. Kuo, and R. Heckel, et al., “DiffuserCam: lensless single-exposure 3D imaging,” Optica, vol. 5, no. 1, pp. 1–9, 2018, https://doi.org/10.1364/optica.5.000001.Search in Google Scholar
 S. Vincent, S. Diamond, Y. Peng, et al., “End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging,” ACM Trans. Graph., vol. 37, no. 4, pp. 1–13, 2018.Search in Google Scholar
 D. Xiong, H. Ikoma, W. Gordon, Z. Wang, X. Cheng, and Y. Peng, “Learned rotationally symmetric diffractive achromat for full-spectrum computational imaging,” Optica, vol. 7, no. 8, pp. 913–922, 2020.Search in Google Scholar
 M. Khorasaninejad, W. T. Chen, R. C. Devlin, J. Oh, A. Y. Zhu, and F. Capasso, “Metalenses at visible wavelengths: diffraction-limited focusing and subwavelength resolution imaging,” Science, vol. 352, no. 6290, pp. 1190–1194, 2016, https://doi.org/10.1126/science.aaf6644.Search in Google Scholar
 T. Albert, Inverse Problem Theory and Methods for Model Parameter Estimation, Philadelphia, PA, SIAM, 2005.Search in Google Scholar
 P. R. Gill and D. G. Stork, “Computationally efficient-based image reconstruction in a lensless diffractive imager,” in Computational Optical Sensing and Imaging, pages CM3E-4, Arlington, VA, Optical Society of America, 2015.Search in Google Scholar
 M. Salman Asif, A. Ali, A. Sankaranarayanan, A. Veeraraghavan, and R. G. Baraniuk, “Flatcam: thin, lensless cameras using coded aperture and computation,” IEEE Trans. Comput. Imag., Vol. 3, no. 3, pp. 384–397, 2016.Search in Google Scholar
 P. Gill, Enabling a Computer to Do the Job of a Lens, Sunnyvale, CA, SPIE Newsroom, 2013.Search in Google Scholar
 A. Wang, P. Gill, and A. Molnar, “Angle sensitive pixels in cmos for lensless 3d imaging,” in 2009 IEEE Custom Integrated Circuits Conference, IEEE, 2009, pp. 371–374.Search in Google Scholar
 W. T. Chen, Y. Z. Alexander, V. Sanjeev, et al.., “A broadband achromatic metalens for focusing and imaging in the visible,” Nat. Nanotechnol., vol. 13, no. 3, p. 220, 2018, https://doi.org/10.1038/s41565-017-0034-6.Search in Google Scholar
 R. Pestourie, C. Pérez-Arancibia, Z. Lin, W. Shin, F. Capasso, and S. G. Johnson, “Inverse design of large-area metasurfaces,” Opt. Express, vol. 26, no. 26, pp. 33732–33747, 2018, https://doi.org/10.1364/oe.26.033732.Search in Google Scholar
 Z. Lin, V. Liu, R. Pestourie, and S. G. Johnson, “Topology optimization of freeform large-area metasurfaces,” Opt. Express, vol. 27, no. 11, pp. 15765–15775, 2019, https://doi.org/10.1364/oe.27.015765.Search in Google Scholar
 Z. Lin and S. G. Johnson, “Overlapping domains for topology optimization of large-area metasurfaces,” Opt. Express, vol. 27, no. 22, pp. 32445–32453, 2019, https://doi.org/10.1364/oe.27.032445.Search in Google Scholar
 Z. Lin, B. Groever, F. Capasso, A. W. Rodriguez, and M. Lončar, “Topology-optimized multilayered metaoptics,” Phys. Rev. Appl., vol. 9, no. 4, p. 044030, 2018, https://doi.org/10.1103/physrevapplied.9.044030.Search in Google Scholar
 C. M. Lalau-Keraly, S. Bhargava, D. M. Owen, and E. Yablonovitch, “Adjoint shape optimization applied to electromagnetic design,” Opt. Express, vol. 21, no. 18, pp. 21693–21701, 2013, https://doi.org/10.1364/oe.21.021693.Search in Google Scholar
 J. Søndergaard Jensen and O. Sigmund, “Topology optimization for nano-photonics,” Laser Photonics Rev., vol. 5, no. 2, pp. 308–321, 2011, https://doi.org/10.1002/lpor.201000014.Search in Google Scholar
 S. Molesky, Z. Lin, Y. P. Alexander, W. Jin, J. Vucković, and A. W. Rodriguez, “Inverse design in nanophotonics,” Nat. Photonics, vol. 12, no. 11, p. 659, 2018, https://doi.org/10.1038/s41566-018-0246-9.Search in Google Scholar
 E. Bayati, R. Pestourie, S. Colburn, Z. Lin, S. G. Johnson, and A. Majumdar, “Inverse designed metalenses with extended depth of focus,” ACS Photonics, vol. 7, no. 4, pp. 873–878, 2020, https://doi.org/10.1021/acsphotonics.9b01703.Search in Google Scholar
 T. Phan, D. Sell, E. W. Wang, et al.., “High-efficiency, large-area, topology-optimized metasurfaces,” Light Sci. Appl., vol. 8, no. 1, pp. 1–9, 2019, https://doi.org/10.1038/s41377-019-0159-5.Search in Google Scholar
 N. A. Rubin, G. D’Aversa, P. Chevalier, Z. Shi, W. T. Chen, and F. Capasso, “Matrix Fourier optics enables a compact full-Stokes polarization camera,” Science, vol. 365, no. 6448, p. eaax1839, 2019, https://doi.org/10.1126/science.aax1839.Search in Google Scholar
 Q. Guo, Z. Shi, Y.-W. Huang, E. Alexander, C.-W. Qiu, F. Capasso, and Z. Todd, “Compact single-shot metalens depth sensors inspired by eyes of jumping spiders,” Proc. Natl. Acad. Sci. U.S.A., vol. 116, no. 46, pp. 22959–22965, 2019, https://doi.org/10.1073/pnas.1912154116.Search in Google Scholar
 S. Colburn and Arka Majumdar, “Simultaneous achromatic and varifocal imaging with quartic metasurfaces in the visible,” ACS Phonics, vol. 7, no. 1, pp. 120–127, 2019, https://doi.org/10.1021/acsphotonics.9b01216.Search in Google Scholar
 S. Colburn and A. Majumdar, “Metasurface generation of paired accelerating and rotating optical beams for passive ranging and scene reconstruction,” ACS Photonics, vol. 7, no. 6, pp. 1529–1536, 2020, https://doi.org/10.1364/cleo_at.2020.am3k.2.Search in Google Scholar
 E. Khoram, A. Chen, D. Liu, et al.., “Nanophotonic media for artificial neural inference,” Photonics Res., vol. 7, no. 8, pp. 823–827, 2019, https://doi.org/10.1364/prj.7.000823.Search in Google Scholar
 G. S. Kino, Acoustic Waves: Devices, Imaging, and Analog Signal Processing, Englewood Cliffs, NJ, PrenticeHall, 1987.Search in Google Scholar
 A. Greenbaum, W. Luo, T.-W. Su, et al.., “Imaging without lenses: achievements and remaining challenges of wide-field on-chip microscopy,” Nat. Methods, vol. 9, no. 9, p. 889, 2012, https://doi.org/10.1038/nmeth.2114.Search in Google Scholar
 D. D. D. Maclaurin and R. P. Adams, “Autograd: Effortless gradients in numpy,” in ICML 2015 AutoML Workshop, vol. 238, 2015.Search in Google Scholar
 J. W. Goodman, Introduction to Fourier Optics, Greenwood Village, CO, Roberts and Company Publishers, 2005.Search in Google Scholar
 L. N. Trefethen and D. BauIII, Numerical Linear Algebra, vol. 50, Philadelphia, PA, SIAM, 1997.Search in Google Scholar
 S. Boyd, S. P. Boyd, and L. Vandenberghe, Convex Optimization, Cambridge, UK, Cambridge University Press, 2004.Search in Google Scholar
 C. Szegedy, T. Alexander, and D. Erhan, “Deep neural networks for object detection,” Adv. Neural Inf. Process. Syst., vol. 2, pp. 2553–2561, 2013, https://doi.org/10.5555/2999792.2999897.Search in Google Scholar
 F. Milletari, N. Navab, and A. Seyed-Ahmad, “V-net: fully convolutional neural networks for volumetric medical image segmentation,” in 2016 Fourth International Conference on 3D Vision (3DV), IEEE, pp. 565–571, 2016.Search in Google Scholar
 J. D. Jackson, Classical Electrodynamics, New York, NY, American Association of Physics Teachers, 1999.Search in Google Scholar
 J.-M. Jin, The Finite Element Method in Electromagnetics, Hoboken, NJ, John Wiley & Sons, 2015.Search in Google Scholar
 C. M. Bishop, Pattern Recognition and Machine Learning, New York, NY, Springer, 2006.Search in Google Scholar
 P. J. Keller, A. D. Schmidt, J. Wittbrodt, and H. K. S. Ernst, “Reconstruction of zebrafish early embryonic development by scanned light sheet microscopy,” Science, vol. 322, no. 5904, pp. 1065–1069, 2008, https://doi.org/10.1126/science.1162493.Search in Google Scholar
 Jian Wang, “High resolution 2D imaging and 3D scanning with line sensors,” PhD thesis, Carnegie Mellon University Pittsburgh, PA, 2018.Search in Google Scholar
 A. Udupa, J. Zhu, and L. L. Goddard, “topology optimization for fabrication-compatible inverse design of 3d photonic devices,” Optics Express, vol. 27, no. 15, pp. 21988–21998, 2019, https://doi.org/10.1364/oe.27.021988.Search in Google Scholar
 T. Bückmann, N. Stenger, M. Kadic, et al.., “Tailored 3d mechanical metamaterials made by dip-in direct-laser-writing optical lithography,” Adv. Mater., vol. 24, no. 20, pp. 2710–2714, 2012, https://doi.org/10.1002/adma.201200584.Search in Google Scholar
 A. Zhan, R. Gibson, J. Whitehead, E. Smith, J. R. Hendrickson, and A. Majumdar, “Controlling three-dimensional optical fields via inverse mie scattering,” Sci. Adv., vol. 5, no. 10, p. eaax4769, 2019, https://doi.org/10.1126/sciadv.aax4769.Search in Google Scholar
 L. Bottou, “Large-scale machine learning with stochastic gradient descent,” in Proceedings of COMPSTAT’2010, Springer, 2010, pp. 177–186.Search in Google Scholar
 T. Gissibl, S. Wagner, J. Sykora, M. Schmid, and H. Giessen, “Refractive index measurements of photo-resists for three-dimensional direct laser writing,” Opt. Mater. Express, vol. 7, no. 7, pp. 2293–2298, 2017, https://doi.org/10.1364/ome.7.002293.Search in Google Scholar
 D. B. Fullager, D. B. Glenn, and T. Hofmann, “Infrared dielectric response of nanoscribe IP-Dip and IP-L monomers after polymerization from 250 cm−1 to 6000 cm−1,” Opt. Mater. Express, vol. 7, no. 3, pp. 888–894, 2017, https://doi.org/10.1364/ome.7.000888.Search in Google Scholar
 C. Yu, S. Fan, Y. Sun, and E. Pickwell-MacPherson, “The potential of terahertz imaging for cancer diagnosis: a review of investigations to date,” Quant. Imag. Med. Surg., vol. 2, no. 1, p. 33, 2012.Search in Google Scholar
 P. Camayd-Muñoz, C. Ballew, G. Roberts, and A. Faraon, “Multifunctional volumetric meta-optics for color and polarization image sensors,” Optica, vol. 7, no. 4, pp. 280–283, 2020, https://doi.org/10.1364/optica.384228.Search in Google Scholar
 T. Strohmer, “Measure what should be measured: progress and challenges in compressive sensing,” IEEE Sig. Process. Lett., vol. 19, no. 12, pp. 887–893, 2012, https://doi.org/10.1109/lsp.2012.2224518.Search in Google Scholar
 G. Tang, B. N. Bhaskar, P. Shah, and B. Recht, “Compressed sensing off the grid,” IEEE Trans. Inf. Theor., vol. 59, no. 11, pp. 7465–7490, 2013, https://doi.org/10.1109/tit.2013.2277451.Search in Google Scholar
 F. Albert and W. Liao, “Coherence pattern–guided compressive sensing with unresolved grids,” SIAM J. Imag. Sci., vol. 5, no. 1, pp. 179–202, 2012.Search in Google Scholar
 S. Chen and A. Moitra, “Algorithmic foundations for the diffraction limit,” arXiv preprint arXiv:2004.07659, 2020.Search in Google Scholar
 M. Fu, Y. Chen, F. J. Garcia de Abajo, and I. H. Z. Nikolay, “Optical super-resolution through super-oscillations,” J. Opt. Pure Appl. Opt., vol. 9, no. 9, p. S285, 2007.Search in Google Scholar
 A. Arbabi, H. Yu, M. Bagheri, and A. Faraon, “Dielectric metasurfaces for complete control of phase and polarization with subwavelength spatial resolution and high transmission,” Nat. Nanotechnol., vol. 10, no. 11, pp. 937–943, 2015, https://doi.org/10.1038/nnano.2015.186.Search in Google Scholar
 J. P. B. Mueller, N. A. Rubin, R. C. Devlin, B. Groever, and F. Capasso, “Metasurface polarization optics: independent phase control of arbitrary orthogonal states of polarization,” Phys. Rev. Lett., vol. 118, no. 11, p. 113901, 2017.Search in Google Scholar
 M. P. Backlund, D. L. Matthew, S. B. Adam, S. J. Sahl, and W. E. Moerner, “The role of molecular dipole orientation in single-molecule fluorescence microscopy and implications for super-resolution imaging,” ChemPhysChem, vol. 15, no. 4, pp. 587–599, 2014, https://doi.org/10.1002/cphc.201300880.Search in Google Scholar
 M. P. Backlund, A. Arbabi, P. N. Petrov, et al.., “Removing orientation-induced localization biases in single-molecule microscopy using a broadband metasurface mask,” Nat. Photonics, vol. 10, no. 7, pp. 459–462, 2016, https://doi.org/10.1038/nphoton.2016.93.Search in Google Scholar
 J. Scott Tyo, “Design of optimal polarimeters: maximization of signal-to-noise ratio and minimization of systematic error,” Appl. Opt., vol. 41, no. 4, pp. 619–630, 2002, https://doi.org/10.1364/ao.41.000619.Search in Google Scholar
 J. N. Damask, Polarization Optics in Telecommunications, vol. 101, New York, NY, Springer Science & Business Media, 2004.Search in Google Scholar
 M. O. Scully and M. S. Zubairy, Quantum Optics, Cambridge, UK, Cambridge University, 1999.Search in Google Scholar
 A. F. Oskooi, D. Roundy, M. Ibanescu, P. Bermel, J. D. Joannopoulos, and S. G. Johnson, “MEEP: a flexible free-software package for electromagnetic simulations by the FDTD method,” Comput. Phys. Commun., vol. 181, no. 3, pp. 687–702, 2010, https://doi.org/10.1016/j.cpc.2009.11.008.Search in Google Scholar
 R. E. Christiansen, Z. Lin, C. Roques-Carmes, et al.., “Fullwave Maxwell inverse design of axisymmetric, tunable, and multi-scale multi-wavelength metalenses,” Opt. Express, vol. 28, no. 23, pp. 33854–33868, 2020, https://doi.org/10.1364/oe.403192.Search in Google Scholar
 H. Xin, D. Erickson, L. Ryan Baugh, et al., “Optofluidic microscopy—a method for implementing a high resolution optical microscope on a chip,” Lab Chip, vol. 6, no. 10, pp. 1274–1276, 2006.Search in Google Scholar
 C.-I. Chang, Hyperspectral Imaging: Techniques for Spectral Detection and Classification, New York, NY, Springer Science & Business Media, 2003.Search in Google Scholar
 K. Monakhova, K. Yanny, N. Aggarwal, and L. Waller, “Spectral DiffuserCam: Lensless snapshot hyperspectral imaging with a spectral filter array,” arXiv preprint arXiv:2006.08565, 2020, https://doi.org/10.1364/cosi.2020.jf2f.4.Search in Google Scholar
 N. Ren, M. Levoy, M. Brédif, et al., “Light field photography with a hand-held plenoptic camera,” Comput. Sci. Tech. Rep. CSTR, vol. 2, no. 11, pp. 1–11, 2005.Search in Google Scholar
 E. Reinhard, W. Heidrich, D. Paul, S. Pattanaik, G. Ward, and K. Myszkowski, High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting, San Francisco, CA, Morgan Kaufmann, 2010.Search in Google Scholar
 G. P. Agrawal, Applications of Nonlinear Fiber Optics, Cambridge, Massachusetts, Elsevier, 2001.Search in Google Scholar
 K. W. Murch, U. Vool, D. Zhou, S. J. Weber, S. M. Girvin, and I. Siddiqi, “Cavity-assisted quantum bath engineering,” Phys. Rev. Lett., vol. 109, no. 18, p. 183602, 2012, https://doi.org/10.1103/physrevlett.109.183602.Search in Google Scholar
 R. Bennett, “Inverse design of environment-induced coherence,” arXiv preprint arXiv:2006.03816, 2020.Search in Google Scholar
 K. Gregor and Y. LeCun, “Learning fast approximations of sparse coding,” in Proceedings of the 27th International Conference on International Conference on Machine Learning, 2010, pp. 399–406.Search in Google Scholar
 M. Gustafsson, S. Kurt, L. Jelinek, and M. Capek, “Upper bounds on absorption and scattering,” New J. Phys., vol. 22, pp. 073013–073038, 2020, https://doi.org/10.1088/1367-2630/ab83d3.Search in Google Scholar
 S. Molesky, P. Chao, and A. W. Rodriguez. “T-operator limits on electromagnetic scattering: bounds on extinguished, absorbed, and scattered power from arbitrary sources,” arXiv preprint arXiv:2001.11531, 2020, https://doi.org/10.1364/cleo_qels.2020.ff1f.6.Search in Google Scholar
© 2020 Zin Lin et al., published by De Gruyter, Berlin/Boston
This work is licensed under the Creative Commons Attribution 4.0 International License.