End-to-end nanophotonic inverse design for imaging and polarimetry

1 Introduction

Computational imaging and computer vision plays an increasingly important role in modern technology, ranging from simplest image de-noising routines to state-of-the-art object recognition, robotic vision and machine intelligence algorithms with widespread demand in defense, medical as well as emerging Internet-of-Things (IoT) industries. Traditional computer vision is exclusively driven by innovating the computational back end, and more recently, via deep learning and AI software. Little attention has been paid to the optical hardware at the front end beyond conventional lenses and diffraction gratings, in which light propagation is designed only by geometric optics [1], [2], [3]. The full potential of wave physics has yet to be exploited for imaging device design in conjunction with computational reconstruction, especially for extracting spectral and polarization information that is mostly discarded by geometric optics. The last decade has seen explosive advances in understanding and manipulation of light waves and light–matter interactions at the most profound level of nanomaterials, abetted by the development of efficient numerical modeling/design techniques as well as the advent of sophisticated nanofabrication machinery. Those capabilities have been exploited for purely optical designs, such as metasurface lenses, that involve little or no computational post-processing [4], [, 5]. In this paper, we demonstrate the potential of 3D nanophotonics in the development of next-generation computer-vision technologies, in which conventional optics hardware is replaced by exquisitely designed nanophotonic structures; in particular, we propose to bring deeper and richer physics to computer vision by optimally tailoring a nanophotonic front end for a computational-imaging back end using a fully coupled inverse-design process, offering ultracompact form factors as well as unprecedented capabilities for physical data acquisition and manipulation.

A conventional all-optical imaging system (Figure 1a) maps each point in a “target” space onto a separate sensor pixel, directly producing a faithful spatial image but requires bulky optics, and also typically fails to capture detailed spectral or polarization content without additional filters. In another extreme, a compact lens-free system (Figure 1b) would directly detect a blurry image of the target and attempt to solve the subsequent “inverse scattering” problem (target reconstruction by, e.g., least square fitting), which is typically very ill-conditioned and hence sensitive to noise [6], [7], [8], [9], [10]. In this paper, we introduce an end-to-end approach for inverse scattering (Figure 1c), in which a compact metaoptical structure is generated by large-scale inverse design of the full Maxwell equations coupled with signal processing for target recovery, both for conventional spatial imaging and for spectral polarimetry. First, we show that noise-tolerant subwavelength (0.2λ) far-field reconstruction of a collection of point sources is possible even with an ultracompact (2λ-thick) imaging device. Second, we demonstrate a “multidimensional” polarimeter that can resolve the full polarization states of multiple point sources at multiple frequencies. Specifically, we design metaoptical structures that generate well-conditioned (noise-robust) inverse-scattering problems, while exploiting a simple Tikhonov-regularization method (Section 3) to obtain subwavelength resolution without subwavelength focusing, or to enable multidimensional information extraction from a single-shot measurement. Accomplishing this requires that the optical “inverse” design problem, involving large-scale optimization over ≈10⁴ degrees of freedom, be coupled with the reconstruction algorithms (Section 2). That is, we perform “end-to-end” design in which the error L(ε, p) of the reconstructed targets is jointly minimized as a function of both the microstructure (ε) and the reconstruction parameters (p). Applying this approach to a two-dimensional (2D) example problem (Section 3), we obtain 0.22λ spatial resolution with a robust condition number (noise sensitivity) of only ≈10, an improvement of 10²–10³ over the condition numbers for lens-free or random (diffusing [1]) scattering structures. Applying similar techniques to the polarimetry problem (Section 4), we obtain a full-3D inverse-designed probe with a robust condition number of ≈6 that can reconstruct nine-parameter polarization-coherence matrices of two point sources emitting at two frequencies.

Figure 1:

Comparison of three imaging modalities.

(a) In traditional all-optical imaging, a bulky optical system focuses each point of the target on a different sensor pixel to directly produce a physical image; however, refractive or diffractive lenses are not designed for capturing the polarization or spectral content of the target. (b) In a compact lens-free system, the sensor directly records a blurry image while signal processing attempts to solve the resulting ill-posed (noise-sensitive) reconstruction problem; in this way, a spatial intensity profile of the target may be accurately reconstructed under sufficiently low noise conditions but polarization and spectral information cannot be retrieved. (c) In this work, we present an end-to-end inverse design approach, which optimizes a nanophotonic structure alongside the signal-processing algorithm leading to an ultracompact, noise-robust “all-in-one” system which may be used for not only imaging but also extracting polarization (small black arrows) and spectral information (color-coded).

Recent work in end-to-end computational imaging achieved improved image quality using regularized least-square image reconstruction in conjunction with scalar diffraction theory (rather than the full Maxwell equations) to design a phase plate (i.e., treated as locally uniform and neglecting multiple scattering) [2]. Flat-optics metalenses [4], [5], [11], in contrast, have utilized more complete wave optics theory ranging from locally periodic [12], [, 13] or overlapping [14] domain approximations to full Maxwell calculations [15], [, 16] coupled with optimization-based inverse design [17], [18], [19], exploiting local resonances and multiple scattering to achieve diffraction-limited focusing [20], [, 21]. Four-parameter Stokes imaging has also been demonstrated using a metaoptics polarization sorter in combination with a refractive lens [22]. However, these works specified the focal point and/or the desired wavefront a priori, even with more complex focal patterns chosen to facilitate subsequent computational processing [23], [24], [25], [26], [27], rather than performing a fully coupled end-to-end design. There is also a vast body of work on computational image reconstruction [28], [, 29], but decoupled from the lens design (taking the optics as an immutable input rather than as design parameters). In contrast, we couple the full Maxwell equations with the post-processing reconstruction during the design process (Section 2), so that an optimal wavefront is determined for each source to maximize reconstruction accuracy. Specifically, we demonstrate imaging with subwavelength resolution and multidimensional information extraction in ultracompact form factors, a feat not possible using previously reported end-to-end computational imaging. In order to perform this optimization, we employ standard adjoint techniques from photonic inverse design [17], [18], [19] combined with automatic-differentiation tools [30] to obtain the sensitivity to changes in structural parameters ε and reconstruction parameters p.

2 End-to-end framework

Figure 2 shows a schematic of our proposed framework which can be applied to any wave-scattering problem including imaging, spectroscopy, polarimetry or any combination thereof. Here, the goal is to reconstruct a target u in a preselected region of interest by computationally analyzing the captured image v on a sensor. In between the sensor and the target region, we place a scattering structure, aka a photonic probe, ε(r) to be designed, at a “working” distance d_u from the target and an “image” distance d_v from the sensor. The state of the target is specified by a vector u=[u1,…,un] containing spatial, spectral and/or polarization information, the details of which depend on the specific problem at hand. The sensor has m pixels with corresponding intensities (raw image) v=[v1,…,vm] given by the forward scattering model v=G(ϵ)u+η where G represents the solution of the Maxwell equations and η is an additive noise vector. For simplicity, we will consider zero-mean half-Gaussian white noise with nonzero standard deviation (η∼|N(0,σ)|; note that intensity noise η is non-negative) [2], although our method can be easily adapted to other noise models (such as Poisson/shot noise) by calibrating the camera beforehand. We consider a planar sensor, which is the most common configuration in imaging, but our framework can readily be extended to arbitrary sensor topologies. The linear kernel G is a m × n matrix whose columns are essentially point spread functions (PSF) [1], [, 31] computed from the underlying Maxwell equations given a structure ε(r).

Figure 2:

A schematic of the end-to-end inverse design framework. The target region of interest is characterized by an intensity vector u of length n, containing spatial, spectral and/or polarization information. The photonic probe has a dielectric profile ε(r) (to be determined via inverse design). The sensor, with m pixels, records the raw image v. u and v are related by the forward scattering model: v=G(ϵ)u+η, where G is a m × n matrix whose columns are extracted from the solution of the full Maxwell equations, and η is a noise vector (e.g., sensor noise). v is then fed into a signal processing algorithm parametrized by a vector p; the overall performance is evaluated by a loss function L (e.g., mean square deviation from the ground truth). The processing may involve any operations including matrix-vector multiplications, nonlinear kernels, integro-differential equations or artificial neural networks; in particular, we consider the inverse scattering problem of estimating u through regularized least-square minimization. End-to-end inverse design seeks optimal ε and p that optimizes the entire work-flow including both the forward model and the inverse problem; the gradients are obtained by backpropagation and adjoint methods.

The raw image v is fed into a signal-processing algorithm to approximately reconstruct u [6], in our case by a regularized least-square fit. That is, we find uˆ such that uˆ=arg minμ‖Gμ−v‖2+R(μ). Here, R is a regularization operator which serves to condition a typically ill-posed inverse problem; essentially, R incorporates any prior assumptions about u (such as smoothness or sparsity) which ensure that the inverse problem has a stable unique solution [6]. In particular, we choose a Tikhonov (L₂) regularization R(μ)=α‖μ‖2 where α > 0 is a regularization parameter to be determined, and uˆ has a closed-form solution uˆ=(GTG+αI)−1GTv [6]. The noise sensitivity of the reconstructed uˆ is characterized by the condition numberκ(G) of the matrix G, which is a dimensionless quantity ≥1 that is roughly proportional to the ratio of the ‖uˆ−u‖/‖u‖ relative error to the input noise ‖η‖/‖v‖ [32] (κ(G) can be computed as the ratio of the largest to smallest singular values of G). Many other variations are possible, such as L₁ “sparse” reconstruction [33] or artificial neural networks [34], [, 35]. As we discuss in Section 5, our approach extends easily to such techniques, even if the reconstruction problem does not have a closed-form solution or it involves a vast number of free parameters to be determined. In general, a reconstruction algorithm is characterized by a vector p of P parameters; in this example, p=[α] and P = 1.

The end-to-end inverse design seeks optimal choices of ε and p, which are tightly coupled by the end-to-end work-flow, in order to minimize the difference between the reconstructed uˆ against the ground truth u. Specifically, we define a loss function L(ϵ,p), here a mean-square error (MSE), such that L=〈‖u−uˆ‖2〉u,η where 〈⋯〉u,η denotes averaging (expected value) over many realizations of u and η. The formulation can be now written as:

Here, the PSF matrix G is extracted from the numerical solution of the Maxwell equations by any method.

In this paper, we consider the frequency-domain Maxwell equations with time-harmonic sources e−iωt [36]:

solved by a finite-difference frequency-domain (FDFD) method [37]. For each voxel in the target region, J is chosen as a point-source situated at the center of the voxel and the corresponding PSF is obtained by simulating the integrated electric field intensities |E|2 over the sensor plane. The optimization over ε, p require their gradients ∂L∂ϵ, ∂L∂p, which can be found by back-propagation through the signal-processing stage [38] and adjoint sensitivity analysis [18], [, 19] of the Maxwell equations. We numerically implement these gradients by coupling an open-source automatic-differentiation packages [30] with our own Maxwell adjoint solvers [13].

3 Imaging at subwavelength resolutions

To demonstrate the capability of our framework, we consider an imaging problem at subwavelength resolutions. We consider a 2D problem ϵ(x,z) (Figure 3) with y-polarized electric fields, so that the Maxwell equations are reduced to a scalar 2D Helmholtz equation. Specifically, we set du=5λ (compact, but in the far field) and dv=λ (where near-field effects may be relevant) (Figure 3a) where λ is the operating wavelength. Also, we discretize a 1D 2λ-wide target region into n = 10 point sources on an equispaced grid. Here, we assume incoherent illumination of the target region (as is common for imaging) so that only intensities need to be considered [31]. Therefore, an arbitrary target residing within the region is described by an intensity vector u=[u1,…,un] with a spatial resolution of Δxu=0.222λ (for targets at “infinity,” such as a photographic scene, the region of interest is an angular field of view and one can consider plane-wave sources instead of point sources). The probe and sensor have a width of 50λ and the sensor contains m = 50 pixels with a pixel size Δxv=λ. Although we have chosen these parameters for ease of demonstration, we note that this scenario is realizable using selective illumination [39], slit apertures, a high-speed scanning mode, and line sensors [40] to produce 2D or even 3D images over a wide field of view. More importantly, this system illustrates the essential ingredients of many important applications as discussed in Section 5.

Figure 3:

(a) Topology-optimized double-layer photonic probe (ϵ≈2.3) used to resolve a target of 10 point sources (not drawn to scale). The probe is 50λ wide and λ thick, and is made up of freeform variable-height geometry. Note the scale bar. (b) Mean square error (MSE, blue line) and inverse condition number κ−1 (red line) of the PSF matrix G, where G is a m × n matrix with m = 50, n = 10. κ−1 steadily increases to around 0.08(κ≈12), showing that the reconstruction becomes robust against noise. (c) Two example ground-truth targets are reconstructed under two different noise levels. A Gaussian noise η∼N(0,σ) is added to the image: v=Gu+η. The standard deviation σ is chosen as a percentage relative to the average PSF intensity G‾; for example 5% noise indicates σ=0.05G‾. (d) The third example represents an off-grid point source situated halfway between two grid nodes. The reconstruction intuitively produces a “nearest-neighbors” approximation.

Although we have set m > n (a nominally “over-determined” inverse problem), it is important to note that not any ε(r) will lead to a well-conditioned (noise-robust) PSF matrix G. It is ill-advised to use a randomly chosen ε profile and directly invert G because not every probe can resolve two point sources separated by a distance of 0.22λ and project measurably-distinct noise-tolerant PSFs onto a coarse-resolution sensor (Δxv≫Δxu) one wavelength away from the probe (small d_v leaves little room for conventional magnification). For example, we checked that a uniform ε leads to G with a condition number κ(G)≈104; even a disordered ε with rapidly varying fine features yields κ(G)≈1000. Both of these values represent orders of magnitude amplification of input noise in the output reconstruction, indicating that radical re-design of ε is required.

We show that our end-to-end framework can discover novel geometries ε(r) with greatly reduced κ(G), thereby rendering the inverse problem robust against noise. Here, the ε degrees of freedom are a set of freeform variable heights [41] at each pixel within a double-layer design region made up of a low-permittivity polymer material ε_polymer ≈ 2.3 in air (see Figure 3a). We have chosen these material settings because of rapidly maturing nanoscale 3D-printing technologies [42], [, 43] that would allow for exploration of such complex 3D geometries and are particularly suited for taking advantage of the full power of freeform topology optimization [14], [, 19].

We employ stochastic gradient descent [44] for optimizing ε and α over ≳104 iterations including random noise η; we found that α stays close to an initial choice of 0.5 while ε evolves considerably during the course of optimization. In practice, we found that it works just as well to fix α with zero noise (η = 0) as to vary α under many realizations of η (note that α is closely related to the noise variance σ² [45]). Figure 3a exhibits a double-layer optimized design in a 3D-printable polymer-matrix (for example, Nanoscribe IP-DIP [46], [, 47]); each layer has thickness λ/2 and the minimum feature size is ≈0.04λ, which may be challenging to fabricate at visible wavelengths but is feasible at longer wavelengths such as mid-wave and far-wave infrared or even millimeter waves [48], [, 49]. Figure 3b demonstrates that optimization rapidly improves both MSE (≈10⁻⁶) and κ(G)≈10. Figure 3c shows two example ground truth targets being reconstructed under different noise levels. A Gaussian noise η∼N(0,σ) is added to the image: v=Gu+η, where the standard deviation σ is chosen as a percentage relative to the average PSF intensity G‾; for example 5% noise indicates σ=0.05G‾. The low condition number ensures that the reconstruction errors are not amplified, remaining at ≈5% and 10%, respectively, for different σ′s.

Like almost all computational imaging [50], this device is designed to reconstruct targets situated at a fixed grid (as in any camera with discrete pixels), but the accuracy degrades gracefully for sources deviating from this grid. As shown in Figure 3d, even for the worst case of a light source that lies halfway between two grid points, the reconstructed image mostly divides the intensity between the two closest points (the error intensity at further points could be reduced if an L1 “sparsifying” reconstruction algorithm [50] were used instead of L2 minimization). This degradation is known as “gridding error” in the computational-imaging community [51]. A number of supplementary algorithms have been proposed to further improve the reconstruction for off-grid sources, including atomic-norm minimization [52] and coherence-inhibition schemes [53], which could be incorporated into end-to-end optical design if desired. Yet another way to improve the reconstruction scenario would be to minimize the error over a collection of offset grids (instead of a single fixed grid of training data).

Our results suggest that a low-index photonic microstructure with a highly complex geometry can faithfully reconstruct an image down to deeply subwavelength resolutions (albeit over a finite array of equispaced calibrated point sources), while maintaining a sufficiently high signal-to-noise ratio. From a fundamental-physics perspective, we note that even though the probe is close to the target, the former is clearly not in the near field of the latter (since du>λ/2), which means evanescent fields from the target have negligible amplitude at the probe. Instead, the subwavelength resolution is made possible by the ability of the computational probe to distinguish the subtle differences in spatial frequency components coming from adjacent point sources [54], [55], [56]. Therefore, our approach is unlike negative-index metamaterial superlenses [57], [, 58] or super-oscillatory lenses [59], which seek perfect point-to-point physical image formation via amplification of evanescent waves or subdiffraction-limit focal spots without the aid of computational reconstruction.

4 Spatial + spectral + polarization extraction

The ability to intimately manipulate the polarization states of light is a hallmark of vectorial Maxwell photonics [60], [, 61], which sets it apart from traditional geometric or diffractive optics. For example, in super-resolution microscopy with traditional lenses, the unresolved polarization state of a fluorescent molecule may affect localization accuracy and degrade the image recovery process, posing a nuisance in many imaging systems [62], [, 63]. Here, we show that end-to-end optimization can be used to design a nanophotonic polarimeter that can resolve the polarization coherence state of a fluorescent molecule, approaching theoretical upper bounds [64]. In particular, the instantaneous polarization state of a point-dipole source (e.g., a fluorescent molecule or a solid-state quantum emitter, such as a quantum dot or color center) is specified by the complex-valued 3-element polarization vector J=[Jx,Jy,Jz]. However, only the time-averaged intensities can be detected at optical frequencies, and the detectable polarization state of the dipole is described by a 3 × 3 coherence matrix [65] (equivalent to a matrix [66]):

Here, 〈⋅〉 denotes a time average. These nine coherence parameters associated are a natural generalization of the familiar four St parameters [65] that characterize the polarization coherence state of a plane wave.

The electric-field response E(r) of a nanophotonic structure ϵ(r) in the presence of an arbitrarily polarized point dipole can be completely specified by the three “basis” fields, u1, u2 and u3, derived from x-, y- and z-polarized test sources at the same location as the dipole: E(r,ϵ)=Jxu1(r,ϵ)+Jyu2(r,ϵ)+Jzu3(r,ϵ). The integrated time-averaged electric-field intensity at the ith pixel on the sensor is then given by:

Hence, it should be possible to extract the full nine-element coherence state from a linear inverse-scattering problem with an appropriately PSF kernel:

In fact, an ultracompact single-piece nanophotonic structure should be able to resolve not only polarization states but also extract spatial and spectral information simultaneously from a single measurement. As a proof of principle, we present in Figure 4 an “all-in-one super-probe” which can extract polarization coherence information from up to two spatial points and up to two spectral lines, in which case the target vector u to be reconstructed consists of 36 entries (9 polarization × 2 spatial × 2 spectral). The nanophotonic probe has an ultracompact volume of 10×10×4.5λ3, comprising four layers of variable-thickness polymer (refractive index ≈ 1.5) (Figure 4a). A sensor of 10 × 10 pixels (pixel⋅area =λ2) is located one λ away on one side of the probe; on the other side, the two dipoles are positioned one λ away from the probe and diagonally separated by 22λ away from each other; the dipoles may emit at λ, 1.1λ, or both. The kernel G is a 100 × 36 matrix, whose condition number has been optimized to κopt≈6.4 (Figure 4b). We note that, in general, more layers lead to better optimal results. For example, we obtained an optimal condition number of 22 when using a single layer, 12 when using two layers, and arriving at 6.5 when using the four layers presented here. Here, it is important to emphasize the pivotal role of optimization which greatly improves the noise sensitivity of reconstruction compared to any other nonoptimized structure. For example, we found that the kernel G of free space has κ≈500 and that of a correlated random mask (with the correlation length chosen to have similar λ − 2λ length-scales to the optimized design) has κ∼100; even modifying the optimized probe by simply discretizing the gray-scale thickness or erasing the thin morphological features may spoil κ by a factor anywhere between 1.3 and 10 (that is, any feature-size or binary-thickness constraints must be incorporated during the optimization process using standard techniques [18], not imposed after the fact). We also found that the power captured by the sensor in the presence of the optimized probe is, on average, ∼3× greater than for free space or a random mask, indicating that the probe also serves to focus light onto the sensor, further enhancing the noise tolerance. Figure 4c shows the reconstruction of two dipoles with two different polarization states and emitting at two wavelengths λ (blue) and 1.1λ (red); the dipoles are circularly polarized along different axes; their coherence matrices are reconstructed under different noise levels σ/G‾=0.05,0.20 with reconstruction errors of ≈5%, 10% respectively. The inverse-design optimization took ∼3 days and was performed using a time-domain Maxwell solver [67] distributed over 384 processors. We expect that further acceleration can be achieved by symmetry considerations [68], overlapping domain decompositions [14], broadband simulations and GPU-accelerated time-domain software.

Figure 4:

(a) Topology-optimized nanophotonic “all-in-one” probe for extracting spatial, spectral and polarization information. The probe can reconstruct the polarization coherence states of up to two point dipoles emitting at up to two spectral lines λ and 1.1λ. The dipoles must be positioned one λ away from the probe and diagonally separated by 22λ away from each other whereas the sensor is located one λ away on the other side of the probe. The probe consists of four layers of variable thickness polymer, whose gray-scale thickness profiles are also shown. The thickness ranges from 0 to 1.125λ (note the colorbar). (b) Mean square error (MSE, blue line) and inverse condition number κ−1 (red line) of the PSF kernel G, where G is a m × n matrix with m = 100, n = 36. κopt is found to be ≈6.4. (c) As an example, two dipoles with two different polarization states and emitting at two wavelengths λ (blue) and 1.1 λ (red) are reconstructed under different noise levels.

We emphasize that although our proof-of-concept nanophotonic probe is only 10λ×10λ in area, it can be used to resolve two points, two spectral lines and nine polarization coherence values (36 in total) with a robust condition number of 6.5. Increasing the number of points and/or spectral lines under the same probe area surely increase the optimal condition number. However, modern image sensors are known for relatively low noise statistics (≲1% noise), suggesting that many more points and/or spectral lines can be accurately reconstructed under a relatively high condition number ∼1000 associated with a small device area. At the same time, a larger probe area (wider field of view) admits even more spatial/spectral data without worsening the condition number; such a large-area probe may be realized by a periodic array of small probes or a domain-decomposition design of a large non-periodic probe [14]. On the other hand, in our paper, we have used the simplest reconstruction algorithm based on Tikhonov regularization; we expect that more advanced methods like compressed sensing or neural networks will allow us to extract even more information.

5 Summary and outlook

The key conclusion of our paper is that optical metastructures designed in conjunction with signal processing result in nonobvious light-scattering patterns that greatly ease the computational reconstruction. These results in devices far more compact compared to optics-only solutions while being robust to noise compared to computation-only designs. By solving the full (Maxwell) wave equations during the design process, our optimized structure can exploit all available wave physics (nonparaxial scattering, near-field interactions, resonances, dispersion, etc.). We illustrated this idea in the context of examples involving subwavelength imaging and for polarization-state reconstruction, but the same essential ideas can be readily applied to many other systems and computational processing techniques. In contrast to the many previous metasurface designs that have attempted to mimic and compete with traditional curved lenses [5], our scattered fields look nothing like a focal pattern and represent a functionality that is fundamentally distinct from that of conventional optics. Fullwave end-to-end optimization is particularly powerful for problems requiring spectral and polarization information that is discarded by geometric optics, such as polarimetry or hyperspectral imaging.

There are many other sensing/imaging problems that could benefit from this approach. Our designs in this paper closely resemble lab-on-a-chip microscopy. Related situations arise in ultracompact optofluidic medical sensors, where the probe and sensor must be tightly integrated, the sample is situated only a few wavelengths away from the sensor, and scanning is naturally provided by sample flow [69]. Inverse design can easily be applied to broadband problems, and we are especially excited about using it for computational spectroscopy [70], hyper-spectral imaging [71], [, 72], and other broadband sensing applications. Our framework can straightforwardly scale to larger 3D freeform structures [13], accommodate complex high-dimensional objects such as plenoptic light-fields [73], facilitate nonlinear mechanisms such as high dynamic-range imaging [74], and generalize to other challenging problems in physics such as nonlinear pulse shaping [75] and quantum coherence engineering [76], [, 77]. Optimization can easily incorporate constraints arising from different fabrication processes [18].

In this paper, our computational-reconstruction stage consisted of Tikhonov-regularized least-squares fitting, but end-to-end optical design can be coupled with many other computational techniques. In under-determined systems (many more targets than sensor pixels), a common approach is compressed sensing [50] for sparse targets, and techniques for end-to-end optimization with compressed sensing may include differentiable unrolled approximations [78] or epigraph formulations of basis pursuit denoising [33]. One could also employ deep learning (neural networks) for imaging and other cognitive tasks (e.g. passive ranging, object recognition); from the perspective of deep learning, the Maxwell solver is simply a specialized “network stage” that is differentiable (via adjoint methods) and hence composable with deep-learning software.

Apart from numerical and experimental endeavors, an important theoretical question is to identify the absolute limits to achievable dispersion (spatial or spectral) and condition numbers, given a desired resolution, a design volume V, and a dielectric contrast Δϵ. Recent approaches for shape-independent bounds to light–matter interactions [79], [80], [81] may be capable of answering these questions.