Abstract
Permutation matrices form an important computational building block frequently used in various fields including, e.g., communications, information security, and data processing. Optical implementation of permutation operators with relatively large number of input–output interconnections based on powerefficient, fast, and compact platforms is highly desirable. Here, we present diffractive optical networks engineered through deep learning to alloptically perform permutation operations that can scale to hundreds of thousands of interconnections between an input and an output fieldofview using passive transmissive layers that are individually structured at the wavelength scale. Our findings indicate that the capacity of the diffractive optical network in approximating a given permutation operation increases proportional to the number of diffractive layers and trainable transmission elements in the system. Such deeper diffractive network designs can pose practical challenges in terms of physical alignment and output diffraction efficiency of the system. We addressed these challenges by designing misalignment tolerant diffractive designs that can alloptically perform arbitrarily selected permutation operations, and experimentally demonstrated, for the first time, a diffractive permutation network that operates at THz part of the spectrum. Diffractive permutation networks might find various applications in, e.g., security, image encryption, and data processing, along with telecommunications; especially with the carrier frequencies in wireless communications approaching THzbands, the presented diffractive permutation networks can potentially serve as channel routing and interconnection panels in wireless networks.
1 Introduction
Permutation is one of the basic computational operations that has played a key role in numerous areas of engineering, e.g., computing [1], communications [2], encryption [3], data storage [4], remote sensing [5], and data processing [6]. Historically, electronic integrated circuits have been the established implementation medium for the permutation operation and other spacevariant linear transformations, while the research on optical computing has been mainly focused on using the Fourier transform approximation of thin lenses covering various applications in spaceinvariant transformations, e.g., convolution/correlation. On the other hand, as photonic switching devices and optical waveguide technology have become the mainstream communication tools on highend applications, e.g., fiber optic networks, supercomputers, and data centers, various approaches have been developed toward alloptical implementation of permutation operation and other spacevariant transformations based on, e.g., Mach–Zehnder interferometers [7], optical switches [8], photonic crystals [9], holographically recorded optical elements [10–12], offaxis lenslet arrays [13, 14], and arrays of periodic gratingmicrolens doublets [15]. The development of compact, lowpower optical permutation and interconnection devices can have significant impact on nextgeneration communication systems, e.g., 6G networks [16, 17], as well as other applications such as optical data storage [18] and image encrypting cameras [19–21].
With the widespread availability of highend graphics processing units (GPU) and the massively growing amounts of data, the past decade has witnessed major advances in deep learning, dominating the field of digital information processing for various engineering applications including, e.g., image segmentation and classification [22–25], natural language processing [26, 27], among others [28]. The statistical inference and function approximation capabilities of deep neural networks have also been exploited to produce stateoftheart performance for computational inverse problems in many imaging and sensing applications including, e.g., microscopy [29–35], quantitative phase imaging [36–41], and others [42–51]. Beyond these data processing tasks, deep learning can also provide taskspecific solutions to challenging inverse optical design problems for numerous applications including nanophotonics [52, 53], metamaterials [54], imaging, and sensing [55–60]. However, as the success and the applications of deep learning grow further, the electronic parallel computing platforms, e.g., GPUs, hosting deep neural networks and other machine learning algorithms have started to bring some limitations due to their power and bandwidthhungry operation. Moreover, the pace of the advances in computational capacity of the integrated circuits has fallen behind the exponential increase predicted by the Moore’s law [61]. These factors have fueled a tremendous amount of effort toward the development of optical machine learning schemes and other photonic computing devices that can partially reduce the computational burden on the electronics leading to powerefficient, massively parallel, highspeed machine learning systems. While most of the arising optical computing techniques rely on integrated photonic devices and systems compatible with the integrated waveguide technology [62–68], an alternative option toward exploiting photons for machine learning and the related computing tasks is to use complex modulation media and freespace light propagation and diffraction, which is particularly suitable for visual computing applications where the information is already carried by the optical waves (e.g., of a scene or target object) in free space [69].
Motivated by these pressing needs, Diffractive Deep Neural Networks (D^{2}NN) [70, 71] have emerged as an optical machine learning framework that utilizes deep learning to engineer light–matter interactions over a series of diffractive surfaces so that a desired statistical inference or deterministic computing task is realized alloptically as the light waves propagate through structured surfaces. According to this framework, the physical parameters determining the phase and/or amplitude of light over each independently controllable unit, i.e., the “diffractive neuron,” are updated through the stochastic gradient descent and errorbackpropagation algorithms based on a loss function tailored specifically for a given task. The weights of the connections between the diffractive neurons/features on successive layers, on the other hand, are dictated by the light diffraction in free space. Once the deep learningbased training, which is a onetime effort, is completed using a computer, the resulting transmissive/reflective diffractive layers are fabricated using, e.g., lithography or 3D printing, to physically form the diffractive network that completes a given inference or computational task at the speed of light using entirely passive modulation surfaces, offering a taskspecific, powerefficient, and fast optical machine learning platform.
Based on the D^{2}NN framework, here we demonstrate diffractive optical network designs that were trained to alloptically perform a given permutation operation between the optical intensities at the input and output fieldsofview, capable of handling hundreds of thousands of interconnections with diffraction limited resolution. We quantified the success of the presented diffractive optical networks in approximating a given, randomly selected permutation operation as a function of the number of diffractive neurons and transmissive layers used in the diffractive network design. We also laid the foundations toward practical implementations of diffractive permutation networks by investigating the impact of various physical error sources, e.g., lateral and axial misalignments and unwanted inplane layer rotations, on the quality/accuracy of the optically realized interconnection weights and the permutation operation. Moreover, we showed that the diffractive optical permutation networks can be trained to be resilient against possible misalignments as well as imperfections in the diffractive layer fabrication and assembly. Finally, we present the first proofofconcept experimental demonstration of diffractive permutation networks by alloptically achieving a permutation matrix of size 25 × 25, effectively realizing 625 interconnections based on 3Dprinted diffractive layers operating at the THz part of the spectrum.
The presented diffractive optical permutation networks can readily find applications in THzband communication systems serving as communication channel patch panels; furthermore, the underlying methods and design principles can be broadly extended to operate at other parts of the electromagnetic spectrum, including the visible and IR wavelengths, by scaling each diffractive feature size proportional to the wavelength of light [72] and can be used for image encryption in security cameras [21] and optical data storage systems, among other applications [73–76].
2 Results
Figure 1 illustrates the presented freespace permutation interconnect concept designed around diffractive optical networks using the D^{2}NN framework. As shown in Figure 1, the presented permutation interconnect scheme does not use any standard optical components such as lenses, and instead relies on a series of passive, phaseonly diffractive surfaces. Due to the passive nature of these layers, the diffractive optical network shown in Figure 1 does not consume any power except for the illumination light, providing a powerefficient permutation operation in a compact footprint of ∼600λ along the longitudinal axis, which could be further squeezed as needed. The 5layer diffractive optical permutation network design shown in Figure 1 was trained through supervised learning to alloptically realize a desired permutation operation, P , between the light intensity signals at the input and output FOVs, each with N _{ i } = N _{ o } = 400 (20 × 20) individual pixels of size 2λ × 2λ. Stated differently, this permutation operation controls in total of N _{ i } N _{ o } = 0.16 million optical intensity connections.
The supervised nature of the training process of the diffractive permutation network necessitates the use of a set of input–output signal pairs (examples that satisfy
P
) to compute a penalty term and the associated gradientbased updates with respect to the physical parameters of each diffractive neuron at every iteration. We set the optical signal of interest at the input and the output of the diffractive permutation scheme to be the light intensity, and as a result, the deep learningbased evolution of the presented diffractive permutation network shown in Figure 1 was driven based on the meansquared error (MSE) (see Methods Section) between the groundtruth and the alloptically synthesized output intensity patterns at a given iteration. Since this loss function acts only on the light intensity, the diffractive optical network can enjoy an output phasefreedom in synthesizing the corresponding transformed optical intensity patterns within the output fieldofview. The light intensity, I, is related to the complexvalued field, U, through a nonlinear operation,
After the training phase, we blindly tested each diffractive permutation network with test inputs that were never used during the training. Figure 2 illustrates 6 different randomly generated blind testing inputs along with the corresponding alloptically permuted output light intensities. In the first two randomly generated input patterns shown in Figure 2A, there is light coming out of all the input pixels/apertures at different levels of intensity. In the next two test input patterns shown in Figure 2A, on the other hand, nearly half of the input apertures have nonzero light intensity and finally, the last two test inputs contain only 10 and 11 pixels/apertures with light propagating toward the 1st layer of the diffractive permutation network. When tested on 20K randomly generated blind testing input intensity patterns with different sparsity levels, the 5layer diffractive permutation network shown in Figure 1 achieves 18.61 dB peaksignaltonoise ratio (PSNR), very well matching the ideal output response. For this randomly generated 20K testing data, Figure 2B also shows the distribution of PSNR as a function of the number of input pixels with nonzero light intensity, which reveals that the diffractive permutation network can permute relatively sparser inputs with a higher output image quality, achieving a PSNR of 25.82 dB.
In addition to randomly generated blind testing inputs, we further tested the diffractive permutation network shown in Figure 1 on 18.75K EMNIST images; note that this diffractive network was trained only using randomly generated input/output intensity patterns that satisfy P and the EMNIST images constitute not only blind testing set but also a significant deviation from the statistical distribution of the training images. The input fieldofview contains the permuted EMNIST images ( P ^{−1}) and the diffractive network inverts that permutation by alloptically performing P to recover the original images at the output plane (see Figure 2). The performance of the diffractive permutation network was quantified based on both PSNR and Structural Similarity Index Measure (SSIM). With N _{L} = 40K diffractive neurons on each layer, the 5layer diffractive permutation network shown in Figure 1 provides 19.18 dB and 0.85 for PSNR and SSIM metrics, respectively, demonstrating the generalization capability of the diffractive network to new types of input image data never seen during the training phase.
2.1 Impact of the number of diffractive layers and features
Next, we investigate the performance of diffractive permutation networks as a function of the number of diffractive neurons trained within the system. Toward this goal, in addition to the 5layer design shown in Figure 1 that has in total of N = 200K trainable diffractive features, we trained diffractive permutation networks consisting of 4, 3, and 2 modulation surfaces. The physical design parameters such as the size/width of the diffractive surfaces, layertolayer distances, and the extent of the input and output fieldsofview were kept identical to the ones in the 5layer network design. In other words, these new diffractive networks are designed and trained exactly in the same way as the previous 5layer network except they contain fewer diffractive layers. Figure 3A and B provides a quantitative comparison between these 4 diffractive permutation networks. While Figure 3A illustrates the mean PSNR and SSIM values achieved by each diffractive network for recovering EMNIST images, Figure 3B demonstrates the meansquarederror (MSE) between the desired permutation operation and its optically realized version (
The benefit of having N ≥ N _{ i } N _{ o } is further revealed in the increased generalization capability of the diffractive network as shown in Figure 3A. Since the EMNIST images were not used during the training, they represent completely new types of input intensity patterns for the presented diffractive optical networks. The SSIM (PSNR) values achieved by the 4layer diffractive network is found as 0.75 (16.41 dB) for the optical recovery of the permuted EMNIST images. These numbers are significantly higher compared to the performance of the 3layer and 2layer diffractive designs that can attain SSIM (PSNR) values of 0.46 (12.91 dB) and 0.30 (12.08 dB) for the same task; furthermore, the 5layer diffractive network design shown in Figure 1 outperforms the others by achieving 0.85 (19.18 dB) for the same performance metrics. The visual comparison of the input–output intensity patterns depicted in Figure 3C further supports this conclusion, where the noise due to the crosstalk between interconnection channels decreases proportional to the number of diffractive layers in the system.
2.2 Vaccination of diffractive permutation networks
With sufficiently large number of phaseonly diffractive neurons/features, the diffractive networks can optically realize permutation operations with, e.g., 0.16 million channels between the input and output pixels as shown in Figure 3. In fact, the number of interconnects that can be optically implemented through diffractive networks can go far beyond 0.16 million, given that the size/width of the diffractive surfaces and the number of diffractive layers can be increased further depending on the fabrication technology and the optomechanical constraints of the system. In addition, as the number of diffractive layers increases in a diffractive network architecture, their forward model can better generalize to new, unseen data as shown in Figure 3A.
On the other hand, deeper diffractive optical network designs are more susceptible to misalignments that are caused by the limitations of the optomechanical assembly and/or the fabrication technology that is utilized. It was shown that diffractive optical networks trained for statistical inference tasks, e.g., alloptical object classification, can be vaccinated against misalignments and other physical error sources, when the factors creating these nonideal conditions were incorporated into the training forward model, which was termed as vaccinatedD^{2}NNs or vD^{2}NNs [77]. Specifically, vD^{2}NN expands on the original D^{2}NN framework by modeling possible error sources as random variables and integrating them as part of the training model so that the deep learningbased evolution of the diffractive surfaces is guided toward solutions that are resilient to nonideal physical conditions and/or fabrication errors. Toward practical applications of diffractive permutation networks, we quantified the impact of optomechanical errors and applied the vD^{2}NN framework to devise robust solutions that can achieve a given interconnect operation despite fabrication tolerances.
In our numerical study depicted in Figure 4, we considered 4 different misalignment components representing the 3D misalignment vector of the lth diffractive layer,
To better highlight the impact of these misalignments and demonstrate the efficacy of the vD^{2}NN framework, we trained a new nonvaccinated, i.e., v
_{tr} = 0, diffractive permutation network that can alloptically realize a given permutation matrix,
P
, representing 10K intensity interconnections between 100 input and 100 output pixels of size 4λ × 4λ. The errorfree training model of this diffractive network with v
_{tr} = 0 implicitly assumes that when the resulting diffractive network is fabricated, the system conditions will exactly match the ideal settings regarding the 3D locations of the layers and their inplane orientations. With an architecture identical to the one shown in Figure 1, containing N = 200K ≫ N
_{
i
}
N
_{
o
} diffractive neurons, this diffractive network can alloptically approximate the permutation matrix,
P
, with an MSE of 1.45 × 10^{−6} in the absence of any misalignment errors, i.e., v
_{test} = 0 (see the green curve in Figure 4B). However, when there is some discrepancy between the training and testing conditions, i.e., v
_{test} > 0, the optically implemented forward transformation,
Unlike the nonvaccinated design, the vaccinated diffractive permutation networks can maintain their approximation capacity and accuracy under erroneous testing conditions as shown in Figure 4A and B. For instance, the SSIM value of 0.49 attained by the nonvaccinated diffractive network for the misalignment uncertainty set by v
_{test} = 0.125 increases to 0.88 in the case of a diffractive permutation network trained with v
_{tr} = 0.25 (red curve in Figure 4A). The difference between the image recovery performances of the vaccinated and the nonvaccinated diffractive network designs increases further as the misalignment levels increase during the blind testing. While the nonvaccinated diffractive network can only achieve SSIM values of 0.3 and 0.19 at v
_{test} = 0.25 and v
_{test} = 0.375, respectively, the output images synthesized by the vaccinated design (v
_{tr} = 0.25) reveals SSIM values of 0.8 at v
_{test} = 0.25 and 0.64 at v
_{test} = 0.375 (see Figure 4D). A similar conclusion can also be drawn from Figure 4B, demonstrating the MSE values between the desired permutation matrix,
P
, and its optically realized counterpart,
The compromise for this misalignment robustness comes in the form of a reduction in the peak performance. While the nonvaccinated diffractive network can solely focus on realizing the given permutation operation with the highest quality and approximation accuracy, the vaccinated diffractive network designs partially allocate their degreesoffreedom to building up resilience against physical misalignments. For example, while the peak SSIM achieved by the nonvaccinated diffractive network is 0.99, it is 0.88 for the diffractive permutation network vaccinated with v _{tr} = 0.25. The key difference, on the other hand, is that the better performance of the nonvaccinated diffractive network is sensitive to the physical implementation errors, while the vaccinated diffractive permutation networks can realize the desired input–output interconnects over a larger range of fabrication errors or tolerances. A comparison between the diffractive layer patterns of the nonvaccinated and vaccinated diffractive permutation networks shown in Figure 4C and D, respectively, also reveals that the vaccination strategy results in smoother light modulation patterns; in other words, the material thickness values over the neighboring diffractive neurons partially lose their independence and become correlated, causing a reduction in the number of independent degreesoffreedom in the system.
2.3 Experimental demonstration of a diffractive permutation network
To experimentally demonstrate the success of the presented diffractive permutation interconnects, we designed a 3layer diffractive permutation network achieving the desired (randomly generated) intensity shuffling operation with N _{ i } = N _{ o } = 5 × 5, optically synthesizing 625 connections between the input and output FOVs; this network was designed to operate at 0.4 THz, corresponding to ∼0.75 mm in wavelength. During the training, the forward model of this diffractive permutation network was vaccinated with v _{tr} = 0.5 against the 4 error sources as detailed in Section 2.2 including the 3D location of each diffractive layer and the inplane rotation angle around the optical axis. In addition to these misalignment components, we also vaccinated this diffractive network model against unwanted material thickness variations that could arise due to the limited lateral and axial resolution of our 3D printer (see the Methods Section for details). To compensate for the reduction in the degreesoffreedom due to the vaccination scheme, the number of phaseonly diffractive features in the permutation network was selected to be N _{ L } = 10K diffractive neurons per layer. Therefore, each diffractive layer shown in Figure 5A contains 100 × 100 phaseonly diffractive neurons of size ∼0.67λ × 0.67λ. Compared to the diffractive surfaces shown in Figures 1 –4, the layers of our experimental system were set to be 2times smaller in both the x and y directions to keep the layertolayer distances smaller while maintaining the level of optical connectivity between the successive diffractive surfaces (see Figure 5B). Figure 5C and D illustrates the 3D printed diffractive permutation network and the schematic of our experimental setup (see the Methods Section for details).
Figure 6A illustrates the targeted 25 × 25 permutation matrix (
P
) that is randomly generated and the numerically predicted
3 Discussion
Beyond optomechanical error sources and fabrication tolerances, another factor that might potentially hinder the utilization of diffractive permutation networks in practical applications is the output diffraction efficiency. For instance, the diffraction efficiency of the 5layer network shown in Figure 1 is ∼0.004% which might be very low for some applications. On the other hand, this can be significantly increased by using an additional loss term, penalizing the poor diffraction efficiency of the network (see the Methods Section for details). Supplementary Figure S3 demonstrates 5layer diffractive networks that are designed to optically realize 0.16 million interconnections between the input and output FOVs with increased diffraction efficiencies, and compares their performance in terms of SSIM values achieved. The training of these diffractive network models is based on a loss function in the form of a linear combination of two different penalty terms,
Also note that, although we solely focused on diffractive network designs composed of dielectric optical modulation surfaces, in principle, some of these layers can be replaced with metasurfaces/metamaterials. While the use of can provide some additional degrees of freedom, including, for example, the engineering of dispersion, there are some challenges to overcome in realizing metamaterialbased diffractive networks. In the presence of fabrication errors and imperfections, the scattered light fields might deviate from the predictions of the numerical forward model. These nonideal waves generated by a metasurface would then excite unwanted diffraction modes over the subsequent layers generating an “avalanche” within the diffractive volume, accumulating substantial field errors, especially for deeper network designs with L ≥ 2. In addition, the physical models of phase and/or amplitude modulation of metaatoms are, in general, valid for waves covering a relatively small numerical aperture (NA). As a result, a high NA diffractive network design that utilizes the entire bandwidth of the propagation medium (NA = 1, for air) would be challenging, as the modulation response of the metaunits might deviate from their ideal small angle responses, introducing errors to the forward model. Although it is possible to restrict a diffractive network design to work with a lower NA, it would increase the overall footprint of the system and reduce the spacebandwidth product that can be processed by the diffractive network.
These challenges, in general, are negligible for dielectric diffractive networks composed of λ/2 features on a substrate, as also highlighted by the close match between the numerically predicted images and their experimentally measured counterparts shown in Figure 6 and our former work [21, 70, 74, 76, 77]. In our optical forward model, the diffractive layers are assumed to be thin modulation surfaces, i.e., there is only a single scattering event converting an incident wave field to an outgoing one after each diffractive layer. Practically, though, there are additional scattered fields that are ignored in our model, especially when there is a substantial material thickness variation between adjacent pixels. However, in our designs, we do not observe a sharp material thickness transition between neighboring pixels. This is mainly due to the nature of our training process. Specifically, the presented diffractive networks are trained through errorbackpropagation, which computes the variable updates by taking the gradient of the loss function with respect to the material thickness over each diffractive unit. In such a process, it is highly unlikely that the gradients of two adjacent pixels (λ/2 apart from each other) deviate significantly from each other, which effectively causes a smoothening effect on the diffractive surface height profiles as they are being optimized through deep learning. This smoothening behavior is even more pronounced in the vaccinated diffractive network designs due to the random lateral translation of the diffractive layers as part of the training forward model. Therefore, the impact of side scattering or field shadowing due to height discontinuities across a given diffractive layer design is negligible. In addition, the backreflected waves can also be ignored, as these are, in general, weak processes unless they are specifically enhanced using, e.g., metamaterials or other special structures. Therefore, the optical forward model of dielectric diffractive networks can be accurately represented within the scalar diffraction theory without needing vectorial modeling of light fields or considering weaker multireflections. Finally, the evanescent waves and the vectorial fields associated with them can be entirely ignored since each successive diffractive layer is axially positioned >λ away from the previous layer.
In summary, we showed that the diffractive networks can optically implement intensity permutation operations between their input and output apertures based on phaseonly light modulation surfaces with N ≥ N
_{
i
}
N
_{
o
} diffractive neurons. Due to the nonlinear nature of the intensity operation, it is crucial to use training input intensity patterns with different levels of sparsity to prevent any type of dataspecific overfitting during the training phase. Diffractive permutation networks with N > N
_{
i
}
N
_{
o
} demonstrate increased generalization capability, synthesizing more accurate outputs with
4 Methods
4.1 Experimental setup
According to the schematic diagram of our experimental setup shown in Figure 5D, the THz wave incident on the input FOV of the diffractive network was generated using a horn antenna attached to the source WR2.2 modulator amplifier/multiplier chain (AMC) from Virginia Diode Inc. (VDI). A 10 dBm RF input signal at 11.111 GHz (f _{RF1}) at the input of the AMC was multiplied 36 times to generate a continuouswave (CW) radiation at 0.4 THz, corresponding to ∼0.75 mm in wavelength. The output of the AMC was modulated with 1 kHz square wave to resolve lownoise output data through lockin detection. Since we did not use any collimating optics in our setup, the distance between the input plane of the 3Dprinted diffractive optical network and the exit aperture of the horn antenna was set to be ∼60 cm approximating a uniform plane wave over the 40λ × 40λ input FOV. At the output plane of the diffractive optical network, the diffracted THz light was collected using a singlepixel Mixer/AMC from Virginia Diode Inc. (VDI). During the measurements, the detector received a 10 dBm sinusoidal signal at 11.083 GHz serving as a local oscillator for mixing, and the downconverted signal was at 1 GHz. The 40λ × 40λ output FOV was scanned by placing the singlepixel detector on an XY stage that was built by combining two linear motorized stages (Thorlabs NRT100). At each scan location, the downconverted signal coming from the singlepixel detector was fed to lownoise amplifiers (MiniCircuits ZRL1150LN+) with a gain of 80 dBm and a 1 GHz (+/−10 MHz) bandpass filter (KL Electronics 3C401000/T10O/O) that erases the noise components coming from unwanted frequency bands. Following the amplification and filtering, the measured signal passed through a tunable attenuator (HP 8495B) and a lownoise power detector (MiniCircuits ZX4760). Finally, the output voltage value was generated by a lockin amplifier (Stanford Research SR830). The modulation signal was used as the reference signal for the lockin amplifier and accordingly, we performed a calibration to convert the lockin amplifier readings at each scan location to linear scale. During our experiments, the scanning step size at the output plane was set to be ∼λ in x and y directions. The smallest pixel of the experimentally targeted permutation grid, i.e., the desired resolution of the diffractive permutation operation was taken as 8λ × 8λ during the training, corresponding to 5 × 5 discrete input and output signals. Therefore, the output signal measured for each input object was integrated over a region of 8λ × 8λ per pixel, resulting in the measured images shown in Figure 6B.
A 3D printer, Objet30 Pro, from Stratasys Ltd., was used to fabricate the layers of the diffractive permutation network shown in Figure 5C as well as the layer holders. The active modulation area of our 3D printed diffractive layers was 5 × 5 cm (∼66.66λ × ∼66.66λ) containing 100 × 100, i.e., 10K, diffractive neurons. These modulation surfaces were printed as insets surrounded by a uniform slab of printing material with a thickness of 2.5 mm and the total size of each printed layer including these uniform regions was 6.2 × 6.2 cm. Following the 3D printing, these additional surrounding regions were coated with aluminum to block the propagation of the light over these areas minimizing the contamination of the output signal with unwanted scattered light.
4.2 Training forward model of diffractive permutation networks
4.2.1 Optical forward model
The material thickness, h, was selected as the physical parameter controlling the complexvalued transmittance values of the diffractive layers of our design. Based on the complexvalued refractive index of the diffractive material, τ = n + jκ, the corresponding transmission coefficient of a diffractive neuron located on the lth layer at a coordinate of (x _{ q }, y _{ q }, z _{ l }) is defined as,
where n _{ m } = 1 denotes the refractive index of the propagation medium (air) between the layers. The real and imaginary parts of the 3D printing material were measured experimentally using a THz spectroscopy system, and they were revealed as n = 1.7227 and κ = 0.031 at 0.4 THz.
The optical forward model of the presented diffractive networks relies on the Rayleigh–Sommerfeld theory of scalar diffraction to represent the propagation of light waves between the successive layers. According to this diffraction formulation, the free space can be interpreted as a linear, shiftinvariant operator with the impulse response,
where
The parameter
is generated. The outlined successive modulation and secondary wave generation processes occur until the waves propagating through the diffractive network reach to the output plane. Although, the forward optical model described by Eqs. (1)–(4) is given over a continuous 3D coordinate system, during our deep learningbased training of the presented diffractive permutation networks, all the wave fields and the modulation surfaces were represented based on their discrete counterparts with a spatial sampling rate of ∼0.67λ on both x and y axes, that is also equal to the size of a diffractive neuron.
4.2.2 Physical architecture of the diffractive permutation networks and training loss functions
The size of the output and input FOVs of the presented diffractive permutation networks were both set to be 40λ × 40λ, defining a unit magnification optical permutation operation. Note that the unit magnification is not a necessary condition for the success of the forward operation of diffractive optical interconnects but rather a design choice. Without loss of generality, the output FOV can be defined centered around the origin,
During the deep learningbased training of all of these diffractive permutation networks, the wave fields and the propagation functions depicted in Eqs. (2)–(4) were sampled at a rate of ∼0.67λ that is also equal to the size of the smallest diffractive units on the modulation surfaces constituting the presented diffractive networks. At this spatial sampling rate, the input and output intensity patterns were represented as 2D discrete vectors of size 60 × 60 denoted by
While the light fields, the diffractive layers, and the impulse response of the free space were all sampled at a rate of ∼0.67λ, the spatial grid/pixel size of a given desired permutation operation was set to be larger. Specifically, the permutation pixel size was taken as 2λ × 2λ for the diffractive networks shown in Figures 1 –3. On the other hand, the input and output pixel size was chosen as 4λ × 4λ for the vaccinated and nonvaccinated diffractive permutation networks shown in Figure 4, and finally, the pixel size was set to be 8λ × 8λ for the fabricated diffractive permutation network model depicted in Figure 5.
To train the presented diffractive permutation networks, a structural loss function,
In Eq. (5),
Note that the presented diffractive permutation networks preserve the relative intensity levels. Stated differently, our training forward model aims to keep the intensity levels over the output and input pixels the same up to a single multiplicative constant, σ.
To improve the diffraction efficiency of diffractive permutation networks, we defined another loss function,
The diffractive permutation networks presented in Figures 1
–4 were trained based on
The supervised deep learningbased training of the presented diffractive permutation networks evaluates the loss function
where h _{ m } and h _{ b } denote the maximum modulation thickness and the base material thickness, respectively. For all the diffractive permutation networks presented in Figures 1 –4 and Supplementary Figure S3, h _{ m } was taken as 2λ. In the design of the 3Dprinted diffractive permutation network, however, h _{ m } was set to be 1.66λ to restrict the material thickness contrast between the neighboring diffractive features. The value of h _{ b } was taken as 0.66λ for all the presented designs including the fabricated diffractive network.
4.2.3 Computation of
P
D
2
N
N
, optical transformation errors and performance quality metrics
For a given diffractive permutation network design trained to optically implement a permutation matrix
P
of size N
_{
i
} × N
_{
o
}, there are two different ways to compute the permutation operation predicted by its numerical forward model. The first way is to propagate N different randomly generated independent inputs with N ≥ N
_{
i
}
N
_{
o
} and solve a linear system of equations for revealing the entries of
The alloptical transformation error,
Denoting the lexicographically ordered vectorized version of a 2D input intensity pattern with I _{in}[s], the ground truth output intensity can be found by P I _{in}[s]. The PSNR between this groundtruth vector and the output vector synthesized by the forward optical operation of a given, trained diffractive network, I _{out}[s], can be calculated as,
where σ is the multiplicative constant defined in Eq. (6). The SSIM values were calculated based on the builtin function in TensorFlow, i.e., tf.image.ssim, where the two inputs were 2D versions of P I _{in}[s] and I _{out}[s], representing the groundtruth image and the permuted, alloptical output signal, respectively. All the parameters of tf.image.ssim were taken equal to default values, except that the size of the Gaussian filter was set to be 5 × 5, instead of 11 × 11, and the width of the Gaussian filter was set to be 0.75.
4.2.4 Vaccination of diffractive permutation networks
vD^{2}NN framework aims to design diffractive optical networks that are resilient against physical error sources, e.g., misalignments, by modeling these factors as random variables and incorporating them into the forward training model. In the training forward model of the vaccinated diffractive networks shown in Figure 4, 4 physical error components were modeled representing the misalignment of each diffractive layer with respect to their ideal location and orientation/angle. The first 3 components represent the statistical variations in the location of each diffractive layer in 3D space. Let the ideal location of a diffractive layer, l, be denoted by the vector
During the training, for each batch of input images, the 3D displacement vector D ^{ l , i } is updated and accordingly, the location of the layer l is set to be X ^{ l , i } = X ^{ l } + D ^{ l , i }, building up robustness to physical misalignments.
Beyond the displacement of diffractive layers, the physical forward model of a diffractive network is also susceptible to variations in the orientation of the diffractive layers. Ideally, one should include all 3 rotational components, yaw, pitch, and roll; however, in this study, we only considered the yaw component since in our experimental systems, the pitch and the roll can be controlled with a high precision. The random angle representing the rotation of a diffractive layer l around the optical axis was defined as
For the design of the experimentally validated diffractive permutation network, on top of these 4 optomechanical error components (with v = 0.5), we also modeled fabrication errors in the form of statistical variations of the material thickness over each diffractive neuron (h). Hence, at a given iteration, i, the material thickness values over each diffractive unit
4.2.5 Training details
The deep learningbased training of the diffractive permutation networks was implemented using Python (v3.6.5) and TensorFlow (v1.15.0, Google Inc.). The backpropagation updates were calculated using the Adam optimizer [78], and its parameters were taken as the default values in TensorFlow and kept identical in each model. The learning rates of the diffractive optical networks were set to be 0.001. The training batch size was taken as 75 during the deep learningbased training of the presented diffractive permutation networks. For the training of the diffractive permutation networks, we generated ∼4.7 million random intensity patterns, providing us 93,750 iterations/errorbackpropagation updates per epoch. The training of a 5layer diffractive permutation network with 40K diffractive neurons per layer for 5 epochs using this randomly created training dataset takes approximately 4 days using a computer with a GeForce GTX 1080 Ti Graphical Processing Unit (GPU, Nvidia Inc.) and Intel^{®} Core™ i78700 Central Processing Unit (CPU, Intel Inc.) with 64 GB of RAM, running Windows 10 operating system (Microsoft).
Funding source: Air Force Office of Scientific Research
Award Identifier / Grant number: FA95502110324
Acknowledgments
The Ozcan Research Group at UCLA acknowledges the support of AFOSR, Materials with Extreme Properties Program funding (FA95502110324).

Author contributions: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.

Research funding: This work was supported by Air Force Office of Scientific Research (grant number FA95502110324).

Conflict of interest statement: The authors declare no conflicts of interest regarding this article.
References
[1] N. Ishikawa, S. Sugiura, and L. Hanzo, “50 Years of permutation, spatial and index modulation: from classic RF to visible light communications and data storage,” IEEE Commun. Surv. Tutor., vol. 20, pp. 1905–1938, 2018. https://doi.org/10.1109/comst.2018.2815642.Search in Google Scholar
[2] S. Ishimura and K. Kikuchi, “Multidimensional permutationmodulation format for coherent optical communications,” Opt. Express, vol. 23, p. 15587, 2015. https://doi.org/10.1364/oe.23.015587.Search in Google Scholar
[3] H. Huang, X. He, Y. Xiang, W. Wen, and Y. Zhang, “A compressiondiffusionpermutation strategy for securing image,” Signal Process., vol. 150, pp. 183–190, 2018. https://doi.org/10.1016/j.sigpro.2018.04.014.Search in Google Scholar
[4] A. Jiang, R. Mateescu, M. Schwartz, and J. Bruck, “Rank modulation for flash memories,” IEEE Trans. Inform. Theory, vol. 55, pp. 2659–2673, 2009. https://doi.org/10.1109/tit.2009.2018336.Search in Google Scholar
[5] X. Huang, G. Ye, H. Chai, and O. Xie, “Compression and encryption for remote sensing image using chaotic system,” Secur. Commun. Netw., vol. 8, pp. 3659–3666, 2015. https://doi.org/10.1002/sec.1289.Search in Google Scholar
[6] T. K. Moon and W. C. Sterling, Mathematical Methods and Algorithms for Signal Processing, Upper Saddle River, NJ, Prentice Hall, 2000.Search in Google Scholar
[7] J. Carolan, C. Harrold, C. Sparrow, et al.., “Universal linear optics,” Science, vol. 349, pp. 711–716, 2015. https://doi.org/10.1126/science.aab3642.Search in Google Scholar PubMed
[8] R. A. Spanke and V. E. Benes, “Nstage planar optical permutation network,” Appl. Opt., vol. 26, p. 1226, 1987. https://doi.org/10.1364/ao.26.001226.Search in Google Scholar PubMed
[9] M. Djavid, M. H. T. Dastjerdi, M. R. Philip, et al.., “Photonic crystalbased permutation switch for optical networks,” Photonic Netw. Commun., vol. 35, pp. 90–96, 2018. https://doi.org/10.1007/s1110701707197.Search in Google Scholar
[10] H. Kobolla, F. Sauer, and R. Volkel, Holographic Tandem Arrays, G. M. Morris, Ed., Bellingham, Washington, USA, SPIE, 1989, p. 146.10.1117/12.961677Search in Google Scholar
[11] B. Robertson, E. J. Restall, M. R. Taghizadeh, and A. C. Walker, “Spacevariant holographic optical elements in dichromated gelatin,” Appl. Opt., vol. 30, p. 2368, 1991. https://doi.org/10.1364/ao.30.002368.Search in Google Scholar PubMed
[12] H. Kobolla, J. T. Sheridan, E. Gluch, et al.., “Holographic 2D mixed polarization deflection elements,” J. Mod. Opt., vol. 40, pp. 613–624, 1993. https://doi.org/10.1080/09500349314550671.Search in Google Scholar
[13] M. C. Hutley, P. Savander, and M. Schrader, “The use of microlenses for making spatially variant optical interconnections,” Pure Appl. Opt., vol. 1, pp. 337–346, 1992. https://doi.org/10.1088/09639659/1/6/006.Search in Google Scholar
[14] J. Jahns and W. Däschner, “Optical cyclic shifter using diffractive lenslet arrays,” Opt. Commun., vol. 79, pp. 407–410, 1990. https://doi.org/10.1016/00304018(90)904726.Search in Google Scholar
[15] F. Sauer, J. Jahns, C. R. Nijander, A. Y. Feldblum, and W. P. Townsend, “Refractivediffractive microoptics for permutation interconnects,” Opt. Eng., vol. 33, p. 1550, 1994. https://doi.org/10.1117/12.166528.Search in Google Scholar
[16] A. Tarable, F. Malandrino, L. Dossi, R. Nebuloni, G. Virone, and A. Nordio, “Metasurface optimization in 6G subTHz communications,” in 2010 IEEE International Conference on Communications Workshops (ICC Workshops), IEEE, 2020, pp. 1–6.10.1109/ICCWorkshops49005.2020.9145267Search in Google Scholar
[17] X. Xu, Q. Chen, X. Mu, Y. Liu, and H. Jiang, “Graphembedded multiagent learning for smart reconfigurable THz MIMONOMA networks,” IEEE J. Sel. Areas Commun., vol. 40, pp. 259–275, 2022. https://doi.org/10.1109/jsac.2021.3126079.Search in Google Scholar
[18] B. M. King and M. A. Neifeld, “Sparse modulation coding for increased capacity in volume holographic storage,” Appl. Opt., vol. 39, p. 6681, 2000. https://doi.org/10.1364/ao.39.006681.Search in Google Scholar PubMed
[19] R. Enayatifar, A. H. Abdullah, I. F. Isnin, A. Altameem, and M. Lee, “Image encryption using a synchronous permutationdiffusion technique,” Opt. Lasers. Eng., vol. 90, pp. 146–154, 2017. https://doi.org/10.1016/j.optlaseng.2016.10.006.Search in Google Scholar
[20] V. Patidar, N. K. Pareek, G. Purohit, and K. K. Sud, “A robust and secure chaotic standard map based pseudorandom permutationsubstitution scheme for image encryption,” Opt. Commun., vol. 284, pp. 4331–4339, 2011. https://doi.org/10.1016/j.optcom.2011.05.028.Search in Google Scholar
[21] B. Bai, Y. Luo, T. Gan, et al.., “To image, or not to image: classspecific diffractive cameras with alloptical erasure of undesired objects,” eLight, vol. 2, no. 1, p. 14, 2022. https://doi.org/10.1186/s43593022000213.Search in Google Scholar
[22] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” Commun. ACM, vol. 60, pp. 84–90, 2017. https://doi.org/10.1145/3065386.Search in Google Scholar
[23] Y. LeCun, L. Bottou, Y. Bengio, and P. Ha, “Gradientbased learning applied to document recognition,” Proc. IEEE, vol. 86, pp. 2278–2374, 1998. https://doi.org/10.1109/5.726791.Search in Google Scholar
[24] L. C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 40, pp. 834–848, 2018. https://doi.org/10.1109/tpami.2017.2699184.Search in Google Scholar PubMed
[25] O. Ronneberger, P. Fischer, and T. Brox, “Unet: convolutional networks for biomedical image segmentation,” in Medical Image Computing and ComputerAssisted Intervention – MICCAI 2015, vol. 9351, N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi, Eds., Springer International Publishing, 2015, pp. 234–241.10.1007/9783319245744_28Search in Google Scholar
[26] R. Collobert and J. Weston, “A unified architecture for natural language processing: deep neural networks with multitask learning,” in Proceedings of the 25th International Conference on Machine Learning – ICML’08, ACM Press, 2008, pp. 160–167.10.1145/1390156.1390177Search in Google Scholar
[27] Y. Goldberg, “Neural network methods for natural language processing,” Synth. Lect. Hum. Lang. Technol., vol. 10, pp. 1–309, 2017. https://doi.org/10.2200/s00762ed1v01y201703hlt037.Search in Google Scholar
[28] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, Cambridge, Massachusetts, The MIT Press, 2016.Search in Google Scholar
[29] Y. Rivenson, Z. Göröcs, H. Günaydin, Y. Zhang, H. Wang, and A. Ozcan, “Deep learning microscopy,” Optica, vol. 4, p. 1437, 2017. https://doi.org/10.1364/optica.4.001437.Search in Google Scholar
[30] Y. Rivenson, H. C. Koydemir, H. Wang, et al.., “Deep learning enhanced mobilephone microscopy,” ACS Photonics, vol. 5, pp. 2354–2364, 2018. https://doi.org/10.1021/acsphotonics.8b00146.Search in Google Scholar
[31] K. de Haan, Z. S. Ballard, Y. Rivenson, Y. Wu, and A. Ozcan, “Resolution enhancement in scanning electron microscopy using deep learning,” Sci. Rep., vol. 9, p. 12050, 2019. https://doi.org/10.1038/s41598019484442.Search in Google Scholar PubMed PubMed Central
[32] E. Nehme, L. E. Weiss, T. Michaeli, and Y. Shechtman, “DeepSTORM: superresolution singlemolecule microscopy by deep learning,” Optica, vol. 5, p. 458, 2018. https://doi.org/10.1364/optica.5.000458.Search in Google Scholar
[33] E. Nehme, D. Freedman, R. Gordon, et al.., “DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning,” Nat. Methods, vol. 17, pp. 734–740, 2020. https://doi.org/10.1038/s4159202008535.Search in Google Scholar PubMed PubMed Central
[34] T. Nguyen, Y. Xue, Y. Li, L. Tian, and G. Nehmetallah, “Deep learning approach for Fourier ptychography microscopy,” Opt. Express, vol. 26, pp. 26470–26484, 2018. https://doi.org/10.1364/oe.26.026470.Search in Google Scholar PubMed
[35] Y. Wu, Y. Rivenson, H. Wang, et al.., “Threedimensional virtual refocusing of fluorescence microscopy images using deep learning,” Nat. Methods, vol. 16, pp. 1323–1331, 2019. https://doi.org/10.1038/s4159201906225.Search in Google Scholar PubMed
[36] Y. Rivenson, Y. Zhang, H. Günaydın, D. Teng, and A. Ozcan, “Phase recovery and holographic image reconstruction using deep learning in neural networks,” Light Sci. Appl., vol. 7, p. 17141, 2018. https://doi.org/10.1038/lsa.2017.141.Search in Google Scholar PubMed PubMed Central
[37] Y. Wu, Y. Rivenson, Y. Zhang, et al.., “Extended depthoffield in holographic imaging using deeplearningbased autofocusing and phase recovery,” Optica, vol. 5, p. 704, 2018. https://doi.org/10.1364/optica.5.000704.Search in Google Scholar
[38] Y. Rivenson, Y. Wu, and A. Ozcan, “Deep learning in holography and coherent imaging,” Light Sci. Appl., vol. 8, pp. 1–8, 2019. https://doi.org/10.1038/s4137701901960.Search in Google Scholar PubMed PubMed Central
[39] Y. Jo, H. Cho, S. Y. Lee, et al.., “Quantitative phase imaging and artificial intelligence: a review,” IEEE J. Sel. Top. Quantum Electron., vol. 25, pp. 1–14, 2019. https://doi.org/10.1109/jstqe.2018.2859234.Search in Google Scholar
[40] Y. Park, C. Depeursinge, and G. Popescu, “Quantitative phase imaging in biomedicine,” Nat. Photonics, vol. 12, pp. 578–589, 2018. https://doi.org/10.1038/s415660180253x.Search in Google Scholar
[41] N. Goswami, Y. R. He, Y.H. Deng, et al.., “Labelfree SARSCoV2 detection and classification using phase imaging with computational specificity,” Light Sci. Appl., vol. 10, p. 176, 2021. https://doi.org/10.1038/s41377021006208.Search in Google Scholar PubMed PubMed Central
[42] B. Rahmani, D. Loterie, G. Konstantinou, D. Psaltis, and C. Moser, “Multimode optical fiber transmission with a deep learning network,” Light Sci. Appl., vol. 7, pp. 1–11, 2018. https://doi.org/10.1038/s4137701800741.Search in Google Scholar PubMed PubMed Central
[43] N. Borhani, E. Kakkava, C. Moser, and D. Psaltis, “Learning to see through multimode fibers,” Optica, vol. 5, p. 960, 2018. https://doi.org/10.1364/optica.5.000960.Search in Google Scholar
[44] V. Bianco, P. L. Mazzeo, M. Paturzo, C. Distante, and P. Ferraro, “Deep learning assisted portable IR active imaging sensor spots and identifies live humans through fire,” Opt. Lasers. Eng., vol. 124, p. 105818, 2020. https://doi.org/10.1016/j.optlaseng.2019.105818.Search in Google Scholar
[45] S. You, E. J. Chaney, H. Tu, Y. Sun, S. Sinha, and S. A. Boppart, “Labelfree deep profiling of the tumor microenvironment,” Cancer Res., vol. 81, pp. 2534–2544, 2021. https://doi.org/10.1158/00085472.can203124.Search in Google Scholar
[46] J. Yoon, Y. Jo, M.H. Kim, et al.., “Identification of nonactivated lymphocytes using threedimensional refractive index tomography and machine learning,” Sci. Rep., vol. 7, p. 6654, 2017. https://doi.org/10.1038/s4159801706311y.Search in Google Scholar PubMed PubMed Central
[47] J. Li, J. Garfinkel, X. Zhang, et al.., “Biopsyfree in vivo virtual histology of skin using deep learning,” Light Sci. Appl., vol. 10, p. 233, 2021. https://doi.org/10.1038/s41377021006748.Search in Google Scholar PubMed PubMed Central
[48] Y. Rivenson, T. Liu, Z. Wei, Y. Zhang, K. de Haan, and A. Ozcan, “PhaseStain: the digital staining of labelfree quantitative phase microscopy images using deep learning,” Light Sci. Appl., vol. 8, p. 23, 2019. https://doi.org/10.1038/s413770190129y.Search in Google Scholar PubMed PubMed Central
[49] J. Wu, L. Cao, L. Cao, G. Barbastathis, and G. Barbastathis, “DNNFZA camera: a deep learning approach toward broadband FZA lensless imaging,” Opt. Lett., vol. 46, pp. 130–133, 2021. https://doi.org/10.1364/ol.411228.Search in Google Scholar
[50] A. Sinha, J. Lee, S. Li, and G. Barbastathis, “Lensless computational imaging through deep learning,” Optica, vol. 4, p. 1117, 2017. https://doi.org/10.1364/optica.4.001117.Search in Google Scholar
[51] Y. Rivenson, H. Wang, Z. Wei, et al.., “Virtual histological staining of unlabelled tissueautofluorescence images via deep learning,” Nat. Biomed. Eng., vol. 3, pp. 466–477, 2019. https://doi.org/10.1038/s415510190362y.Search in Google Scholar PubMed
[52] I. Malkiel, M. Mrejen, A. Nagler, U. Arieli, L. Wolf, and H. Suchowski, “Plasmonic nanostructure design and characterization via Deep Learning,” Light Sci. Appl., vol. 7, p. 60, 2018. https://doi.org/10.1038/s4137701800607.Search in Google Scholar PubMed PubMed Central
[53] W. Ma, F. Cheng, and Y. Liu, “DeeplearningEnabled ondemand design of chiral metamaterials,” ACS Nano, vol. 12, pp. 6326–6334, 2018. https://doi.org/10.1021/acsnano.8b03569.Search in Google Scholar PubMed
[54] W. Ma, F. Cheng, Y. Xu, Q. Wen, and Y. Liu, “Probabilistic representation and inverse design of metamaterials based on a deep generative model with semisupervised learning strategy,” Adv. Mater., vol. 31, p. 1901111, 2019. https://doi.org/10.1002/adma.201901111.Search in Google Scholar PubMed
[55] E. Schwartz, R. Giryes, and A. M. Bronstein, “DeepISP: towards learning an endtoend image processing pipeline,” IEEE Trans. Image Process, vol. 28, pp. 912–923, 2019. https://doi.org/10.1109/tip.2018.2872858.Search in Google Scholar
[56] V. Sitzmann, S. Diamond, Y. Peng, et al.., “Endtoend optimization of optics and image processing for achromatic extended depth of field and superresolution imaging,” ACM Trans. Graph., vol. 37, pp. 1–13, 2018. https://doi.org/10.1145/3197517.3201333.Search in Google Scholar
[57] Z. Ballard, C. Brown, A. M. Madni, and A. Ozcan, “Machine learning and computationenabled intelligent sensor design,” Nat. Mach. Intell., vol. 3, pp. 556–565, 2021. https://doi.org/10.1038/s42256021003609.Search in Google Scholar
[58] H.A. Joung, Z. S. Ballard, J. Wu, et al.., “PointofCare serodiagnostic test for earlystage lyme disease using a multiplexed paperbased immunoassay and machine learning,” ACS Nano, vol. 14, pp. 229–240, 2020. https://doi.org/10.1021/acsnano.9b08151.Search in Google Scholar PubMed
[59] C. A. Metzler, H. Ikoma, Y. Peng, and G. Wetzstein, “Deep optics for singleshot highdynamicrange imaging,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2020, pp. 1372–1382.10.1109/CVPR42600.2020.00145Search in Google Scholar
[60] L. Hu, S. Hu, W. Gong, and K. Si, “Learningbased ShackHartmann wavefront sensor for highorder aberration detection,” Opt. Express, vol. 27, pp. 33504–33517, 2019. https://doi.org/10.1364/oe.27.033504.Search in Google Scholar
[61] M. Schmeisser, B. C. Heisen, M. Luettich, et al.., “Parallel, distributed and GPU computing technologies in singleparticle electron microscopy,” Acta Crystallogr. D Biol. Crystallogr., vol. 65, pp. 659–671, 2009. https://doi.org/10.1107/s0907444909011433.Search in Google Scholar PubMed PubMed Central
[62] B. J. Shastri, A. N. Tait, T. F. de Lima, M. A. Nahmias, H.T. Peng, and P. R. Prucnal, “Neuromorphic photonics, principles of,” in Encyclopedia of Complexity and Systems Science, R. A. Meyers, Ed., Berlin Heidelberg, Springer, 2018, pp. 1–37.10.1007/9783642277375_7021Search in Google Scholar
[63] Y. Shen, N. C. Harris, S. Skirlo, et al.., “Deep learning with coherent nanophotonic circuits,” Nat. Photonics, vol. 11, pp. 441–446, 2017. https://doi.org/10.1038/nphoton.2017.93.Search in Google Scholar
[64] A. N. Tait, T. F. de Lima, E. Zhou, et al.., “Neuromorphic photonic networks using silicon photonic weight banks,” Sci. Rep., vol. 7, no. 1, p. 7430, 2017. https://doi.org/10.1038/s4159801707754z.Search in Google Scholar PubMed PubMed Central
[65] X. Xu, M. Tan, B. Corcoran, et al.., “11 TOPS photonic convolutional accelerator for optical neural networks,” Nature, vol. 589, pp. 44–51, 2021. https://doi.org/10.1038/s41586020030630.Search in Google Scholar PubMed
[66] J. Feldmann, N. Youngblood, C. D. Wright, H. Bhaskaran, and W. H. P. Pernice, “Alloptical spiking neurosynaptic networks with selflearning capabilities,” Nature, vol. 569, pp. 208–214, 2019. https://doi.org/10.1038/s4158601911578.Search in Google Scholar PubMed PubMed Central
[67] A. N. Tait, M. A. Nahmias, B. J. Shastri, and P. R. Prucnal, “Broadcast and weight: an integrated network for scalable photonic spike processing,” J. Lightwave Technol., vol. 32, pp. 4029–4041, 2014. https://doi.org/10.1109/jlt.2014.2345652.Search in Google Scholar
[68] A. N. Tait, T. F. de Lima, M. A. Nahmias, et al.., “Silicon photonic modulator neuron,” Phys. Rev. Appl., vol. 11, p. 064043, 2019. https://doi.org/10.1103/physrevapplied.11.064043.Search in Google Scholar
[69] G. Wetzstein, A. Ozcan, S. Gigan, et al.., “Inference in artificial intelligence with deep optics and photonics,” Nature, vol. 588, pp. 39–47, 2020. https://doi.org/10.1038/s4158602029736.Search in Google Scholar PubMed
[70] X. Lin, Y. Rivenson, N. T. Yardimci, et al.., “Alloptical machine learning using diffractive deep neural networks,” Science, vol. 361, pp. 1004–1008, 2018. https://doi.org/10.1126/science.aat8084.Search in Google Scholar PubMed
[71] O. Kulce, D. Mengu, Y. Rivenson, and A. Ozcan, “Alloptical informationprocessing capacity of diffractive surfaces,” Light Sci. Appl., vol. 10, p. 25, 2021. https://doi.org/10.1038/s41377020004399.Search in Google Scholar PubMed PubMed Central
[72] E. Goi, X. Chen, Q. Zhang, et al.., “Nanoprinted highneurondensity optical linear perceptrons performing nearinfrared inference on a CMOS chip,” Light Sci. Appl., vol. 10, p. 40, 2021. https://doi.org/10.1038/s4137702100483z.Search in Google Scholar PubMed PubMed Central
[73] O. Kulce, D. Mengu, Y. Rivenson, and A. Ozcan, “Alloptical synthesis of an arbitrary linear transformation using diffractive surfaces,” Light Sci. Appl., vol. 10, p. 196, 2021. https://doi.org/10.1038/s41377021006235.Search in Google Scholar PubMed PubMed Central
[74] C. Isil, D. Mengu, Y. Zhao, et al.., “Superresolution image display using diffractive decoders,” arXiv, 2022. https://doi.org/10.48550/ARXIV.2206.07281.Search in Google Scholar
[75] D. Mengu and A. Ozcan, “Alloptical phase recovery: diffractive computing for quantitative phase imaging,” Adv. Opt. Mater., p. 2200281, 2022. https://doi.org/10.1002/adom.202200281.Search in Google Scholar
[76] Y. Luo, Y. Zhao, J. Li, et al.., “Computational imaging without a computer: seeing through random diffusers at the speed of light,” eLight, vol. 2, p. 4, 2022. https://doi.org/10.1186/s43593022000124.Search in Google Scholar
[77] D. Mengu, Y. Zhao, N. T. Yardimci, Y. Rivenson, M. Jarrahi, and A. Ozcan, “Misalignment resilient diffractive optical networks,” Nanophotonics, vol. 9, pp. 4207–4219, 2020. https://doi.org/10.1515/nanoph20200291.Search in Google Scholar
[78] D. P. Kingma and J. Ba, “Adam: a method for stochastic optimization,” in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 79, 2015, Conference Track Proceedings, Y. Bengio and Y. LeCun, Eds., 2015.Search in Google Scholar
Supplementary Material
The online version of this article offers supplementary material (https://doi.org/10.1515/nanoph20220358).
© 2022 the author(s), published by De Gruyter, Berlin/Boston
This work is licensed under the Creative Commons Attribution 4.0 International License.