Diffractive interconnects: all-optical permutation operation using diffractive networks

1 Introduction

Permutation is one of the basic computational operations that has played a key role in numerous areas of engineering, e.g., computing [1], communications [2], encryption [3], data storage [4], remote sensing [5], and data processing [6]. Historically, electronic integrated circuits have been the established implementation medium for the permutation operation and other space-variant linear transformations, while the research on optical computing has been mainly focused on using the Fourier transform approximation of thin lenses covering various applications in space-invariant transformations, e.g., convolution/correlation. On the other hand, as photonic switching devices and optical waveguide technology have become the mainstream communication tools on high-end applications, e.g., fiber optic networks, supercomputers, and data centers, various approaches have been developed toward all-optical implementation of permutation operation and other space-variant transformations based on, e.g., Mach–Zehnder interferometers [7], optical switches [8], photonic crystals [9], holographically recorded optical elements [10–12], off-axis lenslet arrays [13, 14], and arrays of periodic grating-microlens doublets [15]. The development of compact, low-power optical permutation and interconnection devices can have significant impact on next-generation communication systems, e.g., 6G networks [16, 17], as well as other applications such as optical data storage [18] and image encrypting cameras [19–21].

With the widespread availability of high-end graphics processing units (GPU) and the massively growing amounts of data, the past decade has witnessed major advances in deep learning, dominating the field of digital information processing for various engineering applications including, e.g., image segmentation and classification [22–25], natural language processing [26, 27], among others [28]. The statistical inference and function approximation capabilities of deep neural networks have also been exploited to produce state-of-the-art performance for computational inverse problems in many imaging and sensing applications including, e.g., microscopy [29–35], quantitative phase imaging [36–41], and others [42–51]. Beyond these data processing tasks, deep learning can also provide task-specific solutions to challenging inverse optical design problems for numerous applications including nanophotonics [52, 53], metamaterials [54], imaging, and sensing [55–60]. However, as the success and the applications of deep learning grow further, the electronic parallel computing platforms, e.g., GPUs, hosting deep neural networks and other machine learning algorithms have started to bring some limitations due to their power- and bandwidth-hungry operation. Moreover, the pace of the advances in computational capacity of the integrated circuits has fallen behind the exponential increase predicted by the Moore’s law [61]. These factors have fueled a tremendous amount of effort toward the development of optical machine learning schemes and other photonic computing devices that can partially reduce the computational burden on the electronics leading to power-efficient, massively parallel, high-speed machine learning systems. While most of the arising optical computing techniques rely on integrated photonic devices and systems compatible with the integrated waveguide technology [62–68], an alternative option toward exploiting photons for machine learning and the related computing tasks is to use complex modulation media and free-space light propagation and diffraction, which is particularly suitable for visual computing applications where the information is already carried by the optical waves (e.g., of a scene or target object) in free space [69].

Motivated by these pressing needs, Diffractive Deep Neural Networks (D²NN) [70, 71] have emerged as an optical machine learning framework that utilizes deep learning to engineer light–matter interactions over a series of diffractive surfaces so that a desired statistical inference or deterministic computing task is realized all-optically as the light waves propagate through structured surfaces. According to this framework, the physical parameters determining the phase and/or amplitude of light over each independently controllable unit, i.e., the “diffractive neuron,” are updated through the stochastic gradient descent and error-backpropagation algorithms based on a loss function tailored specifically for a given task. The weights of the connections between the diffractive neurons/features on successive layers, on the other hand, are dictated by the light diffraction in free space. Once the deep learning-based training, which is a one-time effort, is completed using a computer, the resulting transmissive/reflective diffractive layers are fabricated using, e.g., lithography or 3D printing, to physically form the diffractive network that completes a given inference or computational task at the speed of light using entirely passive modulation surfaces, offering a task-specific, power-efficient, and fast optical machine learning platform.

Based on the D²NN framework, here we demonstrate diffractive optical network designs that were trained to all-optically perform a given permutation operation between the optical intensities at the input and output fields-of-view, capable of handling hundreds of thousands of interconnections with diffraction limited resolution. We quantified the success of the presented diffractive optical networks in approximating a given, randomly selected permutation operation as a function of the number of diffractive neurons and transmissive layers used in the diffractive network design. We also laid the foundations toward practical implementations of diffractive permutation networks by investigating the impact of various physical error sources, e.g., lateral and axial misalignments and unwanted in-plane layer rotations, on the quality/accuracy of the optically realized interconnection weights and the permutation operation. Moreover, we showed that the diffractive optical permutation networks can be trained to be resilient against possible misalignments as well as imperfections in the diffractive layer fabrication and assembly. Finally, we present the first proof-of-concept experimental demonstration of diffractive permutation networks by all-optically achieving a permutation matrix of size 25 × 25, effectively realizing 625 interconnections based on 3D-printed diffractive layers operating at the THz part of the spectrum.

The presented diffractive optical permutation networks can readily find applications in THz-band communication systems serving as communication channel patch panels; furthermore, the underlying methods and design principles can be broadly extended to operate at other parts of the electromagnetic spectrum, including the visible and IR wavelengths, by scaling each diffractive feature size proportional to the wavelength of light [72] and can be used for image encryption in security cameras [21] and optical data storage systems, among other applications [73–76].

2 Results

Figure 1 illustrates the presented free-space permutation interconnect concept designed around diffractive optical networks using the D²NN framework. As shown in Figure 1, the presented permutation interconnect scheme does not use any standard optical components such as lenses, and instead relies on a series of passive, phase-only diffractive surfaces. Due to the passive nature of these layers, the diffractive optical network shown in Figure 1 does not consume any power except for the illumination light, providing a power-efficient permutation operation in a compact footprint of ∼600λ along the longitudinal axis, which could be further squeezed as needed. The 5-layer diffractive optical permutation network design shown in Figure 1 was trained through supervised learning to all-optically realize a desired permutation operation, P , between the light intensity signals at the input and output FOVs, each with N _i = N _o = 400 (20 × 20) individual pixels of size 2λ × 2λ. Stated differently, this permutation operation controls in total of N _i N _o = 0.16 million optical intensity connections.

Figure 1:

The schematic of a 5-layer diffractive permutation network, all-optically realizing 0.16 million interconnects between an input and output field-of-view. The presented diffractive permutation network was trained to optically realize an arbitrarily selected permutation operation between the light intensities over N _i = 400 = 20 × 20 input and N _o = 400 = 20 × 20 output pixels, establishing N _i N _o = 0.16 million desired interconnections based on 5 phase-only diffractive layers, each containing 40K(200 × 200) diffractive neurons/features.

The supervised nature of the training process of the diffractive permutation network necessitates the use of a set of input–output signal pairs (examples that satisfy P ) to compute a penalty term and the associated gradient-based updates with respect to the physical parameters of each diffractive neuron at every iteration. We set the optical signal of interest at the input and the output of the diffractive permutation scheme to be the light intensity, and as a result, the deep learning-based evolution of the presented diffractive permutation network shown in Figure 1 was driven based on the mean-squared error (MSE) (see Methods Section) between the ground-truth and the all-optically synthesized output intensity patterns at a given iteration. Since this loss function acts only on the light intensity, the diffractive optical network can enjoy an output phase-freedom in synthesizing the corresponding transformed optical intensity patterns within the output field-of-view. The light intensity, I, is related to the complex-valued field, U, through a nonlinear operation, I = U 2 . If a pair of input–output complex fields exists for a given diffractive network, i.e., {U _in, U _out} and { U i n ′ , U out ′ }, then the input field U i n ′ ′ = α U i n + β U i n ′ will create the output field U out ′ ′ = α U out + β U out ′ at the output plane. In terms of the associated intensities, however, this direct linear extension does not hold since α U out 2 + β U out ′ 2 ≠ U out ′ ′ 2 , making it challenging (in terms of the data generalization capability) to design diffractive optical networks for achieving a general purpose intensity-to-intensity transformation such as a permutation operation. To overcome this generalization challenge, we trained our diffractive permutation networks using ∼4.7 million randomly generated input/output intensity patterns that satisfy the desired P , instead of a standard benchmark image dataset (see the Methods).

After the training phase, we blindly tested each diffractive permutation network with test inputs that were never used during the training. Figure 2 illustrates 6 different randomly generated blind testing inputs along with the corresponding all-optically permuted output light intensities. In the first two randomly generated input patterns shown in Figure 2A, there is light coming out of all the input pixels/apertures at different levels of intensity. In the next two test input patterns shown in Figure 2A, on the other hand, nearly half of the input apertures have nonzero light intensity and finally, the last two test inputs contain only 10 and 11 pixels/apertures with light propagating toward the 1st layer of the diffractive permutation network. When tested on 20K randomly generated blind testing input intensity patterns with different sparsity levels, the 5-layer diffractive permutation network shown in Figure 1 achieves 18.61 dB peak-signal-to-noise ratio (PSNR), very well matching the ideal output response. For this randomly generated 20K testing data, Figure 2B also shows the distribution of PSNR as a function of the number of input pixels with nonzero light intensity, which reveals that the diffractive permutation network can permute relatively sparser inputs with a higher output image quality, achieving a PSNR of 25.82 dB.

Figure 2:

Input–output intensity pairs generated by the diffractive permutation network shown in Figure 1. (A) The diffractive permutation network shown in Figure 1 was tested on two different datasets. The first blind testing dataset contains 20K randomly generated inputs. 6 examples from this randomly created testing data are shown here for demonstrating input–output intensity pairs with low, moderate and high signal sparsity levels. Beyond successfully permuting randomly generated intensity patterns, the performance of the diffractive permutation network was also quantified using permuted EMNIST images. None of these test samples were used in the training phase. (B) Output intensity image PSNR with respect to the ground truth intensity patterns as a function of the input signal sparsity in randomly generated test dataset. (C) Same as (B), except for EMNIST test images.

In addition to randomly generated blind testing inputs, we further tested the diffractive permutation network shown in Figure 1 on 18.75K EMNIST images; note that this diffractive network was trained only using randomly generated input/output intensity patterns that satisfy P and the EMNIST images constitute not only blind testing set but also a significant deviation from the statistical distribution of the training images. The input field-of-view contains the permuted EMNIST images ( P ⁻¹) and the diffractive network inverts that permutation by all-optically performing P to recover the original images at the output plane (see Figure 2). The performance of the diffractive permutation network was quantified based on both PSNR and Structural Similarity Index Measure (SSIM). With N _L = 40K diffractive neurons on each layer, the 5-layer diffractive permutation network shown in Figure 1 provides 19.18 dB and 0.85 for PSNR and SSIM metrics, respectively, demonstrating the generalization capability of the diffractive network to new types of input image data never seen during the training phase.

2.1 Impact of the number of diffractive layers and features

Next, we investigate the performance of diffractive permutation networks as a function of the number of diffractive neurons trained within the system. Toward this goal, in addition to the 5-layer design shown in Figure 1 that has in total of N = 200K trainable diffractive features, we trained diffractive permutation networks consisting of 4, 3, and 2 modulation surfaces. The physical design parameters such as the size/width of the diffractive surfaces, layer-to-layer distances, and the extent of the input and output fields-of-view were kept identical to the ones in the 5-layer network design. In other words, these new diffractive networks are designed and trained exactly in the same way as the previous 5-layer network except they contain fewer diffractive layers. Figure 3A and B provides a quantitative comparison between these 4 diffractive permutation networks. While Figure 3A illustrates the mean PSNR and SSIM values achieved by each diffractive network for recovering EMNIST images, Figure 3B demonstrates the mean-squared-error (MSE) between the desired permutation operation and its optically realized version ( P D 2 N N ) as a function of the number of diffractive layers utilized in these designs (see Supplementary Figure S1 and the Methods Section on the estimation of P D 2 N N ). According to the permutation operator error shown in Figure 3B, the performance improvement of the system increases drastically with the additional diffractive layers up to the 4-layer design that represents a critical point in the sense that the inclusion of a 5th diffractive surface brings a relatively small improvement. The reason behind this behavior is the fact that the number of diffractive features, N, in the 4-layer diffractive permutation network matches the space-bandwidth product set by our input and output FOVs, i.e., N _i N _o = 400 × 400 = 160K. In other words, Figure 3B reveals that when the number of phase-only diffractive modulation units N matches or exceeds N _i N _o , the diffractive optical network can achieve a given linear transformation between the input and output intensities with a very low error, i.e., P D 2 N N ≈ P ; for example, the MSE between P D 2 N N and P in the case of a 4-layer design was found to be 6.63 × 10⁻⁵. For N < N _i N _o , the error between P D 2 N N and P increases accordingly, as shown in Figure 3B.

Figure 3:

The impact of the number of diffractive layers on the approximation accuracy of D²NN for a given intensity permutation operation. (A) The average SSIM and PSNR values achieved by the diffractive permutation network designs based on L = 2, L = 3, L = 4 and L = 5 diffractive layers, containing 200 × 200, i.e., 40K, phase-only diffractive neurons/features per layer for the task of optically recovering permuted EMNIST images. (B) The transformation error between the desired intensity permutation (P) and its optically realized counterpart ( P D 2 N N ) for the diffractive networks with L = 2, L = 3, L = 4 and L = 5 diffractive layers. The transformation error decreases as a function of the number of layers in the diffractive network architecture. The L = 4-layer diffractive permutation network design represents a critical point as it matches the space-bandwidth product requirement of the desired permutation operation, i.e., N = N _i N _o = 4 × 40K = 160K, and further increasing the number of layers to L = 5 brings only a minor improvement. (C) Examples of EMNIST test images demonstrating the performance of the diffractive permutation networks as a function of L.

The benefit of having N ≥ N _i N _o is further revealed in the increased generalization capability of the diffractive network as shown in Figure 3A. Since the EMNIST images were not used during the training, they represent completely new types of input intensity patterns for the presented diffractive optical networks. The SSIM (PSNR) values achieved by the 4-layer diffractive network is found as 0.75 (16.41 dB) for the optical recovery of the permuted EMNIST images. These numbers are significantly higher compared to the performance of the 3-layer and 2-layer diffractive designs that can attain SSIM (PSNR) values of 0.46 (12.91 dB) and 0.30 (12.08 dB) for the same task; furthermore, the 5-layer diffractive network design shown in Figure 1 outperforms the others by achieving 0.85 (19.18 dB) for the same performance metrics. The visual comparison of the input–output intensity patterns depicted in Figure 3C further supports this conclusion, where the noise due to the crosstalk between interconnection channels decreases proportional to the number of diffractive layers in the system.

2.2 Vaccination of diffractive permutation networks

With sufficiently large number of phase-only diffractive neurons/features, the diffractive networks can optically realize permutation operations with, e.g., 0.16 million channels between the input and output pixels as shown in Figure 3. In fact, the number of interconnects that can be optically implemented through diffractive networks can go far beyond 0.16 million, given that the size/width of the diffractive surfaces and the number of diffractive layers can be increased further depending on the fabrication technology and the optomechanical constraints of the system. In addition, as the number of diffractive layers increases in a diffractive network architecture, their forward model can better generalize to new, unseen data as shown in Figure 3A.

On the other hand, deeper diffractive optical network designs are more susceptible to misalignments that are caused by the limitations of the optomechanical assembly and/or the fabrication technology that is utilized. It was shown that diffractive optical networks trained for statistical inference tasks, e.g., all-optical object classification, can be vaccinated against misalignments and other physical error sources, when the factors creating these nonideal conditions were incorporated into the training forward model, which was termed as vaccinated-D²NNs or v-D²NNs [77]. Specifically, v-D²NN expands on the original D²NN framework by modeling possible error sources as random variables and integrating them as part of the training model so that the deep learning-based evolution of the diffractive surfaces is guided toward solutions that are resilient to nonideal physical conditions and/or fabrication errors. Toward practical applications of diffractive permutation networks, we quantified the impact of optomechanical errors and applied the v-D²NN framework to devise robust solutions that can achieve a given interconnect operation despite fabrication tolerances.

In our numerical study depicted in Figure 4, we considered 4 different misalignment components representing the 3D misalignment vector of the lth diffractive layer, D x l , D y l , D z l and their in-plane rotation around the optical axis denoted as D θ l . Each of these 4 misalignment components were defined as independent, uniformly distributed random variables, D * l ∼ U ( − Δ * , Δ * ) , with Δ_* defined as a function of a common auxiliary parameter, v. The lateral misalignments parameters, Δ _x and Δ _y , determining the range of D x l and D y l , respectively, were set to be 0.67λv, i.e., D x l ∼ U ( − 0.67 λ v , 0.67 λ v ) and D y l ∼ U ( − 0.67 λ v , 0.67 λ v ) , where λ denotes the wavelength of the illumination light. Similarly, Δ _z and Δ _θ were defined as 24λv and 4°v. For instance, if we take v = 0.5, this means each diffractive layer can independently/randomly shift in both x and y axes within a range of (−0.335λ, 0.335λ). In addition, their location over the z direction and their in-plane orientation can randomly change within the ranges of (−12λ, 12λ) and (−2°, 2°), respectively (see the Methods Section for more details).

Figure 4:

The sensitivity of the diffractive permutation networks against various levels of physical misalignments (A) SSIM values achieved by 5-layer diffractive permutation networks with and without vaccination. (B) Transformation errors between the desired 100 × 100 permutation operation (P) and its optically synthesized counterpart ( P D 2 N N ) at different levels of misalignments denoted by v _test. (C) The layers of a nonvaccinated diffractive permutation network, i.e., v _tr = 0, along with the examples of EMNIST test images recovered optically through the diffractive permutation operation. (D) Same as (C), except for a vaccinated diffractive permutation network based on v _tr = 0.25.

To better highlight the impact of these misalignments and demonstrate the efficacy of the v-D²NN framework, we trained a new nonvaccinated, i.e., v _tr = 0, diffractive permutation network that can all-optically realize a given permutation matrix, P , representing 10K intensity interconnections between 100 input and 100 output pixels of size 4λ × 4λ. The error-free training model of this diffractive network with v _tr = 0 implicitly assumes that when the resulting diffractive network is fabricated, the system conditions will exactly match the ideal settings regarding the 3D locations of the layers and their in-plane orientations. With an architecture identical to the one shown in Figure 1, containing N = 200K ≫ N _i N _o diffractive neurons, this diffractive network can all-optically approximate the permutation matrix, P , with an MSE of 1.45 × 10⁻⁶ in the absence of any misalignment errors, i.e., v _test = 0 (see the green curve in Figure 4B). However, when there is some discrepancy between the training and testing conditions, i.e., v _test > 0, the optically implemented forward transformation, P D 2 N N , starts to deviate from the desired operation P . For instance, at v _test = 0.125, the transformation error, ‖ P D 2 N N − P ‖ , can be computed as 3.1 × 10⁻³. This negative impact of the physical misalignments on the performance of a nonvaccinated diffractive network can also be seen in Figure 4A (green curve), which demonstrates the SSIM values achieved by this diffractive network for recovering permuted EMNIST images under different levels of misalignments. The high-quality of the image recovery (see Figure 4C) at v _test = 0 quantified with an SSIM of 0.99 deteriorates under the presence of misalignments, highlighted by the SSIM value falling to 0.49 and 0.30 at v _test = 0.125 and v _test = 0.25, respectively.

Unlike the nonvaccinated design, the vaccinated diffractive permutation networks can maintain their approximation capacity and accuracy under erroneous testing conditions as shown in Figure 4A and B. For instance, the SSIM value of 0.49 attained by the nonvaccinated diffractive network for the misalignment uncertainty set by v _test = 0.125 increases to 0.88 in the case of a diffractive permutation network trained with v _tr = 0.25 (red curve in Figure 4A). The difference between the image recovery performances of the vaccinated and the nonvaccinated diffractive network designs increases further as the misalignment levels increase during the blind testing. While the nonvaccinated diffractive network can only achieve SSIM values of 0.3 and 0.19 at v _test = 0.25 and v _test = 0.375, respectively, the output images synthesized by the vaccinated design (v _tr = 0.25) reveals SSIM values of 0.8 at v _test = 0.25 and 0.64 at v _test = 0.375 (see Figure 4D). A similar conclusion can also be drawn from Figure 4B, demonstrating the MSE values between the desired permutation matrix, P , and its optically realized counterpart, P D 2 N N . The transformation errors, ‖ P D 2 N N − P ‖ , of the vaccinated diffractive network (v _tr = 0.25) at v _test = 0.125 and at v _test = 0.25 were computed as 5.15 × 10⁻⁴ and 1.2 × 10⁻³, respectively, which are 5–10 times smaller compared to the MSE values provided by the nonvaccinated diffractive design at the same misalignment levels. Supplementary Figure S2 further illustrates the error maps between P and P D 2 N N realized by the nonvaccinated and vaccinated diffractive permutation networks at different misalignment levels.

The compromise for this misalignment robustness comes in the form of a reduction in the peak performance. While the nonvaccinated diffractive network can solely focus on realizing the given permutation operation with the highest quality and approximation accuracy, the vaccinated diffractive network designs partially allocate their degrees-of-freedom to building up resilience against physical misalignments. For example, while the peak SSIM achieved by the nonvaccinated diffractive network is 0.99, it is 0.88 for the diffractive permutation network vaccinated with v _tr = 0.25. The key difference, on the other hand, is that the better performance of the nonvaccinated diffractive network is sensitive to the physical implementation errors, while the vaccinated diffractive permutation networks can realize the desired input–output interconnects over a larger range of fabrication errors or tolerances. A comparison between the diffractive layer patterns of the nonvaccinated and vaccinated diffractive permutation networks shown in Figure 4C and D, respectively, also reveals that the vaccination strategy results in smoother light modulation patterns; in other words, the material thickness values over the neighboring diffractive neurons partially lose their independence and become correlated, causing a reduction in the number of independent degrees-of-freedom in the system.

2.3 Experimental demonstration of a diffractive permutation network

To experimentally demonstrate the success of the presented diffractive permutation interconnects, we designed a 3-layer diffractive permutation network achieving the desired (randomly generated) intensity shuffling operation with N _i = N _o = 5 × 5, optically synthesizing 625 connections between the input and output FOVs; this network was designed to operate at 0.4 THz, corresponding to ∼0.75 mm in wavelength. During the training, the forward model of this diffractive permutation network was vaccinated with v _tr = 0.5 against the 4 error sources as detailed in Section 2.2 including the 3D location of each diffractive layer and the in-plane rotation angle around the optical axis. In addition to these misalignment components, we also vaccinated this diffractive network model against unwanted material thickness variations that could arise due to the limited lateral and axial resolution of our 3D printer (see the Methods Section for details). To compensate for the reduction in the degrees-of-freedom due to the vaccination scheme, the number of phase-only diffractive features in the permutation network was selected to be N _L = 10K diffractive neurons per layer. Therefore, each diffractive layer shown in Figure 5A contains 100 × 100 phase-only diffractive neurons of size ∼0.67λ × 0.67λ. Compared to the diffractive surfaces shown in Figures 1 –4, the layers of our experimental system were set to be 2-times smaller in both the x and y directions to keep the layer-to-layer distances smaller while maintaining the level of optical connectivity between the successive diffractive surfaces (see Figure 5B). Figure 5C and D illustrates the 3D printed diffractive permutation network and the schematic of our experimental setup (see the Methods Section for details).

Figure 5:

Experimental demonstration of a diffractive permutation network. (A) The material thickness profiles of the diffractive surfaces of the fabricated diffractive permutation network. (B) The schematic of the experimental architecture illustrating the forward optical model of the diffractive permutation network. (C) 3D printed diffractive permutation network operating at THz part of the spectrum. (D) The schematic of our experimental system.

Figure 6A illustrates the targeted 25 × 25 permutation matrix ( P ) that is randomly generated and the numerically predicted P D 2 N N along with the absolute difference map between these two matrices. According to the numerical forward model of the trained diffractive network shown in Figure 5, the transformation error between the P and P D 2 N N , i.e., ‖ P D 2 N N − P ‖ is equal to 5.99 × 10⁻⁴ under error-free conditions, i.e., v _test = 0. Furthermore, the forward model of the trained diffractive permutation network shown in Figure 5 provides 17.87 dB PSNR on average for the test letters “U,” “C,” “L,” and “A,” as depicted in Figure 6B. A visual comparison between the numerically predicted and the experimentally measured output images of these 4 input letters (which were never seen by the network before) demonstrates the accuracy of the forward training and testing models as well as the success of the presented diffractive permutation network design. Interestingly, the PSNR of the experimentally measured images was observed to be higher, 19.54 dB, compared to the numerically predicted value, 17.87 dB. Our numerical study reported in Figure 4 suggests that this can be explained based on the vaccination range used during the training and the amount physical error in the system testing. For instance, the SSIM value achieved by the vaccinated diffractive network trained with v _tr = 0.5 (yellow curve) at relatively lower physical misalignment levels, e.g., v _test = 0.125, is higher compared to its performance under the ideal conditions, i.e., v _test = 0.0, as depicted in Figure 4A.

Figure 6:

Experimental results. (A) (left) The desired 25 × 25 permutation matrix, P, (middle) the optically realized permutation operation predicted by the numerical forward model, P D 2 N N , and (right) the absolute error map between the two matrices. (B) Comparison between the numerically predicted and the experimentally measured output images for the task of recovering intensity patterns describing the letters “U,” “C,” “L,” and “A”.

3 Discussion

Beyond optomechanical error sources and fabrication tolerances, another factor that might potentially hinder the utilization of diffractive permutation networks in practical applications is the output diffraction efficiency. For instance, the diffraction efficiency of the 5-layer network shown in Figure 1 is ∼0.004% which might be very low for some applications. On the other hand, this can be significantly increased by using an additional loss term, penalizing the poor diffraction efficiency of the network (see the Methods Section for details). Supplementary Figure S3 demonstrates 5-layer diffractive networks that are designed to optically realize 0.16 million interconnections between the input and output FOVs with increased diffraction efficiencies, and compares their performance in terms of SSIM values achieved. The training of these diffractive network models is based on a loss function in the form of a linear combination of two different penalty terms, L ′ = L + γ L e , where L L e is the diffraction efficiency related penalty term promoting efficient solutions (see the Methods Section). As a general trend, the diffraction efficiency of the underlying diffractive network model increases as a function of the weight (γ) of the efficiency penalty term in the loss function. However, since the number of diffractive neurons, hence, the degrees-of-freedom in these diffractive network models is very close to N _i N _o , the diffraction efficiency either does not improve beyond a certain value or the evolution of the diffractive layers starts to solely focus on the efficiency instead of the desired permutation operation resulting in low performance designs. This unstable behavior can be observed specifically when 0.235 < γ < 0.24. On the other hand, as in the case of vaccinated diffractive network models, if the diffractive network architecture contains N ≫ N _i N _o diffractive neurons, then this instability vanishes, providing significant improvements in the output diffraction efficiency without sacrificing the performance of the all-optical permutation operation. For instance, the 3D printed diffractive permutation network depicted in Figure 6 was trained based on L ′ with γ = 0.15 and it provides 2.45% output diffraction efficiency, despite the fact that 89.37% of the incident power at the input plane is lost due to the absorption of the 3D printing material. With weakly absorbing transparent materials used as part of the diffractive network fabrication, a significantly larger output efficiency can be achieved.

Also note that, although we solely focused on diffractive network designs composed of dielectric optical modulation surfaces, in principle, some of these layers can be replaced with metasurfaces/metamaterials. While the use of can provide some additional degrees of freedom, including, for example, the engineering of dispersion, there are some challenges to overcome in realizing metamaterial-based diffractive networks. In the presence of fabrication errors and imperfections, the scattered light fields might deviate from the predictions of the numerical forward model. These nonideal waves generated by a metasurface would then excite unwanted diffraction modes over the subsequent layers generating an “avalanche” within the diffractive volume, accumulating substantial field errors, especially for deeper network designs with L ≥ 2. In addition, the physical models of phase and/or amplitude modulation of meta-atoms are, in general, valid for waves covering a relatively small numerical aperture (NA). As a result, a high NA diffractive network design that utilizes the entire bandwidth of the propagation medium (NA = 1, for air) would be challenging, as the modulation response of the meta-units might deviate from their ideal small angle responses, introducing errors to the forward model. Although it is possible to restrict a diffractive network design to work with a lower NA, it would increase the overall footprint of the system and reduce the space-bandwidth product that can be processed by the diffractive network.

These challenges, in general, are negligible for dielectric diffractive networks composed of λ/2 features on a substrate, as also highlighted by the close match between the numerically predicted images and their experimentally measured counterparts shown in Figure 6 and our former work [21, 70, 74, 76, 77]. In our optical forward model, the diffractive layers are assumed to be thin modulation surfaces, i.e., there is only a single scattering event converting an incident wave field to an outgoing one after each diffractive layer. Practically, though, there are additional scattered fields that are ignored in our model, especially when there is a substantial material thickness variation between adjacent pixels. However, in our designs, we do not observe a sharp material thickness transition between neighboring pixels. This is mainly due to the nature of our training process. Specifically, the presented diffractive networks are trained through error-backpropagation, which computes the variable updates by taking the gradient of the loss function with respect to the material thickness over each diffractive unit. In such a process, it is highly unlikely that the gradients of two adjacent pixels (λ/2 apart from each other) deviate significantly from each other, which effectively causes a smoothening effect on the diffractive surface height profiles as they are being optimized through deep learning. This smoothening behavior is even more pronounced in the vaccinated diffractive network designs due to the random lateral translation of the diffractive layers as part of the training forward model. Therefore, the impact of side scattering or field shadowing due to height discontinuities across a given diffractive layer design is negligible. In addition, the back-reflected waves can also be ignored, as these are, in general, weak processes unless they are specifically enhanced using, e.g., metamaterials or other special structures. Therefore, the optical forward model of dielectric diffractive networks can be accurately represented within the scalar diffraction theory without needing vectorial modeling of light fields or considering weaker multi-reflections. Finally, the evanescent waves and the vectorial fields associated with them can be entirely ignored since each successive diffractive layer is axially positioned >λ away from the previous layer.

In summary, we showed that the diffractive networks can optically implement intensity permutation operations between their input and output apertures based on phase-only light modulation surfaces with N ≥ N _i N _o diffractive neurons. Due to the nonlinear nature of the intensity operation, it is crucial to use training input intensity patterns with different levels of sparsity to prevent any type of data-specific overfitting during the training phase. Diffractive permutation networks with N > N _i N _o demonstrate increased generalization capability, synthesizing more accurate outputs with ‖ P D 2 N N − P ‖ ≈ 0 . By using N > N _i N _o , one can also design misalignment and fabrication error insensitive, power-efficient diffractive permutation networks, which could play a major role in practical applications, e.g., 6G wireless networks, computational cameras, etc. Although this study demonstrated diffractive optical networks realizing permutation operations with 0.16 million interconnects, with N _i = N _o = 20 × 20, these systems are highly scalable to even larger N _i , N _o combinations depending on the availability of training computer hardware. Since the training of a diffractive optical network is a one-time effort, one can use a computing platform with a significantly larger random-access memory (RAM) to design much bigger diffractive networks. Alternatively, the forward training model of a diffractive network can also be distributed among multiple GPUs for parallel computing with increased memory capacity paving the way to significantly larger permutation operations to be implemented all-optically. Finally, the incorporation of dynamic spatial light modulators to replace some of the diffractive layers in a given design can be used to reconfigure, on demand, the all-optically performed diffractive transformation.

4 Methods