Skip to content
BY 4.0 license Open Access Published by De Gruyter October 7, 2020

Waveguide combiners for mixed reality headsets: a nanophotonics design perspective

  • Bernard C. Kress ORCID logo EMAIL logo and Ishan Chatterjee
From the journal Nanophotonics

Abstract

This paper is a review and analysis of the various implementation architectures of diffractive waveguide combiners for augmented reality (AR), mixed reality (MR) headsets, and smart glasses. Extended reality (XR) is another acronym frequently used to refer to all variants across the MR spectrum. Such devices have the potential to revolutionize how we work, communicate, travel, learn, teach, shop, and are entertained. Already, market analysts show very optimistic expectations on return on investment in MR, for both enterprise and consumer applications. Hardware architectures and technologies for AR and MR have made tremendous progress over the past five years, fueled by recent investment hype in start-ups and accelerated mergers and acquisitions by larger corporations. In order to meet such high market expectations, several challenges must be addressed: first, cementing primary use cases for each specific market segment and, second, achieving greater MR performance out of increasingly size-, weight-, cost- and power-constrained hardware. One such crucial component is the optical combiner. Combiners are often considered as critical optical elements in MR headsets, as they are the direct window to both the digital content and the real world for the user’s eyes.

Two main pillars defining the MR experience are comfort and immersion. Comfort comes in various forms:

  1. wearable comfort—reducing weight and size, pushing back the center of gravity, addressing thermal issues, and so on

  2. visual comfort—providing accurate and natural 3-dimensional cues over a large field of view and a high angular resolution

  3. vestibular comfort—providing stable and realistic virtual overlays that spatially agree with the user’s motion

  4. social comfort—allowing for true eye contact, in a socially acceptable form factor.

Immersion can be defined as the multisensory perceptual experience (including audio, display, gestures, haptics) that conveys to the user a sense of realism and envelopment. In order to effectively address both comfort and immersion challenges through improved hardware architectures and software developments, a deep understanding of the specific features and limitations of the human visual perception system is required. We emphasize the need for a human-centric optical design process, which would allow for the most comfortable headset design (wearable, visual, vestibular, and social comfort) without compromising the user’s sense of immersion (display, sensing, and interaction). Matching the specifics of the display architecture to the human visual perception system is key to bound the constraints of the hardware allowing for headset development and mass production at reasonable costs, while providing a delightful experience to the end user.

Glossary of Terms, Abbreviations, and Acronyms

We provide this glossary for the reader after the abstract section as these acronyms are used extensively in this review paper.

ARAugmented reality, adding virtual content into field of view of reality, can include augmentations created by mixed reality headsets, handhelds, head up displays, smart glasses, camera-projector systems, etc.
MRMixed reality, virtual objects situationalized in 3D in your real space, often interactable
OST-MROptical see-through mixed reality, displays are transparent such that real world is viewable optically through the displays
VST-MRVideo see-through mixed reality, virtual reality turned into the mixed reality with camera pass-through of the real-world into the VR environment
XRExtended reality, a generic term to capture all varieties across MR and AR
VRVirtual reality, blocks out reality and supplants with virtual objects
ImmersionSense of realism and development in delivered experience
IMUInertial measurement unit consisting of at least an accelerometer, and gyroscope, and often a magnetometer
GPUGraphical processing unit, parallel architecture suited for graphics render and other matrix operations
HMDHead-mounted display or helmet-mounted display
HUDHead up display, refers to see-through display that is often mounted externally (such as above a dashboard) allowing user to see both virtual content and subject of focus (e.g., the road ahead) simultaneously
SLMSpatial light modulator
LCDLiquid-crystal display, display technology where electro-sensitive liquid crystal pixels amplitude-modulate light from a global polarized backlight in transmission
LTPS-LCDLow-temperature polysilicon liquid-crystal display, higher resolution and faster switching speed than amorphous Si LCD
IPS-LCDIn-plane switching liquid-crystal display, liquid-crystal structure twist in-plane of display, allowing for higher viewing angles than twisted nematic (TN) LCDs, used in phones and monitors
HTPS-LCDHigh-temperature polysilicon (used for silicon backplanes)
AMOLEDActive-matrix organic light-emitting diode, increased contrast at the cost of lifetime and high brightness, each pixel is its own organic electroluminescent emitter, used commonly in cellphones
mu-OLED, micro-OLEDMicro-organic light-emitting diode, display with emitter size less than 15 μm, used in camera electronic view finders
DLPDigital light processing, Texas Instrument’s colloquially genericized trademark for DMD (digital micromirror device), an array of bi-stable reflective micromirrors, commonly used in projection systems for highly efficiency SLM
LCoSLiquid crystal on silicon, microdisplay with a switchable liquid-crystal matrix on reflective silicon backplane
mu-iLED, micro-iLEDMicro inorganic light-emitting diode, actively addressed inorganic LED array with emitter size <50 mu, NTE displays usually require a magnitude lower; can achieve high brightness and contrast, but challenged in maintaining efficiency, multicolor integration and backplane integration
VCSELVertical-cavity surface-emitting laser, laser diode with lower divergence and current threshold than edge-emitting diodes
MEMSMicroelectromechanical system
LBSLaser beam scanning, type of display where a modulated laser dot is raster scanned across display FOV via system of MEMS mirrors
NTENear-to-eye
3DOF3 degrees of freedom, in the context of tracking usually refers to the rotational axes (pitch, yaw, roll) which can be resolved with only a calibrated IMU
6DOF6 degrees of freedom, in the context of tracking refers to the rotational and translational axes
CGCenter of gravity, important ergonomic metric in head-worn devices
IPDInterpupillary distance
PPDPixels per degree
HDRHigh dynamic range
FOVField of view, provided as an angle
EyeboxThe volume that the user’s pupil can sit in and view the entire virtual image field-of-view. The box may not be a rectangular prism, but is more often a frustrum
Eye reliefThe distance the user’s corneal surface is from the display optic surface
UX/UIUser experience/User interface, refers to the design of the experience and applications
VACVergence accommodation conflict, refers to the mismatch experienced when a stereoscopic display’s image focal plane does not match the stereo disparity of the virtual image.
Pupil swimThe experience of warp and shift of virtual objects as the user’s pupils rove around the eyebox caused by distortion in the projected image across the eyebox
Hard-edge occlusionThe ability for real-world objects to mask virtual content according to the depth the virtual image is in the world
HologramRecording of a interference pattern between a reference and a wavefront off a 3D scene… but in AR/VR forums, a virtual stereo image that appears to be positioned in space like a true hologram
ETEye tracking
HeTHead tracking
TIRTotal internal reflection (principle of how light propagates when trapped in a light guide)
PBSPolarized beam splitter
EPEExit pupil expansion, a technique where a combiner’s exit pupil may be replicated in 1D or 2D space allowing for a larger eyebox
LOELightguide optical element
SRGSurface relief gratings, nanostructure gratings etched into substrate surface, can be blazed, slanted, binary, multilevel, or analog
CGHComputer-generated hologram, hologram whose wavefront has be calculated computationally rather than recorded in analog
RWGResonant waveguide gratings, also known as GMR (guided mode resonant gratings), diffractive, dielectric structures with leaky lateral modes
MetasurfaceSurface with nanofabricated, sub-wavelength structures (often high aspect ratio) that can impart arbitrary phase changes in transmission and/or reflection unlocking unique optical functions.
NILNanoimprint lithography
ALDAtomic layer deposition
MTFModulation transfer functions, represents the effect (usually degradation) on spatial frequencies (in resolution and contrast) through an optical element, higher is better
RCWARigorous couple-wave analysis
FMMFourier modal method
FDTDFinite difference time domain
SAWSurface acoustic wave
AOMAcousto-optical modulator
EOMElectro-optical modulator

1 Introduction

Defense has been historically the first application sector for augmented reality (AR) and virtual reality (VR), as far back as the 1950s [1]. Based on these early developments, the first consumer VR/AR boom expanded in the early 1990s and contracted considerably throughout that decade, a poster child of a technology ahead of its time and ahead of its markets [2]. Notably, due to the lack of available consumer display technologies and related sensors, novel optical display concepts were introduced throughout the 1990s [3], [4] that are still considered as state of the art, such as the “Private Eye” smart glass from Reflection Technology (1989) and the “Virtual Boy” from Nintendo (1995)—both based on scanning displays rather than flat-panel displays. Although such display technologies were well ahead of their time [5], [6], [7], the lack of consumer-grade inertial measurement unit (IMU) sensors, low-power 3-dimensional (3D)-rendering graphical processing units, and wireless data transfer technologies contributed to the end of this first VR boom. The other reason was the lack of digital content or rather the lack of a clear vision of adapted VR/AR content for enterprise or consumer spaces [8], [9].

The only AR/VR sector that saw sustained efforts and developments throughout the next decade was the defense industry (flight simulation and training, helmet-mounted displays [HMDs] for rotary-wing aircrafts, and head-up displays [HUDs] for fixed-wing aircrafts) [10]. The only effective consumer efforts during 2000–2010 were in the field of automotive HUDs and personal binocular headset video players.

The smartphone technology ecosystem, including the associated display, connectivity, and sensor systems, shaped the emergence of the second VR/AR boom and formed the first building blocks used by early product integrators. Today’s engineers, exposed at an early age to ever-present flat-panel display technologies, tend to act as creatures of habit much more than their peers 20 years ago, who had to invent novel immersive display technologies from scratch. We have therefore seen since 2012 the initial implementations of immersive AR/VR HMDs based on readily available smartphone display panels (low-temperature polysilicon liquid-crystal display [LCD], In-plane switching liquid-crystal display, active-matrix organic light-emitting diode) and picoprojector microdisplay panels (High-temperature polysilicon LCD, mu-organic light-emitting diode (OLED), digital light processing (DLP), liquid crystal on silicon (LCoS). (Similarly, the AR/VR industry has been able to leverage the progress made during the smartphone revolution for cheap and reliable sensors as well, such as IMUs and cameras). Currently, HMD display architectures are evolving slowly to more specific technologies, which may be a better fit for immersive requirements than flat panels are, sometimes resembling the display technologies invented throughout the first AR/VR boom two decades earlier (inorganic mu-iLED panels, 1-dimensional [1D] scanned arrays, 2-dimension (2D) laser/vertical-cavity surface-emitting laser [VCSEL] microelectromechanical system [MEMS] scanners, and so on).

Such traditional display technologies will serve as an initial catalyst for what is coming next. The immersive display experience in AR/VR is a paradigm shift from the traditional panel display experiences that have existed for more than half a century, going from cathode ray tube (CRT) TVs to LCD computer monitors and laptop screens, to OLED tablets and smartphones, to LCoS, DLP, and MEMS scanner digital projectors, and to iLED smartwatches (see Figure 1).

Figure 1: Immersive NTE displays: a paradigm shift in personal information display. NTE, near-to-eye.
Figure 1:

Immersive NTE displays: a paradigm shift in personal information display. NTE, near-to-eye.

When flat-panel display technologies and architectures (smartphone or microdisplay panels) are used to implement immersive near-to-eye (NTE) display devices, factors such as etendue, static focus, low contrast, and low brightness become severe limitations. Alternative display technologies are required to address the needs of NTE immersive displays to match the specifics of the human visual system.

The emergence of the second VR/AR/smart glasses boom in the early 2010s introduced new naming trends, more inclusive than AR or VR: mixed (or merged) reality (MR), more generally known today as “XR,” a generic acronym for “extended reality.” The name “smart eyewear” (world locked audio, digital monocular display, and prescription eyewear) tends to replace the initial “smart glass” naming convention.

Figure 2 represents the global MR spectrum continuum, from the real-world experience toward diminished reality (where parts of reality are selectively blocked through hard edge occlusion, such as annoying advertisements while walking or driving through a city, to blinding car headlights while cruising at night on a highway) to AR as in optically see-through MR, to merged reality as in video see-through MR, and to pure virtual worlds (as in VR).

Figure 2: Mixed-reality spectrum continuum.
Figure 2:

Mixed-reality spectrum continuum.

Parallel realities are a new concept that emerged recently with specific optical display hardware, creating from a single-display hardware-specific individual eyeboxes (EBs) with specific information, targeted at multiple viewers detected and tracked through biometrics by sensors around that same display. These dynamic EBs are steered in real time to follow the specific viewers. This is not a wearable display architecture, rather a monitor or transparent window display. In this scenario, different viewers of that same physical display see different information, tuned to their specific interest, depending on their physical location.

2 The emergence of MR as the next computing platform

Smart glasses (also commonly called smart eyewear or digital eyewear) are mainly an extension of prescription eyewear, providing a digital contextual display as an addition to vision prescription correction (see for example Google Glass). This concept is functionally very different from either AR or MR functionality. The typical smart glass field of view remains small (<15°diagonal), is typically monocular, and is often offset from the line of sight. The lack of sensors (apart the IMU) allows for approximate 3 degrees of freedom (3DOF) head tracking, and lack of binocular vision reduces the display to simple, overlaid 2D text and images. Typical 3DOF content is locked relative to the head, while 6 degrees of freedom (6DOF) sensing allows the user to get further and closer to the content.

Monocular displays do not require as much rigidity in the frames as a binocular vision system would (to reduce horizontal and vertical retinal disparity that can produce eye strain). Many smart glass developers also provide prescription correction as a standard feature (e.g., “Focal” by North or Google Glass V2).

The combination of strong connectivity (3G, 4G, WiFi, Bluetooth) and a camera makes it a convincing companion to a smartphone, for contextual display functionality or as a virtual assistant, acting as a global positioning system (GPS)-enabled social network companion. A smart glass does not aim to replace a smartphone, but it intends to contribute as a good addition to it, like a smartwatch.

VR headsets are an extension of simulators and gaming consoles, as shown by major gaming providers such as Sony, Oculus, HTC Vive, and Microsoft Windows MR, with gaming companies such as Valve Corp providing a gaming content ecosystem (Steam VR). The offerings have bifurcated into high-performance personal computer (PC)-tethered headsets (Samsung Odessey, HTC Vive Pro, Oculus Rift) and mobile-first, stand-alone experiences (Oculus Quest). Pancake optics and hybrid lenses will continue to push the form factor of these devices down.

AR and especially MR systems are poised to become the next computing platform, replacing ailing desktop and laptop hardware, and now even the aging tablet computing hardware. Such systems are mostly untethered for most of them (see HoloLens 1) and require high-end optics for the display engine, combiner optics, and sensors (depth scanner camera, head-tracking cameras to provide 6DOF, accurate eye trackers, and gesture sensors). These are currently the most demanding headsets in terms of hardware, especially optical hardware, and are the basis of this review paper.

Eventually, if technology permits, these three categories will merge into a single hardware concept. This will, however, require improvements in connectivity (5G, WiGig), visual comfort (new display technologies), and wearable comfort (battery life, thermal management, weight/size).

The worldwide sales decline for smartphones and tablets in Q3 2018 was an acute signal for major consumer electronics corporations and VC firms to fund and develop the “next big thing.” MR headsets (in all their forms as glasses, goggles or helmets), along with 5G connectivity and subsequent cloud MR services, look like good candidates for many.

2.1 Wearable, visual, vestibular, and social comfort

Comfort, in all four declinations—wearable, visual, vestibular, and social—is key to enabling a large acceptance base of any consumer MR headset candidate architecture. Comfort, especially visual, is a subjective concept. Its impact is therefore difficult to measure or even estimate on a user pool. Careful user testing is required to assess.

Wearable comfort features include the following:

  1. Untethered headset for best mobility.

  2. Small size and light weight.

  3. Thermal management throughout the entire headset (passive or active).

  4. Skin contact management through pressure points.

  5. Breathable fabrics to manage sweat and heat.

  6. Center of gravity (CG) closer to the CG of a human head.

Visual comfort features include the following:

  1. Large EB to allow for wide interpupillary distance (IPD) coverage. The optics might also come in different stock keeping units (SKUs) for consumers (i.e., small, medium, and large IPDs), but for enterprise, because the headset is shared between employees, it needs to accommodate a wide IPD range.

  2. Angular resolution close to 20/20 visual acuity (at least 45 pixels per degree [PPD] in the central foveated region), lowered to a few PPD in the peripheral visual region.

  3. No screen-door effects (large pixel fill factor and high PPD), and no Mura effects.

  4. High dynamic range through high brightness and high contrast (emissive displays such as MEMS scanners and OLEDs/iLEDs versus nonemissive displays such as LCoS and LCD).

  5. Ghost images minimized (<1%).

  6. Unconstrained 200+ degree see-through peripheral vision (especially useful for outdoor activities, defense, and civil engineering).

  7. Active dimming on visor (uniform shutter or soft-edge dimming).

  8. Display brightness control (to accommodate various environmental lightning conditions).

  9. Reduction of any blue remaining ultraviolet (UV) or blue LED light (<415 nm) to limit retinal damage.

  10. Color accuracy and color uniformity over FOV as well as EB are also important vision comfort keys.

Vestibular comfort features include the following:

  1. Motion-to-photon latency (time between head movement and display update) below 10 ms (through optimized sensor fusion).

  2. Spatial stability of holograms in the 3D world across both low and high frequencies.

  3. User experience/user interface considerations to present content motion do not severely disagrees with a user’s sense of motion.

Visual comfort features leveraging eye tracking include the following:

  1. Vergence–accommodation conflict (VAC) mitigation for close objects located in the foveated cone through vergence tracking from differential eye tracking data (as vergence is the trigger to accommodation).

  2. Active pupil swim correction for large-FOV optics.

  3. Active pixel occlusion (hard-edge occlusion) to increase hologram opacity (more realistic).

Social comfort features include the following:

  1. Unaltered eye view of the HMD wearer, allowing for continuous eye contact and eye expression discernment.

  2. No world-side image extraction (present in many waveguide combiners).

  3. Covert multiple-sensor objective cameras pointing to the world (reducing socially unacceptable world spying).

Note: The word “hologram” is used extensively by the AR/VR/MR community as referring to “stereo images.” For the optical engineer, a hologram is either the volume holographic media (dichromated gelatin [DCG] emulsion, Silver Halide film or Photopolymers films, surface relief element, and so on) that can store phase and/or amplitude information as a phase and/or amplitude modulation or the representation of a true diffracted holographic field, forming an amplitude image, a phase object, or a combination thereof. A hologram in the original sense of the world can thus be also an optical element, such as a grating, a lens, a mirror, a beam shaper, a filter, a spot array generator, and so on. However, throughout this review work, we conform to the new (albeit deformed by the AR/VR/MR community) meaning of the world “hologram” as a stereo image.

2.2 Display immersion

Immersion is the other key to the ultimate MR experience and is not based only on FOV, which is a 2D angular concept; immersive FOV is a 3D concept that includes the z distance from the user’s eyes, allowing for arm’s length display interaction through VAC mitigation.

Immersive experiences can come in various forms:

  1. Wide-angle FOV, including peripheral display regions with lower pixels count per degree (resolution) and lower color depth.

  2. Foveated display that is either fixed/static (foveated rendering) or dynamic (through display steering, mechanically or optically).

  3. World-locked holograms, hologram occlusion through accurate and fast spatial mapping and hard-edge see-through occlusion.

  4. World-locked spatial audio.

  5. Accurate eye/gesture/brain sensing through dedicated sensors.

  6. Vivid and realistic hologram colors and shading.

  7. Haptic feedback.

3 Functional optical building blocks of an MR headset

An HMD, and particularly an optically see-through HMD, is a complex system, with at its core various optical subsystems. Once the optical subsystems are defined, such as the choice of the optical engine, the combiner engine and the optical sensors (eye tracking, head tracking, depth scanner, gesture sensors, and so on), all the rest can be engineered around this core.

A typical functional optical building block ecosystem of a MR headset is shown in Figure 3, including display, imaging, and sensing subsystems.

Figure 3: Functional optical building blocks of an MR system.
Figure 3:

Functional optical building blocks of an MR system.

The display engine is where the image is formed and then imaged onwards, forming or not a pupil, and passed through an optical combiner that can include a pupil replication scheme to the eye pupil. Gaze tracking might or might not share optics with the display architecture (which is usually an infinite conjugate system, and eye tracking is usually a finite conjugate system).

3.1 Display engine optical architectures

Once the image is formed over a plane, a surface, or through a scanner, there is a need to form an exit pupil, over which the image is either totally or partially collimated and then presented directly to the eye or to an optical combiner (see Figure 4 for the display engine architecture and subsequent waveguide combiner for HoloLens 1 and HoloLens 2) . In some cases, an intermediate aerial image can be formed to increase the etendue of the system.

Figure 4: Display engines based on an LCoS imager, as in the HoloLens 1 (top, 2016), and a laser MEMS scanner, as in the HoloLens 2 (bottom, 2019).
Figure 4:

Display engines based on an LCoS imager, as in the HoloLens 1 (top, 2016), and a laser MEMS scanner, as in the HoloLens 2 (bottom, 2019).

Because the waveguide input pupils for both eyes are located in the upper nasal area in HoloLens 1, several optical elements of the display engine have been shared with both display engines in order to reduce any binocular image misalignments. In the HoloLens 2, this is not the case since the input pupils are centrally located on the waveguide (as the field propagates by total internal reflection [TIR] in both directions in the guides).

Spatially demultiplexed exit pupils (either color or field separated) can be an interesting option, depending on the combiner architecture used (see the Magic Leap One). Imaging optics or relay optics in the display engine are usually free-space optics but in very compact form, including in many cases polarization beam cubes combined with birdbath architectures [12] to fold the optical path in various directions. Reflective/catadioptric optics are also preferred for their reduced achromatic spread.

3.2 Combiner optics and exit pupil expansion

The optical combiner is often the most complex and most costly optical element in the entire MR display architecture: it is the one component seen directly by the user and the one seen directly by the world. It often defines the size-and-aspect ratio of the entire headset. It is the critical optical element that reduces the quality of the see-through and the one that defines the EB size (and in many cases, also the FOV).

There are three main types of optical combiners used in most MR/AR/smart glasses today:

  1. Free-space optical combiners,

  2. TIR prism optical combiners (and compensators), and

  3. Waveguide-based optical combiners.

When optimizing an HMD display system, the optical engine must be optimized in concert with the combiner engine. Usually, a team that designs an optical engine without fully understanding the limitations and specifics of a combiner engine designed by another team, and vice versa, can result in a suboptimal system or even a failed optical architecture, no matter how well the individual optical building blocks might be designed.

4 Waveguide combiners

Free-form TIR prism combiners are at the interface between free space and waveguide combiners. When the number of TIR bounces increases, one might refer to them as waveguide combiners. Waveguide combiner architectures are the topic of this review paper.

Waveguide combiners are based on TIR propagation of the entire field in an optical guide, essentially acting as a transparent periscope with a single entrance pupil and often many exit pupils.

The primary functional components of a waveguide combiner consist of the input and output couplers. These can be either simple prisms, microprism arrays, embedded mirror arrays, surface relief gratings (SRGs), thin or thick analog holographic gratings, metasurfaces, or resonant waveguide gratings (RWGs). All of these have their specific advantages and limitations, which will be discussed here. Waveguide combiners have been used historically or tasks very different from AR combiner, such as planar optical interconnections [13] and LCD backlights [14].

Waveguide combiners are an old concept, some of the earliest intellectual property (IP) dates back to 1976 and applied to HUDs. Figure 5(a) shows a patent by Juris Upatnieks dating back 1987, a Latvian/American scientist and one of the pioneers of modern holography [16], implemented in a DCG holographic media. A few years later, 1D EB expansion (1D exit pupil expansion [EPE]) architectures were proposed as well as a variety of alternatives for in-coupler and out-coupler technologies, such as SRG couplers by Thomson CSF (Figure 5(b)). Figure 5(c) shows the original 1991 patent for a waveguide-embedded partial mirror combiner and exit pupil replication. (All of these original waveguide combiner patents have been in the public domain for nearly a decade.)

Figure 5: (a) Original waveguide combiner patents including holographic (1987), (b) surface relief grating (1989), and (c) partial mirrors (1991) for HUD and HMD applications.
Figure 5:

(a) Original waveguide combiner patents including holographic (1987), (b) surface relief grating (1989), and (c) partial mirrors (1991) for HUD and HMD applications.

4.1 Curved waveguide combiners and a single exit pupil

If the FOV is small (<20° diagonally), such as in smart glasses, it might not be necessary to use an exit pupil expansion architecture, which would make the waveguide design much simpler and allow for more degrees of freedom, such as curving the waveguide. Indeed, if there is a single output pupil, the waveguide can imprint optical power onto the TIR field, as is done in the curved waveguide Smart Glass by Zeiss in Germany (developed now with Deutsche Telekom and renamed “Tooz”); see Figure 6.

Figure 6: Zeiss “Tooz” smart glass with single exit pupil allowing for curved waveguide.
Figure 6:

Zeiss “Tooz” smart glass with single exit pupil allowing for curved waveguide.

The other waveguide smart glass shown here (flat waveguide cut as a zero-diopter ophthalmic lens) is the early prototype (1995) from Micro-Optical Corp. in which the extractor is an embedded coated prism.

In the Zeiss “Tooz” smart glass, the exit coupler is an embedded off-axis Fresnel reflector. The FOV as well as the out-coupler is excentered from the line of sight. The FOV remains small (11°) and the thickness of the guide relatively thin (3–4 mm).

Single exit pupils have also been implemented in flat guides, as in the Epson Moverio BT100, BT200, and BT300 (temple-mounted optical engine in a 10-mm-thick guide with curved half-tone extractor in the BT300) or in the Konica Minolta smart glasses, with top-down display injection and a flat RGB panchromatic volume holographic extractor (see Figure 7).

Figure 7: Single-exit-pupil flat waveguide combiners (with curved reflective or flat holographic out-couplers).
Figure 7:

Single-exit-pupil flat waveguide combiners (with curved reflective or flat holographic out-couplers).

Single exit pupils (no EPE) are well adapted to small-FOV smart glasses. If the FOV gets larger than 20°, especially in a binocular design, 1D or 2D exit pupil replication is required.

Covering a large IPD range (such as a 95 or 98 percentile of the target consumer population, including various facial types) requires a large horizontal EB, typically 10–15 mm. Also, due to fit issues and nose-pad designs, a similar large and vertical EB is also desirable, ranging from 8–12 mm.

4.2 Continuum from flat to curved waveguides and extractor mirrors

One can take the concept of a flat waveguide with a single curved extractor mirror (Epson Moverio BT300) or free-form prism combiner, or a curved waveguide with curved mirror extractor, to the next level by multiplying the mirrors to increase the EB (see the Lumus lightguide optical element [LOE] waveguide combiner) or fracturing metal mirrors into individual pieces (see the Optinvent ORA waveguide combiner or the LetinAR waveguide combiner)

While fracturing the same mirror into individual pieces can increase see through and depth of focus, the use of more mirrors to replicate the pupil is a bit more complicated, especially in a curved waveguide where the two exit pupils need to be spatially demultiplexed to provide a specific mirror curvature to each pupil to correct for image position: this limits the FOV in one direction so that such overlap does not happen.

Figure 8 summarizes some of the possible design configurations with such waveguide mirror architectures. Note that the grating- or holographic-based waveguide combiners are not listed here; they are the subject of the next sections.

Figure 8: Multiplying or fracturing the extractor mirrors in flat or curved waveguides.
Figure 8:

Multiplying or fracturing the extractor mirrors in flat or curved waveguides.

Figure 8 shows that many of the waveguide combiner architectures mentioned in this section can be listed in this table. Mirrors can be half tone (Google Glass, Epson Moverio), dielectric (Lumus LOE), have volume holographic reflectors (Luminit or Konica Minolta), or the lens can be fractured into a Fresnel element (Zeiss Tooz Smart Glass). In the Optinvent case, we have a hybrid between fractured metal mirrors and cascaded half-tone mirrors. In one implementation, each microprism on the waveguide has one side fully reflective and the other side transparent to allow see through.

In the LetinAR case, all fractured mirrors are reflective, can be flat or curved, and can be inverted to work with a birdbath reflective lens embedded in the guide.

Even though the waveguide might be flat, when using multiple lensed mirrors, the various lens powers will be different since the display is positioned at different distances from these lensed extractors. When the waveguide is curved, everything becomes more complex, and the extractor mirror lenses need also to compensate for the power imprinted on the TIR field at each TIR bounce in the guide. In the case of curved mirrors (either in flat or curved waveguides), the exit pupils over the entire field cannot overlap since the power to be imprinted on each exit pupil (each field position) is different (Moverio BT300 and Zeiss Tooz Smart Glass). This is not the case when the extractors are flat and the field is collimated in the guide (Lumus LOE).

4.3 One-dimensional EB expansion

As the horizontal EB is usually the most critical to accommodate large IPD percentiles, a single-dimensional exit pupil replication might suffice. The first attempts used holographic extractors (Sony Ltd.) [18] with efforts to record RGB holographic extractors as phase-multiplexed volume holograms [19] and also as cascaded half-tone mirror extractors (LOE from Lumus, Israel) or arrays of microprisms (Optinvent, France) [69]. This reduced the 2D footprint of the combiner, which operates only in one direction.

However, to generate a sufficiently large EB in the nonexpanded direction, the input pupil produced by the display engine needs to be quite large in this same direction—larger than the exit pupil in the replicated direction. In many cases, a tall-aspect-ratio input pupil can lead to larger display engines such as in the 1D EPE Lumus LCoS–based enginers. However, a single vertical pupil with natural expansion will provide the best imaging and color uniformity over the EB.

The Lumus LOE has been integrated in various AR glasses at Lumus, as well as in many third-party AR headsets (Daqri, Atheer Labs, Flex, Lenovo). The Lumus LOE can operate in either the vertical direction with the display engine located on the forehead (DK Vision). Lumus is also working on a 2D expansion scheme for its LOE line of combiners (Maximus), with central or lateral input pupils, allowing for a smaller light engine (as the light-engine exit pupil can be symmetric due to 2D expansion). Similarly, the Sony 1D holographic waveguide combiner architecture has been implemented in various products, such as Univet and SwimAR (both using Sony SED 100A waveguide).

4.4 Two-dimensional EB expansion

Two-dimensional EB expansion is desired (or required) when the input pupil cannot be generated by the optical engine over an aspect ratio tall enough to form the 2D EB because of the FOV (etendue limitations) and related size/weight considerations. A 2D EPE is therefore required (see Figure 9).

Figure 9: 2D pupil replication architectures in planar optical waveguide combiners.
Figure 9:

2D pupil replication architectures in planar optical waveguide combiners.

Various types of 2D EPE replication have been developed: from cascaded X/Y expansion (as in the Digilens, Nokia, Vuzix, HoloLens, and Magic Leap One combiner architectures [21], [22], [23]) to combiner 2D expansion [24], [26] (as in the BAE Q-Sight combiner or the WaveOptics Ltd. Phlox 40-degree grating combiner architectures, see Figure 10), to more complex spatially multiplexed gratings (as in the Dispelix combiner).

Figure 10: Smart glasses and AR headsets that use 2D EPE diffractive or holographic waveguide combiners.
Figure 10:

Smart glasses and AR headsets that use 2D EPE diffractive or holographic waveguide combiners.

While holographic recording or holographic volume gratings are usually limited to linear gratings or gratings with slow power (such as off-axis diffractive lenses), SRGs can be either 1D or 2D and either linear or quasi arbitrary in shape. Such structures or structure groups can be optimized by iterative algorithms (topological optimization) rather than designed analytically (WaveOptics computer-generated holograms [CGHs] or Dispelix “mushroom forest” gratings).

Some of these combiners use one guide per color, some use two guides for all three colors, and some use a single guide for RGB; some use glass guides, and others use plastic guides, along with the subsequent compromises one has to make on color uniformity, efficiency, EB, and FOV.

Next, we point out the differences between the various coupler elements and waveguide combiner architecture used in such products. We will also review new coupler technologies that have not yet been applied to enterprise or consumer products. While the basic 2D EPE expansion technique might be straightforward, we will discuss alternative techniques that can allow a larger FOV to be processed by both in-coupler and out-couplers (either as surface gratings or volume holograms). Finally, we will review the mastering and mass replication techniques of such waveguide combiners to allow scaling and consumer cost levels.

4.5 Choosing the right waveguide coupler technology

The coupler element is the key feature of a waveguide combiner. The TIR angle is dictated by the refractive index of the waveguide, not the refractive index of the coupler nanostructures. Very often, the index of the coupler structure (grating or hologram) prescribes the angular and spectral bandwidth over which this coupler can act, thus impacting the color uniformity over the FOV and EB.

Numerous coupler technologies have been used in industry and academia to implement the in-couplers and out-couplers, and they can be defined either as refractive/reflective or diffractive/holographic coupler elements.

4.5.1 Refractive/reflective couplers elements

4.5.1.1 Macroscopic prism

A prism is the simplest TIR in-coupler and can be very efficient. A prism can be bounded on top of the waveguide, or the waveguide itself can be cut at an angle, to allow normal incident light to enter the waveguide and be guided by TIR (depending on the incoming pupil size). Another way uses a reflective prism on the bottom of the waveguide (metal coated). Using a macroscopic prism as an out-coupler is not impossible, and it requires a compensating prism for see through, with either a reflective coating or a low-index glue line, as done in the Oorym (Israel) light guide combiner concept.

4.5.1.2 Embedded cascaded mirrors

Cascaded embedded mirrors with partially reflective coatings are used as out-couplers in the Lumus (Israel) LOE waveguide combiner. The input coupler remains a prism. As the LOE is composed of reflective surfaces, it yields good color uniformity over the entire FOV. As with other coupler technologies, intrinsic constraints in the cascaded mirror design of the LOE might limit the FOV [26]. See through is very important in AR systems: the Louver effects produced by the cascaded mirrors in earlier versions of LOEs have been reduced recently thanks to better cutting/polishing, coating, and designing.

4.5.1.3 Embedded microprism array

Microprism arrays are used in the Optinvent (France) waveguide as out-couplers [20]. The in-coupler here is again a prism. Such microprism arrays can be surface relief or index matched to produce an unaltered see-through experience. The microprisms can all be coated uniformly with a half-tone mirror layer or can have an alternance of totally reflective and transmissive prism facets, provide a resulting 50% transmission see-through experience. The Optinvent waveguide is the only flat waveguide available today as a plastic guide, thus allowing for a consumer-level cost for the optics. The microprism arrays are injection molded in plastic and bounded on top of the guide.

4.5.2 Diffractive/holographic couplers elements

4.5.2.1 Thin reflective holographic coupler

Transparent volume holograms working in reflection mode—as in DCG, bleached silver halides (Slavic or Ultimate Holography by Yves Gentet), or more recent photopolymers such as Bayfol® photopolymer by Covestro/Bayer, (Germany) [27], and photopolymers by DuPont (US), Polygrama (Brazil), or Dai Nippon (Japan)—have been used to implement in-couplers and out-couplers in waveguide combiners. Such photopolymers can be sensitized to work over a specific wavelength or over the entire visible spectrum (panchromatic holograms).

Photopolymer holograms do not need to be developed as DCG, nor do they need to be bleached like silver halides. A full-color hologram based on three phase-multiplexed single-color holograms allows for a single plate waveguide architecture, which can simplify the combiner and reduce weight, size, and costs while increasing yield (no plate alignment required). However, the efficiency of such full-RGB phase-multiplexed holograms is still quite low when compared to single-color photopolymer holograms.

Also, the limited index swing of photopolymer holograms allows them to work more efficiently in reflection mode than in transmission mode (allowing for better confinement of both the wavelength and angular spectrum bandwidths).

Examples of photopolymer couplers include Sony LMX-001 Waveguides for smart glasses and the TrueLife Optics (UK) process of mastering the hologram in silver halide and replicating it in photopolymer.

Replication of the holographic function in photopolymer through a fixed master has proven to be possible in a roll-to-roll operation by Bayer (Germany). Typical photopolymer holographic media thicknesses range from 16–70 µm, depending on the required angular and spectral bandwidths.

4.5.2.2 Thin transmission holographic coupler

When the index swing of the volume hologram can be increased, the efficiency gets higher and the operation in transmission mode becomes possible. This is the case with Digilens’ proprietary holographic polymer-dispersed liquid crystal (H-PDLC) hologram material [28]. Transmission mode requires the hologram to be sandwiched between two plates rather than laminating a layer on top or bottom of the waveguide as with photopolymers, DCG, or silver halides. Digilens’ H-PDLC has the largest index swing today and can therefore produce strong coupling efficiency over a thin layer (typically four microns or less). H-PDLC material can be engineered and recorded to work over a wide range of wavelengths to allow full-color operation.

4.5.2.3 Thick holographic coupler

Increasing the index swing can optimize the efficiency and/or angular and spectral bandwidths of the hologram. However, this is difficult to achieve with most available materials and might also produce parasitic effects such as haze. Increasing the thickness of the hologram is another option, especially when sharp angular or spectral bandwidths are desired, such as in telecom spectral and angular filters. This is not the case for an AR combiner, where both spectral and bandwidths need to be wide (to process a wide FOV over a wide spectral band such as LEDs). However, a thicker hologram layer also allows for phase multiplexing over many different holograms, one on top of another, allowing for multiple Bragg conditions to operate in concert to build a wide synthetic spectral and/or angular bandwidth, as modeled by the Kogelnik theory [30]. This is the technique used by Akonia, Inc. (a US start-up in Colorado, formerly InPhase Inc., which was originally funded and focused to produce high-density holographic page data-storage media, ruled by the same basic holographic phase-multiplexing principles [29]).

Thick holographic layers, as thick as 500 µm, work well in transmission and/or reflection modes, but they need to be sandwiched between two glass plates. In some specific operation modes, the light can be guided inside the thick hologram medium, where it is not limited by the TIR angle dictated by the index of the glass plates. As the various hologram bandwidths build the final FOV, one needs to be cautious in developing such phase-multiplexed holograms when using narrow illumination sources such as lasers.

Replication of such thick volume holograms is difficult in roll-to-roll operation, as done with thinner single holograms (Covestro Photopolymers, H-PDLC), and require multiple successive exposures to build the hundreds of phase-multiplexed holograms that compose the final holographic structure. This can however be relatively easy with highly automated recording setups as the ones developed by the now-defunct holographic page data-storage industry (In-Phase Corp., General Electric, and so on).

Note that although the individual holograms acting in slivers of angular and spectral bandwidth spread the incoming spectrum like any other hologram (especially when using LED illumination), the spectral spread over the limited spectral range of the hologram is not wide enough to alter the modulation transfer function (MTF) of the immersive image and thus does not need to be compensated by a symmetric in-coupler and out-coupler as with all other grating or holographic structures. This feature allows this waveguide architecture to be asymmetric, such as having a strong in-coupler as a simple prism: a strong in-coupler is always a challenge for any grating or holographic waveguide combiner architecture, and a macroscopic prism is the best coupler imaginable.

Figure 11 shows both thin and thick volume holograms operating in reflection and/or transmission modes. The top part of the figure shows a typical 1D EPE expander with a single transmission volume hologram sandwiched between two plates. When the field traverses the hologram downward, it is in off/Bragg condition, and when it traverses the volume hologram upward after a TIR reflection, it is in an on/Bragg condition (or close to it), thereby creating a weak (or strong) diffracted beam that breaks the TIR condition.

Figure 11: Different types of volume holograms acting as in-couplers and out-couplers in waveguide combiners.
Figure 11:

Different types of volume holograms acting as in-couplers and out-couplers in waveguide combiners.

A hologram sandwiched between plates might look more complex to produce than a reflective or transmission laminated version, but it has the advantage that it can operate in both transmission and reflection modes at the same time (e.g., to increase the pupil replication diversity).

4.5.2.4 Surface-relief grating couplers

Figure 12 reviews the various SRGs used in industry today (blazed, slanted, binary, multilevel, and analog) and how they can be integrated in waveguide combiners as incoupling and outcoupling elements.

Figure 12: Surface-relief grating types used as waveguide combiner in-couplers and out-couplers. Solid lines indicate reflective coatings on the grating surface, and dashed lines indicate diffracted orders.
Figure 12:

Surface-relief grating types used as waveguide combiner in-couplers and out-couplers. Solid lines indicate reflective coatings on the grating surface, and dashed lines indicate diffracted orders.

Covering a SRG with a reflective metallic surface (see Figure 12) will increase dramatically its efficiency in reflection mode. A transparent grating (no coating) can also work both in transmission and reflection modes, especially as an out-coupler, in which the field has a strong incident angle.

Increasing the number of phase levels from binary to quarternary or even eight or sixteen levels increases its efficiency as predicted by the scalar diffraction theory, for normal incidence. However, for a strong incidence angle and for small periods, this is no longer true. A strong outcoupling can thus be produced in either reflection or transmission mode.

Slanted gratings are very versatile elements, and their spectral and angular bandwidths can be tuned by the slant angles. Front and back slant angles in a same period (or from period to period) can be carefully tuned to achieve the desired angular and spectral operation.

SRGs have been used as a commodity technology since mastering and mass replication techniques technologies were established and made available in the early 1990s [39]. Typical periods for TIR grating couplers in the visible spectrum are below 500 nm, yielding nanostructures of just a few tens of nanometers if multilevel structures are required. This can be achieved by direct e-beam write, i-line (or deep ultra-violet [DUV]) lithography, or even interference lithography (holographic resist exposure) [37]. SRG structures can be replicated in volumes by nanoimprint, a microlithography wafer fabrication technology developed originally for the integrated circuit (IC) industry. Going from wafer-scale fabrication to panel-scale fabrication will reduce costs, allowing for consumer-grade AR and MR products.

Figure 13 and Figure 14 illustrate how some of the SRGs shown in Figure 12 have been applied to the latest waveguide combiners such as the Microsoft HoloLens 1 and Magic Leap One. Multilevel SRGs have been used by companies such as Dispelix Oy, and quasi-analog surface relief CGHs have been used by others, such as WaveOptics Ltd.

Figure 13: Spatially color-multiplexed input pupils with slanted gratings as in-couplers and out-couplers working in transmission and reflection mode (HoloLens 1 MR headset).
Figure 13:

Spatially color-multiplexed input pupils with slanted gratings as in-couplers and out-couplers working in transmission and reflection mode (HoloLens 1 MR headset).

Figure 14: Spatially color-demultiplexed input pupils with 100% reflective blazed gratings as in-couplers and binary phase gratings as out-couplers (Magic Leap One MR headset).
Figure 14:

Spatially color-demultiplexed input pupils with 100% reflective blazed gratings as in-couplers and binary phase gratings as out-couplers (Magic Leap One MR headset).

Figure 13 shows the waveguide combiner architecture used in the Microsoft HoloLens 1 MR headset (2015). The display engine is located on the opposite side of the EB. The single input pupil carries the entire image over the various colors at infinity (here, only two colors and the central field are depicted for clarity), as in a conventional digital projector architecture. The in-couplers have been chosen to be slanted gratings for their ability to act on a specific spectral range while letting the remaining spectrum unaffected in the zero order, to be processed by the next in-coupler area located on the guide below, and to do this for all three colors. Such uncoated slanted gratings work both in transmission and reflection modes but can be optimized to work more efficiently in a specific mode. The out-couplers here are also slanted gratings, which can be tuned to effectively work over a specific incoming angular range (TIR range) and leave the see-through field quasi unaffected. The part of the see through field that is indeed diffracted by the out-couplers is trapped by TIR and does not make it to the EB. These gratings are modulated in-depth to provide a uniform EB to the user. Note the symmetric in-coupler and out-coupler configuration compensating the spectral spread over the three LED bands.

The redirection gratings are not shown here. Input and output grating slants are set close to 45°, and the redirection grating slants at half this angle. The periods of the gratings are tuned in each guide to produce the right TIR angle for the entire FOV for that specific color (thus the same central diffraction angle in each guide for each RGB LED color band).

Figure 14 depicts the waveguide combiner architecture used in the Magic Leap One MR headset (2018). The display engine is located on the same side as the EB. The input pupils are spatially color demultiplexed, carrying the entire FOV at infinity (here again, only two colors and the central field are depicted for clarity).

Spatial color demultiplexing can be done conveniently with a color sequential LCoS display mode for which the illumination LEDs are also spatially demultiplexed. In this configuration, the input grating couplers are strong blazed gratings, coated with a reflective metal (such as Al). They do not need to work over a specific single-color spectral width since the colors are already demultiplexed. The out-couplers are simple top-down binary gratings, which are also depth modulated to produce a uniform EB for the user. These binary gratings are shallow, acting therefore very little on the see through, but they have much stronger efficiency when working in internal reflection diffraction mode, since the optical path length in this case is longer by a factor of 2ncos(α) than that in transmission mode, (where n is the index of the guide, and α is the angle if there is incidence in the guide). As in the HoloLens 1, most of the see-through field diffracted by the out-couplers is trapped by TIR.

The redirection gratings (not shown here) are also composed of binary top-down structures. The periods of the gratings are tuned in each guide to produce the right TIR angle for the entire FOV for that specific color (same central diffraction angles for each RGB LED color band).

Other companies such as WaveOptics in the UK uses multilevel and/or quasianalog surface relief diffractive structures to implement in-couplers and out-couplers (see Figure 14). This choice is mainly driven by the complexity of the extraction gratings, acting both as redirection gratings and out-coupler gratings, making them therefore more complex than linear or slightly curved (powered) gratings, similar to iteratively optimized CGHs [40]. Allowing multilevel or quasianalog surface relief diffractive structures increases the space bandwidth product of the element to allow more complex optical functionalities to be encoded with relatively high efficiency.

4.5.2.5 RWG couplers

RWGs, also known as guided mode resonant gratings or waveguide-mode resonant gratings [41], are dielectric structures where these resonant diffractive elements benefit from lateral leaky guided modes. A broad range of optical effects are obtained using RWGs such as waveguide coupling, filtering, focusing, field enhancement and nonlinear effects, magneto-optical Kerr effect, or electromagnetically induced transparency. Thanks to their high degree of optical tuning (wavelength, phase, polarization, intensity) and the variety of fabrication processes and materials available, RWGs have been implemented in a broad scope of applications in research and industry. RWGs can therefore also be applied as in-couplers and out-couplers for waveguide gratings.

Figure 15 shows an RWG on top of a lightguide (referred often incorrectly through the popular AR lingo as a “waveguide”), acting as the in-couplers and out-couplers.

Figure 15: Resonant waveguide gratings as in-couplers and out-couplers on a waveguide combiner.
Figure 15:

Resonant waveguide gratings as in-couplers and out-couplers on a waveguide combiner.

Roll-to-roll replication of such grating structures can help bring down overall waveguide combiner costs. The CSEM research center in Switzerland developed the RWG concept back in the 1980s, companies are now actively developing such technologies [90].

4.5.2.6 Metasurface couplers

Metasurfaces are a hot topic in research: they can implement various optical element functionality in an ultraflat form factor by imprinting a specific phase function over the incoming wavefront in reflection or transmission (or both) so that the resulting effect is refractive, reflective, or diffractive or a combination of them. This phase imprint can be done through a traditional optical path difference phase jump or through Pancharatnam–Berry phase gratings/holograms.

Due to their large design space, low track length, and ability to render unconventional optical functions, metalenses could grow out of the laboratory to become an unique item in the engineer’s bag of tools. If one can implement in a fabricable metasurface an optical functionality that cannot be implemented by any other known optical element (diffractives, holographics, or Fresnels), it is particularly interesting. For example, having a true achromatic optical element is very desirable not only in imaging but also in many other tasks such as waveguide coupling. Another example is ultralow track length focal stack for IR cameras from Metalenz Corp. Additionally, if one can simplify the fabrication and replication process by using metasurfaces, the design for manufacturing (DFM) can be compelling. However, optical efficiencies, design tools, and large scale fabrication will need to continue to improve find their way into product.

4.5.3 Achromatic coupler technologies

Waveguide combiners could benefit greatly from a true achromatic coupler functionality: incoupling and/or outcoupling RGB FOVs and matching each color FOV to the maximum angular range (FOV) dictated by the waveguide TIR condition. This would reduce the complexity of multiple waveguide stacks for RGB operation.

When it comes to implementing a waveguide coupler as a true achromatic grating coupler, one can either use embedded partial mirror arrays (as in the Lumus LOE combiner), design a complex hybrid refractive/diffractive prism array, or even record phase-multiplexed volume holograms in a single holographic material. However, in the first case, the 2D exit pupil expansion implementation remains complex; in the second case, the microstructures can get very complex and thick; and in the third case, the diffraction efficiency can drop dramatically (as in the Konica Minolta or Sony RGB photopolymers combiners or in the thick Akonia holographic dual photopolymer combiner, now part of Apple, Inc.).

It has been recently demonstrated in literature that metasurfaces can be engineered to provide a true achromatic behavior in a very thin surface with only binary nanostructures [43]. It is easier to fabricate binary nanostructures than complex analog surface relief diffractives, and it is also easier to replicate them by nanoimprint lithography (NIL) or soft lithography and still implement a true analog diffraction function as a lens or a grating. The high index contrast required for such nanostructures can be generated by either direct imprint in high index inorganic spin-on glass or by NIL resist lift-off after an atomic layer deposition process. Direct dry etching of nanostructured remains a costly option for a product.

It is important to remember that metasurfaces or thick volume holograms are not inherently achromatic elements, and never will be. However, when many narrow band diffraction effects are spatially or phase multiplexed in a metasurface or a thick volume hologram, their overall behavior over a much larger spectral bandwidth can effectively lead the viewer to think they are indeed achromatic: although each single hologram or metasurface operation are strongly dispersive, their cascaded contributions may result in a broadband operation which looks achromatic to the human eye (e.g., the remaining dispersion of each individual hologram or metasurface effect affecting a spectral spread that is below human visual acuity—one arcmin or smaller). It is also possible to phase multiplex surface relief holograms to produce achromatic effects but more difficult than with thick volume holograms or thin metasurfaces.

Mirrors are of course perfect achromatic elements and will therefore produce the best polychromatic MTF (such as with Lumus LOE combiners or LetinAR pin mirror waveguides).

4.5.4 Summary of waveguide coupler technologies

Table 1 summarizes the various waveguide coupler technologies reviewed here, along with their specifics and limitations.

Table 1:

Benchmark of various waveguide coupler technologies.

Waveguide coupler techOperationReflective couplingTransmission couplingEfficiency modulationLensed out- couplerSpectral dispersion.Color uniformityDynamically tunablePolarization maintainingMass productionCompany/Product
Embedded mirrorsReflectiveYesNoComplex coatingsNoMinimalGoodNoYesSlicing, coating, polishing.Lumus Ltd. DK50
Micro-prismsReflectiveYesNoCoatingsNoMinimalGoodNoYesInjection moldingOptinvent SaRL. ORA
Surface relief slanted gratingDiffractiveYesYesDepth, duty cycle, slantYesStrongNeeds comp.Possible with LCNoNIL (wafer, plate)Microsoft HoloLens, Vuzix Inc, Nokia…
Surface relief blazed gratingDiffractiveYesNoDepthNoStrongNeeds comp.Possible with LCNoNIL (wafer, plate)Magic Leap One,
Surface relief binary gratingDiffractiveYesYesDepth, dutyYesStrongNeeds comp.Possible with LCNoNIL (wafer, plate)Magic Leap One
Multilevel surface relief gratingDiffractiveYesYesDepth, duty cycleYesStrongNeeds comp.Possible with LCPossible, but difficultNIL (wafer, plate)WaveOptics Ltd, BAE. Dispelix.
Thin photopolymer hologramDiffractiveYesYesIndex swingYes, but difficultStrongNeeds comp.Possible with shearNoNIL (wafer, plate)Sony Ltd, TruelifeOptics Ltd,
H-PDLC volume holographicDiffractiveNoYesIndex swingYes, but difficultStrongOKYes (electrical)(NoExposureDigilens Corp. (MonoHUD)
Thick photopolymer hologramDiffractiveYesYesIndex swingYes, but difficultMinimalOKNoNoMultiple exposureAkonia Corp (now Apple Inc.)
Resonant waveguide gratingDiffractiveYesYesDepth. Duty cycleYesCan be mitigatedNAPossible with LCPossibleRoll to roll NILCSEM/Resonannt screens
Metasurface couplerMostly diffractiveYesYesVariousYesCan be mitigatedNeeds comp.Possible with LCPossibleNIL (wafer, plate)Metalenz Corp.

Although Table 1 shows a wide variety of optical couplers, most of today’s AR/MR/smart glass products are based on only a handful of traditional coupler technologies such as thin volume holograms, slanted SRGs, and embedded half-tone mirrors. The task of the optical designer (or rather the product program manager) is to choose the right balance and the best compromise between coupling efficiency, color uniformity over the EB and FOV, mass production costs, and size/weight.

Figure 16 shows the various coupler elements and waveguide architectures grouped in a single table, including SRG couplers, thin holographic couplers, and thick holographic couplers in three, two, and single flat guides. For geometric waveguide combiners that use embedded mirrors or other reflective/refractive couplers (such as microprisms).

Figure 16: Summary of waveguide combiner architectures with 1D or 2D EPE schemes.
Figure 16:

Summary of waveguide combiner architectures with 1D or 2D EPE schemes.

5 Design and modeling of optical waveguide combiners

Designing and modeling a waveguide combiner is very different from designing and modeling a freespace optical combiner. As conventional ray trace in standard optical CAD tools such as ZemaxTM, CodeVTM, FredTM, or TraceProTM are sufficient to design effective free space and even TIR prism combiners and to design waveguide combiners, especially when using diffractive or holographic couplers, a hybrid ray-trace/rigorous electromagnetic diffraction mode is usually necessary.

The modeling efforts is shared between two different tasks:

  • Local rigorous EM light interaction with micro- and anno-optics couplers (gratings, holograms, metasurfaces, RWGs, and so on).

  • Global architecture design of the waveguide combiner, building up FOV, resolution, color and EB, by the use of more traditional ray trace algorithms.

5.1 Waveguide coupler design, optimization, and modeling

5.1.1 Coupler/light interaction model

Modeling of the angular and spectral Bragg selectivity of volume holograms, thin or thick, in reflection and transmission modes, can be performed with the couple wave theory developed by Kogelnik in 1969 [31], [32].

Similarly, modeling of the efficiency of SRGs can be performed accurately with rigorous coupled-wave analysis (RCWA) [33], [34], especially the Fourier modal method (FMM). The finite difference time domain (FDTD) method—also a rigorous EM nanostructure modeling method—can in many cases be a more accurate modeling technique but also much heavier and more CPU time consuming. However, the FDTD will show all the diffracted fields, the polarization conversions, and the entire complex field, whereas the Kogelnik model and the RCWA will only give efficiency values for particular diffraction orders.

The FDTD can model nonperiodic nanostructures, while RCWA can accurately model quasiperiodic structures. Thus, the FDTD might help with modeling k-vector variations (rolled k-vector) along the grating, slant, depths, and duty cycle variations, as well as random and systematic fabrication errors in the mastering and replication steps. The Kogelnik theory is best suited for slowly varying index modulations with moderate index swings (i.e., photopolymer volume holograms).

Free versions of the RCWA-FMM [35] and FDTD [36] codes can be found on the Internet. The Kogelnik theory can be easily implemented as a straightforward equation set for transmission and reflection modes. Commercial software suites implementing FDTD and RCWA are R-Soft from Synopsys and Lumerical.

These models predict the efficiency in each order for a single interaction of the light with the coupler element. In order to model the entire waveguide combiner, especially when a pupil replication scheme is used, conventional ray tracing optical design software can be used, such as Zemax, or more specific light-propagation software modules, such as the ones by LightTrans, Germany [37] (see Figure 17 for ray tracing through 2D EPE grating waveguides).

Figure 17: Waveguide grating combiner modeling by LightTrans (Germany) in 2D EPE version.
Figure 17:

Waveguide grating combiner modeling by LightTrans (Germany) in 2D EPE version.

The interaction of the EM field with the coupler regions (surface relief structures or index modulations) modeled through the RCWA or Kogelnik can be implemented via a dynamically linked library (DLL) in conventional optical design software based on ray tracing (e.g., C or Matlab code). As the FDTD numerical algorithm propagates the entire complex field rather than predicting only efficiency values (as in the RCWA or Kogelnik model), it is therefore more difficult to implement as a DLL.

Raytrace optimization of the high-level waveguide combiner architecture with accurate EM light/coupler interactions modeling are both required to design a combiner with good color uniformity over the FOV, a uniform EB over a target area at a desired eye relief, and high efficiency (in one or both polarizations). Inverse propagation from the EB to the optical engine exit pupil is a good way to simplify the optimization process. The design process can also make use of an iterative algorithm to optimize color over the FOV/EB and/or efficiency or even reduce the space of the grating areas by making sure that no light gets lost outside the effective EB.

Waveguide couplers have specific angular and spectral bandwidths that affect both the FOV and the EB uniformity. A typical breakdown of the effects of a 2D EPE waveguide architecture on both spectral and angular bandwidths on the resulting immersive display is shown in Figure 18.

Figure 18: Cascaded effects of the field/coupler interactions on the FOV uniformity.
Figure 18:

Cascaded effects of the field/coupler interactions on the FOV uniformity.

Figure 18 shows that the coupler’s spectral and angular bandwidths are critical to the FOV uniformity, especially color uniformity. While embedded mirrors and microprisms have a quasiuniform effect on color and FOV, others do not, such as gratings and holograms. It is therefore interesting to have the flattest and widest spectral and angular bandwidths possible. For volume holograms, this means operating in reflection mode and having a strong index swing (Kogelnik), and for surface gratings, this means a high index (as predicted by the RCWA-FMM or FDTD). The angular bandwidth location can be tuned by the slant angle in both holograms and surface gratings. Multiplexing bandwidths can help to build a larger overall bandwidth, bot spectral, and angular and is used in various implementations today. Such multiplexing can be done in phase, in space, or in time or a combination of the above. Finally, as spectral and angular bandwidths are closely linked, altering the spectral input over the field can have a strong impact on FOV and vice versa.

Polarization and degree of coherence are two other dimensions one should need to investigate especially when lasers or VCSELs are used in the optical engine or if polarization maintaining (or rather polarization conversion) is required. The multiple interactions in the R-E regions can produce multiple miniature Mach–Zehnder interferometers, which might modulate the intensity of the particular fields.

5.1.2 Increasing FOV by using illumination spectrum

The ultimate task for a holographic or grating coupler is to provide the widest FOV coupling possible, matching the FOV limit dictated by the TIR condition in the underlying waveguide (linked to the refractive index of the waveguide material).

We have seen that volume holographic combiners have been used extensively to provide a decent angular incoupling and outcoupling into the guide. However, most of the available holographic materials today have a low index swing and thus yield a relatively small angular bandwidth in the propagation direction. In this case, the FOV bottleneck is the coupler not the TIR condition in the waveguide.

A typical Kogelnik efficiency plot in the angular/spectral space for a reflection photopolymer volume holographic coupler is shown Figure 19 (spectral dimension vertical and angular dimension horizontal).

Figure 19: Spectral source bandwidth building larger FOV (angular bandwidth) for photopolymer volume holographic coupler in waveguide combiners.
Figure 19:

Spectral source bandwidth building larger FOV (angular bandwidth) for photopolymer volume holographic coupler in waveguide combiners.

The hologram specifications and exposure setup in Figure 19 are listed below:

  • Mean holographic material index: 1.53,

  • Holographic index swing: 0.03,

  • Photopolymer thickness: 16 µm,

  • Operation mode: reflective,

  • Polarization: (“s” but very little change when moving to “p” polarization),

  • Design wavelength: 550 nm,

  • Reconstruction wavelength: LED light from 540–560 nm (20-nm bandwidth),

  • Normal incidence coupling angle: 50° in air.

When using a laser (<1-nm line) as a display source (such as in a laser MEMS display engine), the max FOV is the horizontal cross section of the Kogelnik curved above (17-degree FWHM). However, when using the same color as an LED source (20 nm wide, such as in an LED-lit LCoS micro-display light engine, the resulting FOV is a slanted cross-section (in this case increased to 34-degree FWHM), and a 2× FOV gain is achieved without changing the waveguide index or the holographic coupler, only the illumination’s spectral characteristics.

However, this comes at the cost of color uniformity: the lower angles (left side of the FOV) will have more contributions from the shorter wavelengths (540 nm), and the higher angles (right side of the FOV) will have more contributions from the longer wavelengths (560 nm). This slight color nonuniformity over the FOV is typical for volume holographic couplers.

5.1.3 Increasing FOV by optimizing grating coupler parameters

Unlike holographic couplers, which are originated and replicated by holographic interference in a phase change media (see previous section), SRGs are rather originated by traditional IC lithographic techniques and replicated by NIL or soft lithography. The topological structure of the gratings can therefore be optimized digitally to achieve the best functionality in both spectral and angular dimensions. Topological optimization needs to account for DFM and typical lithographic fabrication limitations. The angular bandwidth of an SRG coupler (i.e., the FOV that can be processed by this SRG) can be tuned by optimizing the various parameters of such a grating structure, such as the front and back slant angles, the grating fill factor, the potential coating(s), the grating depth, and of course the period of the grating (Figure20). Additional material variables are the refractive indices of the grating structure, grating base, grating coating, grating top layer, and underlying waveguide.

Figure 20: Optimizing the grating parameters to optimize color uniformity over the FOV.
Figure 20:

Optimizing the grating parameters to optimize color uniformity over the FOV.

Figure 20 shows how the SRG grating parameters can be optimized to provide a larger FOV, albeit with a lower overall efficiency, matching better the available angular bandwidth provided by the TIR condition in the guide. Lower efficiency is okay over the out-couplers since they are tuned in the low-efficiency range to produce a uniform EB (the in-coupler, however, needs to be highly efficient since there is only one grating interaction to couple the entire field into TIR mode).

Calculations of coupling efficiency have been carried out with an RCWA FMM algorithm and topological optimization by a steepest descent algorithm. Note that both unoptimized and optimized gratings have the same grating periods as well as the same central slant angle to position respectively the spectral and the angular bandwidths on identical system design points (with the FOV generated by the display engine and wavelength of the illumination source).

The bottleneck in FOV with the unoptimized grating structure is not the TIR condition (i.e., the index of the waveguide) but rather the grating geometry and the index of the grating. The angular bandwidth of the optimized grating should overlap the angular bandwidth of the waveguide TIR condition for best results over the largest possible FOV. Also, a “top hat” bandwidth makes the color uniformity over the FOV less sensitive to systematic and random fabrication errors in the mastering and the NIL replication of the gratings. Increasing the index of the grating and reducing the back slant while increasing the front slant angle can provide such an improvement.

Additional optimizations over a longer stretch of the grating can include depth modulations, slant modulations (rolling k-vector), or duty cycle modulations to produce an even wider bandwidth over a large, uniform EB.

5.1.4 Using dynamic couplers to increase waveguide combiner functionality

Switchable or tunable TIR couplers can be used to optimize any waveguide combiner architecture, as in

  • Increasing the FOV by temporal sub-FOV stitching at double the refresh rate,

  • Increasing the brightness at the eye by steering a reduced size EB to the pupil position (thus also increasing the perceived EB size), and

  • Increasing the compactness of the waveguide combiner by switching multiple single-color couplers in color sequence in a single guide.

Dynamic couplers can be integrated in various ways: polarization diversity with polarization-dependent couplers (the polarization switching occurring in the optical engine), reconfigurable surface acoustic wave or acousto-optical modulator couplers, electro-optical modulation of buried gratings, switchable SRGs in an LC layer, switchable metasurfaces in an multilayer LC layer, tunable volume holograms (by shearing, pressure, pulling), or switchable H-PDLC, as in Digilens’ volume holographic couplers.

5.2 High-level waveguide-combiner design

The previous section discussed ways to model and optimize the performance of individual couplers, in either grating or holographic form. We now go a step further and look at how to design and optimize the overall waveguide combiner architectures.

5.2.1 Choosing the waveguide coupler layout architecture

We have seen that couplers can work in either transmission or reflection mode to create a more diverse exit-pupil replication scheme (producing a more uniform EB) or to improve the compactness of the waveguide by using both surfaces, front and back. The various couplers might direct the field in a single direction or in two or more directions, potentially increasing the FOV that can propagate in the waveguide without necessarily increasing its index.

Figure 21shows how the optical designer can expand the functionality of in-couplers or out-couplers, with architectures ranging from bidimensional coupling to dual reflective/transmission operation in the same guide with sandwiched volume holograms or top/bottom grating couplers.

Figure 21: More functional coupler architectures that yield compact and efficient waveguide combiners.
Figure 21:

More functional coupler architectures that yield compact and efficient waveguide combiners.

More complex and more functional coupler architectures have specific effects on MTF, efficiency, color uniformity, and FOV. For example, while the index of the guide allows for a larger FOV to propagate, the index of the grating structures in air would increase the spectral and angular bandwidths to process a larger FOV without compromising color uniformity or efficiency. The waviness of the waveguide itself will impact the MTF as random cylindrical powers added to the field. Multiple stacked waveguides might be efficient at processing single colors, but their misalignment will impact the MTF as misaligned color frames. Similarly, hybrid top/bottom couplers will affect the MTF if they are not perfectly aligned (angular alignment within a few arc seconds).

5.2.2 Building a uniform EB

As the TIR field gets depleted when the image gets extracted along the out-coupler region, the extraction efficiency of the out-coupler needs to gradually increase in the propagation direction to produce a uniform EB. This complicates the fabrication process of the couplers, especially when the gradual increase in efficiency needs to happen in both pupil replication directions.

For volume holograms, the efficiency can be increased by a stronger index swing in the photopolymer or PDLC (through a longer exposure or a thickness modulation). For SRGs, there are a few options, as shown in Figure 22. This is true for the redirection grating (R-E) as well as the out-coupler (O-E).

Figure 22: Modulation of the outcoupling efficiency to build up a uniform EB.
Figure 22:

Modulation of the outcoupling efficiency to build up a uniform EB.

Groove depth and duty cycle modulation can be performed on all type of gratings, binary, multilevel, blazed, and slanted (see Figure 22). Duty cycle modulation has the advantage of modulating only the lateral structures, not the depth, which makes it an easier mastering process. Modulating the depth of the gratings can be done in binary steps (as in the Magic Leap One, Figure 22—right) or in a continuous way (Digilens waveguide combiners).

Grating front- and back-slant angle modulation (in a single grating period or over a larger grating length) can change the angular and spectral bandwidths to modulate efficiency and other aspects of the coupling (angular, spectral, polarization). Periodic modulation of the slant angles is sometimes also called the “rolling k-vector” technique and can allow for larger FOV processing due to specific angular bandwidth management over the grating area. Once the master has been fabricated with the correct nanostructure modulation, the NIL replication process of the gratings is the same no matter the complexity of the nanostructures (caution is warranted for slanted gratings where the NIL process must resolve the undercut structures; however, the slanted grating NIL process (with slants up to 50°) has been mastered by many foundries around the world [37]).

5.2.3 Spectral spread compensation in diffractive waveguide combiners

Spectral spread comes to mind as soon as one speaks about gratings or holographic elements. It was the first and is still the main application pool for gratings and holograms: spectroscopy. Spectral spread is especially critical when the display illumination is broadband, such as with LEDs (as in most of the waveguide grating combiner devices today, such as the HoloLens 1, Vuzix, Magic Leap, Digilens, Nokia, and so on), with a notable difference in the HoloLens 2 (laser MEMS display engine). The straightforward technique to compensate for the inevitable spectral spread is to use a symmetric in-coupler/out-coupler configuration in which the gratings or holograms work in opposite direction and thus compensate in the out-coupler any spectral spread impacted in the in-coupler (Figure 23).

Figure 23: Spectral spread compensation in a symmetric in-coupler/out-coupler waveguide combiner.
Figure 23:

Spectral spread compensation in a symmetric in-coupler/out-coupler waveguide combiner.

Although the spectral spread might be compensated, one can notice in Figure 23 that the individual spectral bands are spatially demultiplexed at the exit ports while multiplexed at the entry port. Strong exit-pupil replication diversity is thus required to smooth out any color nonuniformities generated over the EB.

This symmetric technique might not be used to compensate for spectral spread across different colors (RGB LEDs) but rather for the spread around a single LED color. The spread across colors might stretch the RGB exit pupils too far apart and reduce the FOV over which all RGB colors can propagate by TIR.

The pupil replication diversity can also be increased by introducing a partial reflective layer in the waveguide (by combining two plates with a reflective surface), thus producing a more uniform EB in color and field.

5.2.4 Field spread in waveguide combiners

The different fields propagating by TIR down the guide are also spread out, no matter the coupler technology (mirrors, prisms, gratings, holograms, and so on), see Figure 24.

Figure 24: Fractional field spread in a waveguide combiner.
Figure 24:

Fractional field spread in a waveguide combiner.

A uniform FOV (i.e., all fields appearing) can be formed over the EB with a strong exit pupil diversity scheme. This is a concept often misunderstood as in many cases, only one field is represented when schematizing a waveguide combiner. Figure 24 shows the field spread occurring in a diffractive waveguide combiner. The number of replicated fields is also contingent on the size of the human eye pupil. If the ambient light gets bright, i.e., the human eye pupil gets smaller, then only part of the FOV might appear to the user, missing a few fields.

5.2.5 Focus spread in waveguide combiners

When a pupil replication scheme is used in a waveguide combiner, no matter the coupler, the input pupil needs to be formed over a collimated field (image at infinity/far field). If the focus is set to the near field instead of the far field in the display engine, each waveguide exit pupil will produce an image at a slightly different distance, thereby producing a mixed visual experience, overlapping the same image with different focal depths. It is quasi-impossible to compensate for such focus shift over the exit pupils because of both spectral spread and field spread over the exit pupils, as discussed previously. Figure 25 shows such a focus spread over the EB from an input pupil over which the image is formed in the near field.

Figure 25: Focus spread in a waveguide combiner with a noncollimated input field.
Figure 25:

Focus spread in a waveguide combiner with a noncollimated input field.

The image over the input pupil can, however, be located in the near field when no pupil replication scheme is performed in the guide, such as in the Epson Moverio BT300 or in the Zeiss “Tooz” Smart Glass (yielding a small FOV and small EB).

When pupil replication is used in the guide, the virtual image can be set at a closer distance for better visual comfort by using a static (or even tunable) negative lens acting over the entire EB. For an unperturbed see-through experience, such a lens needs to be compensated by its conjugate placed on the world side of the combiner waveguide. This is the architecture used in the Microsoft HoloLens 1 (2015) [37].

Another, more compact, way would introduce a slight optical power in the O-E, so that this coupler takes the functionality of an off-axis lens (or an off-axis diffractive lens) rather than that of a simple linear grating extractor or linear mirror/prism array. Although this is difficult to implement with a mirror array (as in an LOE), it is fairly easy to implement with a grating or holographic coupler. The grating lens power does not affect the zeroth diffraction order that travels by TIR down the guide but affects only the outcoupled (or diffracted) field. The see-through field is also not affected by such a lensed out-coupler since the see-through field diffracted by such an element would be trapped by TIR and thus not enter the eye pupil of the user.

All three configurations (no lens for image at infinity, static lens with its compensator, and powered O-E grating) are shown in Figure 26. The left part of the EB shows an extracted field with image at infinity (as in the Lumus DK40—2016), the center part shows an extracted field with image at infinity that passes through a negative lens to form a virtual image closer to the user and its counterpart positive lens to compensate for see-through (as in the Microsoft HoloLens 1, 2015), and the right part of the EB shows an extracted field with the image directly located in the near field through a powered grating extractor (as with an off-axis diffractive lens, e.g., the Magic Leap One, 2018).

Figure 26: Two out-coupler architectures positioning the virtual image in the near field over all exit pupils.
Figure 26:

Two out-coupler architectures positioning the virtual image in the near field over all exit pupils.

For example, a half-diopter negative lens power would position the original extracted far field image to a more comfortable 2-m distance, uniformly over the entire EB.

A powered out-coupler grating might reduce the MTF of the image, especially in the direction of the lens offset (direction of TIR propagation), since the input (I-E) and output (O-E) couplers are no more perfectly symmetric (the input coupler being a linear grating in both cases, and the out-coupler an off-axis diffractive lens). Thus, the spectral spread of the image in each color band cannot be compensated perfectly and will produce lateral coromatic aberations (LCA) in the direction of the lens offset. This can be critical when using an LED as an illumination source, but it would affect the MTF much less when using narrower spectral sources, such as lasers or VCSELs.

One of the main problems with such a lensed out-coupler grating configuration when attempting to propagate two colors in the same guide (for example, a two-guide RGB waveguide architecture, as in Figure 34) is the generation of longitudinal chromatic aberrations (due to the focus changing with color since the lens is diffractive). Using a single color per guide and a laser source can greatly simplify the design task.

5.2.6 Propagating full color images in the waveguide combiner over a maximum FOV

We have seen in the previous paragraphs that the spectral spread of grating and holographic couplers can be perfectly compensated with a symmetric in-coupler and out-coupler configuration. This is possible over a single-color band but will considerably reduce the FOV if used over the various color bands (assuming that the couplers will work over these various spectral bands).

In order to maximize the RGB FOV in a waveguide combiner, one solution is to use stacked guides optimized each for a single-color band, each coupling a maximum FOV by tuning the diffraction angle of the in-couplers and out-couplers accordingly. This is the architecture used in both HoloLens 1 and Magic Leap One (see Figure 27, although the position of the input pupil (light engine) is opposite in both devices.

Figure 27: Stacked waveguides combiners that provide the largest FOV TIR propagation over three colors.
Figure 27:

Stacked waveguides combiners that provide the largest FOV TIR propagation over three colors.

Air gaps between all plates are required to produce the TIR condition. Such gaps also allow for additional potential filtering in between plates for enhanced performance (such as spectral and polarization filtering).

Figure 28 shows the functional diagram of such a single-color plate as a top view as well as its k-vector space depiction. Here again, I-E refers to the in-coupler, R-E refers to the leaky 90-degree redirection element, and O-E refers to the leaky out-coupler that forms the final EB (for 2D pupil replications).

Figure 28: k-vector diagram and lateral pupil replication layout for a single guide and single color.
Figure 28:

k-vector diagram and lateral pupil replication layout for a single guide and single color.

Note that the entire FOV is shown on the k-vector diagram (Figure 28), but only a single field (central pixel in the FOV, with entry normal to the guide) is shown in the EB expansion schematic.

The FOV in the direction of the incoupling can be increased by a factor of two when using a symmetric incoupling configuration in which the input grating or hologram (or even prism[s]) would attempt to couple the entire FOV to both sides, with one of the input configurations described in Figure 13 or Figure 14.

As the TIR angular range does not support such an enlarged FOV, part of the FOV is coupled to the right and part of the FOV is coupled to the left. Due to the opposite directions, opposite sides of the FOV travel in each direction. If such TIR fields are then joined back with a single out-coupler, the original FOV can be reconstructed by overlapping both partial FOVs, as in Figure 29.

Figure 29: Symmetric incoupling for FOV increase in the direction of incoupling.
Figure 29:

Symmetric incoupling for FOV increase in the direction of incoupling.

In the orthogonal direction, the FOV that can be coupled by TIR remains unchanged. This concept can be taken to more than one dimension, but the coupler space on the waveguide can become prohibitive.

5.2.7 Waveguide-coupler lateral geometries

We have reviewed the various coupler technologies that can be used in waveguide combiners, as well as the 2D exit pupil expansion that can be performed in waveguide combiners. Waveguide combiners are desirable since their thickness is not impacted by the FOV, unlike other combiner architectures such as free-space or TIR prisms. However, the lateral dimensions of the waveguide (especially the redirection coupler and out-coupler areas over the waveguide) are closely linked to size of the incoupled FOV, as shown in Figure 30. For example, the R-E region geometry is dictated by the FOV in the waveguide medium: it expands in the direction orthogonal to the TIR propagation, forming a conical shape.

Figure 30: Redirection and out-coupler areas as dictated by the incoupled FOV.
Figure 30:

Redirection and out-coupler areas as dictated by the incoupled FOV.

The largest coupler area requirement is usually the out-coupler element (center), aiming at processing all FOVs and building up the entire EB. Eye relief also strongly impacts this factor. However, its size can be reduced in a “human-centric optical design” approach: the right part of the FOV at the left edge of the EB as well as the left part of the FOV at the right edge of the EB can be discarded, thus considerably reducing the size of the O-E without compromising the image over the EB. Note that in Figure 29, the k-vector diagram (a) shows the FOV, whereas the lateral schematics of the waveguide in (b) and (c) show the actual size of the coupler regions.

Reducing the input pupil can help to reduce the overall size and thickness of the combiner. However, the thickness of the guide must be large enough not to allow for a second I-E interaction with the incoming pupil after the first TIR bounce. If there is a second interaction, then by the principle of time reversal, part of the light will be outcoupled and form a partial pupil (partial moon if the input pupil is circular) propagating down the guide instead of the full one. This is more pronounced for the smallest field angle, as depicted in Figure 31.

Figure 31: Effects of the input pupil size (and size of the I-E) and thickness of guide on a single field TIR pupil bouncing down the guide.
Figure 31:

Effects of the input pupil size (and size of the I-E) and thickness of guide on a single field TIR pupil bouncing down the guide.

However, if the polarization of the field is altered after the first TIR reflection at the bottom of the guide, the parasitic outcoupling can be reduced if the I-E is made to be highly polarization sensitive.

Reducing the waveguide thickness can also produce stronger pupil diversity over the EB and thus better EB uniformity. If reducing the guide is not an option (for parasitic outcoupling of the input pupil and also for etendue limitations in the display engine), a semitransparent Fresnel surface can be used inside the guide (as in two guides bounded together), which would reflect only part of the field and leave the other part unperturbed, effectively increasing the exit pupil diversity.

Figure 32 shows how the space of the out-coupler grating is dictated solely by the FOV and the EB. Note that many fields can be canceled at the edges and toward the edges of the EB, as they will not enter the eye pupil (right fields on the left EB edge and left fields on the right EB edge). This can also reduce the size of the redirection grating considerably. This holds true for both EB dimensions.

Figure 32: Eyebox and FOV dictate the size of the out-coupler area.
Figure 32:

Eyebox and FOV dictate the size of the out-coupler area.

5.2.8 Reducing the number of plates for RGB display and maintain FOV reach

Reducing the number of plates without altering the color of the image while propagating the maximum FOV allowed by the index of the guide is a desirable feature since it reduces the weight, size, and complexity of the combiner and make it also less prone to MTF reductions due to guide misalignments. Both lateral and longitudinal angular waveguide misalignments will contribute to a reduction of the MTF built by the display engine. Waveguide surface flatness issues are yet more cause for MTF reduction.

Due to the strong spectral spread of the in-coupler elements (gratings, holograms, RWGs, or metasurfaces), the individual color fields are coupled at higher angles as the wavelength increases, which reduces the overall RGB FOV overlap that can propagate in the guide within the TIR conditions (smallest angle dictated by the TIR condition and largest angle dictated by pupil replication requirements for a uniform EB). This issue is best depicted in the k-vector diagram (Figure 33).

Figure 33: k-vector diagram of a single-plate waveguide combiner using (a) RGB FOV coupling over a single-color TIR angular range condition and (b) RGB reduced FOV sharing the same TIR range.
Figure 33:

k-vector diagram of a single-plate waveguide combiner using (a) RGB FOV coupling over a single-color TIR angular range condition and (b) RGB reduced FOV sharing the same TIR range.

A lower spectral spread, such as through a prism in-coupler, would increase the RGB FOV overlap in a single guide, such as in an LOE (embedded partial mirrors out-couplers) from Lumus or in the microprism array couplers from Optinvent.

The left configuration in Figure 33 acts as a hybrid spatial/spectral filter, filtering the left part of the blue FOV, allowing the entire green FOV to be propagated (if the grating coupler periods have been tuned to match the green wavelength), and filtering the right part of the red FOV. The configuration in Figure 33 propagates the entire RGB FOV (assuming the couplers can diffract uniformly over the entire spectrum) at the cost of the FOV extending in the direction of the propagation. However, when considering binocular vision, this limitation could be mitigated by engineering a symmetric color vigneting in each eye (particularly on blue and red), providing a uniform stereo color vision in a single RGB guide with high FOV (e.g., Dispelix Oy).

Recently, two plate RGB waveguide combiner architectures have been investigated, reducing by one third the weight and size of traditional three-guide combiners, where the green FOV is shared between the top and bottom layer (see Figure 34. Various companies are using this two-plate RGB waveguide combiner architecture today, including Vuzix, WaveOptics, and Digilens.

Figure 34: Two-guide RGB waveguide combiner configuration.
Figure 34:

Two-guide RGB waveguide combiner configuration.

However, this requires the grating (or holograms, RWGs, or metasurfaces) to be efficient over a larger spectral band, which implies that SRGs are to be replicated in a higher refractive index, widening their spectral (and angular) bandwidths. High-index grating replication by NIL stretches the traditional wafer-scale NIL resin material science (inclusion of TiO2 or ZrO2 nanofiller particles). Nanoimprint at a Gen2 panel size of higher-index inorganic spin-on glass material might be the best fit, which also solves the resin or photopolymer reliability issues over various environmental conditions (temperature, pressure, shear, UV exposure, and humidity).

This two-guide RGB configuration splits the green FOV in two at the in-coupler region and merges them again over the out-coupler region. For good color uniformity over the FOV and the EB, especially in the green field, this technique requires perfect control of the two-guide efficiency balance. Pre-emphasis compensation of the guide mismatch is possible using the display dynamic range, but this requires precise calibration, reduces the final color depth, and does not solve the stitching region issue where the two fields overlap.

An alternative to the architecture uses the first guide to propagate green and blue FOVs and the second guide to propagate only the red FOV, as green and blue are closer spectrally to each other than red. This change, however, slightly reduces the allowed FOV traveling without vignetting but solves the green FOV stitching problem.

Although going from three plates to two plates brings a small benefit in size, weight, and cost, the added complexity of the color split geometry and the resulting color nonuniformities over the EB might overshadow the initial small benefits.

A single-plate RGB waveguide combiner would provide a much stronger benefit, as there is no need to align multiple guides anymore because everything is aligned lithographically by NIL inside the single plate (potentially also front and back). This would also yield the best possible MTF and the lowest costs.

One single-plate solution is to phase multiplex three different color couplers with three different periods into a single layer and then tune it so that there is no spectral overlap (no color ghost images over the EB). Such phase multiplexing is theoretically possible in volume holograms. This might be achieved in the Akonia (now Apple) thick holographic material (500 µm). If a thinner photopolymer (less than 20 µm) is desired for better reliability and easier mass production, a large holographic index swing is required. Standard photopolymers can be panchromatic and can also be phase multiplexed, but the resulting efficiency remains low, and color cross-contamination between holograms is an additional issue. This is also theoretically possible with SRGs, but it is difficult to simultaneously achieve high efficiency and a high extinction ratio over the three color bands. Metasurfaces and RWGs can theoretically produce such phase-multiplexed layers but with the same limitations.

Another solution is to spatially interleave various grating configurations by varying the periods, depths, and slant angles. This is, however, difficult to achieve practically. Yet another solution to solve the single RGB guide problem would time multiplex RGB gratings through a switchable hologram, such as the ones produced by Digilens Corp. This switching technique could also produce much larger FOVs multiplexed in the time domain and fused in the integration time of the human eye.

6 Conclusion

The aim of this review paper was to capture the state of the art in waveguide combiner optics for AR and MR headsets, especially as diffractive waveguide combiners (surface relief diffractives, volume holographic, and others such as metasurfaces, resonant gratings, and so on). We also reviewed the various geometric waveguides combiner architectures which are rather based on refractive and reflective elements.

We showed that for optimum results, the waveguide grating, display engine, and sensors need to be codesigned as a global system to closely match the optical performances and the specific features and limitations of the human visual system, through human-centric optical design.

The coming years will be an exciting time for MR hardware. A full ecosystem to allow for commodity mass production and lower costs of waveguide grating combiners is growing worldwide, comprising high-index ultraflat glass wafer manufacturers, high-index resin material developers, process equipment developers, NIL equipment developers, and also dedicated software design tools developers allow finally this technology to emerge as a viable option for the upcoming consumer MR and smart glass market.

However, one has to remember that delivering on the promises of the ultimate wearable display hardware is only one aspect of the trial and opportunity ahead for MR, delivering on strong use cases, especially for consumer markets, will be the other critical item to consider.


Corresponding author: Bernard C. Kress, Microsoft Corp. HoloLens Team, 1 Microsoft Way, Redmond, 98052, WA, USA, E-mail:

  1. Author contribution: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.

  2. Research funding: None declared.

  3. Conflict of interest statement: The authors declare no conflicts of interest regarding this article.

References

[1] W. S. Colburn and B. J. Chang, “Holographic combiners for head up displays,” Tech Report No. AFAL-TR-77-110, 1977.Search in Google Scholar

[2] J. Jerald, The VR Book: Human Centered Design for Virtual Reality, ACM Books, 2016, 978-1-97000-112-9.Search in Google Scholar

[3] W. Barfield, Fundamentals of Wearable Computers and Augmented Reality, 2nd ed., CRC Press, Taylor and Francis Group, 2015, 978-1-482243595.10.1201/b18703Search in Google Scholar

[4] L. Inzerillo, “Augmented reality: past, present and future,” in The Engineering Reality of Virtual Reality, Vol. 8649, M. Dolinsky and I. E. McDowall, Eds. Proc. of SPIE-IS&T Electronic Imaging, SPIE, 2013.Search in Google Scholar

[5] R. T. Azuma, “A survey of augmented reality,” in Presence, Teleoperators and Virtual Environments, vol. 6, pp. 355–385, 1997.10.1162/pres.1997.6.4.355Search in Google Scholar

[6] O. Cakmakci, J. Rolland, “Head-worn displays: a review,” J. Display. Technol., vol. 2, pp. 199–216, 2006.10.1109/JDT.2006.879846Search in Google Scholar

[7] J. Rolland and O. Cakmakci, “Head-worn displays: the future through new eyes,” Opt. Phot. News, vol. 20, pp. 20–27, 2009, https://doi.org/10.1364/opn.20.4.000020.Search in Google Scholar

[8] D. W. F. Van Krevelen and R. Poelman, “A survey of augmented reality technologies, applications and limitations,” Int. J. Virtual Real., vol. 9, pp. 1–20, 2010, https://doi.org/10.20870/ijvr.2010.9.2.2767.Search in Google Scholar

[9] K.-L. Low, A. Ilie, G. Welch, A. Lastra, “Combining head-mounted and projector-based displays for surgical training,” IEEE Virtual Real., 2003, Proceedings 10.1109/VR.2003, https://doi.org/10.1109/VR.2003.1191128.Search in Google Scholar

[10] Y. Amitai, A. Friesem, and V. Weiss, “Holographic elements with high efficiency and low aberrations for helmet displays,” Appl. Opt., vol. 28, pp. 3405–3416, 1989, https://doi.org/10.1364/ao.28.003405.Search in Google Scholar

[11] N. Baker, Mixed Reality Keynote at Hot Chips HC28 – Symposium for High Performance Chips, Aug. 21-23 2016, www.hotchips.org.Search in Google Scholar

[12] B. Kress and W. Cummins, Towards the Ultimate Mixed Reality Experience: HoloLens Display Architecture Choices, SID 2017 Book 1: Session 11: AR/VR Invited Session II.10.1002/sdtp.11586Search in Google Scholar

[13] J. Michael Miller, N. de Beaucoudrey, P. Chavel, J. Turunen, and E. Cambril, “Design and fabrication of binary slanted surface-relief gratings for a planar optical interconnection,” Appl. Opt., vol. 36, pp. 5717–5727, 10 August 1997, https://doi.org/10.1364/AO.36.005717.Search in Google Scholar

[14] J. Kimmel, T. Levola, P. Saarikko, and J. Bergquist, “A novel diffractive backlight concept for mobile displays,” J. SID, vol. 16, no. 2, 2008.10.1889/1.2841870Search in Google Scholar

[15] J. Kimmel and T. Levola, “Diffractive backlight light guide plates in mobile electrowetting display applications,” SID 09 Paper 471 Page 2.10.1889/1.3256921Search in Google Scholar

[16] J. Liu, N. Zhang, J. Han, et al., “An improved holographic waveguide display system,” Appl. Optic., vol. 54, no. 12, pp. 3645–3649, 2015.10.1364/AO.54.003645Search in Google Scholar

[17] T. Yoshida, K. Tokuyama, Y. Takai, et al., “A plastic holographic waveguide combiner for light-weight and highly-transparent augmented reality glasses,” J. SID, vol. 26, no. 5, 2018.10.1002/jsid.659Search in Google Scholar

[18] H. Mukawa, K. Akutsu, I. Matsumura et al., “A full color eyewear display using holographic planar waveguides” 8.4, SID 08 DIGEST 2008.10.1889/1.3069819Search in Google Scholar

[19] T. Oku, K. Akutsu, M. Kuwahara et al., “High-luminance see-through eyewear display with novel volume hologram waveguide technology,” 15.2, 192 • SID DIGEST 2015.10.1002/sdtp.10308Search in Google Scholar

[20] K. Sarayeddine, P. Benoit, G. Dubroca, and X. Hugel, “Monolithic low-cost plastic light guide for full colour see through personal video glasses,” in ISSN-L 1883-2490/17/1433 ITE and SID (IDW 10), 2010, pp. 1433–1435.Search in Google Scholar

[21] T. Levola, “Exit pupil expander with a large field of view based on diffractive optics,” J. Soc. Inf. Disp., vol. 17, pp. 659–664, 2009.10.1889/JSID17.8.659Search in Google Scholar

[22] T. Levola, “Diffractive optics for virtual reality displays,” J. SID, vol. 14, no. 5, 2006.10.1889/1.2206112Search in Google Scholar

[23] B. Kress, “Diffractive and holographic optics as optical combiners in head mounted displays,” in Proceedings of the 2013 ACM Conference on Pervasive and Ubiquitous Computing – Ubicomp’13, 2013, pp. 1479–1482.10.1145/2494091.2499572Search in Google Scholar

[24] A. Cameron, “Optical waveguide technology & its application in head mounted displays,” in Head- and Helmet-Mounted Displays XVII; and Display Technologies and Applications for Defense, Security, and Avionics VI, Vol. 8383, P. L. Marasco, P. R. HavigII, D. D. Desjardins, and K. R. Sarma, Eds., Proc. of SPIE, p. 83830E.10.1117/12.923660Search in Google Scholar

[25] M. Homan, “The use of optical waveguides in Head up Display (HUD) applications,” in Display Technologies and Applications for Defense, Security, and Avionics VII, Vol. 8736, D. D. Desjardins and K. R. Sarma, Eds, Proc. of SPIE.10.1117/12.2014513Search in Google Scholar

[26] D. Cheng, Y. Wang, C. Xu, W. Song, and G. Jin, “Design of an ultra-thin near-eye display with geometrical waveguide and freeform optics,” Opt. Express, vol. 22, no. 17, pp. 20705–20719, 2014, https://doi.org/10.1364/oe.22.020705.Search in Google Scholar

[27] D. Jurbergs, F.-K. Bruder, F. Deuber, et al., “New recording materials for the holographic industry,” in Practical Holography XXIII: Materials and Applications, Vol. 7233, H. I. Bjelkhagen and R. K. Kostuk, Eds. Proc. of SPIE.10.1117/12.809579Search in Google Scholar

[28] www.digilens.com.Search in Google Scholar

[29] K. Curtis and D. Psaltis, “Cross talk in phase coded holographic memories,” J. Opt. Soc. Am. A, vol. 10, no. 12, December 1993, https://doi.org/10.1364/JOSAA.10.002547.Search in Google Scholar

[30] H. Kogelnik, “Coupled wave theory for thick hologram gratings,” Bell Syst. Tech. J., vol. 48, 1969, https://doi.org/10.1002/j.1538-7305.1969.tb01198.x.Search in Google Scholar

[31] M. A. Golub, A. A. Friesem, and L. Eisen, “Bragg properties of efficient surface relief gratings in the resonance domain,” Optic Commun., vol. 235, pp. 261–267, 2004, https://doi.org/10.1016/j.optcom.2004.02.069.Search in Google Scholar

[32] M. G. Moharam, “Stable implementation of the rigorous coupled wave analysis for surface relief gratings: enhanced transmittance matric approach,” J. Opt. Soc. Am. A, vol. 12, no. 5, pp. 1077–1086, 1995, https://doi.org/10.1364/josaa.12.001077.Search in Google Scholar

[33] L. Alberto Estepa, C. Neipp, J. Francés, et al., “Corrected coupled-wave theory for non-slanted reflection gratings,” in Physical Optics, Vol. 8171, D. G. Smith, F. Wyrowski, and A. Erdmann, Eds, Proc. of SPIE.Search in Google Scholar

[34] http://www.kjinnovation.com/.Search in Google Scholar

[35] https://meep.readthedocs.io/en/latest/.Search in Google Scholar

[36] T. Levola and P. Laakkonen, “Replicated slanted gratings with a high refractive index material for in and outcoupling of light,” Opt. Express, vol. 15, pp. 2067–2074, 2007.10.1364/OE.15.002067Search in Google Scholar PubMed

[37] https://www.lighttrans.com/applications/virtual-mixed-reality/waveguide-huds.html.Search in Google Scholar

[38] M. W. Farn, “Binary gratings with increased efficiency,” Appl. Opt., vol. 31, no. 22, pp. 4453–4458, 1992, https://doi.org/10.1364/ao.31.004453.Search in Google Scholar

[39] B. Kress and P. Meyrueis, Applied Digital Optics: From Micro-optics to Nanophotonics, 1st ed., John Wiley and Sons Publisher, 2007, -10: 0470022639.Search in Google Scholar

[40] G. Quaranta, G. Basset, O. J. F. Martin, and B. Gallinet, “Steering and filtering white light with resonant waveguide gratings,” in Proc. SPIE 10354, Nanoengineering: Fabrication, Properties, Optics, and Devices XIV, 2017, p. 1035408.Search in Google Scholar

[41] G. Basset, “Resonant screens focus on the optics of AR,” in Proc. SPIE 10676, Digital Optics for Immersive Displays, 2018, p. 106760I.10.1117/12.2305932Search in Google Scholar

[42] P. Genevet, F. Capasso, F. Aieta, M. Khorasaninejad, and R. Devlin, “Recent advances in planar optics: from plasmonic to dielectric metasurfaces,” Optica, vol. 4, no. 1, pp. 139–152, 2017, https://doi.org/10.1364/optica.4.000139.Search in Google Scholar

[43] F. Capasso, “The future and promise of flat optics: a personal perspective,” Nanophotonics, vol. 7, no. 6, 2018, https://doi.org/10.1515/nanoph-2018-0004.Search in Google Scholar

[44] W. T. Chen, A. Y. Zhu, J. Sisler, et al., “Broadband Achromatic metasurface-refractive optics,” Nano Lett., vol. 18, no. 12, pp. 7801–7808, 2018, https://doi.org/10.1021/acs.nanolett.8b03567.Search in Google Scholar


Supplementary Material

The online version of this article offers supplementary material (https://doi.org/10.1515/nanoph-2020-0410).


Received: 2020-07-21
Accepted: 2020-09-16
Published Online: 2020-10-07

© 2020 Bernard C. Kress and Ishan Chatterjee, published by De Gruyter, Berlin/Boston

This work is licensed under the Creative Commons Attribution 4.0 International License.

Downloaded on 5.3.2024 from https://www.degruyter.com/document/doi/10.1515/nanoph-2020-0410/html?lang=en
Scroll to top button