This year, the French-German Research Institute of Saint-Louis (ISL) is celebrating its 60th anniversary, but the roots of the institute go back to the early end of the Second World War when a team of German scientists led by Professor Schardin, from the Air Force Technical Academy (TechnischeAkademie der Luftwaffe) in Berlin-Gatow, came to the Upper Rhine to work for the French Army. There, Professor Schardin worked in the field of ballistics and became renowned worldwide for his work in high-speed physics. In his early years, as a permanent assistant to the eminent German ballistics Professor Carl Cranz, he developed the famous Cranz-Schardin camera, a revolutionary electro-optical photography and high-speed cinematography method, which used electric sparks or X-ray flash as an illuminator, and was able to work at frame rates over 106 images/s. This technique brought huge improvements in the comprehension of ballistic phenomena and, moreover, in high-speed physics. Since 1945, the LRSL (Laboratoire de Recherches de Saint-Louis), and afterward the ISL, maintained its leading position in the domain of high-speed phenomena. In the beginning of the 1960s, the invention of the laser began a true revolution and allowed the emergence of new techniques like holography and interferometric holography. Today, with the introduction of semiconductor lasers, ISL has a leading position on range-gated active imaging and deploys all its research effort in a newly emerging domain: computational imaging.
2 High-speed imaging with spark or X-ray illumination
2.1 Cranz-Schardin camera or spark chronolens
In 1929, Schardin developed the Cranz-Schardin camera also known as the spark chronolens. This revolutionary camera is still used for the visualization of ultra-fast physical phenomena . Even nowadays, hundreds of scientific papers continue to mention this technique each year. Figure 1 shows the setup as it was used at that time.
It consists of an illumination, a camera, and a spherical mirror. The illuminator is composed of 24 spark cells (0.3 μs of illumination duration, Figure 1 left) coupled with a 24-lens camera (center, Figure 1) capable of imaging a chronogram of 24 images of the same scene at an imaging rate of >1 million images/s, i.e. >1 MHz. It is amazing to note that this imaging rate remains challenging, even today. Each of the 24 spark illuminators is conjugated to one of the 24 lenses of the camera through a convex mirror (foreground, Figure 1), and the phenomenon to study is placed between the system and the mirror. Figure 2 shows two sequences of images taken with the Cranz-Schardin camera.
Figure 3 presents one of the fastest video sequences made by Schardin with a frame rate of 6.5 MHz. This movie taken in the late 1930s gives a visualization of the shock wave produced by the collision of a projectile on a glass plate in polarized light.
At the same time, Schardin was developing Kerr cells , which were used as very fast shutters.
2.2 X-ray imaging for ballistics application
The first basic studies on X-ray imaging were conducted by Schardin in Berlin-Gatow to study the effect of hollow charges on armoring. In fact, it was the advent of flash radiography that allowed the visualization of the deformation of the coating and its transformation into a projectile. The first X-ray generators emitting very short pulses, well below 1 μs, were designed in 1938–1939 by the Siemens Company where Dr. Thomer worked. At the request of Schardin, he was assigned to Gatow’s laboratory in 1939. The first radiographic photographs of the functioning of hemispheric charges were, thus, obtained in 1940, and the researchers had the opportunity to understand the formation of a projectile through the deformation of a shaped charge and to study their penetration power (Figure 4).
3 Laser imaging
The advent of lasers in the early 1960s allowed the development of new visualization techniques particularly adapted to the study of flows. ISL researchers developed a lot of flow visualization techniques that can be classified in different categories depending on whether they allow a global flow visualization or a local measurement of the flow characteristics, and whether there is a need of seeding the flow or not. Table 1 lists the different techniques according to these characteristics.
Flow visualization techniques developed at ISL.
|Global flow visualization||Global measurements||Local measurements|
|Without seeding||Shadowgraphy, Schlieren technique, classical or holographic interferometry, differential interferometry||RAMAN scattering|
|With seeding||Photography or holography of particle fluorescence induced by laser||Interferometric laser velocimetry, particle image velocimetry, holographic particle image velocimetry||Laser anemometry|
For example, differential interferometry with polarized laser light brought very nice results in aerodynamics (Figure 5). With this method, the fringe deformation is proportional to the difference of the optical path in the airflow.
At the same time, ISL developed the technique of holographic cinematography . In 1983, ISL made the first holographic movie at 24 Hz on a 35-mm film (small train impacting a wall of plastic cubes). This movie was awarded the Gaumont price in 1985. The same year, ISL was recording the first holographic movie of a living person at 25 Hz on a 126-mm film. ‘Christiane et les holobulles’ presented a young woman blowing soap bubbles . This movie was awarded the ‘Great international price of the future’ during the International Exhibition of Techniques and Energy of the Future in Toulouse, France.
Figure 6 shows the recording setup and the three-dimensional (3D) holographic image restitution used to build up the movie and some images from the movie ‘Christiane et les holobulles’.
However, the main field of research to benefit from the achievements in holography was non-destructive testing and control. On this topic, ISL developed collaborations with a lot of companies to test different kinds of materials and structures. Holographic interferometry could put in evidence small defects in a very wide diversity of materials subject to different mechanical stresses. Material defects in rocket propellants, gluing defects in composite materials or on helicopter blades could be evidenced by submitting the structure to a mechanical, thermal, pressure, or vibration load. The advent of holographic interferometry with a double reference beam allowed to access the quantitative displacement measurement. In Figure 7, we can see different results of holographic interferometry including a vibration map of a car brake disk , a detection of a buried mine by visualization of a seismic wave propagation , and a visualization of a crack in a concrete wall .
4 Range-gated active imaging
The advent of digital holography familiarized the teams with the use of low-light cameras and lighting techniques, which naturally led the ISL to focus on active imaging for surveillance applications. The research on active imaging began in 2001. Very rapidly, the first high-TRL prototypes of range-gated active imaging systems allowed new military applications, and a lot of new fields of research related to this topic could be explored. Figure 8 shows some of the prototypes that have been developed at ISL over the years for different kinds of applications .
4.1 Performance modeling
To be able to fit with the operational needs based on the user specifications, an appropriate modeling of the system has to be worked out. For this, the ISL scientists developed a modeling environment under Matlab. The software takes into account most of the system and environmental parameters in order to be able to predict the global performance of a system before its development (Figure 9).
It includes the source, optics, intensifier, camera, and display parameters to provide the image quality and the energetic balance of the global system. For the propagation medium, the visibility and the turbulence level are the two main parameters that are taken into account.
4.2 Illumination techniques
The performance of a range-gated active imaging system depends on the quality of the illumination. Depending on the nature of the laser source (solid state laser or semiconductor laser), the range-gated active imaging system will work in flash mode (one laser pulse/image) or in accumulation mode (multiple laser pulses/image) to collect a sufficient amount of reflected photons , .
4.2.1 Semiconductor laser source
Laser diodes are the most adapted laser source to build an efficient laser illuminator. Their advantages are numerous: high electrical-to-optical efficiency, high mean power, compactness of the laser emission component and of the power management. The only drawback is their low and asymmetrical beam parameter product (BPP), but this can be overcome by different collimation techniques. With these techniques, a high-quality illumination can be produced with a high homogeneity and, most importantly, without speckle.
A lot of work has been done at ISL to increase the illumination quality , , , , . It consists in equalizing, as much as possible, the BPP between the two axes using a light duct or a light fiber and then to collimate their output with a collimation lens. Figure 10 shows two techniques developed at ISL: the light duct and the restacking glass plates.
This results in a very homogeneous and top-hat distribution of light in the far field. These laser illuminators use high-power laser diodes, emitting a laser beam at a wavelength between 800 and 860 nm. The laser divergence can be adjusted from some hundreds of mrads down to 2 mrads for the smallest field of view, and the mean illumination power can be as high as 30 W. With this technique, we are able to visualize objects at distances over 20 km.
4.2.2 Solid state laser source
In some cases, in particular for SWIR imaging, when the sensor cannot work in accumulation mode, the use of a solid state laser as an illumination source cannot be avoided. In this case, one image corresponds to one laser pulse. However, solid state lasers are very coherent: this leads to an increase in the inhomogeneity of the illuminated field due to speckle and also to parasite interferences and diffraction effects.
In this case, it is important to find solutions that are able to break down the coherence of the source by increasing the spatial diversity. The trick is to sample the incident laser beam into a high number of beamlets to create uncorrelated beamlets. This can be done using adapted light pipes , optical steppers , or both . The result is a very homogeneous field of illumination with a top-hat illumination (Figure 11). The left image is illuminated with coherent light and has a high speckle contrast. The right image was recorded with homogenized light. The speckle contrast is significantly reduced.
Figure 12 gives some examples of images taken with a SWIR active imaging system. It is easy to see that depending on the integration time, the depth of the visualized range gate is different. Figure 12B shows the result with a long integration time (500 ns). All the objects in the background and the foreground are visible. From Figure 12C–E, we used a small integration time, which leads to select only a short-range gate in space and, therefore, to visualize a small depth of the scene. The different images are taken with different delays between the laser pulse and the opening of the camera. In this case, a simple processing of the images can give the distance of the object for each pixel of the image. We will take a closer look at this property in the next paragraph.
4.3 3D range imaging
When using a range-gated active imaging system, the gate distance is given by the delay between the laser pulse and the camera opening time. As a matter of fact, each image of a sequence is time stamped. This leads to the possibility of using this information for a 3D reconstruction of the scene. In the last years, ISL developed two techniques for 3D scene visualization . The tomographic method uses a very small slice of light (small pulse width and gate duration), which is translated through the depth of the scene due to the variation of the sensor delay .
As discussed in the literature , all the images produced are processed and used to reconstruct a virtual 3D model of the scene. Thus, the precision of the 3D model depends on the light slice thickness and on the value of the step used to scan the scene. As a matter of fact, one has to compromise between precision, processing time, and reconstruction algorithm (weighted mean, correlation, maximum detection…). Figure 13 shows the principle of this method, the resulting range gates at different ranges, and the 3D reconstructed landscape. In this example, the reconstructed field sampled by 200 images has a depth of about 1 km.
Figure 14 shows two more results of 3D reconstruction . By adapting the pulse width, this method can digitize either a kilometric scene or a centimetric scene. The left-hand image was acquired with a pulse width and exposure time equal to 200 ns, i.e. a range gate of 60 m; the right-hand image was acquired with a pulse width and exposure time equal to 2 ns, i.e. a range gate of 60 cm depth. In both cases, this method gives very good results in terms of image quality, but even if the image acquisition and processing is automated, the translation of the range gate through the depth of the scene is still lasting for a few seconds.
To overcome this drawback and to enable the use of active imaging in stealthy application (military application where the use of a laser illuminator has to be as furtive as possible), we developed a second method based on the use of two images only. Here, the image acquisition process will only need two short pulses of light and exposure times longer than the laser pulse width. In this case, the depth-intensity profile will include a plateau (see Figure 15).
In this case, for a given sensor delay, i.e. for a given positioning of the depth-intensity profile, the rising and falling edges of the profile contain the 3D information modulated by the object reflectance, and the plateau will only contain the reflectance information. Based on this intrinsic property of the depth-intensity profile, one can extract the 3D information of a scene using a minimum of two images. Figure 15 shows that, using two images and an exposure time two times greater than the pulse width, it is possible to have an equal width for the rising edge, the plateau, and the falling edge. The index 0 denotes the sensor delay, and the indices i and i+1 denote the two succeeding images where the gated viewing delay is shifted by a factor Δzr or Δzf.
Comparing the intensity of the plateau Ip,i from image i with the linear rising edge Ip,i+1 from image i+1, and the linear falling edge If,i from image i with intensity of the plateau Ip,i+1 from image i+1, the depth of the scene can be calculated from (F, et al., 2014):
As it is shown by Eqs. (1) and (2), the 3D reconstruction will consist in acquiring two images (i and i+1) of the scene, shifted in depth from a half of the pulse width, and in dividing the two images one by the other. A simple test on the gray levels will determine the equation that applies. Figure 16 shows an example of a 3D reconstruction with the two-image method.
In this example, a pulse width of 2 μs and an exposure time of 4 μs were used; z0,i was placed at 950 m. These values allow to reconstruct a scene of a 600-m depth. The same method can also apply for small objects. In this case, one has to use smaller pulse widths and exposure times. Figure 17 gives an example of 3D reconstruction of a head model. In this case, we used a pulse width of 1 ns and an exposure time of 2 ns. The reconstructed depth was about 30 cm.
5 Computational imaging to see around the corner
In the previous sections of this paper, it was assumed that the target to visualize was in the line-of-sight of the vision system, even if the range was very high. For instance, by recording intensity images, we measure the photon flux impinging a focal plane array (FPA) from a certain observation direction. The amount of photons or the intensity is interpreted as a convolution between the illumination laser pulse and the surface reflectance properties of the line-in-sight scenario. In non-line-of-sight (NLoS) imaging, we take into account that a typical photon bounces off multiple surfaces before our FPA detector can detect it. The aim of the computational imaging approach is to reveal information about objects outside the direct-line-of-sight and carried by multi-bounce photons. Thus, we assume every surface to have a Lambertian bidirectional reflection distribution function (BRDF) leading to diffuse omnidirectional reflection of the light and a total loss of imaging information.
At each diffuse reflection, the photons, which are coming from the target surface, spread into the whole space and the amount of photons that are traveling into the direction of the detector reduces rapidly. As depicted in Figure 18, we typically assume a three-bounce scenario. In such a scenario, we neglect the presence of higher numbers of bounces and assume these as a blurred background signal. Further in our analysis, we assume that the photons are bouncing off three surfaces by diffuse reflection and are propagating from a light source to a spot on a relay surface, then to the target surface to a sensing area, and, finally, to the detector.
At ISL, we are investigating two different computational imaging approaches based on ordinary intensity images or based on highly precise measurements of the photon’s times of flight. Figure 19 illustrates the first approach. A continuous laser pointer illuminates a spot on a relay surface, and the light propagates into the hidden scene (fiber-coupled laser diode SemiNex 4PN-108, SemiNex, Peabody, MA, USA).
Then, an ordinary intensity camera (Xenics Xeva-1.7-320, Xenics, Leuven, Belgium) samples the blurred intensity distribution of light, which is coming from the target, illuminating the relay surface. In our approach, we use an analysis-by-synthesis algorithm to estimate the position and orientation of the target with six degrees of freedom (x, y, z, θx, θy, θz). This method renders a synthetic image from a scene hypothesis and compares the result to the measured data. Then, the algorithm calculates the most probable position and orientation of the target by optimizing the scene hypothesis iteratively. Our algorithm is able to track an object within an area of several meters in real time (25 Hz) with centimetric precision .
Furthermore, the transient imaging approach uses high precise measurements of the photon roundtrip time. By knowing the time and the direction of arrival, we can reconstruct the position and some surface shapes by back-projection of the recorded information. In Figure 20, an example shows a raw data set and the result of the back-projection. In this case, the computational process reveals the position of three hidden targets .
From the Cranz-Schardin camera to computational imaging, ISL never stopped innovating. During its 60 years of existence, the Institute has always taken advantage of the scientific and technological innovations of its time to put them at the service of its two governing ministries and, thus, at the service of the military and security forces of our two countries. The science of photon, photonics, still has a bright future ahead, and there is no doubt that the latest discoveries and technological advances in this field will be applied to systems that will allow our armed forces to maintain their technological superiority and our soldiers to carry out their missions with greater security.
H. Schardin, J. SMPTE 71, 329–334 (1962).
H. Schardin, J. SMPTE 61, 273–285 (1953).
H. Pâques and P. Smigielski, in ‘Cinéholographie’ (C.R. Académie des Sciences, Paris, 1965) 260, pp. 6562–6564.
P. Smigielski, H. Fagot and F. Albe, Proc. SPIE 600, 186–193 (1986).
P. Smigielski, Proc. SPIE 1553, 436–446 (1992).
F. Christnacher, P. Smigielski, A. Matwyschuk, M. Bastide and D. Fusco, Proc. SPIE 3745, 361–365 (1999).
F. Christnacher, P. Smigielski, A. Matwyschuk, D. Fusco and Y. Guillard, Proc. SPIE 3745, 239–243 (1999).
M. Laurenzis and F. Christnacher, Adv. Opt. Technol. 2, 397–405 (2013).
F. Christnacher, M. Laurenzis and S. Schertzer, Proc. SPIE 8896, (2013).
F. Christnacher, M. Laurenzis and S. Schertzer, Opt. Eng. 53, 043106 (2014).
Y. Lutz, J.-M. Poyet and N. Metzger, Proc. SPIE 8896, 889608 (2013).
Y. Lutz and N. Metzger, Proc. SPIE 8170, 81700C (2011).
Y. Lutz and M. Laurenzis, Adv. Opt. Technol. 3, 179–185 (2014).
Y. Lutz and J.-M. Poyet, Opt. Laser Technol. 57, 90–95 (2014).
Y. Lutz, E. Bacher and S. Schertzer, Opt. Laser Technol. 96, 1–6 (2017).
M. Laurenzis, Y. Lutz, F. Christnacher, A. Matwyschuk and J.-M. Poyet, Opt. Eng. 51, 061302 (2012).
Y. Lutz, SPIE Newsroom (2015). Available at: http://spie.org/news/6197-instantaneous-generation-of-spatial-diversity-for-speckle-contrast-reduction.
J.-M. Poyet and Y. Lutz, Opt. Eng. 55, 075103 (2016).
F. Christnacher, M. Laurenzis, D. Monnin, S. Schertzer and G. Schmitt, in ‘8th NATO Symposium on Military Sensing, RTO-MP-SET 169, Friedrichshafen’ (Germany, 2011).
D. Monnin, A. L. Schneider, F. Christnacher and Y. Lutz, in ‘A 3D outdoor scene scanner based on a night-vision range-gated active imaging system. 3rd Int. Symposium on 3DPVT’ (Chapel Hill, US, 2006).
J. Klein, C. Peters, J. Martín, M. Laurenzis and M. B. Hullin, Nat. Sci. Rep. 6, 32491 (2016).
M. Laurenzis and A. Velten, J. Electron. Imaging 23, 063003 (2014).