. Real time defect detection during composite layup via Tactile Shape Sensing. Science and Engineering of Composite Materials, 28(1), 1-10.

: In this study an automated composite layup end effector is presented which is the first to be able to find defects in real time during layup using tactile shape sensing. Based around an existing sensor concept developed by the Bristol Robot Laboratory known as the ‘TacTip’, a new end effector is developed, replacing the soft gel core of the original sensor was replaced by a much firmer elas-tomer, enabling it to apply up to 400N of compaction force. In this paper it is shown to successfully detect typical defects such as wrinkles, foreign objects, layup errors or incorrect material types while simultaneously compacting preimpregnated composite materials over complex mould shapes.


Introduction
Composite materials generally consist of thousands of carbon or glass fibres which individually have excellent specific structural properties.These are typically arranged into sheets or tapes of aligned fibres which need to be formed into the shape of the finished components.In order to make the most of these properties in structural applications the fibres need to be strongly bonded together and remain as straight and aligned as possible in the finished product.Defects such as wrinkles, bridges across concave corners, foreign objects or misplaced plies can severely compromise the integrity of the material [1].
Composite products have traditionally been manufactured by hand, with skilled workers placing and compacting layers of fibres onto moulds to make high performance, small production volume products.There has been a recent expansion of composites use into high volume industries such as civilian aerospace and mass-produced cars.Manual production cannot keep up with increased demands and as a result a wide variety of automated solutions have been developed [2,3].Despite this, hand layup remains a crucial part of the composites industry, primarily due to its ability to layup composite material onto the most complex, highly curved moulds which automated processes cannot successfully tackle.Research into what enables hand layup to stay ahead in this field was conducted by the author in a previous study [4].It was concluded that one of the main advantages of manual layup was the ability of the human workers to gather real time feedback from their work via both visual and tactile sensing to ensure the products are defects free.This allows them to constantly adjust their actions in a closed-loop feedback system, correcting any defects or unwanted features as soon as they begin to arise.
In contrast most automated processes only gather feedback once the layup has been completed.This is often still completed manually, sequentially inspecting each ply after it has been laid up, leading to up to 4-6 hours of dead time during manufacture of large high-performance parts according to Krombholz et al. [5].Because the inspection process can add a significant delay to the manufacturing time there has been a drive to automate the inspection processes.The ideal situation would be for the inspection to run in parallel to the layup process rather than in a stopstart sequential manner.As well as reducing downtime, this could facilitate some level of closed-loop feedback in the automated process.
Automated quality assurance has typically been limited to vision-based systems such as Aligned Vision's 'Laser Vision TM ' system, which is being commercially used to detect misplaced plies, gaps and fibre angles during automated tape laying [6].A prototype system has also been developed which uses laser parallax to gauge the surface profile of the composite material in order to detect wrinkles, bridging and foreign objects.Airbus has proposed a similar system which uses a projected laser line and camera mounted directly behind an Automated Fibre Placement (AFP) head to scan tapes of composite material laid onto a tool, but only minimal details have been published [7].Three-dimensional laser scanning equipment which could map a surface is also commercially available but can be prohibitively expensive [8].Tactile sensing however remains a relatively untouched area for automated systems.

Tactile sensing
The importance of tactile sensing in robotics has been cited by many other studies [9] and it has been a goal for over 30 years to enable the detection of properties that are difficult to detect via vision alone, such as contact force or 3D shape [10].For example robotics researchers have used tactile force sensing to try and replicate the feeling in a finger, often for use in prosthetic limbs to give the user more feedback to make the use of the hand more intuitive [11].The generic term 'Tactile sensing' actually covers a range of properties which are typically measured during contact with an object.As outlined by Nichollas et al. [12] these include contact detection, force measurement, threedimensional shape, slip or temperature.Multiple reviews on tactile sensing are available covering a wide range of different sensor types [13,14].It could be argued that all of these properties have some importance during the layup of composite materials.The thermosets resins typically used in advanced composites are thermo-visco-elastic, hence temperature, applied force and contact time will all influence the behaviour of the material.In this study the focus will be on detecting defects that manifest as threedimensional shapes.
There are a wide range of devices that can detect pressure or force, but crucially cannot detect three-dimensional shape.Many of these use pressure sensitive films [15,16], or pressure sensors embedded into a rubber matrix [10].In the case of defects occurring during composites layup, the vast majority will manifest as a 3D shape, such as a wrinkle, a bridge across a concave corner or a foreign object trapped in the layup [1].It could be argued that an out-of-plane defect such as a wrinkle may also effect the contact pattern or total force applied by an end effector.A project to give composite end effectors tactile force feedback has been undertaken by Airbus [17] citing motivations of 'anticipating and/or preventing laying errors' as well using the data for longer term optimisation of the process.They present a concept for an AFP or Automated Tape Laying (ATL) roller equipped with pressure sensing piezo-electric transducers to give real time feedback.No actual performance data or examples are available at present.Alternatively compaction force has been used previously to inspect composite materials, using the data to infer the through-thickness permeability dry fibre preforms [18].A promising option that can analyse three dimensional shape rather than force has been developed at Bristol Robotics Laboratory (BRL) [19].This optical tactile sensor known as the 'TacTip' has been shown to be capable of analysing a wide range three-dimensional shapes in real time and was identified as having potential to operate in a composite layup environment [20].

The TacTip
The TacTip is an optical tactile sensor, capable of analysing the three-dimensional shape of a surface.A schematic diagram of a sensor is shown in Figure 1.The TacTip assembly is typically mounted onto a six-axis robotic arm which can precisely and repeatedly position the sensor.In operation the TacTip is lowered onto a surface to typically 6mm deflection and then raised again, an action which will be referred to as a 'Tap' from here on.The only electronic components in the sensor are a ring of LED lights and a Microsoft Cinema HD webcam.The lower part of the sensor which deforms around a surface profile during a tap is a hemispherical membrane filled with a clear gel.On the inner surface 'pins' which protrude perpendicularly to the surface of the membrane into the gel core.The tip of each pin is painted white to act as an optical marker, contrasting against the black membrane material.As the hemisphere deforms around a shape, the pins move and rotate relative to their original position.Lit by the ring of LEDs, the movement of the white pins tips is recorded by the inbuilt camera, examples of which are shown in Figure 2. The camera footage is then processed in real time using OpenCV for Python to output the position of every pin in terms of X and Y pixel locations in each frame.This data is then passed onto MAT-LAB for further analysis which is outlined separately in section 6.The spatial resolution available from the sensor is much smaller than the spacing of pins might suggest, with the sensors capable of 'super-resolution', creating a sub millimetre sensitivity [21].In a previous study by the author the standard form of the TacTip has been proven to be able to detect a range of composite manufacturing defects including wrinkles, foreign objects and bridged concave corners [20].In that work it was acting purely as a sensor, and the aim of this new study is to develop a version that can apply significant compaction forces at the same time as tactile shape sensing to create a functional layup end effector.

Sensor design
The sensor design chosen for the basis of this project was similar to that used in a range of other studies, featuring a 40 mm diameter hemispherical skin, which is 1mm thick with 127 pins on the inner surface.The current materials in use are a rubberlike silicone (TangoBlack plus (FLX980)) for the skin, shore hardness (A) 30, and silicone gel (RTV27906) as the medium.The original versions of the TacTip were designed solely as a sensor and the gel filled tip is only capable of applying 10N or an average pressure of approximately 0.06MPa, just over half atmospheric pressure.To enable effective compaction of composite plies, especially into and around tight corners, this force needed to be dramatically increased.A range of approaches to making a 'hard' TacTip such as thicker membranes or alternative materials were tested in a master's project by Almas [23].The most successful of these designs changed the Gel medium for a Polydimethylsiloxane (PDMS) elastomer (Sylgard184, manufactured by Dow Chemicals [24]).The new harder sensor was demonstrated as capable of applying enough pressure to consolidate prepreg onto a mould such that it could be then cured into a finished laminate.To quantify the applied force, a compression test of an elastomer filled TacTip against a flat surface has been completed and the results are presented in Figure 3.It shows the TacTip is now able to apply over 400N of force, generating an average con-tact pressure of approximately 2.26 MPa, an approximately 40-fold increase compared to the original TacTip.As a sensor, the hard elastomer filled TacTip was shown in the previous study to retain the basic functionality of the original sensor by differentiating between basic shapes such as flat surfaces or small three-dimensional features.This new study aims to prove the harder TacTip is sensitive enough to detect a range of real composite layup defects.

Data collection and analysis
The data output from the TacTip is the X and Y coordinates of each of the 127 pins.While there have been recent efforts to directly calculate the deformed shape of the hemisphere from the pin positions [25], the most successful applications of the TacTip have used a different approach.This is based around 'training' the TacTip by exposing it to range of known stimuli and recording the responses prior to it being applied.When the TacTip is being applied as a sensor (referred to as 'Testing' from here on) the incoming X/Y coordinates are compared with those from the pre-recorded library of data collected during the training.If the tap is significantly alike to any of the training taps for that location, then a match is identified, and the shape of the test surface can therefore be inferred.For the training to work effectively in many of the previous application of the TacTip the 'training' data has consisted of a large number of repeats.For the specific application of composite layup the data has to be collected during the layup a process rather than just repeatedly tapping onto a static object.This makes collecting such large volumes of training data very costly and ideally it should be avoided.In this study we set a target of maximum three repeats for each tap to fully train the system.This limited some of the techniques that could be applied such as neural networks or other machine learning tools which have been used in previous studies.

Anomaly detection
The detection of defects during the manufacturing process can be broken down to an anomaly detection problem, where it is possible to train on a 'correct' layup, and the task of the TacTip is to detect if test layups differ in any way, which would suggest a defect has occurred.A wide-ranging review of techniques for anomaly detection is presented by Chandola et al. [26] and it was concluded that given the limited training opportunities, that the best approach would be to directly compare the location of the pins in testing and train data, utilising a 'nearest neighbour approach'.Two taps are compared by selecting the position of each pin in one tap and calculating the linear distance to the position of the same pin in the other tap, as shown schematically in   The 'nearest neighbour' approach is applied when comparing a single 'test' tap to the three 'training' taps from the same location.For every pin within the test tap, the distance to that particular pin in each of the three training taps is calculated, but only the smallest of the three distances is input in the final tap 'score', as shown schematically in Figure 4.This approach can reduce the impact of any erroneous training data.A data analysis program was written in MATLAB, capable of processing training and testing data in real time on a standard laptop.An image of the TacTip being applied while real time data is displayed on the MATLAB user interface is shown in Figure 5 with a green to red colourmap showing the acceptance level for the tap.

Fault tolerance
Fault tolerance is a key characteristic for a successful sensor, such that it can continue to operate even if some elements of the sensors should fail [14].The standard TacTip uses a robust and proven pin tracking algorithm but the analysis software has been developed such that in the unlikely case of one or more pins being temporarily lost during tracking or becoming damaged, it can still function.This is achieved in the software by selectively removing the data for pins which are missing in either the test or training data, or which show a displacement much greater than the surrounding pins, suggesting an error.This creates a sensor that rather than completely failing as more pins are removed, would just gradually reduce in sensitivity.With a total of 127 pins in the head, removing up to around five pins will only have a very small detrimental effect of the performance in most sensing applications.Only in the very unlikely event of having to remove a significantly greater number of pins, all of which are located around a small localised defect, would the sensor fail to function correctly.

Testing methodology
The robotic arm was programmed to tap the TacTip onto the sample layups in a Rasta-scan pattern such that the entire ply was compacted onto the mould.The TacTip was trained by scanning three 'correct' layups and during the testing phase the exact same tap patterns were repeated on five examples of each test scenario.The TacTip was trained and tested across a range of composite layup and the results are presented in section 8.For every tap the TacTip was raised and lowered at 80mms −1 and paused for 0.2s at the lowest point of each tap to allow collection of a few video frames of the TacTip in a steady state.Each dot in the plotted results corresponds to a tap of the TacTip.All the figures shown in Tables 1-3 were generated in real time as the TacTip was applied to prepreg material, and are all plotted on the same Colour axis and colour map in MATLAB, with the exception of the AFP tow overlap and gap trials in Table 3 which use a narrower limit to highlight very subtle defects.

Flat Panel layup
The first round of the tests was on a flat rectangular layup.Five 'correct' versions of the flat layup were placed in the correct location and then compacted and scanned by the TacTip, with the outer taps going beyond the edge of the ply.All five versions produced 100% matches to the training data.Further layups containing a wrinkle, plies or films placed below the surface or incorrect positioning or ma- terial type were all trialled using the same method.In all cases where there was a defect it is was identified by the Tac-Tip.Examples of the results are shown in Table 1 alongside schematics of the defective plies.

Complex layup
The TacTip was also trialled on a layup featuring convex and concave corners with sharp almost 0mm radii.The end effector could compact the woven prepreg onto the mould without any bridging or defects as shown in Figure 6.Two

Automated Fibre Placement
In AFP, one of the most common defects is dropped tows, where one of the tows breaks or stops being laid down [27].
The TacTip was trained on a layup of Uni-directional 6.35mm (1/4") tapes as used in AFP.In the test layup the last third of one of the tapes was removed to simulate a dropped tow and this was successfully detected by the sensor.Another common defect is gaps or overlaps between the tows.A gap and an overlap of 3.18mm (1/8") were created manually and were successfully detected by the sensor.

Repeatability
Every trial presented in section 8 was repeated five times with excellent repeatability.Figure 7 shows a selection of  showing location of these selected taps, for the full plot see Table 1.
data from one of the flat panel tests in bar graph format.
The error bars represent one standard deviation from the five repeats at this trial.Of the 3630 Taps completed during the repeats of the results shown in section 8, there was only a single tap which gave a single false defect decision.

Conclusions and Future work.
A new version of the 'TacTip' tactile shape sensor featuring an elastomer core rather than the original soft gel core has been shown to be capable of detecting a range of composite defects while simultaneously compacting prepreg material onto a mould surface.It successfully detected wrinkles, bridges and a range of foreign objects including thing films placed below the ply.The new elastomer core increased the force that can be applied from under 10N to over 400N.The detection was all completed in real time using a standard laptop PC running Python and MATLAB.This successful study has shown that tactile shape sensing with TacTip technology could be a genuine option for quality control in future automated systems.
9 Future work

Hardware optimisation
Despite substituting the Gel core for an Elastomer the Tac-Tip has retained sufficient tactile shape sensing capability to detect the defects during layup.However the sensitivity of the elastomer sensor is lower than that of the original gel based sensor but there is further scope to fine tune the hardness of the elastomer by adjusting the hardness/mixer ratio or changing material entirely to tune the balance between application force and sensitivity [28].
In addition to adjusting the hardness, the shape of the sensor can be optimised.The new Elastomer filled version of the TacTip presented in this study uses the same shape as the existing versions which have been designed to operate on a wide range of tactile shape sensing tasks but it may not be optimal for layup tasks.The TacTip technology could be translated into different shapes like those used in composite layup technologies such as that developed by Elkington et al. [29] or Bjornson et al. [30].A cylindrical version of the TacTip has been successfully demonstrated, designed for medical use in detecting tumours on the inside of the Gastrointestinal gland [31].This has the potential to form the basis for a sensor that can roll across a surface rather than probe, like those used in automated processes such as AFP or ATL.

Analysis optimisation
The difficulties in collecting a large set of a training data limits the data processing approaches that can be applied to this anomaly detection problem.However, the current version uses a very stripped down but effective and efficient processing system which could be built upon in further studies, adding in rigorous statistical analysis.For example, the current approach only considers each tap result in isolation and does not consider the spatial relationship between the taps.For example an isolated tap with a small difference might be ignored, however if a significant num-ber of adjacent or aligning taps all show a low level defect, this data could be flagged as a potential error.
The current version operates only as an anomaly detection system and does not attempt to classify the 'type' of defect.The TacTip and other tactile sensors has been demonstrated in multiple other studies as being able to differentiate between different object types [32,33].There is potential therefore to build a version that could classify whether a defect is wrinkle, misplaced ply or other type of defect.This data would be useful for informing the resulting decision about how to correct the defect.TacTip data has been used to adjust position of an object in a grasp [34] or to the follow the edge of an unknown shape [34] showing the potential for real time use of the data to inform later automated actions.Information about the defect could be used to adjust parameters like the speed, temperature or force.The next step would be using the information to inform automated corrective actions similar to how humans operate in hand layup.

Figure 1 :Figure 2 :
Figure 1: (Left) Schematic of the TacTip construction, (Right) Image of the TacTip tapping down onto a prepreg ply with wrinkle.

Figure 3 :
Figure 3: Force displacement graph showing how the Elastomer core TacTip can apply up to 400N at its typical displacement depth of 6mm.

Figure 4 :
Figure 4: (Left) Schematic of how difference in pin location between taps is used to detect anomalies.Here the Black dots show the TacTip response to a correct flat surface while the yellow dots show the response to an anomalous contoured surface.(Right) Schematic Diagram of the 'Nearest Neighbour' Technique, in this case the test tap (Black) was closest to the third of the training taps.

Figure 5 :
Figure 5: (Left) Image of the TacTip being applied to a layup while (Right) Real time data is displayed to the user via a Matlab scatter plot.Green = Correct, Red = Defect found.

Figure 4 .
Figure 4.The overall 'score' for each tap is the average pin training-testing position difference across all the pins.The 'nearest neighbour' approach is applied when comparing a single 'test' tap to the three 'training' taps from the same location.For every pin within the test tap, the distance to that particular pin in each of the three training taps is calculated, but only the smallest of the three distances is input in the final tap 'score', as shown schematically in Figure4.This approach can reduce the impact of any erroneous training data.A data analysis program was written in MATLAB, capable of processing training and testing data in real time on a standard laptop.An image of the TacTip being applied while real time data is displayed on the MATLAB user interface is shown in Figure5with a green to red colourmap showing the acceptance level for the tap.

Table 1 :
Results from Trialling the TacTip on a range of Flat layups.1A -Correct ply 1B -Wrinkled ply 1C -Small square of prepreg beneath the layup 1D -Small square of backing film beneath the layup 1E -Ply misplaced by 3mm down and left 1F -Incorrect ply: Different material thickness Good Match: Passed Medium match: Possible Defect Poor Match: Failed

2 :
Results of TacTip scans on the Complex curved layup Correct ply Bridged ply Extra strip of Ply under the surface Good Match: Passed Medium match: Possible Defect Poor Match: Failed defective versions of the layup were scanned, one featuring a strip of prepreg placed below the layup and another with a filler placed in the concave corner to simulate a bridge.As shown in Table2, the TacTip successfully detected these defects during the layup of the ply.

Figure 6 :
Figure 6: Image of a ply successfully laid onto a curved mould shape by the TacTip.

Figure 7 :
Figure 7: (Left) Selection of tap data presented with error bars showing the standard deviation across five samples.(Right) Illustrationshowing location of these selected taps, for the full plot see Table1.

Table 3 :
Results of TacTip on a simulated AFP layup.