Auralization of Concert Halls for Touristic Purposes

1 Introduction

Modern information technologies have become an integral element in the landscape of experiencing art and cultural heritage. Apps for a museum visit or live streaming of plays or concerts into private homes or public places belong to the common toolbox used for enhancing the audience experience. Virtual walks through 3D models created from real-world heritage sites have been employed in many areas, offering a valuable alternative to the physical experience, e. g. in architecture, archaeology, art, games, or tourism.

This paper will describe a novel acoustic experience of culture with auralization as an innovative technology-based technique. Auralization is a procedure designed to model and simulate the experience of acoustic phenomena rendered as a sound field in a virtualized space. Virtual walks through 3D models created from real-world heritage sites have meanwhile become a common occurrence in many areas, offering a valuable alternative to the physical experience, e. g. in the COVID-19 time, people have attended all kinds of acoustic-based events, such as concerts, theatre plays, seminars, etc. with the help of streaming technologies [4].

Our approach will merge the digital acoustic experience with the 3D visual experience, creating a virtual soundscape that will allow different stakeholders and target groups to experience sound exactly as it occurs in a specific physical environment or space. A person will be able to perceive the difference in acoustics at the back of a concert hall or in the first row or the balcony or the middle of the orchestra on stage.

The first step is the 3D modeling of the concert hall with all acoustic parameters. The second step is the simulation of the listening and viewing simulation from all possible places in the concert hall for different target groups:

1. For the artists i. e. the orchestras, soloists, singers, conductors, directors, choreographs, actors, etc. by providing them a better understanding of the acoustics of the spaces they perform in.

2. For the audience e. g. aficionados of wanting to experience certain music in different halls or theatres.

3. For the culture and tourist market e. g. new ways of monetizing digital events, or using auralized spaces to enhance their marketing strategies or ticket sales.

2 State of the Art

The central part of the AURA project is the auralization of digitalized models of different concert halls from three cities. Auralization describes the process of rendering audio within a sound field in a way that simulates the binaural listening experience at any position in the 3D environment [7]. Its main goal is the recreation of acoustic characteristics of an indoor or outdoor space [12]. Source materials and room geometry will influence how audio will be filtered within the environment.

Before auralizing said models, the concert halls need to be digitalized and recreated in 3D. The technological development in the field of digital surveys such as LIDAR (Laser Imaging Detection and Ranging) allows the realization of three-dimensional data highly descriptive, which can be combined with the technologies of Augmented and Virtual Reality (AR and VR). The conjuncted application of these technologies offers the possibility to develop tools for the dissemination of information and data collected during scientific research and to improve the classic workflow between researchers, providing realistic looking 3D models implementable and navigable in remote [6].

Digital laser-scanning 3D has developed a lot in the recent period allowing to process highly accurate point clouds and providing more and more reliable and descriptive data thanks to the integration of high-resolution cameras which acquired color and illumination, providing a 3D model with a photorealistic appearance. Thanks to the improvement and the diffusion of AR and VR technologies which are now easily accessible to the public, the great amount of data acquired by laser-scanning can be used to establish a virtual environment. The challenge of the research is to manage and process the great data volume acquired by LIDAR for visualizing the dataset in a highly realistic and immersive way [9], using several devices such as smartphones or tablets. The aforementioned workflow can improve a lot of the work of research in the field of cultural heritage conservation, archaeology, architecture, and engineering, and offers the opportunity of disseminating culture in a widespread and more accessible way.

In particular, the AURA project aims to create multi-sensory models, reliable and performing both in terms of graphic rendering and virtual fruition, both in terms of acoustics, exploiting the integration of point clouds within gaming technologies. This allows reducing the long modeling processes of complex virtual environments, modeling only the elements with which the user interacts in a multi-sensory way.

When creating auralized versions of 3D models different plugins can be used. Many of which allow easy parameterization to determine how certain parts of a room geometry interact with audio. Since not all plugins are still actively supported and developed a chosen selection will be analyzed in more detail in section 4.

3 Digitalization of Concert Halls

The AURA project involves the development of auralized 3D models of three musical venues: the Konzerthaus in Berlin, the Teatro del Maggio in Florence, and the Opera in Lviv. The case studies examined allow the development of a methodology applicable at the European level, taking into account three emblematic architectures, belonging to three different historical periods and characterized by three different architectural styles. The Konzerthaus in Berlin, built on a project by Karl Friedrich Schinkel with the name of Schauspielhaus in neoclassical style at the beginning of the nineteenth century; the Lviv Opera House, built at the end of the nineteenth century in neo-baroque style with some decorative elements Art Nouveau, enriched with stucco, statues and oil paintings. The theatre of the Maggio Musicale Fiorentino is a contemporary architecture, built between 2009 and 2014, at the entrance of the Cascine Park, in Florence, designed by the architect Paolo Desideri.

The first step of the project is the digitalization of the Built Heritage, which requires a preliminary analysis of the architecture and its urban and environmental context to plan digital survey campaigns. The project involves the combined use of point clouds and NURBS 3D models, therefore the digital survey is made using contemporary 3D laser-scanner technologies, that provide a high reliable metric data useful for the steps of data discretization and NURBS modeling.

Figure 1

Laserscan of the Teatro del Maggio Florence.

The three locations show several differences which influence the choice of the methodology of analysis and technical instruments to be applied. The two historical theaters of Berlin and Lviv required the integrated use of LIDAR and SfM (Structure from Motion) photogrammetry, which allows the creation of both the colored point cloud for the virtual environment and the photographic ortho-images of the surfaces, needed to map the NURBS model properly and reliably. The theater of the Maggio Musicale Fiorentino instead, due to the recent construction and the presence of contemporary materials, required the sampling of each surface to be reproduced in the 3D model using sampled material recreated with rendering software. SfM is a photogrammetric range imaging technique that estimates three-dimensional structures from two-dimensional image sequences which can be acquired close range, using cameras or aerial, using UAV (Unmanned aerial vehicle) such as drones. This technique provides 3D models which can be used in this project to extract photographic textures which, appropriately optimized, allow to map the surfaces of the NURBS models and return a photorealistic environment to set up virtual experiences. The highly reliable metric data obtained through the processes of post-production of the point cloud, after a careful discretization of morphometric information, allows developing 2D elaborates in the CAD environment necessary for the successive phases of modeling NURBS using software such as Rhinoceros.

The advantage of using LIDAR systems is the high graphic rendering of the model, which uses photogrammetry technologies by coloring the points, returning the image of the monument at the exact moment it was acquired. The variation of the shadows throughout the day would require the acquisition of the survey in an overcast sky condition to obtain a neutral model in which it is possible to apply lights and shadows in the last rendering processes. The 3D laser scanner data acquisition required careful planning of the positioning of each scan to avoid the formation of shadow cones on the surfaces of interest, also in consideration of the furnishings and decorative elements present in the interiors of the three theaters. Furthermore, the field of view of the instrument, which rotate generally between 360 ° on the horizontal axis and 270 °–300 ° approximately on the vertical axis, creates in each scan a circular area without data that can be offset by the integration of side scans. Indeed, an accurate acquisition for the virtualization requires usually more scans despite those for architectural purposes (such as redrawing two-dimensional plans and sections). This great amount of acquired data needs to be processed and optimized to make it usable by different types of devices (e. g. desktop, mobile, ...), requiring a choice of scans to be used for the creation of the virtual environment.

The field survey is followed by the post-production of the data through the registration of the individual scans and the creation of the overall point cloud, using software such as Autodesk ReCap or Leica Cyclone. This first complete registered point-cloud, which reliability must be properly verified through the analysis of the error report of the registration, includes a large amount of improperly generated data due to the nature of materials such as glass, metal, and shiny materials that can generate uncontrolled reflexion and distortions of the laser pulse. This requires a careful cleaning phase of the registered point cloud, which can be carried out within the management software through the use of section plans and the limit box, an instrument of Autodesk ReCap or Leica Cyclone, to isolate improper elements from those acquired correctly.

To extract manageable data for the virtual experience, it is necessary to develop a storyboard of the final experience, evaluating the visiting paths and their optimal visualizations to verify the fluidity of these routes through remote navigation of the point cloud. Once the point cloud has been optimized, it is possible to identify, through the visit path, the scans necessary to recreate the virtual setting. The point cloud can then be further optimized and regularized through the use of special management software, such as Rapidform XOR, which allows to unify the individual scans and lighten the original file by arranging the points according to a regular grid. This appropriately processed file is then imported into software such as Unreal Engine, equipped with the Lidar Point Cloud plug-in, or Unity 3D, which allows managing the atmosphere, inserting lights, objects, and so on.

The choice to integrate point-cloud models with NURBS models is due to the need to apply the acoustic parameters to the surfaces for auralization. In fact, for the auralization of the models has been developed a special process of modeling based on the digital survey, which provides an accurate process of semantic subdivision of the elements, which must take into account the nature of the materials, surfaces, and construction characteristics of the panels and coatings. Together with the computerization of the morphological and dimensional characteristics of the detected elements, through the application of the specific characteristics and the relative acoustic coefficients, contribute to the definition of the model from the acoustic point of view.

4 Auralization of 3D Models

The main goal of the project is the creation of auralized versions of the digitalized concert halls. Therefore certain tools are needed. Editors like Unity are best to work on the auralization. The 3D models can easily be imported when in .fbx or .obj format. Unity comes with build-in 3D sound which can be activated in the audio source component. Parameters like spread, doppler level, the type of volume roll-off, and minimum and maximum distance can be regulated (Figure 2). This helps when creating spatialized audio for a game or application. However, since Unity does not take into account how room geometry or materials affect the sound, when creating a fully auralized scene extra audio plugins are needed.

Figure 2

Audio source component in Unity.

Firstly, different auralization plugins were researched and tested. The Oculus Spatializer by Facebook/Oculus, Resonance Audio by Google, and Steam Audio by Valve were chosen, because they met the necessary criteria in being free to use, actively developed and supported, and being compatible with Unity. All of them come with certain strengths and weaknesses, so the choice is mainly based on the specific use case the plugin is being used for.

The Oculus Spatialzer offers a clear directionality of objects and has effects for acoustic shadowing and near-field sounds. But since there is no form of occlusion, it was not suitable for the use case of auralizing 3D concert hall models. Occlusion means how sound is affected when passing through solid objects [5].

Resonance Audio benefits from consistently good spatialization and near-field effects. Occlusion can be calculated in real-time, but since there is only a basic on- or off-setting, it can be quite unrealistic and therefore not suitable for the use case [5].

Steam Audio comes with the best sounding occlusion system and has options for setting geometry and acoustic materials. It allows for gradual amounts of occlusion and offers frequency-dependent transmission [5]. Integration into Unity is straightforward, since only a package needs to be imported, making it the best fit for the AURA project.

After successfully importing the Steam Audio package into Unity and preparing a demo scene, the Steam Audio source component needs to be added to the game object containing the audio source. This component contains direct sound settings like occlusion or the type of head-related transfer function (HRTF) interpolation, as well as an option for regulating the directivity of sound. Settings for indirect sound like reflections or whether real-time or baked simulation should be used [1] can also be found here (Figure 3).

Figure 3

Steam Audio source component.

Any part of the room, for example, the floor, walls, ceiling, and all objects within the room, are tagged with a Steam Audio material and Steam Audio geometry component. Depending on the object, materials like carpet, wood, glass, or custom ones can be set (Figure 4). This will affect how the audio will sound when traveling through the environment.

Figure 4

Steam Audio material and geometry component.

Lastly, after setting up the audio sources and room geometry the scene needs to be exported and can then be tested (Figure 5). While moving through the environment clear differences can be heard when being very close or further away from the sound. Changing parameters like the material will result in very different audio. Choosing carpet for example will make the audio sound a lot more muffled than selecting glass.

Figure 5

Unity scene after exporting.

After experimenting with various parameter settings, the environment can be realistically set up to represent real-world conditions.

5 Usage of Auralization for Touristic Purposes

The common challenge is improving and fully utilizing the tourist potential in Berlin, Florence, and Lviv. Theatre and music performances are essential things in cultural heritage. The fostering of new markets and new audience approaches for the cultural industry can be realized by analyzing the preferences of opera visitors in each city. The specifics of the Opera House building, its acoustic, the available cast, and the city’s aura are decisive in the popularity of specific performances. Knowing the nature of the sound of instruments and sound reproduction in the Opera House, we can increase the number of visitors. Artificial intelligence methods are used for preference analysis. Preference analysis is organized based on hidden dependencies mining. One of the possible techniques to find them is the use of associative rules. Associative rules are an effective mechanism for finding relationships between the personal parameters of the visitor and the parameters of music performed in a particular Opera House.

Only a limited choice of datasets related to classical music was available. The chosen data file^[1] consists of 1010 rows and 150 columns (139 integer and 11 categorical).

1. The data contains missing values.

2. All participants were of Slovakian nationality, aged between 15–30.

1. Music preferences (19 items)

2. Movie preferences (12 items)

3. Hobbies & interests (32 items)

4. Phobias (10 items)

5. Health habits (3 items)

6. Personality traits, views on life & opinions (57 items)

7. Spending habits (7 items)

8. Demographics (10 items)

The task is formulated as follows: To find the relations (preferences) between features for instances of classical music performance visitors. All variables given above are taken into account. The target variable is “Opera”.

First, data preprocessing was organized. Multiple Imputation by Chained Equations (MICE) [8] were used for missing data imputation. In total, more than 1000 values were imputed. Attributes Punctuality, Lying, Alcohol were removed.

Next, one-hot-encoding was used for categorical attributes. Totally 257 attributes are taken into account.

Statistical analysis is given below:

1. Music preferences (women or men) (Figure 6)

2. Clustering using k-means [2] Figure 7. Number of clusters is chosen using gap-statistics [11]

Figure 6

Music preferences (women or men).

Figure 7

Clustering using k-means.

Several predictors based on Machine Learning (ML) algorithms are used. Root Mean Square Error (RMSE) and Mean Absolute Percentage Error (MAPE). The best models are random forest and knn with minimal error.

The regression tree looks like the following (Figure 8). It can be used for feature selection. The significant features are given below.

Figure 8

Regression tree.

The significant features for classical music audience are variables used in tree construction:

[1] Active.sport, Celebrities, Changing.the.past, Classical.music

[5] Elections, Gardening, Hip-hop..Rap, Musical

[9] Musical.instruments, Thinking.ahead, Unpopularity

Next, the preference analysis is developed using associative rules. The task of finding associative rules is to identify sets of objects that are commonly encountered in a large number of objects. The task of sequential analysis is to search for frequent sequences. The main difference between the tasks of sequential analysis from the search for associative rules is to establish a relationship of order between objects. The presence of a hierarchy in objects and its use in the task of finding associative rules allows you to perform a more flexible analysis and obtain additional knowledge. The results of the problem’s solution are presented in the form of associative rules, conditional and the final part of which contains sets of objects. Associative rules mining and interpretation allow building user scenarios and related business models for sustainable implementation.

Association rules (AR) are a set of special rules that allow you to find and describe conformities in large sets of data. Basic concepts in the theory of associative rules are subject set and transaction. A subject set is a non-empty set of elements that can be part of a transaction:

where i_k are elements included in the subject sets k=1, n is the number of elements of the set I.

The transaction represents a certain set that has some elements of the set I occurring together. The transaction also has a unique identifier TID (Transaction ID).

There is a certain set of transactions in the database:

where t_i is relevant transaction, m is total number of transactions.

Between the elements of transactions, you can set the patterns in the form of associative rules, consisting of two sets of objects: X={jk∣jk∈I}, that is named a condition and Y={jk∣jk∈I}, is called a consequence. An associative rule describes the relationship between sets of objects that are conditional and consequential and recorded X→Y. In this case, the sets X and Y must not intersect: X∩Y=∅. The main indicators of the relevance of the associative rule are support and confidence.

Have a set X and an associative rule X→Y. The setup support for X will be:

where

The concepts of set support and associative rule are closely related to another characteristic of an associative rule, the confidence, which is calculated as the ratio of the support of the set, which has both a condition and a consequence (in other words, it is the support of the associative rule), to support the set which has only a condition.

To determine the significance of the rules, there are used threshold values of minimum support and authenticity of MinSupp and MinConf, that is usually determined by the users of the system or experts, is based on their own experience:

Methods for finding associative rules find all associations that meet the constraints of support and confidence. However, this leads to the need to revise a sufficiently large number of associative rules, which is desirable to be reduced in such a way as to analyze only the most significant of them.

There are objective and subjective measures of the relevance of the associative rule. Objectives are the above support and confidence. The subjective measures of meaningfulness are an elevator and leverage. The lift is determined by the ratio of maintaining the associative rule to the product support condition and effect separately:

One of the adapting notions according to associative riles needs is not only of relational data or NoSQL. The followers of the concept of NoSQL language emphasize that it is not a complete negation of SQL and the relational model. Still, the project comes from the fact that SQL – is an essential and handy tool that cannot be considered universal. One problem that points to a classical relational database is dealing with massive data and projects with a high load. Therefore, the primary objective approach is to extend the database if SQL is flexible enough and not displace it wherever it performs its tasks.

The concept of R- “curiosity” of the associative rule is expanded, which is used to select the identified associative dependencies to reduce their number. R- “curiosity” is defined as exceeding the level of trust or support of expected values by R times. Expected values are calculated for associations that have at least two features on the left side of the rule, assuming that they are statistically independent. Informativeness makes it possible to assess “curiosity” if, for example, the level of support for the rule is R1 times higher than the expected value, and the level of trust is R2 times lower and vice versa [10].

The visualization of hidden dependencies is given in Figure 9.

The “interest” of the association is defined as the excess of the informativeness of the association of the expected value. Thus informativeness is considered as a generalization of the concept of R- “curiosity” of the associative rule.

Figure 9

Visualization of association dependencies.

In addition, monetizing digital events or using auralised spaces to enhance their marketing strategies or ticket sales can be realized. It is built on acoustic specific of the particular Opera House and preferences of each visitor. Ticket purchasing might be supported by giving an impression of the different acoustic experiences in the differently priced seats.

Figure 10

3D laser-scanner digital survey of the theater of Maggio Musicale Fiorentino. On the left: perspective view of two color laser-scanner scans that highlight the presence of shadow cones. On the right: section and plan with the positioning of the individual laser-scanner scans.

6 Demonstration

The first example, concerning the Florentine Maggio Theater, illustrates the data acquisition phases carried out during the digital survey campaigns and the preliminary steps for the establishment of a simplified model divided into layers based on the materials, which will be used for the implementation of acoustic data aimed at auralization. The first phases of the digital survey were carried out in the theater of Maggio Fiorentino through the acquisition of 177 color scans through the use of a laser scanner Z+F IMAGER 5016 and 109 color scans using a Faro Focus M70 (Figure 10).

Both the instrument not only provide position information but also color data associated with each point, establishing realistic-looking 3D point clouds integrated by well-balanced high-resolution images in the representation of the original color of the elements and therefore highly descriptive, which required a post-process production and cleaning of the data to remove the parts of point cloud generated improperly. The refined data has been used for the realization of 2D CAD elaborates, plants, and sections necessary for the realization of the 3D model NURBS [3]. The next phases of construction of the model of the theater of Maggio Fiorentino and the attribution of the chromatic and acoustic characteristics of the materials are currently underway. As already seen above, in the case of the newly built Florentine venue, the model will require sampling of the textures of the materials and the development of materials for rendering, while in the case of Konzerthaus and Lviv Opera it will be necessary to map the individual exploded surfaces of the NURBS model through SfM photogrammetry campaigns, which will allow, through the establishment of 3D mesh models, to extrapolate highly reliable photographic photo planes of surfaces.

Figure 11

Import of the point cloud model of the theater of Maggio Musicale Fiorentino into the Unreal Engine software. Detailed comparison of the point cloud model in a virtual environment. Above: detail and perspective view of the interior with the higher density point cloud. Below: detail and perspective view of the exterior.

For the creation of the virtual environment, it was decided to use the portions of the point cloud that concern the square in front of the theater and the central hall, from which it will be possible to access the main hall modeled in NURBS and auralised. Based on the planimetry with the positioning of the scans, it was decided to use only those made using the laser scanner Z+F IMAGER 5016, which thanks to the planning of the survey campaign had been used for the environments provided for the visit. This laser, which acquires over 1 million points per second at a maximum distance of 360 meters and has a wider than average field of view of 360 ° × 320 °, offers precision even at long distances and is equipped with a camera HDR which returns a well-balanced color data. The necessary scans, identified for the establishment of the final model, were unified and exported in the .LAS format, supported by the Lidar Point Cloud plug-in of the Unreal Engine software together with the .PTS, .TXT and .XYZ formats. Within the software, it is possible to modify the display parameters of the point cloud, in particular the total number of points displayed, the size of the point, and the shape of the point. While for the exteriors it was seen that an optimal display was obtainable with a total of 5,000,000 points and a point size of 1, for the interiors it was necessary to increase the number of points to 100,000,000 with a point size than 0.6, creating a heavier file (Figure 11). Similarly, the square shape of the point is more suitable for distant environments, filling more the space of the backdrop, while for the interiors it has been noticed that the visualization is more pleasant with the circle point. The further phases of the research foresee the integration between the point-cloud model with the auralised NURBS model of the internal main hall and the creation of a VR app for visiting the theater and listening to the concerts.

To test the mentioned auralization plugins a first auralized prototype has been created using an already existing 3D model of the Konzerthaus Berlin. The Great Hall of the Konzerthaus was first used to give an impression of how the auralization plugin will work and how the audio will sound like. The Unity test scene can be seen in figure 12.

Figure 12

Different perspective in Unity test scenes of the Great Hall of the Konzerthaus Berlin.

Changing the position within the scene influences how the audio will sound. Different selections for material parameters will also have a big impact on the sound. During testing several settings have been tried out. To furthermore test the plugin and its possibilities a second test scene was implemented. The Small Hall of the Konzerthaus Berlin was auralized using the same method as for the Great Hall. The same good results were achieved. The Unity scene can be seen in the following figure 13.

Figure 13

Different perspective in Unity test scenes of the Small Hall of the Konzerthaus Berlin.

7 Conclusion

The project aims at the construction of multisensory models, focused on the aspects of photorealistic visualization and acoustic rendering of site-specific music. For their construction, it is necessary to provide, in addition to the perfect technical realization of the models and their acoustic and visual features, a path of access to the theater and the possibility to choose the seat by the user. For this reason, several outputs derive from the digital survey, exploiting their morpho-dimensional and perceptive features. The models must be performative to represent the atmosphere of the theater itself and must be able to reconstruct the access to the building and to ensure the opportunity to choose the user’s positioning within the places available in the virtual room.

The creation of said models will be the central goal of the AURA project. Fully auralized versions of the three mentioned concert halls will be implemented. A second and third digital survey will be carried out, first of the Konzerthaus Berlin and lastly of the Lviv Opera House. The auralized models will then be used to conduct several case studies, analyzing the best uses and opportunities for auralization within the cultural context.

The implemented machine learning model and data mining techniques can be used for audience prediction. During the further course of the project, these models will help to develop business and monetization methods. Resulting in the publication of reports introducing the newly developed business plans as well as the analyzed case studies.