The metaphorical collision between chocolate and peanut butter has been applied from the academically metaphysical to the commercially slapstick, but that particular confectionary equation has the benefit of years’ worth of experience, consumer focus groups, and the kinds of cookies that reside in your pantry instead of your hard drive. Two other unique universes that have avidly pursed their own evolutionary paths remain still largely apart, but plenty of people have been wondering if there’s some kind of technological meet-cute ready to happen between the worlds of three-dimensional imagery and immersive audio.
“From our perspective, the sense of hearing is something we process subconsciously, and it’s actually faster than visual processing,” states Sid Bhargava, Product Lead at Spatial, which develops immersive soundscapes for wellness, entertainment, and other verticals from its offices in Scottsdale, AZ and Emeryville, CA. “I think we can use audio as a visceral way to create a connection with an end user that lasts beyond the visual augmented reality. [Immersive] audio can bring a sense of depth and compel the user to engage and explore an experience more deeply. So I don't think you can have one or the other by itself for a fully immersive experience.”
Bhargava points to the ongoing series of immersive fine-arts “experiences,” such as the ubiquitous Immersive Van Gogh pop-up exhibits that envelop visitors in images of an artist’s work while a gossamer soundtrack acts as an immersive film score. He also cites Spatial’s own collaborations with Meow Wolf, a Santa Fe-based underground art collective that’s using similar media technology to convey complex narratives, such as its Mega Mart experience in Las Vegas that playfully (and perhaps ominously) suggests a future retail experience closer to Minority Report than Walmart (See “Meow Wolf Builds Artful Mega-Experiences” by Barbara Robertson in the April 7, 2022 edition of HoloWire).
These artistic immersives have done well enough as entertainment, but Bhargava says the combination of immersive sound and visuals has significant practical applications, as well, particularly for health and wellness. “We've been working with places that have workers that are on the front lines of stressful situations,” he says, pointing to Spatial’s development of stress-shedding “wellness rooms” with hospitals and police departments. “We're seeing a lot of actual positive impact on people using biophilic nature sounds and helping reduce stress levels by using just audio and visual elements.
For some in the audio sector, the combination of immersive sound and image feels natural. “They're developing independently and that's not a shock: audio and video have always developed independently, but they've also developed in a symbiotic relationship,” contends Scott Sugden, director of product management for electronics and software at L-Acoustics, a Paris-based live-sound system manufacturer whose PAs system are one of a handful of “rider-friendly” packages on most of the top-100 music tours every year. “Movies didn’t have sound or color in the beginning; then they went from mono to stereo to surround. One medium pulls the other along. They eventually become reliant on each other to evolve. At the same time, it's not like the same person who’s developing holograms is also developing immersive formats for audio. That’s where the separation is.”
The company has been collaborating with visual artists and technologists to create immersive environments, using its L-ISA object-based system for live events. These have included media artist Refik Anado’s Machine Hallucination, a “mixed-reality” immersive installation at New York City’s ARTECHOUSE venue, in which 32 discrete channels/objects combine pre-recorded and live sound sources that score 150-megapixel laser-projected images across the entire interior space. Another was the dystopian production Cages, which utilized a holograph as the cast’s second lead and about which the Los Angeles Times wondered, “Is it theater? Is it film?” Sugden says the range of immersive-sound users keeps broadening and now includes music concerts and corporate events such as product introductions and, surprisingly or not, a growing number of mega-churches, which as a category are heavily invested in seasonal-religious theatrical productions.
However, Sugden cautions, while each side of the audio/video equation has its own technologists and visionaries, it will need bridge builders to make artistically and commercially coherent connections between them. “Imagine a beautiful hologram, but you can’t understand what's being said or the audio’s completely disconnected and it doesn't feel real anymore,” he says. “There needs to be synergy between the sound and the images; it can’t be one and the other for their own sake. Both holography and immersive sound technologies have to move beyond the novelty stage, technically speaking. Now they need a unifying vision and a viable business model.”
Hyper-granular LED displays and 8K projection mapping are providing the visual components for many immersive projects, but audio remains a challenge, divided between those who are fine working with appliances such as headphones and earbuds, and those who contend that there shouldn’t be any barriers between the sound source and the ear. As Spatial’s Bhargava points out, the sonic side of immersive has to choose between headphones and loudspeakers as its reproductive media. The slide-rule contingent of this question contends that headphones personalized using the head-related transfer function (HRTF) algorithm, a mathematical model that characterizes how an ear perceives a sound from a point in space, can best render the localization needed to make immersive sound authentic. Personalization of immersive audio would allow spectators to move about a holographic environment and not lose directional perspective of its associated audio.
The live-sound faction believes that not only can loudspeakers (enough of them, anyway) better create an immersive environment that can approximate the kind of realistic physical circumstances our pinna evolved to monitor, but also that it encourages a sense of community that headphones work against. “Multichannel sound systems give people a reason to go to the venue, while headphones are kind of an excuse for staying home,” observes Steve Ellison, director of spatial sound for Meyer Sound, a Berkeley, CA sound-system developer whose Constellation platform is used to create artificial ambience in production and performance venues, and whose Spacemap Go is its entry in the rapidly expanding sector of immersive live-sound design and mixing systems. “What we're trying to do is mimic what real-world sounds would do in a physical space using speakers only.”
“We don't believe that [headphones are] the right way to go,” adds Bhargav, more directly. “We want to inherently build social experiences; we want to let somebody walk into a physical space and be able to partake in that same experience what you're seeing and what you're hearing. I think the best way to do that would be through a physical space with speakers, not through headphones.” Although headphones are the way he perceives that some on the visual side of the equation look at sound: that is, as hardware rather than moving air. “We’d like to break that habit.”
The convergence of immersive sound and image might be an art show in search of a gallery. Certain types of venues, such as National Sawdust, an arts center in Brooklyn that installed 102 loudspeakers in its walls for Meyer Spacemap Go and Constellation systems, have become test cases for it. Another was Vend, which aspired to do the same in the Los Angeles area but fell victim to the vicissitudes of the city’s real-estate realties. However, its conceptual progenitor, KamranV — an amiable polymath who has produced surround versions of LPs for artists including Beck and Nine Inch Nails and co-founded Moogfest, the nerdy Woodstock for electronic music techies, says artistic venues will be crucial as developmental loci for converging immersive sound and images.
“The practice of a technology is what advances it,” he says, in a kind of instant aphorism, asserting that musicians and other artists having a space with simplified, accessible versions of media technologies would be the optimal way to accelerate those technologies’ evolution and uptake. He cites Intel’s now-shuttered volumetric-capture stage in Los Angeles as an example of a cutting-edge media technology venue that was out of reach financially and otherwise for the kinds of artists who might have best pushed its envelope.
An instance of how increased access can reduce costs and propel a technology is how the cost of accessing Dolby’s Atmos immersive-audio system for music has dropped considerably, with mixing studios now offering the service for a fraction of what the rendering process cost just a few years ago, in part because Dolby proactively brought thought-leading audio mixers into its fold and made the technology more affordable.
“Accessibility is really the key,” says KamranV. “[Audio and video] technologies tend to have their separate lanes, but once you can put it in the hands of artists, it becomes more…interesting.” That, he believes, will also make it more attractive to more productions with higher budgets, such as corporate events, with art and commerce feeding on each other. Not unlike, HEADDS, what immersive audio and video could accomplish once they really find each other. “Then,” he says, “stand back.”