How hologram technology is changing the future of the music industry

“Imagine a world where you can experience iconic events like Woodstock, The Beatles’ debut at the Cavern Club or the last performance of David Bowie’s Ziggy Stardust at the Hammersmith Apollo,” says the British Phonographic Industry (BPI)’s Gennaro Castaldo.

Inside the realm of music entertainment in 2019, anything seems possible. From mind-warping visual effects to advanced 3D projections, our live-entertainment universe is expanding at a rapid rate. As Gennaro suggests, there will soon be no limit on what – and who – we can pay to see, living or dead. “It’s so exciting that artificial intelligence (AI), virtual and augmented reality, CGI and holograms all have the potential to keep enhancing and evolving the music experience.”

A star is reborn

In October, music icons Roy Orbison and Buddy Holly will be hitting the road on a joint headline jaunt visiting some of the UK’s most prestigious venues. Their Rock ’N’ Roll Dream Tour follows a series of 2018 solo ‘live’ dates from Roy, playing his hit-laden back catalogue from the sort of stages you’d associate with an artist of his magnitude and influence. There’s just one crucial difference to live outings from similar legends; Roy has been dead for more than 30 years.

These late rock ’n’ roll artists are among an ever-growing number of musical pioneers to be brought back to life, albeit digitally. Hip-hop legend Tupac Shakur was the first to take the public consciousness by storm at California’s Coachella Festival back in 2012. Michael Jackson soon followed and negotiations are in place to reincarnate soul songstress Amy Winehouse.

The upcoming double headline tour by Roy and Buddy has been put together by BASE Hologram, one of a number of global tech companies competing for dominance within this new live music field. “As a company, BASE has always been at the forefront of what’s new and exciting. To us, holograms are the natural next step in live entertainment,” says Brian Becker, chairman and CEO of BASE Entertainment and BASE Hologram.  

“Our focus is to combine the substantial and still-growing demand for live concerts, theatricals and spectacles with the accelerating interest in virtual reality (VR), augmented reality and holographic film technology.”

But despite the obvious excitement of witnessing peerless stars ‘perform’ their biggest hits live, these ventures raise new questions which go beyond mere entertainment. Is this new style of musical nostalgia potentially going to prevent emerging talent from taking centre stage? And is there something that could be construed as disrespectful in treating dead artists in this way? It’s arguably a constant reminder of the talent we’ve lost…

Pepper’s ghost

Let’s look at the technology in use here. Holograms are 3D images created by an interference of light beams and have existed in various forms for decades. While new takes on this are suddenly lighting up contemporary music stages, these kind of optical illusions have been a go-to for magicians down the ages – and one that is still harnessed by entertainers to this day.  

The ‘Pepper’s Ghost’ technique is not technically a hologram; instead, it’s a 2D visual trick jointly invented by the London-based engineer Henry Dircks and scientist John Henry Pepper. First shown in a stage production in the 1860s, it’s based on an optical illusion using the reflection of an unseen figure to give the impression they are floating on stage.

This effect was behind the most jaw-dropping moment from Coachella 2012. Out in the Californian desert at one of the festival world’s biggest events, Tupac Shakur emerged to share the stage with Snoop Dogg and Dr Dre to perform two of his legendary cuts. It was a moment that really rattled the cage of the music industry, and brought the potential of hologram technology into the minds of many.

All eyes on me

Digital Domain’s Janelle Croshaw was visual effects supervisor on the digital recreation of Tupac, working closely with collaborator Steve Preeg on the project. Both had been visual supervisors on the Hollywood hit film The Curious Case Of Benjamin Button. More recently, Janelle has busied herself working on a film directed by music star Sia as well as the latest superhero hit Captain Marvel.

“Tupac was one of those projects that terrified most people, because of the difficult nature of recreating human beings digitally. Plus, we only had six weeks in which to do it,” she reveals.    

The tight deadline proved to be one of many challenges. This was one of the first attempts to achieve such an ambitious recreation of a much-loved superstar under the glare of the world’s media. “Often times, when making a digital human, you’ll start with a scan of the actor’s face or a sculpted maquette of the person. But we didn’t have that luxury with Tupac. We used old pictures as reference and had to rely heavily on our eyes alone to achieve his likeness,” she explains.

“It’s a whole different story making the rest of the shapes to make him speak,” she continues. “It’s possible to use already-existing shape libraries from previously built characters as a base to drive the performance, but enough shapes have to match Tupac to really capture his essence. With so little reference, we were struggling to really nail the look.”

A chance encounter with a gym user in Los Angeles who bore an uncanny resemblance to Tupac in terms of facial structure and proportions offered the team a lifeline. “He was really cooperative and let us scan him and take a lot of reference photos of him doing specific Tupac expressions,” says Janelle. “From there, visual-effects company Digital Domain’s super-solid tracking, shading, lighting and computer pipelines were used for the 18,000-frame single shot. It’s a miracle we pushed the project through in the amount of time we had.”

The visual coordinators were just one part of a huge team of collaborators involved in pulling the groundbreaking project together from projections to staging and performance. Janelle continues: “AV Concepts from San Diego was in charge of the actual projection and stage setup at Coachella. Director Philip Atwell and producer Dylan Brown oversaw and guided the process. Our image sequence was then projected onto a clear Mylar foil.  

“It’s actually pretty old technology, but it had to be engineered in a way where a curtain could go down and in minutes, it could be set up and solid. There were big issues early on with it swaying in the wind and also vibrating with Dre’s bass cranked up. They remedied that situation, for the most part. The late Demfis Fyssicopulos, lighting designer, played a huge part in orchestrating it.”  

New digital domains

Regardless of the challenges, Tupac’s appearance was a huge success, having a massive global impact, amassing millions of YouTube views, Google searches and column inches. Leading visual-effects company Digital Domain oversaw the project; despite experience in the world of visual effects in Hollywood blockbusters, it was its first experience dealing with holograms.   

John Canning, executive producer, new media and experiential at Digital Domain, says: “To bring back Tupac really leveraged our skill and expertise from years in the visual-effects business creating characters such as Benjamin Button, the Beast [for Beauty And The Beast] and Thanos [for Avengers: Infinity War and Endgame]. Over the years, we have created other characters such as [late Taiwanese/Chinese singer] Teresa Teng, where we’ve continued to leverage the process and technology of our core visual-effects work in films.”

So how has technology changed in the last seven years to improve how these dearly departed stars are presented? “The increase of processing and graphic capabilities of computers has certainly had a huge influence,” explains John. “The rise of sophisticated AI systems such as machine learning has also played a significant factor.

“We take the visual data information from a camera on someone’s face, have the system learn how the face works and then apply that information to driving a character’s face. This led to Josh Brolin’s performance through the face of Thanos in Avengers: Infinity War. We are creating digital humans today with amazing lifelike fidelity.”

Back to black

Seven years after hologram Tupac’s performance, industry chatter around holograms, or at least visual illusions, is getting louder as increasing numbers of deceased performers return to virtual life. While BASE will host its Roy and Buddy tour in the autumn, negotiations are also ongoing with Amy Winehouse’s estate to bring the Back To Black star to the world once again.

With their roots hardwired into the worlds of VR and special effects, BASE is one of the leading proponents of this craze. So why did it choose to bring Roy back to reality? “Roy Orbison is a very unique and special performer. His music is best experienced in intimate settings, which made him a natural fit for this type of experience,” says Brian.

With the blessing of Roy’s estate and family, BASE worked with Eric Schaeffer, director of Broadway musical, Million Dollar Quartet, to help create the stage show. Brian is excited by the tech’s potential for inspiring younger generations of music fans.

“Roy influenced so many of the world’s musical heroes from the likes of Bob Dylan to Bruce Springsteen. So we believed it was important to introduce him to a new generation that may have heard his music, but are unaware of his legacy.”

In dreams

When this virtual Roy Orbison returns to the UK in the autumn, the tour will feature an accompaniment including a combination of pre-recorded tracks and live music from the legendary Philly Pops orchestra. These tours are created utilising cutting-edge techniques to bring these artists to the stage. While Brian is reluctant to give too much away, there is some essential groundwork BASE undertakes to ensure the visual effects are as realistic as possible.

“We start with a body double who works closely with our director to choreograph the performances. Then we take the results and go to work on them digitally, along with, in many cases, cleaned up and remastered cuts of the songs,” he states. “The technology has evolved so the team can now for the first time strip out the vocals and separate the tracks from both orchestra and other singers. From there, it’s marrying that audio with digital and laser imaging, CGI techniques and spectacular showmanship to create these projects.”

Along with this brave new world of music entertainment, there remain some slightly uncomfortable questions for the tech companies behind them. Is it morally questionable to bring back Roy Orbison without his say? And doesn’t this all feel just a little bit… creepy? Additionally, the ‘uncanny valley’ concept suggests people feel unsettled by humanoid objects which appear almost but not quite exactly like humans. So does this worry BASE?

For Brian, it’s essential their work is undertaken in the best possible taste and with the full permission and support of the artist’s estates. “We are honoured that the families of these beloved performers have entrusted us with their legacies,” he explains.“We wouldn’t want to do this without having them onboard as collaborative partners. These projects are deeply personal to them and that close relationship is invaluable to the show’s authenticity.”

The right reason

For music fans, the concept of hologram tours is divisive. There are those who react as if this were an insult to a dead legend. Other new listeners are excited by the chance to experience long-gone icons in the flesh, regardless of the form or medium.

“Typically, artist estates are behind the use of these technology-led events, and while legally and technically, hologram tours may be achievable and without issue, you have to hope they are motivated to do it for the right reasons,” says Gennaro from the BPI.

As the ability to create characters improves, artists will need to increasingly think about permissions around the use of their image after death. At the same time, as the tech behind this visual magic develops, better-quality experiences and more realistic tours should be within grasp. Gennaro says: “It means the more the public will respond to them and be prepared to genuinely buy into the idea.”

Jason Edwards, head of music at DICE, the market-leading live-music ticketing app, agrees there will be challenges for the industry to ward against cynical cash-ins from hard-up artist estates. “Ultimately, fans will vote with their feet. If the ethics are unsound, then diehard fans are going to be really put off and won’t buy tickets.”

Novelty factor

Elsewhere, naysayers have suggested the technology is a mere fad and not something capable of rivalling the thrill of new live music. But if the tech becomes powerful enough, could it be the case that in years to come the live industry is dominated increasingly by hologram tours, with less space for living artists to perform?

Controversial Chicago hip-hop star Chief Keef organised The Chief Keef And The Hologram Icons Of American Music Tour with another tech leader, Hologram USA, in a bid to bypass legal bans on him performing. Still, Jason from DICE is sceptical about the potential for this experience to enter the mainstream. “For me, the novelty factor remains. The hologram tech looks impressive, but it can’t really compare to a real artist on stage, reacting to a crowd. At the same time, younger audiences do seem more open to new forms of live entertainment. Just look at the impact of the Marshmello Fortnite shows – 10 million viewers watching an artist perform in a virtual world.”

EDM artist Marshmello certainly has music-industry talking heads staring hard into their crystal balls attempting to predict the future. His ‘appearance’ in the wildly popular video game saw user weapons disabled so they could enjoy the immersive 10-minute concert in-game. Is this where live music is heading next?

Dr Timothy Jung is founder and director of the Creative AR & VR Hub at Manchester Metropolitan University and also head of MMU AR & VR Research Group. He’s a VR expert and speaks on the subject at a range of music-industry events. He tells us that: “While there is genuine interest in this technology, it may not recreate the thrill or excitement of an actual live performance. For the time being, this may struggle to replace actual human emotions.”

Cutting edge

Still, Dr Jung is excited by what could happen next within live music. He and his team are working at the cutting edge of VR, exploring the potential for projects within entertainment, retail, training, education and more.

“The live-music industry will explore a combination of multi-sensory (eg, vision, audition, gustation, olfaction, tactician) immersive experiences using augmented reality, VR as well as holograms, which is one of mixed-reality technology, in order to touch fans’ emotions,” he states.

“It seems holograms will be one of the most powerful pieces of technology for the future of the live-music industry, but the quality of hologram and harmony between physical music performance is essential. It has to go beyond gimmick to survive.”

So with Roy and Buddy expected to touch down in the UK this autumn, proof of the lifetime of this new approach to live music will come from ticket sales. For BASE, the possibilities are endless, stretching beyond entertainment. For the music industry, live music is the most obvious field to benefit from this new technology, but it doesn’t need to stop there. The BPI’s Gennaro Castaldo believes there are implications for the recorded sector, too.

“Imagine a world where as you stream your favourite song or album, the artist or group performs it to entertain you in the form of a hologram. So streaming continues as a core format, but our experience keeps getting enriched as tech increasingly makes it possible,” he says. “It might sound rather far-fetched and part of the realm of sci-fi fantasy rather than real life. But when you consider the revolutionary changes brought about tech in just a few years, who’s to say what will be possible?”