Play Along: Virtual Orchestra

The challenge

Many schoolchildren grow up with no access to real orchestral instruments and no touch point with classical music. The Philharmonia wanted an experience that could travel into those classrooms – inspiring and educating children who might never otherwise sit inside a symphony orchestra. It needed to be genuinely fun to pick up, teach real musical ideas along the way, and sound authentic to musicians whose ears are trained to hear timing down to fractions of a millisecond. And it had to run not only on the latest headsets, but on the more modest hardware that schools can realistically deploy at scale.

The solution

The project began with Meta Camera capturing a live performance of the William Tell Overture using the Meta Three and Meta Four cameras. Some of those camera positions were placed exactly where the players would sit – so in the finished experience you take that musician’s seat, see what they see, watch the audience and follow the conductor from the heart of the orchestra.

Working with the Philharmonia, we chose three instruments that are accessible, engaging and instantly rewarding to play: the triangle, the timpani and the hand cymbals. Visualise then built the full experience in Unity for Meta Quest, from the interaction design through to a bespoke physics and audio system.

Inside the experience

You are welcomed by Philharmonia patron Love Ssega, who introduces classical music for first-timers and sets the tone with a highlight reel of the orchestra’s other 360 films. Pick an instrument from the menu and a spoken fun fact explains why it is worth playing – the first of many educational touches woven through the journey.

Next comes a tutorial filmed in 360 with Philharmonia percussionist Tom, who demonstrates each instrument in front of you. From there you step into a fully 3D rehearsal room we modelled and built in Unity – a six-degrees-of-freedom space with warm mood lighting, Philharmonia posters on the walls, and your instrument waiting in front of you, modelled from photo references so it looks and feels real.

Learning to play, Guitar Hero style

Inspired by rhythm games, notes scroll along a musical stave and fly toward you; the goal is to strike your instrument in time as each note reaches the action bar. Tom guides you through three tutorial stages of rising difficulty, introducing real musical notation along the way – what the symbols mean and where they sit on the stave. Once you have passed the skill checks, you play the full William Tell Overture, either in a simplified arrangement or the true professional score the Philharmonia’s own players perform.

Real players, real sound

The detail the Philharmonia cared about most is also what sets this apart. Rather than computer-generated MIDI sounds – as most play-along apps use – every note you hear is a real audio sample recorded from Philharmonia players and their instruments. In a dedicated recording session we captured each instrument struck in different places and at different dynamics, from the quietest touch to a full-force hit, including the ‘wrong’ sounds. So the harder you hit the timpani the louder it rings, the sweet spot sings while the edge dings, and the triangle and cymbals respond to exactly where and how you play them.

The hardest part: physics and sync

Making the instruments feel right meant building a bespoke physics system: a drumstick that bounces back naturally, a triangle that swings because it hangs loose on its stand, and cymbals given a convincing sense of weight despite being held with hand controllers. On top of that sat a detection system that works out exactly where you struck the instrument and triggers the right sample.

The biggest engineering effort was synchronisation. Professional musicians can hear timing errors down to hundredths of a millisecond, so we drove the latency between hitting a virtual instrument and hearing it – and scoring it – as low as possible. Because the underlying recording is a live performance led by a conductor, its tempo naturally ebbs and flows, so notes could not simply be copied along a grid. Every note on the timeline was placed by hand, refined in close collaboration with the Philharmonia’s musicians until it was exactly right.

Built for the classroom

A scoring system rates how accurately you hit each note, and a branching feedback tree responds to both your score and the difficulty you chose – gently sending beginners back to the tutorial, and reserving its highest praise for a strong run on the hardest setting. Throughout the build we consulted the Philharmonia’s education team and ran repeated focus groups with musical and non-musical staff, tuning the experience until it felt right for schools. Crucially, it runs on Meta Quest 2 as well as Quest 3, so the same physics and learning work on the headsets schools actually own.

The result

Play Along: Virtual Orchestra launched publicly at the Southbank Centre and is now being delivered into schools, with whole classes picking up a virtual instrument and playing alongside the Philharmonia. No musical experience required, and whether you are eight or eighty, it offers a rare chance to sit inside a world-class orchestra – and maybe discover a hidden talent in someone you least expect.

The Results

The experience clocked up over 2,000 plays in its opening week, with an average play time of over five minutes per session – a long stretch by any measure for a public installation.

Credits:

Production led by Meta Camera with the Visualise team. VR application development, ideation and delivery led by Visualise, delivered to Virtual Umbrella for schools. Client: Philharmonia Orchestra.