Shakespeare in your living room: How AR can transform teaching

It’s a scene that could be observed in any drama student’s room in lockdown. An online performance of Shakespeare’s comedy A Midsummer Night’s Dream. A young actor recites the moment where Pyramus stabs himself with his own sword: “Thus I die. Thus, thus, thus. Now I am dead.”

But there’s a difference. The student watching at home holds their phone in front of the laptop screen. Suddenly, an image of the actor’s sword appears on their phone in 3D. The student is immediately engaged as the play comes to life in their room.

This performance is the culmination of a six-month teaching R&D project led by academics from Royal Holloway, University of London exploring how augmented reality (AR) could enhance online teaching and learning.

Our interdisciplinary team of four was made up of experienced and award-winning academics from drama, engineering and media arts. I was the newbie. A director of virtual reality experiences, I launched Royal Holloway’s new MA in immersive storytelling in September 2020.

THE Campus: Immersification – a fresh dimension to learning design

Even before lockdown, like many higher education institutions in the UK, Royal Holloway realised it needed to start researching and teaching this brave new world of discovery – immersive production. Reflecting the state of the nascent industry itself, immersive study in universities is scattered across many disciplines – engineering, media arts, computer science. It brings together artists, engineers, gaming developers, psychologists, filmmakers, writers, theatre producers and more.

The immersive industry is still a bit Wild West or the beginnings of television – the technology is still being discovered and the language of production defined − but in the spring of 2020, deep in the UK’s first pandemic lockdown, the switch to online teaching brought our team’s brainstorming into focus. Our AR project was funded by StoryFutures Academy’s Train the Trainer scheme, which enables higher education institutions to explore new ways of creating teaching materials and developing courses in immersive storytelling.

For drama lecturer Will Shuler, the intimacy and presence created in live performance had been removed overnight. Our challenge was to explore how an online theatre production could be more than simply a bad TV show − and how to teach this online.

Many academics were completely overwhelmed by the switch to online teaching, so we were determined to democratise the technology – any lecturer or student should be able to use our lessons, no matter their technical level. Nuno Barreiro (engineering) and David Young (media arts) led our technical development, working alongside immersive production company Mnemoscene. Early on, they worked out that we should create a website anyone could access, avoiding the hassle of downloading an app.

Fast forward three months and ARstorydecks.com was born, enabling educators to use AR to enhance their online teaching. The website includes a set of 10 “image cards” and a corresponding tool trained to detect them as AR markers. Markers are visual cues, a bit like QR codes, that trigger the virtual information, in this case 3D images. These cards (storydecks) are created with free 3D assets, using open-source archive Google Poly. The lecturer can assign specific 3D assets to each individual card, particular to their lecture’s learning objectives. The cards can then be used by a lecturer/performer in a video conference, with students able to see the AR assets by accessing the tool and pointing their smartphone camera at their computer screen. 

ARstorydecks.com is far from perfect. We had issues with some mobile brands not recognising the markers, and sometimes the 3D asset just doesn’t appear when you scan it. The process is still pretty “clunky” – holding up cards while trying to perform Shakespeare can detract from the performance. We are also still experimenting with the distance between the marker and the 3D assets, so that the marker isn’t a distraction.

THE Campus: Remote vs virtual labs

But as a proof of concept it’s a real step forward for AR and teaching. We achieved our main goal – creating an open-source toolkit for educators to use AR in online teaching.

The development process was a huge learning curve for the team. Using the decks in a lesson and seeing the students engaging with this new way of learning was enormously rewarding. A real highlight for the team was the interaction between a group of lecturers whose paths might not normally cross. It was brilliant trying to get inside the brain of an engineer and seeing how the technical development process works. For them, it was fascinating to see how far they could push the coding and technology to create successful arts classes.

ARstorydecks.com will launch on 29 January, but that’s not the end of the project. The team is keen to continue developing the quality of the website for students and teachers. Our next challenge is to secure further funding to smooth out the website’s user experience and to explore alternative 3D asset sources (Google Poly is set to close in June 2021).

The UK has the largest immersive industry in Europe, with market growth expected to rise from £118 million to £294 million by 2023. We’ve seen an upsurge in immersive production during the lockdown, with music festivals, corporate conferences and global events all being held online. Everyone is looking for innovative techniques to engage audiences, and many are turning to augmented or virtual reality.

Academia is no exception. Immersive production relates to several disciplines, from game designers and engineers building the technology to artists and filmmakers learning to create content. But it goes beyond these fields and it won’t be long before we’re all looking to immersive technology to further students’ learning.

In 2021, mobile phones are so integral to all our lives, particularly young people. So why not exploit this and bring 3D Shakespeare into their rooms?

Mary Matheson is an award-winning immersive director, creator and executive producer. She is a teaching fellow and course director of the Royal Holloway, University of London MA in immersive storytelling.