SFU Film Studio Employs AI to Advance Film Production
As technology continues to change the way art is created, SFU computing science professor Yağız Aksoy is using AI to improve film production for artists and editors.
Aksoy will be using his computational photography expertise to help artists and film producers in British Columbia. To do so, he will be leading the advancement of a new film studio at SFU that will be used for both film production and research development.
Aksoy, who leads the Computational Photography Lab at SFU, specialises in computer graphics and computer vision. He and his team research how artificial intelligence (AI) can be used to assist artists in the process of realistic image editing and film production.
With funding of $200,000 from Canada Foundation for Innovation’s (CFI) John R. Evans Leaders Fund (JELF), Aksoy and his team will be purchasing equipment to build a fully functional film studio at SFU, expected to be ready in 2022. They will use this studio to film educational videos that focus on their research.
This will help Aksoy and his team accomplish multiple goals. For one, this practice will allow the researchers understand some of the challenges that artists face while filming and editing. Secondly, these videos will be used to teach artists what can be made possible through AI research in this area. Last but not least, the researchers will be able to build data sets from these videos that they will use to train AI to perform various computer vision tasks relevant to film production such as monocular depth estimation. This is defined as the process of estimating scene depth from a single image.
“With the hardware we are getting from the CFI JELF grant, we will be able to generate better and higher quality training data which will have a large impact on the applicability of our machine learning models,” says Aksoy.
Recently, Aksoy and his lab published a research paper titled Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging in which they proposed a method for how to create high-resolution depth estimations from a single image. Their method extracts the best qualities from low-resolution and high-resolution networks to produce a depth estimation that is both high-detailed and without inconsistent structure. With the development of the new film studio, the researchers will be able to expand upon this research with new and improved data.
“We model the statistical distribution in our neural networks based on the data set that we have,” says Aksoy.
“That is why the data set itself is very important and why we are investing in data set capture technology.”
The ability to estimate scene depth is crucial for AI to be able to recognize and understand scenes, a longstanding challenge for computing vision researchers. Aksoy plans to harness this ability to help artists in BC with photo and film editing.
In order to develop tools that are truly impactful for the end-user, Aksoy wants the artist community involved in the research process. His previous experience at Disney Research during his PhD taught him the importance of receiving feedback from artists early and often.
For this reason, he has built connections with the SFU School of Interactive Arts and Technology, the Vancouver Film School, local start-ups working with photography and film professionals, and more. As much as Aksoy and his team will be educating the artist community on what can be made possible through their research, they will learn from the artists on how this research can best serve the community.
“When the community is not involved in research, it is hard to conduct research that will serve and be accepted by the community,” says Aksoy.
He plans on eventually developing AI recommendation systems for editing during post-production in film through which film editors will be more easily able to arrange scenes based on their artistic discretion.
Long term, he hopes that this research can also be used to develop tools to help artists create in new areas such as augmented reality.
“I dream of seeing artists create things that I can’t even imagine using the tools that we develop,” says Aksoy.
For now, Aksoy and his team will be focused on getting their new film studio ready.
Simon Fraser University’s School of Computing Science is mobilizing brilliant minds to create business and societal innovation for good. For more information, visit sfu.ca/csresearch
