Through our recent R&D work, we came across a CGI video from Eijun Kimura which showed an AR extension to an Oreo box. We wanted to challenge ourselves to recreate this in real life by using Spark AR to publish to Instagram, showing what is possible for social AR.

As social AR is growing in size and ability, many creators are beginning to develop effects for the likes of Snapchat, Facebook and Instagram. These filters can easily be accessed through the camera in their respective apps, ranging from a simple colour overlay, right the way through to interactive 3D models that are placed in the real world. As well as having a bit of fun with a “Which ______ Are You” filter or trying on some new sunglasses, the creative community has been developing some interesting outputs that have been really pushing the boundaries of the technology provided by the social media platforms.

Although it may seem simple on the surface, there are multiple hurdles that began to evolve through the project. Giving a flat surface tracker the ability to display depth both behind and in front of the original tracking marker provided some interesting workaround and tricks of perspective. Although the AR layer looks to be sinking back into the box, the use of masks offset from the front of the box are required to ensure the sides of the physical box remain visible, even when viewing from extreme angles.

Social AR comes with restrictions. Not only are you working to the policies and overall guidance set out by their social media company, the technology is not always as expansive as working directly with native Android or Apple AR technologies. One of the key limitations currently in place is the overall file size for effects: Facebook allowing up to 10MB and Instagram only 4MB or less. To further challenge ourselves, this effect was created solely for Instagram, working within that 4 MB cap, which is equivalent to less than two selfies! This means graphics and 3D models used needed to be highly compressed to meet the restriction, whilst not detracting from the visual clarity of the piece.

The glowing outline trace from Eijun’s original effect was one of the key elements that caught our eye. This isn’t something that can be natively created within Spark AR, so we had to get creative. Using reference photos of the front of the box, we used After Effects to animate 2D renderings of these traces and movements. Once complete, we had to bring these into Spark as a PNG image sequence to preserve transparency and run them as a stop frame animation. At this point, our previous file size limitation returned as we had hundreds of images to import and avoid heavy compression, we decided to use an old animation trick of halving the frame rate to reduce the amount of images in the sequence. The graphics were then played back at 15 frames per second and a reduced quality to limit the overall file size.

Furthermore, the 3D models used within the scene have to be simplified in order to comfortably display and run on social AR. The Oreo model we had was a lot larger than would be allowed, so we have to use a number of techniques to reduce the polygon count of these elements, whilst still attempting to preserve the life-like texture and quality. Although some of the high detail on the surface of the biscuit is lacking, smoothing effects allow the 3D topography to remain to be caught by lighting within the scene.

Overall, this form of AR is interesting to offer additional information and experiences to a user from simple product packaging, especially when it can be implemented directly within the social platforms they are already so familiar with. By connecting an AR engagement through a physical product, brands can drive customers to purchase the product in order to try the effect, whilst customers who already have made a purchase will be driven to that brand’s social channels to ‘unlock’ the experience. This is a really interesting space with constant developments made to the techniques and possibilities of offer to us.

https://www.instagram.com/p/CAPuvhYh_Ii/?utm_source=ig_web_copy_link

Original Inspiration Publisher:

Eijun Kimura || きむらえいじゅん || @kuro_40
Kuro40.myportfolio.com
www.instagram.com/kuro_40
twitter.com/kuro_40
www.youtube.com/user/EijunKimura