Annotator: Anchoring AR Experiences In Bloomingdales
Company
Client
Date & Length
Role
Tools
Framework
About
Bloomingdales customer base is primarily gen X customers (older folks). If that doesn't change in couple years, we will no longer have Bloomingdales.
Problem
How do we make Bloomingdales feel accessible and relevant to Gen Z kids?
Solution
In an effort to attract a younger customer base, we created away for Bloomingdales shoppers find deals and host snapchat-like store experiences at exact points in their departments using augmented reality called Holopop (now HoloGrid).
But, In order for these experiences to exist, Bloomingdales stores needed a tool that allows them to anchor holograms to exact points in their stores, such as a coach handbag. To do this, we made Annotator, an iOS app that uses computer vision and outside-in point clouds to recognize points in their stores.
Discovery
We started with a stakeholder workshop called Post Up. We then affinity mapped the problems into trends.
Then as team, we mapped out the Risk Using a Risk/Reward Matrix.
From the workshop, we came up with the following Epic. As an associate:
- I need to be able to log in to the Bloomingdales account so I can create a computer vision model for my store/microlocations.
- I need to be able to record a video and mark the points I want to annotate.
- My recordings needs to train the model in the background.
- I need to be able to view places, microlocations, and data added in WorldRegistry.
- I need to be able to attach my video to a places microlocation, and some data.
- I need to be able to Edit and view old recording information.
Ideation
We started out with sketching and a paper prototype, showing how the user would mark the object in the video recording to train the AI model.
Following the ContextGrid Branding
Using our style guide, I created the visual designs to match our web app.
Prototyping
Using Adobe XD
We created a lo-fi prototype of annotator and tested it with Bloomingdales employees, which underwent 3 rounds.
Testing
Improving how users create anchors
After we had a version to test, we did ADHOC testing in the Bloomingdales stores with the associates. What we found was that the modal creating part of taking a video and marking it up, took too long. Store Associates have other tasks at hand, and they need to be able to create an anchor in under 1 minute to make it worth their time.
The other problem was there was a lack of feedback in regards to the success of creating an anchor. This was because it took 6 hours for the system to validate the creation of the anchor using machine learning with our system.
In short, we had an inefficient, non scalable solution.
ARKit2 Beta: Apple’s Point Cloud System
Luckily, Apple’s ARKit 2 Beta had an out-the-box solution for creating outside-in point clouds (this allowed us to also create an anchor for a single object). The creation process which gamified for the user, quick, and it creates the modal instantly. We made a test app and got feedback from the store associates and they found it to be a massively more productive and enjoyable experiencel. We quickly replaced our computer vision/machine learning tech with ARKit2’s Process.
Final Product
Results
The app was proved usable, easy to navigate, and solved Bloomingdale’s problem of not being able to annotate a store with data in quickly, at scale. Unfortunately, due to a change in leadership, the project was discontinued with talks of reconsidering the pilot stores in the Bay Area of California Q2 2019. More info coming soon!
Demo
View on the Appstore
