The Future of Augmented Reality Glasses

Meta

  • Five years ago, plans were announced to create AR glasses to blend the digital and physical worlds.
  • Orion, the latest AR glasses from Meta, aims to enhance presence, connectivity, and empowerment.
  • AR glasses offer unrestricted digital experiences with large holographic displays.
  • Contextual AI integration allows for proactive addressing of user needs.
  • Orion stands out for its lightweight design, indoor/outdoor versatility, and emphasis on interpersonal interactions.
  • Orion represents a significant advancement in AR glasses technology, combining wearability with advanced features.
  • The groundbreaking AR display in Orion offers immersive experiences and a wide field of view.
  • Orion’s unique design maintains a glasses-like appearance, allowing users to see others’ expressions.
  • Orion’s capabilities include smart assistant integration, hands-free communication, and immersive social experiences.
  • While not yet available to consumers, Orion serves as a polished product prototype for future AR glasses development.

OP: Introducing Orion, [Meta’s] First True Augmented Reality Glasses

Snap

Sophia Dominguez, the Director of AR Platform at Snap, discussed Snap Spectacles and the company’s AR initiatives at the Snap Lens Fest original post here.

  • Snap Spectacles can be connected to a battery pack for extended use beyond the standard 45 minutes, with a focus on B2B interactions that directly engage consumers.
  • There is a push for Snap to collaborate with various businesses, such as those in location-based entertainment or museums, to expand the Snap Spectacles ecosystem.
  • Sophia Dominguez has been involved in AR for over a decade, starting with Google Glass, and now oversees developers and partners creating lenses on Snapchat.
  • Snap’s approach to AR emphasizes personal self-expression as a catalyst for AR lenses, transitioning to world-facing AR lenses like those in Snap Spectacles.
  • Snap’s long-term vision is to make AR ubiquitous and profitable for developers, aiming to integrate digital objects seamlessly into the real world.
  • Snap’s focus on consumer-level AR use cases includes self-expression as a core feature, offering a variety of options for users to engage with AR content.
  • Snap’s AR platform also caters to enterprise and B2B applications, collaborating with stadiums, museums, and other businesses for unique AR experiences beyond consumer-facing lenses.
  • Snap’s technology, like Snapchat cam, is designed for venues to integrate into large screens or jumbotrons, focusing on consumer desires for virality and joy rather than just enterprise solutions.
  • The company aims to increase ubiquity by making lenses fun and approachable, partnering with entities like the Louvre to explore augmented reality possibilities in a consumer-friendly manner.
    HTC Vive has delved into location-based entertainment more than Meta, and Snap is prioritizing connected experiences, ensuring fast connectivity and optimizing for various use cases like museum activations.
  • Snap collaborates closely with developers, offering grants and support without strings attached to foster innovation in the augmented reality space, aiming to be the most developer-friendly platform globally.
  • Snap’s Spectacles have evolved over the years, from simple camera glasses to AR display developer kits, with the latest fifth generation focusing on wearability, developer excitement, and paving the way for consumer adoption.
  • The company has revamped Lens Studio to encompass mobile and Spectacles lenses, emphasizing ease of use and spatial experiences, aiming to create a seamless ecosystem for developers across different platforms.
  • Snap values feedback and collaboration with developers, striving to provide pathways for monetization and support for creators building on both mobile and Spectacles platforms.
  • Snap’s Spectacles offer a unique immersive experience, leveraging standalone capabilities and spatial interactions, aiming to enable emergent social dynamics and experiences not possible on other devices.
  • Developers are considering the length of time users spend on devices like Zoom calls or workouts, with a focus on creating a seamless experience for users on the go.
  • The new SnapOS manages a dual processing architecture for Spectacles, with Lens Studio being the primary pipeline for developers to create content for the device.
  • Snap is actively listening to developer feedback and working on enabling WebXR on Spectacles to support a variety of use cases and experiences.
  • The operating system for Spectacles includes features like connected lenses, hands and voice-based UI, and social elements out of the box to facilitate easier development.
  • The ultimate potential of spatial computing is envisioned as a way to break free from the limitations of screens, allowing for more natural interactions and connections in the real world.
  • Snap aims to empower developers to explore the possibilities of augmented reality and spatial computing, emphasizing ease of use and continuous improvement based on user feedback.

Snap Lens Studio

Hello World Building Augmented Reality for Snapchat

Fun fact – 30000+ lenses created by Snapchatters, leading to over a billion views of lense content
Table of Contents
Lens Studio
Hello World! Lens Studio Live Demo
High-Quality Rendering with Allegorithmic
Chester Fetch with Klei Entertainment
Cuphead and Mugman with Studio MDHR

Travis Chen

  • Worked at Bad Robot, Neversoft, and Blizzard

Lens Studio

  • Snapchat has always opened up to the camera, which has positively affected their engagement
  • Pair your phone with Lens Studio
  • A community forum on the site exists where devs q & a
  • Has been out for less than 4 months today, and the lenses have resulted in over a billion experiences
  • The tool has been used for a variety of things hamburger photogrammetry, full-screen 2d experiences,
    r/snaplenses
  • Distributing your lenses is really easy

  • Within, snap you can discover a lense where you can see more lenses by the same creators or you
    can pull up on the base of a story to figure out what lense was used.

Lense Boost – All users see the Snapchat carousel, a Lens Boost to get your lense into this carousel

Find which template best fits your creative intent

Templates

  • Static object
  • Animated object
  • Interactive templates (tap, approach, look at)
  • Immersive (look around, window)
  • For 2D creators (cutout, picture frame, fullscreen, soundboard)
  • Interactive path (idle, walk, and arrival states necessary) coming soon

Examples

  • Brian Garcia, Neon Book
  • Pinot, 2D textures, cutout template, then character animator to animate
  • DFace, DDog, imported into lenstudio (from camera reflections feature)
  • Jordan & Snapchat, ‘88 static Jordan 3D model
  • Netflix & Snapchat, Stranger Things – turning on the TV, or spelling your name out,
    awakiening the demi gorgon

Hello World

Lens Studio is made up of panels:

  • Live Preview, to see what it will be like, it includes tracked content and interaction support
  • Objects panel, like the Unity scene view, it shows you what is in the preview
  • Resources panel, all your resources and where you’d import stuff

Workflow

Start with the animated object template
Select an object in the resources panel and move it to the objects panel
Google blocks + mixamo + export free animations from Adobe and import your character animated from
mixamo
File import monitor and astronaut
Child the imported 3D model to the fox as a child and delete the fox
Add a shadow
Sprite

High-Quality Rendering

Substance Painter is an app to apply materials or paint textures for 2D or 3D.
Any material you bring in you can apply to an object, they apply uniformly, but there’s
also, smart materials which applies intelligently to geometry (rust example).

The layers tab is like the scene view the place to drag and drop

Alphas provides a cutout, you can apply materials to the cutout

Upon clicking export

lensstudio
Challenge: Rubber Ducky

Chester Fetch with Klei Entertainment

Games studio since 2005

Why is AR interesting for Klei?

  • AR is about bringing the virtual world out to the player.
  • Shareable
  • Limited bandwidth
  • Seems hard
  • Would require too much time from others at the Studio

Cuphead and Mugman with Studio MDHR

  • Cuphead and Mugman wanted to build and snap a boss battle
  • All of the lenses used in cuphead were from assets created directly from the game
  • Chains together 5 2D animations

 

Questions


  • Within the Snap app, I noticed you can rent/create a lense “as a service” how does this pertain to lens studio?
  • A question I had was, looking forward to a day when you can use targets like people for further interactable and shareable content like the examples shown in Mugman, when will person/object recognition be available to developers and users of Snap?   
  • What is the github account for Snap?