Today, High Fidelity and doob
announce a partnership to enable the importing of doob full body scans into High Fidelity virtual reality environments as fully-rigged user avatars.
I myself have made an appointment and will post later on how my model comes out by uploading it to Sketchfab with a high likelihood.
from Pocket
via Did you enjoy this article? Then read the full version from the author’s website.
For my Machine Learning class project, I decided to look at whether or not we can refine images by using techniques for visualizing what a Convolutional Neural Network (CNN) trained on an image recognition task has learned.
from Pocket
via Did you enjoy this article? Then read the full version from the author’s website.
S.L. Benfica—Portugal’s top football team and one of the best teams in the world—makes as much money from carefully nurturing, training, and selling players as actually playing football.
Football teams have always sold and traded players, of course, but Sport Lisboa e Benfica has turned it into an art form: buying young talent; using advanced technology, data science, and training to improve their health and performance; and then selling them for tens of millions of pounds—sometimes as much as 10 or 20 times the original fee.
Let me give you a few examples. Benfica signed 17-year-old Jan Oblak in 2010 for €1.7 million; in 2014, as he blossomed into one of the best goalies in the world, Atlético Madrid picked him up for a cool €16 million. In 2007 David Luiz joined Benfica for €1.5 million; just four years later, Luiz was traded to Chelsea for €25 million and player Nemanja Matic. Then, three years after that, Matic returned to Chelsea for another €25 million. All told, S.L. Benfica raised more than £270 million (€320m) from player transfers over the last six years.
At Benfica’s Caixa Futebol Campus there are seven grass pitches, two artificial fields, an indoor test lab, and accommodation for 65 youth team members. With three top-level football teams (SL Benfica, SL Benfica B, and SL Benfica Juniors) and other youth levels below that, there are over 100 players actively training at the campus—and almost every aspect of their lives is tracked, analyzed, and improved by technology. How much they eat and sleep, how fast they run, tire, and recover, their mental health—everything is ingested into a giant data lake.
With machine learning and predictive analytics running on Microsoft Azure, combined with Benfica’s expert data scientists and the learned experience of the trainers, each player receives a personalized training regime where weaknesses are ironed out, strengths enhanced, and the chance of injury significantly reduced.
Before any kind of analysis can occur, Benfica has to gather lots and lots of data—mostly from sensors, but some data points (psychology, diet) have to be surveyed manually. Because small, low-power sensors are a relatively new area with lots of competition, there’s very little standardization to speak of: every sensor (or sensor system) uses its own wireless protocol or file format. “Hundreds of thousands” of data points are collected from a single match or training session.
Processing all of that data wouldn’t be so bad if there were just three or four different sensors, but we counted almost a dozen disparate systems—Datatrax for match day tracking, Prozone, Philips Actiware biosensors, StatSports GPS tracking, OptoGait gait analysis, Biodex physiotherapy machines, the list goes on—and each one outputs data in a different format, or has to be connected to its own proprietary base station.
Benfica uses a custom middleware layer that sanitises the output from each sensor into a single format (yes, XKCD 927 is in full force here). The sanitised data is then ingested into a giant SQL data lake hosted on the team’s own data centre. There might even be
a few Excel spreadsheets along the way, Benfica’s chief information officer Joao Copeto tells Ars—”they exist in every club,” he says with a laugh—but they are in the process of moving everything to the cloud with Dynamics 365 and Microsoft Azure.
Once everything is floating around in the data lake, maintaining the security and privacy of that data is very important. “Access to the data is segregated, to protect confidentiality,” says Copeto. “Detailed information is only available to a very restricted group of professionals.” Benfica’s data scientists, which are mostly interested in patterns in the data, only have access to anonymised player data—they can see the player’s position, but not much else.
Players have full access to their own data, which they can compare to team or position averages, to see how they’re doing in the grand scheme of things. Benfica is very careful to comply with existing EU data protection laws and is ready to embrace the even-more-stringent General Data Protection Regulation (GPDR) when it comes into force in 2018.
via Did you enjoy this article? Then read the full version from the author’s website.
Project Tango has been around for a while, from the developer tablet and demos at MWC and Google I/O to the Lenovo Phab2 Pro and the new Asus Zenfone AR. While we’ve seen it progress quite steadily in that time, we’ve never seen it look as convincing as it did at I/O this week.
from Pocket
via Did you enjoy this article? Then read the full version from the author’s website.
At Google, we have successfully applied deep learning models to many applications, from image recognition to speech recognition to machine translation. Typically, our machine learning models are painstakingly designed by a team of engineers and scientists. This process of manually designing machine learning models is difficult because the search space of all possible models can be combinatorially large — a typical 10-layer network can have ~1010 candidate networks! For this reason, the process of designing networks often takes a significant amount of time and experimentation by those with significant machine learning expertise.
 |
| Our GoogleNet architecture. Design of this network required many years of careful experimentation and refinement from initial versions of convolutional architectures. |
To make this process of designing machine learning models much more accessible, we’ve been exploring ways to automate the design of machine learning models. Among many algorithms we’ve studied, evolutionary algorithms [1] and reinforcement learning algorithms [2] have shown great promise. But in this blog post, we’ll focus on our reinforcement learning approach and the early results we’ve gotten so far.
In our approach (which we call “AutoML”), a controller neural net can propose a “child” model architecture, which can then be trained and evaluated for quality on a particular task. That feedback is then used to inform the controller how to improve its proposals for the next round. We repeat this process thousands of times — generating new architectures, testing them, and giving that feedback to the controller to learn from. Eventually, the controller learns to assign a high probability to areas of architecture space that achieve better accuracy on a held-out validation dataset, and low probability to areas of architecture space that score poorly. Here’s what the process looks like:
We’ve applied this approach to two heavily benchmarked datasets in deep learning: image recognition with CIFAR-10 and language modeling with Penn Treebank.
from Pocket
via Did you enjoy this article? Then read the full version from the author’s website.
Musings about when and how this research might manifest in human products are welcome. Tweet to me @dilan_shah.
At VRLA this past month, I had the opportunity to see first-hand how the technology gap is closing in terms of photorealistic rendering in virtual reality. Using the ODG R-9 Smartglasses, Otoy was showing a CG scene rendered using Octane Renderer that was so realistic I couldn’t tell whether or not it was real. The ORBX VR media file that results when you build a scene using Octane can be played back at 18K on the GearVR. Unity and Otoy are actively working to integrate their rendering pipeline in the Unity2017 release of the engine. And in short, with a light-field render option, you can move your head around if the device’s positional tracking allows for it.
Octane is an unbiased renderer. In computer graphics, unbiased rendering is a method of rendering that does not introduce systematic errors or distortions in the estimation of illumination. Octane became a pipeline mainly used for visualization work, everything from a tree to a building for architects, in the early 2010s. About 13 years ago, Sony Pictures Imageworks enabled VFX knowledge that is coming to VR content from Magnopus. How VR, AR, & MR Are Driving True Pipeline Convergence, Presented by Foundry
change of pace with the voxel, so named as a shortened form of “volume element”, is kind of like an atom. It represents a value on a regular grid in three-dimensional space.
This is analogous to a texel, which represents 2D image data in a bitmap (which is sometimes referred to as a pixmap). As with pixels in a bitmap, voxels themselves do not typically have their position (their coordinates) explicitly encoded along with their values. Instead, the position of a voxel is inferred based upon its position relative to other voxels (i.e., its position in the data structure that makes up a single volumetric image). In contrast to pixels and voxels, points and polygons are often explicitly represented by the coordinates of their vertices. A direct consequence of this difference is that polygons are able to efficiently represent simple 3D structures with lots of empty or homogeneously filled space, while voxels are good at representing regularly sampled spaces that are non-homogeneously filled. [1]
Within the last year, I’ve seen Google’s Tango launch in the Lenovo Phab 2 Pro (and soon the Asus ZenFone AR), Improbable’s SpatialOS Demo Live, Otoy’s ODG R-9’s at Otoy’s booth at VRLA of an incredibly realistic scene completely rendered using some form of point cloud data.
“The ability to more accurately model reality in this manner should come as no surprise, given that reality is also voxel based. The difference being that our voxels are exceedingly small, and we call them subatomic particles.”
[1] Wikipedia : Voxel
from Pocket
via Did you enjoy this article? Then read the full version from the author’s website.
At Unity Vision Summit a couple days ago, Unity announced that 360˚ video compositing will be available in “Unity2017”. Unity2017 is the next stable release of their engine, said to focus on artists and designers.
With this new feature in Unity engine, anyone can add graphics effects such as lens flares, digital animations, and interactivity in real-time to a video. The presenter Natalie Grant, a Senior Product Marketing Manager @Unity of VR/AR/Film, said one of the most important aspects about VR is that it achieves the “feeling like you are actually there.” She continued, “These are a few small ways to make a regular 360˚ video interactive and immersive”.
The purpose of this post is to explain that I posit that consumers and creators alike will learn more about computer graphics topics like (General Purpose GPU usage) and virtualization of the real-world. WebVR and 360˚ content on laptops and phones will “bring people up the immersion curve” as Mike Schroepfer says. This approach where content is composed of 360˚ video and real-time 3D model content will contribute to that.
Compositing is combining of visual elements from separate sources into single images.
As described in the talk*, 360˚ composited with real-time CG elements is essentially two spheres in a scene and 360˚ videos on the interior of each sphere with a camera at the shared center point. To explain, imagine the layers of the Earth as an analogy.
The inner core is essentially the user’s head or main camera looking around the environment. The outer core is the first 360˚ video player with a shader** applied to it masking some of the video but not all of it. Skipping the lower mantle temporarily, the upper mantle is where the second 360˚ video player is. This upper mantle is showing the same 360˚ video as the inner player but normally, without a shader. In between, the lower mantle is where users can now place digital animations, 3D objects, and UI elements that are interactable. This is where all the magic happens–specifically because the space between the two concentric 360˚ video player spheres allows for CG content to really seem in the scene. Both copies of the composited 360˚ video are exactly in alignment–meaning that as long as the user’s view position is confined to 3DoF, the user can’t tell there are two copies of this video file playing when viewing this.
Finally, for more immersion, the Crust and the rest of Space, is also a layer where any Unity object can be positioned.
Natalie shows this is important in the use case of matching a Unity directional light source to the position, direction, and intensity of the sun as captured in a 360˚ video. This means that because of Unity’s physics based rendering, the CG elements (in the lower mantle or outer crust with standard shaders) should have shadows, color, reflections and more produced in a realistic way (because they are affected by the light source). This increases the effectiveness of the illusion that footage and real-time elements are composited.
Another way to think about this approach is as the Russian nesting dolls of spheres (credit: Ann Greenberg). In this comparison, each doll corresponds to a 360˚ video sphere, and just like the dolls, the spheres are concentrically nested and aligned with the same rotation.
As demonstrated by Natalie on stage, when done deliberately enough the 3D content will actually look like it’s in the camera-captured shot. Creating the illusion that 3D objects are occluded or hidden behind the inner sphere playing video (see below a 3D dinosaur moving behind the first 360 video).
In the short-term, I think this will help people engage with more 360˚ video content and potentially excite people about mixing camera captures and virtual content.
When demonstrating “locomotion in 360˚ video”, Natalie Grant of Unity showed that one can click to move to another 360˚ video. For starters, this isn’t exactly like movement with teleportation in a completely digital environment, where one can teleport anywhere a pointer collides with a plane. Remember that the creative behind the project must capture each 360˚ video using a camera and tripod, and that’s still a constraint on the freedom of choice for location. However, with potentially a lot less work, the creative can begin making compelling 360˚ video experiences with an interactive component (i.e. switching the 360˚ video) and layers of spatially accurate CG objects.
Also at this year’s F8 developer conference Facebook announced a new camera, the Surround 360 video camera, that will let users move around inside live-action scenes. The product can infer 3D point and vector data of its surroundings using overlapping video image data from adjacent cameras. So a reasonable implication is that we may even have 6 DoF live action scenes eventually with CG elements composited***.
However, I’d imagine that blind spots would exist once a user has moved significantly from the original center of the two spheres, and that will also impact the integrity of the illusion that both CG and video are composited.
I look forward to seeing some creative applications of this method.
*found at the 35-minute mark https://www.youtube.com/watch?v=ODXMhaNIF5E
** A Unity Standard Shader attempts to light objects in a “physically-accurate” way. This technique is called Physically-Based Rendering or PBR for short. Instead of defining how an object looks in one lighting environment, one needs only to specify the properties of the object (e.g. how metal or plastic it is).
Then, the shader computes its shading based on those factors.
***The original Surround 360 was estimated to cost about $30,000 at using the company’s exact schematics.
# Virtual Virtual Reality
In Virtual Virtual Reality the setting is Activitude, where you play the role of a human trying to appease various “A.I.” characters. These characters end up being personified objects like a brick of butter, tumbleweed, or sailboat.
You will quickly be shown the button layout by the host, who calls you Bee. Your duties each take place in separate “virtual labor access points” which are really virtual realities within virtual reality and to enter and exit you will put VR headsets to your face using the Daydream Controller. Ipso facto, you will peel these headsets off to exit different virtual realities. As you begin your duties––chores that seem unmanageable, even by the most dexterous Daydream users standards––you are subjected to funny, quirky characters’ demands.
Progress will reveal that the virtual realities are holographic projections absconding a larger scheme…
Perhaps what I liked most about it was its design.
For the first time in a long while, I was given a reminder that we can create completely surreal worlds for users. Later on, I was on a train in Shanghai and looking down the train as we moved through turns the train cars reminded me of the vertebrae of a spine or something…
I thought it’s almost like being inside of a snake. This immediately became food for thought for a VR piece–climbing into a snake as a mode of transportation.
Virtual Virtual Reality is designed for longer sessions (you can spend nearly half an hour
in the app) affording you the ability to try many things and headsets on without exhausting all possibilities. It follows, of course, that the app and daydream are super comfortable, thoughtfully crafted so you don’t need to view behind you that much. Tender Claws also made fantastic use of the controller and defined a mechanic that harks back to
Portal.
Teleportation is great with the application button and uses vectors to illustrate a warping effect.
I’m left wanting to play more. The character voice-overs, spatial audio, and sense of humor are especially notable.
The following is a synthesis of a talk given by Greenlight Insight’s Alexis Macklin and Unity’s Tony Parisi along with my own experiences at Oculus Developer Day 2017. The notes on VR experiences are important as the industry grows because we can reflect on what strategies are yielding better experiences for our users.
It’s clear that Unity brings a lot of different toolsets together… namely, the gaming and cinematic arts sectors seem to be bridged by Unity
Now there’s a video player that supports 360-degree playback. Also supported is 4K video playback, ingest a 360-degree video and then augment that video for enhanced viewing.
My Snapcode
