We're experimenting with new ways for you to comment and discuss stories that you read.
You can now read and write comments directly on sentences, paragraphs, images and quotes.
Highlight text and click the icon or click the icon at the end of each paragraph.
A mockup shows HoloLens being used to provide remote help with home repairs.
I’ve seen two competing visions for a future in which virtual objects are merged seamlessly with the real world. Both were impressive in part, but they also made me wonder whether augmented reality will become a successful commercial reality anytime soon.
I’m the only person I know of to have tried both Microsoft’s HoloLens and the system being developed by a secretive startup called Magic Leap.
I got a peek at what Magic Leap is building back in December (see “10 Breakthrough Technologies 2015: Magic Leap”). In that demonstration, 3-D monsters and robots looked amazingly detailed and crisp, fitting in well with the surrounding world, though they were visible only with lenses attached to bulky hardware sitting on a cart, and no release date has yet been revealed.
I had my chance to see HoloLens during a recent visit to Microsoft’s headquarters in Redmond, Washington. HoloLens is a holographic system that the company plans to pack into a visor about the size of a pair of bulbous ski goggles. In January, Microsoft said HoloLens would be available “in the Windows 10 time frame,” and the company said this week that the new operating system will be released this summer.
I experienced three HoloLens demos. The first, HoloStudio, showed the possibilities for 3-D modeling and manipulation. The second let me explore the surface of Mars with a virtually present NASA scientist. And the third gave a sense of how I might use HoloLens in combination with a Skype video chat to get help with a real-world problem (in this case, installing a light switch).
Unlike with some stereoscopic virtual-reality 3-D technologies I’ve tried, such as Oculus Rift, HoloLens did not make me feel nauseous, which bodes well. Microsoft would not say how it works, but a bit of explanation in a Wired piece makes it sound as if it may be doing something somewhat similar to Magic Leap, which is using a tiny projector to shine light at your eyes that blends in very well with the light you get from the real world around you.
The final form Microsoft’s HoloLens will take.
But I was not blown away by what I saw in Redmond. The holograms looked great in a couple of instances, such as when I peered at the underside of a rock on a reconstruction of the surface of Mars, created with data from the Curiosity rover. More often, though, images appeared distractingly transparent and not nearly as crisp as the creatures Magic Leap showed me some months before. What’s more, the relatively narrow viewing area in front of my face meant the 3-D imagery seen through HoloLens was often interrupted by glimpses of the unenhanced world on the periphery. The headset also wasn’t closed off to the world around me, so I still had my natural peripheral vision of the unenhanced room. This was okay when looking at smaller or farther-away 3-D images, like an underwater scene I was shown during my first demo, or while moving around to inspect images close-up from different angles. The illusion got screwed up, though, when it came to looking at something larger than my field of view.
Microsoft is also still working on packing everything into the HoloLens form it has promised. Unlike the untethered headset that the company demonstrated in January, the device I tried was unwieldy and unfinished: it had see-through lenses attached to a heavy mass of electronics and plastic straps, tethered to a softly whirring rectangular box (Microsoft’s holographic processing unit) that I had to wear around my neck and to a nearby computer. I was instructed to touch only a plastic strap that fit over the top of my head; demo minders placed it on me and took it off at the end of each experience.
Even this level of limited mobility was more than I got at Magic Leap, but it’s clear the HoloLens team has a big task in getting the technology to fit into its smaller, consumer-ready design.
For instance, during the Mars demo, the room around me was blanketed with realistic images of the surface of the planet, and a detailed-looking rover sat in front of me, slightly to my right. But I could only see it one rectangle at a time; if my eyes strayed beyond that rectangle in front of me, I’d see bits of the room, but no hologram.
The issues extended to the opacity of the images, too. The demos were all held in rooms that had no windows, but lighting was kept at a normal level of brightness, and the rooms were decorated with furniture, knick-knacks, and other items on and near the walls—not unlike your average living room, and the kind of environment in which you’d be likely to use a HoloLens if you bought one for yourself. Yet I could often see bits of the room peeking through the images themselves in a way that interrupted, rather than worked with, the illusion.
The most impressive part of the HoloLens demos was the use of sensors to track where I was looking and gesturing, as well as what I was saying. My gaze was effectively a mouse, accurately highlighting what I was looking at. An up-and-down motion with my index finger—dubbed an “air tap” by the HoloLens crew—functioned as the mouse click to do things like paint a fish or place a flag in a certain spot on Mars. (I screwed this up a number of times; mostly because I wasn’t holding my finger up high enough.) Simple voice commands like “copy” and “rotate” worked well, too.
Software able to read a synopsis of Lord of the Rings and answer questions about it could beef up Facebook search. Facebook is working on artificial intelligence software that can process text and then answer questions about it. The effort could eventually lead to anything from better search on Facebook itself to more accurate and useful personal assistant software.
The social network’s chief technology officer, Mike Schroepfer, introduced the software, called Memory Network, in a talk at Facebook’s F8 developer conference in San Francisco on Thursday. He demonstrated how the software could acquire knowledge from text by showing how it was fed a super-simple synopsis of the book “Lord of the Rings”, in the form of phrases including “Bilbo travelled to the cave” and “Gollum dropped the ring there.” After that, the software could answer questions that required following the flow of events in the text, such as “Where is the ring?” and “Where is Frodo now?”
Extracting information from text and figuring out how to put it together into brand-new facts is a difficult task for computers to do–as Shroepfer noted in his demo, it requires the machine to understand the relationships between objects over time.
Facebook is making this work with a new twist on a recently-popular approach to machine learning called deep learning (see “10 Breakthrough Technologies 2014: Deep Learning”). That technique involves using networks of crude “neurons” to process data. Facebook added what Schroepfer described as a “multimillion-slot memory system” to such a network, which functions essentially as a short-term memory where facts can be stored and processed.
Facebook set up a research group dedicated to deep learning in 2013 (see “Facebook Launches Advanced AI Effort”). Like similar groups at Google and elsewhere, it has largely focused on using the technique to make software able to figure out what’s going on in images. In his talk, Schroepfer also showed results from a project in which his researchers taught deep learning software to classify 487 different sports from looking at video clips. He said it is good enough to differentiate between figure skating, speed skating, artistic roller skating, and ice hockey.
That such an apparently easy task is a major achievement for software is a reminder that even deep learning software is far from very intelligent. Such software could be valuable, though. If applied to the many videos uploaded to Facebook, it might make it possible to, say, show you ads that are closely targeted to what you’re watching.
