Facebook's future looks a bit like Pokémon Go

Someday you’ll likely be able to slip on a pair of special glasses and see realistic digital objects — from cartoon monsters to your friends’ faces — superimposed on the real world.

But that vision may be years away from coming to fruition, and Facebook CEO Mark Zuckerberg doesn’t want to wait. So on Tuesday, he announced that Facebook is taking a shortcut: creating software that gives ordinary smartphones sophisticated augmented-reality capabilities. Zuckerberg envisions users holding their phones in front of their faces to see virtual objects superimposed on the physical world.

Of course, Facebook isn’t the first company to have this idea. Pokémon Go pioneered a primitive form of augmented reality last year, and Snapchat has lenses that superimpose dog ears or other virtual objects on your face as you take a selfie. But Facebook sees those as two specific applications of what could become a powerful new computing platform — one that thousands of third-party developers can use to build augmented-reality apps of their own.

Facebook is particularly well positioned to bring this vision to life. It has already added a Snapchat-like camera feature to the Facebook mobile app complete with Snapchat-style filters, providing a foundation on which the company can build further camera-based functionality in the future. At the same time, Facebook owns Oculus, a company building one of the leading virtual reality headsets. Those headsets are still too expensive and cumbersome for mainstream use. But by using the smartphone as a kind of proto-AR platform, Facebook hopes it will be well prepared when augmented reality finally becomes a mainstream technology some time in the 2020s.

Augmented reality can be a lot better than Pokémon Go

Photo by Michael Loccisano/Getty Images

When Nintendo released Pokémon Go last summer, there was a lot of hype about it as the first augmented reality app to reach a wide audience. When you pointed your smartphone in the direction of a virtual Pokémon, the app would show the creature superimposed on your real-world surroundings. If you were seated at a desk, the Pokémon would appear to be bouncing on top of the desk.

But this was the most primitive possible version of augmented reality, which became clear if you walked around the room. The Pokémon would appear to float awkwardly above your surroundings — not fixed to any particular place in the room. If you went over by the window, the Pokémon might suddenly appear to be floating in space outside the building.

But much better augmented reality is possible. Better augmented-reality platforms construct detailed 3D maps of the world, allowing virtual objects to interact with physical objects in realistic ways. That allows objects to appear fixed in one spot even as the user walks around the world. A virtual object on a table appears to stay in the same spot on the table as you walk around it. If a virtual object goes “behind” a real object, it becomes obscured by the closer object just like a real one would.

For several years, Microsoft has been working on an augmented-reality headset called HoloLens that has these capabilities. Facebook is trying to bring those same sophisticated capabilities to ordinary smartphones, which have less processing power and less sophisticated sensors than high-end augmented headsets.

Not only that, but Facebook is also working on the capability for its app to identify individual objects in a scene. Zuckerberg showed a smartphone pointed at a table that had a coffee cup, a wine bottle, and a potted plant on it. Tap on the plant and the app recognizes it as a plant and offers plant-specific options, like having flowers sprout out of the pot or having a virtual raincloud coalesce over the plant. Tap on the coffee cup and you can have virtual steam float out of it. (Snapchat just announced a new feature called World Lenses that does something very similar.)

These are whimsical applications, but the technology could lead to compelling real-world capabilities. An obvious area is games. With the ability to recognize common objects like couches, tables, and windows, it could become easy for developers to build games that feature bad guys climbing in through the windows and hiding behind furniture.

Facebook believes these platforms will usher in an era of virtual art. Zuckerberg showed off a virtual art installation in which virtual paint gushed from the ceiling of a hallway in Facebook headquarters. He said that Facebook employees had begun wandering around the company’s offices holding their smartphones in front of their faces so they could look at virtual objects around them.

A big question, of course, is whether users actually want to do any of this stuff. It’s possible that Facebook will put a lot of work into this technology and it’ll fall flat with consumers. The experience will be more than a little awkward as long as people have to consume augmented reality by holding their phones in front of them.

But Facebook is betting that some users will be interested in the technology despite the awkwardness of doing it with a smartphone. And eventually, someone will invent lightweight augmented-reality glasses that let people see virtual objects while they keep their hands free. Facebook’s Oculus subsidiary builds virtual reality headsets and might eventually create glasses like that.

But Zuckerberg believes it would be a mistake for Facebook to wait for this kind of hardware to become available. Facebook is giving itself a head start by building phone-based AR software now. Once affordable AR glasses become available, Zuckerberg believes, it will be easy enough to adapt the phone-based software to the new platform.

Zuckerberg emphasized that this was a work in progress, not a completed project. Right now Facebook is focusing on building the basic software that allows a smartphone to understand the world around it and insert virtual objects into physical space — and it’s opening up these capabilities to third parties so they can start playing around with them. But it will be a while yet before Facebook offers a finished app that takes full advantage of these capabilities.

Facebook’s larger vision is social virtual reality

When Facebook bought Oculus back in 2014, a lot of people wondered why a social media company was buying a maker of virtual reality headsets. On Tuesday, Facebook announced that it was releasing a beta version of an Oculus app called Facebook Spaces that provides a clearer idea of Facebook’s vision for virtual reality.

The Verge / Facebook

Oculus users wear a headset and hold a small controller in each hand. In Spaces, users are represented by avatars whose motions in virtual space mirror their users’ real-world motions. While in virtual space, you can talk, doodle, share photos, and watch videos together on a shared virtual screen.

People can also be transported to a variety of virtual environments. In one demo, people were transported to an apartment one of them had just rented so the others could look around. Users can even hold out a virtual “selfie stick” to take a photo of their virtual selves in their virtual environment.

If you’ve never tried virtual reality (and I haven’t) it probably sounds goofy and kind of pointless. But people who have tried it say it starts to feel natural surprisingly quickly. Even though avatars have cartoon faces and bodies, the realistic way their bodies move and interact creates a surprisingly realistic vibe.

The beta version of the software is being released today for Oculus users. But right now, the Oculus isn’t yet a mainstream technology — the headset costs hundreds of dollars and requires a PC that is likely to cost more than $1,000. But we can expect the technology to get steadily more affordable in the coming years.

And over time, Facebook will look for ways to cross-pollinate ideas between its high-end Oculus headset and its low-end smartphone app. Because eventually these won’t be two separate products. Augmented-reality glasses will be cheap enough that everyone can afford them, and they’ll be able to transport your friends either to fully immersive virtual environments or to your living room.

Will you support Vox’s explanatory journalism?

Most news outlets make their money through advertising or subscriptions. But when it comes to what we’re trying to do at Vox, there are a couple reasons that we can't rely only on ads and subscriptions to keep the lights on.

First, advertising dollars go up and down with the economy. We often only know a few months out what our advertising revenue will be, which makes it hard to plan ahead.

Second, we’re not in the subscriptions business. Vox is here to help everyone understand the complex issues shaping the world — not just the people who can afford to pay for a subscription. We believe that’s an important part of building a more equal society. We can’t do that if we have a paywall.

That’s why we also turn to you, our readers, to help us keep Vox free. If you also believe that everyone deserves access to trusted high-quality information, will you make a gift to Vox today?

One-Time Monthly Annual

$5/month

$10/month

$25/month

$50/month

Other

$

Yes, I'll give $5/month

Yes, I'll give $5/month

We accept credit card, Apple Pay, and Google Pay. You can also contribute via