China’s Tech Firms to Soon Blur Bounds of ‘Reality’ Using AR | That’s Shanghai

---

Imagine the Optimists’ Future

Technology progresses by leaps and bounds, radically changing everything. It frees us from our fossil fuel addiction; it even filters out the poisons and excess carbon in our air. Our livestock are spared as we grow new proteins people actually want to eat. Our lives are freed from labor, as ever-smarter robots do the heavy lifting and relieve us from our tedium. We’re free to expand our minds, internalize our technology or change our DNA to achieve impossible things. We join Elon Musk on Mars and live among the stars.

While this could all come to pass – assuming we avoid the worst devastation, natural or manmade – one realm of technology has the potential to blur our understanding of reality itself.

Augmented reality (AR), or mixed reality, is distinct from the full sensory-replacement of virtual reality (VR). In VR, our sight and hearing are completely filled with an artificial world of code, where nothing real remains. It’s exciting and potentially limitless, but most futurists consider it a niche technology, best suited for games and training exercises. With mixed reality, the world is still visible and audible, but enhanced with signs and sounds to show us more.

“It’s not just a gimmick. Today, it’s around games and marketing, but I really believe it’s a shift. We’re not locked into a small 2D screen. We’re 3D creatures in 3D space – it can be augmented in the way it’s connected, way more than just a small tag. We’re trying to get closer to what we naturally do – we grab an object, we don’t click on it.”

This eager movement to the future comes from Loïc Kobes, co-founder and CEO of the Shenzhen-based startup CoolHobo. The firm is working to create augmented stores for shops like CityShop or even Walmart. When it’s ready to launch, the app will provide info on the history and rankings for products: the IBU of a beer, the organic content of a packaged meal and much more. But they’ll also provide a 3D model of the store, visible through your mobile device, so you can name an item and a virtual character will guide you there. True to form, Kobes serves as our guide to the state of AR today and the wonders it could become.

“Tech isn’t going to stop – it’s still accelerating, with more GPU, more brain power working on it,” Kobes explains. “AI is working on it – analyzing algorithms, machine learning to classify what it sees.” And in mobile-saturated places like China, those machines can see a lot. “If 20,000 people run their cameras over a space, it’s incredibly accurate. You can start to make it very interesting in 3D.”

The whole world could be mapped that way. Imagine going for a run in the near future, wearing comfortable mixed reality glasses (or contact lenses). The sky is a brighter blue, the path looks clear and you can see and hear a rushing river where otherwise there’d be an empty drainage bed. Your device recognizes potholes before you do and they glow to help you avoid them… or, if your autopilot is on, your headset adjusts your internal equilibrium so you veer to avoid the obstacles without ever knowing it.

Illustration by Dave Alber for That’s

Configreality, also based in Shenzhen, has already developed algorithms to take the first steps toward that. Their ‘5×10’ project, commissioned by Shenzhen’s Design Society, shows the power of visual perception – shifted by VR headgear, novel space design and complicated math – to change a person’s walking path. “For now the algorithm is not in realtime scanning the physical space. So if you suddenly put a chair somewhere… the audience will have no way to avoid it,” Leo Liu of Configreality tells us. We can’t run on autopilot today because “the current headset is not providing the related function… But yes, probably in the future.”

Or imagine stepping before a mirror. With a gesture you see how you’d look in a dozen hair styles, various cosmetics or fashion statements. You can even reach out and feel them. Actimirror is giving customers some of those options right now. In your virtual reflection you could see that exclusive virtual pet that cost you thousands of points, or a tiny angel of your spiritual persuasion sitting on your shoulder with situationally-appropriate advice.

Or you could visit with friends and family, who live on the other side of the world but are sitting in the chair right before you.

Or consider work. “You won’t need people to be physically in the same room,” Kobes explains. “Give the sense of presence and there’s no need to take a flight for all those hours. It will change the way we work and perceive information. Helping humans to collaborate more because they can see the same thing at the same time.”

Dealing with anger, trauma or despair? Your device could read your pulse and the chemicals in your sweat, then manifest that stress into view while a therapist helps you through calming exercises. Real-time monitoring would let you watch that distress dissipate. Or that AR mirror could reflect back your image as a person of another gender or race, helping to break down our biases.

And of course any new technology will be conscripted to the search for love. When viewing potential mates through your device, you could see their availability, orientation, preferences, birth sign, diet and aura (of course there’ll be an app for that). Gaze into a crowd and the artificial intelligence can reveal your perfect potential match, enhanced and shining brightly as everyone else is dulled to grey.

Ahead of the Game

We’ve already seen the first dawn of this new era. Google Glass catapulted augmented reality into public awareness in 2013, though all agree that this was only a primordial version of what we’ll see. The display was tiny (a 13-degree field of view) and only offered text in the peripheral vision of one eye. But it was proof of concept and designers continued to innovate.

In the years since, startups and tech behemoths have thrown money at the problem, with varying success. After a wave of VR headsets were released to general indifference in 2016, attention turned to AR options. Microsoft released its HoloLens visor in 2016 with industrial applications – 3D modeling, architectural design – and a more consumer-friendly version is planned for 2019. Apple is rumored to release glasses of their own in 2020 and, sometime before 2025, Beijing’s nreal.ai should release “the lightest and most immersive MR smartglasses in the world,” according to Product Manager Zhao Zhihao. “Our glasses will have a 1080p high resolution and 52 [degree field of vision] and only weigh about 70 grams, twice as much as a normal pair of glasses.”

“There’s huge hype on these new technologies — people have too high expectations, too early”

The latest whizz-bang device is Magic Leap – funded by behemoths like Alibaba and Google, and promised to reach consumers by the end of summer 2018. But Magic Leap has suffered through the cycle of overinflated expectations. An earlier promo, showing a fully-rendered whale splashing into a school gymnasium, seems a far cry from the reality that people have been able to demo: a tiny monster appears on your desk to toss rocks at your head.

“There’s huge hype on these new technologies — people have too high expectations, too early,” Kobes notes. “You’ll have to be patient with the developer that the demo isn’t as impressive as they want… It’s easy to make a cool video about it – so there’s disappointment when you actually get it. It requires patience. It will happen, eventually.”

The Future is in Your Hand (for Now)

While we wait for amazing new headware, developers in China and abroad have turned to the supercomputer that’s already in the palm of our hands. Pokémon Go was 2016’s killer app for this round of mobile device-based AR… though, of course, the app was banned in the Middle Kingdom. No worries, though: a Journey to the West-tinged copycat, ‘一起来捉妖’ (Let’s Hunt Monsters), hit China’s Android app store in May 2018, courtesy of Tencent.

Alibaba made a splash partnering with Starbucks for an AR tour of the coffee chain’s Shanghai flagship. Scan your mobile device over certain areas to watch videos or scan the menu to order instantly. In its Taobao app, the promise of augmented reality shopping is postponed as the company focuses on VR stores, available when shoppers buy a simple Buy+ headset – similar to Google Cardboard – slip their mobile device inside, press it to their face, and check out a few stores’ items. But if you want to handle all the different varieties of an item, you’ll have to wait.

“If you are just creating a VR setting for one dress, it’s relatively easy. But if you need to adjust the model of the dress to fit different sizes and achieve it at scale, it’s extremely technically challenging,” Zhuang Zhuoran, senior director at Alibaba, acknowledged to Vice News in 2016. “We need to make some technological breakthroughs.”

Baidu, the third part of China’s tech triumvirate, launched an AR Lab in January 2017, but hasn’t yet realized its marketing, healthcare and education aspirations. Instead, the company’s DuSee feature displays fixed 3D images and animations on your screen. Advertisers can add markers to print advertisements, which reveal animated displays when seen through your smartphone. Or passengers riding Line 2 of the Beijing Metro can use their smartphone to view renderings of the capital’s nine city gates in their original form. It’s cool enough, but instead of a towering edifice that you could walk through and explore, the 3D image just pops up in your hand at that spot in the subway station.

Illustration by Dave Alber for That’s

Of the BAT trifecta, Tencent is investing the most in technologies that could pay off with workably fantastic AR experiences. Besides their monster game, they’re spreading their massive cash reserve as venture capital to SnapChat’s World Lenses – 3D stickers you add to the world, as seen through your phone – and ObEN, a startup making strides in turning avatars from their current cartoonish proportions into something more lifelike. ObEN’s aim is to build the online representations of real-life celebrities – and create virtual celebrities of their own, recognizable and ubiquitous as Black Mirror’s Waldo the bear. Each technology will be essential if we’re to make people and virtual objects real in any meaningful way.

Tencent also has a USD3 million stake in Innovega, a startup developing the least obtrusive headware we’ve seen. Their eMacula device combines contact lenses with normal-looking sunglasses for a lightweight device that’s ready for extended wear. The company has reported successful clinical trials and is now waiting for review by the US Food & Drug Administration, since contact lenses are involved.

More promising still is Tencent’s patent-exchange agreement with Google, announced January 2018. Though tight-lipped about what exactly is included, Tencent likely received early access to ARCore, Google’s platform that shifts AR from only recognizing specific markers to a much more dynamic system that recognizes surfaces and, eventually, everyday objects.

The marker-based platform has been the standard for the past decade but “it’s very limited, if it’s not in the database,” Kobes reflects. “What Google [with its ARCore] and Apple [with its similar ARKit] did was recognize surfaces like tables and floors. You can place your experience and walk around.” But Google also gave developers in America and Europe an advantage in its March 2018 release. “For the first six months, that upgrade was not available in Asian phones. Because it’s a US company that controls the software, they get six months’ lead on experimenting.”

Now that the advantage is starting to expire, Beijing-based Xiaomi announced the addition of ARCore to the company’s Mix 2S devices while TechCrunch reports that Shenzhen’s Huawei is in line to ink a similar deal for their devices.

“When it’s here, the adoption rate by users is huge. Hundreds of millions using it, and all that feedback,” Kobes explains. “That’s why China is so much better at the applications, using all that feedback. If you don’t need to be concerned about the user and how you’re using their data – you have to accept that the government accesses all your data and that’s the end of the story – then just get things done without having to worry.

“China is beating everyone on applications and using existing technologies. But not that deeper tech. For pure tech, other countries still have an edge. China is putting a lot of money [towards developing] this, but you can’t change a whole culture that quickly. Western scientists will push to the end, no 50-percent chabuduo.”

When Money Talks, Robots Listen

But where there’s a will (and cash), there’s a way. As part of China’s 13th Five Year Plan, governments from the local to the national level are investing in technical research – for artificial intelligence (the foundation for any workable AR system), neural networks inspired by the structure of the brain, interactive film and other fields that could help mixed reality take off.

SenseTime, with offices in Shenzhen, Hong Kong, Beijing, Shanghai, Chengdu, Hangzhou and Japan, has emerged as the world’s highest-valued AI company. With over USD1-billion in investment and huge contracts with the Chinese government, it’s working to process the firehose of data coming from the nation’s surveillance cameras, to recognize individual faces and connect them with relevant databases. Beyond facial recognition, they’re researching technology for image recognition (‘computer vision’) in hospitals and everyday settings, like self-driving cars.

This, too, is AR. Our future cars don’t need to augment reality, Kobes explains, “but they need to understand it. And that’s what’s going on behind AR.”

And nreal.ai, which sees “the latest computing platform – the smartphone – has almost reached its full potential, manifested by the declining sales figures and the slowdown of innovation,” according to Zhao. To capture the immense market opportunities of AR, they’ve built a team of “seasoned experts from Google, Microsoft, Yale University, Columbia University, Zhejiang University and Peking University, with deep expertise in software and hardware development.”

Illustration by Dave Alber for That’s

So, with all this promise and investment, why are all the current AR applications so underwhelming? What’s standing in the way?

There are still “many hurdles in optics, software algorithms, design and hardware integration [that] need to be overcome,” Zhao admits. And the Director of Strategy and Planning at Futurewei, Huawei’s US outpost, points to “very serious network challenges.” In a 2015 interview, José Roberto Alvarez explained that “Tremendous bandwidth is needed to be able to capture reality and then superimpose virtual objects on top of that reality.

“Tracking means that I need to analyze the image, perhaps, over the network, perform a computationally-intensive process, and then relay it back to this display. That is latency. That latency has to be on the order of much less than 100 milliseconds. Today, we are not approaching that. It’s one of our biggest challenges in the network.”

So even if we could get the best cameras and displays to rest like a feather on the bridge of our nose, we’re still looking at a computing challenge.

Happily for us, that’s where China is leading the way.

Faster than the Speed of Thought

Tencent Cloud currently holds the world record in four categories of computing speed, as measured by Sort Benchmark. In 2016, WeChat’s parent company smashed the previous year’s records, processing up to 60 terabytes per minute, or roughly 1,000 gigabytes a second. This was more than three times as fast as the 2015 record.

Though no tests were announced for 2017 and the results for 2018 are still pending, we can expect the internet behemoth to stay on top. In celebrating the 2016 benchmark, the general manager for Tencent’s Data Platform Department vowed to “continue to improve the backend technology for our cloud service by optimizing architecture, software and hardware to help global enterprises solve complex business challenges by leveraging hyper-scale computing platforms.”

China’s scientists have already announced the stronger, faster microprocessing chips needed to stay ahead, ready to power devices equipped for faster-than-ever 5G mobile service. Huawei researchers, along with others in China, are issuing more patents than ever for 5G-enabling technology, still just a collective 10 percent of the world’s total, but enough to start lobbying for international standards with Chinese characteristics. Within five years, 5G is projected to be fully mainstreamed, with CCS Insight predicting 1.3 billion people on these networks worldwide. Chinese users will make up more than half.

The world of 2023 isn’t far away, but the changes in store are potentially more radical than any we’ve yet seen. Remember that five years ago, China had neither Ofo nor Mobike, and WeChat Wallet was only just introduced.

If research continues at its current pace, by the mid-2020s the world will have achieved some of the crucial underlying technology to make AR living a reality. Most exciting is the combination of open-air gesture control and ultrahaptics. With the first, sensors (including tiny radar) will track the subtle motions of our hands, allowing us to swipe and pinch our way through mixed reality. The second will let us actually feel those virtual objects. With enough computing power, our devices can manipulate sound waves beyond the range of human hearing. Condense enough of those sound waves onto the same location, and we can feel it with our fingertips. No special gloves required.

By 2025, China will have access to these technologies – and probably the massive computational power of quantum computing, thanks to the USD10-billion National Laboratory for Quantum Information Sciences opening in 2020 in Hefei, Anhui province. Combine those breakthroughs with the advances from wide-reaching 13th and 14th five-year plans, along with the Made in China 2025 program that aims to shift Chinese tech onto domestic platforms from Huawei, Tencent and the like, and China stands ready to make possible all those futurist dreams.

But is there anything that should keep us up at night?

Electronic Sheep for Robot Dreams

Illustration by Dave Alber for That’s

Beyond the dangers of distraction and advertisement overload that come with on-your-face media – beautifully and horribly portrayed in Keiichi Matsuda’s Hyper-Reality – China faces its own unique possibilities.

In the country’s recent tech boom, convenience and prosperity have flourished alongside the decay of privacy and anonymity. The facial recognition technology that alllows you to unlock your phone with a glance also allows the cameras on the street to track you anywhere in the city. One of the first pure AR wearables in China is produced domestically by LLVision for use by police at a train station in Zhengzhou, Henan province, and highway checkpoints in Beijing. With a glance, authorities can match a person’s face against a database of suspects, displaying a red warning if there’s a match.

By 2020, China will also see a nationwide social credit system. Currently, the system is a variety of lists at the city, regional and national level, recording actions that are commendable or discouraged. Reporting by Foreign Policy, Business Insider and others indicate that citizens are added to the blacklists for violations like shirking debts, acting badly on a flight, smoking in non-smoking sections, jaywalking, driving badly, or “buying too many video games and posting fake news online.” Right now the consequences vary and are limited to inconveniences like throttled internet speed, or more serious restrictions regarding which trains they can ride, which jobs they can take and which schools their children can attend.

“Once untrustworthy, always restricted” is the heart of the plan. In a future where the world you experience can change with a few lines of code, that principle could breed extreme consequences.

In our augmented future, people with high social credit could simply live a better life – not just better schools and better jobs, but a world that literally looks and sounds better: full access to landscape beautification, lifelike AR interactions and all the unbridled creativity that follows from a limitless virtual world.

With a low score, access could be denied, throttled, left cartoonish and pixelated, or dimmed and gray. Fall far enough in your social credit and it’s easy to imagine a Black Mirror scenario where you’re persona non grata: instead of you, others would see a grey blob. Or worse, through their glasses your fellow citizens don’t see you at all and the device adjusts their equilibrium so they unconsciously walk around you. When AR penetrates our reality, the process of removing undesirable info from the internet can be transferred into reality. We could be unpersoned with a flick of a switch.

Of course, any vision of the future is bound to fall short. There are too many variables to be completely certain. But the course of recent history is clear: with these leaps in technology, massive adoption and unrestricted access, China’s technical wizards are at the cusp of shifting our reality in ways only our computers can imagine.

READ MORE: Virtual Reality: China Bets Billions on Troubled Tech

[Cover illustration by Dave Alber for That’s]