How a Stealth Startup Hopes to Change Video Calls and VR

Video calls became part of daily life during the pandemic and, while the tech has improved in recent years, they’re still far from ideal. ViewMagic, a Bay Area startup that’s been operating in stealth since 2014, has been focusing intensely on overcoming one shortcoming in particular: the difficulty that people have in making eye contact with one another on a video call.

Now there are signs it is getting somewhere. ViewMagic has developed hardware and software that lets video callers look each other in the eye, through their screens. An offshoot of the same tech allows for 3D video, live or prerecorded, to be viewed in virtual reality. Rolf Herken, ViewMagic’s CEO, told me that he hopes to license the tech to larger companies for use in their video calling services. That could lead to ViewMagic getting acquired at some point down the road. The startup has raised about $10.2 million to date. 

I was the first reporter to try out ViewMagic’s tech, using it to talk to ViewMagic’s CTO James McCombe. Both the video call demo and a live streaming demo in VR were impressive—and the fact that they worked using my ordinary home internet connection, running on hardware that will be outdated in a few years time, came as a surprise.

A Virtual Window Into ViewMagic’s Office

When I looked at McCombe on the display, it was indeed as if he was looking directly at me through the screen. This video ViewMagic provided demonstrates how video calls look using its technology.

While I was using an iPhone 12, McCombe was using a prototype device: a tablet with four cameras mounted next to the screen. Instead of using depth sensors, ViewMagic’s algorithms construct a 3D light field from the four camera feeds. ViewMagic says there are no tricks being done to adjust the position of a caller’s eyes, as is the case with recent AI-powered gaze correction tools.

To show off the VR potential, McCombe spoke to me while I used an Oculus Rift S headset, which connected to a laptop with an external graphics card. In VR, I was surrounded by a room made to look like the holodeck from Star Trek. Inside that room, right in front of me, was McCombe behind a desk. A sliver of ViewMagic’s office was being projected into VR (McCombe narrates this video demonstrating the system).

As McCombe and I talked, I could move my head around a few feet in any direction inside my view of the office scene, this time captured with a PC using an eight camera array. There was some video artifacting (similar to the fuzziness or shimmer when watching YouTube on a poor internet connection) but nothing overly distracting. This branch of the tech, which the company calls ViewMagic 4D, is being developed for live event broadcasts and pre-recorded VR films.

Compelling as it is, ViewMagic’s software isn’t quite ready for consumer use. The iPhone I used for the video call was noticeably warm after a few minutes on the call, but that could be chalked up to the app not being optimized yet. As for the VR experience, it’s reasonably difficult to render on the viewer’s side. McCombe said he thinks it could conceivably run on successors to current standalone VR headsets like the Oculus Quest 2.

What Comes Next

At my request, ViewMagic connected me to a few outsiders who saw the same demos (Herken said that several leading companies have now seen ViewMagic’s tech, but can’t name names). 

Avi Bar-Zeev, an AR/VR designer and engineer whose experience includes stints at Microsoft, Apple and Disney, spoke positively of the demos. “There’s a lot of great potential in the approach they’re taking,” Bar-Zeev said of ViewMagic’s video calling solution in particular.

Gary Yost, known for leading the creation of Autodesk’s 3ds Max graphics software, said he sees promise in the tech for his new creative pursuit, VR storytelling. “As a documentary filmmaker, putting someone in a soundstage and doing a 3D capture of them is just an incredibly artificial way to present context,” said Yost. By contrast, ViewMagic’s prototype camera array for VR is already portable. “And it can provide streaming, live 6 degrees of freedom content,” Yost added.

Meanwhile, others are working on similar 3D streaming solutions that require more sophisticated hardware than ViewMagic’s prototypes. Google’s Project Starline call booths appear to adopt a similar approach (albeit using more sensors and massive 3D displays) and premium headset maker Varjo recently announced plans for a cloud-based platform that will use a LIDAR-equipped headset to stream people and their surroundings.

Facebook and Microsoft also see realistic AR/VR presence—the ability to see someone not as a stylized avatar, but as they actually appear in real life—as a goal worth pursuing. Mark Zuckerberg likened it to a “holy grail.” Whether or not this kind of tech really does end up shaping the future of digital communication, the demos I took showed that ViewMagic is perhaps closer to making usable versions of it than anyone else. 

Other news:

• Niantic’s Pokemon Go has brought in $5 billion in lifetime revenue according to data from Sensor Tower. The AR game turned five years old earlier this week.

• Osso VR, a platform for VR surgery training, raised a $27 million Series B round led by GSR Ventures. The company recently hired its 100th employee and its training content is used in residency programs at Johns Hopkins and Brown University.

• WebAR platform 8th Wall is launching a partner program. Similar to efforts from Facebook and Snap, the network is intended to help brands find reputable studios and creators that can create browser-based AR artwork, games and applications.

Thanks for reading Reality Check. I’d love to hear your thoughts any time at [email protected]. And if you know someone who might like this newsletter, just forward it along and point them to www.theinformation.com/reality-check so they can sign up for it.

See you tomorrow,

Mathew