Apple CEO Tim Cook. Photo illustration: 731; Photo: Reuters/Redux
The device has strayed from Tim Cook’s original vision, but it will still define the field.
Thu May 18 2023 04:00:13 GMT-0700 (PDT)
Many times throughout its history, Apple Inc. has redefined consumer technology by breathing new life into an existing category of gadgetry that has yet to fulfill its promise. On June 5, Apple Chief Executive Officer Tim Cook will take the virtual stage at Apple’s Worldwide Developers Conference and try to do for mixed-reality headsets what his company has previously done for music players, smartphones, smartwatches and earbuds. Despite Apple’s track record, an air of doubt surrounds the device, expected to be sold under the Reality name. It will run on a new xrOS operating system and cost about $3,000. Billions of dollars of investment into computers you wear on your face from the tech industry’s largest companies and well-funded startups have yet to produce a breakout success.
The device Cook will present, say people familiar with a development process that spread over seven years, has deviated far from his initial vision. Initially imagined as a pair of unobtrusive eyeglasses that could be worn all day, Apple’s device has morphed into a headset that resembles a pair of ski goggles and requires a separate battery pack.
The stakes are high. For Cook, it’s the release of a long-awaited product that could be one of his last big swings as Apple CEO and will affect his legacy, either by giving him another major achievement or underscoring the narrative that the company’s biggest victories were initiated under his predecessor, Apple co-founder Steve Jobs. For Apple, it’s the culmination of a multibillion-dollar development process, and some people within the company have described it as the potential foundation of a post-iPhone era. For others pursuing mixed reality and the metaverse, Apple’s headset could finally prove that the technology can live up to its long-promised, never-quite-realized potential.
“Having an entrant like Apple, who usually doesn’t jump into a market until it’s ready, feels good,” says Peggy Johnson, CEO of Magic Leap Inc., an augmented-reality company that raised $3.5 billion with a sweeping plan to build a general-use mixed-reality headset and then pivoted to a more modest vision focused on the enterprise market. Johnson says she wouldn’t underestimate Apple’s ability to develop a mass consumer audience for its device. “It can be tough to start a market, but you have to hand it to Apple. They’re good at things like this,” she says.
Magic Leap isn’t the only mixed-reality company to have trouble achieving its largest ambitions. Facebook’s 2021 transformation into Meta Platforms Inc. was based on the idea that it could turn its Quest line of virtual-reality headsets into a primary gateway to the metaverse, but that computing paradigm seems no closer today, and there are signs that the company’s commitment to its vision is wavering. Microsoft Corp. released the first version of its HoloLens in 2016, hoping it could become a powerhouse for gaming and consumer applications; the company is now selling it mostly as a tool for businesses, and a major contract with the US Army has run into significant trouble.
Companies have spent years trying to find the right formula for mixed reality
The details in this story are drawn from conversations with people involved in the development of Apple’s headset, who spoke on the condition of anonymity because of the company’s prohibitions on discussing unreleased products. They describe how Apple started the project convinced that mixed reality would be important but concerned that the devices would be socially isolating. After initially setting its sights on a lightweight pair of augmented-reality glasses, Apple gradually drifted toward something that felt more like existing devices because of technological constraints, the desire to get a product on the market and internal disagreements. The company declined to comment.
Apple’s ambition is that customers will eventually wear the device continuously all day, replacing daily tasks done on an iPhone or a Mac such as playing games, browsing the web, emailing, doing FaceTime video calls while collaborating in apps, working out and even meditating. It will feature hand and eye control and run many of the kinds of apps found on Apple’s other devices.
The company doesn’t see the headset being as immediately transformative as the iPhone. But internal projections give it the potential to eventually be as big as the iPad or the Apple Watch, as the company adds features and reduces the price with subsequent versions. That could mean a contribution of more than $25 billion annually to the company’s revenue. Apple knows this will take time. It initially hoped it could sell about 3 million units a year out of the gate, but it’s pared back those estimates to about 1 million, then to 900,000 units. By comparison, the company sells more than 200 million iPhones a year.
The lower initial estimates in part reflect Apple’s decision to sell the headset roughly at cost instead of at a loss, something it once considered. The product’s design is also a tacit admission that the company, like others that have made mixed-reality headsets, hasn’t been able to solve some core technological problems. Features such as the ability to function as an external Mac monitor and to make multiperson video calls are less advanced than the company initially intended, though it hopes to improve them. Apple also had wanted to integrate the battery into the headset, according to people familiar with the project. But to reduce weight and keep the device from overheating, it made a very un-Apple-like design compromise: It redesigned the battery as an iPhone-size pack that sits in a user’s pocket, attached by a power cord.
Michael Gartenberg, a former Apple marketing executive who’s now an independent consultant, warns that the device could be “one of the great tech flops of all time,” citing the lack of a real market for mixed-reality headsets and the performance of the Magic Leap and HoloLens devices. “I suspect there’s a lot of internal pressure for the next big thing,” he says.
The term “mixed reality” emerged as a way to describe a class of related visual technologies. One of them, virtual reality, seals off users from much of the real world to immerse them in a digital experience. The other, augmented reality, makes digital items appear to inhabit the physical world. For years, technology companies have been predicting that these technologies will mark the next major shift in personal computing.
Apple began considering building a headset around 2015. It started with other products, including Samsung’s Gear VR and the HTC Vive, as the foundation for its own prototypes and experiments, giving demos to top executives and board members. Cook was adamant in his preference for augmented reality, preferably in the form of lightweight glasses. “Nobody in here—few people in here—think it’s acceptable to be tethered to a computer walking in here and sitting down. Few people are going to view that it’s acceptable to be enclosed in something, because we’re all social people at heart,” he told a group of students at a 2016 technology conference in Utah.
Despite his strong views, Cook wasn’t deeply engaged in the specific design of the headset, say people who have worked with him. This was notably different from Jobs, who was famous for imposing his strong design sensibilities onto Apple products, down to the feel of a touchscreen or the shade of blue used in a Mac app icon. Cook, in contrast, made his name overseeing operations and has never been known as a “product guy.” His more distant approach was consistent with his role in the development of the Apple Watch and AirPods. “The closest Cook gets to product development is a demo,” says one of the people. “But even then, he’s not the type of guy who says it should do X and not Y. He’s the complete opposite of Steve in terms of having strong opinions on the minutiae.”
Still, some people involved in the headset project say Cook’s relative noninvolvement has been more consequential this time, given the stakes—the budget has exceeded $1 billion annually, with more than 1,000 engineers dedicated to the project—and the extent to which the direction of the project has changed. His approach was sometimes perceived as indecision, leading to delays and concerns about obtaining sufficient resources. “Tim didn’t throw his weight around the project at all, and this frustrated people,” says another person who worked on the project.
Other key figures in Apple’s top ranks, such as Craig Federighi, senior vice president for software engineering, have also kept their distance and seemed wary of the headset, according to people familiar with the project. Johny Srouji, Apple’s senior vice president for hardware technologies, has privately been a skeptic, likening it to a science project. Internally he’s warned that building the high-performing chips needed for the device could distract from new iPhone chips, which would probably drive more revenue. Srouji’s group did end up developing some of Apple’s most advanced chips to date for the headset, while iPhone speed gains have indeed slowed in recent years.
FederighiPhoto illustration: 731; Photo: Bloomberg
Dan Riccio, who was Apple’s hardware leader as the headset project began, hired former Dolby Laboratories Inc. Chief Technology Officer Mike Rockwell in 2015 to work on device displays before Rockwell pivoted to assemble a team, which was dubbed the Technology Development Group, or T288, to explore head-worn devices. As Rockwell was getting started, Apple’s industrial design team, led by Jony Ive, the chief design officer at the time, was coming off the watch’s introduction and looking for its next initiative. It, too, was exploring head-worn devices and quickly got involved with Rockwell’s team.
Rockwell’s and Ive’s teams quickly disagreed on the project’s direction. The headset team initially wanted to build a device that would display virtual-reality content in video-realistic form. This goal required shipping a base station the size of a Mac mini that would beam over the most powerful graphics, enabling top-flight video games and hyperrealistic content. This setup has been a common way to increase the power of VR headsets, but device makers have increasingly sought to avoid it.
Ive, who remained involved in development until about a year ago, preferred a stand-alone, maximally portable device, even if this meant sacrificing some performance. He also expressed concern that Apple would end up creating a product that isolated humans from one another. Ive’s vision, which hewed much closer to Cook’s conception of a glasses-like device, eventually won over Apple’s executive team. (Ive didn’t respond to an interview request.)
SroujiPhoto illustration: 731; Photo: Bloomberg
To get as close as possible to Ive and Cook’s vision, the team developed a compromise: a VR device, code-named N301, that would function in some ways like an AR device. In contrast to other augmented-reality headsets such as HoloLens and Magic Leap, users wouldn’t see their surroundings directly. Instead, external video cameras would capture their environment and display it on a screen when users switched the headset from VR mode to AR mode, a feature known as “video pass-through.”
In an attempt to keep headset wearers engaged with the real world, the device will have an outward-facing display showing their eye movements and facial expressions. Apple regards this feature as a key differentiator from enclosed VR headsets. One person familiar with the device says the exterior screens allow people to interact with a headset wearer without feeling as if they’re talking to a robot.
By the end of 2017, Apple said it believed it could begin selling the device by 2020. But the project was slowed by challenges in both hardware and software development, as well as a lack of agreement about which applications were most likely to resonate with consumers. The onset of the Covid-19 pandemic delayed progress further. In 2021, Riccio, the hardware chief, left his broader position but stayed on solely to finish work on the headset debut.
Apple had realized soon into the project that it wasn’t feasible to build AR glasses that would be sufficiently powerful to be useful. The company’s engineers determined it would have to replicate the performance of an iPhone while using only a tenth of the power, to keep from getting too hot. Despite this obstacle, Rockwell’s team continued to describe its work as laying the foundation for AR glasses—a project internally dubbed N421—to secure resources, even as only about 10% of those resources were going toward that end.
A person on the project describes a running joke that engineers were working on the hopeless N421 just to keep Cook happy. By 2019 the company had made little or no headway on developing a viable plan to make AR glasses. Yet, in an all-hands meeting at the end of that year, Rockwell told hundreds of his colleagues that Apple could introduce glasses one year after it began selling the first headset, say people who were at the event.
Apple eventually postponed any serious product development on stand-alone glasses for years, all but killing the idea, according to people involved in the process. They say that Apple is at least four years away from introducing any such product, if it ever happens. In March, Apple gave an in-depth preview of the mixed-reality headset to its top 100 executives, and the company plans to begin selling it in the coming months.
In many important ways, Apple has followed the path of other companies pursuing mixed-reality tech. During the planning stages, it had high hopes for a self-contained, comfortable wearable device that would feel more like a fashion accessory than a computer strapped to your face. The engineers had faith that the technical challenges of shrinking the components while maintaining processing power and battery capacity weren’t insurmountable. As the project progressed, though, the solutions to key issues never emerged, and the need to ship a product drove engineers to find the best compromises they could.
One open question is how people who buy the device will use it. “It was very clear what the iPhone and iPad would do, but the watch meandered all over the place,” a person with knowledge of the product says. “The headset will be similar, but there is hope that third-party apps will save it.” (One internal presentation suggested that people will wear the headsets to parties in the physical world, interacting with people through the external devices.) Apple is already engaging with software and game developers, as well as other entertainment companies to have content ready once the device goes on sale.
When Cook makes his pitch in June, he’s unlikely to dwell on how much Apple’s headset resembles the bulky ones he criticized at the outset of this process. Still, having something to sell is just one of the necessary steps along the way to finding what works, says one person familiar with the project. “You land a beachhead,” this person says, “and improve on it.”