Hands-On With Apple’s ‘ARKit’ Augmented Reality Demo for Developers

With iOS 11, Apple is delving into augmented reality in a big way, introducing an ARKit development platform that will allow developers to quickly and easily build augmented reality experiences into their apps and games.

ARKit is positioned to be the largest AR platform in the world when it launches this fall, using the camera, processors, and motion sensors in the iPhone and iPad to create some incredibly impressive augmented reality interactions.

While we won't see the first augmented reality apps and games built on ARKit for a couple of months, Apple has an ARKit demo app to show off what ARKit can do. We went hands-on with the demo to give MacRumors readers just a small taste of what to expect.

Subscribe to the MacRumors YouTube channel for more videos.

ARKit uses technology called Visual Inertial Odometry to track the world around an iPad or iPhone, allowing a device to sense how it moves in a room. ARKit automatically analyzes a room's layout, detecting horizontal planes like tables and floors, which then allows virtual objects to be placed upon those surfaces.

With ARKit able to place any virtual object within a physical room, developers can create all kinds of unique experiences, and developers have already released several demos showing what might be possible.

Apple already has at least one major retailer on board to use ARKit -- IKEA. IKEA is developing a new augmented reality app built on ARKit that will let customers preview IKEA products in their own homes before making a purchase. IKEA has offered augmented reality functionality for a few years now, but the company says Apple's new platform will much improve the experience. With ARKit available, IKEA says augmented reality will now "play a key role" in new product lines.

For additional details on other developer tools and features coming in iOS 11, make sure to check out our full iOS 11 roundup.

Related Roundups: Apple VR Project, iOS 11
Tag: ARKit

Discuss this article in our forums

AR/VR Headset Shipments Forecast to Hit 100 Million Units By 2021, Buoyed By Apple, Microsoft, and Others

Earlier this month at the Worldwide Developers Conference, Apple revealed its first big push into the augmented reality and virtual reality spaces, with a new ARKit developer framework and high-performance iMacs with native support for VR content creation.

On Monday, research firm IDC published new data forecasting significant growth in both markets, with dedicated AR and VR headset adoption expected to increase from just under 10 million units last year to 100 million units in 2021.


VR headsets account for much of the device volume so far, with VR headsets powered by a smartphone proving the most popular, according to IDC. The second half of 2016 also saw an increase in volume of Sony PlayStation VR, HTC Vive, and Facebook's Oculus Rift.
"The next six to 18 months will further stimulate the VR market as PC vendors, along with Microsoft, introduce tethered headsets and high-end standalone VR headsets also enter the market," said Jitesh Ubrani senior research analyst for IDC Mobile Device Trackers. "With lower hardware requirements on the PC and lower prices on headsets, VR will be more accessible than ever before. And the introduction of additional motion tracking and hand tracking will help further blur the line between digital and physical reality."
Although IDC believes VR headsets will continue to dominate the market in terms of volume for the foreseeable future, the firm believes AR will have a much bigger impact on the industry as a whole, in part thanks to Apple's recent entry onto the scene.
In terms of dedicated devices, AR continues to sit slightly in the background of VR. The reason for this is not that AR is less important, but rather it is harder to achieve. IDC believes VR headsets will continue to lead in terms of volume throughout the forecast, but maintains that AR in general will have a much bigger impact overall on the industry. Consumers are very likely to have their first AR experience via a mobile phone or tablet rather than a dedicated headset, and Apple's recent introduction of ARKit further supports this.
IDC believes AR headsets will become increasingly popular in markets such as healthcare, manufacturing, field service workers, and design, with commercial shipments to account for just over 80 percent of all AR headsets shipped in the next 5 years. "We believe that many industrial jobs will fundamentally change because of AR in the next 5-years," said Ryan Reith, IDC program vice president. "These are much more opportunistic markets for dedicated AR headsets than the consumer market."

Mobile app developers have been sharing early creations using Apple's new ARKit, suggesting huge enthusiasm for the possibilities for AR on iOS devices. As for VR, Apple's enthusiasm was clear during its WWDC keynote, with the company showing off the power of its new iMacs through a live demo of VR content creation using a HTC Vive, made possible via a new Metal 2 developer kit that has provisions for external GPUs and VR headsets.

Related Roundup: Apple VR Project
Tags: IDC, ARKit

Discuss this article in our forums

Developers Share First Augmented Reality Creations Using Apple’s ARKit

Apple only announced its augmented reality developer platform ARKit last week at the Worldwide Developers Conference, but some video clips have already been posted online by developers eager to showcase the impressive potential of the software.

ARKit enables iPhones and iPads running iOS 11 to superimpose computer-generated graphics over the real world, allowing developers to take their apps beyond the screen and into the user's environment.

Apple ARKit running Unity and Overwatch Widowmaker, by Cody Brown

By using the built-in camera, processors, and motion sensors found in iOS devices, virtual content appears on top of real-world scenes, and users don't need any special equipment to enjoy them because ARKit does the heavy lifting.

One developer who spoke to Motherboard said Apple had improved upon existing AR solutions like Hololens and Google Tango by making the ARKit framework elegant and simple to use.
"The most impressive aspect of ARKit is that it tends to just work," said Cody Brown, founder of virtual reality production studio IRL, in an online interview with Motherboard. "Other AR software often requires some sort of physical tracking mechanism (like a QR code), which inevitably becomes a major piece of friction if you are trying to get anyone to use this stuff.

"Another incredible aspect of ARKit is how it handles lighting adjustments in real time, continued Brown. "I can only imagine the math and magic underneath this tech to make it work."
During its keynote at WWDC, Apple demoed a range of effects that ARKit is capable of. One demo by Wingnut Studios wowed the audience with a tabletop sci-fi raider assault. Another showed a user placing a steaming digital coffee mug and a lamp on a table and moving the objects around to show off the tracking and shadow effects.

Several other ARKit developer demos have since appeared online, including a virtual "iPhone 8", roving StarWars character BB-8, dog-stalking zombies, office floor naval battles, dancing Candy rock stars, and more.

Rainforest garage by 8ninths

Apple has invested heavily in augmented reality, making several VR/AR company acquisitions in recent years including PrimeSense, FaceShift, and Metaio. Apple's much-anticipated "iPhone 8", which is expected to be announced in September, is rumored to have 3D sensing capabilities, with augmented reality being one possible use for the technology.

ARKit will come to compatible devices as part of iOS 11, which Apple is set to release in the fall.

Related Roundups: Apple VR Project, iOS 11
Tag: ARKit

Discuss this article in our forums

Apple Offers External GPU Enclosure to Developers As Valve Opens Door to VR Mac Gaming

Apple announced a new version of its Metal graphics technology during Monday's keynote at the Worldwide Developers Conference. As part of macOS High Sierra, Metal 2 will officially support external GPUs, allowing any Mac with a Thunderbolt 3 port to benefit from graphics hardware powerful enough to run demanding virtual reality applications and games.

The native VR support in macOS High Sierra also opens up the possibility for Mac owners to hook up VR headsets to their computer for the first time. And in a concurrent related announcement, Steam game platform creator Valve also revealed in a blog post on Monday that it is making a beta version of its SteamVR software development kit available on Mac, offering players the same 360-degree, room-scale tracking as the Windows and Linux variants.

On the development side, we have worked closely with Epic and Unity to make Mac extensions of content built on those engine technologies as simple as possible. Extension tools for those engines, and others, are available as part of this beta.

We've also worked with Mozilla to help enable WebVR support on Firefox, so macOS-based web developers can start trying out VR.
In addition to the Valve partnership, Apple announced it is also selling its own external graphics enclosure to developers who want to work on graphically intensive VR and 3D applications and games, although Apple noted that external GPU support likely won't arrive for consumers until spring 2018.
Apps that use Metal, OpenCL, and OpenGL can now take advantage of the increased performance that external graphics processors can bring. The External Graphics Development Kit includes everything you need to start optimizing advanced VR and 3D apps on external graphics processors with macOS High Sierra.
Apple's External Graphics Development Kit comes with a Sonnet external GPU chassis with Thunderbolt 3 and 350W power supply, an AMD Radeon RX 580 8GB graphics card, a Belkin USB-C to 4-port USB-A hub, and a promo code for $100 towards the purchase of a HTC Vive VR headset.

The External Graphics Development Kit costs $599 and requires a Mac with Thunderbolt 3 running the latest beta version of macOS High Sierra. The other caveat is that customers have to be a member of the Apple Developer Program to be eligible to purchase the kit.

The kit can be bought directly from Apple's website, although Apple cautions that the HTC Vive promo codes have limited availability and are distributed on a first-come, first-served basis.


Discuss this article in our forums

Apple Hires NASA Augmented Reality Expert Jeff Norris

Apple has hired Jeff Norris, an augmented reality expert who founded the Mission Operations Innovation Office at NASA's Jet Propulsion Lab, reports Bloomberg.

Norris has reportedly joined Apple as a senior manager working on the augmented reality team led by Mike Rockwell, who formerly ran Dolby Labs. The team is said to be working on the previously-rumored augmented reality smart glasses as well as AR features for future versions of the iPhone.

Prior to joining Apple, Norris worked at the NASA Jet Propulsion Laboratory, where he was employed since 1999. Along with founding the Mission Operations Innovation Office, he founded the JPL Ops Lab for developing human-system interfaces for mission operations, and he led multiple projects focused on human-system interaction with an emphasis on virtual and augmented reality.

On his website, Norris features a speech he gave on augmented reality and Nasa's JPL Ops Lab, much of which was focused on augmented reality headsets and their uses.


Under Norris' leadership, the JPL Ops Lab provided the Microsoft HoloLens to astronauts onboard the International Space Station and developed software for virtually working on Mars with the HoloLens.


For the last couple of years, Apple has taken a deep interest in augmented and virtual reality, and is said to have a large team of employees working on the technologies and exploring ways they could be used in future Apple products.

Apple has been working on both virtual reality headsets and augmented reality smart glasses, with the aim of launching the latter in 2018. We've also heard rumors suggesting augmented reality functionality could be incorporated into the iPhone, perhaps as early as the iPhone 8 set to be released this September.

Apple CEO Tim Cook has expressed his excitement about augmented reality several times in recent months. "I think AR is that big, it's huge. I get excited because of the things that could be done that could improve a lot of lives," he said in February of 2017.

Related Roundup: Apple VR Project
Tags: bloomberg.com, NASA

Discuss this article in our forums

Apple Increasing Focus on Augmented Reality Eyewear, Any Launch Still Over a Year Away

Apple is "stepping up" its augmented reality eyewear efforts, with an eventual goal of releasing a consumer-facing product, according to the Financial Times.
Apple first began to build a team to examine the feasibility of a head-worn device more than a year ago. Now, it is devoting more resources to its augmented-reality efforts, with the aim of taking it from a science project towards a consumer product, according to people familiar with the company’s plans.
Don't expect to be wearing a pair of Apple-branded augmented reality glasses in the near future, however, as the report said any potential launch remains at least a year away, or "perhaps much longer."

Bloomberg was first to report that Apple is exploring digital glasses that would connect wirelessly to iPhones and "show images and other information in the wearer's field of vision." The report said the digital glasses, which may use augmented reality, would not launch until 2018 at the earliest.

One person who expects a sooner launch is tech evangelist Robert Scoble, who insists that Apple is working on a pair of "mixed reality" glasses that will debut alongside the 2017 iPhone lineup, according to his sources. He also said Apple and German company Carl Zeiss are working together on augmented reality optics.

Scoble's information has yet to be corroborated by other sources, so 2018 or later remains a more likely timeline at this point.

While a standalone product appears to remain far off, most analysts agree that Apple will start incorporating AR-related features into its existing products in the near future. A future iPhone camera, for example, could be able to detect faces and apply Snapchat-like filters using augmented reality.

Apple CEO Tim Cook has shown significant interest in augmented reality over the past few years, calling the technology everything from "profound" to a "big idea like the smartphone" in recent interviews. He also believes that augmented reality is essentially more important than virtual reality.

Last year, Cook said that Apple continues to "invest a lot" in augmented reality, and the company has filed several patents related to the technology over the past decade, confirming its interest in the field. However, Apple routinely tests new products and technologies that are never publicly released.

Apple's augmented reality efforts have been preceded by the Microsoft HoloLens, a cordless, self-contained Windows 10 holographic headset that mixes virtual reality with augmented reality. Microsoft began shipping the HoloLens Development Edition in March 2016 for $3,000 in the United States and Canada.

Related Roundup: Apple VR Project

Discuss this article in our forums

Apple Exploring AR in Israel as Robert Scoble Insists ‘Mixed Reality’ Glasses Coming This Year

Augmented reality could be the "next big thing" for Apple, according to the latest research from UBS analyst Steven Milunovich.

Milunovich believes that Apple could equip the iPad and iPhone with AR-enabled hardware such as sensors, 3D cameras, and custom chipsets over the next two to three years.

He speculates the iPhone 8, for example, could have "moderate 3D mapping using stereoscopic vision" based on a technology called Simultaneous Localization and Mapping.

An excerpt from his research note obtained by MacRumors:
SLAM solves the chicken and egg problem of mapping a space without prior knowledge. The device starts from one angle and collects points of data. As the device moves, it observes how objects shift and begins to fuse data to create an environment. As the 3D map is forming, the device begins to understand its own positon in the context of the environment it is trying to map.

SLAM underlies most of the advanced AR technologies available today and will likely be critical in the future. In 2015, Apple bought software company Metaio, which had one of the most advanced SLAM-based AR engines available at the time. The company has been silent since being acquired, but according to our industry checks it's likely the core team has remained and the IP could play a key role in Apple's strategy.
Milunovich, citing "some industry sources," added "there may be over 1,000 engineers in Israel working on projects that could be related to AR," but he expects Apple to slowly roll out augmented reality features until the technology becomes more mainstream and "reaches a level of maturity suitable for Apple's brand."

Perhaps related, iPhone 8 development is said to be ongoing at Apple's offices in Israel. Earlier this month, Apple acquired Israeli startup RealFace, a cybersecurity and machine learning firm specializing in facial recognition technology. And in November 2013, Apple acquired Israeli 3D sensor company PrimeSense.

Milunovich said that because new technologies must fit within and grow the Apple ecosystem, the company will likely release an augmented reality SDK that is tightly integrated with iOS.

Beyond the next two to three years, Milunovich thinks an additional AR-enabled hardware device is possible, citing tech evangelist Robert Scoble's belief that Apple will launch a pair of smart glasses in partnership with German optics company Carl Zeiss. The report said the glasses could launch as early as 2017.

In November 2016, Bloomberg likewise reported about Apple working on a pair of smart glasses that would connect wirelessly to the iPhone and display "images and other information" to the wearer.

Scoble recently told MacRumors that Apple is "readying a three-ounce pair of glasses that pair with the iPhone 8 for mixed reality." He insists that "mixed reality," which he refers to as "next-generation augmented reality," will be key to Apple's future products. Treat this info with a proverbial grain of salt for now.

Apple CEO Tim Cook has repeatedly expressed his interest in augmented reality. He recently said AR is "a big idea like the smartphone," but noted "there are things to discover before that technology is good enough for the mainstream." In the past, he said AR is "profound" and should "amplify" human contact.


Discuss this article in our forums

Tim Cook Thinks Augmented Reality is a ‘Huge Idea’ Like Smartphones

Apple CEO Tim Cook has been on a whirlwind of a trip in Europe this week, making stops in France, Germany, and the United Kingdom to meet with everyone from students and developers to Apple retail employees and government officials.


On Thursday, Cook visited the Ustwo Games offices in London to meet the creators of popular iPhone game Monument Valley. There, he later sat down for a chat with The Independent's David Phelan to discuss a wide variety of topics, ranging from Apple's hardware roadmap to augmented reality.

Responding to a question about how difficult it is to stay ahead of the curve, and to put the right hardware in place for the next generation of developers, Cook said Apple will "continue to push" not only on the hardware side, but also on the software side with its Swift programming language.
We try to continually push ourselves to do more and more, not just on the hardware side but also in terms of developers’ tools so they can take advantage of the hardware that’s there, in the best way. That’s the heart of what the coding software Swift is about. We’ve created the language and our hope was that you can get a lot more people coding, and then secondly have people push more to take advantage of the latest hardware.
Cook said the importance of Apple being able to design its own hardware is "rising exponentially" due to things like machine learning and the company's desire to "maintain a level of privacy" for its users.

In terms of augmented reality, Cook reiterated that he is "excited" about the technology because it "allows individuals to be present in the world but hopefully allows an improvement on what’s happening presently." But he added there are "things to discover" before the technology is "good enough" for the masses.
I regard it as a big idea like the smartphone. The smartphone is for everyone, we don't have to think the iPhone is about a certain demographic, or country or vertical market: it’s for everyone. I think AR is that big, it’s huge. I get excited because of the things that could be done that could improve a lot of lives. And be entertaining. I view AR like I view the silicon here in my iPhone, it’s not a product per se, it’s a core technology. But there are things to discover before that technology is good enough for the mainstream. I do think there can be a lot of things that really help people out in daily life, real-life things, that’s why I get so excited about it.
Cook said the developers, among others, he met on his trip were "incredibly uplifting," particularly with Brexit looming in the United Kingdom.
You can really see the start-up community gaining momentum. That’s important any time but with Brexit hanging over – from some people’s point of view – it’s even more important that these stories get out where people have something to look at and say “You know, times are not really awful, there’s some great things happening.” It gives me a lot of energy to talk to developers, or meet students in classrooms who are using our technology to help learn faster and better. Watching them pursue their passion.
Earlier today, Cook visited the BRIT School for Performing Arts and Technology to check out students' digital design work. He also visited the Woodberry Down primary school and met with London mayor Sadiq Khan to discuss access to talent, digital skills, and Apple's investment in the city.

Full Interview: Apple CEO Tim Cook: As Brexit Hangs Over UK… at The Independent

Related Roundup: Apple VR Project
Tag: Tim Cook

Discuss this article in our forums

Apple Patents Detail AR/VR System Suitable for Smartphones, Explore Object Recognition Challenges

Apple was granted a pair of patents on Tuesday that depict a mobile augmented reality system that can detect objects in the surrounding environment and overlay them with virtual information.

Picked up by AppleInsider, the first patent published by the U.S. Patent and Trademark Office is titled "Wearable information system having at least one camera" and describes an AR device with intelligent object recognition capabilities.

AR patent
Combining cameras, a screen, and a user interface, the device is described as being ideal for a head-mounted display, but the patent also suggests a use for the system in future smartphones, owing to the power efficient way in which it monitors the environment.

The device is described as having a default low-power scanning mode for normal operation, with high-power modes activated for short periods – when downloading and displaying AR content, for example. The rest of the patent gets quite technical as it details methods of optical tracking and determining camera orientation, two areas of AR that will prove particularly difficult to perfect.

Detecting, matching, and describing environmental features quickly and accurately are two challenges that Apple suggests could be overcome using a combination of dedicated hardware and pre-learned data. One aspect of such a system would use depth detection, achieved via dual-lens cameras similar to those found in the iPhone 7 Plus, which also utilizes depth mapping algorithms to produce Portrait Mode photos.

Apple AR patent
The second patent, titled "Method for representing virtual information in a real environment", details a way of labeling points in a topological real-world space while taking in environmental features such as object occlusion as the user moves around.

In one example, a user views an AR city map on a transparent display – or on a smartphone screen feeding a live image from a rear-facing camera – and pulls up information relating to nearby buildings or landmarks.

The patent goes on to suggest using geometry models, depth-sensing, positioning data and other advanced technologies to compensate for shortcomings in current visual modeling systems. The second patent also relies on a dual-camera system for creating a depth map of the immediate environment onto which the virtual data is superimposed.

The described technologies may or may not appear in future Apple products, but it's worth noting that both patents were originally filed for by German AR specialist Metaio just before Apple acquired the firm in 2015. Existing staff would have likely joined Apple's team dedicated to AR/VR research, to help explore how the technologies can be incorporated into future products.

Several times over the last year, Tim Cook has spoken of Apple's deep interest in augmented reality. "AR can be really great. We have been and continue to invest a lot in this. We're high on AR in the long run," he said in July.

Related Roundup: Apple VR Project
Tag: patent

Discuss this article in our forums

Apple Allegedly Partnering With Carl Zeiss for Augmented Reality Glasses

Apple and German company Carl Zeiss AG are working together on a pair of augmented reality smart glasses that could be announced as soon as this year, claims well-known blogger Robert Scoble in a Facebook post.

According to Scoble, a Zeiss employed allegedly confirmed the partnership between the two companies in a conversation that took place at the Consumer Electronics Show in Las Vegas, Nevada last week.

zeissbooth
Zeiss' booth at CES. Image via Robert Scoble

Scoble questioned the Zeiss employees at the show after noticing Zeiss had a booth in the augmented reality section of CES without showing off an augmented reality product. He believes Apple forced Zeiss to keep quiet about an upcoming release.
Exclusive news: Apple and Zeiss working together on augmented reality optics. [...]

A Zeiss employee confirmed the rumors that Apple and Carl Zeiss AG are working on a light pair of augmented reality/mixed reality glasses that may be announced this year. (I thought it was next year but now that I saw this I believe it will happen this year).
It's not clear how reliable Scoble's information is, but previous rumors have suggested Apple is working on a set of smart glasses that would connect wirelessly to the iPhone and display images and other information to the wearer.

Apple is said to have contacted suppliers about the project, which appears to be in the exploratory prototyping phase of development. While Scoble believes the glasses could be released in 2017, previous information points towards a 2018 or later release.

Apple has a full team dedicated to AR/VR research and exploring how the technologies can be incorporated into future products. Multiple times over the past year, Tim Cook has said that Apple has a deep interest in augmented reality. "AR can be really great. We have been and continue to invest a lot in this. We're high on AR in the long run," he said in July.

Related Roundup: Apple VR Project
Tags: Robert Scoble, Zeiss

Discuss this article in our forums