Latest ARKit Demo Showcases Virtual Drawing

Ahead of the launch of iOS 11, developers have been tinkering with ARKit, Apple's upcoming augmented reality platform, and showcasing the myriad ways that it can be used in apps and games.

The latest demo comes courtesy of Osama Abdel-Karim, who uses ARKit to virtually paint on a notepad using his fingers.


According to Abdel-Karim, an iOS 11 library named Vision was used to develop the virtual drawing feature. Vision includes an object tracking feature that is able to detect the thumbnail of a finger and track its movement to enable the drawing.

Abdel-Karim has outlined the steps he used to create his ARKit demo and provided the full source code for the project.

When it launches this fall, ARKit is positioned to become the largest AR Platform in the world, using the camera, processors, and motion sensors in the iPhone and iPad to create some incredibly impressive augmented reality interactions.

As outlined in our video covering ARKit, the feature uses technology called Visual Inertial Odometry to track the world around an iPad or iPhone, allowing a device to sense how it moves in a room. ARKit automatically analyzes a room's layout, detecting horizontal planes like tables and floors, which then allows virtual objects to be placed upon those surfaces.

Subscribe to the MacRumors YouTube channel for more videos.

The first apps and games with ARKit won't be available until iOS 11 is officially available to the public, but we've seen what ARKit can be used for, with developers demonstrating everything from live filter applications in a recreation of A-ha's Take On Me video to live measurements of furniture and room spaces.

Check out all of our previous ARKit coverage below to see what else developers can do with it:

- ARKit Roundup: Turn-by-Turn Directions, Precise Room Measurements, and Pac-Man
- Apple's ARKit Used to Recreate Classic A-ha 'Take On Me' Video
- Apple Users' Mixed Reality Future Teased in Latest ARKit Demo
- Latest Apps to Showcase Apple's ARKit Include Simple Measuring Tape and Minecraft
- Developers Share First Augmented Reality Creations Using Apple's ARKit


Discuss this article in our forums

Apple’s ARKit Used to Recreate Classic A-ha ‘Take On Me’ Video

Developers have been creating some impressive augmented reality apps and games with Apple's upcoming ARKit API, and the latest proof-of-concept video reimagines the video accompanying A-ha's 80s hit "Take On Me."

The video, created by augmented reality studio Trixi, uses ARKit, Unity 3D assets, GameFlow, and animations made using Mixamo to combine real world visuals with black and white sketched characters in the "Take On Me" video style.


The original A-Ha video featuring a sketched comic book character that comes to life took several months to create, but Trixi founder Chip Sineni told The Verge that the ARKit version took just a couple of days.

"The hardest part was not having the all the live video receive the post effect," said Sineni. "But [we] really wanted that effect of turning your world into the 'Take on Me' experience, not a baked in experience."

ARKit is set to position the iPhone and the iPad as the largest augmented reality platform when it launches alongside iOS 11. Developers have created a wide range of content using ARKit, ranging from the practical applications like room measurements and furniture placement to games.

The first apps able to take advantage of ARKit will launch this fall.


Discuss this article in our forums

ARKit Roundup: Turn-by-Turn Directions, Precise Room Measurements, and Pac-Man

Since the announcement of Apple's new augmented reality developer platform at WWDC in June, developers have been sharing interesting new AR experiences on iOS devices, including practical applications like measuring tape apps and basic character model demos.

Today, we've rounded up the newest examples of how ARKit could work in real-world scenarios, starting off with a maps addition that could bolster directions in Apple Maps. As with all ARKit demos, today's examples are not confirmed to be the final launch products for augmented reality apps coming down the line, but they are intriguing glimpses into what users can expect when the AR features debut on iOS 11 this fall.

Images via @AndrewProjDent

Shared by iOS developer Andrew Hart on Twitter, the first example of the AR-enhanced maps software overlays destinations on points of interest when looked at through the camera of your iPhone or iPad, giving an estimation of how far you are from each location.

Burrowing deeper into getting directions to a specific location, Hart used ARKit and Apple developer framework Core Location -- which lets developers integrate the geographic location and orientation of a device directly into their software -- to create augmented reality turn-by-turn directions.

Acquisitions of mapping companies and patent filings dating back to 2009 have long suggested that Apple is interested in adding AR features into Apple Maps, but the technology prior to ARKit has likely not been promising enough for such an implementation.


Continuing on the measuring AR app trend, a new tool was shared on the Made With ARKit Twitter account recently, allowing users to perform precise square foot measurements of an entire room. The last few measuring apps detailed in our ARKit roundup in June centered upon AR-enabled measuring tapes that could only provide distance estimates in a straight line.


For those interested in gaming AR apps, developer Kobi Snir shared a real-life version of Pac-Man that uses ARKit to place users directly within the game's maze, filled with dots and ghosts. The players take on the role of Pac-Man, and move around the maze to eat every dot while avoiding the ghosts. Another recent gaming-related ARKit example showcased what Minecraft would look like in the real world.


Games have been a core part of ARKit from the day it was announced, with Apple's senior vice president of Software Engineering Craig Federighi touting Pokémon Go as one of the first apps that will receive ARKit-related enhancements this fall. "The Pokémon is so real, he's right there on the ground," Federighi said at WWDC. "As the ball bounces, it actually bounces right there in the real environment. It's AR like you've never seen it before."

Of course, these are just a handful of recent examples of ARKit that developers have shared. Others include a graffiti doodling app, a shopping app (similar to IKEA's planned ARKit app), and an inter-dimensional portal. Apple CEO Tim Cook has said AR makes him so excited that he just wants to "yell out and scream," telling Bloomberg Businessweek last month that, "When people begin to see what’s possible, it’s going to get them very excited—like we are, like we’ve been."

Tag: ARKit

Discuss this article in our forums

Apple Users’ Mixed Reality Future Teased in Latest ARKit Demo

Developers have wasted no time testing the limits of Apple's new ARKit platform, as each passing week serves up tantalizing previews of what's possible on an iPhone or iPad now that developers can quickly and easily build augmented reality experiences into their apps.

The latest demo to raise the bar comes courtesy of New York-based virtual reality developer Normal VR and offers a sneak peek at what's potentially in store for anyone with an iOS device, an iMac, and a VR headset.


The video clip, recorded on a 10-inch iPad Pro, shows the company using ARKit and the Unity game engine to capture the physical movements of a VR artist and project her virtual painting onto the real world via the mimicking actions of a digital avatar. The interaction also appears to be two-way, with scaling and repositioning functions controlled by the external viewer using iPad gestures.

It's unknown whether the HTC Vive is connected to a Mac or linked to some other supporting machine, but either way the potential uses for such a setup could be pretty far-reaching.

We already know Apple's latest top-of-the-line iMacs will include VR headset support out of the box once High Sierra gets its final release. Steam game platform creator Valve has also developed a version of its SteamVR software development kit for Mac, offering Apple users the same 360-degree, room-scale tracking as the Windows and Linux variants, with the added bonus of Oculus Rift support.

If ARKit-wielding developers can create apps that allow VR content to be viewed and interacted with outside of a headset, cross-platform Apple users could be looking at an exciting future of shared mixed reality experiences.

The first apps powered by ARKit are set to launch on the new-look App Store alongside iOS 11 this fall.

Related Roundups: Apple VR Project, iOS 11
Tag: ARKit

Discuss this article in our forums

Apple Marketing Lead Greg Joswiak Talks ARKit in New Interview

Apple's vice president of product marketing Greg Joswiak this week sat down for an interview with The Australian, where he spoke about the company's augmented reality efforts, including ARKit and the response that its gotten from developers so far.

Enthusiasm about ARKit has been "unbelievable" says Joswiak, who went on to describe some of the things developers have built so far, including virtual tape measures that can accurately measure real-world objects. "It's absolutely incredible what people are doing in so little time," he said. Joswiak declined to speak about Apple's future AR plans, but said the company is going to "start at zero" with the iPhone and the iPad.

Subscribe to the MacRumors YouTube channel for more videos.
Apple was not talking about any plan to build AR glasses or a headset, but would instead promote its use on the iPhone and iPad, he said. "I think there is a gigantic runway that we have here with the iPhone and the iPad. The fact we have a billion of these devices out there is quite an opportunity for developers.

"Who knows the kind of things coming down the road, but whatever those things are, we're going to start at zero."
Joswiak also spoke about HomeKit, where he said he expects "plenty" of compatible HomeKit devices to be in homes in Australia before the HomePod speaker launches in December, and he highlighted both the new Siri voices in iOS 11 and Apple's work on machine learning. Apple has been using machine learning techniques since 2007, when the original iPhone launched. "We used machine learning to learn what you typed back in 2007," he said.

Machine learning, ARKit, and artificial intelligence play a major role in iOS 11. With ARKit, developers are able to quickly and easily build augmented reality experiences into their apps and games. When it launches this fall, ARKit will be the largest AR platform in the world, thanks to the myriad iPhones and iPads that are out on the market.

Related Roundup: iOS 11
Tags: Greg Joswiak, ARKit

Discuss this article in our forums

Latest Apps to Showcase Apple’s ARKit Include Simple Measuring Tape and Minecraft

iOS developers have already gotten their hands on ARKit, Apple's latest platform which enables developers to quickly and easily build augmented reality experiences into their apps. ARKit blog Made With ARKit has been sharing even more examples of the augmented reality apps that developers are toying around with, coming a few months ahead of when the first apps will launch to the public alongside iOS 11 this fall.

In two new videos shared on Twitter, developers have created useful measuring apps with ARKit, using the camera, processors, and motion sensors in an iPhone or iPad to calculate the size of various objects. In the first video, the app requires users to tap two locations and then shows the total distance between the spots as a floating number in the air.


In the second video (seen above), users choose a starting point for the virtual measuring tape, and then pan to where they want the end point to be located. The video compares the virtual AR measuring experience to a real measuring tape, and then tries it out on a picture frame and armoire. The app in the second video was built by Laan Labs, and they have a few other ARKit videos on their twitter, including one where they make a 3D drawing.

Gaming apps have already begun to take shape as well, with developer Matthew Hallberg building a Minecraft AR app with ARKit and Unity. The app lets players place Minecraft blocks around their real-world environment, and then destroy what they've built with Minecraft's traditional crafting and destruction mechanics.


One well-known partner for ARKit is furniture company IKEA, which plans to launch a new iPhone and iPad app that will let customers view objects in their own home through ARKit before they make a purchase. For more on ARKit, check out a hands-on video of the platform's demo that Apple created for developers and used to showcase the technology this year at WWDC.


Discuss this article in our forums

Supply Chain Evidence Mounts for Advanced 3D Sensing Abilities Coming to iPhone 8

Ahead of the iPhone 8's predicted September announcement date, supply chain reports over the past few days have indicated a ramp up of 3D sensing components that are predicted to be destined for augmented reality and biometric security applications in 2017's high-end iPhone 8. In a research note by BlueFin Research Partners, "winners" for iPhone 8 component production are said to include Finisar and Lumentum for 3D sensing modules and Broadcom for wireless charging components (via Barron's).

Reports began building last week when iPhone camera component supplier Largan Precision confirmed that it will ship lenses for 3D sensing modules in the second half of 2017, while refraining from directly mentioning Apple and iPhone 8. In a recent quarterly earnings report, supplier Finisar made remarks that heavily suggested it will be "one of two or three suppliers" of vertical-cavity surface-emitting lasers (VCSEL) in the iPhone 8, according to LoupVentures.

iPhone 8 rendering by Benjamin Geskin

Such technology could be implemented into the iPhone 8's rumored 3D sensing module and help "enrich the augmented reality experience" in the next-generation iPhone. Apple and CEO Tim Cook earlier this month detailed plans to back AR and its potential as an indispensable future technology with the reveal of ARKit, a developer platform that will provide the building blocks for consumer AR iPhone apps.

Likewise, Lumentum has announced that it expects to see volume shipments for its VCSEL lasers begin in the second half of 2017, timed with the annual production ramp up of iPhone manufacturing.
On last nights earnings call, Finisar management did not mention Apple by name, but they highlighted they expect to see volume VCSEL orders in their second fiscal quarter, which is the October quarter end of this calendar year. The company anticipates shipping “millions” of units during the quarter, but management also went on to say they anticipate unit shipments to be in the “10s of millions” in future quarters, which gives us further confidence 3D sensing and AR applications will be one of the focus features in the next generation of iPhones. In May of this year, Lumentum was the first VCSEL supplier to announce they anticipate volume shipments to begin in the second half of 2017.

We also want to highlight Finisar acknowledged they are shipping VCSEL lasers to multiple customers, but one customer (aka Apple) is accounting for the majority of total demand. We believe Finisar and others supplying VCSEL lasers are supply constraint and shipping everything they can manufacture. We believe Apple has secured a high percentage of all VCSEL lasers created, which we view as a large competitive advantage and will make Apple a leading AR player in the smartphone space.
LoupVentures further suggested that the significantly updated iPhone 8 will have a front-facing, low-end VCSEL laser and a rear-facing, high-end VCSEL laser, meaning that AR applications could take advantage of either camera. A long-running rumor for the iPhone 8 has been a dual-lens rear camera that could provide increased separation between the two lenses and allow for a greater difference in perspective, helping fuel better AR experiences. A front-facing dual-lens camera has been suggested as a possibility in the new iPhone as well.

In estimating the iPhone 8's cost, the site believes that Apple will add around $100 onto the cost of the smartphone specifically due to incorporating 3D sensing modules. That price comes to around $950 (with no storage tier size references), and largely falling into line with the new iPhone's predicted "premium" status.

For its part, Broadcom is said to have a predominant place in the supply chain for wireless charging components of the iPhone 8, a feature fellow supplier Wistron recently confirmed will be coming to at least one of 2017's iPhone models. iPhone 8's wireless charging is expected to be fueled by inductive technology, because long-range wireless charging for an iPhone is still expected to be years away from feasibility.

In total, reports surrounding iPhone 8 component manufacturing have been increasing recently, with Apple said to be dominating the NAND memory chip supply chain and increasing shortages for an already strained section of component supplies.

Related Roundup: iPhone 8
Tags: ARKit, augmented reality

Discuss this article in our forums

Hands-On With Apple’s ‘ARKit’ Augmented Reality Demo for Developers

With iOS 11, Apple is delving into augmented reality in a big way, introducing an ARKit development platform that will allow developers to quickly and easily build augmented reality experiences into their apps and games.

ARKit is positioned to be the largest AR platform in the world when it launches this fall, using the camera, processors, and motion sensors in the iPhone and iPad to create some incredibly impressive augmented reality interactions.

While we won't see the first augmented reality apps and games built on ARKit for a couple of months, Apple has an ARKit demo app to show off what ARKit can do. We went hands-on with the demo to give MacRumors readers just a small taste of what to expect.

Subscribe to the MacRumors YouTube channel for more videos.

ARKit uses technology called Visual Inertial Odometry to track the world around an iPad or iPhone, allowing a device to sense how it moves in a room. ARKit automatically analyzes a room's layout, detecting horizontal planes like tables and floors, which then allows virtual objects to be placed upon those surfaces.

With ARKit able to place any virtual object within a physical room, developers can create all kinds of unique experiences, and developers have already released several demos showing what might be possible.

Apple already has at least one major retailer on board to use ARKit -- IKEA. IKEA is developing a new augmented reality app built on ARKit that will let customers preview IKEA products in their own homes before making a purchase. IKEA has offered augmented reality functionality for a few years now, but the company says Apple's new platform will much improve the experience. With ARKit available, IKEA says augmented reality will now "play a key role" in new product lines.

For additional details on other developer tools and features coming in iOS 11, make sure to check out our full iOS 11 roundup.

Related Roundups: Apple VR Project, iOS 11
Tag: ARKit

Discuss this article in our forums

AR/VR Headset Shipments Forecast to Hit 100 Million Units By 2021, Buoyed By Apple, Microsoft, and Others

Earlier this month at the Worldwide Developers Conference, Apple revealed its first big push into the augmented reality and virtual reality spaces, with a new ARKit developer framework and high-performance iMacs with native support for VR content creation.

On Monday, research firm IDC published new data forecasting significant growth in both markets, with dedicated AR and VR headset adoption expected to increase from just under 10 million units last year to 100 million units in 2021.


VR headsets account for much of the device volume so far, with VR headsets powered by a smartphone proving the most popular, according to IDC. The second half of 2016 also saw an increase in volume of Sony PlayStation VR, HTC Vive, and Facebook's Oculus Rift.
"The next six to 18 months will further stimulate the VR market as PC vendors, along with Microsoft, introduce tethered headsets and high-end standalone VR headsets also enter the market," said Jitesh Ubrani senior research analyst for IDC Mobile Device Trackers. "With lower hardware requirements on the PC and lower prices on headsets, VR will be more accessible than ever before. And the introduction of additional motion tracking and hand tracking will help further blur the line between digital and physical reality."
Although IDC believes VR headsets will continue to dominate the market in terms of volume for the foreseeable future, the firm believes AR will have a much bigger impact on the industry as a whole, in part thanks to Apple's recent entry onto the scene.
In terms of dedicated devices, AR continues to sit slightly in the background of VR. The reason for this is not that AR is less important, but rather it is harder to achieve. IDC believes VR headsets will continue to lead in terms of volume throughout the forecast, but maintains that AR in general will have a much bigger impact overall on the industry. Consumers are very likely to have their first AR experience via a mobile phone or tablet rather than a dedicated headset, and Apple's recent introduction of ARKit further supports this.
IDC believes AR headsets will become increasingly popular in markets such as healthcare, manufacturing, field service workers, and design, with commercial shipments to account for just over 80 percent of all AR headsets shipped in the next 5 years. "We believe that many industrial jobs will fundamentally change because of AR in the next 5-years," said Ryan Reith, IDC program vice president. "These are much more opportunistic markets for dedicated AR headsets than the consumer market."

Mobile app developers have been sharing early creations using Apple's new ARKit, suggesting huge enthusiasm for the possibilities for AR on iOS devices. As for VR, Apple's enthusiasm was clear during its WWDC keynote, with the company showing off the power of its new iMacs through a live demo of VR content creation using a HTC Vive, made possible via a new Metal 2 developer kit that has provisions for external GPUs and VR headsets.

Related Roundup: Apple VR Project
Tags: IDC, ARKit

Discuss this article in our forums

IKEA Details Plans for Furniture Placement App Powered by Apple’s ARKit

At WWDC this year, Apple senior vice president of software engineering Craig Federighi performed a demo of the company's new augmented reality platform, ARKit, while mentioning popular furniture company IKEA as an upcoming partner in the technology. Similarly, Apple CEO Tim Cook referenced an IKEA AR partnership in a recent interview with Bloomberg Businessweek.

Now, Ikea executive Michael Valdsgaard has spoken about the company's partnership with Apple and ARKit, describing an all-new augmented reality app that will help customers make "reliable buying decisions" for IKEA's big ticket items (via Digital.di) [Google Translate].

Image via Digital.di

When it launches, the app will let customers choose which IKEA product they want and then use an iPhone or iPad powered by ARKit to see how the IKEA furniture looks in their own home before it's purchased. IKEA has been doing this for years now, including an early version which required users to scan pages of an IKEA catalogue to view AR furniture, but the company said that Apple's platform will greatly increase the consistency and quality of the experience.

Valdsgaard said that the app will support between 500 and 600 IKEA products at launch with more being added afterwards. The augmented reality experience will even directly play into the development and launch of new products, as Valdsgaard explained that the company plans to first debut new pieces of furniture in the app to give customers a taste of what's coming.
"This will be the first augmented reality app that allows you to make reliable buying decisions," said Michael Valdsgaard, digital transformation manager at Inter Ikea Systems, the company that owns the brand Ikea.

Going forward, it will play a key role in new product lines. "When we launch new products, they will be first in the AR app."
Selling furniture directly within the app is a possibility as well, but isn't the company's focus for the first iteration of the software. Valdsgaard said, "Ideally, you could put a sofa in your home with the help of the app, and then with one click add it in the shopping cart on the site. But we have a tight deadline."

Several teams are working on the technology behind the augmented reality app, including one "innovation team" of 150 people located in Älmhult, Sweden -- where the first IKEA opened -- as well as a twenty-person team specifically responsible for the 3D modeling of furniture. The ARKit-powered IKEA app is expected to launch this fall, following the iOS 11 update.

Tags: Ikea, ARKit

Discuss this article in our forums