IP Library Granted Patent US 10,825,253
Granted Patent B2
US 10,825,253 · App. 16/375,549 · Granted Nov 3, 2020

Generating augmented reality objects on real-world surfaces using a digital writing device

Inventors: Tenell Rhodes (Campbell, CA); Gavin S. P. Miller (Los Altos, CA); Duygu Ceylan Aksit (Mountain View, CA); Daichi Ito (Los Gatos, CA)
Assignee: ADOBE INC.
G06T19/006G06F3/0383G06F3/03545G06F3/0304G06F3/0346
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,825,253
App. No.
16/375,549
Granted
Nov 3, 2020
Kind
B2
Abstract

The present disclosure includes systems, methods, computer readable media, and devices that can generate accurate augmented reality objects based on tracking a writing device in relation to a real-world surface. In particular, the systems and methods described herein can detect an initial location of a writing device, and further track movement of the writing device on a real-world surface based on one or more sensory inputs. For example, disclosed systems and methods can generate an augmented reality object based on pressure detected at a tip of a writing device, based on orientation of the writing device, based on motion detector elements of the writing device (e.g., reflective materials, emitters, or object tracking shapes), and/or optical sensors. The systems and methods further render augmented reality objects within an augmented reality environment that appear on the real-world surface based on tracking the movement of the writing device.

Claims (79)

1. In a digital medium environment for generating augmented reality objects using a digital writing device, a computer-implemented method for creating a digital object in an augmented reality environment corresponding to a real-world surface, the computer-implemented method comprising:

determining an initial location of a writing device on a real-world surface;

detecting real-world movement of the writing device relative to the initial location by:

determining that the real-world movement of the writing device is above a speed threshold;

in response to determining that the real-world movement is above the speed threshold, determining the real-world movement of the writing device based on a first combination of two or more tracking techniques; and

in response to determining that the real-world movement is at or below the speed threshold, determining the real-world movement of the writing device based on a second combination of two or more tracking techniques; and

rendering, within an augmented reality environment generated by way of an augmented reality device, an augmented reality object that appears to be drawn on the real-world surface based on the real-world movement of the writing device.

2. The computer-implemented method of claim 1 , wherein the two or more tracking techniques comprise two or more of:

tracking real-world movement of a reflective element located on the writing device;

utilizing an emitter associated with the writing device to track real-world movement;

tracking real-world movement utilizing an optical sensor located on the writing device;

utilizing an inertial measurement unit to track real-world movement of the writing device;

utilizing an object recognition technique to track real-world movement of the writing device; or

implementing a machine learning model to track real-world movement of the writing device.

3. The computer-implemented method of claim 1 , further comprising:

analyzing, by way of the augmented reality device, a plurality of real-world surfaces within a real-world environment to identify, from the plurality of real-world surfaces, the real-world surface as sufficiently large and sufficiently flat for composing; and

providing, within the augmented reality environment and in response to identifying the real-world surface as sufficiently large and sufficiently flat for composing, a visual indication that the real-world surface is acceptable for composing.

4. The computer-implemented method of claim 3 , further comprising:

identifying, from the plurality of real-world surfaces, one or more real-world surfaces that are unacceptable for composing; and

providing, within the augmented reality environment and in response to identifying the one or more real-world surfaces as unacceptable for composing, a visual indication that the one or more real-world surfaces are unacceptable for composing.

5. The computer-implemented method of claim 1 , further comprising:

determining environmental factors associated with tracking the writing device; and

utilizing the two or more tracking techniques based on the environmental factors.

6. The computer-implemented method of claim 5 , wherein determining the environmental factors comprises one or more of determining a reflectiveness of the real-world surface or detecting lighting conditions associated with the augmented reality environment.

7. The computer-implemented method of claim 1 , wherein rendering the augmented reality object comprises:

detecting, by way of a pressure sensor associated with the writing device, that the writing device is contacting the real-world surface; and

generating, within the augmented reality environment and in response to detecting that the writing device is contacting the real-world surface, a digital mark that appears to be drawn on the real-world surface and that follows the real-world movement of the writing device.

8. The computer-implemented method of claim 7 , further comprising:

detecting, by way of the pressure sensor associated with the writing device, a pressure force applied to the writing device;

detecting, by way of an inertial measurement unit associated with the writing device, a tilt of the writing device; and

adjusting a compositional attribute of the digital mark based on the pressure force applied to the writing device and the tilt of the writing device.

9. A system comprising:

at least one processor; and

a non-transitory computer readable medium comprising instructions that, when executed by the at least one processor, cause the system to:

determine an initial location of a writing device on a real-world surface by detecting a locator element associated with the writing device in response to receiving an indication of an initial contact of the writing device on the real-world surface;

detect real-world movement of the writing device relative to the initial location by:

determining that the real-world movement of the writing device is above a speed threshold;

in response to determining that the real-world movement is above the speed threshold, determining the real-world movement of the writing device based on a first combination of two or more tracking techniques; and

in response to determining that the real-world movement is at or below the speed threshold, determining the real-world movement of the writing device based on a second combination of two or more tracking techniques; and

render, within an augmented reality environment generated by way of an augmented reality device, an augmented reality object that appears to be drawn on the real-world surface based on the real-world movement of the writing device.

10. The system of claim 9 , wherein the two or more tracking techniques comprise two or more of:

tracking real-world movement of a reflective element located on the writing device;

utilizing an emitter associated with the writing device to track real-world movement;

tracking real-world movement utilizing an optical sensor located on the writing device;

utilizing an inertial measurement unit to track real-world movement of the writing device;

utilizing an object recognition technique to track real-world movement of the writing device; or

implementing a machine learning model to track real-world movement of the writing device.

11. The system of claim 9 , further comprising instructions that, when executed by the at least one processor, cause the system to analyze, by way of the augmented reality device, a real-world environment to identify at least one real-world surface that is sufficiently large and sufficiently flat for composing.

12. The system of claim 11 , further comprising instructions that, when executed by the at least one processor, cause the system to provide, within the augmented reality environment and in response to analyzing the at least one real-world surface, an indication that the at least one real-world surface is acceptable for composing.

13. The system of claim 12 , further comprising instructions that, when executed by the at least one processor, cause the system to:

identify, from a plurality of real-world surfaces, one or more real-world surfaces that are unacceptable for composing; and

provide, within the augmented reality environment and in response to identifying the one or more real-world surfaces as unacceptable for composing, a visual indication that the one or more real-world surfaces are unacceptable for composing.

14. The system of claim 9 , further comprising instructions that, when executed by the at least one process, cause the system to:

determine environmental factors associated with tracking the writing device; and

utilize the two or more tracking techniques based on the environmental factors.

15. The system of claim 9 , wherein the instructions cause the system to determine the initial location of the writing device by detecting, by way of a pressure sensor of the writing device, an initial contact of the writing device with the real-world surface.

16. The system of claim 9 , wherein the instructions cause the system to render the augmented reality object by:

detecting, by way of a pressure sensor associated with the writing device, that the writing device is contacting the real-world surface; and

generating, within the augmented reality environment and in response to detecting that the writing device is contacting the real-world surface, a digital mark that appears to be drawn on the real-world surface and that follows the real-world movement of the writing device.

17. A non-transitory computer readable medium comprising instructions that, when executed by at least one processor, cause a computing device to:

determine an initial location of a writing device on a real-world surface;

detect real-world movement of the writing device relative to the initial location by:

determining that the real-world movement of the writing device is above a speed threshold;

in response to determining that the real-world movement is above the speed threshold, determining the real-world movement of the writing device based on a first combination of two or more tracking techniques; and

in response to determining that the real-world movement is at or below the speed threshold, determining the real-world movement of the writing device based on a second combination of two or more tracking techniques; and

render, within an augmented reality environment generated by way of an augmented reality device, an augmented reality object that appears to be drawn on the real-world surface based on the real-world movement of the writing device.

18. The non-transitory computer readable medium of claim 17 , wherein the two or more tracking techniques comprise two or more of:

tracking real-world movement of a reflective element located on the writing device;

utilizing an emitter associated with the writing device to track real-world movement;

tracking real-world movement utilizing an optical sensor located on the writing device;

utilizing an inertial measurement unit to track real-world movement of the writing device;

utilizing an object recognition technique to track real-world movement of the writing device; or

implementing a machine learning model to track real-world movement of the writing device.

19. The non-transitory computer readable medium of claim 17 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:

analyze, by way of the augmented reality device, a plurality of real-world surfaces within a real-world environment to identify, from the plurality of real-world surfaces, the real-world surface as sufficiently large and sufficiently flat for composing; and

provide, within the augmented reality environment and in response to identifying the real-world surface as sufficiently large and sufficiently flat for composing, a visual indication that the real-world surface is acceptable for composing.

20. The non-transitory computer readable medium of claim 19 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:

identify, from the plurality of real-world surfaces, one or more real-world surfaces that are unacceptable for composing; and

provide, within the augmented reality environment and in response to identifying the one or more real-world surfaces as unacceptable for composing, a visual indication that the one or more real-world surfaces are unacceptable for composing.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 4, 2019
From: RHODES, TENELL; MILLER, GAVIN S.P.; AKSIT, DUYGU CEYLAN; ITO, DAICHI
To: ADOBE SYSTEMS INCORPORATED
Reel/Frame 048797/0934 →
CHANGE OF NAME Recorded Apr 4, 2019
From: ADOBE SYSTEMS INCORPORATED
To: ADOBE INC.
Reel/Frame 048805/0975 →
Continuity (2)
Continuation 15716450 · Sep 26, 2017
Related Publication 20190236845A1 · Aug 1, 2019