IP Library Granted Patent US 10,033,943
Granted Patent B1
US 10,033,943 · App. 14/965,776 · Granted Jul 24, 2018

Activity surface detection, display and enhancement

Inventors: Pramod Kumar Sharma (San Jose, CA); Jerome Scholler (San Francisco, CA); Conrad Verkler (East Palo Alto, CA)
Assignee: Tangible Play, Inc.
H04N5/272G06T1/20G06T11/60H04N5/142H04N5/23293H04N5/2628H04N7/183
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,033,943
App. No.
14/965,776
Granted
Jul 24, 2018
Kind
B1
Abstract

Activity surface detection, display, and enhancement implementations are described. In an example implementation, a method determines, using a processor of a computing device, a traceable image and presenting the traceable image in an interface on a display of the computing device; captures, using a video capture device coupled to the computing device, a video stream of a physical activity surface proximate to the computing device; and displays, on the display of the computing device, the captured video stream overlaid with the traceable image in the interface.

Claims (61)

1. A computer-implemented method for monitoring user activity in a physical activity scene, comprising:

determining, using a processor of a computing device, a traceable image and presenting the traceable image in an interface on a display of the computing device;

capturing, using a video capture device coupled to the computing device, a video stream of a physical activity surface proximate to the computing device;

displaying, on the display of the computing device, the captured video stream overlaid with the traceable image in the interface;

processing, using the processor of the computing device, the video stream captured by the video capture device to detect a contour of a work being created by a user on the physical activity surface;

comparing, using the processor of the computing device, the contour of the work with a corresponding portion of the traceable image;

determining, using the processor of the computing device, an accuracy of the contour relative to the corresponding portion of the traceable image based on the comparison;

determining, using the processor of the computing device, a progress of the user in creating a work on the physical activity surface based on the accuracy; and

updating, using the processor of the computing device, the interface to include the progress of the user.

2. The method of claim 1 , further comprising:

determining, using the processor of the computing device, directive feedback based on the comparison between the contour of the work and the corresponding portion of the traceable image; and

updating, using the processor of the computing device, the interface to include the directive feedback.

3. The method of claim 2 , wherein the comparison indicates a difference between the contour of the work and the corresponding portion of the traceable image, and the directive feedback highlights the difference.

4. The method of claim 1 further comprising:

detecting, using the processor of the computing device, a new contour on the physical activity surface;

generating, using the processor of the computing device, a visualization of the new contour; and

updating the interface with the visualization of the new contour overlaid on a corresponding portion of the traceable image.

5. The method of claim 1 , wherein displaying the captured video stream overlaid with the traceable image includes displaying an image frame from the captured video stream overlaid with the traceable image.

6. The method of claim 1 , wherein the traceable image is semi-transparent.

7. The method of claim 1 wherein generating a traceable image further comprises:

receiving, using the processor of the computing device, a captured image;

detecting, using the processor of the computing device, contours in the captured image; and

generating, using the processor of the computing device, the traceable image based on the contours in the captured image.

8. The method of claim 1 , further comprising:

detecting, using the processor of the computing device, one or more of a resize instruction and a reposition instruction;

modifying, using the processor of the computing device, the traceable image based on the one or more of the resize instruction and the reposition instruction; and

updating the interface to display the modified traceable image.

9. A computer-implemented method comprising:

determining, using a processor of a computing device, a traceable image and presenting the traceable image in an interface on a display of the computing device;

capturing, using a video capture device coupled to the computing device, a video stream of a physical activity surface proximate to the computing device;

displaying, on the display of the computing device, the captured video stream overlaid with the traceable image in the interface, the video stream depicting a work being created by a user on the physical activity surface;

determining, using the processor of the computing device, an accuracy of the work relative to the traceable image;

determining, using the processor of the computing device, a progress of the user in creating the work on the physical activity surface based on the accuracy; and

updating, using the processor of the computing device, the interface to include the progress of the user.

10. A computer-implemented method of claim 9 , further comprising:

processing, using a processor of a computing device, the video stream to detect a change in the work being created by the user in the physical activity surface;

tracking, using the computing device, the progress of the work in comparison to the traceable image depicted in the interface displayed on the computing device, by evaluating which portion of the traceable image the change in the work relates to; and

updating, using a processor of a computing device, the interface displayed on the computing device with a visual indication of the tracked progress.

11. The method of claim 9 , further comprising:

generating, using a processor of a computing device, the traceable image by detecting contour lines of an image and generating the traceable image based on the contour lines.

12. The method of claim 10 , wherein detecting the change in the work includes detecting, using the processor of the computing device, a change in a contour.

13. The method of claim 9 , wherein the computing device is a tablet computer, a smartphone, a laptop computer, or a desktop computer.

14. The method of claim 10 , wherein the visual indication of the tracked progress includes generating corresponding virtual contours based on the change in the contour and rendering the corresponding virtual contours overlaid on the traceable image displayed in the interface.

15. The method of claim 9 , further comprising:

processing, using the processor of the computing device, the video stream captured by the video capture device to detect a contour of a work being created by a user on the physical activity surface;

comparing, using the processor of the computing device, the contour of the work with a corresponding portion of the traceable image;

determining, using the processor of the computing device, directive feedback based on the comparison between the contour of the work and the corresponding portion of the traceable image; and

updating, using the processor of the computing device, the interface to include the directive feedback.

16. The method of claim 15 , wherein the comparison indicates a difference between the contour of the work and the corresponding portion of the traceable image, and the directive feedback highlights the difference.

17. The method of claim 9 , further comprising:

processing, using the processor of the computing device, the video stream to detect an object in the physical activity surface;

determining, using the processor of the computing device, a relative position of the object in relation to the traceable image displayed on the interface; and

displaying on the interface, a visualization of the object in relation to the traceable image.

18. The method of claim 9 , wherein the physical activity surface is located in front of the display of the computing device or behind the display of the computing device.

19. A physical activity surface visualization system, comprising:

a detection engine coupled via a communications bus to a video capture device configured to capture a video stream of a physical activity surface proximate to a computing device, the video stream depicting a work being created by a user on the physical activity surface, the detection engine further configured to process the video stream to detect a change in the work being created by a user in the physical activity surface; and

an activity application executable by a processor of the computing device and configured to display an instructional image in an interface on a display of the computing device, the activity application further configured to determine an accuracy of the work relative to the instructional image, determine a progress of the work based on the accuracy, the interface displaying a visual indicator of the tracked progress and the instructional image.

20. The physical activity surface visualization system of claim 19 , wherein the computing device is a tablet computer, a smartphone, a laptop computer, or a desktop computer.

21. The physical activity surface visualization system of claim 19 , further comprising:

a detachable camera adapter coupled to adapt the video capture device, the detachable camera adapter configured to alter a field of view of the video capture device to include the physical activity surface.

22. The physical activity surface visualization system of claim 21 , wherein the detachable camera adapter includes one or more mirrors for redirecting the field of view of the video capture device to the physical activity surface.

Assignments (5)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 30, 2026
From: TANGIBLE PLAY, INC.
To: PLAY OSMO INC.
Reel/Frame 074224/0665 →
RELEASE OF SECURITY INTEREST Recorded May 30, 2025
From: GLAS TRUST COMPANY LLC
To: CLAUDIA Z. SPRINGER, AS CHAPTER 11 TRUSTEE OF TANGIBLE PLAY, INC., ON BEHALF OF THE ESTATE OF DEBTOR TANGIBLE PLAY, INC.
Reel/Frame 071473/0897 →
SECURITY INTEREST Recorded Nov 1, 2024
From: CLAUDIA Z. SPRINGER, AS CHAPTER 11 TRUSTEE OF EPIC! CREATIONS, INC., ON BEHALF OF THE ESTATE OF DEBTOR EPIC! CREATIONS, INC.; CLAUDIA Z. SPRINGER, AS CHAPTER 11 TRUSTEE OF NEURON FUEL, INC., ON BEHALF OF THE ESTATE OF DEBTOR NEURON FUEL, INC.; CLAUDIA Z. SPRINGER, AS CHAPTER 11 TRUSTEE OF TANGIBLE PLAY, INC., ON BEHALF OF THE ESTATE OF DEBTOR TANGIBLE PLAY, INC.
To: GLAS TRUST COMPANY LLC
Reel/Frame 069290/0456 →
SECURITY INTEREST Recorded May 16, 2022
From: TANGIBLE PLAY, INC.
To: GLAS TRUST COMPANY LLC
Reel/Frame 060257/0811 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 26, 2016
From: SHARMA, PRAMOD KUMAR; SCHOLLER, JEROME; VERKLER, CONRAD
To: TANGIBLE PLAY, INC.
Reel/Frame 037589/0386 →
Continuity (4)
Continuation In Part 14842777 · Sep 1, 2015
Continuation 13928111 · Jun 26, 2013
Provisional Application 62090345 · Dec 10, 2014
Provisional Application 61714138 · Oct 15, 2012
Cited By (1)
US 12,400,375