IP Library › Granted Patent US 9,046,925
Granted Patent B2
US 9,046,925 · App. 13/609,446 · Granted Jun 2, 2015

Method for using the GPU to create haptic friction maps

Inventors: Richard Schuckle (Austin, TX); Thomas Lanzoni (Cedar Park, TX)
Assignee: Dell Products L.P.
G06F3/016G06T15/005
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,046,925
App. No.
13/609,446
Filed
Sep 11, 2012
Granted
Jun 2, 2015
Kind
B2
Art Unit
2616
USPC
345/419
Abstract

A haptics rendering system comprises: a display device having a display screen and one or more haptic feedback mechanisms associated with at least one tactile output peripheral; and a graphics processing unit (GPU) communicatively coupled to the display device. The GPU: receives one or more portions of display image software code for a three dimensional display image; generates a three dimensional (3D) visual output of the display image for display on the display device; and concurrently generates one or more specific friction maps to provide haptic feedback of depth and/or texture associated with specific portions of the display image that comprise at least one portion having at least one of a different depth and a different texture than other portions of the display image. The GPU renders the display image and the friction map from a same set of 3D commands within the display image software code.

Claims (74)

1. An information handling system comprising:

a processor capable of executing software code;

a display device having a display screen;

one or more haptic feedback mechanisms associated with at least one tactile response peripheral and which generate one or more sensations associated with a displayed image; and

a graphics processing unit (GPU) communicatively coupled to the processor and to the display device and which:

receives one or more portions of the software code corresponding to the display image;

generates one of a two dimensional ( 2 D) and a three dimensional ( 3 D) visual output of the display image for display on the display device, wherein the three dimensional visual output contains a height, a width and a depth dimension relative to a 2 D plane; and

concurrently generates one or more specific friction maps to provide haptic feedback associated with specific portions of the displayed image that comprise at least one portion having at least one of a different depth and a different texture than other portions of the displayed image, wherein the friction maps are created using a z-level function by which an amount of friction is correlated to a closeness on a depth-plane of a surface of an object within the display image relative to a background plane of the display image;

a haptic subsystem for generating a haptic response within at least one of the tactile output peripherals, utilizing haptic response mechanisms associated with the particular tactile output peripherals; and

a haptic friction buffer coupled between the GPU and the haptic subsystem and which temporarily holds friction maps generated by the GPU;

wherein the GPU passes the generated friction map to the haptic friction buffer; and

wherein the haptic subsystem:

processes the generated friction map within the haptic friction buffer; and

in response to detecting at least one of a presence, a contact and a functional interfacing of an interface tool with a surface of the display, applies a haptic response to represent edges and surfaces of objects visually represented below the interface tool that are at a higher depth dimension than the background plane of the display image.

2. The information handling system of claim 1 , wherein the GPU renders the displayed image and the friction map from a same set of 3D commands within display image software code received from one or more of the processor and a display data input source.

3. The information handling system of claim 1 , wherein the haptic subsystem further:

changes a level of haptic feedback based on a relative depth dimension of each point of an object over which the interface tool passes, wherein a higher object on a z-plane triggers a greater haptic feedback than a lower object on the z-plane, wherein objects that appear closer in the displayed image are provided with a higher level of resistance to a detected contact of the screen interface stylus at that location of the display screen, wherein when the haptic feedback includes vibrations, the haptic subsystem changes an intensity of the vibration based on at least one of (a) a relative location of the surface within the z-plane and (b) a texture of the surface.

4. The information handling system of claim 1 , wherein the GPU:

generates the haptic map with depth/height dimension parameter values, which trigger a higher level of haptic feedback at one or more points of the display in response to detection of a presence, contact, or functional interfacing of an interface tool at a location of the display screen at which a surface of an object within the displayed image is visibly closer in a depth-plane than a background of the displayed image, wherein a level of haptic feedback, including an increase in vibration intensity, correlates to a height of the surface of the object relative to the background.

5. The information handling system of claim 1 , wherein the GPU further:

provides within the haptic map one or more parameter values that represent different textures, wherein the different textures are applied to one or more of a surface of an object within a window and an entire window to enable an interface tool to detect an applied texture across an entire surface of the object and the window.

6. An information handling system comprising:

a processor capable of executing software code;

a display device having a display screen;

one or more haptic feedback mechanisms associated with at least one tactile response peripheral and which generate one or more sensations associated with a displayed image; and

a graphics processing unit (GPU) communicatively coupled to the processor and to the display device and which:

receives one or more portions of the software code corresponding to the displayed image;

generates one of a two dimensional (2D) and a three dimensional (3D) visual output of the displayed image for display on the display device, wherein the three dimensional visual output contains a height, a width and a depth dimension relative to a 2D plane;

concurrently generates one or more specific friction maps to provide haptic feedback associated with specific portions of the display image that comprise at least one portion having at least one of a different depth and a different texture than other portions of the displayed image;

receives, within display image software code, an indication of specific friction textures to assign to one or more rendered objects; and

generates the haptic map, based on the received software code indications, to include one or more parameter value that represent the specific friction textures to apply to a surface of each of the one or more rendered objects to enable an interface tool to detect the specific friction texture of each of the one or more rendered objects;

wherein the GPU renders the display, places the display in a normal frame buffer for display, and places one of the display and the haptic map correlated to the display into a texture buffer, whose contents are passed to the haptics subsystem for mapping onto one or more of the tactile output peripherals.

7. A method performed within an information handling system having a display device with a display screen and one or more haptic feedback mechanisms associated with at least one tactile response peripheral and which generates one or more sensations associated with a display, the method comprising:

a graphics processing unit (GPU) receiving one or more portions of software code corresponding to a display;

generating one of a two dimensional (2D) and a three dimensional (3D) visual output of the display for display on the display device, wherein the three dimensional visual output contains a height, a width, and a depth dimension relative to a 2D plane; and

concurrently generating one or more specific friction maps to provide haptic feedback associated with specific portions of the display that comprise at least one portion having at least one of a different depth and a different texture than other portions of the display, wherein the generating includes creating the friction map utilizing a z-level function, wherein an amount of friction is correlated to a closeness on a depth-plane of a surface of an object within the displayed image relative to a background plane of the displayed image;

wherein the information handling system further comprises: a haptic subsystem for generating a haptic response within at least one of the tactile output peripherals, utilizing haptic response mechanisms associated with the specific tactile output peripherals; and a haptic friction buffer coupled between the GPU and the haptic subsystem and which temporarily holds friction maps generated by the GPU; and

passing the generated friction map to the haptic friction buffer to trigger the haptic subsystem to: process the generated friction map within the haptic friction buffer; and in response to detecting at least one of a presence of, a contact, and a functional interfacing by an interface tool with a surface of the display, apply a haptic feedback to represent edges and surfaces of objects visually represented below the screen interface stylus that are at a higher depth dimension than the background plane of the displayed image.

8. The method of claim 7 , further comprising rendering the displayed image and the friction map from a same set of 3D commands received from one or more of a processor of the information handling system and a display data input source.

9. The method of claim 7 , further comprising:

changing, via the haptic subsystem, a level of haptic feedback based on a relative depth dimension of each point of an object over which the screen interface stylus passes, wherein a higher object on a z-plane triggers a greater haptic feedback than a lower object on the z-plane, wherein objects that appear closer in the displayed image are provided with a higher level of resistance to a detected contact of the interface tool at that location of the display screen;

wherein when the haptic feedback includes vibrations, the method comprises changing an intensity of the vibration based on at least one of (a) a relative location of the surface within the z-plane and (b) a texture of the surface.

10. The method of claim 7 , further comprising:

generating the haptic map with depth/height dimension parameter values, which trigger a higher level of haptic feedback at one or more points of the display in response to detection of a presence, contact, or functional interfacing of an interface tool at a location of the display screen at which a surface of an object within the displayed image is visibly closer in a depth-plane than a background of the displayed image, wherein an amount of increase in the feedback intensity correlates to a height of the surface of the object relative to the background.

11. The method of claim 7 , further comprising:

providing within the haptic map one or more parameter values that represent different textures, wherein the different textures are applied to one or more of a surface of an object within a window and an entire window to enable an interface tool to detect an applied texture across an entire surface of the object and the window.

12. A method performed within an information handling system having a display device with a display screen and one or more haptic feedback mechanisms associated with at least one tactile response peripheral and which generates one or more sensations associated with a displayed image, the method comprising:

a graphics processing unit (GPU) receiving one or more portions of software code corresponding to a displayed image;

generating one of a two dimensional (2D) and a three dimensional (3D) visual output of the displayed image for display on the display device, wherein the three dimensional visual output contains a height, a width, and a depth dimension relative to a 2D plane;

concurrently generating one or more specific friction maps to provide haptic feedback associated with specific portions of the displayed image that comprise at least one portion having at least one of a different depth and a different texture than other portions of the displayed image;

receiving, within display image software code, an indication of specific friction textures to assign to one or more rendered objects; and

generating the haptic map, based on the received software code indications, to include one or more parameter values that represent the specific friction textures to apply to a surface of each of the one or more rendered objects to enable an interface tool to detect the specific friction texture of each of the one or more rendered objects;

wherein the GPU renders the displayed, places the displayed image in a normal frame buffer for display, and places one of the displayed image and the haptic map correlated to the displayed image into a texture buffer, whose contents are passed to the haptics subsystem for mapping onto one or more of the tactile output peripherals.

13. A haptics rendering system comprising:

a touch screen display device having a display screen and one or more haptic feedback mechanisms disposed proximate to the display screen for generating one or more sensations associated with a displayed image; and

a graphics processing unit (GPU) communicatively coupled to the display device and which:

receives one or more portions of the display image software code of a three dimensional display image;

generates a three dimensional (3D) visual output of the displayed image for display on the display device, wherein the three dimensional visual output contains a height, a width and a depth dimension; and

concurrently generates one or more specific friction maps to provide haptic feedback associated with specific portions of the displayed image that comprise at least one portion having at least one of a different depth and a different texture than other portions of the displayed image;

wherein the GPU renders the displayed image and the friction map from a same set of 3D commands within the display image software code;

wherein the GPU generates the haptic map with depth/height dimension parameter values, which trigger an increase in a feedback intensity at one or more points of the display image in response to detection of a presence, contact, or functional interfacing of an interface tool at a location of the display screen at which a surface of an object within the displayed image is visibly closer in a depth-plane than a background of the displayed image, wherein an amount of increase in the feedback intensity correlates to a height of the surface of the object relative to the background; and

wherein the haptic subsystem further changes an intensity of the haptic response based on a relative depth dimension of each point of an object over which the interface tool passes, wherein a higher object on a z-plane triggers a greater haptic response than a lower object on the z-plane, wherein objects that appear closer in the displayed image are provided with a higher level of resistance to a detected presence, contact, or functional interfacing of the interface tool at that location of the display.

14. The haptics rendering system of claim 13 , wherein:

the GPU further creates the friction map utilizing a z-level function, wherein an amount of friction is correlated to a closeness on a depth-plane of a surface of an object within the displayed image relative to a background plane of the displayed image; and

the system further comprises:

a haptic subsystem for generating a haptic response within at least one tactile output peripheral, utilizing haptic response mechanisms associated with the tactile output peripheral;

a haptic friction buffer coupled between the GPU and the haptic subsystem and which temporarily holds friction maps generated by the GPU;

wherein the GPU passes the generated friction map to the haptic friction buffer; and

wherein the haptic subsystem: processes the generated friction map within the haptic friction buffer; and in response to detecting a presence, contact, or functional interfacing of an interface tool across a surface of the display, applies a haptic response to represent edges and surfaces of objects visually represented below the interface tool that are at a higher depth dimension than the background plane of the displayed image.

15. The haptics rendering system of claim 13 , wherein the GPU further:

receives, within display image software code, an indication of specific friction textures to assign to one or more rendered objects;

generates the haptic map, based on the received software code indications, to include one or more parameter values that represent the specific friction textures to apply to a surface of each of the one or more rendered objects to enable an interface tool to detect the specific friction texture of each of the one or more rendered objects; and

provides within the haptic map one or more parameter values that represent different textures, wherein the different textures are applied to one or more of a surface of an object within a window and an entire window to enable the interface tool to detect an applied texture across an entire surface of the object and the window;

wherein the GPU renders the displayed image, places the displayed image in a normal frame buffer for display, and places one of the display and the haptic map correlated to the displayed image into a texture buffer, whose contents are passed to the haptics subsystem for mapping onto a tactile output device.

Assignments (15)
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (053546/0001) Recorded Jun 23, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.); DELL INTERNATIONAL L.L.C.; DELL PRODUCTS L.P.; DELL USA L.P.; EMC CORPORATION; DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.); EMC IP HOLDING COMPANY LLC
Reel/Frame 071642/0001 →
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001) Recorded May 20, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.); DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.); DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL PRODUCTS L.P.; DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.); EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC); EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.); SCALEIO LLC
Reel/Frame 061753/0001 →
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001) Recorded Apr 26, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.); DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.); DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL PRODUCTS L.P.; DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.); EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC); EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.); SCALEIO LLC
Reel/Frame 061324/0001 →
RELEASE OF SECURITY INTEREST Recorded Nov 3, 2021
From: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH
To: ASAP SOFTWARE EXPRESS, INC.; AVENTAIL LLC; CREDANT TECHNOLOGIES, INC.; DELL USA L.P.; DELL INTERNATIONAL, L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL SOFTWARE INC.; DELL SYSTEMS CORPORATION; EMC CORPORATION; EMC IP HOLDING COMPANY LLC; FORCE10 NETWORKS, INC.; MAGINATICS LLC; MOZY, INC.; SCALEIO LLC; WYSE TECHNOLOGY L.L.C.
Reel/Frame 058216/0001 →
SECURITY AGREEMENT Recorded Apr 22, 2020
From: CREDANT TECHNOLOGIES INC.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL USA L.P.; EMC CORPORATION; FORCE10 NETWORKS, INC.; WYSE TECHNOLOGY L.L.C.; EMC IP HOLDING COMPANY LLC
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A.
Reel/Frame 053546/0001 →
SECURITY AGREEMENT Recorded Mar 21, 2019
From: CREDANT TECHNOLOGIES, INC.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL USA L.P.; EMC CORPORATION; FORCE10 NETWORKS, INC.; WYSE TECHNOLOGY L.L.C.; EMC IP HOLDING COMPANY LLC
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A.
Reel/Frame 049452/0223 →
SECURITY AGREEMENT Recorded Sep 21, 2016
From: ASAP SOFTWARE EXPRESS, INC.; AVENTAIL LLC; CREDANT TECHNOLOGIES, INC.; DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL SOFTWARE INC.; DELL SYSTEMS CORPORATION; EMC CORPORATION; EMC IP HOLDING COMPANY LLC; FORCE10 NETWORKS, INC.; MAGINATICS LLC; MOZY, INC.; SCALEIO LLC; SPANNING CLOUD APPS LLC; WYSE TECHNOLOGY L.L.C.
To: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, AS COLLATERAL AGENT
Reel/Frame 040134/0001 →
SECURITY AGREEMENT Recorded Sep 21, 2016
From: ASAP SOFTWARE EXPRESS, INC.; AVENTAIL LLC; CREDANT TECHNOLOGIES, INC.; DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL SOFTWARE INC.; DELL SYSTEMS CORPORATION; EMC CORPORATION; EMC IP HOLDING COMPANY LLC; FORCE10 NETWORKS, INC.; MAGINATICS LLC; MOZY, INC.; SCALEIO LLC; SPANNING CLOUD APPS LLC; WYSE TECHNOLOGY L.L.C.
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
Reel/Frame 040136/0001 →
RELEASE OF SECURITY INTEREST Recorded Sep 14, 2016
From: BANK OF AMERICA, N.A., AS COLLATERAL AGENT
To: DELL MARKETING L.P.; ASAP SOFTWARE EXPRESS, INC.; APPASSURE SOFTWARE, INC.; COMPELLENT TECHNOLOGIES, INC.; CREDANT TECHNOLOGIES, INC.; DELL INC.; DELL PRODUCTS L.P.; DELL USA L.P.; DELL SOFTWARE INC.; FORCE10 NETWORKS, INC.; PEROT SYSTEMS CORPORATION; SECUREWORKS, INC.; WYSE TECHNOLOGY L.L.C.
Reel/Frame 040040/0001 →
RELEASE OF SECURITY INTEREST Recorded Sep 14, 2016
From: BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS COLLATERAL AGENT
To: DELL MARKETING L.P.; ASAP SOFTWARE EXPRESS, INC.; APPASSURE SOFTWARE, INC.; COMPELLENT TECHNOLOGIES, INC.; CREDANT TECHNOLOGIES, INC.; DELL INC.; DELL PRODUCTS L.P.; DELL USA L.P.; DELL SOFTWARE INC.; FORCE10 NETWORKS, INC.; PEROT SYSTEMS CORPORATION; SECUREWORKS, INC.; WYSE TECHNOLOGY L.L.C.
Reel/Frame 040065/0618 →
RELEASE OF SECURITY INTEREST Recorded Sep 13, 2016
From: BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT
To: DELL MARKETING L.P.; ASAP SOFTWARE EXPRESS, INC.; APPASSURE SOFTWARE, INC.; COMPELLANT TECHNOLOGIES, INC.; CREDANT TECHNOLOGIES, INC.; DELL INC.; DELL PRODUCTS L.P.; DELL USA L.P.; DELL SOFTWARE INC.; FORCE10 NETWORKS, INC.; PEROT SYSTEMS CORPORATION; SECUREWORKS, INC.; WYSE TECHNOLOGY L.L.C.
Reel/Frame 040065/0216 →
PATENT SECURITY AGREEMENT (ABL) Recorded Jan 2, 2014
From: DELL INC.; APPASSURE SOFTWARE, INC.; ASAP SOFTWARE EXPRESS, INC.; BOOMI, INC.; COMPELLENT TECHNOLOGIES, INC.; CREDANT TECHNOLOGIES, INC.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL SOFTWARE INC.; DELL USA L.P.; FORCE10 NETWORKS, INC.; GALE TECHNOLOGIES, INC.; PEROT SYSTEMS CORPORATION; SECUREWORKS, INC.; WYSE TECHNOLOGY L.L.C.
To: BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 031898/0001 →
PATENT SECURITY AGREEMENT (TERM LOAN) Recorded Jan 2, 2014
From: DELL INC.; APPASSURE SOFTWARE, INC.; ASAP SOFTWARE EXPRESS, INC.; BOOMI, INC.; COMPELLENT TECHNOLOGIES, INC.; CREDANT TECHNOLOGIES, INC.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL SOFTWARE INC.; DELL USA L.P.; FORCE10 NETWORKS, INC.; GALE TECHNOLOGIES, INC.; PEROT SYSTEMS CORPORATION; SECUREWORKS, INC.; WYSE TECHNOLOGY L.L.C.
To: BANK OF AMERICA, N.A., AS COLLATERAL AGENT
Reel/Frame 031899/0261 →
PATENT SECURITY AGREEMENT (NOTES) Recorded Jan 2, 2014
From: APPASSURE SOFTWARE, INC.; ASAP SOFTWARE EXPRESS, INC.; BOOMI, INC.; COMPELLENT TECHNOLOGIES, INC.; CREDANT TECHNOLOGIES, INC.; DELL INC.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL SOFTWARE INC.; DELL USA L.P.; FORCE10 NETWORKS, INC.; GALE TECHNOLOGIES, INC.; PEROT SYSTEMS CORPORATION; SECUREWORKS, INC.; WYSE TECHNOLOGY L.L.C.
To: BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS FIRST LIEN COLLATERAL AGENT
Reel/Frame 031897/0348 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 5, 2013
From: SCHUCKLE, RICHARD; LANZONI, THOMAS
To: DELL PRODUCTS L.P.
Reel/Frame 031147/0649 →
Continuity (1)
Related Publication 20140071117A1 · Mar 13, 2014