IP Library › Granted Patent US 12,456,272
Granted Patent B2
US 12,456,272 · App. 18/066,745 · Granted Oct 28, 2025

Object modification in a smart environment

Inventors: Jennifer M. Hatfield (San Francisco, CA); Jeremy R. Fox (Georgetown, TX); Tushar Agrawal (West Fargo, ND); Sarbajit K. Rakshit (Kolkata, IN)
Assignee: International Business Machines Corporation
G06T19/20G06T19/003G06T19/006
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,456,272
App. No.
18/066,745
Granted
Oct 28, 2025
Kind
B2
Abstract

A processor may receive an object dataset based on an object. The processor may generate an interactive model of the object to display to a user using the object dataset. The processor may receive an interaction dataset associated with the user interacting with the interactive model. The processor may analyze the interactive data to identify one or more modifications to the one or more parameters of the object, based on the user interacting with the interactive model.

Claims (59)

1. A computer-implemented method for comprising:

receiving, by a processor, an object dataset associated with an object;

generating, using a Generative Adversarial Network (GAN), an interactive model of the object using the object dataset, wherein the interactive model is a 3 dimensional (3D) interactive model that is displayed to a user via a Virtual Reality (VR) headset;

receiving, at least partially from a wearable device tracking hand motion, an interaction dataset associated with the user interacting with the interactive model, wherein the interaction dataset includes information related to a directional force of the hand motion on the interactive model of the object;

analyzing the interaction dataset to identify one or more modifications to the object, based on the user interacting with the interactive model;

generating a modified design plan of the object based on the one or more modifications the user made to the interactive model;

analyzing the modified design plan of the object for a level of object integrity, wherein the level of object integrity relates to being producible;

determining the level of object integrity does not exceed a minimum threshold level;

analyzing the modified design plan and the object dataset to determine one or more recommended modifications; and

issuing the one or more recommended modifications to the user in a notification, wherein the one or more recommended modifications changes the modified design plan to make it producible.

2. The computer-implemented method of claim 1 , the method further comprising:

updating the interactive model based, at least in part, on the one or more modifications, wherein an updated interactive model is formed by updating the interactive model.

3. The computer-implemented method of claim 1 , the method further comprising:

producing the object based on the modified design plan with the one or more recommended modifications.

4. The computer-implemented method of claim 1 , the method further comprising:

analyzing a user profile associated with the user, wherein the user profile includes one or more user preferences; and

automatically generating one or more object options to the user based, at least in part on, the one or more user preferences and the object dataset.

5. The computer-implemented method of claim 1 , wherein the interaction dataset further includes a hand speed as the user interacts with the interactive model of the object.

6. The computer-implemented method of claim 1 , wherein the interaction dataset is received, at least partially, from an ultrasound scanning device.

7. A system comprising:

a memory; and

a processor in communication with the memory, the processor being configured to perform operations comprising:

receiving an object dataset associated with an object;

generating, using a Generative Adversarial Network (GAN), an interactive model of the object using the object dataset, wherein the interactive model is a 3 dimensional (3D) interactive model that is displayed to a user via a Virtual Reality (VR) headset;

receiving, at least partially from a wearable device tracking hand motion, an interaction dataset associated with the user interacting with the interactive model, wherein the interaction dataset includes information related to a directional force of the hand motion on the interactive model of the object;

analyzing the interaction dataset to identify one or more modifications to the object, based on the user interacting with the interactive model;

generating a modified design plan of the object based on the one or more modifications the user made to the interactive model;

analyzing the modified design plan of the object for a level of object integrity, wherein the level of object integrity relates to being producible;

determining the level of object integrity does not exceed a minimum threshold level;

analyzing the modified design plan and the object dataset to determine one or more recommended modifications; and

issuing the one or more recommended modifications to the user in a notification, wherein the one or more recommended modifications changes the modified design plan to make it producible.

8. The system of claim 7 , the processor being configured to perform operations further comprising:

updating the interactive model based, at least in part, on the one or more modifications, wherein an updated interactive model is formed by updating the interactive model.

9. The system of claim 7 , the processor being configured to perform operations further comprising:

producing the object based on the modified design plan with the one or more recommended modifications.

10. The system of claim 7 , the processor being configured to perform operations further comprising:

analyzing a user profile associated with the user, wherein the user profile includes one or more user preferences; and

automatically generating one or more object options to the user based, at least in part on, the one or more user preferences and the object dataset.

11. The system of claim 7 , wherein the interaction dataset further includes a hand speed as the user interacts with the interactive model of the object.

12. The system of claim 7 , wherein the interaction dataset is received, at least partially, from an ultrasound scanning device.

13. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform operations, the operations comprising:

receiving an object dataset associated with an object;

generating, using a Generative Adversarial Network (GAN), an interactive model of the object using the object dataset, wherein the interactive model is a 3 dimensional (3D) interactive model that is displayed to a user via a Virtual Reality (VR) headset;

receiving, at least partially from a wearable device tracking hand motion, an interaction dataset associated with the user interacting with the interactive model, wherein the interaction dataset includes information related to a directional force of the hand motion on the interactive model of the object;

analyzing the interaction dataset to identify one or more modifications to the object, based on the user interacting with the interactive model;

generating a modified design plan of the object based on the one or more modifications the user made to the interactive model;

analyzing the modified design plan of the object for a level of object integrity, wherein the level of object integrity relates to being producible;

determining the level of object integrity does not exceed a minimum threshold level;

analyzing the modified design plan and the object dataset to determine one or more recommended modifications; and

issuing the one or more recommended modifications to the user in a notification, wherein the one or more recommended modifications changes the modified design plan to make it producible.

14. The computer program product of claim 13 , the program instructions executable by the processor to cause the processor to perform operations further comprising:

producing the object based on the modified design plan with the one or more recommended modifications.

15. The computer program product of claim 13 , the program instructions executable by the processor to cause the processor to perform operations further comprising:

updating the interactive model based, at least in part, on the one or more modifications, wherein an updated interactive model is formed by updating the interactive model.

16. The computer program product of claim 13 , the program instructions executable by the processor to cause the processor to perform operations further comprising:

analyzing a user profile associated with the user, wherein the user profile includes one or more user preferences; and

automatically generating one or more object options to the user based, at least in part on, the one or more user preferences and the object dataset.

17. The computer program product of claim 13 , wherein the interaction dataset further includes a hand speed as the user interacts with the interactive model of the object.

18. The computer program product of claim 13 , wherein the interaction dataset is received, at least partially, from an ultrasound scanning device.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 15, 2022
From: HATFIELD, JENNIFER M.; FOX, JEREMY R.; AGRAWAL, TUSHAR; RAKSHIT, SARBAJIT K.
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 062109/0227 →
Continuity (1)
Related Publication 20240203076A1 · Jun 20, 2024
References Cited (32)
US 6768486B1 · Szabo · 2004 [cited by examiner]
US 9911243B2 · Vats · 2018 [cited by applicant]
US 10254941B2 · Bowen · 2019 [cited by applicant]
US 10534809B2 · Wiesel · 2020 [cited by applicant]
US 10817921B2 · Collier · 2020 [cited by applicant]
US 20180059644A1 · Lection · 2018 [cited by applicant]
US 20180284692A1 · Kline · 2018 [cited by applicant]
US 20200004225A1 · Buller · 2020 [cited by examiner]
US 20220032548A1 · Goyal · 2022 [cited by applicant]
US 20220245903A1 · Saxena · 2022 [cited by examiner]
US 20220255995A1 · Berliner · 2022 [cited by examiner]
Amir, et al., “A Low Power, Fully Event-Based Gesture Recognition System”, CVPR 2017, Nov. 6, 2017, pp. 7243-7252, <https://research.ibm.com/interactive/dvsgesture/>. [cited by applicant]
Anonymous, “A Method and System for Enabling 3D Printing with Feedback and Modification in VR”, An IP.com Prior Art Database Technical Disclosure, IP.com No. IPCOM000253627D, IP.com Electronic Publication Date: Apr. 17,… [cited by applicant]
Baron, Jessica, “How Digital Twins Will Revolutionize 3D Manufacturing”, [Accessed May 27, 2022], Published Feb. 26, 2021, 10 pages, <https://all3dp.com/1/digital-twins-in-3d-manufacturing/>. [cited by applicant]
Brownlee, Jason, “A Tour of Generative Adversarial Network Models”, Machine Learning Mastery, Last Updated on Jul. 12, 2019, 22 pages, <https://machinelearningmastery.com/tour-of-generative adversarial-network-models/>. [cited by applicant]
IBM, “Computer Vision & Multimedia”, IBM Research blog, last updated May 3, 2018, 7 pages, <https://researcher.watson.ibm.com/researcher/view_group_subpage.php?id=9482>. [cited by applicant]
IBM, “IBM Maximo Application Suite: Remote monitoring”, retrieved from internet Sep. 30, 2022, 6 pages, <https://www.ibm.com/products/maximo/remote-monitoring>. [cited by applicant]
IBM, “IBM Sterling Configure, Price, Quote”, last retrieved from internet Sep. 30, 2022, 2 pages, <https://www.ibm.com/products/configure-price-quote>. [cited by applicant]
Kantaros, et al., “3D Printing and Implementation of Digital Twins: Current Trends and Limitations”, Applied System Innovation, vol. 5, Issue 1, Article 7, 13 pages, <https://www.mdpi.com/2571-5577/5/1/7>. [cited by applicant]
Karpathy, et al., “Generative Models”, OpenAI, Jun. 16, 2016, 12 pages, <https://openai.com/blog/generative-models/>. [cited by applicant]
Khan, Jamshed, “StyleGAN: Use machine learning to generate and customize realistic images”, Hearbeat, Jul. 31, 2019, 15 pages, <https://heartbeat.comet.ml/stylegans-use-machine-learning-to-generate-and-customize-realist… [cited by applicant]
Li, et al., “Image-Based 3D Shape Generation Used for 3D Printing”, In: Li, L., et al., (eds), Methods and Applications for Modeling and Simulation of Complex Systems, AsiaSim 2018, Communications in Computer and Inform… [cited by applicant]
Lin, et al., “Design of Hand Gestures for Manipulating Objects in Virtual Reality”, HCl 2017: Human-Computer Interaction. User Interface Design, Development and Multimodality, First Online: May 14, 2017, pp. 584-592, <h… [cited by applicant]
Oh, et al., “Deep Generative Design: Integration of Topology Optimization and Generative Models”, Computer Science, Machine Learning, Version 2, Last revised May 20, 2019, 22 pages, <https://arxiv.org/abs/1903.01548>. [cited by applicant]
Okutani et al., “Stereoscopic Capture in Projection Mapping”, in IEEE Access, vol. 6, Nov. 30, 2018, pp. 65894-65900, <https://ieeexplore.ieee.org/document/8491263>. [cited by applicant]
PACKT_PUB, “Inside the Generative Adversarial Networks (GAN) architecture”, Medium.com, Nov. 19, 2019, 6 pages, <https://medium.com/@Packt_Pub/inside-the-generative-adversarial-networks-gan-architecture-2435afbd6b3b>. [cited by applicant]
Pappu, et al., “Haptic interaction with holographic video images”, Proceedings of SPIE, vol. 3293, Practical Holography XII, Mar. 18, 1998, pp. 38-45. [cited by applicant]
Slambekova, et al., “Gaze and Gesture Based Object Manipulation in Virtual Worlds”, VRST '12: Proceedings of the 18th ACM symposium on Virtual Reality Software and Technology, Dec. 2012, pp. 203-204, <https://dl.acm.org… [cited by applicant]
Unkown, “Play with Generative Adversarial Networks in Your Browser!”, Polo Club, Github, last retrieved from internet Sep. 30, 2022, 8 pages, <https://poloclub.github.io/ganlab/>. [cited by applicant]
IBM “Predictive maintenance with IBM Maximo Application Suite”, Advance your journey to predictive maintenance, Apr. 9, 2025, 07 pages. [cited by applicant]
Venture Radar “Top holograms Companies”, Apr. 9, 2025, 06 pages, https://www.ventureradar.com/keyword/holograms. [cited by applicant]
No Author, “Watson Visual Recognition”, https://web.archive.org/web/20200718085354/https://www.ibm.com/dk-en/cloud/watson-visual-recognition, dated Jun. 2, 2025, 3 pages. [cited by applicant]