IP Library › Granted Patent US 12,455,761
Granted Patent B2
US 12,455,761 · App. 16/705,140 · Granted Oct 28, 2025

Techniques for workflow analysis and design task optimization

Inventors: Tovi Grossman (Toronto, CA); George Fitzmaurice (Toronto, CA); Benjamin Lafreniere (Toronto, CA); Juho Kim (Daejeon, KR); Minsuk Chang (Daejeon, KR); Kaveh Hassani (Toronto, CA)
Assignee: AUTODESK, INC.
G06F9/4881G06F16/9024G06F30/12G06N20/00G06Q10/0633
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,455,761
App. No.
16/705,140
Granted
Oct 28, 2025
Kind
B2
Abstract

A W-graph system comprising a server connected with a plurality of clients via a network. Each client/user performs a design task via a design application while the server collects timestamped event data. The server generates a plurality of W-graphs for a plurality of tasks based on the collected event data. Each W-graph comprises one or more representative workflows, each representative workflow comprising at least one merged node representing nodes from different workflows for different users performing the same task. A W-graph for a task selected by the user may be viewed in a W-graph GUI. A user may also select a W-suggest function to have a current workflow for a task analyzed for optimization based on a W-graph generated for the same task. A modified current workflow is generated that highlights user techniques in the current workflow that are less efficient than user techniques in the W-graph.

Claims (70)

1. A computer-implemented method for analyzing software application workflows, the method comprising:

receiving a selection of a first task that is performed via a design application;

executing, on a first node of a first user workflow for performing the first task and a second node of a first workflow graph that graphically represents a plurality of user workflows for performing the first task, a trained autoencoder to generate a first vector comprising a compact representation of three-dimensional (3D) data associated with the first node and a second vector comprising a compact representation of 3D data associated with the second node;

determining, via the trained autoencoder, that the first node and the second node are similar based on a comparison between the first vector and the second vector;

subsequent to determining that the first node and the second node are similar, determining that a first edge associated with the first node should be modified based on a comparison between a feature of the first edge and a similar feature of a second edge associated with the second node;

generating a modified first user workflow based on a modification to the first edge; and

causing the modified first user workflow to be displayed via a user interface.

2. The computer-implemented method of claim 1 , wherein:

the first task comprises modeling a first object; and

the modified first user workflow comprises a sequence of edges and nodes, each node in the modified first user workflow representing a state of the modeling of the first object.

3. The computer-implemented method of claim 1 , wherein:

the first workflow graph comprises a plurality of representative workflows including a primary representative workflow, the primary representative workflow having a lowest number of total invoked commands relative to all other representative workflows included in the plurality of representative workflows;

the first user workflow comprises a plurality of edges representing a plurality of user techniques; and

the modified first user workflow comprises a modified version of the first user workflow and includes a first highlighted edge that represents a first user technique that has been determined to be less efficient than all other user techniques included in the plurality of user techniques based on a comparison between the first user workflow and the primary representative workflow.

4. The computer-implemented method of claim 1 , wherein:

the first user workflow comprises a plurality of edges representing a plurality of user techniques;

the first workflow graph includes a primary representative workflow comprising a plurality of representative edges representing a plurality of user techniques, each representative edge in the primary representative workflow corresponding to an edge in the first user workflow; and

the modified first user workflow comprises a modified version of the first user workflow and includes a first highlighted edge that represents a first user technique that has been determined to be less efficient than a corresponding user technique represented by a corresponding representative edge in the primary representative workflow.

5. The computer-implemented method of claim 4 , wherein:

the first workflow graph comprises a plurality of representative workflows; and

the primary representative workflow has a lowest number of total invoked commands relative to all other representative workflows included in the plurality of representative workflows.

6. The computer-implemented method of claim 4 , wherein:

the first workflow graph comprises a plurality of representative workflows; and

the primary representative workflow has a lowest performance time relative to all other representative workflows included in the plurality of representative workflows.

7. The computer-implemented method of claim 4 , wherein the first user technique is associated with a first number of invoked commands, the corresponding representative edge is associated with a second number of invoked commands, and the first number is greater than the second number.

8. The computer-implemented method of claim 4 , wherein the first user technique is associated with a first performance time, the corresponding representative edge is associated with a second performance time, and the first performance time is greater than the second performance time.

9. The computer-implemented method of claim 1 , wherein the modified first user workflow comprises a plurality of edges, and further comprising:

receiving from a client device, via the user interface, a user selection of a first edge included in the modified first user workflow;

retrieving edge information associated with the first edge; and

causing the edge information to be displayed, at the client device, via the user interface.

10. The computer-implemented method of claim 9 , wherein the edge information comprises at least one of compiled metrics, command data, and a screen recording associated with a user technique represented by the first edge.

11. One or more non-transitory computer-readable media including instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:

receiving a selection of a first task that is performed via a design application;

executing, on a first node of a first user workflow for performing the first task and a second node of a first workflow graph that graphically represents a plurality of user workflows for performing the first task, a trained autoencoder to generate a first vector comprising a compact representation of three-dimensional (3D) data associated with the first node and a second vector comprising a compact representation of 3D data associated with the second node;

determining, via the trained autoencoder, that the first node and the second node are similar based on a comparison between the first vector and the second vector;

subsequent to determining that the first node and the second node are similar, determining that a first edge associated with the first node should be modified based on a comparison between a feature of the first edge and a similar feature of a second edge associated with the second node;

generating a modified first user workflow based on a modification to the first edge; and

causing the modified first user workflow to be displayed via a user interface.

12. The one or more non-transitory computer-readable media of claim 11 , wherein:

the first task comprises modeling a first object; and

the modified first user workflow comprises a sequence of edges and nodes, each node in the modified first user workflow representing a state of the modeling of the first object.

13. The one or more non-transitory computer-readable media of claim 11 , wherein:

the first workflow graph comprises a plurality of representative workflows including a primary representative workflow, the primary representative workflow having a lowest number of total invoked commands relative to all other representative workflows included in the plurality of representative workflows;

the first user workflow comprises a plurality of edges representing a plurality of user techniques; and

the modified first user workflow comprises a modified version of the first user workflow and includes a first highlighted edge that represents a first user technique that has been determined to be less efficient than all other user techniques included in the plurality of user techniques based on a comparison between the first user workflow and the primary representative workflow.

14. The one or more non-transitory computer-readable media of claim 11 , wherein:

the first user workflow comprises a plurality of edges representing a plurality of user techniques;

the first workflow graph includes a primary representative workflow comprising a plurality of representative edges representing a plurality of user techniques, each representative edge in the primary representative workflow corresponding to an edge in the first user workflow; and

the modified first user workflow comprises a modified version of the first user workflow and includes a first highlighted edge that represents a first user technique that has been determined to be less efficient than a corresponding user technique represented by a corresponding representative edge in the primary representative workflow.

15. The one or more non-transitory computer-readable media of claim 14 , wherein:

the first workflow graph comprises a plurality of representative workflows; and

the primary representative workflow has a lowest number of total invoked commands relative to all other representative workflows included in the plurality of representative workflows.

16. The one or more non-transitory computer-readable media of claim 11 , wherein:

the first user workflow comprises a plurality of edges representing a plurality of user techniques; and

the modified first user workflow comprises a modified version of the first user workflow comprising a first highlighted edge determined to represent a first user technique comprising a least efficient user technique in the plurality of user techniques based on a comparison between the first user workflow and the first workflow graph.

17. The one or more non-transitory computer-readable media of claim 16 , wherein the first highlighted edge is determined to represent the least efficient user technique in the plurality of user techniques based on a total number of invoked commands associated with the first user technique.

18. The one or more non-transitory computer-readable media of claim 16 , wherein the first highlighted edge is determined to represent the least efficient user technique in the plurality of user techniques based on a total performance time associated with the first user technique.

19. The one or more non-transitory computer-readable media of claim 11 , wherein the modified first user workflow comprises a plurality of edges, further comprising the steps of:

receiving from a client device, via the user interface, a user selection of a first edge included in the modified first user workflow;

retrieving edge information associated with the first edge; and

causing the edge information to be displayed, at the client device, via the user interface.

20. A computing system comprising:

one or more memories that includes instructions; and

one or more processors that are coupled to the one or more memories and, upon executing the instructions, perform the steps of:

receiving a selection of a first task that is performed via a design application;

executing, on a first node of a first user workflow for performing the first task and a second node of a first workflow graph that graphically represents a plurality of user workflows for performing the first task, a trained autoencoder to generate a first vector comprising a compact representation of three-dimensional (3D) data associated with the first node and a second vector comprising a compact representation of 3D data associated with the second node;

determining, via the trained autoencoder, that the first node and the second node are similar based on a comparison between the first vector and the second vector;

subsequent to determining that the first node and the second node are similar, determining that a first edge associated with the first node should be modified based on a comparison between a feature of the first edge and a similar feature of a second edge associated with the second node;

generating a modified first user workflow based on a modification to the first edge; and

causing the modified first user workflow to be displayed via a user interface.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 26, 2019
From: GROSSMAN, TOVI; FITZMAURICE, GEORGE; LAFRENIERE, BENJAMIN; KIM, JUHO; CHANG, MINSUK; HASSANI, KAVEH
To: AUTODESK, INC.
Reel/Frame 051370/0966 →
Continuity (2)
Provisional Application 62842271 · May 2, 2019
Related Publication 20200349482A1 · Nov 5, 2020
References Cited (66)
US 11354352B1 · Keisler · 2022 [cited by examiner]
US 20030004770A1 · Miller et al. · 2003 [cited by applicant]
US 20070055558A1 · Shanahan et al. · 2007 [cited by applicant]
US 20080065448A1 · Hull et al. · 2008 [cited by applicant]
US 20080312963A1 · Reiner · 2008 [cited by examiner]
US 20090222817A1 · Faatz et al. · 2009 [cited by applicant]
US 20110301732A1 · Gao · 2011 [cited by examiner]
US 20130311927A1 · Grossman et al. · 2013 [cited by applicant]
US 20140129285A1 · Wu et al. · 2014 [cited by applicant]
US 20140257835A1 · Kashyap · 2014 [cited by applicant]
US 20170024436A1 · Look et al. · 2017 [cited by applicant]
US 20170068816A1 · Cavazos · 2017 [cited by examiner]
US 20170364843A1 · Haligowski · 2017 [cited by examiner]
US 20190026634A1 · Homeyer et al. · 2019 [cited by applicant]
US 20190279114A1 · Deshpande et al. · 2019 [cited by applicant]
Akers et al., “Backtracking Events As Indicators of Usability Problems in Creation-Oriented Applications”, ACM Transactions on Computer-Human Interaction, DOI 10.1145/2240156.2240164, vol. 19, No. 2, Article 16, Jul. 20… [cited by applicant]
Aspert et al., “MESH : Measuring Errors Between Surfaces Using the Hausdorff Distance”, In Proceedings IEEE International Conference on Multimedia and Expo, vol. 1, Aug. 26-29, 2002, pp. 705-708. [cited by applicant]
Birant et al., “ST-DBSCAN: An algorithm for clustering spatial-temporal data”, Data & Knowledge Engineering, doi:10.1016/j.datak.2006.01.013, vol. 60, No. 1, 2007, pp. 208-221. [cited by applicant]
Breunig et al., “LOF: Identifying Density-Based Local Outliers”, In ACM SIGMOD record, vol. 29. ACM, 2000, pp. 93-104. [cited by applicant]
Carroll et al., “The Nurnberg Funnel: Designing Minimalist Instruction for Practical Computer Skill”, IEEE Transactions on Professional Communication, vol. 33, No. 4, Dec. 1990, pp. 182-187. [cited by applicant]
Carroll et al., “Paradox of the Active User”, In Interfacing Thought: Cognitive Aspects of Human-Computer Interaction, http://dl.acm.org/citation.cfm?id=28446.28451, MIT Press, 1987, pp. 80-111. [cited by applicant]
Chang et al., “RecipeScape: An Interactive Tool for Analyzing Cooking Instructions at Scale”, In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, DOI: https://doi.org/10.1145/3173574.3174025… [cited by applicant]
Chi et al., “MixT: Automatic Generation of Step-by-Step Mixed Media Tutorials”, In Proceedings of the 25th annual ACM symposium on User interface software and technology (UIST '12), ACM, Oct. 7-10, 2012, pp. 93-102. [cited by applicant]
Denning et al., “MeshFlow: Interactive Visualization of Mesh Construction Sequences”, DOI: http://dx.doi.org/10.1145/2010324.1964961, ACM Transactions on Graphics, vol. 30, No. 4, Article 66, Jul. 2011, pp. 66:1-66.8. [cited by applicant]
Ester et al., “A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise”, In Proceedings of 2nd International Conference on Knowledge Discovery and Data Mining, vol. 96, 1996, pp. 226-231. [cited by applicant]
Fan et al., “A Point Set Generation Network for 3D Object Reconstruction from a Single Image”, In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 605-613. [cited by applicant]
Fernquist et al., “Sketch-Sketch Revolution: An Engaging Tutorial System for Guided Sketching and Application Learning” In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11… [cited by applicant]
Fourney et al., “InterTwine: Creating Interapplication Information Scent to Support Coordinated Use of Software”, In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14), htt… [cited by applicant]
Fourney et al., “Query-Feature Graphs: Bridging User Vocabulary and System Functionality”, In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11), Oct. 16-19, 2011, pp. 207-… [cited by applicant]
Fraser et al., “RePlay: Contextually Presenting Learning Videos Across Software Applications”, In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, https://doi.org/10.1145/3290605.3300527, May … [cited by applicant]
Fu et al., “Resolving the paradox of the active user: stable suboptimal performance in interactive tasks”, DOI: http://dx.doi.org/10.1016/j.cogsci.2004.03.005, Cognitive Science, vol. 28, No. 6, Nov. 2004, pp. 901-935. [cited by applicant]
Glassman et al., “Learnersourcing Personalized Hints”, In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16), DOI: http://dx.doi.org/10.1145/2818048.2820011, Feb.… [cited by applicant]
Grabler et al., “Generating Photo Manipulation Tutorials by Demonstration”, DOI: http://dx.doi.org/10.1145/1531326.1531372, ACM Transactions on Graphics, vol. 28, No. 3, Article 66, Aug. 2009, pp. 66:1-66:9. [cited by applicant]
Grossman et al., “Strategies for Accelerating On-line Learning of Hotkeys”, DOI: http://dx.doi.org/10.1145/1240624.1240865, In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '07), Apr. 2… [cited by applicant]
Grossman et al., “ToolClips: An Investigation of Contextual Video Assistance for Functionality Understanding”, In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10), DOI: http://dx.doi.… [cited by applicant]
Grossman et al., “A Survey of Software Learnability: Metrics, Methodologies and Guidelines”, In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09), DOI:http://dx.doi.org/10.1145/1518701… [cited by applicant]
Grossman et al., “Chronicle: Capture, Exploration, and Playback of Document Workflow Histories”, In Proceedings of the 23nd annual ACM symposium on User interface software and technology (UIST '10), DOI: http://dx.doi.o… [cited by applicant]
Guo, Philip J., “Codeopticon: Real-Time, One-To-Many Human Tutoring for Computer Programming”, In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15), DOI: http://dx.doi.org/1… [cited by applicant]
Jin et al., “SolveDeep: A System for Supporting Subgoal Learning in Online Math Problem Solving”, In Proceedings of the SIGCHI Conference Extended Abstracts on Human Factors in Computing Systems, https://doi.org/10.1145… [cited by applicant]
Kim, Juho., “Learnersourcing: Improving Learning with Collective Learner Activity”, PhD thesis, Massachusetts Institute of Technology, Sep. 2015, 213 pages. [cited by applicant]
Kong et al., “Delta: A Tool For Representing and Comparing Workflows”, In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12), DOI: http://dx.doi.org/10.1145/2208516.2208549, May 5-10, 2… [cited by applicant]
Lafreniere et al., “Task-Centric Interfaces for Feature-Rich Software”, In Proceedings of the 26th Australian Computer-Human Interaction Conference on Designing Futures: The Future of Design (OzCHI '14), DOI:http://dx.d… [cited by applicant]
Lafreniere et al., “Community Enhanced Tutorials: Improving Tutorials with Multiple Demonstrations”, In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM CHI 2013, Apr. 27-May 2, 2013, pp. … [cited by applicant]
Li et al., “GamiCAD: A Gamified Tutorial System for First Time AutoCAD Users”, In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST '12), DOI: http://dx.doi.org/10.1145/2380116… [cited by applicant]
Li et al., “ CADament: A Gamified Multiplayer Software Tutorial System”, In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14), DOI: http://dx.doi.org/10.1145/2556288.2556954, Apr. 26-M… [cited by applicant]
Li et al., “TutorialPlan: Automated Tutorial Generation from CAD Drawings”, In Twenty-Third International Joint Conference on Artificial Intelligence, 2013, pp. 2020-2027. [cited by applicant]
Malacria et al., “Skillometers: Reflective Widgets That Motivate and Help Users to Improve Performance”, In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13), DOI: http://… [cited by applicant]
Matejka et al., “Ambient Help”, In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, DOI: http://dx.doi.org/10.1145/1978942.1979349, (CHI '11), May 7-12, 2011, pp. 2751-2760. [cited by applicant]
Matejka et al., “CommunityCommands: Command Recommendations for Software Applications”, In Proceedings of the 20th annual ACM symposium on User interface software and technology (UIST '09), DOI: http://dx.doi.org/10.114… [cited by applicant]
Pongnumkul et al., “Pause-and-Play: Automatically Linking Screencast Video Tutorials with Applications”, In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11), DOI: http://… [cited by applicant]
Qi et al., “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation”, arXiv:1612.00593, 2016, 19 pages. [cited by applicant]
Sander et al., “Density-Based Clustering in Spatial Databases: The Algorithm GDBSCAN and Its Applications”, Data Mining and Knowledge Discovery 2, vol. 2, 1998, pp. 169-194. [cited by applicant]
Scarr et al., “Dips and Ceilings: Understanding and Supporting Transitions to Expertise in User Interfaces”, In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11), DOI: http://dx.doi.or… [cited by applicant]
Wang et al., “Leveraging Community-Generated Videos and Command Logs to Classify and Recommend Software Workflows”, CHI 2018, https://doi.org/10.1145/3173574.3173859, Apr. 21-26, 2018, pp. 1-13. [cited by applicant]
Whitehill et al., “A Crowdsourcing Approach to Collecting Tutorial Videos-Toward Personalized Learning-at-Scale”, In Proceedings of the Fourth (2017) ACM Conference on Learning@ Scale, DOI: http://dx.doi.org/10.1145/305… [cited by applicant]
Williams et al., “AXIS: Generating Explanations at Scale with Learnersourcing and Machine Learning”, In Proceedings of the Third (2016) ACM Conference on Learning@Scale (L@S '16), DOI: http://dx.doi.org/10.1145/2876034.… [cited by applicant]
Yan et al., “The PyramidSnapshot Challenge: Understanding student process from visual output of programs”, In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (SIGCSEâ ̌AZ'19), https://doi.or… [cited by applicant]
Yang et al., “Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clustering”, In Proceedings of the 34th International Conference on Machine Learning, JMLR. org, vol. 70, 2017, pp. 3861-3870. [cited by applicant]
Yi et al., “A Scalable Active Framework for Region Annotation in 3D Shape Collections”, SIGGRAPH Asia (2016), DOI 10.1145/2980179.2980238, ACM Transactions on Graphics, vol. 35, No. 6, Article 210, Nov. 2016, pp. 210:1-… [cited by applicant]
International Search Report for application No. PCT/US2020/030679 dated Aug. 13, 2020. [cited by applicant]
Zhang et al., “Collaborative Scientific Workflow Composition as a Service: An Infrastructure Supporting Collaborative Data Analytics Workflow Design and Management”, 2016 IEEE 2nd International Conference on Collaborati… [cited by applicant]
Mo et al., “PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding”, XP080990049, arxiv:1812.02713v1, Dec. 6, 2018, pp. 1-23. [cited by applicant]
Tosta et al., “Improving workflow design by mining reusable tasks”, Journal of the Brazilian Computer Society, DOI 10.1186/S13173-015-0035-y, vol. 21, No. 1, XP035552887, ISSN: 0104-6500, Oct. 5, 2015, pp. 1-16. [cited by applicant]
Non Final Office Action received for U.S. Appl. No. 16/705,133 dated Jun. 15, 2022, 39 pages. [cited by applicant]
Notice of Allowance received for U.S. Appl. No. 16/705,133 dated Oct. 18, 2022, 19 pages. [cited by applicant]
Wei et al., “Anomaly Prediction Approach in Business Process Based on Machine Learning”, doi:10.13196/j.cims, Apr. 2019, 9 pages. [cited by applicant]