Granted Patent B2

US 12,639,719 · App. 17/960,574 · Granted May 26, 2026

Inventors: Karl Buhariwala (Sammamish, WA); Adity Agarwal (Redmond, WA); Ganga Narayanan (Kirkland, WA); Kiran Nallabothula (Redmond, WA)

Assignee: Microsoft Technology Licensing, LLC

G06Q30/016G06F11/366

Abstract

Claims (67)

1 . A method for generating passive feedback to improve a software product, the method comprising:

determining at each of a plurality of customer devices and by a processor that tracks quantities of time a user has spent interacting with various different elements of a flow of visual elements presented by the software product;

collecting, by the plurality of customer devices, engagement data pertaining to interactions between customers and the flow of visual elements presented by the software product, the engagement data indicating quantities of time that users have spent interacting with different elements within the flow of visual elements;

aggregating the engagement data to create aggregated engagement data;

detecting, by a first machine learning model, a pattern in the aggregated engagement data that has been learned by the first machine learning model as being indicative of atypical user behavior and inconsistent with patterns associated with behavior expected from customers satisfied with the software product, the pattern corresponding to a location within the flow of visual elements;

in response to detecting the pattern, identifying a potentially deficient content item by correlating the location within the flow of visual elements with a particular content item viewed at a time of the atypical user behavior;

in response to identifying the potentially deficient content item, determining a reason for the atypical user behavior by providing the potentially deficient content item to a second machine learning model trained to detect a predefined type of content deficiency, the second machine learning model trained on content elements and user ratings of the content elements;

receiving an output from the second machine learning model indicating that the potentially deficient content item is characterized by the predefined type of content deficiency, and

in response to receiving the output from the second machine learning model, autogenerating a repair ticket for review by a developer, the repair ticket identifying the potentially deficient content item and the predefined type of content deficiency identified by the second machine learning model.

2 . The method of claim 1 , wherein the software product is a debug and diagnostic software tool and the method further comprises:

receiving a first indication that a customer has experienced a particular technical problem;

determining that the customer has used the debug and diagnostic software tool in an effort to address the particular technical problem; and

receiving a second indication that the customer is dissatisfied with the debug and diagnostic software tool in relation to the particular technical problem, wherein identifying the location occurs in response to receipt of the second indication.

3 . The method of claim 1 , wherein the pattern is indicative of an average user interaction time that is of atypical length or a tendency to abort the flow of visual elements at a location that is not a natural endpoint.

4 . The method of claim 1 , further comprising:

receiving a report of a technical problem encountered by a customer; and

recommending, based on content of the report, a debug and diagnostic software tool to help the customer resolve the technical problem, wherein the software product is the debug and diagnostic software tool and collecting the engagement data further comprises:

collecting engagement data pertaining to customer interactions with the debug and diagnostic software tool.

5 . The method of claim 1 ,

wherein identifying the location further comprises identifying a visual element within the flow of elements that is characterized, within the aggregated engagement data, by an anomalous average interaction time, wherein the repair ticket identifies a potential deficiency with the visual element.

6 . The method of claim 1 , wherein identifying the location further comprises identifying an element in the flow of visual elements corresponding to a last-interacted-with element for a subset of the customers, wherein the repair ticket identifies a potential deficiency with the last-interacted-with element.

7 . The method of claim 1 , further comprising:

receiving a request for technical support from a customer;

automatically determining, from content of the request, a characteristic of a technical problem identified by the customer;

searching content within the flow of visual elements for a reference to the characteristic of the technical problem; and

determining that the flow of visual elements does not include the reference to the characteristic of the technical problem, wherein the repair ticket indicates the flow of visual elements is deficient for failing to reference the characteristic of the technical problem reported by the customer.

8 . The method of claim 1 , further comprising:

determining that a customer of the software product subsequently received assistance with the software product from a human operator;

analyzing state data received from a computer of the customer;

detecting, based on the analyzing of the state data, system alterations to the computer that occurred during a time interval corresponding to a time in which the customer received the assistance from the human operator; and

automatically analyzing the visual elements in the flow for reference to the detected system alterations, wherein the repair ticket indicates that the software product is deficient for failing to address the system alterations.

9 . An automated system for identifying a root cause of customer dissatisfaction with a software product, the automated system comprising:

an engagement data collection tool stored in memory that collects engagement data pertaining to interactions between a customer and a flow of visual elements presented by the software product, wherein collecting the engagement data entails using a processor to track quantities of time a user has spent interacting with different elements within the flow of visual elements and the engagement data indicates the quantities of time that the user has spent interacting with the different elements;

a passive feedback generator stored in memory that automatically:

creates aggregated engagement data by aggregating the engagement data collected from a plurality of customer devices;

analyses the aggregated engagement data to detect a pattern in the aggregated engagement data corresponding to a location within the flow of visual elements, the pattern being indicative of an atypical user behavior;

in response to detecting the pattern, identifies a potentially deficient content item by correlating the location within the flow of visual elements with a particular content item being viewed at a time of the atypical user behavior;

in response to identifying the potentially deficient content item, determines a reason for the atypical user behavior by providing the potentially deficient content item to a first machine learning model trained to detect a predefined type of content deficiency;

receives an output from the first machine learning model indicating that the potentially deficient content item is characterized by the predefined type of content deficiency; and

in response to receiving the output from the first machine learning model, autogenerates a repair ticket for review by a developer the repair ticket identifying the potentially deficient content item and the predefined type of content deficiency identified by the first machine learning model.

10 . The automated system of claim 9 , wherein the system is further configured to perform operations comprising:

receiving a report of a technical problem encountered by the customer;

recommending, based on content of the report, a debug and diagnostic software tool to help resolve the customer resolve the technical problem, wherein the software product is the debug and diagnostic software tool and collecting the engagement data further comprises:

collecting engagement data pertaining to customer interactions with the debug and diagnostic software tool.

11 . The automated system of claim 10 , wherein the pattern is indicative of an average user interaction time that is of atypical length or a tendency to abort the flow of visual elements at a location that is not a natural endpoint.

12 . The automated system of claim 9 , wherein the passive feedback generator is configured to:

identify a visual element within the flow of elements that is characterized, within the aggregated engagement data, by an anomalous average interaction time, wherein the repair ticket identifies a potential deficiency with the visual element.

13 . The automated system of claim 9 , wherein the passive feedback generator is configured to: identify a visual element in the flow of elements corresponding to a last-interacted-with element for a subset of customers providing the engagement data, wherein the repair ticket identifies a potential deficiency with the last-interacted-with element.

14 . The automated system of claim 9 , wherein the software product is a debug and diagnostic software tool and the passive feedback generator is further configured to:

analyze content of a support ticket opened by the customer after interacting with the debug and diagnostic software tool, the support ticket indicating a characteristic of a technical problem identified by the customer;

analyze the flow of visual elements for a reference to the characteristic of the technical problem; and

determine that the flow of visual elements does not include the reference to the characteristic of the technical problem, wherein the repair ticket indicates that the debug and diagnostic software tool is deficient for failing to reference the characteristic of the technical problem reported by the customer.

15 . One or more tangible computer-readable storage media encoding computer-executable instructions for executing a computer process that automatically identifies a root cause of customer dissatisfaction with a software product, the computer process comprising:

determining at each of a plurality of customer devices and by a processor that tracks quantities of time a user has spent interacting with various different elements of a flow of visual elements presented by the software product;

collecting, by the plurality of customer devices, engagement data pertaining to interactions between customers and the flow of visual elements presented by the software product, the engagement data indicating quantities of time that users have spent interacting with different elements within the flow of visual elements;

aggregating the engagement data to create aggregated engagement data;

detecting a trigger event indicating that a select customer is dissatisfied with the software product;

in response to the trigger event, analyzing the aggregated engagement data to identify a pattern corresponding to a location within the flow of visual elements, the pattern being indicative of an atypical user behavior;

in response to detecting the pattern, identifying a potentially deficient content item by correlating the location within the flow of visual elements with a particular content item being viewed at a time of the atypical user behavior;

in response to identifying the potentially deficient content item, determining a reason for the atypical user behavior by providing the potentially deficient content item to a first machine learning model trained to detect a predefined type of content deficiency;

receiving an output from the first machine learning model indicating that the potentially deficient content item is characterized by the predefined type of content deficiency; and

in response to receiving the output from the first machine learning model, generating a repair ticket for a development team, the repair ticket identifying the software product and the predefined type of content deficiency identified by the first machine learning model.

16 . The one or more tangible computer-readable storage media of claim 15 , wherein the trigger event is a help support request identifying a technical problem encountered by a customer.

17 . The one or more tangible computer-readable storage media of claim 16 , wherein the computer process further comprises:

identifying a visual element within the flow of element that is characterized, within the aggregated engagement data, by an anomalous average interaction time, wherein the repair ticket identifies a potential deficiency with the visual element.

18 . The one or more tangible computer-readable storage media of claim 16 , wherein the computer process further comprises: identifying a visual element in the flow of elements corresponding to a last-interacted-with element for a subset of the customers, wherein the repair ticket identifies a potential deficiency with the last-interacted-with element.

19 . The one or more tangible computer-readable storage media of claim 16 , wherein the first machine learning model is trained on content elements and user ratings of the content elements.

Assignments (1)

Continuity (1)

References Cited (19)