Granted Patent B2

US 12,737,651 · App. 17/146,258 · Granted Sep 15, 2026

Inventors: Mark Roden (Los Angeles, CA); Mahesh Goud Tandarpally (Los Angeles, CA); James Healy Mirkovich (Los Angeles, CA)

Assignee: Live Nation Entertainment, Inc.

G06N5/04G06N20/00H04L63/102

Abstract

Claims (100)

1 . A computer-implemented method comprising:

collecting a feature set that includes one or more features associated with a plurality of users, each feature of the feature set corresponding to an attribute or characteristic of a previous interaction between a plurality of user devices and a primary load management system, and the primary load management system managing access to and assignment of a plurality of access rights to a plurality of resources;

storing, for each user of the plurality of users, a log line representing a user, the log line being a structured data string representing at least a subset of the feature set, each log line including a label representing whether or not the user or a data element within the structured data string is associated with an invalid user activity;

training a machine-learning model using the log lines and one or more machine-learning algorithms, the training of the machine-learning model including automatically detecting patterns of features within the feature set corresponding to the plurality of users, wherein the detected patterns define a criteria for the invalid user activity performed by bot traffic;

receiving a communication from a user device over the primary load management system, the communication corresponding to a request for one or more access rights to a resource, wherein the one or more access rights to the resource corresponds to a ticket for an event, and wherein the request corresponds to purchasing of the ticket;

adding the request in a digital queue generated by a queuing system associated with the primary load management system;

retrieving the log line corresponding to the user device;

inputting the log line corresponding to the user device into the trained machine-learning model;

in response to inputting the log line into the trained machine-learning model, processing the log line, via the trained machine learning model, to determine association of the request within the criteria for the invalid user activity,

generating an output predictive of an invalidity of the request associated with inputs from the bot traffic for the one or more access rights to the resource, wherein the output is generated based on a score generated by the trained machine-learning model, the score being:

generated after initiation and prior to completion of the purchasing of the ticket, and

a representation of a likelihood that the request is associated with the invalid user activity; and

performing an action based on the generated output, wherein the action includes:

removing the request from the digital queue, and

blocking the request from being placed on a digital interface of the primary load management for processing.

2 . The computer-implemented method of claim 1 , further comprising:

generating a relevance parameter for each access-right request of a plurality of access-right requests, wherein the generation of the relevance parameter is based on a combination of a total value associated with the request for the one or more access rights and a time from a query to a resource start time.

3 . The computer-implemented method of claim 2 , further comprising:

receiving a rank for each access-right request of the plurality of access-right requests; and

training a learn-to-rank model based on the ranked plurality of access-right requests.

4 . The computer-implemented method of claim 3 , further comprising:

feeding a new access-right request into the learn-to-rank model;

generating a relevance parameter for the new access-right request;

ranking the new access-right request with the ranked plurality of access-right requests based on the relevance parameters; and

determining a metric representing an invalid user activity associated with the new access-right request based at least in part on the relevance parameter of the new access-right request and the rank of the new access-right request.

5 . The computer-implemented method of claim 3 , further comprising:

ranking the each access-right request according to the corresponding output, the output representing or being associated with the generated relevance parameter;

feeding the ranked access-right requests in a group, along with other ranked access-right requests to one or more agents, wherein each agent of the one or more agents is configured to independently rank each access-right request of the grouped access-right requests based on an invalid user activity assessment;

receiving the access-right requests ranked by the one or more agents; and

using the learn-to-rank model to learn the rankings of the access-right requests conducted by the one or more agents.

6 . The computer-implemented method of claim 1 , wherein the training of the machine-learning model further comprises:

using a specialized learn-to-rank model to perform a pairwise comparison on labeled access-right requests and learning a labeled order.

7 . The computer-implemented method of claim 1 , wherein the log line corresponding to the user device includes a set of data elements that characterize a user associated with the user device or the user device itself, wherein each data element of the set of data elements corresponds to a value generated based on or in response to one or more interactions between the user device and the primary load management system.

8 . A system, comprising:

one or more processors; and

a non-transitory computer-readable storage medium containing instructions which, when executed on the one or more processors, cause the one or more processors to perform operations including:

collecting a feature set that includes one or more features associated with a plurality of users, each feature of the feature set corresponding to an attribute or characteristic of a previous interaction between a plurality of user devices and a primary load management system, and the primary load management system managing access to and assignment of a plurality of access rights to a plurality of resources;

storing, for each user of the plurality of users, a log line representing a user, the log line being a structured data string representing at least a subset of the feature set, each log line including a label representing whether or not the user or a data element within the structured data string is associated with an invalid user activity;

training a machine-learning model using the log lines and one or more machine-learning algorithms, the training of the machine-learning model including automatically detecting patterns of features within the feature set corresponding to the plurality of users, wherein the detected patterns define a criteria for the invalid user activity performed by bot traffic;

receiving a communication from a user device over the primary load management system, the communication corresponding to a request for one or more access rights to a resource, wherein the one or more access rights to the resource corresponds to a ticket for an event, and wherein the request corresponds to purchasing of the ticket;

adding the request in a digital queue generated by a queuing system associated with the primary load management system;

retrieving the log line corresponding to the user device;

inputting the log line corresponding to the user device into the trained machine-learning model;

in response to inputting the log line into the trained machine-learning model, processing the log line, via the trained machine learning model, to determine association of the request within the criteria for the invalid user activity,

generating an output predictive of an invalidity of the request associated with inputs from the bot traffic for the one or more access rights to the resource, wherein the output is generated based on a score generated by the trained machine-learning model, the score being:

generated after initiation and prior to completion of the purchasing of the ticket, and

a representation of a likelihood that the request is associated with the invalid user activity; and

performing an action based on the generated output, wherein the action includes:

removing the request from the digital queue, and

blocking the request from being placed on a digital interface of the primary load management for processing.

9 . The system of claim 8 , wherein the operations further comprise:

generating a relevance parameter for each access-right request of a plurality of access-right requests, wherein the generation of the relevance parameter is based on a combination of a total value associated with the request for the one or more access rights and a time from a query to a resource start time.

10 . The system of claim 9 , wherein the operations further comprise:

receiving a rank for each access-right request of the plurality of access-right requests; and

training a learn-to-rank model based on the ranked plurality of access-right requests.

11 . The system of claim 10 , wherein the operations further comprise:

feeding a new access-right request into the learn-to-rank model;

generating a relevance parameter for the new access-right request;

ranking the new access-right request with the ranked plurality of access-right requests based on the relevance parameters; and

determining a metric representing an invalid user activity associated with the new access-right request based at least in part on the relevance parameter of the new access-right request and the rank of the new access-right request.

12 . The system of claim 10 , wherein the operations further comprise:

ranking the each access-right request according to the corresponding output, the output representing or being associated with the generated relevance parameter;

feeding the ranked access-right requests in a group, along with other ranked access-right requests to one or more agents, wherein each agent of the one or more agents is configured to independently rank each access-right request of the grouped access-right requests based on an invalid user activity assessment;

receiving the access-right requests ranked by the one or more agents; and

using the learn-to-rank model to learn the rankings of the access-right requests conducted by the one or more agents.

13 . The system of claim 8 , wherein the training of the machine-learning model further comprises:

using a specialized learn-to-rank model to perform a pairwise comparison on labeled access-right requests and learning a labeled order.

14 . The system of claim 8 , wherein the log line corresponding to the user device includes a set of data elements that characterize a user associated with the user device or the user device itself, wherein each data element of the set of data elements corresponds to a value generated based on or in response to one or more interactions between the user device and the primary load management system.

15 . A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause a data processing apparatus to perform operations including:

collecting a feature set that includes one or more features associated with a plurality of users, each feature of the feature set corresponding to an attribute or characteristic of a previous interaction between a plurality of user devices and a primary load management system, and the primary load management system managing access to and assignment of a plurality of access rights to a plurality of resources;

storing, for each user of the plurality of users, a log line representing a user, the log line being a structured data string representing at least a subset of the feature set, each log line including a label representing whether or not the user or a data element within the structured data string is associated with an invalid user activity;

training a machine-learning model using the log lines and one or more machine-learning algorithms, the training of the machine-learning model including automatically detecting patterns of features within the feature set corresponding to the plurality of users, wherein the detected patterns define a criteria for the invalid user activity performed by bot traffic;

receiving a communication from a user device over the primary load management system, the communication corresponding to a request for one or more access rights to a resource, wherein the one or more access rights to the resource corresponds to a ticket for an event, and wherein the request corresponds to purchasing of the ticket;

adding the request in a digital queue generated by a queuing system associated with the primary load management system;

retrieving the log line corresponding to the user device;

inputting the log line corresponding to the user device into the trained machine-learning model;

in response to inputting the log line into the trained machine-learning model, processing the log line, via the trained machine learning model, to determine association of the request within the criteria for the invalid user activity,

generating an output predictive of an invalidity of the request associated with inputs from the bot traffic for the one or more access rights to the resource, wherein the output is generated based on a score generated by the trained machine-learning model, the score being:

generated after initiation and prior to completion of the purchasing of the ticket, and

a representation of a likelihood that the request is associated with the invalid user activity; and

performing an action based on the generated output, wherein the action includes:

removing the request from the digital queue, and

blocking the request from being placed on a digital interface of the primary load management for processing.

16 . The computer-program product of claim 15 , wherein the operations further comprise:

generating a relevance parameter for each access-right request of a plurality of access-right requests, wherein the generation of the relevance parameter is based on a combination of a total value associated with the request for the one or more access rights and a time from a query to a resource start time.

17 . The computer-program product of claim 16 , wherein the operations further comprise:

receiving a rank for each access-right request of the plurality of access-right requests; and

training a learn-to-rank model based on the ranked plurality of access-right requests.

18 . The computer-program product of claim 17 , wherein the operations further comprise:

feeding a new access-right request into the learn-to-rank model;

generating a relevance parameter for the new access-right request;

ranking the new access-right request with the ranked plurality of access-right requests based on the relevance parameters; and

determining a metric representing an invalid user activity associated with the new access-right request based at least in part on the relevance parameter of the new access-right request and the rank of the new access-right request.

19 . The computer-program product of claim 17 , wherein the operations further comprise:

ranking each access-right request according to the corresponding output, the output representing or being associated with the generated relevance parameter;

feeding the ranked access-right requests in a group, along with other ranked access-right requests to one or more agents, wherein each agent of the one or more agents is configured to independently rank each access-right request of the grouped access-right requests based on an invalid user activity assessment;

receiving the access-right requests ranked by the one or more agents; and

using the learn-to-rank model to learn the rankings of the access-right requests conducted by the one or more agents.

20 . The computer-program product of claim 15 , wherein the training of the machine-learning model further comprises:

using a specialized learn-to-rank model to perform a pairwise comparison on labeled access-right requests and learning a labeled order.

Assignments (3)

Continuity (2)

References Cited (21)