IP Library Granted Patent US 11,475,207
Granted Patent B2
US 11,475,207 · App. 16/901,656 · Granted Oct 18, 2022

Subject line tester

Inventors: Kexin Xie (San Mateo, CA); Gokhan Cagrici (Gilroy, CA); Daniel Keith Wilson (San Francisco, CA); Shrestha Basu Mallick (San Francisco, CA); Jonathan Daniel Showers Belkowitz (Brooklyn, NY); Jason Lestina (San Francisco, CA); James Brewer (Indianapolis, IN); Daniel Louis Gasperut (Indianapolis, IN); Jeffery Allen Zickgraf (Indianapolis, IN); Greg Lyman (Jacksonville, IL); Michael Ronald Brewer (Somers, MT); Evan Black (Carmel, IN); Austin Rauschuber (Hamden, CT); Victoria Schultz (Indianapolis, IN); Matthew David Trepina (Westfield, IN); Peter Stadlinger (San Francisco, CA)
Assignee: Salesforce, Inc.
G06F40/166G06F40/216G06N20/00H04L51/52
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,475,207
App. No.
16/901,656
Granted
Oct 18, 2022
Kind
B2
Abstract

Methods, systems, and devices supporting data processing are described. In some systems, a data processing platform may support communication message analysis using machine learning. For example, a system may receive a set of communication messages (e.g., social media messages) and perform a machine learning process on the message contents and message interaction data to train a machine learned model. The system may further receive a subject line for a communication message for analysis, input the subject line into the machine learned model, and receive, as an output of the machine learned model, an engagement score based on the subject line. The engagement score may indicate an estimated probability that a user receiving the communication message opens the communication message (e.g., based on the subject line). A user—or the system—may modify the subject line based on the analysis to improve the engagement score.

Claims (61)

1. A method for communication message analysis using machine learning, comprising:

receiving a plurality of social media messages comprising a plurality of respective message contents and corresponding to respective interaction data indicating how users interact with a social media message on a social media platform;

performing a machine learning process on the plurality of respective message contents and the respective interaction data corresponding to the plurality of social media messages to generate a machine learned model;

receiving a subject line for a communication message for analysis;

inputting the subject line into the machine learned model; and

receiving, as an output of the machine learned model, an engagement score based at least in part on the subject line, wherein the engagement score indicates an estimated probability that a user receiving the communication message on an email platform different from the social media platform opens the communication message.

2. The method of claim 1 , further comprising:

receiving, as an additional output of the machine learned model, one or more suggested changes to the subject line.

3. The method of claim 2 , further comprising:

receiving a user input indicating a suggested change of the one or more suggested changes to the subject line; and

updating the communication message to comprise an updated subject line based at least in part on the subject line and the suggested change.

4. The method of claim 2 , wherein receiving, as the additional output of the machine learned model, the one or more suggested changes to the subject line further comprises:

receiving one or more additional engagement scores corresponding to the one or more suggested changes to the subject line.

5. The method of claim 1 , further comprising:

receiving, as an additional output of the machine learned model, an indicated segmentation of the subject line into a plurality of portions and a respective indication of how each portion of the plurality of portions affects the engagement score for the subject line.

6. The method of claim 1 , further comprising:

sending, for display in a user interface of a user device, the engagement score.

7. The method of claim 1 , further comprising:

transmitting the communication message to one or more users;

receiving, from at least one user of the one or more users, feedback information indicating user engagement with the communication message; and

updating the machine learned model based at least in part on the feedback information.

8. The method of claim 7 , wherein the user engagement with the communication message comprises opening the communication message, replying to the communication message, forwarding the communication message, performing an action associated with the communication message, or a combination thereof.

9. The method of claim 7 , wherein the engagement score comprises a predicted user engagement score, the method further comprising:

determining an actual user engagement score based at least in part on the feedback information, wherein the machine learned model is updated based at least in part on a difference between the predicted user engagement score and the actual user engagement score.

10. The method of claim 1 , wherein performing the machine learning process further comprises:

determining an association between the plurality of respective message contents and the respective interaction data corresponding to the plurality of social media messages, wherein the machine learned model is based at least in part on the association.

11. The method of claim 1 , further comprising:

filtering a total set of social media messages to remove one or more outlier social media messages with message engagement data greater than a first threshold engagement amount or lower than a second threshold engagement amount, wherein the plurality of social media messages is received from the filtered total set of social media messages.

12. The method of claim 1 , wherein performing the machine learning process further comprises:

training the machine learned model using a first subset of the plurality of social media messages; and

validating the machine learned model using a second subset of the plurality of social media messages.

13. The method of claim 12 , wherein validating the machine learned model comprises:

inputting respective message contents for each message of the second subset of social media messages into the machine learned model to obtain respective predicted message interaction data for each message of the second subset of social media messages;

comparing the respective predicted message interaction data for each message of the second subset of social media messages to respective actual message interaction data corresponding to each message of the second subset of social media messages; and

further training the machine learned model based at least in part on the comparing.

14. The method of claim 1 , wherein receiving the plurality of social media messages further comprises:

identifying a criterion for an account based at least in part on a number of other accounts connected to the account, a number of social media messages associated with the account, content of social media messages associated with the account, or a combination thereof; and

selecting a plurality of accounts based at least in part on the criterion, wherein the plurality of social media messages is received from the selected plurality of accounts.

15. The method of claim 1 , wherein receiving the plurality of social media messages further comprises:

identifying a first social media message of the plurality of social media messages with a highest message interaction rate and a second social media message of the plurality of social media messages with a lowest message interaction rate, wherein the first social media message and the second social media message correspond to a range for the engagement score.

16. The method of claim 1 , wherein the plurality of social media messages is received based at least in part on a set of users generating the plurality of social media messages, a common subset of message contents of the plurality of social media messages, the respective interaction data corresponding to the plurality of social media messages, or a combination thereof.

17. The method of claim 1 , wherein the respective interaction data comprises a number of shares, a number of responses, a number of views, or a combination thereof.

18. The method of claim 1 , wherein:

the plurality of respective message contents comprises text, a link, an emoticon, an image, or a combination thereof;

the communication message comprises an email message; and

the subject line comprises text, a link, or a combination thereof.

19. An apparatus for communication message analysis using machine learning, comprising:

a processor;

memory coupled with the processor; and

instructions stored in the memory and executable by the processor to cause the apparatus to:

receive a plurality of social media messages comprising a plurality of respective message contents and corresponding to respective interaction data indicating how users interact with a social media message on a social media platform;

perform a machine learning process on the plurality of respective message contents and the respective interaction data corresponding to the plurality of social media messages to generate a machine learned model;

receive a subject line for a communication message for analysis;

input the subject line into the machine learned model; and

receive, as an output of the machine learned model, an engagement score based at least in part on the subject line, wherein the engagement score indicates an estimated probability that a user receiving the communication message on an email platform different from the social media platform opens the communication message.

20. A non-transitory computer-readable medium storing code for communication message analysis using machine learning, the code comprising instructions executable by a processor to:

receive a plurality of social media messages comprising a plurality of respective message contents and corresponding to respective interaction data indicating how users interact with a social media message on a social media platform;

perform a machine learning process on the plurality of respective message contents and the respective interaction data corresponding to the plurality of social media messages to generate a machine learned model;

receive a subject line for a communication message for analysis;

input the subject line into the machine learned model; and

receive, as an output of the machine learned model, an engagement score based at least in part on the subject line, wherein the engagement score indicates an estimated probability that a user receiving the communication message on an email platform different from the social media platform opens the communication message.

Assignments (3)
CHANGE OF NAME Recorded Dec 18, 2024
From: SALESFORCE.COM, INC.
To: SALESFORCE, INC.
Reel/Frame 069717/0499 →
CORRECTIVE ASSIGNMENT TO CORRECT THE NINTH INVENTOR'S FIRST NAME PREVIOUSLY RECORDED ON REEL 052941 FRAME 0746. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Sep 6, 2022
From: XIE, KEXIN; CAGRICI, GOKHAN; WILSON, DANIEL KEITH; MALLICK, SHRESTHA BASU; BELKOWITZ, JONATHAN DANIEL SHOWERS; LESTINA, JASON; BREWER, JAMES; GASPERUT, DANIEL LOUIS; ZICKGRAF, JEFFERY ALLEN; LYMAN, GREG; BREWER, MICHAEL RONALD; BLACK, EVAN; RAUSCHUBER, AUSTIN; SCHULTZ, VICTORIA; TREPINA, MATTHEW DAVID; STADLINGER, PETER
To: SALESFORCE.COM, INC.
Reel/Frame 061603/0961 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 15, 2020
From: XIE, KEXIN; CAGRICI, GOKHAN; WILSON, DANIEL KEITH; MALLICK, SHRESTHA BASU; BELKOWITZ, JONATHAN DANIEL SHOWERS; LESTINA, JASON; BREWER, JAMES; GASPERUT, DANIEL LOUIS; ZICKGRAF, JEFFREY ALLEN; LYMAN, GREG; BREWER, MICHAEL RONALD; BLACK, EVAN; RAUSCHUBER, AUSTIN; SCHULTZ, VICTORIA; TREPINA, MATTHEW DAVID; STADLINGER, PETER
To: SALESFORCE.COM, INC.
Reel/Frame 052941/0746 →
Continuity (2)
Provisional Application 62938949 · Nov 21, 2019
Related Publication 20210157974A1 · May 27, 2021
Cited By (8)
US 1,093,421 US 12,387,236 US 12,511,473 US 12,536,561 US 12,632,442 US 12,645,674 US 12,657,399 US 12,670,151