Granted Patent B2

US 12,361,207 · App. 17/976,823 · Granted Jul 15, 2025

Inventors: Michael T. Lawson (Boston, MA); Woody Austin (Boston, MA); Ally Hangartner (Sarasota, FL); Carola Leiva (Boston, MA); Nicholas Hartmann (Swampscott, MA); Thomas Culp (Cambridge, MA); Justin Xu (Boston, MA); Maritza Ebling (Boston, MA)

Assignee: Klaviyo, Inc

G06F40/186G06Q30/02G06Q30/0201H04W4/021H04W4/12H04W4/14

Abstract

Claims (41)

1. A computer-implemented method for testing of a mobile message, comprising:

generating, by a server, at least a first template of the mobile message and a second template of the mobile message, each of the first template and the second template having at least a different content or a different send time;

determining, by the server, information associated with the mobile message;

identifying and sending, by the server, guidance for a marketing manager based on the information associated with the mobile message, wherein the marketing manager selects user-selected configurations in response to the guidance;

receiving, by the server, the user-selected configurations from the marketing manager in response to the guidance;

assigning, by the server, a first respective group of mobile message recipients to the first template of the mobile message and a second respective group of mobile message recipients to the second template of the mobile message, wherein the assigning is based on the information associated with the mobile message and the user-selected configurations received from the marketing manager;

determining automatically, by the server, whether the first template of the mobile message has a statistical advantage over the second template of the mobile message through testing based on the assigning, wherein the testing comprises;

collecting test data from testing including mobile message recipient actions of the first template of the mobile message and the second template of the mobile message;

generating estimates of mobile message recipient actions for the first template and the second template including applying an analysis method to the collected test data to determine a win probability difference between a win probability for the first template and a win probability for the second template, and determine that the win probability difference is greater than a selected win threshold;

determining that a precision of the estimates of a success rate is greater than a precision threshold; and

sending the mobile message of the first template to a remainder of planned mobile message recipients when the first template of the mobile message is determined to have the statistical advantage over the second template of the mobile message.

2. The method of claim 1 , wherein the information associated with the mobile message is stored in a database that is accessible by the server and includes information pertaining to a merchant of the marketing manager and mobile message recipients of the merchant.

3. The method of claim 2 , wherein the information associated with the mobile message comprises a message content and a send time.

4. The method of claim 2 , wherein the information associated with the mobile message comprises a geolocation of a cell carrier that operates to send the mobile message.

5. The method of claim 2 , wherein the information associated with the mobile message comprises legal restrictions in selected regions in which the mobile message is to be sent.

6. The method of claim 2 , wherein the information associated with the mobile message comprises latency of reported subsequent actions by cellular carriers.

7. The method of claim 2 , wherein the information associated with the mobile message comprises platform-wide event rates and latency.

8. The method of claim 1 , wherein identifying and sending the guidance for the marketing manager based on the information associated with the mobile message comprises:

generating, by the server, guidance to the marketing manager based on the information associated with the mobile message; and

communicating, by the server, the guidance to the marketing manager.

9. The method of claim 8 , wherein the guidance is based on SMS (short message service) compliance laws.

10. The method of claim 8 , wherein the guidance is based on an expected number of mobile message recipients included in a testing of the first and second templates.

11. The method of claim 8 , wherein the guidance is based on a metric to be used to determine a winner of a testing of the first and second templates.

12. The method of claim 1 , wherein the received user-selected configurations include a size of a testing of at least the first template and the second template.

13. The method of claim 1 , wherein the received user-selected configurations include a duration of testing of at least the first template and the second template.

14. The method of claim 1 , wherein the received user-selected configurations include a send time of the mobile message including at least the first template and the second template of testing of at least the first template and the second template.

15. The method of claim 1 , wherein the assigning is statistically random across each of a plurality of geolocations of cell carriers.

16. A system for automated testing a mobile message, comprising:

a marketing manager server configured to operate and manage a website;

a plurality of mobile devices configured to receive mobile messages; and

a server electronically connected to the marketing manager server and the plurality of mobile devices, the server configured to:

generate at least a first template of the mobile message and a second template of the mobile message, each of the first template and the second template having at least a different content or a different send time;

determine information associated with the mobile message;

identify and send guidance for a marketing manager based on the information associated with the mobile message;

receive user-selected configurations from the marketing manager in response to the guidance;

assign a first respective group of mobile message recipients to the first template of the mobile message and a second respective group of mobile message recipients to the second template of the mobile message, wherein the assigning is based on the information associated with the mobile message and the user-selected configurations received from the marketing manager;

determine automatically whether the first template of the mobile message has a statistical advantage over the second template of the mobile message through testing based on the assigning, wherein the testing comprises;

collect test data from testing including mobile message recipient actions of the first template of the mobile message and the second template of the mobile message;

generate estimates of mobile message recipient actions for the first template and the second template including applying an analysis method to the collected test data to determine a win probability difference between a win probability for the first template and a win probability for the second template, and determine that the win probability difference is greater than a selected win threshold;

determine that a precision of the estimates of a success rate is greater than a precision threshold; and

send the mobile message of the first template to a remainder of planned mobile message recipients when the first template of the mobile message is determined to have the statistical advantage over the second template of the mobile message.

Assignments (1)

Continuity (2)

References Cited (47)