Testing of templates of electronic messages
Apparatuses, methods, and systems for testing templates of electronic messages. One method includes continuously sending, by a server, a main electronic message, wherein at least a content of the main electronic message is determined by a main message template, creating a plurality of test templates including at least an A template and a B template, initiating, by the server, testing of test electronic messages, wherein at least a content of one test electronic message is determined by the A template, and at least the content of another of the test electronic message is determined by the B template, wherein editing of the test electronic messages occurs simultaneous with the continuous sending of the main electronic message, and selecting, by the server, a winning template of the A template and the B template.
1. A computer-implemented method for testing templates of electronic messages, comprising:
continuously electronically sending, by a server, a main electronic message, wherein at least a content of the main electronic message is determined by a main message template;
creating a plurality of test templates including at least an A template and a B template;
initiating, by the server, testing of test electronic messages, wherein at least a content of one test electronic message is determined by the A template, and at least the content of another of the test electronic message is determined by the B template;
wherein editing of the test electronic messages occurs simultaneous with the continuous sending of the main electronic message;
sensing, by sensors of computing devices of recipients of the test electronic messages, actions of the recipients in response to receiving the test electronic messages;
selecting, by the server, a winning template of the A template and the B template based on at least the sensed actions of the recipients of the test electronic messages;
replacing, by the server, the main message template with the winner of the A template or the B template causing a dynamic update of an image displayed to the recipients based on an electronic message of the winner of the A template or the B template displayed on the computing devices of the recipients, wherein the electronic message includes a link for selection by the recipients; and
displaying, by the server, results of a series of completed testing of messages according to templates grouped with a live test to a merchant, thereby allowing the merchant to make decisions based on the completed tests and the live test.
2. The method of claim 1 , further comprising:
switching, by the server, between the sending of the main message and the sending of the test messages, wherein no customer of the merchant receives both the main message and any one of the test messages and all customers receive at least one of the main message or one of the test messages.
3. The method of claim 1 , further comprising:
transitioning from editing at least the A template and the B template, to testing the test electronic messages according to the A template and the B template, wherein the transition is initiated by the merchant.
4. The method of claim 3 , wherein sending of the main electronic message ceases during the testing.
5. The method of claim 1 , further comprising the merchant editing at least one of the main message template, the A template, or the B template during the continuous sending of the main electronic message.
6. The method of claim 1 , further comprising coordinating the sending of the main electronic message, editing of the A template and the B template, testing of messages according to the A template and the B template, and selecting a winning template over time.
7. The method of claim 1 , wherein continuously sending the main electronic message comprises triggering the sending of the main electronic message based on at least one of a customer being added to a list of customers, a customer being added to a dynamically generated group of customers based on an arbitrary set of criteria (segment), a dynamically ingested webhook event associated with the merchant, an action that a customer has taken that the server has just associated with the customer, a date associated with the customer or the merchant.
8. The method of claim 1 , further comprising the merchant selecting a weighting between the A template and the B template, wherein the weighting sets allocations of test messages being sent to customers according to each of the A template and the B template.
9. The method of claim 1 , further comprising the server adaptively selecting a weighting between the A template and the B template, wherein the weighting sets allocations of customers to each of the A template and the B template.
10. The method of claim 9 , wherein the server adaptively selects the weighting based on at least which test electronic message is winning the testing of the A template and the B template.
11. The method of claim 1 , wherein the A template and the B template are continuously tested.
12. The method of claim 11 , further comprising continuously displaying results of the testing of the A template and the B template, wherein a weighting of each of the template and the B template is continuously updated.
13. The method of claim 12 , wherein displaying the results comprises displaying results of the testing in real-time for templates A and B.
14. The method of claim 1 , wherein displaying the results comprises:
displaying real-time results for data associated with past testing, current testing, past main messages, or a live main electronic message.
15. The method of claim 1 , wherein the testing ends when one template is determined to have statistical advantage over another template, comprising;
collecting test data from testing including deliveries of the test electronic messages and interactions with test electronic message of the A template and the test electronic message of the B template;
generating estimates of a success rates for the A template and the B template including applying an analysis method to the collected test data to determine a win probability difference between a win probability for the A template and a win probability for the B template, and determine that the win probability difference is greater than a selected win threshold;
determining that a precision of the estimates of the success rate is greater than a precision threshold;
determining that greater than a volume threshold of test data has been collected;
determining, by the server, the statistical advantage when the selected win threshold, the precision threshold, and the volume threshold are all met.
16. The method of claim 1 , wherein the testing ends as selected by the merchant.
17. The method of claim 1 , further comprising:
selecting a winning template of a first testing of electronic messages;
initiating a second testing of electronic messages, wherein at least a content of one test electronic message is determined by the winning template of the first testing, and at least the content of another of the test electronic message is determined by a C template; and
selecting, by the server, a second winning template of the winning template of the first testing and the C template.
18. A system for testing templates of electronic messages, comprising:
a merchant server configured to operate and manage a website;
a plurality of customer devices configured to receive electronic messages; and
a server electronically connected to the merchant server and the plurality of customer devices, the server configured to:
continuously send a main electronic message, wherein at least a content of the main electronic message is determined by a main message template;
create a plurality of test templates including at least an A template and a B template;
initiate testing of test electronic messages, wherein at least a content of one test electronic message is determined by the A template, and at least the content of another of the test electronic message is determined by the B template;
wherein editing of the test electronic messages occurs simultaneous with the continuous sending of the main electronic message;
wherein sensors of computing devices of recipients of the test electronic messages sense actions of the recipients in response to receiving the test electronic messages;
wherein the server is further configured to:
select a winning template of the A template and the B template based on at least the sensed actions of the recipients of the test electronic messages; and
replace the main message template with the winner of the A template or the B template causing a dynamic update of an image displayed to the recipients based on an electronic message of the winner of the A template or the B template displayed on the computing devices of the recipients, wherein the electronic message includes a link for selection by the recipients; and
display results of a series of completed testing of messages according to templates grouped with a live test to a merchant, thereby allowing the merchant to make decisions based on the completed tests and the live test.