IP Library › Granted Patent US 12,412,189
Granted Patent B2
US 12,412,189 · App. 18/213,651 · Granted Sep 9, 2025

Predictively and dynamically updating content using nested A/B testing

Inventors: Ajay Jain (San Jose, CA); Eric Kienle (Portland, OR)
Assignee: Adobe Inc.
G06Q30/0251
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,412,189
App. No.
18/213,651
Granted
Sep 9, 2025
Kind
B2
Abstract

Systems and methods for predictively and dynamically updating content using nested A/B testing are provided. In certain embodiments, a system may assign a content item to a dynamic content portion of a communication. The communication may be sent to a first plurality of recipients. The system may receive information about interactions of a second plurality of recipients with the communication, wherein the second plurality of recipients is a subset of the first plurality of recipients. The system can determine a first conversion metric. The system can then compare the first conversion metric to a conversion metric threshold and based on the comparison, modify the dynamic content portion of the communication to generate a modified communication. The modified communication may be sent to a third plurality of recipients, wherein the third plurality of recipients is a subset of the first plurality of recipients not including the second plurality of recipients.

Claims (81)

1. A method, comprising:

assigning a first content item to a dynamic content portion of a communication, wherein a first network location of the first content item is mapped to the dynamic content portion to cause the first content item to be loaded from the first network location;

sending a copy of the communication to a first plurality of recipients;

receiving information about interactions of a second plurality of recipients with the copy of the communication, wherein the second plurality of recipients is a subset of the first plurality of recipients;

determining, based on the interactions of the second plurality of recipients, a first conversion metric, wherein:

the first conversion metric is not determined until an interaction count of the second plurality of recipients exceeds an interaction count threshold based on a predetermined level of statistical confidence; and

the first conversion metric indicates a portion of the second plurality of recipients that have performed one or more conversion interactions;

comparing the first conversion metric to a conversion metric threshold;

based on the comparing the first conversion metric to the conversion metric threshold, assigning a second content item to the dynamic content portion of the communication, to generate a first modified communication, wherein a second network location of the second content item is mapped to the dynamic content portion to cause the second content item to be loaded from the second network location when the first modified communication is rendered; and

sending a copy of the first modified communication to a third plurality of recipients, wherein the third plurality of recipients is a subset of the first plurality of recipients not including the second plurality of recipients.

2. The method of claim 1 , wherein:

the communication comprises a plurality of dynamic content portions, wherein each of the dynamic content portions includes at least one content item; and

generating the first modified communication comprises modifying at least one of the plurality of dynamic content portions of the communication.

3. The method of claim 1 , wherein modifying the dynamic content portion of the communication includes at least one of:

updating a first mapping to the first content item with a second mapping to the second content item;

updating an assignment of the first content item to the dynamic content portion with an assignment of the second content item to the dynamic content portion of the communication;

modifying the first content item; or

removing the first content item.

4. The method of claim 1 , wherein each recipient of the third plurality of recipients has not yet interacted with the copy of the communication.

5. The method of claim 4 , further comprising:

receiving information about interactions of a fourth plurality of recipients with the copy of the first modified communication;

determining, based on the interactions of the fourth plurality of recipients, a second conversion metric, wherein the second conversion metric indicates a portion of the fourth plurality of recipients that have performed one or more conversion interactions;

comparing the second conversion metric to the conversion metric threshold;

based on the comparing the second conversion metric to the conversion metric threshold, modifying the dynamic content portion of the first modified communication, to generate a second modified communication; and

sending a copy of the second modified communication to a fifth plurality of recipients, wherein the fifth plurality of recipients is a subset of the third plurality of recipients not including the fourth plurality of recipients.

6. The method of claim 1 , wherein the first conversion metric is compared to the conversion metric threshold after a fixed time interval.

7. The method of claim 1 , wherein the first plurality of recipients is determined based on one or more details about each of the recipients of the first plurality of recipients.

8. The method of claim 1 , wherein the third plurality of recipients is determined based on one or more details about each of the recipients of the second plurality of recipients and the interactions of the second plurality of recipients with the copy of the communication.

9. A system, comprising:

a processing device; and

a memory device that includes instructions executable by the processing device for causing the processing device to perform operations comprising:

assigning a first content item to a dynamic content portion of a communication, wherein a first network location of the first content item is mapped to the dynamic content portion to cause the first content item to be loaded from the first network location;

sending a copy of the communication to a first plurality of recipients;

receiving information about interactions of a second plurality of recipients with the copy of the communication, wherein the second plurality of recipients is a subset of the first plurality of recipients;

determining, based on the interactions of the second plurality of recipients, a first conversion metric, wherein:

the first conversion metric is not determined until an interaction count of the second plurality of recipients exceeds an interaction count threshold based on a predetermined level of statistical confidence; and

the first conversion metric indicates a portion of the second plurality of recipients that have performed one or more conversion interactions;

comparing the first conversion metric to a conversion metric threshold;

based on the comparing the first conversion metric to the conversion metric threshold, assigning a second content item to the dynamic content portion of the communication, to generate a first modified communication, wherein a second network location of the second content item is mapped to the dynamic content portion to cause the second content item to be loaded from the second network location when the first modified communication is rendered; and

sending a copy of the first modified communication to a third plurality of recipients, wherein:

the third plurality of recipients is a subset of the first plurality of recipients not including the second plurality of recipients; and

each recipient of the third plurality of recipients has not yet interacted with the copy of the communication.

10. The system of claim 9 , wherein modifying the dynamic content portion of the communication includes at least one of:

updating a first mapping to the first content item with a second mapping to the second content item;

updating an assignment of the first content item to the dynamic content portion with an assignment of the second content item to the dynamic content portion of the communication;

modifying the first content item; or

removing the first content item.

11. The system of claim 10 , wherein the second content item is determined by a machine learning model based on one or more details about each of the recipients of the third plurality of recipients.

12. The system of claim 9 , further comprising:

receiving information about interactions of a fourth plurality of recipients with the copy of the first modified communication;

determining, based on the interactions of the fourth plurality of recipients, a second conversion metric, wherein the second conversion metric indicates a portion of the fourth plurality of recipients that have performed one or more conversion interactions;

comparing the second conversion metric to the conversion metric threshold;

based on the comparing the second conversion metric to the conversion metric threshold, modifying the dynamic content portion of the first modified communication, to generate a second modified communication; and

sending a copy of the second modified communication to a fifth plurality of recipients, wherein the fifth plurality of recipients is a subset of the third plurality of recipients not including the fourth plurality of recipients.

13. The system of claim 9 , wherein the first plurality of recipients is determined based on one or more details about each of the recipients of the first plurality of recipients.

14. The system of claim 9 , wherein the third plurality of recipients is determined by a machine learning model based the interactions of the second plurality of recipients with the copy of the communication.

15. A non-transitory computer-readable medium comprising instructions that are executable by a processing device for causing the processing device to perform operations comprising:

assigning a first content item to a dynamic content portion of a communication, wherein a first network location of the first content item is mapped to the dynamic content portion to cause the first content item to be loaded from the first network location;

sending a copy of the communication to a first plurality of recipients;

receiving information about interactions of a second plurality of recipients with the copy of the communication, wherein the second plurality of recipients is a subset of the first plurality of recipients;

determining, based on the interactions of the second plurality of recipients, a first conversion metric, wherein:

the first conversion metric is not determined until an interaction count of the second plurality of recipients exceeds an interaction count threshold based on a predetermined level of statistical confidence; and

the first conversion metric indicates a portion of the second plurality of recipients that have performed one or more conversion interactions;

comparing the first conversion metric to a conversion metric threshold;

based on the comparing the first conversion metric to the conversion metric threshold, assigning a second content item to the dynamic content portion of the communication, to generate a first modified communication, wherein a second network location of the second content item is mapped to the dynamic content portion to cause the second content item to be loaded from the second network location when the first modified communication is rendered; and

sending a copy of the first modified communication to a third plurality of recipients, wherein:

the third plurality of recipients is a subset of the first plurality of recipients not including the second plurality of recipients; and

each recipient of the third plurality of recipients has not yet interacted with the copy of the communication.

16. The non-transitory computer-readable medium of claim 15 , wherein modifying the dynamic content portion of the communication includes at least one of:

updating a first mapping to the first content item with a second mapping to the second content item;

updating an assignment of the first content item to the dynamic content portion with an assignment of the second content item to the dynamic content portion of the communication;

modifying the first content item; or

removing the first content item.

17. The non-transitory computer-readable medium of claim 15 , wherein:

the third plurality of recipients is determined, using a machine learning model, based on one or more details about each of the recipients of the second plurality of recipients and the interactions of the second plurality of recipients with the copy of the communication.

18. The non-transitory computer-readable medium of claim 15 , further comprising:

receiving information about interactions of a fourth plurality of recipients with the copy of the first modified communication;

determining, based on the interactions of the fourth plurality of recipients, a second conversion metric, wherein the second conversion metric indicates a portion of the fourth plurality of recipients that have performed one or more conversion interactions;

comparing the second conversion metric to the conversion metric threshold;

based on the comparing the second conversion metric to the conversion metric threshold, modifying the dynamic content portion of the first modified communication, to generate a second modified communication; and

sending a copy of the second modified communication to a fifth plurality of recipients, wherein the fifth plurality of recipients is a subset of the third plurality of recipients not including the fourth plurality of recipients.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 23, 2023
From: JAIN, AJAY; KIENLE, ERIC
To: ADOBE INC.
Reel/Frame 064046/0628 →
Continuity (1)
Related Publication 20240428291A1 · Dec 26, 2024
References Cited (19)
US 20150025959A1 · Raghuveer · 2015 [cited by examiner]
US 20150227983A1 · Eustace · 2015 [cited by examiner]
US 20160117740A1 · Linden · 2016 [cited by examiner]
US 20170178179A1 · Bauman · 2017 [cited by examiner]
US 20170323331A1 · Malek · 2017 [cited by examiner]
US 20190087852A1 · Blom · 2019 [cited by examiner]
US 20190236621A1 · Liu · 2019 [cited by examiner]
US 20200272791A1 · Jonnalagadda · 2020 [cited by examiner]
US 20210142357A1 · Truong · 2021 [cited by examiner]
US 20210390578A1 · Calabrese · 2021 [cited by examiner]
WO WO2016130803A1 · 2016 [cited by examiner]
Singh, Nripendra, et al. “A/B Testing and Audience Creation for Effective Digital Marketing: Evidences from Facebook Analytics.” 2023 6th International Conference on Information Systems and Computer Networks (ISCON). IE… [cited by examiner]
Kumar, Saurabh. “The Role of A/B Testing in Advancing Marketing Analytics: A Systematic Approach.” Journal of Scientific and Engineering Research 8.12 (2021): 323-330. (Year: 2021). [cited by examiner]
Content Marketing Institute, B2B Content Marketing, “2016 Benchmarks, Budget, and Trends—North America” Available online at https://contentmarketinginstitute.com/, 2016, pp. 1-32. [cited by applicant]
Zoominfo, “It's Our Business to Grow Yours”, available online at https://www.zoominfo.com/, 2023, 10 pages. [cited by applicant]
Adobe Experience Cloud, “Email Marketing”, Available at https://business.adobe.com/products/marketo/email-marketing.html, 2023, 9 pages. [cited by applicant]
Adobe Experience Cloud, “Predictive Content”, Available at https://business.adobe.com/products/marketo/predictive-content.html, 2023, 8 pages. [cited by applicant]
Adobe Experience Cloud Blog, “Lead Management—Definition Process, and Best Practices”, available online at https://business.adobe.com/blog/basics/lead-management, Apr. 26, 2023, 11 pages. [cited by applicant]
Adobe Experience Cloud, Adobe Marketo Engage, “Leave No Lead Behind With Advance Lead Management”, available online at https://business.adobe.com/products/marketo/lead-management.html, 2023 11 pages. [cited by applicant]