Data pipeline for managing and determining personalized data
Disclosed is a data pipeline for managing data for determining personalized data for many users. The pipeline can include a startup manager microservice that receives user input to begin batch processing user data, receives the user data in near real-time, and runs data definition language (DDL) and data manipulation language (DML) scripts at a data store to prepare the data store to be used by other pipeline microservices. The pipeline can include rule engines that generate personalized data based on the received user data, a metrics aggregation microservice that can receive the user data from the startup manager microservice and aggregate the user data in near real-time, a message repository microservice that can store, in the data store, messages generated by the rule engines using the corresponding message templates, and a synchronization microservice to provide communication between the pipeline and user devices.
1 . A system for managing data for determining personalized data for many users, wherein the personalized data indicates trends based on aggregated user data and aggregated historic data, wherein the personalized data comprises a score selected from the group consisting of a sleep score, a quality of sleep score, and a user health score, the system comprising one or more processors:
a startup manager microservice configured to:
receive user input to begin batch processing of user data;
receive the user data in real-time; and
run data definition language (DDL) and data manipulation language (DML) scripts at a data store to prepare the data store to be used by one or more other microservices;
one or more rule engines that generate personalized data based at least in part on the received user data, wherein the one or more rule engines are configured to:
import, by a rules importer microservice of the rule engine, one or more rules to the data store, wherein the one or more imported rules include message templates storing data in a machine-readable format from which readable messages can be generated for output generated by applying the one or more imported rules to the received user data;
parse, by a rules partitioning microservice of the rule engine, the one or more rules to identify the message templates corresponding to each of the one or more rules; and
store, by the rules partitioning microservice of the rule engine, the message templates and the parsed rules in persistent metadata;
a metrics aggregation microservice that is configured to receive the user data from a current sleep session from the startup manager microservice and aggregate the user data in real-time;
a historic import microservice configured to:
retrieve, from a data repository, historic data about a user's previous sleep sessions; and
aggregate the historic data;
a message repository microservice configured to store, in the data store, messages generated by the one or more rule engines using the corresponding message templates; and
a synchronization microservice configured to provide communication between the system and one or more user devices, wherein the messages generated by the one or more rule engines and stored by the message repository microservice are retrieved, by the synchronization microservice, and transmitted to the one or more user devices for presentation in a graphical user interface (GUI) display of the one or more user devices,
wherein each of the one or more rule engines is further configured to:
receive, from the metrics aggregation microservice, the aggregated user data;
receive, from the historic import microservice, the aggregated historic data;
determine, based on the aggregated user data and the aggregated historic data, whether one or more of the parsed rules are satisfied;
generate, based on determining whether one or more of the parsed rules are satisfied, personalized data for the user associated with the aggregated user data;
generate, based on determining whether one or more of the parsed rules are satisfied, messages to be presented at the one or more user devices using one or more of the corresponding message templates, wherein the messages include the generated personalized data; and
transmit the generated messages to the message repository microservice for storage.
2 . The system of claim 1 , wherein each of the one or more rule engines includes a rule file that indicates (i) a type of personalized data that is determined by the rule engine, (ii) criteria for determining the type of personalized data, and (iii) criteria for a type of output to generate using a corresponding message template.
3 . The system of claim 1 , wherein the message templates are used, by the one or more rule engines, to generate the messages in a readable format for users of the one or more user devices.
4 . The system of claim 1 , wherein at least one of the rule engines is configured to generate personalized data for the user that indicates monthly or weekly trends based on the aggregated user data and the aggregated historic data.
5 . The system of claim 1 , wherein at least one of the rule engines is a rule-based engine that is configured to generate monthly health quality scores for the user based on the aggregated user data and the aggregated historic data.
6 . The system of claim 1 , wherein at least one of the rule engines is a rule-based engine that is configured to generate monthly sleep quality scores for the user based on the aggregated user data and the aggregated historic data.
7 . The system of claim 1 , wherein at least one of the rule engines is artificial intelligence (AI)-based and is configured to:
generate messages about the personalized data for the user; and
prioritize, using models, trained with machine learning, and based on the personalized data, sharing of the messages with the one or more user devices.
8 . The system of claim 1 , wherein at least one of the rule engines is a hybrid of a rule-based and AI-based engine that is configured to generate personalized data about the user based on the aggregated user data and the aggregated historic data.
9 . The system of claim 1 , wherein the one or more rule engines are executed in parallel.
10 . The system of claim 1 , wherein each of the one or more rule engines is further configured to randomize delivery of the messages to the one or more user devices.
11 . The system of claim 1 , wherein the user data includes at least one of sleep data, health data, and biometrics data.
12 . The system of claim 1 , wherein the user data is detected by components of a bed system and transmitted to the startup manager microservice by a controller of the bed system.
13 . The system of claim 1 , wherein the metrics aggregation microservice is further configured to aggregate the user data on at least one of a weekend, weekday, daily, weekly, monthly, and yearly basis.
14 . The system of claim 1 , wherein the message repository microservice is configured to:
store the generated messages in the data store for a predetermined period of time; and
archive the generated messages once the predetermined period of time ends.
15 . The system of claim 14 , wherein the predetermined period of time is 3 months from at least one of collection of the user data and determination of the personalized data.
16 . The system of claim 1 , wherein one or more of the microservices of the system communicate by a simple queue service (SQS) message queue.
17 . The system of claim 1 , wherein the personalized data includes one or more suggestions for the user to improve at least one of their sleep and health.
18 . A system for managing data for determining personalized data for many users, wherein the personalized data indicates trends based on aggregated user data and aggregated historic data, wherein the personalized data comprises a score selected from the group consisting of a sleep score, a quality of sleep score, and a user health score, the system comprising one or more processors:
a startup manager microservice configured to:
receive user input to begin batch processing of user data;
receive the user data in near real-time; and
run data definition language (DDL) and data manipulation language (DML) scripts at a data store to prepare the data store to be used by one or more other microservices;
one or more rule engines that generate personalized data based at least in part on the received user data, wherein the one or more rule engines are configured to:
import, by a rules importer microservice of the rule engine, one or more rules to the data store, wherein the one or more imported rules include message templates for output generated by applying the one or more imported rules to the received user data;
parse, by a rules partitioning microservice of the rule engine, the one or more rules to identify the message templates corresponding to each of the one or more rules; and
store, by the rules partitioning microservice of the rule engine, the message templates and the parsed rules in persistent metadata;
a metrics aggregation microservice that is configured to receive the user data from the startup manager microservice and aggregate the user data in near real-time;
a historic import microservice configured to:
retrieve, from a data repository, historic data about a user that is associated with the user data; and
aggregate the historic data;
a message repository microservice configured to store, in the data store, messages generated by the one or more rule engines using the corresponding message templates; and
a synchronization microservice configured to provide communication between the system and one or more user devices, wherein the messages generated by the one or more rule engines and stored by the message repository microservice are retrieved, by the synchronization microservice, and transmitted to the one or more user devices for presentation in a graphical user interface (GUI) display of the one or more user devices,
wherein each of the one or more rule engines is further configured to:
receive, from the metrics aggregation microservice, the aggregated user data;
receive, from the historic import microservice, the aggregated historic data;
determine, based on the aggregated user data and the aggregated historic data, whether one or more of the parsed rules are satisfied;
generate, based on determining whether one or more of the parsed rules are satisfied, personalized data for the user associated with the aggregated user data;
generate, based on determining whether one or more of the parsed rules are satisfied, messages to be presented at the one or more user devices using one or more of the corresponding message templates, wherein the messages include the generated personalized data; and
transmit the generated messages to the message repository microservice for storage;
wherein each of the one or more rule engines includes a rule file that indicates (i) a type of personalized data that is determined by the rule engine, (ii) criteria for determining the type of personalized data, and (iii) criteria for a type of output to generate using a corresponding message template;
wherein (ii) further includes a time period associated with the personalized data, wherein the time period is at least one of a weekend, a day, a group of days, a week, a month, or a year.
19 . A system for managing data for determining personalized data for many users, wherein the personalized data indicates trends based on aggregated user data and aggregated historic data, wherein the personalized data comprises a score selected from the group consisting of a sleep score, a quality of sleep score, and a user health score, the system comprising one or more processors:
a startup manager microservice configured to:
receive user input to begin batch processing of user data;
receive the user data in near real-time; and
run data definition language (DDL) and data manipulation language (DML) scripts at a data store to prepare the data store to be used by one or more other microservices;
one or more rule engines that generate personalized data based at least in part on the received user data, wherein the one or more rule engines are configured to:
import, by a rules importer microservice of the rule engine, one or more rules to the data store, wherein the one or more imported rules include message templates for output generated by applying the one or more imported rules to the received user data;
parse, by a rules partitioning microservice of the rule engine, the one or more rules to identify the message templates corresponding to each of the one or more rules; and
store, by the rules partitioning microservice of the rule engine, the message templates and the parsed rules in persistent metadata;
a metrics aggregation microservice that is configured to receive the user data from the startup manager microservice and aggregate the user data in near real-time;
a historic import microservice configured to:
retrieve, from a data repository, historic data about a user that is associated with the user data; and
aggregate the historic data;
a message repository microservice configured to store, in the data store, messages generated by the one or more rule engines using the corresponding message templates; and
a synchronization microservice configured to provide communication between the system and one or more user devices, wherein the messages generated by the one or more rule engines and stored by the message repository microservice are retrieved, by the synchronization microservice, and transmitted to the one or more user devices for presentation in a graphical user interface (GUI) display of the one or more user devices,
wherein each of the one or more rule engines is further configured to:
receive, from the metrics aggregation microservice, the aggregated user data;
receive, from the historic import microservice, the aggregated historic data;
determine, based on the aggregated user data and the aggregated historic data, whether one or more of the parsed rules are satisfied;
generate, based on determining whether one or more of the parsed rules are satisfied, personalized data for the user associated with the aggregated user data;
generate, based on determining whether one or more of the parsed rules are satisfied, messages to be presented at the one or more user devices using one or more of the corresponding message templates, wherein the messages include the generated personalized data; and
transmit the generated messages to the message repository microservice for storage;
wherein the user data is received, by the startup manager microservice, in real-time during sleep sessions of the user.