Data aggregation with self-configuring drivers
View Patent ↗A data aggregation implementation includes self-configuring drivers. From the viewpoint of a Network Operation Center (NOC), a plurality of heterogenous content sources provide content that may be of a variety of different types and formats. All of this content must be ingested and stored for retrieval and reporting, analysis, and/or presentation despite many differences in their collection, format, transmission, and quality. In some embodiments, the NOC includes or cooperates with one or more servers to, among other functions, receive content from content sources, request object reflection by the driver of each content source, receive driver attributes in response, and map the metadata of the content for each content source to a universal schema, thereby self-configuring the driver.
1. One or more non-transitory computer-readable storage media storing computer-executable instructions that, if executed by one or more processors, cause the one or more processors to perform acts comprising:
sending a request to a driver to perform object reflection to obtain attribute information on at least one software object associated with the driver;
receiving from the driver the attribute information obtained by the reflection in accordance with the request;
mapping the received attribute information to corresponding fields in a schema such that data received corresponding to each attribute is configured to be stored to its respective corresponding field thereby self-configuring the driver,
wherein the mapping of the received attribute information to corresponding fields in a schema is based at least on a lookup table indicating likely synonyms of field attributes of objects from the driver to the fields in the schema; and
marking the driver as configured.
2. The one or more non-transitory computer-readable storage media of claim 1 , wherein the driver is programmed with an object-oriented programming language that supports objects having attributes.
3. The one or more non-transitory computer-readable storage media of claim 1 , wherein the lookup table is generated from a machine learning algorithm based on prior successful mappings.
4. The one or more non-transitory computer-readable storage media of claim 1 , wherein the acts further comprise:
receiving one or more data records from the driver; and
storing the received one or more data records from the driver into the schema according to the mapping of the attributes to the fields in the schema.
5. The one or more non-transitory computer-readable storage media of claim 4 , wherein the one or more data records have respective fields of metadata describing packetized data corresponding to the respective data record, and the metadata includes, for each of the one or more data records, a datatype, a name of each field, and a size of each field.
6. The one or more non-transitory computer-readable media of claim 5 , wherein the acts further comprise streaming the packetized data,
wherein the packetized data comprises sensor data configured from a single source, and the metadata includes time of data capture of the sensor data by the single source.
7. The one or more non-transitory computer-readable media of claim 5 , wherein a returning of a driver configuration information includes returning a default value for a field having no data that can be returned.
8. The one or more non-transitory computer-readable media of claim 7 , wherein the new driver configuration information includes a value to override the default value.
9. The one or more non-transitory computer-readable media of claim 1 , wherein the acts further comprise:
determining occurrence of an event;
formatting metadata in one or more data records with a schema in accordance with the new driver configuration information; and
outputting the formatted metadata in the one or more data records in response to the determining of the occurrence.
10. A computer-implemented method comprising:
sending a request to a driver to perform object reflection to obtain attribute information on at least one software object associated with the driver;
receiving from the driver the attribute information obtained by the reflection in accordance with the request;
mapping the received attribute information to corresponding fields in a schema such that data received corresponding to each attribute is configured to be stored to its respective corresponding field thereby self-configuring the driver,
wherein the mapping of the received attribute information to corresponding fields in a schema is based at least on a lookup table indicating likely synonyms of field attributes of objects from the driver to the fields in the schema; and
marking the driver as configured.
11. The computer-implemented method of claim 10 , wherein the driver is programmed with an object-oriented programming language that supports objects having attributes.
12. The computer-implemented method of claim 10 , wherein the lookup table is generated from a machine learning algorithm based on prior successful mappings.
13. The computer-implemented method of claim 10 , further comprising:
receiving one or more data records from the driver; and
storing the received one or more data records from the driver into the schema according to the mapping of the attributes to the fields in the schema.
14. The computer-implemented method of claim 13 , wherein the one or more data records have respective fields of metadata describing packetized data corresponding to the respective data record, and the metadata includes, for each of the one or more data records, a datatype, a name of each field, and a size of each field.
15. The computer-implemented method of claim 13 , wherein a returning of a driver configuration information includes returning a default value for a field having no data that can be returned.
16. The computer-implemented method of claim 15 , wherein the new driver configuration information includes a value to override the default value.
17. A system, comprising:
one or more processors; and
memory storing one or more computer-readable instructions that, if executed by the one or more processors, cause the one or more processors to perform acts comprising:
receiving, from a driver of a first content source, attribute information of the driver;
mapping the received attribute information to corresponding fields in a schema such that data received corresponding to each attribute is configured to be stored to its respective corresponding field, wherein the mapping of the received attribute information to corresponding fields in a schema is based at least on a lookup table indicating likely synonyms of field attributes of objects from the driver to the fields in the schema; and
self-configuring drivers.
18. The system of claim 17 , wherein:
the schema defines respective fields of metadata describing packetized data corresponding to one or more data records, the metadata including, for each of the one or more data records, a datatype, a name of each field, and a size of each field.
19. The system of claim 17 , wherein the acts further comprise:
modifying a driver configuration received from the driver of the first content source to fit the schema different from that of the driver configuration received from the driver of the first data source.
20. The system of claim 17 , wherein the lookup table is generated from a machine learning algorithm based on prior successful mappings.