Data sample template (DST) management for enabling fog-based data processing
A Data Sample Collection Service (DSCS) may be configured to receive various data sample collection requests from users and to process the one or more requests on behalf of the users. In order to properly describe what kind of Ready-to-Use Data Samples (RDS) a user needs, the DSCS may adopt a Data Sample Template (DST). By using the DST, the user can clearly depict what their desired data samples look like, along with various quality requirements regarding the data samples. The DSCS can further provide a DST Management function to users. In general, DST management involves the processes of DST creation, DST update, DST deletion and DST activation/de-activation. For example, a user may create a DST based on its data sample collection needs. Later, the created DST can also be updated and/or deleted based on dynamic changes to the needs of the user.
1. A method performed at a first agent of a data sample collection service, the method comprising:
receiving one or more requests to create a data sample template, the one or more requests to create the data sample template comprising information associated with one or more data elements, wherein the one or more requests to create the data sample template comprises an indication of a type of data sample to be created and an indication of a second agent of the data sample collection service associated with the type of data sample, the second agent is located on a local fog node, and a data processing operation is performed by the second agent on the one or more data elements based on the type of data sample;
creating, based on the information associated with the one or more data elements, the data sample template comprising the indication of the type of data sample to be created and the indication of the second agent on the local fog node;
sending, to the second agent on the local fog node, the data sample template to identify one or more data sources associated with the one or more data elements; and
receiving, from the second agent on the local fog node based on the data sample template, data samples associated with the one or more data sources.
2. The method of claim 1 , wherein the one or more requests to create the data sample template further comprises an indication of one or more parameters, the one or more parameters comprising:
a targeted region associated with the data sample;
a frequency associated with the data sample;
a production schedule associated with the data sample; and
a context associated with the data sample.
3. The method of claim 2 , further comprising determining, based on the one or more parameters, which of a plurality of local fog nodes to target.
4. The method of claim 1 , wherein the information associated with the one or more data elements comprises one or more of:
a raw data type of the data element;
a unit of the data element;
a data processing operation of the data element;
one or more customized processing details of the data element; and
one or more quality requirements of the data element.
5. The method of claim 1 , wherein the second agent on the local fog node is configured to generate, based on the configured data sample template, a ready to use data sample.
6. An apparatus comprising a processor and a memory, the memory storing computer-executable instructions which, when executed by the processor, cause the apparatus to implement a first agent of a data sample collection service, the first agent of the data sample collection service being configured to perform operations comprising:
receiving one or more requests to create a data sample template, the one or more requests to create the data sample template comprising information associated with one or more data elements, wherein the one or more requests to create the data sample template comprises an indication of a type of data sample to be created and an indication of a second agent of the data sample collection service associated with the type of data sample, wherein the second agent is located on a local fog node, and a data processing operation is performed by the second agent on the one or more data elements based on the type of data sample;
creating, based on the information associated with the one or more data elements, the data sample template comprising the indication of the type of data sample to be created and the indication of the second agent on the local fog node;
sending, to the second agent on the local fog node, the data sample template to identify one or more data sources associated with the one or more data elements; and
receiving, from the second agent on the local fog node based on the data sample template, data samples associated with the one or more data sources.
7. The apparatus of claim 6 , wherein the one or more requests to create the data sample template further comprises an indication of one or more parameters, the one or more parameters comprising:
a targeted region associated with the data sample;
a frequency associated with the data sample;
a production schedule associated with the data sample; and
a context associated with the data sample.
8. The apparatus of claim 7 , wherein the instructions, when executed, further cause the first agent of the data sample collection service to perform operations comprising determining, based on the one or more parameters, which of a plurality of local fog nodes to target.
9. The apparatus of claim 6 , wherein the information associated with the one or more data elements comprises one or more of:
a raw data type of the data element;
a unit of the data element;
a data processing operation of the data element;
one or more customized processing details of the data element; and
one or more quality requirements of the data element.
10. The apparatus of claim 6 , wherein the second agent on the local fog node is configured to generate, based on the configured data sample template, a ready to use data sample.
11. A computer-readable storage medium storing instructions which, when executed by a processor, cause a first agent of a data sample collection service to perform operations comprising:
receiving a request to create a data sample template, the one or more requests to create the data sample template comprising information associated with one or more data elements, wherein the one or more requests to create the data sample template comprises an indication of a type of data sample to be created and an indication of a second agent of the data sample collection service associated with the type of data sample, wherein the second agent is located on a local fog node, and a data processing operation is performed by the second agent on the one or more data elements based on the type of data sample;
creating, based on the information associated with the one or more data elements, the data sample template comprising the indication of the type of data sample to be created and the indication of the second agent on the local fog node;
sending, to the second agent on the local fog node, the data sample template to identify one or more data sources associated with the one or more data elements;
receiving, from the second agent on the local fog node based on the data sample template data samples associated with the one or more data sources.
12. The computer-readable storage medium of claim 11 , wherein the one or more requests to create the data sample template further comprises an indication of one or more parameters, the one or more parameters comprising:
a targeted region associated with the data sample;
a frequency associated with the data sample;
a production schedule associated with the data sample; and
a context associated with the data sample.
13. The computer-readable storage medium of claim 12 , wherein the instructions, when executed, further cause the first agent of the data sample collection service to perform operations comprising determining, based on the one or more parameters, which of a plurality of local fog nodes to target.
14. The computer-readable storage medium of claim 11 , wherein the information associated with the one or more data elements comprises one or more of:
a raw data type of the data element;
a unit of the data element;
a data processing operation of the data element;
one or more customized processing details of the data element; and
one or more quality requirements of the data element.