Systems and methods for processing different data types

1. A computer-implemented method for data processing, comprising:

receiving data from a plurality of different data sources in canonical form;

provisioning one or more nodes to accommodate the data as the volume of data increases, and balancing the data among the one or more nodes;

partitioning the data of each of the one or more nodes into two or more different data types, wherein the two or more different data types comprise (1) a first type of data comprising structured or slow-changing data, and (2) a second type of data comprising dynamically streaming data;

storing the first type of data in a relational database and separately storing the second type of data in a non-relational database; and

applying different steps for processing the first and second types of data based on their types, wherein the different steps comprise:

applying a machine learning algorithm to the first and second types of data to identify energy meters associated with usage or consumption trends or abnormal events; and

using empirical data as supervised feedback to adjust the machine learning algorithm, wherein the empirical data comprises data generated from investigation of the energy meters by field personnel.

2. The method of claim 1 , wherein the non-relational database comprises a key/value store, and the streaming data comprises time-series data.

3. The method of claim 1 , wherein the structured or slow-changing data is stored only in the relational database, and the streaming data is stored only in the non-relational database.

4. The method of claim 1 , wherein a post-processing step is performed on the first type of data, wherein a normalization step is performed on the second type of data, and wherein the normalization step comprises filling in missing data points and addressing outliers in the second type of data.

5. The method of claim 1 , wherein the streaming data is derived from usage or consumption of a resource by one or more users.

6. The method of claim 1 , wherein the streaming data comprises high frequency data from a meter, sub-meter or grid sensor.

7. The method of claim 1 , wherein a validation step is performed on the received data prior to converting the data in accordance with a data model, wherein the validation step comprises examining a structure of the received data to ensure that (a) required fields are present and (b) that the received data matches at least one data type from the data model.

8. The method of claim 7 , wherein (i) data that is successfully validated by the validation step is converted in accordance with the data model, and (ii) data that is not successfully validated by the validation step is discarded without converting in accordance with the data model.

9. The method of claim 1 , wherein applying the machine learning algorithm to the first and second types of data comprises feature extraction for identifying the usage or consumption trends or the abnormal events.

10. The method of claim 9 , wherein applying the machine learning algorithm to the first and second types of data comprises feature classification in which different features are selectively merged or weighted, and elements are grouped to generate a set of follow-up opportunities.

11. The method of claim 10 , wherein applying the machine learning algorithm to the first and second types of data further comprises feature ranking in which the follow-up opportunities within the set are prioritized based on preferences and business operations of one or more customers.

12. The method of claim 1 , wherein the canonical form is based on industry standards or specifications.

13. The method of claim 1 , wherein the structured or slow-changing data comprises customer information, billing accounts and service agreements.

14. The method of claim 4 , wherein the post-processing step is performed directly on the first type of data in the relational database, and the normalization step is performed directly on the second type of data in the non-relational database.

15. The method of claim 1 , wherein the dynamically streaming data is received in streams, and wherein a data stream is persisted until the processing of the data stream is complete, at which time the processed data stream is discarded.

16. The method of claim 1 , wherein the two or more different data types are associated with conceptual models of attributes and processes related to different entities or domains.

17. The method of claim 1 , wherein the received data comprises a plurality of canonical objects having different type definitions and description.

18. The method of claim 17 , wherein the plurality of canonical objects are associated with an organization, facility, service, billing, usage-point, meter-reading, energy conservation measure, external benchmark, and/or region.

19. A system comprising:

a server in communication with a plurality of data sources; and

a memory comprising instructions stored thereon that, when executed by the server, perform operations comprising:

receiving data from the plurality of different data sources in canonical form;

provisioning one or more nodes to accommodate the data as the volume of data increases, and balancing the data among the one or more nodes;

partitioning the data of each of the one or more nodes into two or more different data types, wherein the two or more different data types comprises (1) a first type of data comprising of structured or slow-changing data, and (2) a second type of data comprising of dynamically streaming data;

storing the first type of data in a relational database and separately storing the second type of data in a non-relational database; and

applying different steps for processing the first and second types of data based on their types, wherein the different steps comprise:

applying a machine learning algorithm to the first and second types of data to identify energy meters associated with usage or consumption trends or abnormal events, and

using empirical data as supervised feedback to adjust the machine learning algorithm, wherein the empirical data comprises data generated from investigation of the energy meters by field personnel.

20. A non-transitory computer-readable storage medium comprising instructions that, when executed by a server, causes the server to perform a method for data processing, the method comprising:

receiving data from a plurality of different data sources in canonical form;

provisioning one or more nodes to accommodate the data as the volume of data increases, and balancing the data among the one or more nodes;

partitioning the data of each of the one or more nodes into two or more different data types, wherein the two or more different data types comprises (1) a first type of data comprising of structured or slow-changing data, and (2) a second type of data comprising of dynamically streaming data;

storing the first type of data in a relational database and separately storing the second type of data in a non-relational database; and

applying different steps for processing the first and second types of data based on their types, wherein the different steps comprise:

applying a machine learning algorithm to the first and second types of data to identify energy meters associated with usage or consumption trends or abnormal events, and

using empirical data as supervised feedback to train the machine learning algorithm, wherein the empirical data comprises data generated from investigation of the energy meters by field personnel.