Method and system for managing power grid data
A system and method of managing time-series data for smart grids is disclosed. Data is collected from a plurality of sensors. An index is modified for a newly created block. A one disk operation per read or write is performed. The one disk operation per read includes accessing and looking up the index to locate the data without movement of an arm of the disk, and obtaining the data. The one disk operation per write includes searching the disk for free space, calculating an offset, modifying the index, and writing the data contiguously into a block of the disk the index points to.
1. A method of managing time-series data for smart grids, comprising:
a. collecting data from a plurality of sensors;
b. modifying an index for a newly created block; and
c. performing a one disk operation per read or write.
2. The method of claim 1 further comprising adding a look-up capability to the index.
3. The method of claim 1 wherein the index is stored in at least one of the following: main memory of a local machine, main memory from a remote machine, a solid-state storage device (SSD) from the local machine, and the SSD from the remote machine.
4. The method of claim 1 wherein the performing a one disk operation per read comprises accessing and looking up the index to locate the data without movement of an arm of a disk, and obtaining the data.
5. The method of claim 1 wherein the performing a one disk operation per write comprises searching a disk for free space, calculating an offset, modifying the index, and writing the data contiguously into a block of the disk the index points to.
6. The method of claim 5 wherein the data is first written into a main memory buffer before being written into the disk.
7. The method of claim 1 wherein the collecting data from a plurality of sensors further comprises organizing the data contiguously in a disk.
8. The method of claim 7 wherein the data is reorganized contiguously in main memory before being written into the disk.
9. The method of claim 1 further comprising estimating a number of disk operations and staying below a maximum storage capacity.
10. The method of claim 1 further comprising distributing the data into multiple disks to enhance efficiency and reliability.
11. The method of claim 1 wherein the data is power grid data.
12. A system for scalable real time data management for smart grids comprising:
a. data collected from a plurality of sensors;
b. an index for data retrieval via one disk access;
c. a disk including data blocks for storage of the data; and
d. a device for caching the data, wherein the index is stored in the device.
13. The system of claim 12 wherein the data is collected in a time series format and organized contiguously in the disk.
14. The system of claim 12 wherein the index includes a look-up capability.
15. The system of claim 12 wherein the data is distributed into multiple disks to enhance efficiency and reliability.
16. The system of claim 12 wherein the index keeps track of stored data.
17. The system of claim 16 wherein the index keeps track of and looks up free disk space.
18. The system of claim 12 wherein the device is at least one of the following:
main memory of a local machine, main memory from a remote machine, a solid-state storage device (SSD) from the local machine, and the SSD from the remote machine.
19. The system of claim 12 wherein the system performs one disk operation per read or write.
20. The system of claim 12 wherein the data is reorganized contiguously in main memory before being written into the disk.
21. The system of claim 12 wherein the data is power grid data.
22. A method of managing time-series data for smart grids, comprising:
a. collecting data from a plurality of sensors;
b. organizing the data contiguously in a disk;
c. modifying an index for a newly created block;
d. adding a look-up capability to the index; and
e. estimating a number of disk operations and staying below a maximum storage capacity; wherein the data is collected in a time series format and first written into a main memory buffer before being written into the disk.