Processing and archiving data from edge nodes across distributed systems
A computer-implemented method, according to one approach, includes: detecting new data generated at a first edge node, and causing a first copy of the new data to be transferred to a central data storage location in real-time. A second copy of the new data is also stored in a first location at the first edge node. In response to a predetermined condition being met, a determination is made as to whether any information in the first copy of the new data transferred is missing at the central data storage location. The first copy of the new data is removed from the first location at the first edge node in response to determining that no information in the first copy of the new data transferred is missing at the central data storage location.
1. A computer-implemented method, comprising:
detecting new data generated at a first edge node;
causing a first copy of the new data to be transferred to a central data storage location in real-time;
causing a second copy of the new data to be stored in a first location at the first edge node;
sending, to the central data storage location, a request to determine whether a first storage location at the central data storage location includes a mirrored duplicate of the first copy of the new data; and
in response to receiving a determination that the first storage location includes a mirrored duplicate of the first copy of the new data, releasing the second copy of the new data from the first edge node.
2. The computer-implemented method of claim 1 , wherein the computer-implemented method is performed by an archive handler, wherein the archive handler includes a set of observers and a remover.
3. The computer-implemented method of claim 2 , wherein the detecting new data generated at the first edge node includes an observer from the set of observers identifying the new data being written into a folder that corresponds to the observer, wherein the first copy of the new data is transferred to the central data storage location in real-time by the observer, wherein the second copy of the new data is stored in the first location at the first edge node by the observer.
4. The computer-implemented method of claim 3 , comprising:
in response to a predetermined condition being met, determining whether any information in the first copy of the new data transferred is missing at the central data storage location;
in response to determining that no information in the first copy of the new data transferred is missing at the central data storage location, causing the second copy of the new data to be removed from the first location at the first edge node;
in response to determining that at least some information in the first copy of the new data is missing at the central data storage location, transferring, by the observer, the missing information to the central data storage location; and
removing the first copy of the new data from the first location in response to transferring the missing information to the central data storage location.
5. The computer-implemented method of claim 2 , wherein the archive handler is configured to manage an edge file hierarchy at the first edge node.
6. The computer-implemented method of claim 5 , wherein the observers in the set of observers correspond to respective folders at the first edge node, wherein the new data generated at the first edge node is detected in response to receiving a notification from the observer.
7. The computer-implemented method of claim 5 , wherein releasing the second copy of the new data to be removed from the first location at the first edge node includes sending one or more instructions to the remover, the one or more instructions causing the remover component to remove the second copy of the new data from the first location.
8. The computer-implemented method of claim 4 , wherein meeting the predetermined condition includes maintaining the second copy of the new data in the first location at the first edge node for a first predetermined amount of time.
9. The computer-implemented method of claim 8 , wherein the first predetermined amount of time is based at least in part on a type and/or amount of information included in the second copy of the new data.
10. A computer program product, comprising a computer readable storage medium having program instructions embodied therewith, the program instructions readable by a processor, executable by the processor, or readable and executable by the processor, to cause the processor to:
detect new data generated at a first edge node;
cause a first copy of the new data to be transferred to a central data storage location in real-time;
cause a second copy of the new data to be stored in a first location at the first edge node;
send, to the central data storage location, a request to determine whether a first storage location at the central data storage location includes a mirrored duplicate of the first copy of the new data; and
in response to receiving a determination that the first storage location includes a mirrored duplicate of the first copy of the new data, release the second copy of the new data from the first edge node.
11. The computer program product of claim 10 , wherein the central data storage location is configured to receive and store one or more additional copies of data received from one or more additional edge nodes, wherein the central data storage location is configured to maintain real-time mirrored copies of data that is received at the respective one or more additional edge nodes.
12. The computer program product of claim 10 , wherein the detecting new data generated at the first edge node includes an observer identifying the new data being written into a folder that corresponds to the observer, wherein the first copy of the new data is transferred to the central data storage location in real-time by the observer, wherein the second copy of the new data is stored in the first location at the first edge node by the observer, wherein the observer is from a set of observers in an archive handler.
13. The computer program product of claim 10 , wherein the program instructions are readable and/or executable by the processor to cause the processor to:
in response to a predetermined condition being met, determine whether any information in the first copy of the new data transferred is missing at the central data storage location;
in response to determining that no information in the first copy of the new data transferred is missing at the central data storage location, cause the second copy of the new data to be removed from the first location at the first edge node;
in response to determining that at least some information in the first copy of the new data is missing at the central data storage location, transfer the missing information to the central data storage location; and
remove the first copy of the new data from the first location in response to transferring the missing information to the central data storage location.
14. The computer program product of claim 12 , wherein the archive handler is configured to manage an edge file hierarchy at the first edge node, wherein the archive handler includes the set of observer components and a remover component.
15. The computer program product of claim 14 , wherein the observers in the set of observers correspond to respective folders at the first edge node, wherein the new data generated at the first edge node is detected in response to receiving a notification from a respective one of the observer components.
16. The computer program product of claim 14 , wherein causing the first copy of the new data to be removed from the first location at the first edge node includes sending one or more instructions to the remover component, the one or more instructions causing the remover component to remove the first copy from the first location.
17. The computer program product of claim 10 , wherein meeting the predetermined condition includes maintaining the first copy of the new data in the first location at the first edge node for a first predetermined amount of time.
18. The computer program product of claim 17 , wherein the first predetermined amount of time is based at least in part on a type and/or amount of information included in the first copy of the new data.
19. A system, comprising:
a first edge node connected to a central data storage location;
a processor; and
logic integrated with the processor, executable by the processor, or integrated with and executable by the processor, the logic being configured to:
detect new data generated at a first edge node;
cause a first copy of the new data to be transferred to a central data storage location in real-time;
cause a second copy of the new data to be stored in a first location at the first edge node;
send, to the central data storage location, a request to determine whether a first storage location at the central data storage location includes a mirrored duplicate of the first copy of the new data; and
in response to receiving a determination that the first storage location includes a mirrored duplicate of the first copy of the new data, release the second copy of the new data from the first edge node.
20. The system of claim 19 ,
wherein the central data storage location is configured to receive and store one or more additional copies of data received from one or more additional edge nodes,
wherein the central data storage location is configured to maintain real-time mirrored copies of data that is received at the respective one or more additional edge nodes.
21. A computer program product, comprising a computer readable storage medium having program instructions embodied therewith, the program instructions readable by a processor, executable by the processor, or readable and executable by the processor, to cause the processor to:
receive a first copy of data from a first edge node;
store the first copy of data in a first storage location at a central data storage location;
receive a second copy of data from a second edge node;
store the second copy of data in a second storage location at the central data storage location;
receive a request from the first edge node to determine whether the first storage location includes a mirrored duplicate of the first copy of data received from the first edge node; and
in response to determining that the first storage location includes a mirrored duplicate of the first copy of data, cause the first copy of data to be released from the first edge node by sending an acknowledgement to the first edge node.
22. The computer program product of claim 21 , wherein the program instructions are readable and/or executable by the processor to cause the processor to:
in response to determining that the first storage location does not include a mirrored duplicate of the first copy of data, issue a request to the first edge node for missing portions of the first copy of data;
receive the missing portions of the first copy of data; and
use the missing portions to complete the mirrored duplicate of the first copy of data at the first storage location.
23. The computer program product of claim 21 , wherein the program instructions are readable and/or executable by the processor to cause the processor to:
receive a request from the second edge node to determine whether the second storage location includes a mirrored duplicate of the second copy of data received from the second edge node; and
in response to determining that the second storage location includes a mirrored duplicate of the second copy of data, cause the second copy of data to be released from the second edge node by sending an acknowledgement to the second edge node.
24. The computer program product of claim 23 , wherein the program instructions are readable and/or executable by the processor to cause the processor to:
in response to determining that the second storage location does not include a mirrored duplicate of the second copy of data, issue a request to the second edge node for missing portions of the second copy of data;
receive the missing portions of the second copy of data; and
use the missing portions to complete the mirrored duplicate of the second copy of data at the second storage location,
wherein the first storage location is in non-volatile memory at the central data storage location, wherein the second storage location is in non-volatile memory at the central data storage location.
25. A system, comprising:
a central data storage location connected to a first edge node and a second edge node;
a processor; and
logic integrated with the processor, executable by the processor, or integrated with and executable by the processor, the logic being configured to:
receive a first copy of data from a first edge node;
store the first copy of data in a first storage location at the central data storage location;
receive a second copy of data from a second edge node;
store the second copy of data in a second storage location at the central data storage location;
receive a request from the first edge node to determine whether the first storage location includes a mirrored duplicate of the first copy of data received from the first edge node; and
in response to determining that the first storage location includes a mirrored duplicate of the first copy of data, cause the first copy of data to be released from the first edge node by sending an acknowledgement to the first edge node.