Flexible synchronous file system replication
Embodiments include an apparatus for replicating a file system (FS) that stores a file in a first location in a first data storage device, the apparatus comprising a query circuit configured to generate a spatial query that identifies a primary directory tree in the FS, and to receive a response to the query from the FS, where the response identifies a range of the primary directory tree in which a changed file may be found; a scan filter circuit configured to generate a scan filter based on the response, where the scan filter defines a portion of the primary directory tree that is smaller than the primary directory tree to search for a changed file; and a replication circuit configured to scan the portion of the primary directory tree, and upon detecting a changed file, replicate the changed file from the first location to a different location.
1. A non-transitory computer-readable storage device storing computer-executable instructions that when executed by a computer cause the computer to perform a method for replicating data in a file system (FS) that stores a file in a first location in a first data storage device, the method comprising:
generating a query that identifies a primary directory tree in the FS, where the FS includes a metadata archive, where the metadata archive includes a spatial database, and where the metadata archive stores metadata about the file stored in the FS in the spatial database;
querying the metadata archive with the query;
receiving a response to the query from the metadata archive, where the response indicates a range of the primary directory tree in which a changed file may be found;
generating a scan filter based, at least in part, on the response to the query from the metadata archive, where the scan filter defines a portion of the primary directory tree to search for a changed file, where the portion of the primary directory tree is smaller than the primary directory tree;
scanning the portion of the primary directory tree defined by the scan filter for a changed file; and
upon detecting a changed file:
replicating the changed file from the first location to a second, different location.
2. The non-transitory computer-readable storage device of claim 1 , where scanning the portion of the primary directory tree defined by the scan filter for a changed file includes walking less than the entire primary directory tree.
3. The non-transitory computer-readable storage device of claim 1 , where the FS has a journal, and where the metadata archive is updated synchronously in real time by the journal.
4. The non-transitory computer-readable storage device of claim 1 , where the query that identifies the primary directory tree in the FS is a spatial query.
5. The non-transitory computer-readable storage device of claim 1 , where the query that identifies the primary directory tree in the FS further identifies a time window within which to search for a changed file.
6. The non-transitory computer-readable storage device of claim 1 , where querying the metadata archive includes querying the metadata archive using Farey sequence encoded information as an index.
7. The non-transitory computer-readable storage device of claim 1 , where replicating the changed file includes copying the changed file from the first location in the first data storage device to a second, different location in a second, different data storage device.
8. The non-transitory computer-readable storage device of claim 1 , where replicating the changed file includes copying the changed file from the first location in the first data storage device to a second, different location in the first data storage device.
9. The non-transitory computer-readable storage device of claim 1 , the method further comprising:
upon detecting that the changed file is subject to a cool-down condition:
upon detecting that the cool-down condition has been met:
replicating the changed file from the first location to a second, different location.
10. A non-transitory computer-readable storage device storing computer-executable instructions that in response to execution cause a file system replication system to perform operations, the operations comprising:
generating a spatial query that identifies a primary directory tree in a file system (FS) in which to search for a changed file, where the FS stores a file in a first location in a first data storage device, where the FS includes a metadata archive, where the metadata archive includes a spatial database, and where the metadata archive stores metadata about a file stored in the FS in the spatial database;
querying the metadata archive with the spatial query;
receiving a response to the spatial query from the metadata archive, where the response indicates a range of the primary directory tree in which a changed file may be found;
generating a scan filter based, at least in part, on the response to the spatial query from the metadata archive, where the scan filter defines a portion of the primary directory tree to search for a changed file, where the portion of the primary directory tree is smaller than the primary directory tree;
scanning the portion of the primary directory tree defined by the scan filter for a changed file; and
upon detecting a changed file:
replicating the changed file to a second, different location.
11. The non-transitory computer-readable storage device of claim 10 , where scanning the portion of the primary directory tree defined by the scan filter for a changed file includes scanning less than the entire primary directory tree.
12. The non-transitory computer-readable storage device of claim 10 , where the FS has a journal, and where the metadata archive is updated synchronously in real time by the journal.
13. The non-transitory computer-readable storage device of claim 10 , where the query that identifies the primary directory tree in the FS in which to search for a changed file further identifies a time window within which search for a changed file.
14. The non-transitory computer-readable storage device of claim 10 , where querying the metadata archive includes querying the metadata archive using Farey sequence encoded information as an index.
15. The non-transitory computer-readable storage device of claim 10 , where replicating the changed file to a second different location includes copying the changed file from the first location in the first data storage device to a second, different location in the first data storage device, or copying the changed file from the first location in the first data storage device to a second, different location in a second, different data storage device.
16. The non-transitory computer-readable storage device of claim 10 , the operations further comprising:
upon detecting that the changed file is subject to a cool-down condition:
upon detecting that the cool-down condition has been met:
replicating the changed file from the first location to a second, different location.
17. An apparatus for replicating a file system (FS) that stores a file in a first location in a first data storage device, the apparatus comprising:
a memory configured to store a replication policy;
a processor;
a set of circuits; and
an interface that connects the memory, the processor, and the set of circuits, the set of circuits comprising:
a query circuit configured to generate a spatial query that identifies a primary directory tree in the FS in which to search for a changed file, where the query circuit generates the spatial query based, at least in part, on the replication policy, and
where the query circuit is further configured to receive a response to the spatial query from the FS, where the response identifies a range of the primary directory tree in which a changed file may be found;
a scan filter circuit configured to:
generate a scan filter based, at least in part, on the response to the spatial query, where the scan filter defines a portion of the primary directory tree to search for a changed file, where the portion of the primary directory tree is smaller than the primary directory tree; and
a replication circuit configured to:
scan the portion of the primary directory tree defined by the scan filter for a changed file, and
upon detecting a changed file:
replicate the changed file from the first location to a second, different location.
18. The apparatus of claim 17 , where scanning the portion of the primary directory tree defined by the scan filter for a changed file includes walking less than the entire primary directory tree.
19. The apparatus of claim 17 , where replicating the changed file from the first location to a second, different location includes copying the changed file from the first location in the first data storage device to a second, different location in the first data storage device, or copying the changed file from the first location in the first data storage device to a second, different location in a second, different data storage device.
20. The apparatus of claim 17 , the set of circuits further comprising a cool-down circuit configured to:
upon the replication circuit detecting a changed file:
detect if the changed file is subject to a cool-down condition; and
upon detecting that the changed file is subject to a cool-down condition:
upon detecting that the cool-down condition has been met:
controlling the replication circuit to replicate the changed file from the first location to a second, different location.