Published file system and method
A designated orchestrator configured to publish multiple snapshot data versions configured to be exposed to designated reader server/s and thus allow for a multi-task data writing ability which is able to be simultaneously conducted while a previous data snapshot is being iterated.
1. A non-centralized published file system, comprising:
(i) at least one orchestrator designated to control a control plane (CP) of a data plane (DP) network,
(ii) a single writer server configured to perform a writing procedure to write data, run an operating system and control the orchestrator,
(iii) at least one storage media designated to host data produced by the writer server and accessible over the DP network, and
(iv) a plurality of reader servers designated to have a read-only access to the storage media over the DP network,
wherein the plurality of reader servers are not clustered nor configured with metadata that groups them with respect to the writer server, wherein the writing procedure configured to be conducted by the writer server results in at least one data snapshot version, wherein the operations of the plurality of reader servers are not locked by the writer server nor configured with respect to a metadata server, wherein the data written by the writer server is not required to be divided into blocks, wherein the orchestrator is configured to accept a publish command that allows the plurality of reader servers to access the DP network and expose the data snapshot version, wherein said at least one orchestrator is logically situated within the control plane and is the only feature of the network permitted to: access (a) the complete topology of the network; (b) access telemetry data of communications on the network; (c) define and amend policy relating to nodes on the network; and (d) allocate resources of the network,
wherein the said single writer server is the only feature in the network permitted to write and edit data hosted on the plurality of reader servers, and wherein new data version may be updated by the writer server and be hidden from the reader server until another publish command is accepted by the orchestrator.
2. The system of claim 1 , wherein exposing a data snapshot over the DP network allows an immediate system response and reduced latency.
3. The system of claim 1 , wherein the at least one media storage is a read-only media storage stack.
4. The system of claim 1 , wherein the data is backed upon at least two media storages.
5. The system of claim 1 , wherein multiple data snapshot versions may be written while the reader server is exposed to another, already published data snapshot version over the DP network.
6. The system of claim 1 , wherein the orchestrator is configured to recognize a specific snapshot data version made by the writer server and thus, configured to provide a uniform and reliable data stack for multiple ones of the at least one reader server via the DP network.
7. The system of claim 1 , wherein upon publishing a new data snapshot version by the orchestrator, the at least one reader server is configured to refresh the allocation metadata.
8. The system of claim 1 , wherein f the system is configured to be utilized in order to perform an artificial intelligence (AI) training conducted by the at least one reader server.
9. The system of claim 1 , wherein the path to the media storage is direct and wherein each media storage is configured to run required storage stack services.
10. The system of claim 1 , wherein the reader server is configured to cache metadata and data in random access memory (RAM) in order to provide reduced latency.
11. The system of claim 1 , further configured to use a logical volume manager (LVM) in order to create a data snapshot version and enable parallel read by the at least one reader server using a single writer server.
12. The system of claim 11 , further configured to use a thin provisioning layer in order to create a data snapshot and enable parallel read by the at least one reader server using a single writer server.
13. The system of claim 1 , wherein the writer server is configured to interact with at least two reader servers using a multipath connection.
14. The system of claim 1 , wherein the communication between either a reader or a writer server accessible via the DP network, and the orchestrator is done using a designated software component installed on each of said devices.
15. The system of claim 1 , wherein the writer server is configured to utilize a redundant array of independent disk (RAID) storage stack component (SSC) configured to provide data redundancy originated from multiple designated portions of the storage media.
16. The system of claim 1 , wherein the at least one reader server comprises at least two reader servers located at different locations, wherein the orchestration is allocated across different resiliency domains.
17. The system of claim 16 , wherein the orchestration allocated while considering different resiliency domains, and further considers maintenance of system balance.
18. The system of claim 1 , wherein the orchestrator is configured to interact with each server using an administration protocol.
19. The system of claim 1 , wherein the operations on each server may be implemented, wholly or partially, by a data processing unit (DPU).
20. A method for operating a non-centralized published file system, comprising the following steps:
(i) using at least one orchestrator designated to control a control plane (CP) of a data plane (DP) network,
(ii) configuring a single writer server to perform a writing procedure to write data, run an operating system and control the orchestrator,
(iii) designating at least one storage media to host data produced by the configured writer server and having it accessible over the DP network, and
(iv) designating a plurality of reader servers to have a read-only access to the storage media over the DP network,
wherein the configuring of the writing procedure is to be conducted by the writer server is to result in at least one data snapshot version,
wherein the plurality of reader servers are neither clustered nor configured with metadata that groups them with respect to the writer server, wherein the designation of the orchestrator is configured to accept a publish command that allows the plurality of reader servers to access the DP network and expose the data snapshot version, wherein the operations of the plurality of reader servers are not locked by the writer server nor configured with respect to a metadata server,
wherein the data written by the single writer server is not required to be divided into blocks,
wherein the orchestrator is configured to accept a publish command that allows the plurality of reader servers to access the DP network and expose the data snapshot version, wherein said at least one orchestrator is logically situated within the control plane and is the only feature of the network permitted to: access (a) the complete topology of the network; (b) access telemetry data of communications on the network; (c) define and amend policy relating to nodes on the network; and (d) allocate resources of the network, and
wherein updating of a new data version may be obtained by the writer server and be hidden from the reader server until another publish command is accepted by the orchestrator.