Data objects in a distributed file system
Approaches for writing data objects to a distributed file system are described. In an example, a size of a data object to be created within a distributed file system may be determined based on a client request. Based on the size of the data object, data content of the data object is inserted into one of the object location table, object data table or a data container within the distributed file system. In an example, metadata corresponding to the data object may be written to an object location table.
1. A system comprising:
a processor; and
a non-transitory machine-readable storage medium comprising instructions executable on the processor to:
determine, in response to a receiving a client request, that a size of a first data object to be created, including first data content, is less than a first size threshold;
based on determining that the size of the first data object including the first data content is less than the first size threshold, insert the first data content of the first data object together with metadata of the first data object into an object location table, wherein the object location table stores metadata pertaining to a plurality of data objects within a distributed file system;
determine a second data object to be created within the distributed file system, wherein the second data object, including second data content, is of a size greater than or equal to the first size threshold;
write metadata of the second data object into the object location table; and
insert the second data content of the second data object into an object data table, wherein the metadata of the second data object written into the object location table comprises a reference to the object data table.
2. The system of claim 1 , wherein the instructions are executable on the processor to further:
set an in-progress flag in the object location table to true while inserting the second data content of the second data object into the object data table.
3. The system of claim 1 , wherein the object location table comprises a data presence flag indicating whether the first data content of the first data object is inserted into the object location table.
4. The system of claim 1 , wherein the size of the second data object is less than a second size threshold, and the instructions are executable on the processor to:
determine a third data object to be created within the distributed file system, wherein the third data object, including third data content, is of a size greater than or equal to the second size threshold and less than a third size threshold;
write metadata of the third data object into the object location table; and
insert the third data content of the third data object into a data container within the distributed file system.
5. The system of claim 4 , wherein the object location table comprises a record containing:
an object location field that points to a location of the third data content stored in the data container within the distributed file system, and
a metadata field containing the metadata of the third data object.
6. The system of claim 4 , wherein the instructions are executable on the processor to:
determine a fourth data object to be created within the distributed file system, wherein the fourth data object, including fourth data content, is of a size greater than or equal to the third size threshold;
write metadata of the fourth data object into the object location table;
insert the fourth data content of the fourth data object as a plurality of data chunks into a further data container within the distributed file system; and
update the object location table with a reference to a table of pointers that reference the plurality of data chunks corresponding to the fourth data object.
7. The system of claim 6 , wherein to update the object location table, the instructions are executable on the processor to update a pointer of the pointers concurrently with inserting a corresponding data chunk of the plurality of data chunks into the further data container.
8. A system comprising:
a processor; and
a non-transitory machine-readable storage medium comprising instructions executable on the processor to:
determine, in response to a receiving a client request, that a size of a first data object to be created, including data content, is less than a first size threshold;
based on determining that the size of the first data object including the data content is less than the first size threshold, insert the data content of the first data object together with metadata of the first data object into an object location table, wherein the object location table stores metadata pertaining to a plurality of data objects within a distributed file system; and
for respective data objects of the plurality of data objects that are of sizes greater than the first size threshold, store the data content of the respective data objects outside of the object location table,
wherein the object location table is stored in a main memory of a computing device of the distributed file system, and the object location table comprises a plurality of records containing the metadata of respective data objects of the plurality of data objects, and wherein the metadata of the first data object is inserted into a first field of a first record of the plurality of records, and the data content of the first data object is inserted into a second field of the first record.
9. The system of claim 8 , wherein the inserting of the data content together with the metadata of the first data object into the object location table comprises:
performing a single row update that writes the metadata of the first data object to a metadata field of a row of the object location table, and inserts the data content of the first data object into an object data field of the row.
10. A method comprising:
identifying, by a system comprising a hardware processor, a first data object and a second data object to be created within a distributed file system, the first data object including first data content, and the second data object including second data content;
determining, by the system, a first size of the first data object and a second size of the second data object, the second size greater than the first size; and
based on determining the first size of the first data object and the second size of the second data object:
writing, by the system, metadata of the first data object into a first row of an object location table, and inserting the first data content of the first data object into the first row of the object location table, and
writing, by the system, metadata of the second data object into a second row of the object location table, and inserting the second data content of the second data object into a storage location outside the object location table.
11. The method of claim 10 , wherein the storage location outside the object location table comprises an object data table or a data container within the distributed file system, the method further comprising:
responsive to a completion of the inserting of the second data content into the object data table or the data container, setting an in-progress flag to indicate that writing of the second data content has completed.
12. The method of claim 11 , comprising:
detecting a client request seeking to access the second data content; and
returning the second data content based on a state of the in-progress flag.
13. The method of claim 11 , wherein the in-progress flag is set to a first state to indicate that the writing of the second data content has completed, and to a different second state to indicate that the writing of the second data content is in progress.
14. The method of claim 10 , wherein the first row comprises a metadata field and an object data field, the method comprising:
performing a single row update that writes the metadata of the first data object to the metadata field of the first row, and inserts the first data content of the first data object into the object data field of the first row.
15. The method of claim 10 , wherein:
the inserting of the first data content of the first data object into the first row of the object location table is based on the first size of the first data object being less than a first size threshold, and
the inserting of the second data content of the second data object into the storage location outside the object location table is based on the second size of the second data object being greater than the first size threshold.
16. The method of claim 15 , wherein the storage location comprises an object data table, and the second data content is inserted into the object data table based on the second size of the second data object being greater than or equal the first size threshold and less than a second size threshold, the method further comprising:
determining, by the system, that a third size of a third data object including third data content is greater than or equal to the second size threshold and less than a third size threshold; and
based on determining that the third size of the third data object is greater than or equal the second size threshold and less than the third size threshold, inserting, by the system, the third data content of the third data object into a data container within the distributed file system, and updating an object location field in a third row of the object location table with a reference to the data container.
17. The method of claim 10 , further comprising:
receiving, by the system, a request to access the first data object; and
returning, by the system, the first data content from the first row of the object location table as a response to the request.
18. A non-transitory computer-readable medium comprising instructions that, when executed on a processor, cause a system to:
determine a size of a data object, including data content, to be created within a distributed file system based on a client request; and
based on the size of the data object being less than a first size threshold:
perform a single row update in an object location table by writing metadata of the data object into a row of the object location table, and inserting the data content of the data object into the row of the object location table, wherein the object location table is to store metadata pertaining to a plurality of data objects within the distributed file system.
19. The non-transitory computer-readable medium of claim 18 , wherein the instructions when executed on the processor cause the system to:
determine a size of a second data object, including second data content, to be created within the distributed file system; and
based on the size of the second data object being greater than or equal the first size threshold and less than a second size threshold:
write metadata of the second data object into a second row of the object location table,
insert the second data content into a storage location outside the object location table, and
set an in-progress flag in the second row of the object location table to indicate that the second data content is in process of being inserted into the storage location outside the object location table.
20. The non-transitory computer-readable medium of claim 19 , wherein the storage location outside the object location table comprises an object data table, and the instructions when executed on the processor cause the system to:
determine a size of a third data object, including third data content, to be created within the distributed file system; and
based on the size of the third data object being greater than or equal the second size threshold and less than a third size threshold:
write metadata of the third data object into a third row of the object location table, and
insert the third data content of the third data object into a data container within the distributed file system.