Efficient file system directory reverse name lookup
Techniques are disclosed for efficient file system directory reverse name lookup. An example method includes storing, at one or more of a plurality of storage locations of a key map, a file system directory including one or more key-value pairs, wherein each key-value pair of the file system directory maps a respective object identifier to corresponding object metadata; and in response to detecting an indication of a file system event for a file: storing, to a storage location of the key map, a key-value pair that maps an object identifier of the file to object metadata of the file; generating, based on the object identifier, an indication of the storage location; including, by the data platform, the indication of the storage location in the object metadata; and storing, by the data platform, the object metadata to a data store.
1 . A method comprising:
storing, by a data platform implemented by a computing system and at one or more of a plurality of storage locations of a key map, a file system directory including one or more key-value pairs, wherein each key-value pair of the file system directory maps a respective object identifier to corresponding object metadata, wherein the respective object identifier identifies a respective file of a plurality of files stored to the data platform;
detecting, by the data platform, an indication of a file system event for a file of the plurality of files; and
in response to detecting the indication of the file system event:
storing, by the data platform and to a storage location of the key map, a key-value pair that maps an object identifier of the file to object metadata of the file;
generating, by the data platform and based on the object identifier of the file, an indication of the storage location of the key map;
including, by the data platform, the indication of the storage location of the key map in the object metadata for the file, wherein the indication of the storage location of the key map is used to permit retrieval of the key-value pair that maps the object identifier of the file to the object metadata of the file using the object metadata for the file; and
storing, by the data platform, the object metadata for the file to a data store.
2 . The method of claim 1 , further comprising:
receiving, by the data platform, a request including an indication of the object metadata of the file;
retrieving, by the data platform and based on the indication of the object metadata of the file, the object metadata of the file from the data store; and
determining, by the data platform and from the object metadata, the storage location of the key map.
3 . The method of claim 2 , further comprising:
retrieving, by the data platform and from the storage location of the key map, the key-value pair that maps the object identifier of the file to the object metadata of the file; and
sending, by the data platform, a key of the key-value pair to respond to the request, wherein the key corresponds to an object name of the file.
4 . The method of claim 1 , wherein generating the indication of the storage location of the key map comprises generating a hash of a key of the key-value pair.
5 . The method of claim 4 , wherein generating the indication of the storage location of the key map comprises truncating the hash of the key of the key-value pair.
6 . The method of claim 1 , wherein the object identifier of the file includes at least a portion of a file system path of the file.
7 . The method of claim 1 , wherein the indication of the file system event for the file includes one or more of a creation event for the file, a renaming event for the file, or a linking event for the file.
8 . A computing system comprising:
a memory storing instructions; and
processing circuitry that executes the instructions to:
store, at one or more of a plurality of storage locations of a key map, a file system directory including one or more key-value pairs, wherein each key-value pair of the file system directory maps a respective object identifier to corresponding object metadata, wherein the respective object identifier identifies a respective file of a plurality of files stored to a data platform executed by one or more computing devices;
detect an indication of a file system event for a file of the plurality of files; and
in response to detecting the indication of the file system event:
store, to a storage location of the key map, a key-value pair that maps an object identifier of the file to object metadata of the file;
generate, based on the object identifier of the file, an indication of the storage location of the key map;
include the indication of the storage location of the key map in the object metadata for the file, wherein the indication of the storage location of the key map is used to permit retrieval of the key-value pair that maps the object identifier of the file to the object metadata of the file using the object metadata for the file; and
store the object metadata for the file to a data store.
9 . The computing system of claim 8 , wherein the processing circuitry executes the instructions to:
receive a request including an indication of the object metadata of the file;
retrieve, based on the indication of the object metadata of the file, the object metadata of the file from the data store; and
determine, from the object metadata, the storage location of the key map.
10 . The computing system of claim 9 , wherein the processing circuitry executes the instructions to:
retrieve, from the storage location of the key map, the key-value pair that maps the object identifier of the file to the object metadata of the file; and
send a key of the key-value pair to respond to the request, wherein the key corresponds to an object name of the file.
11 . The computing system of claim 8 , wherein to generate the indication of the storage location of the key map the processing circuitry executes the instructions to generate a hash of a key of the key-value pair.
12 . The computing system of claim 11 , wherein to generate the indication of the storage location of the key map the processing circuitry executes the instructions to truncate the hash of the key of the key-value pair.
13 . The computing system of claim 8 , wherein the object identifier of the file includes at least a portion of a file system path of the file.
14 . The computing system of claim 8 , wherein the indication of the file system event for the file includes one or more of a creation event for the file, a renaming event for the file, or a linking event for the file.
15 . Non-transitory computer-readable storage media comprising instructions that, when executed, cause processing circuitry of a computing system to:
store, at one or more of a plurality of storage locations of a key map, a file system directory including one or more key-value pairs, wherein each key-value pair of the file system directory maps a respective object identifier to corresponding object metadata, wherein the respective object identifier identifies a respective file of a plurality of files stored to a data platform executed by one or more computing devices;
detect an indication of a file system event for a file of the plurality of files; and
in response to detecting the indication of the file system event:
store, to a storage location of the key map, a key-value pair that maps an object identifier of the file to object metadata of the file;
generate, based on the object identifier of the file, an indication of the storage location of the key map;
include the indication of the storage location of the key map in the object metadata for the file, wherein the indication of the storage location of the key map is used to permit retrieval of the key-value pair that maps the object identifier of the file to the object metadata of the file using the object metadata for the file; and
store the object metadata for the file to a data store.
16 . The non-transitory computer-readable storage media of claim 15 , wherein the instructions, when executed, cause the processing circuitry to:
receive a request including an indication of the object metadata of the file;
retrieve, based on the indication of the object metadata of the file, the object metadata of the file from the data store; and
determine, from the object metadata, the storage location of the key map.
17 . The non-transitory computer-readable storage media of claim 16 , wherein the instructions, when executed, cause the processing circuitry to:
retrieve, from the storage location of the key map, the key-value pair that maps the object identifier of the file to the object metadata of the file; and
send a key of the key-value pair to respond to the request, wherein the key corresponds to an object name of the file.
18 . The non-transitory computer-readable storage media of claim 15 , wherein to generate the indication of the storage location of the key map the instructions, when executed, cause the processing circuitry to generate a hash of a key of the key-value pair.
19 . The non-transitory computer-readable storage media of claim 18 , wherein to generate the indication of the storage location of the key map the instructions, when executed, cause the processing circuitry to truncate the hash of the key of the key-value pair.
20 . The non-transitory computer-readable storage media of claim 15 , wherein the object identifier of the file includes at least a portion of a file system path of the file.