SYSTEM AND METHODS FOR CAPTURING AND STORING METADATA FROM ACCESS LOGS AND STORAGE SYSTEMS AND IMPROVING STORAGE EFFICIENCY OF DATA AND METHOD THEREFOR
An electronic file storage system has a processor. A memory is coupled to the processor, the memory storing program instructions that when executed by the processor, causes the processor to: read an access log; and infer access time, modify time, create time, delete time, other metadata from the access log unavailable on the access log, wherein access time, modify time, create time, delete time and other metadata about an object is inferred from a timestamp and record of an operation on an object recorded on the access log.
1 . An electronic file storage system comprising:
a processor;
a memory coupled to the processor, the memory storing program instructions that when executed by the processor, causes the processor to:
read an access log; and
infer access time, modify time, create time, delete time, other metadata from the access log unavailable on the access log, wherein access time, modify time, create time, delete time and other metadata is inferred from a timestamp of an object recorded on the access log.
2 . The electronic file storage system of claim 1 , wherein the memory storing program instructions that when executed by the processor, causes the processor to:
infer metadata for each object from the access log that is unavailable on the object itself, wherein time is inferred from the timestamp of the object record, wherein access time is inferred from logged reads, create time and modify times are inferred from logged writes, and other times are inferred from other logged operations metadata or other data update times.
3 . The electronic file storage system of claim 1 , wherein the memory storing program instructions that when executed by the processor, causes the processor to:
store the access time, the modify time, the create time, and other metadata created in a separate database of object metadata that can be queried by a separate computer system.
4 . The electronic file storage system of claim 1 , wherein the memory storing program instructions that when executed by the processor, causes the processor to:
capture the metadata stored by an object store for each object, and record the metadata in a separate database.
5 . The electronic file storage system of claim 1 , wherein the memory storing program instructions that when executed by the processor, causes the processor to:
recognize a location of the object in the system along with times of access, create, modify, delete, and metadata update determined from the parsing of the access log;
send a direct request for object metadata from an object store of a newly parsed object; and
record the metadata determined from parsing the access log along with the metadata captured from the object store for each object.
6 . The electronic file storage system of claim 5 , wherein the memory storing program instructions that when executed by the processor, causes the processor to:
record the metadata determined from parsing the access log along with the metadata captured from the object store for each object in a separate database.
7 . The electronic file storage system of claim 1 , wherein the memory storing program instructions that when executed by the processor, causes the processor to:
capture a series of access, create, modify, and delete times for the object to store the history of the object in a separate database.
8 . The electronic file storage system of claim 1 , wherein the memory storing program instructions that when executed by the processor, causes the processor to:
determine a probability of the object being accessed or modified in a future time based on a series of access and modify times of the objects; and
store the probability of the object in a separate database.
9 . The electronic file storage system of claim 8 , wherein the memory storing program instructions that when executed by the processor, causes the processor to:
determine the probability of the object being accessed or modified using recency of access and historical pattern of access.
10 . The electronic file storage system of claim 1 , wherein the memory storing program instructions that when executed by the processor, causes the processor to:
label the object as one of a “hot” object or a “cold” object.
11 . The electronic file storage system of claim 8 , wherein the memory storing program instructions that when executed by the processor, causes the processor to:
label the object as one of a “hot” object or a “cold” object;
label the object as the a “hot” object when the object has been accessed recently within a predetermined timeframe, has a probability of future access above a predetermined threshold level or a combination thereof; and
label the object as the a “cold” object when the object has not been accessed within a predetermined timeframe, has a probability of future access below predetermined number or a combination thereof.
12 . The electronic file storage system of claim 10 , wherein the memory storing program instructions that when executed by the processor, causes the processor to:
storing the “hot” object in higher performing storage tiers; and
storing the “cold” object in lower performing storage tiers.
13 . An electronic file storage system comprising:
a processor;
a memory coupled to the processor, the memory storing program instructions that when executed by the processor, causes the processor to:
parse an access log;
infer access time, modify time, create time, delete time, other metadata from the access log unavailable on the access log, wherein access time, modify time, create time, delete time and other metadata is inferred from a timestamp of an object recorded on the access log, wherein time is inferred from the timestamp of the object record, wherein access time is inferred from logged reads, create time and modify times are inferred from logged writes, and other times are inferred from other logged operations metadata or other data update times;
capture a series of access, create, modify, and delete times for the object to store a history of the object;
determine the probability of the object being accessed or modified using recency of access and historical pattern of access; and
label the object as one of a “hot” object or a “cold” object based on the probability.
14 . The electronic file storage system of claim 13 , wherein the memory storing program instructions that when executed by the processor, causes the processor to:
store the access time, the modify time, the create time, and other metadata created in a separate database of object metadata that can be queried by a separate computer system.
15 . The electronic file storage system of claim 13 , wherein the memory storing program instructions that when executed by the processor, causes the processor to:
capture the metadata stored by an object store for each object, and record the metadata in a separate database.
16 . The electronic file storage system of claim 13 , wherein the memory storing program instructions that when executed by the processor, causes the processor to:
recognize a location of the object in the system along with times of access, create, modify, delete, and metadata update determined from the parsing of the access log;
send a direct request for object metadata from an object store of a newly parsed object; and
record the metadata determined from parsing the access log along with the metadata captured from the object store for each object.
17 . The electronic file storage system of claim 13 , wherein the memory storing program instructions that when executed by the processor, causes the processor to:
record the metadata determined from parsing the access log along with the metadata captured from the object store for each object in a separate database.
18 . The electronic file storage system of claim 13 , wherein the memory storing program instructions that when executed by the processor, causes the processor to:
label the object as one of a “hot” object or a “cold” object;
label the object as the a “hot” object when the object has been accessed recently within a predetermined timeframe, has a probability of future access above a predetermined threshold level or a combination thereof; and
label the object as the a “cold” object when the object has not been accessed within a predetermined timeframe, has a probability of future access below predetermined number or a combination thereof.
19 . The electronic file storage system of claim 13 , wherein the memory storing program instructions that when executed by the processor, causes the processor to:
storing the “hot” object in higher performing storage tiers; and
storing the “cold” object in lower performing storage tiers.