Data processing performance enhancement in a distributed file system
View Patent ↗Systems and methods of data processing performance enhancement are disclosed. One embodiment includes, invoking operating system calls to optimize cache management by an I/O component; wherein, the operating system calls are invoked to perform one or more of; proactive triggering of readaheads for sequential read requests of a disk; purging data out of buffer cache after writing to the disk or performing sequential reads from the desk; and/or eliminating a delay between when a write is performed and when written data from the write is flushed to the disk from the buffer cache.
1. A method for enhancing performance for data processing in a distributed file system, the method comprising:
instantiating an input/output (I/O) manager on a machine among a plurality of machines that implement the distributed file system; and
utilizing the I/O manager to perform cache management optimization including:
(a) determining that the machine employs a heuristic for triggering readaheads for sequential read requests;
overriding the heuristic so as to deterministically trigger the readaheads for all sequential read requests;
(b) determining that the machine is configured to automatically cache data into a buffer on the machine after the data is accessed;
detecting that a specific size of data has been accessed;
instructing the machine to invalidate the cached data in the buffer; and
(c) determining that the machine is configured to include a time delay before committing data from the buffer to a disk on the machine;
overriding the time delay so that the machine commits the data from the buffer to the disk without the time delay.
2. The method of claim 1 , wherein, the cache management is optimized for reads and writes in Hbase or MapReduce.
3. The method of claim 1 , wherein, the cache management is optimized for large sequential reads and writes.
4. The method of claim 1 , wherein an amount of the readaheads for sequential read requests is dynamically adjusted based on the sequential read requests.
5. The method of claim 1 , wherein the method further comprising:
instantiating a connection manager on the machine, wherein the connection manager is configured to instruct the machine perform steps including:
upon receiving an access request from a client, establishing a connection with a node associated with the access request in response to the access request;
determining that the access request is associated with data that is smaller than a select size;
detecting whether operations requested by the access request have been completed;
after the operations requested by the access request have been completed, holding the connection with the node open for a select time period,
receiving, by the connection manager from the client, a subsequent access request that is associated with the node associated with the access request before the select time period expires; and
performing operations requested by the subsequent access request through the held open connection.
6. The method of claim 1 , wherein, data size of the data that is invalidated from the buffer is configurable or dynamically adjustable.
7. The method of claim 1 , wherein, data size of the data that is invalidated from the buffer is between 4-8 MB or 8-16 MB.
8. The method of claim 1 , wherein the cache management optimization are performed through operating system calls are native to an operating system running on the machine.
9. The method of claim 1 , further comprising decreasing checksum overhead in a read path of the distributed file system by modifying a polynomial that is used in generating the checksum.
10. The method of claim 9 , wherein, the checksum is performed in hardware by a processor supporting CRC32.
11. The method of claim 1 , wherein, the operating system is Linux or Unix-based operating systems.
12. The method of claim 1 , wherein, the operating system conforms to POSIX.1-2001.
13. The method of claim 1 , wherein, the distributed file system is the Hadoop distributed file system.
14. A system for distributed computing, the system comprising:
a set of machines forming a distributed file system cluster, a given machine in the set of machines having:
a processor;
a disk;
memory having stored there on instructions which when executed by the processor, causes the given machine to perform:
(a) determining that the machine employs a heuristic for triggering readaheads for sequential read requests;
overriding the heuristic so as to deterministically trigger the readaheads for all sequential read requests;
(b) determining that the machine is configured to automatically cache data into a buffer on the machine after the data is accessed;
detecting that a specific size of data has been accessed;
instructing the machine to invalidate the cached data in the buffer; and
(c) determining that the machine is configured to include a time delay before committing data from the buffer to a disk on the machine;
overriding the time delay so that the machine commits the data from the buffer to the disk without the time delay.
15. The system of claim 14 , wherein the given machine is further caused to perform:
instantiating a connection manager on the machine, wherein the connection manager is configured to instruct the machine perform steps including:
upon receiving an access request from a client, establishing a connection with node associated with the access request in response to the access request;
determining that the access request is associated with data that is smaller than a select size;
detecting whether operations requested by the access request have been completed;
after the operations requested by the access request have been completed, holding the connection with the node open for a select time period,
receiving, by the connection manager from the client, a subsequent access request that is associated with the node associated with the access request before the select time period expires; and
performing operations requested by the subsequent access request through the held open connection.
16. The system of claim 14 , wherein the connection is held open for 0.5-1 seconds, 1-2, seconds, or 2-5 seconds.
17. The system of claim 14 , wherein an amount of the readaheads for sequential read requests is dynamically adjusted based on the sequential read requests.
18. The system of claim 14 , wherein, the distributed file system is the Hadoop distributed file system (HDFS).
19. The system of claim 14 , wherein, the cache management is optimized for reads and writes in Hbase or MapReduce.
20. The system of claim 14 , wherein the given system is further cased to perform:
decreasing checksum overhead in a read path of the distributed file system by modifying a polynomial that is used in generating the checksum.
21. The system of claim 20 , wherein, the checksum uses CRC32C algorithm.