Method and system of mapreduce implementations on indexed datasets in a distributed database environment
View Patent ↗In one exemplary embodiment, a method of a distributed database system includes the step receiving a query in a query language from a client with a distributed database system. An index that matches the query is located. The index is pre-generated from a database table in the distributed database system. A map function of a MapReduce programming model is implemented using the index. A reduce function of the MapReduce programming model is implemented using the output of the map function. Optionally, a finalize function can be implemented using the output of the reduce function. The distributed database system can be a scalable NoSQL database. The reduce function can be optional when the value of the output of the map function is guaranteed to be unique.
1. A method in a distributed database system, the distributed database system comprising a processor, the method comprising:
receiving a query in a query language from a client with a distributed database system;
locating an index that matches the query, wherein the index is pre-generated from a database table in the distributed database system;
implementing a map function of a MapReduce programming model using the index; and
implementing a reduce function of the Map Reduce programming model using the output of the map function.
2. The method of claim 1 further comprising:
implementing a finalize function using the output of the reduce function.
3. The method of claim 1 wherein the map function comprises:
obtaining a series of key/value pairs; and
processing each key/value pair to generate zero or more output key/value pairs.
4. The method of claim 1 , wherein the distributed database system comprises a scalable NoSQL database.
5. The method of claim 4 , wherein the NoSQL database is implemented with a shared-nothing architecture.
6. The method of claim 1 , wherein the query comprises a WHERE clause.
7. The method of claim 6 , wherein the index comprises a data structure created using one or more columns of the database table based on the WHERE clause.
8. The method of claim 1 , wherein the reduce function is not performed when the value of the output of the map function is unique.
9. A non-transitory computer-readable storage medium comprising program instructions for MapReduce implementations on indexed datasets in a distributed database environment, wherein execution of the program instructions by one or more processors of a computer system causes the one or more processors to carry out the steps of:
receiving a query in a query language from a client with a distributed database system;
locating an index that matches the query, wherein the index is pre-generated from a database table in the distributed database system, and wherein the index is maintained substantially current with a relevant modification of the database table;
executing a map function of a MapReduce programming model using the index; and
executing a reduce function of the MapReduce programming model using the output of the map function.
10. The non-transitory computer-readable storage medium of claim 9 , wherein the distributed database system comprises a scalable NoSQL database.
11. The non-transitory computer-readable storage medium of claim 9 , comprising further program instructions for the step of:
executing a finalize function on an output of the reduce function.
12. The non-transitory computer-readable storage medium of claim 11 , wherein the finalize function comprises
receiving a key and a value of the reduce function output and returns a finalized value in a list format.
13. A distributed database system comprising:
a processor;
a client node that provides a query in a query language to the distributed database system;
a distributed database system comprising a cluster of server nodes: and
a server node that:
creates at least one index from a data store in the distributed database,
maintains the at least one index in a substantially current state based on the data store,
receives a query from a client node,
matches the query to the at least one index when the query is received from the client node, and
implements a map function of a MapReduce programming model using the index.
14. The distributed database system of claim 13 , wherein the server node provides a map function output to the client node.
15. The distributed database system of claim 14 , wherein the client node implements a reduce function of the MapReduce programming model using the output of the map function.
16. The distributed database system of claim 14 , wherein the client node implements a finalize function using the output of the reduce function.
17. The distributed database system of claim 13 , wherein the server node implements the reduce function of the MapReduce programming model using the output of the map function.
18. The distributed database system of claim 17 , wherein the server node implements the finalize function using the output of the reduce function.
19. The distributed database system of claim 13 , wherein the server node provide the output of the finalize function or the output of the reduce function to the client node.
20. The distributed database system of claim 13 , wherein the server node creates at least one index based on the query from the client node.