System and method for performing efficient searches and queries in a storage node
A system and method of providing enhanced data processing and analysis in a storage node, such as a solid state drive (SSD). The SSD includes flash memory and an SSD processing unit capable of executing searches and analysis on the data in the flash memory without returning all of the data to be searched to a host CPU outside of the SSD. Other processing capabilities incorporated into the SSD may include encryption and decryption, compression and decompression, and in-line indexing of data, and data analytics.
1. A solid state drive comprising:
a solid state drive host interface;
a processing unit; and
a flash memory;
the processing unit being in communication with the flash memory, and comprising:
a hardware engine; and
a microcontroller;
the solid state drive host interface being selected from the group consisting of: PCIe, NVMe, SATA, Fibre Channel, SAS, SCSI over PCIe, and Ethernet,
the solid state drive configured to receive a query through the solid state drive host interface, and to return, through the solid state drive host interface, a result of the query, and
the hardware engine configured to be controlled by the microcontroller, and to generate a result by applying a processing algorithm to data received from the flash memory,
the processing unit configured to execute a search analysis on the processed data, the processed data comprising unstructured data, the search analysis comprising an analysis selected from the group consisting of:
comparing the processed data to a specified pattern, wherein the comparing comprises first testing the processed data with a first set of criteria, and second testing the processed data with a second set of criteria in response to the first testing only when the processed data meet the first set of criteria,
counting words in the processed data, wherein the counting comprises counting the number of occurrences of a word in the processed data,
performing in-line indexing of the processed data,
and combinations thereof,
wherein the search analysis comprises
the comparing of the processed data to the specified pattern, wherein the comparing comprises the first testing of the processed data with the first set of criteria, and the second testing of the processed data with the second set of criteria in response to the first testing only when the processed data meet the first set of criteria,
wherein the unstructured data comprises a header and a body, and
wherein the first testing comprises first testing the header with the first set of criteria, and the second testing comprises second testing the body with the second set of criteria in response to the first testing only when the header meets the first set of criteria.
2. The drive of claim 1 , wherein the processing algorithm comprises a comparison algorithm configured to count the number of occurrences of the specified pattern in the data received from the flash memory.
3. The drive of claim 2 , wherein the hardware engine comprises a search engine.
4. The drive of claim 3 , wherein the search engine comprises a comparator.
5. The drive of claim 2 , wherein the hardware engine comprises a plurality of search engines.
6. The drive of claim 1 , wherein the processing unit comprises a buffer.
7. The drive of claim 6 , wherein the hardware engine, the microcontroller, and the buffer are integrated on a single semiconductor chip.
8. The drive of claim 7 , wherein the semiconductor chip is a single silicon chip.
9. The drive of claim 6 , wherein the processing unit comprises a flash interface.
10. The drive of claim 6 , wherein the processing unit comprises a plurality of flash interfaces.
11. The drive of claim 10 , wherein the hardware engine, the microcontroller, the buffer, and the flash interfaces are integrated on a single semiconductor chip.
12. The solid state drive of claim 1 , wherein the search analysis further comprises:
the counting of words in the processed data, and
the performing of the in-line indexing of the processed data.
13. A method of searching data stored in a solid state drive comprising a flash memory, a processing unit, and a solid state drive host interface, the method comprising:
receiving, by the solid state drive processing unit, a search request through the solid state drive host interface;
retrieving raw data, by the solid state drive processing unit, from the flash memory, in response to the received search request;
processing, by the solid state drive processing unit, the raw data, to generate a result; and
providing, by the solid state drive processing unit, the generated result through the solid state drive host interface,
wherein the solid state drive host interface is selected from the group consisting of: PCIe, NVMe, SATA, Fibre Channel, SAS, SCSI over PCIe, and Ethernet,
wherein the processing unit is configured to execute a first search analysis on the processed data, the processed data being unstructured data, the first search analysis comprising comparing the unstructured data to a specified pattern,
wherein the processing unit is further configured to execute a second search analysis concurrently with the first search analysis, the second search analysis comprising comparing the unstructured data to a specified pattern, and
wherein the first search analysis is on a header of the processed data, and the second search analysis is on a body of the processed data.
14. The method of claim 13 , wherein the first search analysis begins at a first storage location in a data storage area of the processed data and progresses through the data storage area, and the second search analysis begins at a second storage location in the data storage area and progresses through the data storage area, the first storage location being different from the second storage location.
15. The method of claim 14 , wherein the second search analysis begins when the first search has reached the second storage location in the data area.
16. The method of claim 13 , wherein the search request comprises the specified pattern of the first search analysis, and the processing, by the solid state drive, of the raw data, comprises comparing the raw data to the specified pattern of the first search analysis.
17. The method of claim 13 , wherein the processing unit is in communication with the flash memory.
18. The method of claim 13 , wherein the processing unit comprises a hardware engine and a microcontroller.
19. The method of claim 18 , wherein the hardware engine and the microcontroller are integrated on a single semiconductor chip.
20. The method of claim 18 , wherein the processing unit comprises a plurality of hardware engines.