PATTERN TRIGGERS WHILE SEARCHING AN INDEX OF DATA BEING INGESTED INTO A DISTRIBUTED COMPUTING SYSTEM
A storage unit (SU) includes an interface configured to interface and communicate with a dispersed storage network (DSN), a memory that stores operational instructions, and a processing module operably coupled to the interface and memory such that the processing module, when operable within the SU based on the operational instructions, is configured to perform various operations. The SU obtains a data index search request to search a data index and identifies a portion of the data index to search. The SU also identifies a DSN storage location corresponding to the portion of the data index and determines whether the DSN storage location is local. When the DSN storage location is not local, the SU generates a task request and transmits the task request to another SU associated with the DSN storage location. The SU then receives a result from the other SU based in response to the task request.
1 . A storage unit (SU) comprising:
an interface configured to interface and communicate with a dispersed or distributed storage network (DSN) that includes a plurality of storage units (SUs);
memory that stores operational instructions; and
a processing module operably coupled to the interface and to the memory, wherein the processing module, when operable within the SU based on the operational instructions, is configured to:
obtain a data index search request to search a data index;
identify a portion of the data index to search based on the data index search request;
identify a DSN storage location corresponding to the portion of the data index;
determine whether the DSN storage location is local based on at least one of a directory lookup, a query, or a memory map; and
based on an unfavorable local determination of the DSN storage location:
generate a task request;
transmit the task request to another SU associated with the DSN storage location; and
receive a result from the another SU based in response to the task request.
2 . The SU of claim 1 , wherein:
the data index search request includes at least one of a data index identifier of the data index to search, one or more search terms, subsequent search terms for subsequent searches based on a search term match, or subsequent search terms for subsequent searches based on an unfavorable search term match; and
the search terms include at least one of a trigger, a pattern, a value, a range, match criteria, or failure criteria.
3 . The SU of claim 1 , wherein obtaining of the data index search to search the data index includes at least one of receiving the data index search, determining the data index search based on at least one of a previous data index search, a modification of a search term based on a previous result to obtain the data index search, a predetermination of the data index search, a query to obtain the data index search, or a list to obtain the data index search.
4 . The SU of claim 1 , wherein identifying of the DSN storage location is based on at least one of the data index search request, a data index directory, execution resource availability, or a search timeframe requirement.
5 . The SU of claim 1 , wherein generation of the task request is based on at least one of the data index search request, the portion of the data index to search, or the DSN storage location.
6 . The SU of claim 1 , wherein the processing module, when operable within the SU based on the operational instructions, is further configured to:
store at least one encoded data slices (EDSs) of a set of EDSs associated with a data object; and
search the portion of the data index to produce another result based on a favorable local determination of the DSN storage location; and wherein:
the data object is segmented into a plurality of data segments;
a data segment of the plurality of data segments is dispersed error encoded in accordance with dispersed error encoding parameters to produce the set of EDSs;
a decode threshold number of EDSs are needed to recover the data segment;
a read threshold number of EDSs provides for reconstruction of the data segment;
a write threshold number of EDSs provides for a successful transfer of the set of EDSs from a first at least one location in the DSN to a second at least one location in the DSN;
the set of EDSs is of pillar width and includes a pillar number of EDSs;
each of the decode threshold number, the read threshold number, and the write threshold number is less than the pillar number; and
the write threshold number is greater than or equal to the read threshold number that is greater than or equal to the decode threshold number.
7 . The SU of claim 1 , wherein the processing module, when operable within the SU based on the operational instructions, is further configured to:
receive the data index search from a computing device that includes at least one other SU of the plurality of SUs within the DSN, a wireless smart phone, a laptop, a tablet, a personal computers (PC), a work station, or a video game device.
8 . The SU of claim 1 , wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).
9 . A storage unit (SU) comprising:
an interface configured to interface and communicate with a dispersed or distributed storage network (DSN) that includes a plurality of storage units (SUs);
memory that stores operational instructions; and
a processing module operably coupled to the interface and to the memory, wherein the processing module, when operable within the SU based on the operational instructions, is configured to:
store at least one encoded data slices (EDSs) of a set of EDSs associated with a data object, wherein the data object is segmented into a plurality of data segments, wherein a data segment of the plurality of data segments is dispersed error encoded in accordance with dispersed error encoding parameters to produce the set of EDSs, and wherein a decode threshold number of EDSs are needed to recover the data segment;
obtain a data index search request to search a data index;
identify a portion of the data index to search based on the data index search request;
identify a DSN storage location corresponding to the portion of the data index based on at least one of the data index search request, a data index directory, execution resource availability, or a search timeframe requirement;
determine whether the DSN storage location is local based on at least one of a directory lookup, a query, or a memory map; and
search the portion of the data index to produce a result based on a favorable local determination of the DSN storage location;
based on an unfavorable local determination of the DSN storage location:
generate a task request;
transmit the task request to another SU associated with the DSN storage location; and
receive another result from the another SU based in response to the task request.
10 . The SU of claim 9 , wherein:
the data index search request includes at least one of a data index identifier of the data index to search, one or more search terms, subsequent search terms for subsequent searches based on a search term match, or subsequent search terms for subsequent searches based on an unfavorable search term match; and
the search terms include at least one of a trigger, a pattern, a value, a range, match criteria, or failure criteria.
11 . The SU of claim 9 , wherein obtaining of the data index search to search the data index includes at least one of receiving the data index search, determining the data index search based on at least one of a previous data index search, a modification of a search term based on a previous result to obtain the data index search, a predetermination of the data index search, a query to obtain the data index search, or a list to obtain the data index search.
12 . The SU of claim 9 , wherein:
a read threshold number of EDSs provides for reconstruction of the data segment;
a write threshold number of EDSs provides for a successful transfer of the set of EDSs from a first at least one location in the DSN to a second at least one location in the DSN;
the set of EDSs is of pillar width and includes a pillar number of EDSs;
each of the decode threshold number, the read threshold number, and the write threshold number is less than the pillar number; and
the write threshold number is greater than or equal to the read threshold number that is greater than or equal to the decode threshold number.
13 . The SU of claim 9 , wherein:
the SU is located at a first premises that is remotely located from at least one other SU of the plurality of SUs within the DSN; and
the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).
14 . A method for execution by a storage unit (SU), the method comprising:
obtaining a data index search request to search a data index;
identifying a portion of the data index to search based on the data index search request;
identifying a DSN storage location corresponding to the portion of the data index;
determining whether the DSN storage location is local based on at least one of a directory lookup, a query, or a memory map; and
based on an unfavorable local determination of the DSN storage location:
generating a task request;
transmitting, via an interface of the SU that is configured to interface and communicate with a dispersed or distributed storage network (DSN) that includes a plurality of storage units (SUs), the task request to another SU associated with the DSN storage location; and
receiving, via the interface, a result from the another SU based in response to the task request.
15 . The method of claim 14 , wherein:
the data index search request includes at least one of a data index identifier of the data index to search, one or more search terms, subsequent search terms for subsequent searches based on a search term match, or subsequent search terms for subsequent searches based on an unfavorable search term match; and
the search terms include at least one of a trigger, a pattern, a value, a range, match criteria, or failure criteria.
16 . The method of claim 14 , wherein obtaining of the data index search to search the data index includes at least one of receiving the data index search, determining the data index search based on at least one of a previous data index search, a modification of a search term based on a previous result to obtain the data index search, a predetermination of the data index search, a query to obtain the data index search, or a list to obtain the data index search.
17 . The method of claim 14 , wherein identifying of the DSN storage location is based on at least one of the data index search request, a data index directory, execution resource availability, or a search timeframe requirement.
18 . The method of claim 14 , wherein generation of the task request is based on at least one of the data index search request, the portion of the data index to search, or the DSN storage location.
19 . The method of claim 14 further comprising:
storing at least one encoded data slices (EDSs) of a set of EDSs associated with a data object; and
searching the portion of the data index to produce another result based on a favorable local determination of the DSN storage location; and wherein:
the data object is segmented into a plurality of data segments;
a data segment of the plurality of data segments is dispersed error encoded in accordance with dispersed error encoding parameters to produce the set of EDSs;
a decode threshold number of EDSs are needed to recover the data segment;
a read threshold number of EDSs provides for reconstruction of the data segment;
a write threshold number of EDSs provides for a successful transfer of the set of EDSs from a first at least one location in the DSN to a second at least one location in the DSN;
the set of EDSs is of pillar width and includes a pillar number of EDSs;
each of the decode threshold number, the read threshold number, and the write threshold number is less than the pillar number; and
the write threshold number is greater than or equal to the read threshold number that is greater than or equal to the decode threshold number.
20 . The method of claim 14 further comprising receiving the data index search from a computing device that includes at least one other SU of the plurality of SUs within the DSN, a wireless smart phone, a laptop, a tablet, a personal computers (PC), a work station, or a video game device; and wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).