Data presence/absence determination apparatus and computer-readable storage medium storing program for determination of data presence/absence

1. A data processing apparatus comprising:

a memory configured to store a data set including a plurality of data elements and a first Bloom filter used in a membership test to determine whether a query-target data element is present in the data set and having specific bits set to a specific value, the specific bits being determined by predetermined calculation using the plurality of data elements, respectively; and

a processor configured to execute a process including:

deleting, when some data elements are deleted from the data set, a first bit array from an end of a bit array of the first Bloom filter, the first bit array having a number of bits corresponding to a number of the some data elements;

adding, when a first data element is entered as the query-target data element, a second bit array temporarily at an end of a bit array of a second Bloom filter, which is obtained by the deleting of the first bit array from the first Bloom filter, the second bit array having a same number of bits as the first bit array, with all bits set to the specific value; and

running the membership test, using the second Bloom filter with the second bit array added thereto, to determine whether the first data element is present in the data set from which the some data elements have been deleted.

2. The data processing apparatus according to claim 1 , wherein: the number of bits of the first bit array corresponds to a proportion of the number of the some data elements to a number of the plurality of data elements included in the data set.

3. The data processing apparatus according to claim 1 , wherein:

the memory stores a first level Bloom filter used in a membership test to determine whether the query-target data element is present in a first data set and a plurality of second level Bloom filters used in a membership test to determine whether the query-target data element is present in respective second data sets, which is formed by splitting the first data set, the first Bloom filter being one second level Bloom filter amongst the plurality of second level Bloom filters, the data set being one second data set corresponding to the one second level Bloom filter amongst the plurality of second data sets; and

the process further includes:

deleting, when the some data elements are deleted from the data set, a third bit array from an end of a bit array of the first level Bloom filter, the third bit array having the same number of bits as the first bit array;

adding, when the first data element is entered as the query-target data element, a fourth bit array temporarily at an end of a bit array of a third Bloom filter, which is obtained by the deleting of the third bit array from the first level Bloom filter, the fourth bit array being same as the second bit array; and

running the membership test, using the third Bloom filter with the fourth bit array added thereto, to determine whether the first data element is present in the first data set from which the some data elements have been deleted.

4. The data processing apparatus according to claim 1 , wherein:

the memory has write data storage space including storage space of the data set; and

the process further includes:

storing, in the write data storage space, a plurality of write data elements requested from an external apparatus to be written, after deduplicating the plurality of write data elements, and also recording, in the memory, information indicating a correspondence relationship between at least each of the plurality of data elements and the plurality of write data elements;

recording, for each of the plurality of data elements, a first count value in the memory, the first count value indicating how many of the plurality of write data elements the plurality of data elements each corresponds to;

decrementing, in response to a request from the external apparatus to update or delete one of the plurality of write data elements that corresponds to one of the plurality of data elements, the first count value corresponding to the one of the plurality of data elements;

recording, in the memory, a second count value indicating a number of data elements each having the first count value set to zero, amongst the plurality of data elements; and

deleting, when the second count value exceeds a predetermined threshold, the data elements each having the first count value set to zero as the some data elements.

5. The data processing apparatus according to claim 4 , wherein: the membership test to determine whether the first data element is present is run, in response to a request to write the first data element as one of the plurality of write data elements, to determine whether a same data element as the first data element is present in the data set.

6. The data processing apparatus according to claim 4 , wherein: each of the plurality of write data elements is partial data of a file requested from the external apparatus to be written.

7. A non-transitory computer-readable storage medium storing a computer program that causes a computer to execute a process comprising:

deleting, when some data elements are deleted from a data set including a plurality of data elements, stored in memory, a first bit array from an end of a bit array of a first Bloom filter used in a membership test to determine whether a query-target data element is present in the data set and having specific bits set to a specific value, the first bit array having a number of bits corresponding to a number of the some data elements, the specific bits being determined by predetermined calculation using the plurality of data elements, respectively;

adding, when a first data element is entered as the query-target data element, a second bit array temporarily at an end of a bit array of a second Bloom filter, which is obtained by the deleting of the first bit array from the first Bloom filter, the second bit array having a same number of bits as the first bit array, with all bits set to the specific value; and

running the membership test, using the second Bloom filter with the second bit array added thereto, to determine whether the first data element is present in the data set from which the some data elements have been deleted.

8. The non-transitory computer-readable storage medium according to claim 7 , wherein: the number of bits of the first bit array corresponds to a proportion of the number of the some data elements to a number of the plurality of data elements included in the data set.

9. The non-transitory computer-readable storage medium according to claim 7 , wherein:

the memory stores a first level Bloom filter used in a membership test to determine whether the query-target data element is present in a first data set and a plurality of second level Bloom filters used in a membership test to determine whether the query-target data element is present in respective second data sets, which is formed by splitting the first data set, the first Bloom filter being one second level Bloom filter amongst the plurality of second level Bloom filters, the data set being one second data set corresponding to the one second level Bloom filter amongst the plurality of second data sets; and

the process further includes:

deleting, when the some data elements are deleted from the data set, a third bit array from an end of a bit array of the first level Bloom filter, the third bit array having the same number of bits as the first bit array;

adding, when the first data element is entered as the query-target data element, a fourth bit array temporarily at an end of a bit array of a third Bloom filter, which is obtained by the deleting of the third bit array from the first level Bloom filter, the fourth bit array being same as the second bit array; and

running the membership test, using the third Bloom filter with the fourth bit array added thereto, to determine whether the first data element is present in the first data set from which the some data elements have been deleted.

10. The non-transitory computer-readable storage medium according to claim 7 , wherein:

the memory has write data storage space including storage space of the data set; and

the process further includes:

storing, in the write data storage space, a plurality of write data elements requested from an external apparatus to be written, after deduplicating the plurality of write data elements, and also recording, in the memory, information indicating a correspondence relationship between at least each of the plurality of data elements and the plurality of write data elements;

recording, for each of the plurality of data elements, a first count value in the memory, the first count value indicating how many of the plurality of write data elements the plurality of data elements each corresponds to;

decrementing, in response to a request from the external apparatus to update or delete one of the plurality of write data elements that corresponds to one of the plurality of data elements, the first count value corresponding to the one of the plurality of data elements;

recording, in the memory, a second count value indicating a number of data elements each having the first count value set to zero, amongst the plurality of data elements; and

deleting, when the second count value exceeds a predetermined threshold, the data elements each having the first count value set to zero as the some data elements.

11. The non-transitory computer-readable storage medium according to claim 10 , wherein: the membership test to determine whether the first data element is present is run, in response to a request to write the first data element as one of the plurality of write data elements, to determine whether a same data element as the first data element is present in the data set.