Duplicate incident detection using dynamic similarity threshold
Methods, apparatus, and processor-readable storage media for duplicate incident detection using a dynamic similarity threshold are provided herein. An example computer-implemented method includes obtaining a request including information associated with tracking at least a first incident in a database; generating a first representation of the first incident that encodes at least a portion of the information; computing a set of similarity scores for the first incident, where a given similarity score is based on a comparison between the first representation and a second representation generated for one of a plurality of additional incidents in the database; detecting that the first incident is a duplicate of at least one of the plurality of additional incidents based on a comparison of the set of similarity scores to a similarity threshold, where the similarity threshold is updated over time; and initiating an update in the database in response to the detecting.
1. A computer-implemented method comprising:
obtaining a request comprising information associated with tracking at least a first incident in an incident database;
generating a first numeric representation of the first incident that encodes at least a portion of the information;
computing a set of one or more similarity scores for the first incident, wherein a given similarity score in the set is based at least in part on a comparison between the first numeric representation and a second numeric representation generated for one of a plurality of additional incidents in the incident database;
detecting that the first incident is a duplicate of at least one of the plurality of additional incidents based on a comparison of the set of similarity scores to a similarity threshold, wherein the similarity threshold is dynamically updated over time based on at least one of: one or more updates to one or more of the additional incidents in the incident database and one or more new incidents added to the incident database; and
in response to the detecting, initiating an update to the incident database, wherein the update comprises at least one of: (i) assigning a flag to at least one of: (a) the first incident and (b) the at least one of the plurality of additional incidents as being a duplicate incident, and (ii) merging the first incident and the at least one of the plurality of additional incidents in the incident database;
wherein the method is performed by at least one processing device comprising a processor coupled to a memory.
2. The computer-implemented method of claim 1 , wherein the first incident relates to a software defect in one or more software code repositories, and wherein the information associated with tracking the first incident comprises at least one of:
a title of the first incident;
one or more user comments related to the software defect;
a location of the software defect within software code associated with the one or more software code repositories; and
a release version corresponding to the first incident.
3. The computer-implemented method of claim 2 , wherein the similarity threshold is separately determined for at least one of: the release version corresponding to the first incident and the location of the software defect.
4. The computer-implemented method of claim 1 , wherein the first numeric representation comprises a vector representation that is generated using a term frequency-inverse document frequency process.
5. The computer-implemented method of claim 1 , wherein the similarity threshold is updated over time based at least in part on information provided by one or more users, wherein the information verifies that at least one of: (i) the first incident in the incident database is the duplicate of the at least one of the plurality of additional incidents and (two or more other incidents in the incident database are duplicates.
6. The computer-implemented method of claim 1 , wherein the similarity threshold is computed at least in part by:
generating at least one numeric representation for each of the plurality of additional incidents, wherein at least a portion of the plurality of additional incidents are labeled as duplicate incidents;
computing similarity scores between each pair of numeric representations of the plurality of numeric representations, wherein the computing comprises computing a cosine similarity between each pair of numeric representations of the plurality of numeric representations; and
calculating the similarity threshold based on the similarity scores corresponding to the pairs of incidents labeled as duplicates.
7. The computer-implemented method of claim 1 , wherein the similarity threshold is updated in response to detecting that the first incident is the duplicate of the at least one of the plurality of additional incidents.
8. The computer-implemented method of claim 1 , wherein the initiating the update to the incident database, further comprises:
adding one or more comments related to at least one of the first incident and the at least one of the plurality of additional incidents in the incident database.
9. The computer-implemented method claim 1 , wherein the similarity threshold is dynamically updated by recomputing a value based on an average similarity of incident pairs currently labeled as duplicates in the incident database.
10. The computer-implemented method of claim 1 , wherein the similarity threshold is applied across each incident in the incident database.
11. A non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device:
to obtain a request comprising information associated with tracking at least a first incident in an incident database;
to generate a first numeric representation of the first incident that encodes at least a portion of the information;
to compute a set of one or more similarity scores for the first incident, wherein a given similarity score in the set is based at least in part on a comparison between the first numeric representation and a second numeric representation generated for one of a plurality of additional incidents in the incident database;
to detect that the first incident is a duplicate of at least one of the plurality of additional incidents based on a comparison of the set of similarity scores to a similarity threshold, wherein the similarity threshold is dynamically updated over time based on at least one of: one or more updates to one or more of the additional incidents in the incident database and one or more new incidents added to the incident database; and
in response to the detecting, to initiate an update to the incident database, wherein the update comprises at least one of: (i) assigning a flag to at least one of: (a) the first incident and (b) the at least one of the plurality of additional incidents as being a duplicate incident, and (ii) merging the first incident and the at least one of the plurality of additional incidents in the incident database.
12. The non-transitory processor-readable storage medium of claim 11 , wherein the first incident relates to a software defect in one or more software code repositories.
13. The non-transitory processor-readable storage medium of claim 12 , wherein the information associated with tracking the first incident comprises at least one of:
a title of the first incident;
one or more user comments related to the software defect;
a location of the software defect within software code associated with the one or more software code repositories; and
a release version corresponding to the first incident.
14. The non-transitory processor-readable storage medium of claim 13 , wherein the similarity threshold is separately determined for at least one of: the release version corresponding to the first incident and the location of the software defect.
15. The non-transitory processor-readable storage medium of claim 11 , wherein the similarity threshold is updated over time based at least in part on information provided by one or more users, wherein the information verifies that at least one of: (i) the first incident in the incident database is the duplicate of the at least one of the plurality of additional incidents and (ii) two or more other incidents in the incident database are duplicates.
16. The non-transitory processor-readable storage medium of claim 11 , wherein the similarity threshold is computed at least in part by:
generating at least one numeric representation for each of the plurality of additional incidents, wherein at least a portion of the plurality of additional incidents are labeled as duplicate incidents;
computing similarity scores between each pair of numeric representations of the plurality of numeric representations; and
calculating the similarity threshold based on the similarity scores corresponding to the pairs of incidents labeled as duplicates.
17. An apparatus comprising:
at least one processing device comprising a processor coupled to a memory;
the at least one processing device being configured:
to obtain a request comprising information associated with tracking at least a first incident in an incident database;
to generate a first numeric representation of the first incident that encodes at least a portion of the information;
to compute a set of one or more similarity scores for the first incident, wherein a given similarity score in the set is based at least in part on a comparison between the first numeric representation and a second numeric representation generated for one of a plurality of additional incidents in the incident database;
to detect that the first incident is a duplicate of at least one of the plurality of additional incidents based on a comparison of the set of similarity scores to a similarity threshold, wherein the similarity threshold is dynamically updated over time based on at least one of: one or more updates to one or more of the additional incidents in the incident database and one or more new incidents added to the incident database; and
in response to the detecting, to initiate an update to the incident database, wherein the update comprises at least one of: (i) assigning a flag to at least one of: (a) the first incident and (b) the at least one of the plurality of additional incidents as being a duplicate incident, and (ii) merging the first incident and the at least one of the plurality of additional incidents in the incident database.
18. The apparatus of claim 17 , wherein the first incident relates to a software defect in one or more software code repositories.
19. The apparatus of claim 17 , wherein the similarity threshold is updated over time based at least in part on information provided by one or more users, wherein the information verifies that at least one of: the first incident in the incident database is a duplicate and two or more other incidents in the incident database are duplicates.
20. The apparatus of claim 17 , wherein the similarity threshold is computed at least in part by:
generating at least one numeric representation for each of a plurality of incidents, wherein at least a portion of the plurality of incidents are labeled as duplicates;
computing similarity scores between each pair of numeric representations of the plurality of numeric representations; and
calculating the similarity threshold based on the similarity scores corresponding to the pairs of incidents labeled as duplicates.