Unit group generation and relationship establishment
This disclosure describes systems and methods for establishing a unit group dictionary based on user provided annotations. The unit group dictionary may be used to identify relationships between multiple items in a corpus. Those relationships may facilitate the display of object identifiers and/or other aspects used and/or provided by the object management service.
1. A computer-implemented method, comprising:
under control of one or more computing systems configured with executable instructions,
identifying a corpus of items, wherein a plurality of the corpus of items include annotations;
processing the annotations to determine a plurality of unit groups;
determining a plurality of non-random unit groups from the plurality of unit groups at least in part by determining that a joint probability that a plurality of units of the plurality of unit groups are more likely to appear together in the corpus of items is greater than a probability of the plurality of units appearing separately in the corpus of items;
for each of the plurality of non-random unit groups, determining a discount factor based at least in part on a number of times the non-random unit group appears in the corpus of items; and
generating a unit group dictionary that includes non-random unit groups with a discount factor above a defined threshold.
2. The computer-implemented method of claim 1 , wherein each of the plurality of non-random unit groups includes at least two units.
3. The computer-implemented method of claim 1 , wherein each item of the corpus of items include at least one of a representation, an object identifier, a set, or a user.
4. The computer-implemented method of claim 1 , wherein the units of at least one non-random unit group of the plurality of non-random unit groups e include at least one of a character, a word, a symbol, or a number.
5. The computer-implemented method of claim 1 , wherein at least one of the non-random unit groups includes at least two units included in the annotations.
6. The computer-implemented method of claim 1 , wherein the joint probability is determined based at least in part on a number of times the units of the plurality of unit groups appear together in the corpus of items.
7. The computer-implemented method of claim 1 , wherein the probability is determined based at least in part on a number of times the units of the plurality of unit groups appear independently in the corpus of items.
8. The computer-implemented method of claim 1 , wherein determining a discount factor further includes determining a quotient of the number of times the non-random unit groups appear in the corpus of items and a number of item in the corpus of items.
9. The computer-implemented method of claim 1 , wherein a unit is any form of a character, a symbol, a number, or other identifier.
10. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to at least:
identify a corpus of items, wherein a plurality of the corpus of items include annotations;
determine a plurality of non-random unit groups at least in part by determining that a joint probability that a plurality of units of a unit group are more likely to appear together in the corpus of items is greater than a probability of the plurality of units of the unit group appearing separately in the corpus of items;
for each of the plurality of non-random unit groups, determine a discount factor based at least in part on a number of times the non-random unit groups appear in the corpus of items; and
generate a unit group dictionary that includes at least a portion of the plurality of non-random unit groups and corresponding discount factors.
11. The non-transitory computer-readable storage medium of claim 10 , wherein each of the plurality of non-random unit groups include at least two units.
12. The non-transitory computer-readable storage medium of claim 10 , wherein each item of the corpus of items include at least one of a representation, an object identifier, a set, or a user.
13. The non-transitory computer-readable storage medium of claim 10 , wherein the units of the non-random unit groups each include at least one of a character, a word, a symbol, or a number.
14. The non-transitory computer-readable storage medium of claim 10 , wherein the joint probability is determined based at least in part on a number of times the units of the unit group appear together in the corpus of items.
15. The non-transitory computer-readable storage medium of claim 10 , wherein the probability is determined based at least in part on a number of times the units of the unit group appear independently in the corpus of items.
16. The non-transitory computer-readable storage medium of claim 10 , wherein determining a discount factor further includes determining a quotient of the number of times the non-random unit groups appear in the corpus of items and a number of items in the corpus of items.
17. The non-transitory computer-readable storage medium of claim 10 , wherein the instructions, that when executed by the one or more processors, further cause the one or more processors to at least:
exclude from the unit group dictionary at least one non-random unit group with a corresponding discount factor that exceeds a threshold.
18. The non-transitory computer-readable storage medium of claim 10 , wherein the instructions, that when executed by the one or more processors, further cause the one or more processors to at least:
determine at least one pre-defined unit group from the plurality of non-random unit groups; and
exclude the at least one pre-defined unit group from the unit group dictionary.