Granted Patent B2

US 12,282,847 · App. 17/234,011 · Granted Apr 22, 2025

Inventors: Salah Ait-Mokhtar (Montbonnot-Saint-Martin, FR); Caroline Brun (Grenoble, FR); Agnes Sandor (Meylan, FR)

Assignee: NAVER CORPORATION

G06N3/08G06F16/24578G06F16/285G06F40/284G06F40/30G06N3/04G06F3/04842

Abstract

Claims (49)

1. An interactive annotation method for assisting a user in creation of annotated machine learning training data, the annotation method comprising:

providing, using a processor, a sequence of tokens to be annotated for display to a user in a graphical annotation interface, wherein the tokens comprise text;

receiving a span selection from the user via the graphical annotation interface, the span selection comprising one or more tokens from the sequence of tokens, wherein the user interacts with the graphical annotation interface using one or more selection devices;

in response to said received span selection, computing, by an artificial neural network using the processor, class probabilities for each token of the sequence of tokens, the class probabilities for a token corresponding to probabilities for the token to fall under respective classes of a predefined ontology;

computing, using the processor, scores for semantic attribute paths of the span selection, the scores for the semantic attribute paths being based on the class probabilities, wherein the semantic attribute paths correspond to paths in the predefined ontology;

providing a set of semantic attribute paths for the span selection for display to the user via the graphical annotation interface as an interactive graphical element;

receiving a user selection of a semantic attribute path from the set of displayed semantic attribute paths via the graphical annotation interface, wherein the user interacts with the graphical annotation interface using one or more selection devices; and

in response to said received user selection of a semantic attribute path, storing the sequence of tokens and the selected semantic attribute path for the span selection in the annotated machine learning training data.

2. The annotation method of claim 1 , wherein the displayed set of semantic attribute paths in the interactive graphical element ranks the semantic attribute paths by their respective score.

3. The annotation method of claim 1 , wherein computing the scores for semantic attribute paths of the span selection is based on the class probabilities for the tokens of the span selection and on the class probabilities of tokens surrounding the tokens of the span selection.

4. The annotation method of claim 1 , further comprising re-training the artificial neural network employing the stored machine learning training data.

5. The annotation method of claim 1 , wherein the displayed set of semantic attribute paths in the interactive graphical element comprises a scroll-down list or a drop-down menu of semantic attribute paths.

6. The annotation method of claim 5 , wherein receiving the user selection of a semantic attribute path comprises receiving a user input indicating acceptance of a semantic attribute path ranked highest in the list of displayed semantic attribute paths or receiving a selection of an alternative semantic attribute path from the list of semantic attribute paths.

7. The annotation method of claim 5 , wherein the semantic attribute path ranked highest in the list of semantic attribute paths is displayed in the scroll-down list or drop-down menu as a default user selection.

8. The annotation method of claim 5 , wherein receiving the user input indicating acceptance of the semantic attribute path ranked highest comprises receiving another span selection via the graphical annotation interface.

9. The annotation method of claim 1 , wherein the text is taken from a user generated comment.

10. The annotation method of claim 9 , wherein the user generated comment relates to a point of interest (POI).

11. The annotation method of claim 9 , wherein the user generated comment relates to a retailer.

12. The method of claim 1 , further comprising:

pre-training the artificial neural network for information extraction, said pre-training comprising:

assigning seed tokens for each class of a predefined ontology;

computing, using a processor, a similarity of all seed tokens assigned to the classes with vocabulary tokens in a vocabulary of pre-trained token vectors, and selecting a number of least similar vocabulary tokens as negative tokens;

creating, using the processor, artificial texts by combining the seed tokens with tokens sampled from the list of negative tokens; and

using the processor, employing the classes of the seed tokens as annotations of the artificial texts for training an initial artificial neural network to provide the pre-trained artificial neural network.

13. The method of claim 12 , further comprising optimizing, using the processor, the number of least similar vocabulary tokens.

14. The method of claim 12 , wherein the predefined ontology contains a first number of top classes and a second number of semantic classes.

15. An apparatus for assisting in creation of annotated machine learning training data, the apparatus comprising:

a processor; and

a memory;

wherein the processor is configured using code instructions stored in the memory to:

cause to be displayed to a user a sequence of tokens to be annotated via a graphical annotation interface, wherein the tokens comprise text, and wherein the graphical annotation interface is configured for receiving a span selection by the user;

assign class probabilities to each token of the sequence of tokens using an artificial neural network of an automatic span tagger, the class probabilities for a token corresponding to probabilities for the token to fall under respective classes of the predefined ontology;

compute scores for semantic attribute paths of the span selection using a path-scoring algorithm of the automatic span tagger, the scores for the semantic attribute paths being based on the class probabilities, wherein the semantic attribute paths correspond to paths in a predefined ontology;

cause the scored semantic attribute paths to be displayed to the user via the graphical annotation interface;

receive the selection among the scored semantic attribute paths from the user via the graphical annotation interface; and

store in the memory the sequence of tokens and the selected semantic attribute path for the span selection as annotated machine learning data.

16. The apparatus of claim 15 , wherein the graphical annotation interface comprises a train button to initiate re-training of the artificial neural network employing the stored machine learning training data.

17. The apparatus of claim 16 , wherein the path-scoring algorithm is configured to compute the scored semantic attribute paths for the span selection based on the class probabilities by employing the class probabilities of the tokens in the span selection and the class probabilities of tokens surrounding the tokens of the span selection as optimal paths in the predefined ontology.

18. The apparatus of claim 16 , wherein the artificial neural network is a multi-label multi-class classification neural network trained for classifying spans of texts into the classes.

19. The apparatus of claim 15 , further comprising:

a display for displaying the sequence of tokens to be annotated and the scored semantic attribute paths to the user; and

one or more selection devices for enabling the user to select the span selection and the selection among the scored semantic attribute paths.

20. A non-transitory computer-readable medium storing an interactive annotation program including instructions that, when executed by a processor, causes a graphical annotation interface and memory forming part of an apparatus for assisting a user in creation of annotated machine learning training data, to:

cause to be displayed to the user a sequence of tokens to be annotated via the graphical annotation interface, wherein the tokens comprise text, and wherein the graphical annotation interface is configured for receiving a span selection by the user;

assign class probabilities to each token of the sequence of tokens using an artificial neural network of an automatic span tagger, the class probabilities for a token corresponding to probabilities for the token to fall under respective classes of a predefined ontology;

compute scores for semantic attribute paths of the span selection using a path-scoring algorithm of the automatic span tagger, the scores for the semantic attribute paths being based on the class probabilities, wherein the semantic attribute paths correspond to paths in the predefined ontology;

cause the scored semantic attribute paths to be displayed to the user via the graphical annotation interface;

receive the selection among the scored semantic attribute paths from the user via the graphical annotation interface; and

store in the memory the sequence of tokens and the selected semantic attribute path for the span selection as annotated machine learning data.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (17)