Methods and systems for reidentification in a user interface

1 . A computer-implemented method ( 100 ) for performing description mining for a concept, the method comprising:

acquiring (S 110 ) a plurality of reference descriptions, wherein each of the plurality of reference descriptions is associated with a user interface (UI) element corresponding to a type of UI element represented by the concept;

generating (S 120 ), based on the plurality of reference descriptions, at least one of a local description for the concept and a global description for the concept, wherein the local description for the concept comprises a description component that is specific to a respective UI, and wherein the global description for the concept comprises a description component that is not specific to a UI;

storing (S 130 ) the at least one of generated local description and global description such that it is associated with a concept identifier corresponding to the concept;

executing a workflow comprising an action involving the UI element, wherein executing the workflow comprises:

retrieving, for the UI element, at least one of the stored local description and the stored global description, based on the concept identifier; and

querying a representation of a current UI using the at least one of stored local description and stored global description to determine whether the respective description can be used to reidentify the UI element;

wherein the method further comprises:

enabling, via the current UI, an interactive prompt requesting user identification when the UI element cannot be reidentified using the respective description; and

enabling a user input to identify the UI element via a highlighting action on the current UI in response to the interactive prompt.

2 . The method ( 100 ) according to claim 1 , wherein the concept identifier uniquely identifies the concept in a collection of concepts, the collection of concepts being an ontology of a plurality of different types of UI elements.

3 . The method ( 100 ) according to claim 1 , wherein each of the plurality of reference descriptions is in the form of one of: a Cascading Style Sheets, CSS, selector, an Extensible Markup Language Path Language, XPath, expression, and an image and/or audio description.

4 . The method ( 100 ) according to claim 1 , further comprising:

acquiring, for each of the plurality of reference descriptions, a corresponding UI representation,

wherein generating a local description for the concept or a global description for the concept is further based on an interrelationship between a reference description and the corresponding UI representation.

5 . The method ( 100 ) according to claim 4 , further comprising, prior to acquiring a representation of a corresponding UI for a respective reference description:

generating an updated UI representation by removing information content from an original UI representation,

wherein the acquired representation is the updated representation.

6 . The method ( 100 ) according to claim 1 , wherein generating a local description for the concept comprises determining a common description component from a plurality of local historical reference descriptions, or wherein generating a global description for the concept comprises determining a common description component from a plurality of non-local reference descriptions.

7 . The method ( 100 ) according to claim 1 , further comprising, prior to generating at least one of a local description and a global description for the concept:

determining whether a predetermined trigger condition is met,

wherein generation of at least one of a local description and a global description is only performed if it is determined that the predetermined trigger condition is met.

8 . The method ( 100 ) according to claim 7 , further comprising:

acquiring a user input, wherein the user input corresponds to assignment of the concept identifier to a first UI element in a respective UI,

storing a concept instance for the concept identifier, the concept instance comprising: a representation of the respective UI, a description of the first UI element, and the concept identifier;

wherein storage of the concept instance is a predetermined trigger condition.

9 . The method ( 100 ) according to claim 1 , further comprising:

acquiring a user input accepting or rejecting the at least one of generated local description and generated global description,

wherein storing of the generated local description or the generated global description is only performed upon acquiring the user input accepting the respective generated local description or generated global description.

10 . A computer-implemented method ( 200 ) for performing reidentification check for a user interface (UI) element, in a current UI, the method comprising:

retrieving (S 210 ), for the UI element, at least one of a stored first local description and a stored first global description, based on a concept identifier representing the UI element, wherein each of the at least one of the stored first local description and the stored first global description is generated based on a plurality of reference descriptions, each of the plurality of reference descriptions being associated with a UI element corresponding to a type of UI element corresponding to the concept identifier, wherein the stored first local description comprises a description component that is specific to a respective UI, and wherein the stored first global description comprises a description component that is not specific to a UI; and

querying (S 220 ) a representation of the current UI using the at least one first stored local description and first stored global description to determine whether the respective description can be used to reidentify the UI element;

enabling, via the current UI, an interactive prompt requesting user identification when the UI element cannot be reidentified using the respective description; and

enabling a user input to identify the UI element via a highlighting action on the current UI in response to the interactive prompt.

11 . The method ( 200 ) according to claim 10 , further comprising, if it is determined that the first local description and/or the first global description can be used to reidentify the UI element, storing the representation of the current UI and the first local description and/or the first global description such that they are associated with the concept identifier.

12 . The method ( 200 ) according to claim 10 , further comprising, if it is determined that the first local description and/or the first global description cannot be used to reidentify the UI element:

retrieving at least one of: a stored second local description and a stored second global description based on the concept identifier, wherein the second local description is different from the first local description and wherein the second global description is different from the first global description; and

querying the representation of the current UI using the at least one of second local description and second global description to determine whether the respective description can be used to reidentify the UI element.

13 . The method ( 200 ) according to claim 10 , further comprising, if it is determined that the first local description or the first global description cannot be used to reidentifying the UI element, performing at least one of:

requesting user input to indicate the UI element in the current UI; and

performing description mining for the concept, wherein performing description mining for the concept comprises:

acquiring a second plurality of reference descriptions, wherein each of the second plurality of reference descriptions is associated with a UI element represented by the concept;

generating, based on the second plurality of reference descriptions, at least one of a second local description for the concept and a second global description for the concept, wherein the second local description for the concept comprises a description component that is specific to a respective UI, and wherein the second global description for the concept comprises a description component that is not specific to a UI; and

storing the at least one of generated second local description and second global description such that it is associated with a concept identifier corresponding to the concept.

14 . A system ( 300 ) for performing description mining for a concept, the system comprising:

at least one processor; and

at least one memory storing instructions that, when executed by the at least one processor, cause the system to:

acquire a plurality of reference descriptions, wherein each of the plurality of reference descriptions is associated with a user interface (UI) element corresponding to a type of UI element represented by the concept;

generate, based on the plurality of reference descriptions, at least one of a local description for the concept and a global description for the concept, wherein the local description for the concept comprises a description component that is specific to a respective UI, and wherein the global description for the concept comprises a description component that is not specific to a UI; and

store the at least one of generated local description and global description such that it is associated with a concept identifier corresponding to the concept,

execute workflow comprising an action involving the UI element, wherein executing the workflow comprises:

retrieving, for the UI element, at least one of the stored local description and the stored global description, based on the concept identifier; and

querying a representation of a current UI using the at least one of stored local description and stored global description to determine whether the respective description can be used to reidentify the UI element;

enable, via the current UI, an interactive prompt requesting user identification when the UI element cannot be reidentified using the respective description; and

enable a user input to identify the UI element via a highlighting action on the current UI in response to the interactive prompt.

15 . A system ( 400 ) for performing reidentification check for a user interface (UI) element, in a current UI, the system comprising:

at least one processor; and

at least one memory storing instructions that, when executed by the at least one processor, cause the system to:

retrieve, for the UI element, at least one of a stored first local description and a stored first global description, based on a concept identifier representing the UI element, wherein each of the at least one of the stored first local description and the stored first global description is generated based on a plurality of reference descriptions, each of the plurality of reference descriptions being associated with a UI element corresponding to a type of UI element corresponding to the concept identifier, wherein the stored first local description comprises a description component that is specific to a respective UI, and wherein the stored first global description comprises a description component that is not specific to a UI; and

query a representation of the current UI using the at least one first stored local description and first stored global description to determine whether the respective description can be used to reidentify the UI element,

enable, via the current UI, an interactive prompt requesting user identification when the UI element cannot be reidentified using the respective description; and

enable a user input to identify the UI element via a highlighting action on the current UI in response to the interactive prompt.

16 . A non-transitory computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the steps of the method of claim 1 .