Granted Patent B2

US 12,321,254 · App. 17/435,506 · Granted Jun 3, 2025

Inventors: Er-Xin Shang (Shanghai, CN); Qian-Ru Zhai (Shanghai, CN); Hao-Bin Cui (Shanghai, CN)

Assignee: Micro Focus LLC

G06F11/3684G06F9/454G06V10/20G06V10/40G06V30/10

Abstract

Claims (76)

1. A method comprising:

recording a test of a graphical user interface (GUI), wherein the GUI includes a user interface (UI) element, the UI element comprising an image feature and first text in a first human language;

obtaining an image of the GUI based on the recorded test;

locating the first text in the image of the GUI;

processing the image to remove the first text;

detecting the image feature of the UI element based on the removal of the first text, wherein the detected image feature is to be used to identify the UI element in the GUI;

generating an automated test script in the first human language based on the recorded test, information relating to the first text, and the image feature, wherein the automated test script is internationalized into a second human language;

obtaining a second image of a second version of the GUI, the second image comprising the second text in the second human language;

determining a distance between a first word or phrase in a second text and a second word or phrase in the second text;

grouping the first word or phrase into a first group and the second word or phrase into a second group based on the determined distance; and

comparing a first translation of the first group with the first text and a second translation of the second group with the first text to determine which one of the first group or the second group corresponds to the first text.

2. The method of claim 1 , further comprising:

detecting a text boundary of the first text, wherein the first text is removed from the UI element based on the detected text boundary.

3. The method of claim 1 , further comprising:

generating a second version of the automated test script to automatically test the second version of the GUI in the second human language.

4. The method of claim 3 , wherein

identifying the UI element in the second version of the GUI responsive to a determination that the translated second text corresponds to the first text.

5. The method of claim 4 , further comprising:

replaying the second version of the automated test script on the second version of the GUI, wherein the replay includes execution of a user interaction with the identified UI element in the second version of the GUI.

6. The method of claim 4 , further comprising:

providing the first text as a first input to a natural language processor; and

obtaining a first output from the natural language processor, the first output including a first meaning of the first text, wherein the information relating to the first text comprises the first output;

wherein determining whether the translated second text corresponds to the first text, further comprises:

providing the translated second text as a second input to the natural language processor;

obtaining a second output from the natural language processor, the second output including a second meaning of the second text; and

comparing the first meaning with the second meaning.

7. The method of claim 1 , wherein identifying the first text further comprises performing optical character recognition on the image of the GUI.

8. The method of claim 1 , further comprising:

determining that the second UI element has no text; and

detecting the second image feature of the second UI element without text removal from the second image, wherein the detected second image feature is to be used to identify the second UI element in the GUI.

9. The method of claim 1 , wherein the second human language comprises a different character set compared to the first human language.

10. An apparatus comprising:

a processor; and

a non-transitory computer readable medium on which is stored instructions that, when executed by the processor, cause the processor to:

record a test of a graphical user interface (GUI), wherein the GUI includes a user interface (UI) element, the UI element comprising an image feature and first text in a first human language;

obtain an image of the GUI based on the recorded test;

locate the first text in the image of the GUI;

process the image to remove the first text;

detect the image feature of the UI element based on the removal of the first text, wherein the detected image feature is to be used to identify the UI element in the GUI;

generate an automated test script in the first human language based on the recorded test, information relating to the first text, and the image feature, wherein the automated test script is internationalized into a second human language;

obtain a second image of a second version of the GUI, the second image comprising the second text in the second human language;

determine a distance between a first word or phrase in the second text and a second word or phrase in the second text;

group the first word or phrase into a first group and the second word or phrase into a second group based on the determined distance; and

compare a first translation of the first group with the first text and a second translation of the second group with the first text to determine which one of the first group or the second group corresponds to the first text.

11. The apparatus of claim 10 , wherein the instructions that, when executed by the processor, further cause the processor to:

detect a text boundary of the first text, wherein the first text is removed from the UI element based on the detected text boundary.

12. The apparatus of claim 10 , wherein the instructions further cause the processor to:

generate a second version of the automated test script to automatically test the second version of the GUI in the second human language.

13. The apparatus of claim 12 , wherein the instructions further cause the processor to:

identify the UI element in the second version of the GUI responsive to a determination that the translated second text corresponds to the first text.

14. The apparatus of claim 13 , wherein the instructions further cause the processor to:

replay the second version of the automated test script on the second version of the GUI, wherein the replay includes execution of a user interaction with the identified UI element in the second version of the GUI.

15. The apparatus of claim 13 , wherein the instructions are further to cause the processor to:

provide the first text as a first input to a natural language processor; and

obtain a first output from the natural language processor, the first output including a first meaning of the first text, wherein the information relating to the first text comprises the first output;

wherein to determine whether the translated second text corresponds to the first text, the instructions are further to cause the processor to:

provide the translated second text as a second input to the natural language processor;

obtain a second output from the natural language processor, the second output including a second meaning of the second text; and

compare the first meaning with the second meaning.

16. The apparatus of claim 10 , wherein to identify the first text, the instructions are further to cause the processor to:

perform optical character recognition on the image of the GUI.

17. The apparatus of claim 10 , wherein the instructions are further to cause the processor to:

determine that a second UI element has no text; and

detect a second image feature of the second UI element without text removal from the second image, wherein the detected second image feature is to be used to identify the second UI element in the GUI.

18. The apparatus of claim 10 , wherein the second human language comprises a different character set compared to the first human language.

19. A non-transitory computer readable medium comprising instructions that, when read by a machine, cause the machine to perform:

recording a test of a graphical user interface (GUI), wherein the GUI includes a user interface (UI) element, the UI element comprising an image feature and first text in a first human language;

obtaining an image of the GUI based on the recorded test;

locating the first text in the image of the GUI;

processing the image to remove the first text;

detecting the image feature of the UI element based on the removal of the first text, wherein the detected image feature is to be used to identify the UI element in the GUI;

generating an automated test script in the first human language based on the recorded test, information relating to the first text, and the image feature, wherein the automated test script is internationalized into a second human language;

obtaining a second image of a second version of the GUI, the second image comprising the second text in the second human language

determining a distance between a first word or phrase in a second text and a second word or phrase in the second text;

grouping the first word or phrase into a first group and the second word or phrase into a second group based on the determined distance; and

comparing a first translation of the first group with the first text and a second translation of the second group with the first text to determine which one of the first group or the second group corresponds to the first text.

Assignments (2)

Continuity (1)

References Cited (23)