Image data augmentation using user interface element attributes

1. A client computer system configured to augment training, testing, or validation images of software objects, the client computer system comprising:

a memory; and

at least one processor coupled to the memory and configured to:

receive a user selection of a portion of a display view;

iteratively select, within a first respective iteration, a first respective attribute value from a predetermined set of attribute values;

iteratively modify an attribute of a software object according to the first respective attribute value, wherein the attribute comprises a foreground position of the software object, and wherein the software object comprises at least one portion of a locally executed application, a web page, a web application, or a software as a service (SaaS) application;

iteratively generate a respective augmented image of the software object with the attribute modified according to the first respective attribute value of the first respective iteration and based on the user selection of the portion of the display view; and

train, test, or validate a machine learning (ML) process based on the respective augmented image.

2. The client computer system of claim 1 , wherein to iteratively modify the attribute of the software object comprises:

to receive:

a user selection of:

one or more user interface (UI) elements belonging to the software object; and

the attribute, wherein the attribute characterizes the one or more UI elements; and

a user specification of the predetermined set of attribute values; and

to iteratively modify the attribute characterizing the one or more UI elements according to the first respective attribute value.

3. The client computer system of claim 1 , wherein to modify the attribute of the software object is based on one or more of:

a UI automation application programming interface (API);

hooking;

a modification of a hypertext markup language (HTML) document associated with the software object;

a modification of a document object model (DOM) associated with the software object;

a modification of a cascading style sheet (CSS) attribute associated with the attribute of the software object; or

a data augmentation toolbox.

4. The client computer system of claim 1 , wherein to modify the attribute of the software object comprises to modify an HTML document corresponding to the software object via a headless browser session.

5. The client computer system of claim 1 , wherein the attribute of the software object further comprises one or more of:

a window size or aspect ratio;

text contents;

a text size or color;

a font style;

a theme;

an icon;

a window title size or color; or

a button appearance or button text.

6. The client computer system of claim 1 , wherein the attribute is a first attribute and the at least one processor is further configured to:

iteratively select a second respective attribute value from a second predetermined set of attribute values; and

iteratively modify a second attribute of the software object according to the second respective attribute value; and

wherein to iteratively generate the respective augmented image comprises to iteratively generate the respective augmented image with the first attribute modified according to the first respective attribute value and the second attribute modified according to the second respective attribute value.

7. The client computer system of claim 1 , wherein the software object comprises one or more of: an executable application, a window, a dialog box, another UI element, a web page, a web app, or a software as a service (SaaS) app.

8. The client computer system of claim 1 , wherein

to iteratively modify the attribute of the software object comprises to overlay the software object at the foreground position over a background object, while a position of the background object remains unmodified.

9. A method of augmenting training, testing, or validation images of software objects, the method comprising:

receiving a user selection of a portion of a display view;

iteratively selecting, by a processing device and within a first respective iteration, a first respective attribute value from a predetermined set of attribute values;

iteratively modifying an attribute of a software object according to the first respective attribute value, wherein the attribute comprises a foreground position of the software object, and wherein the software object comprises at least one portion of a locally executed application, a web page, a web application, or a software as a service (SaaS) application;

iteratively generating a respective augmented image of the software object with the attribute modified according to the first respective attribute value of the first respective iteration and based on the user selection of the portion of the display view;

iteratively storing the respective augmented image within a training, testing, or validation dataset in a non-transitory storage; and

training, testing, or validating a machine learning (ML) process based on the training, testing, or validation dataset.

10. The method of claim 9 , wherein iteratively modifying the attribute of the software object comprises:

receiving:

a user selection of:

one or more user interface (UI) elements belonging to the software object; and

the attribute, wherein the attribute characterizes the one or more UI elements; and

a user specification of the predetermined set of attribute values; and

iteratively modifying the attribute characterizing the one or more UI elements according to the first respective attribute value.

11. The method of claim 9 , wherein modifying the attribute of the software object is based on one or more of:

a UI automation application programming interface (API);

hooking;

modifying a hypertext markup language (HTML) document associated with the software object;

modifying a document object model (DOM) associated with the software object;

modifying a cascading style sheet (CSS) attribute associated with the software object; or a data augmentation toolbox.

12. The method of claim 9 , wherein the attribute of the software object further comprises one or more of:

a window size or aspect ratio;

text contents;

a text size or color;

a font style;

a theme;

an icon;

a window title size or color; or

a button appearance or button text.

13. The method of claim 9 , wherein the attribute is a first attribute, and further comprising:

iteratively selecting a second respective attribute value from a second predetermined set of attribute values; and

iteratively modifying a second attribute of the software object according to the second respective attribute value; and

wherein iteratively generating the respective augmented image comprises iteratively generating the respective augmented image with the first attribute modified according to the first respective attribute value and the second attribute modified according to the second respective attribute value.

14. The method of claim 9 , further comprising: wherein:

training the ML process comprises training the ML process based on the generated augmented image to obtain trained parameter values;

testing the ML process comprises testing the ML process based on the generated augmented image to obtain an evaluated model; or

validating the ML process comprises validating the ML process based on the generated augmented image to obtain tuned hyperparameter values.

15. A non-transitory computer readable medium storing executable sequences of instructions to augment training, testing, or validation images of software objects, the sequences of instructions comprising instructions to:

receive a user selection of a portion of a display view;

iteratively select, within a first respective iteration, a first respective attribute value from a predetermined set of attribute values;

iteratively modify an attribute of a software object according to the first respective attribute value, wherein the attribute comprises a foreground position of the software object, and wherein the software object comprises at least one portion of a locally executed application, a web page, a web application, or a software as a service (SaaS) application;

iteratively generate a respective augmented image of the software object with the attribute modified according to the first respective attribute value of the first respective iteration and based on the user selection of the portion of the display view; and

train, test, or validate a machine learning (ML) process based on the respective augmented image.

16. The non-transitory computer readable medium of claim 15 , wherein to iteratively modify the attribute of the software object comprises:

to receive:

a user selection of:

one or more user interface (UI) elements belonging to the software object; and

the attribute, wherein the attribute characterizes the one or more UI elements; and

a user specification of the predetermined set of attribute values; and

to iteratively modify the attribute characterizing the one or more UI elements according to the first respective attribute value.

17. The non-transitory computer readable medium of claim 15 , wherein to modify the attribute of the software object is based on one or more of:

a UI automation application programming interface (API);

hooking;

a modification of a hypertext markup language (HTML) document associated with the software object;

a modification of a document object model (DOM) associated with the software object;

a modification of a cascading style sheet (CSS) attribute associated with the software object; or a data augmentation toolbox.

18. The non-transitory computer readable medium of claim 15 , wherein the attribute is a first attribute, and the sequences of instructions further comprise instructions to:

iteratively select a second respective attribute value from a second predetermined set of attribute values; and

iteratively modify a second attribute of the software object according to the second respective attribute value; and

wherein to iteratively generate the respective augmented image comprises to iteratively generate the respective augmented image with the first attribute modified according to the first respective attribute value and the second attribute modified according to the second respective attribute value.

19. The client computer system of claim 1 , wherein the processor is further configured to:

iteratively select, in a second respective iteration, a second respective attribute value from the predetermined set of attribute values;

iteratively modify the attribute according to the second respective attribute value; and

iteratively generate a second respective augmented image of the software object with the attribute modified according to the second respective attribute value of the second respective iteration.