IP Library › Granted Patent US 12,067,708
Granted Patent B2
US 12,067,708 · App. 17/419,285 · Granted Aug 20, 2024

Image judgement apparatus, image judgement method and non-transitory computer readable medium

Inventors: Yeongnam Chae (Tokyo, JP); Preetham Prakasha (Tokyo, JP)
Assignee: RAKUTEN GROUP, INC.
G06T7/0006G06N3/045G06V10/774G06V10/82G06N3/08G06T2207/20081G06T2207/20084
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,067,708
App. No.
17/419,285
Granted
Aug 20, 2024
Kind
B2
Abstract

To easily and correctly determine substantial identity of a plurality of images in each of which an object is placed, provided is an image judgement method including steps of obtaining first object data from a first image with a use of an R-CNN 12 , which is a first machine learning model, where the first object data indicates an attribute and a layout of an object in the first image, obtaining second object data from a second image with a use of the R-CNN 12 , where the second object data indicates an attribute and a layout of an object in the second image, and determining substantial identity of the first image and the second image based on the first object data and the second object data with a use of a CNN, which is a second machine learning model.

Claims (79)

1. An image judgement apparatus comprising:

one or more processors;

a memory storing instructions which, when executed by said one or more processor, cause said one or more processors to perform operations comprising:

obtaining first object data from a first image and second object data from a second image with a use of a first machine learning model in which an image is entered and which outputs object data indicating an attribute and a layout of an object in the image, the first object data indicating an attribute and a layout of an object in the first image, the second object data indicating an attribute and a layout of an object in the second image;

determining whether the first image and the second image are substantially identical with a use of a second machine learning model in which the first object data and the second object data are entered and which outputs whether the first image and the second image are substantially identical;

wherein the first image and the second image are substantially identical if a number of objects included in the first image and the second image, the respective attributes of the objects included in the first image and the second image, and the layout of the objects included in the first image and the second image satisfy a predetermined positive example rule;

wherein the first image is a screen image which is displayed by a computer application for a first execution environment;

wherein the second image is a screen image which is displayed by a computer application for a second execution environment; and

wherein the second execution environment is different from the first execution environment.

2. The image judgement apparatus according to claim 1 , wherein

the first machine learning model includes an R-CNN.

3. The image judgement apparatus according to claim 1 , wherein

the first machine learning model is trained by a training image that is generated by overlaying one or more objects on a predetermined base image.

4. The image judgement apparatus according to claim 1 , wherein

the second machine learning model includes fully connected layers.

5. The image judgement apparatus according to claim 4 , wherein

the second machine learning model includes a convolutional layer and a pooling layer, which reduce dimensionality of input data based on the first object data and the second object data, on an upstream side of the fully connected layers.

6. The image judgement apparatus according to claim 4 , wherein

the second machine learning model is trained by first learning object data indicating an attribute and a layout of an object in a first training image and second learning object data indicating an attribute and a layout of an object in a second training image, the first learning object data and the second learning object data being respectively obtained from the first training image and the second training image that are generated by overlaying a predetermined object on each of identical or similar a first base image and a second base image according to a predetermined layout rule.

7. An image judgement method comprising:

obtaining first object data from a first image with a use of a first machine learning model, the first object data indicating an attribute and a layout of an object in the first image;

obtaining second object data from a second image with a use of the first machine learning model, the second object data indicating an attribute and a layout of an object in the second image;

determining whether the first image and the second image are substantially identical based on the first object data and the second object data with a use of a second machine learning model,

wherein the first image and the second image are substantially identical if a number of objects included in the first image and the second image, the respective attributes of the objects included in the first image and the second image, and the layout of the objects included in the first image and the second image satisfy a predetermined positive example rule;

wherein the first image is a screen image which is displayed by a computer application for a first execution environment;

wherein the second image is a screen image which is displayed by a computer application for a second execution environment; and

wherein the second execution environment is different from the first execution environment.

8. The image judgement method according to claim 7 , wherein

the first machine learning model includes an R-CNN.

9. The image judgement method according to claim 7 further comprising training the first machine learning model by a training image that is generated by overlaying one or more objects on a predetermined base image.

10. The image judgement method according to claim 7 , wherein

the second machine learning model includes fully connected layers.

11. The image judgement method according to claim 10 , wherein

the second machine learning model includes a convolutional layer and a pooling layer, which reduce dimensionality of input data based on the first object data and the second object data, on an upstream side of the fully connected layers.

12. The image judgement method according to claim 10 further comprising:

overlaying a predetermined object on each of identical or similar a first base image and a second base image according to a predetermined layout rule so as to generate a first training image and a second training image;

inputting the first training image and the second training image to the first machine learning model so as to obtain first learning object data and second learning object data, the first learning object data indicating an attribute and a layout of an object in the first training image, the second learning object data indicating an attribute and a layout of an object in the second training image; and

training the second machine learning model by the first learning object data and the second learning object data.

13. A non-transitory computer-readable information storage medium storing a program for causing a computer to:

obtain first object data from a first image with a use of a first machine learning model, the first object data indicating an attribute and a layout of an object in the first image;

obtain second object data from a second image with a use of the first machine learning model, the second object data indicating an attribute and a layout of an object in the second image;

determine whether the first image and the second image are substantially identical based on the first object data and the second object data with a use of a second machine learning model;

wherein the first image and the second image are substantially identical if a number of objects included in the first image and the second image, the respective attributes of the objects included in the first image and the second image, and the layout of the objects included in the first image and the second image satisfy a predetermined positive example rule;

wherein the first image is a screen image which is displayed by a computer application for a first execution environment;

wherein the second image is a screen image which is displayed by a computer application for a second execution environment; and

wherein the second execution environment is different from the first execution environment.

14. An image judgement apparatus comprising:

one or more processors;

a memory storing instructions which, when executed by said one or more processor, cause said one or more processors to perform operations comprising:

obtaining first object data from a first image and second object data from a second image with a use of a first machine learning model in which an image is entered and which outputs object data indicating an attribute and a layout of an object in the image, the first object data indicating an attribute and a layout of an object in the first image, the second object data indicating an attribute and a layout of an object in the second image;

determining whether the first image and the second image are substantially identical with a use of a second machine learning model in which the first object data and the second object data are entered and which outputs whether the first image and the second image are substantially identical;

wherein the first image and the second image are substantially identical if a number of objects included in the first image and the second image, the respective attributes of the objects included in the first image and the second image, and the layout of the objects included in the first image and the second image satisfy a predetermined positive example rule;

wherein the second machine learning model includes fully connected layers;

wherein the second machine learning model is trained by first learning object data indicating an attribute and a layout of an object in a first training image and second learning object data indicating an attribute and a layout of an object in a second training image, the first learning object data and the second learning object data being respectively obtained from the first training image and the second training image that are generated by overlaying a predetermined object on each of identical or similar a first base image and a second base image according to a predetermined layout rule;

wherein the first training image is generated by overlaying the predetermined object at a predetermined position and in a predetermined size on the first base image;

wherein the second training image is generated without overlaying the predetermined object on the second base image; and

wherein the second machine learning model is trained that the first training image and the second training image are not substantially identical.

15. An image judgement apparatus comprising:

one or more processors;

a memory storing instructions which, when executed by said one or more processor, cause said one or more processors to perform operations comprising:

obtaining first object data from a first image and second object data from a second image with a use of a first machine learning model in which an image is entered and which outputs object data indicating an attribute and a layout of an object in the image, the first object data indicating an attribute and a layout of an object in the first image, the second object data indicating an attribute and a layout of an object in the second image;

determining whether the first image and the second image are substantially identical with a use of a second machine learning model in which the first object data and the second object data are entered and which outputs whether the first image and the second image are substantially identical;

wherein the first image and the second image are substantially identical if a number of objects included in the first image and the second image, the respective attributes of the objects included in the first image and the second image, and the layout of the objects included in the first image and the second image satisfy a predetermined positive example rule;

wherein the second machine learning model includes fully connected layers;

wherein the second machine learning model is trained by first learning object data indicating an attribute and a layout of an object in a first training image and second learning object data indicating an attribute and a layout of an object in a second training image, the first learning object data and the second learning object data being respectively obtained from the first training image and the second training image that are generated by overlaying a predetermined object on each of identical or similar a first base image and a second base image according to a predetermined layout rule;

wherein the first training image is generated by overlaying the predetermined object at a predetermined position and in a predetermined size on the first base image;

wherein the second training image is generated by overlaying at a different position from the predetermined position on the second base image; and

wherein the second machine learning model is trained that the first training image and the second training image are not substantially identical.

16. An image judgement apparatus comprising:

one or more processors;

a memory storing instructions which, when executed by said one or more processor, cause said one or more processors to perform operations comprising:

obtaining first object data from a first image and second object data from a second image with a use of a first machine learning model in which an image is entered and which outputs object data indicating an attribute and a layout of an object in the image, the first object data indicating an attribute and a layout of an object in the first image, the second object data indicating an attribute and a layout of an object in the second image;

determining whether the first image and the second image are substantially identical with a use of a second machine learning model in which the first object data and the second object data are entered and which outputs whether the first image and the second image are substantially identical;

wherein the first image and the second image are substantially identical if a number of objects included in the first image and the second image, the respective attributes of the objects included in the first image and the second image, and the layout of the objects included in the first image and the second image satisfy a predetermined positive example rule;

wherein the second machine learning model includes fully connected layers;

wherein the second machine learning model is trained by first learning object data indicating an attribute and a layout of an object in a first training image and second learning object data indicating an attribute and a layout of an object in a second training image, the first learning object data and the second learning object data being respectively obtained from the first training image and the second training image that are generated by overlaying a predetermined object on each of identical or similar a first base image and a second base image according to a predetermined layout rule;

wherein the first training image is generated by overlaying the predetermined object at a predetermined position and in a predetermined size on the first base image;

wherein the second training image is generated by overlaying in a different size from the predetermined size on the second base image; and

wherein the second machine learning model is trained that the first training image and the second training image are substantially identical.

Assignments (2)
CHANGE OF NAME Recorded Jul 13, 2021
From: RAKUTEN, INC.
To: RAKUTEN GROUP, INC.
Reel/Frame 056845/0831 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 29, 2021
From: CHAE, YEONGNAM; PRAKASHA, PREETHAM
To: RAKUTEN, INC.
Reel/Frame 056694/0905 →
Continuity (1)
Related Publication 20220309648A1 · Sep 29, 2022