Granted Patent B2

US 12,354,027 · App. 15/943,947 · Granted Jul 8, 2025

Inventors: Mark Bishop Ring (Anaheim, CA); Satinder Baveja (Ann Arbor, MI); Peter Stone (Austin, TX); James MacGlashan (Riverside, RI); Samuel Barrett (Somerville, MA); Roberto Capobianco (Itri, IT); Varun Kompella (Aachen, DE); Kaushik Subramanian (Richmond, CA); Peter Wurman (Acton, MA)

Assignee: SONY GROUP CORPORATION

G06N5/043G06N20/00

Abstract

Claims (44)

1. A method for training an artificial intelligent agent to recognize a goal configuration, comprising:

placing the agent in the goal configuration and identifying a resulting state as a positive example;

providing negative examples to the agent that demonstrate the agent in a state failing to achieve the goal configuration;

extracting key state features when the agent is in the goal configuration, the key state features including at least one of a room feature, object positioning, ambient lighting, and ambient sounds;

determining what feature categories are important in the goal configuration during receipt of positive examples to the agent;

learning and recognizing, by the agent, the goal configuration based on the extracted key state features and the determined important feature categories;

creating policies, by the agent, based on the learned goal configuration;

converting state features into a distance function to determine how far the agent is from the goal configuration;

using goal detection as a final reward; and

using a goal distance as an intermediate reward.

2. The method of claim 1 , wherein an interface is used to indicate an example as being either the positive example or the negative example, the interface includes at least one of a spoken word received by the agent, an electronic signal received from a computing device, and a physical button on the agent.

3. The method of claim 1 , wherein the step of extracting key state features includes looking for similarity in state features in each of the positive and negative examples.

4. The method of claim 3 , further comprising increasing a confidence of the agent as the positive and negative examples are received by the agent.

5. The method of claim 4 , wherein the agent takes an action upon reaching a predetermined level of confidence.

6. The method of claim 1 , wherein the key state features are weighted according to a predetermined weight value.

7. The method of claim 1 , further comprising asking, by the agent, for human feedback regarding whether the agent is in a goal state.

8. A system comprising a processor and a computer-usable medium embodying a computer program code, the computer program code comprising instructions executable by the processor and configured to provide a method of learning to recognize a goal configuration of an artificial agent, the method comprising:

placing the agent in the goal configuration and identifying a resulting state as a positive example;

providing negative examples to the agent that demonstrate the agent in a state failing to achieve the goal configuration;

extracting key state features when the agent is in the goal configuration, the key state features including at least one of a room feature, object positioning, ambient lighting, and ambient sounds;

determining what feature categories are important in the goal configuration during receipt of positive examples to the agent;

learning and recognizing, by the agent, the goal configuration based on the extracted key state features and the determined important feature categories;

creating policies, by the agent, based on the learned goal configuration;

converting state features into a distance function to determine how far the agent is from the goal configuration;

using the distance function as an intermediate reward for the agent; and

using goal detection as a final reward.

9. The system of claim 8 , wherein the method further comprises recognizing whether the agent is in an initialization state.

10. The system of claim 8 , wherein the method further comprises self-practice by the agent.

11. The system of claim 10 , wherein a selected goal configuration for self-practice is selected based on at least one of a random determination, which goal configuration needs the most improvement, which goal configuration is most likely to improve, which goal configuration has been used least recently, and which goal configuration is most used.

12. The system of claim 10 , wherein the method further comprises biasing action choices based on which actions are important to achieving the goal configuration.

13. The system of claim 8 , wherein the method further comprises updating a policy for achieving a goal configuration based on performance of the agent.

14. A non-transitory computer-readable storage medium with an executable program stored thereon, wherein the program instructs one or more processors to perform the following steps to cause an agent to recognize and learn a goal configuration:

placing the agent in the goal configuration and identifying a resulting state as a positive example;

providing negative examples to the agent that demonstrate the agent in a state failing to achieve the goal configuration;

extracting key state features when the agent is in the goal configuration, the key state features including at least one of a room feature, object positioning, ambient lighting, and ambient sounds;

determining what feature categories are important in the goal configuration during receipt of positive examples to the agent;

learning and recognizing, by the agent, the goal configuration based on the extracted key state features and the determined important feature categories;

creating policies, by the agent, based on the learned goal configuration;

converting state features into a distance function to determine how far the agent is from the goal configuration;

using the distance function as an intermediate reward for the agent; and using goal detection as a final reward.

15. The non-transitory computer-readable storage medium of claim 14 , wherein the step of extracting key state features includes looking for similarity of the key state features in each of the positive and negative examples.

16. The non-transitory computer-readable storage medium of claim 14 , wherein the program instructs one or more processors to perform the following steps:

increasing a confidence of the agent as additional ones of the positive and negative examples are received by the agent; and

taking an action by the agent upon reaching a predetermined level of confidence.

Assignments (4)

Continuity (1)

References Cited (42)

Cited By (1)

US 12,650,895