Method and apparatus for providing real-time machine learning to computer-controlled agents used in video games
View Patent ↗A method for providing real-time machine learning to agents used in video games is disclosed. Initially, adjusted fitnesses of a set of agents within a population are determined. An agent having a relatively low adjusted fitness is then removed from the population. Subsequently, a parent agent is probabilistically selected from the population to generate an offspring agent. Finally, the removed agent is replaced by the offspring agent, maintaining a continuous real-time replacement cycle.
1. A method for providing real-time machine learning to agents used in an interactive application having multiple agents, said method comprising:
determining adjusted fitnesses of a plurality of agents within a population;
removing from said population an agent having a relatively low adjusted fitness;
probabilistically selecting a parent agent from said population to generate an offspring agent; and
replacing said removed agent with said offspring agent.
2. The method of claim 1 , wherein said removing further includes removing from said population an agent having the lowest adjusted fitness.
3. The method of claim 1 , wherein said selecting further includes
dividing said population into a plurality of species;
determining an average fitness for each of said species; and
probabilistically selecting one of said plurality of species based on said determined average fitnesses, wherein said parent agent is chosen from said selected species.
4. The method of claim 3 , wherein said selecting is performed by
Pr
(
S
k
)
=
F
k
_
F
_
tot
where Pr(S k ) is the probability of choosing a species S k that contain said parent agent, F k is the average fitness of species k, and F tot is the total of all of said species' average fitnesses.
5. The method of claim 3 , wherein said method further includes
in response to a determination that the number of species within said population exceeds a first predetermined threshold, decreasing a species compatibility threshold;
in response to a determination that the number of species within said population falls below a second predetermined threshold, increasing said species compatibility threshold; and
reassigning said plurality of agents to corresponding species.
6. An apparatus for providing real-time machine learning to agents used in an interactive application having multiple agents, said apparatus comprising:
means for determining adjusted fitnesses of a plurality of agents within a population;
means for removing from said population an agent having a relatively low adjusted fitness;
means for probabilistically selecting a parent agent from said population to generate an offspring agent; and
means for replacing said removed agent with said offspring agent.
7. The apparatus of claim 6 , wherein said removing means further includes means for removing from said population an agent having the lowest adjusted fitness.
8. The apparatus of claim 6 , wherein said selecting means further includes
means for dividing said population into a plurality of species;
means for determining an average fitness for each of said species; and
means for probabilistically selecting one of said plurality of species based on said determined average fitnesses, wherein said parent agent is chosen from said selected species.
9. The apparatus of claim 8 , wherein said selecting means is performed by
Pr
(
S
k
)
=
F
k
_
F
_
tot
where Pr(S k ) is the probability of choosing a species S k that contain said parent agent, F k is the average fitness of species k, and F tot is the total of all of said species' average fitnesses.
10. The apparatus of claim 8 , wherein said apparatus further includes
in response to a determination that the number of species within said population exceeds a first predetermined threshold, means for decreasing a species compatibility threshold;
in response to a determination that the number of species within said population falls below a second predetermined threshold, means for increasing said species compatibility threshold; and
means for reassigning said plurality of agents to corresponding species.
11. A computer storage medium having a computer program product for providing real-time machine learning to agents used in an interactive application having multiple agents, said computer storage medium comprising:
computer program code for determining adjusted fitnesses of a plurality of agents within a population;
computer program code for removing from said population an agent having a relatively low adjusted fitness;
computer program code for probabilistically selecting a parent agent from said population to generate an offspring agent; and
computer program code for replacing said removed agent with said offspring agent.
12. The computer storage medium of claim 11 , wherein said computer program code for removing further includes computer program code for removing from said population an agent having the lowest adjusted fitness.
13. The computer storage medium of claim 11 , wherein said program code means for selecting further includes
computer program code for dividing said population into a plurality of species;
computer program code for determining an average fitness for each of said species; and
computer program code for probabilistically selecting one of said plurality of species based on said determined average fitnesses, wherein said parent agent is chosen from said selected species.
14. The computer storage medium of claim 13 , wherein said computer program code for selecting is performed by
Pr
(
S
k
)
=
F
k
_
F
_
tot
where Pr(S k ) is the probability of choosing a species S k that contain said parent agent, F k is the average fitness of species k, and F tot is the total of all of said species' average fitnesses.
15. The computer storage medium of claim 13 , wherein said computer storage medium further includes
computer program code, in response to a determination that the number of species within said population exceeds a first predetermined threshold, for decreasing a species compatibility threshold;
computer program code, in response to a determination that the number of species within said population falls below a second predetermined threshold, for increasing said species compatibility threshold; and
computer program code for reassigning said plurality of agents to corresponding species.
16. A method for managing agent population within a video game, said method comprising:
continuously collecting fitness statistics of agents within a video game that is being played by a player, wherein said agent population evolves continuously during said game play;
determining a number of ticks of a simulation clock between replacements n of said agents within said video game by
assigning a minimum lifetime m for any agent;
choosing a number of agents |P| within said agent population;
choosing a fraction I of said agent population, wherein said fraction is ineligible for replacement at any given simulation clock tick due to being below said minimum lifetime; and
dividing said minimum lifetime m by the product of said number of agents |P| and said fraction I; and
replacing a subset of said agents within said agent population after every n simulation clock ticks.
17. The method of claim 16 , wherein said assigning is performed based on a minimum amount of time in simulation clock ticks necessary to complete a task.
18. The method of claim 16 , wherein said choosing is performed based on the number of simultaneous agents a processor is capable of handling and the number most desirable for said player's experience.