Evolving Cooperative Strategies in Multi-Agent Systems Using a Coevolutionary Algorithm

1. Evolving Cooperative Strategies in Multi-Agent Systems Using a Coevolutionary Algorithm Cesario C. Julaton III, Ramanathan S. Thinniyam, Una-May O’Reilly Computer Science and Artifical Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA 94720 Background Coevolutionary Algorithm • Cooperative Coevolutionary Genetic Algorithms • Used in search, optimization, and machine learning • Solution parameters are encoded as genes in chromosomes • Chromosomes are evaluated using a fitness function to see how well the corresponding solution answers the problem • Highly fit chromosomes are selected and participate in crossover and mutation to give rise to new potential solutions. • Relatively new subfield of genetic algorithms that can be broken down into subcomponents • Goals • Have a better understanding of coevolutionary algorithms • Create strategies in fixed environments that succeed in modified environments Problem Formulation • Simulation • Predators cooperatively work together to capture a prey in a square toroid • Predators • Move horizontally or vertically according to their relative positions from the prey • Except one, every predator also moves according to one other predator’s relative position from the prey • Prey Behavior • Prey 1: move away from • nearest predator • Prey 2: move to maximize • the mean distance from predators • Prey 3: move in a position • that is not adjacent to a predator • Genotype Encodings • 1 Predator: 8 genes • Remaining Predators: 64 genes 1 if captured Fitness = 0 otherwise 1 1 1 3 1 3 1 3 Results • Grid Sizes • 10x10 grown: 60% capture rate on a 10x10 grid, but the rate approaches 0 as the grid size increases • 50x50 grown: performs better in smaller grids than in larger grids • 100x100 grown: higher capture rate in larger grid sizes, including grids larger than training size • Adding/Removing a Predator • Adding a predator to an evolved strategy that either moves directly towards the prey or remains stationary reduces the team’s ability to capture the prey • Removing a predator from an evolved team significantly reduces the rate at which the strategy captures the prey • Prey Behavior • Using prey 1 during training gave rise to a strategy that performs at its best against prey 1 and performs better on prey 2 than on prey 3 • The strategy obtained by training with prey 2 performs close to perfection when tested on prey 2 (capture percentage above 98%), but it performs poorly when tested on prey 1 and on prey 3 (capture percentages close to 0) • The strategy grown on prey 3 does not have a capture percentage greater than 50% for prey 3, but it has capture percentages above 7% for prey 1 and prey 2 Discussion Future Research • Allow the predators to follow different rulesets • Introduce a prey that evolves during evolution • Compare coevolutionary algorithm with other evolutionary algorithms • Vary parameters affecting the simulation, such as the number of chromosomes in a subpopulation, number of generations, selection, crossover, mutation rate, etc. • Coevolutionary algorithm evolved a cohesive unit and the introduction of new, unseen agents in a team reduces the efficiency of the strategy • We saw the emergence of behavioral patterns humans can relate to • One predator waited at the bottom while others attacked from the top and from the left • To exploit prey 2’s weakness, the majority of the predators remain in one portion of the grid while one approaches the prey. Acknowledgements: C. C. Julaton III acknowledges support from the MIT Summer Research Program