GA-based algorithms for finding equilibrium

1. GA-based algorithms for finding equilibrium Presenter: Chih-Yuan Chou

2. Agenda • “The evolution of strategies in the iterated prisoner's dilemma” (Axelrod, 1987). • “Genetic algorithm learning and evolutionary games” (Riechmann, 2001). • “Hybrid coevolutionary programming for Nash equilibrium search in games with local optima” (Son, 2004).

3. Agenda • “The evolution of strategies in the iterated prisoner's dilemma” (Axelrod, 1987). • “Genetic algorithm learning and evolutionary games” (Riechmann, 2001). • “Hybrid coevolutionary programming for Nash equilibrium search in games with local optima” (Son, 2004).

4. prisoner's dilemma

5. Iterated Prisoner's Dilemma • The same two individuals may meet more than once. • An individual can recognize a previous interactant and remember some aspects of the prior outcomes. • A strategy is a decision rule which specifies the probability of cooperation or defection as a function of the history of the interaction so far.

6. 5-stage simulation • 1. Initialize the population by generate random strings of C's and D's. • 2. each individual player uses the strategy defined by its chromosome to play in iterated Prisoner's Dilemma with other strategies • 3. The relatively successful individuals are selected to have more offspring.

7. 5-stage simulation • 4. crossover and mutation • 5. The new population will display patterns of behavior that are more like those of the successful individuals of the previous generation.

8. Simulation results • Cooperate after three mutual cooperations. • Defect after three mutual defections. • Defect when the other player defects out of the blue. • Continue to cooperate after cooperation has been restored.

9. TIT for tat • Cooperate on the first move • Do whatever the other player did on the preceding move

10. Agenda • “The evolution of strategies in the iterated prisoner's dilemma” (Axelrod, 1987). • “Genetic algorithm learning and evolutionary games” (Riechmann, 2001). • “Hybrid coevolutionary programming for Nash equilibrium search in games with local optima” (Son, 2004).

11. Canonical ga

12. economic GA • Definition: An economic genetic algorithm is a genetic algorithm with a state-dependent fitness function. • The fitness of an economic agent does not only depend on her own strategy, but also on the strategies of all other agents involved in the model.

13. Genetic populations as near Nash equilibrium • Apopulation can be described by counting how often each of the differentpossible strategies is used by the members of the population. • The population tends over time to move to a Nash equilibrium without fully reaching it.

14. evolutionarily stable states(ESS) • a strategy is evolutionarily stable if, relative to its population, it performs better than any ‘invading’ strategy. • ESS are based on the notion that invading ‘mutant’ strategies are somehow rejected or eliminated from the population.

15. set up a concept of Evolutionary stabilityfor gA • A genetic population is evolutionarily stable if the process of the genetic algorithm rejects an invasion by one or more strategies from the genetic population. • Invasion itself can either take the form of a totally new strategy entering the population or it can simply mean a change in the frequency of the strategies already contained within the population. • A more precise definition of an evolutionarily stable population might be: A population is evolutionarily stable if it is resistant against changes in its composition.

16. set up a concept of Evolutionary stabilityfor gA • If a genetic population is evolutionarily stable, it will recover from an invasion within a finite number of steps of the GA, which means that in the long run the population will not lastingly be changed.

17. Conclusion of this paper • Economic genetic algorithm learning can be shown to be a specific form of an evolutionary game. • The results of this paper demonstrate that research in the dynamics of economic GA learning models can be equipped with the whole tool box of evolutionary game theory.

18. Agenda • “The evolution of strategies in the iterated prisoner's dilemma” (Axelrod, 1987). • “Genetic algorithm learning and evolutionary games” (Riechmann, 2001). • “Hybrid coevolutionary programming for Nash equilibrium search in games with local optima” (Son, 2004).

19. motivation • The conventional NE search algorithms and coevolutionary algorithms can not differentiate real NE and “local NE traps”

20. Contributions of this paper • Prove that any conventional iterative NE search algorithms applied to games with local optima can misidentify NE by following a local optimization path. • Introduce the concept of “local NE trap” that attracts the conventional iterative NE search algorithms based on local optimization tools. • Develop an enhanced version of coevolutionary programming to overcome the problem.

21. Local NE trap • Definition 1 (Nash Equilibrium): is an NE if . • Definition 2 (Local NE): is a local NE if where .

22. Local NE trap • Definition 3 (Local NE Trap): is a local NE trap if but

23. Local NE trap

24. local NE trap

25. Model of iterative NE search algorithms • 1) Initialize each player’s strategy choice set. • 2) Given other players’ decisions, solve each player’s profit maximization problem. Do this step for all players. • 3) Repeat Step 2) until the NE condition is satisfied or the maximum iteration number is reached.

26. Model of iterative NE search algorithms • Proposition: Every local NE, including any local NE trap, can be the solution of any conventional iterative NE search algorithm.

27. example

28. example

29. coevolutionary programming • Each generation of the genetic algorithm-based coevolutionary programming consists of two parts: • The random matching • The evolutionary process of each player

30. coevolutionary programming

31. coevolutionary programming

32. Whole process of coevolutionary programming • 1) Set the basic parameters of maximum generation number, population size, crossover rate, and mutation rate. • 2) Initialize the strategy population of each player. • 3) Choose one strategy from the strategy population of each player randomly from among the strategies that have not already been assigned profits. Input the strategy information to the tournament. The result of the tournament will decide profit values for these chosen strategies. • 4) Repeat Step 3) until every strategy is assigned a profit value. • 5) Apply the evolutionary operators to each strategy population. Keep the best strategy of the current generation alive (elitism). • 6) Repeat Steps 3)–5) until the maximum generation number is reached.

33. Hybrid coevolutionary programming • “best rival matching and fine tuning.” • The chosen individual will be matched against the best strategy of the other populations in the current generation.

34. The algorithmic steps • 1) Randomly choose a strategy from the strategy population of a player. • 2) Take the best strategies of all other rivals from the previous generation. • 3) Apply a local hill climber for fine tuning of the chosen strategy of the player. • 4) Repeat Steps 1)–3) as many times as set by the best rival matching rate. For example, with a best rival matching rate of 0.2, 20% of strategies in each population will be chosen. • 5) Repeat Steps 1)–4) for every other player in turn.

35. The algorithmic steps

36. Simulation • This paper compared three algorithms: • Conventional NE search algorithm • Simple coevolutionary programming • Hybrid coevolutionary programming

37. Game configuration • The profit functions for player A and player B:

38. Game configuration • NE and local NE trap of the numerical game example

39. Conventional ne search • Find multiple solutions including NE and local NE trap without being able to differentiate the NE from the local NE traps.

40. Local iterative search • Iterative NE search result

41. Coevolutionary programming

42. Hybrid Coevolutionary programming

43. Conclusion of this paper

44. Conclusion of this paper • Prove that any iterative NE search algorithms based on local optimization can not differentiate between the real NE and the “local NE traps” • Suggested an enhancement of coevolutionaryprogramming to solve games with local optima. • Hybrid coevolutionaryprogramming converged to the NE, if it existed, and did not converge to any local NE traps.

45. reference • “The evolution of strategies in the iterated prisoner's dilemma” Axelrod, 1987. • “Genetic algorithm learning and evolutionary games” Riechmann, 2001. • “Hybrid coevolutionary programming for Nash equilibrium search in games with local optima” Son, 2004.