Enhanced social learning via trust and reputation mechanisms in multi agent systems
Download
1 / 74

Enhanced Social Learning via Trust and Reputation Mechanisms in Multi-agent Systems - PowerPoint PPT Presentation


  • 126 Views
  • Uploaded on

Enhanced Social Learning via Trust and Reputation Mechanisms in Multi-agent Systems. PhD Completion Seminar Golriz Rezaei Supervisors: Dr. Michael Kirley Dr. Shanika Karunasekera Dept. Computer Science and Software Engineering The University of Melbourne, Australia 20 April 2011. Outline.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Enhanced Social Learning via Trust and Reputation Mechanisms in Multi-agent Systems' - gada


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Enhanced social learning via trust and reputation mechanisms in multi agent systems

Enhanced Social Learning via Trust and Reputation Mechanisms in Multi-agent Systems

PhD Completion Seminar

Golriz Rezaei

Supervisors:

Dr. Michael Kirley

Dr. Shanika Karunasekera

Dept. Computer Science and Software Engineering

The University of Melbourne, Australia

20 April 2011


Outline
Outline in Multi-agent Systems

  • Overview

    • Motivation

    • Enhanced Social Learning

    • Research Goals / Questions / contributions / publications

  • Background

    • Trust and Reputation in Multi-agent Systems

    • Trust and Reputation in Evolutionary Game Theory

    • Evolutionary Games on Graphs

  • The Research work

    • First Model

    • Second Model

    • Third Model

  • Concluding Discussion

  • Acknowledgment and Questions?


  • Motivation
    Motivation in Multi-agent Systems

    • Multi-agent Systems (MAS)?

      • Interacting autonomous agents

      • Different geographical locations

      • Varying cognitive / processing abilities

      • Limited information / partial knowledge

    • Perform tasks  Receive utility

    • Difficult tasks  Beyond individual agent capacity

    • Maximise utility  Interact (collaboration / resource sharing)

      Problem?

      • Appropriate partners  Successful performance  Maximise utility

      • Open dynamic MAS  Uncertainty + Partial knowledge

        Establishing strategic connections is difficult!


    Enhanced social learning
    Enhanced Social Learning in Multi-agent Systems

    • Social Learning (biological background)?

      • Learning through observation / interaction with others

      • Knowledge transmission without genetic materials

      • Acquire knowledge from others without incurring the cost/time

    • Major mechanism  Imitation(perceive and reproduce behaviour)

    • Why good?

      • keep track of beneficial interaction partners

      • save time / energy / cost

      • Improve long term performance (individual / system)

      • Problem?  error-prone / outdated / inappropriate information


    Enhanced social learning cont
    Enhanced Social Learning cont. in Multi-agent Systems

    • Solution?  selective

      • When  High individual trial-and-error cost

      • Intermediate environment change rate

      • How  Mixed with personal innovation

      • From whom 

        • Agents are heterogeneous

        • Appropriate role models  Important for performance

        • Partner selection


    Enhanced social learning via trust and reputation mechanisms in multi agent systems

    Enhanced Social Learning cont. in Multi-agent Systems

    • Top-down

      • Plan at design time

      • Ability of the designer  predict optimal connections in advance

      • Fixed structure of relations (random / particular topology)

      • Autonomy condition + Environmental condition  not realistic

    • Automatic learning 

      • Build and sustained adaptively at run time

      • Trust & Reputation  Formal definition?

      • Evaluate before interaction  Partner selection / Decision making

      • Relations evolve  Partner’s reliability / trustworthiness

        Survey in Ch2

    Evolutionary game theory

    Concrete App MAS


    Coevolutionary endogenous social networks
    Coevolutionary Endogenous Social Networks in Multi-agent Systems

    Dynamic relation formation

    Social

    ties

    Agents’

    strategies

    Topology

    Behaviour


    Proposed framework
    Proposed framework in Multi-agent Systems

    Enhanced Social Learning

    Trust & Reputation

    • Life-experiences

    • Endogenous Evolving

    • Social Networks

    •  Evaluation

    ?

    Social

    Learning

    1) Social Dilemma  Evolutionary Games

    2) Advice-seeking in

    Distributed Service Provision Applications


    Research goals and questions
    Research goals and questions in Multi-agent Systems

    Central hypothesis:

    “Does incorporating concepts of trust and reputation within a social learning framework help to enhance the agents’ interactions in a MAS? And consequently does it help to improve their long term performance?”

    (Life-experiences / Aging) + (Coevolutionary endogenous social networks)Trust / Reputation?  Effective social learning approaches?

    Encourage cooperation in social dilemmas? Broader perspective of general MAS applications (Advice-Seeking for Resource Discovery in Distributed Service Provision)

    Impacts of agents’ heterogeneity (behaviour/attributes/preferences)

    Structural characteristics of the underlying evolved relationship networks?

    Interaction patterns system's behaviour?

    Interaction pattern

    System behaviour


    Publications
    Publications in Multi-agent Systems

    • Life Experiences in Spatial 2-player Prisoners’ Dilemma Game

  • G. Rezaei and M. Kirley (2008). Heterogeneous payoffs and social diversity in the spatial prisoner's dilemma game. In X. Li, M. Kirley, and M. Zhang, editors, Proceedings of 7th International Conference on Simulated Evolution and Learning (SEAL), volume 5361 of Lecture Notes in Computer Science, pages 585--594, Springer.

  • G. Rezaei and M. Kirley (2009). The effects of time varying rewards on the evolution of cooperation. Evolutionary Intelligence, 2(4):207-218.

  • First Model


    Publications cont
    Publications cont. in Multi-agent Systems

    • N-player Prisoners' Dilemma Game on an Evolving Social Network

  • G. Rezaei, M. Kirley and J. Pfau (2009). Evolving cooperation in the N-player prisoner's dilemma: A social network model. In K. B. Korb, M. Randall, and T. Hendtlass, editors, Artificial Life: Borrowing from Biology (ACAL), volume 5865 of Lecture Notes in Computer Science, pages 32-42, Springer Verlag, Berlin.

  • An extended version is under preparation (2011).

  • Distributed Advice-Seeking on an Evolving Social Network

    • G. Rezaei, J. Pfau and M. Kirley (2010). In Distributed Advice-Seeking on an Evolving Social Network. 2010 IEEE/WIC/ACM International Conference on Intelligent Agent Technology.

  • Second Model

    Third Model


    Outline1
    Outline in Multi-agent Systems

    • Overview

      • Motivation

      • Enhanced Social Learning

      • Research Goals / Questions / contributions / publications

    • Background

      • Trust and Reputation in Multi-agent Systems

      • Trust and Reputation in Evolutionary Game Theory

      • Evolutionary Games on Graphs

  • The Research work

    • First Model

    • Second Model

    • Third Model

  • Concluding Discussion

  • Acknowledgment and Questions?


  • Background trust and reputation in mas
    Background in Multi-agent SystemsTrust and Reputation in MAS

    • Trust: [Gambetta 1988]

    • Subjective probability expects performs a given action  welfare depends on.

    • Reputation:Information about an agent’s behavioural history.

    • [Ismail et. al. 2007]

    • Challenging

    • Confusing

    • Inconsistent

      Typology 

    A

    B

    A

    Survey in Ch2


    Background cont typology
    Background cont. in Multi-agent SystemsTypology

    • Suitable

    • mechanisms

    1) Variety of sources of information

    2) Individuals/distributed evaluation

    3) Robust against possible lying/fraud


    Enhanced social learning via trust and reputation mechanisms in multi agent systems

    Background cont. in Multi-agent Systems Evolutionary Games

    • Game Theory (GT)?

    • Evolutionary GT?

    • Social Dilemmas?“Cooperation”  “Tragedy of the commons”

      • Autonomous individuals

      • Theory  individuals behave selfishly

      • Nature  cooperation exists

    • Abstract framework  many real-life scenarios

    • Simple games + rich dynamics 

    • Appropriate mathematical tools 

    • Study complex Strategic interactive scenarios

    [Hardin 1968]

    • Biology, Economics, Sociology (IEEE Trans, Statistical Physics, Nature, CEC, GECCO …)

    • Distributed systems (P2P) (DAI)

    • Crucial for performance of MAS

    act cooperatively  contribute to the social welfare

    Still an open ended question!

    • (AAMAS)

    behave selfishly (not investing anything )  enjoy the free benefits shared among all the members (free-riding)

    • Mechanisms?


    Enhanced social learning via trust and reputation mechanisms in multi agent systems

    Background cont. in Multi-agent Systems

    Prisoners’ Dilemma

    • Why?

      • The most difficult settings for cooperation

      • Robust and fundamental method of modelling

      • Simplicity of statement and design MAS

    • (2-PD)

      • 2 players / agents

      • 2 choices (C or D)

      • Payoff joint actions

      • Actual values  order

      • Order change  game change

    • (D,D)  Nash Equilibrium

    i) T > R > P > S

    ii) 2R >= (T + S)


    Trust and reputation in evolutionary games
    Trust and Reputation in in Multi-agent SystemsEvolutionary Games

    • 5 Fundamental mechanisms  Evolution of “Cooperation”

      • Kin selection vs. Group selection

      • Direct Reciprocity

      • -Iterated encounters

      • -Return of altruistic act / punishment

      • -“You scratch my back, I’ll scratch yours!”

      • Indirect Reciprocity

      • -Unlikely repeated interactions

      • -Return from third parties

      • -Image/Reputation score

    • -“You scratch his back, I'll scratch yours!”

      • Network Reciprocity

      • -Social / spatial constraints  Non-uniform / Local neighbourhood interactions

      • -Clustering effect (community structure)  Enhances cooperation

    [Nowak 2006]

    Compare  Trust & Reputation


    Background cont basics of the networks
    Background cont. in Multi-agent SystemsBasics of the Networks

    • Network graph, G(N, E),

    • N finite set of nodes (vertices)

    • E finite set of edges (links)

    • G represented by N×N adjacency matrix

      • aij = 1 there is an edge between node i and j

      • aij = 0 otherwise

    A graph with 8 vertices and 10 edges

    Network of computers


    Background cont topological properties
    Background cont. in Multi-agent SystemsTopological properties

    • Degree, ki, of a node

    • Path length, L average separation between any two nodes

    • Clustering coefficient, Ci , of a node

    • probability that two nearest neighbours of a node are also nearest neighbours of each other.


    Background cont types of networks
    Background cont. in Multi-agent SystemsTypes of Networks

    ?

    • Random  uniform probability p

    • Mathematical objects  Comparison only (not good for real social network)

    • Regular 

    • Not good for real networks

    • Small-World 

    • Regular lattice Random graph

  • One end of each link  rewired small probability p

  • Highly clustered + Short path length

    • Scale-Free

    • Grow  preferential attachment

    • Power-law degree distribution

  • Most nodes very few links, small nodes highly connected

  • The same degree

    2-D square grid (lattice)

    transition

    1-D circular

    0  p  1

    Small-world graph


    Background cont evolutionary games on graphs
    Background cont. in Multi-agent SystemsEvolutionary Games on Graphs

    • Local neighbourhood interaction

      • Population Structure  system dynamics

    • Clusters of cooperators Enhance cooperation

      • Developmental stages

      • -scaffolding interaction  different types of network topology

      • -parameters (magnitude rewards/punishments, population size, initial condition, update rules)

      • -mathematical analysis difficult  Computational simulations

    Socio-biological

    Uniform interactions

    Non-uniform interactions

    Dynamic Networks

    Non-uniform interactions

    Static Networks

    Realistic Social Net

    2-D Grids


    Outline2
    Outline in Multi-agent Systems

    • Overview

      • Motivation

      • Enhanced Social Learning

      • Research Goals / Questions / contributions / publications

    • Background

      • Trust and Reputation in Multi-agent Systems

      • Trust and Reputation in Evolutionary Game Theory

      • Evolutionary Games on Graphs

  • The Research work

    • First Model

    • Second Model

    • Third Model

  • Concluding Discussion

  • Acknowledgment and Questions?


  • First model life experiences in spatial 2 pd game
    First Model in Multi-agent SystemsLife Experiences in Spatial 2-PD Game

    • Only Decision making

    • NoPartner selection

    • Cooperative behaviour

    Enhanced Social Learning

    Trust & Reputation

    Life-experiences

    &

    Age

    Fixed Network (grid)

    ?

    Social

    Learning

    • Local neighbourhood interaction  Moore

    • Accumulates received payoffs  Fitness

    • End of each round  Imitate

    • the most successful neighbour (MSN)

    • Clusters of cooperators 

    • outweigh losses against defectors


    First model cont the challenge
    First Model cont. in Multi-agent SystemsThe challenge

    • Typically  “Universal fixed payoff matrix”

    • Hypothesis  Introducing “social diversity”

    • alters trajectory of the population

    • Adaptive rewards  (Individual agent strategies + Life-experiences)

    • Given a limited agent life span

    • MSN (Highest accumulated normalized utility + Older)

    • Role model trustworthiness!

    • Ageαi(t+1) = αi(t) + 1

    • Life-span λi randomly from a uniform distribution [min, max]

    • (αi(t) == λi  dies and replaced by a new random agent)

    • Personal version of payoff matrix  updated at each time step

    • based on experience level

    Each agent

    ?

    Contributions

    Update rule


    First model cont adaptive rewards
    First Model cont. in Multi-agent SystemsAdaptive rewards

    • Update 

    • Where is the payoff values for agent iat time t

    • is the default payoff matrix values T, R, P, S

    • is the magnitude of the rescaled values

    • is the age of agent i at time t

    • is the expected life time of agent i

    • is limiting factor and characterises the uncertainty related to

    • the environment

    1)

    2)


    First model cont scenarios
    First Model cont. in Multi-agent SystemsScenarios

    • Standard PD Universal fixed Payoffs + Age

    • Homogeneous modelUniversal fixed Payoffs+Age

    • Heterogeneous model Individual Adaptive Payoffs + Age

    • (3 versions: update 4 elements / update 1 element / update 1 element capped)

    • What is the equilibrium state?

    • Coevolution

    • Altruistic behaviour + Non-stationary dynamic rewards

    (S)

    (HOM)

    (Het 1)

    (Het 2)

    (Het 3)


    First model cont experimental setup
    First Model cont. in Multi-agent SystemsExperimental setup

    • 2-D grid (32*32)  Implemented in Netlogo 4.0 [Wilensky 2002]

    • Population initialization  (20% C – 80% D) / (50% C – 50% D)

    • Payoff (small: T=1, R=1, P=0, S=0) / (Big: T=5, R=3, P=1, S=0)

    • Life-span distributions (λi )  [0,50] / [0,100] / [50,100]

    • Environmental constraint K  [0.1 : 0.025 : 0.2]

    • Each trial  10000 iterations & All configurations  30 times

    • Statistical results are reported


    First model cont sensitivity to the base payoff values
    First Model cont. in Multi-agent SystemsSensitivity to the base payoff values

    Payoff (small: T=1, R=1, P=0, S=0) / (Big: T=5, R=3, P=1, S=0)

    Standard

    (S)

    Homogeneous

    (HOM)


    First model cont heterogeneous vs homogeneous
    First Model cont. in Multi-agent SystemsHeterogeneous vs. Homogeneous

    Payoff: (Big: T=5, R=3, P=1, S=0) / Population initialization

    (20% C – 80% D)

    (50% C – 50% D)


    First model cont snapshots
    First Model cont. in Multi-agent SystemsSnapshots

    Payoff: (Big: T=5, R=3, P=1, S=0) / Population initialization (20%C – 80% D)

    (Het 1)

    Varying size clusters of cooperators (black)

    (Het 2)

    (Het 3)

    Other extra results for different parameters K, life-span, replacement …

    (HOM)


    Outline3
    Outline in Multi-agent Systems

    • Overview

      • Motivation

      • Enhanced Social Learning

      • Research Goals / Questions / contributions / publications

    • Background

      • Trust and Reputation in Multi-agent Systems

      • Trust and Reputation in Evolutionary Game Theory

      • Evolutionary Games on Graphs

  • The Research work

    • First Model

    • Second Model

    • Third Model

  • Concluding Discussion

  • Acknowledgment and Questions?


  • Second model n pd on an evolving social network
    Second Model in Multi-agent SystemsN-PD on an Evolving Social Network

    • Decision making

    • Partner selection

    • Coevolution (Interaction network + Individuals’ strategy)

    Enhanced Social Learning

    • 2-PD  N-PD

    • Cooperative behaviour in larger groups  More difficult ! (N > 2)

    • Real-world social communities

    • Fixed underlying network  Relaxed

    • Relations evolve over time

    • Link weights  Trust & Reputation

    Trust & Reputation

    Social

    Learning

    Endogenous Evolving Social Networks


    Second model cont n player prisoners dilemma
    Second Model cont. in Multi-agent SystemsN-player Prisoners’ Dilemma

    • Natural extension of 2-PD

    • Utility 

    • [Boyd and Richerson 1988]

    • Conditions

    • defection is preferred for individuals

    • contribution to social welfare is beneficial for the group

    Conventional EG (D,D, … all D)


    Second model cont evolving relations
    Second Model cont. in Multi-agent SystemsEvolving Relations

    • Agents play cooperatively  form social links (reinforced)

    • One agent defects breaks his links with the opponents

    slow positive / fast negative


    Second model cont contribution hypothesis
    Second Model cont. in Multi-agent SystemsContribution - Hypothesis

    • Incorporating “social network” into N-player PD 

    • Network evolves by cooperative behaviour

    • Introducing “cognitive” agents 

    • Decision making based on some function of the opponents

    • Encourage high levels of cooperation

    • Persist for longer

    • Analyse the state of the underlying network


    Second model cont schematic algorithm
    Second Model cont. in Multi-agent SystemsSchematic Algorithm

    Algorithm: Social network based N-PD model

    Require:Population of agents P, iteration = imax, players N 2

    1: fori = 0 to imaxdo

    2: G = 0;

    3: while g = NextGame(P,G, N) do

    4: G = G {g}

    5: PlayGame(g)

    6: AdaptLinks(g)

    7: endwhile

    8: a,b = Random Sample(P)

    9: CompareUtilityAndSelect(a,b)

    10: end for

    Partner selection

    Decision making


    Second model cont game formation
    Second Model cont. in Multi-agent SystemsGame Formation

    Partner selection

    • First agent  Randomly from remaining population

    • Two Scenarios

    • (N-1) partners

    Randomly from remaining population

    From the first agent remaining social contacts probabilistically


    Second model cont game execution
    Second Model cont. in Multi-agent SystemsGame Execution

    Decision making

    • Two scenarios (cognitive abilities)

      • Pure strategy (always cooperate/defect)

      • Mixed strategy (play probabilistically)

      • Discriminators function of

      • Agents receive corresponding

      • payoff based on outcomes

      • (Boyd and Richerson function)

    gradient

    generosity

    • Average links weight


    Second model cont snapshots
    Second Model cont. in Multi-agent SystemsSnapshots

    |P| = 25, N = 3, Defector, Cooperator, Discriminator

    • Self-organize social ties based on their self-interest

    • Strategy update cultural evolution


    Second model cont scenarios
    Second Model cont. in Multi-agent SystemsScenarios

    •  Partner selection + Decision making

      • (Random matching) (Pure strategy)

  •  Partner selection + Decision making

    • (Social Network game formation) (Pure strategy)

  •  Partner selection + Decision making

    • (Random matching) (Pure strategy + Discriminators)

  •  Partner selection + Decision making

    • (Social Network game formation) (Pure strategy + Discriminators)

  • Step 1

    Step 2

    Step 3

    Step 4


    Second model cont experimental setup
    Second Model cont. in Multi-agent SystemsExperimental Setup

    • Population size = 1000

    • Group sizes = (2, 4, 5, 10, 15, 20)

    • ε = 0.9 Game formation probability

    • b = 5 and c = 3 (payoff values benefit & cost)

    • Pure strategy scenario (50% pure C – 50% pure D)

    • Mixed strategy scenario (33.3% each)

    • α = 1.5 and β = 0.1 (decision function)

    • average 20 independent trials up to 40000 iterations

    What is the equilibrium state and network topology?


    Second model cont group size vs strategy
    Second Model cont. in Multi-agent SystemsGroup size vs. Strategy

    Step 1

    Step 2

    Step 3

    Step 4


    Second model cont emergent social networks
    Second Model cont. in Multi-agent SystemsEmergent Social Networks

    Clustering

    Coefficient

    Step 2

    Step 3

    Step 4


    Second model cont final degree distribution
    Second Model cont. in Multi-agent SystemsFinal Degree Distribution

    Step 4

    N=2

    Step 4

    N=5

    • Cooperation higher  degree distribution higher

    • Size & shape  depend on N


    Outline4
    Outline in Multi-agent Systems

    • Overview

      • Motivation

      • Enhanced Social Learning

      • Research Goals / Questions / contributions / publications

    • Background

      • Trust and Reputation in Multi-agent Systems

      • Trust and Reputation in Evolutionary Game Theory

      • Evolutionary Games on Graphs

  • The Research work

    • First Model

    • Second Model

    • Third Model

  • Concluding Discussion

  • Acknowledgment and Questions?


  • Third model distributed advice seeking on an evolving social network
    Third Model in Multi-agent SystemsDistributed Advice-Seeking on an Evolving Social Network

    • Decision making

    • Partner selection

    • Coevolution (Interaction network + System’s behaviour)

    Enhanced Social Learning

    Trust & Reputation

    • Games  Advice-Seeking in Distributed Service Provision

    • Relations evolve over time (Link weights  Trust & Reputation)

    Life-experiences

    Social

    Learning

    ?

    Endogenous Evolving Social Networks


    Third model cont distributed infrastructure technology
    Third Model cont. in Multi-agent SystemsDistributed Infrastructure Technology

    • Characteristics

      • Unknown large environment

      • Varieties of selection options

      • Users are heterogeneous

      • Exact characteristics not available

      • until accessed, if it is made explicit at all

      • Ex./ Specialized protein search engines, Netflix

    • Approaches

      • Individual try & error

      • Central registration directory (Brokers, Web Service [Facciorusso et. al. 2003])

      • Advice seeking Direct exchange of “selection advice” beneficial!

      • ex./ Learning [Nunes and Oliveira 2003 ], Distributed Recommender Systems

    Question?


    Third model cont advice seeking
    Third Model cont. in Multi-agent SystemsAdvice-Seeking

    • Question:

      • Heterogeneousindividual requirements Whom?

    • Challenge:Identify other suitable users difficult!

    - Large number of them

    - Preferences not publicly available

    - Not in a position to make their own preferences explicit

    Social Networks!

    • Social contacts serve as valuable resources

    • Manage improve long term payoff gains


    Third model cont abstract framework
    Third Model cont. in Multi-agent SystemsAbstract Framework

    • Agent-based simulation (resources + agents)

    • Repeatedly

    • Subjective Utility

    • Goal = Maximize long term utility, limited selections

    • Challenge = Identify appropriate resources

    • Evolving Social Network

    • - Connect with similar minded  Autonomously

    • based on local information only

      - Receive advice  improve resource selection

      - Learn their own subjective utility  advice accuracy

      decide retain / drop the contact

    • - Form new connections Seek referrals

    Match?


    Third model cont what we study
    Third Model cont. in Multi-agent SystemsWhat we study?

    • This capability

    • Connection network Advice exchange

    • Agents’ interactions Social relationships

    • The evolving social network Utility gain

    Affect the match?

    How co-evolve?

    Change?

    Improve?


    Third model cont schematic algorithm
    Third Model cont. in Multi-agent Systems Schematic Algorithm

    Algorithm: Evolving Social Network Advice seeking

    Require:Population of agents , set of resources , rounds , evolutionary rate , maximum out degree , recommendation threshold t, default edge weight

    1: Weighted Graph = InitializeGraph ( , , )

    2: for r = 1 to do

    3: foreach a∈ in random order do

    4:

    5:if Random() > then

    6: AccessResource(a, )

    7: else

    8: Query (a, , , t)

    9: end if

    10: if Random() < then

    11: AdaptLinks(a, , RANDOM() < , )

    12: end if

    1-Initialization

    2-Exploitation/Exploration

    4-Assessment *

    3-Advice selection

    5-Network Adaptation *


    Third model cont
    Third Model cont. in Multi-agent Systems

    1-Initialization

    • Heterogeneous pool of resources

    • n-dimensional binary feature vector frinitialized randomly

    • Heterogeneous agent population

    • n-dimensional binary preference vector painitialized randomly

    • Initialize Graph( , , )

    • 2 scenarios:

      • random agents no structural restriction

      • social agents outgoing edges, default weight ( = 0.5)


    Third model cont1
    Third Model cont. in Multi-agent Systems

    2-Exploitation/Exploration

    • Selection based on personal knowledge / Query others!

    • Probabilistic Quality of the agent’s acquired knowledge

      • Exploit Access the largest utility resource it knows so far

      • Explore Seek advice (resource, utility)

      • Random agents other random agents

        Social agentsoutgoing edges, social contacts


    Third model cont2
    Third Model cont. in Multi-agent Systems

    3-Advice selection

    • A suggestion probabilistically

      • Advisor Link’s weight

      • One of his suggestions Reported utility

  • Subjective utility of accessed resource

    • Similarity between pa & fr

    • Normalized Hamming distance mapped to [-1,1]

    • Positive values better than average random selection

    • Negative values random selectionwould have done better


  • Third model cont3
    Third Model cont. in Multi-agent Systems

    4-Assessment *

    • Social agents learn from their interactions adjust the weight of links

    • Following a particular suggestion

      - Positive | ua (r) – urep (r)| < thrdis

      - Negative

    • Adjust the link weight with multiple advisors

      - the link weight

      - w(a,b) < thrtolerance remove the edge, free slot!


    Third model cont4
    Third Model cont. in Multi-agent Systems

    5-Network Adaptation *

    • Social agents

      • opportunity to change their links probabilistically!

      • Link to a random agent with default weight

      • Ask for referrals Trust propagation

        [Massa and Avesani 2007, Vidal 2005]


    Third model cont snapshots
    Third Model cont. in Multi-agent SystemsSnapshots

    • Steps 4 & 5 eventually make link with similar preferences

    • Similar-minded community spot beneficial resources faster


    Third model cont experimental setup
    Third Model cont. in Multi-agent SystemsExperimental Setup

    • Monte-Carlo simulations, various parameter settings

    • Scenarios (Social agents only and Random agents only)

    • Population sizes (small = 100, large = 300 agents)

    • Environmental complexity  |R| = (1000, 5000, 10000, 50000)

    • Heterogeneity  |pa| & |fr| = (2, 3, 4, and 5)

    • First 1000 iterations  Average over 30 independent trials

    • (Note! exhaustive exploration will find eventually)


    Third model cont basic model behaviour
    Third Model cont. in Multi-agent SystemsBasic Model behaviour

    • Social agents gain higher utilities?

    • (|A| = 100, |pa| & |fr| = 3, |R| = 5000)


    Third model cont environmental complexity
    Third Model cont. in Multi-agent SystemsEnvironmental Complexity

    • Efficiency of social and random scenarios

      Facing more complex environments?

      |A| = 100 |pa| & |fr| = 3

      |R| = (1000,5000,10000,50000)


    Third model cont analysis the underlying network
    Third Model cont. in Multi-agent SystemsAnalysis the underlying Network

    |A| = (100,300) / |R| = 5000/ |pa| & |fr| = (2, 3,4, 5)  Modularity Score

    Small population

    Large population


    Outline5
    Outline in Multi-agent Systems

    • Overview

      • Motivation

      • Enhanced Social Learning

      • Research Goals / Questions / contributions / publications

    • Background

      • Trust and Reputation in Multi-agent Systems

      • Trust and Reputation in Evolutionary Game Theory

      • Evolutionary Games on Graphs

  • The Research work

    • First Model

    • Second Model

    • Third Model

  • Concluding Discussion

  • Acknowledgment and Questions?


  • Summary thesis contributions
    Summary in Multi-agent SystemsThesis contributions

    Efficacy of Enhanced Social learning approaches 

    Agents interactions Individuals’ and System’s long term (utility) performance

    Life-experiences + Endogenous Evolving Social Networks  Trust and Reputation ESL

    First Model (2-PD on Fix Grid Structure): Adaptive rewards  Life-experiences / Age

    Innovative notion of role model trustworthiness / Heterogeneous social diversity  Cooperation

    Second Model(N-PD on an Evolving Social Network):

    Endogenous network formation  Partner selection + Decision making  (Cooperation)

    Emergent Social Networks  High average clustering + Broad-Scale heterogeneity

    Third Model(Distributed Advice-Seeking for Resource Discovery):

    Life-experiences + Endogenous network formation  Similar minded (appropriate role models)

    Strongly connected communities with similar preferences  Higher utility


    Limitations
    Limitations in Multi-agent Systems

    Generality of Adaptive rewards on Fixed interaction networks

    2-PD on simple Grid Other classes of games (Hawk-Dove / Stag-Hunt / …)

    Age attribute Heterogeneity Other concepts?  How encourage Cooperation?

    Simple Grid  Other fixed topologies? Effect of different neighbourhood structures

    Generality of Adaptive rewards on Evolving Social Networks

    Dynamic Payoffs  N-PD framework  Not satisfying! (limited parameter settings)

    Extensive analysis  Determine why it was not helpful / If it is helpful at all / How?

    (Ex./ Bigger ranges of life-span / different time scales for update rules + evolution interaction network)

    Realistic approaches for Advice-Seeking framework

    Generic model  Inspired by several distributed service provision systems

    Synthetic date  Set up specific, controlled platform  Represent semi-realistic

    MAS  Evaluate performance of the ESL  Not solution for particular application!

    Exploit such techniques  real technological systems  real data sets  real users

    preference profiles  binary preferences  Not realistic!

    Dynamic Environment  Dynamic relations / Users / Preferences / Resources?


    Future work
    Future work in Multi-agent Systems

    N-PD  fixed group sizes + similar for all agents

    Dynamic group formation + heterogeneous sizes  different communities in real-world

    Advice-Seeking model  similarities with Recommender Systems

    Different purpose here  BUT!

    Interesting to Modify and apply in such context  Comparison with other models

    Enhanced Social Learning  Imitation (basic cultural learning)

    Extend to other methods of MAS learning  ex./ Reinforcement Learning

    Evolutionary Game Theory + Advice-Seeking  Investigation domains

    Potential domains (MAS)  P2P / Mobile Ad-hoc Networks / Grid Computing

    Robustness of the proposed mechanisms

    Different scales of dynamicity in real-world environment


    Acknowledgment
    Acknowledgment in Multi-agent Systems

    Michael, Shanika, Adrian

    Jens

    Les, Ed, Leon, Liz, …

    Agent lab members, Rebecca, …

    Dept. Computer Sci / Uni Melb

    Rahil, Leila, Parvin, Toktam, …

    Lab colleagues (Saeed/Raymond/…)


    Questions
    Questions? in Multi-agent Systems

    Thank you


    References
    References in Multi-agent Systems

    • D. Gambetta. Can We Trust Trust? In D. Gambetta, editor, Trust: Making and Breaking Cooperative Relations, pages 213--237. Basil Blackwell, 1988.

    • R. Ismail, A. Jøsang, and C. Boyd. A survey of trust and reputation systems for online service provision. Decision Support Systems, 43:618644, 2007.

    • M. A. Nowak. Five rules for the evolution of cooperation. Science, 314:1560-1563, 2006.

    • R. Boyd and P. Richerson. The evolution of reciprocity in sizeable groups. Journal of Theoretical Biology, 132:337--356, 1988.

    • C. Facciorusso, S. Field, R. Hauser, Y. Hoffner, R. Humbel, R. Pawlitzek, W. Rjaibi, and C. Siminitz. A Web Services Matchmaking Engine for Web Services. In E-Commerce and Web Technologies, Lecture Notes in Computer Science, pages 37--49, 2003.

    • L. Nunes and E. Oliveira. Advice-exchange in heterogeneous groups of learning agents. In Proceedings of the second international joint conference on Autonomous agents and multiagent systems, pages 1084--1085, 2003.

    • P. Massa and P. Avesani. Trust-aware recommender systems. In Proceedings of the 2007 ACM conference on Recommender systems, pages 17--24, 2007.

    • J. M. Vidal. A Protocol for a Distributed Recommender System. In J. Sabater R. Falcone, S. Barber and M. Singh, editors, Trusting Agents for Trusting Electronic Societies. Springer, 2005.

    • G. Hardin. The Tragedy of the Commons. Science, 162:1243{1248, 1968.

    • U. Wilensky. Modelling Nature's Emergent Patterns with Multi-agent Languages. In Proceedings of EuroLogo, 2002. NetLogo is a cross-platform multi-agent programmable modelling environment. See http://ccl.northwestern.edu/netlogo/.


    Enhanced social learning via trust and reputation mechanisms in multi agent systems

    Backup Slides in Multi-agent Systems


    First model cont sensitivity to the magnitude of k
    First Model cont. in Multi-agent SystemsSensitivity to the magnitude of K

    Payoff: (Big: T=5, R=3, P=1, S=0) / Population initialization (20%C – 80% D)

    1)

    2)

    (Het 1)


    First model cont sensitivity to the life span i
    First Model cont. in Multi-agent SystemsSensitivity to the Life-span (λi)

    Payoff: (Big: T=5, R=3, P=1, S=0) / Population initialization (20%C – 80% D)

    (Het 1)


    First model cont sensitivity to the replacement strategy
    First Model cont. in Multi-agent SystemsSensitivity to the replacement strategy

    Payoff: (Big: T=5, R=3, P=1, S=0) / Population initialization (20%C – 80% D)

    (Het 1)


    Third model cont metrics
    Third Model cont. in Multi-agent Systems Metrics

    • Average utility

    • Average error rate

    • Efficiency


    Third model cont the influence of heterogeneity
    Third Model cont. in Multi-agent Systems The influence of Heterogeneity

    • Finding similar-minded agents important role

    • How heterogeneity

      in |pa| & |fr| affect the

      performance of

      social agents?

      |A| = (100 , 300)

      |R| = 5000

      |pa| & |fr| =

      (2, 3, 4, 5)

      T = 1000

      Averaged accumulated utility