Swarm Collaborative Intelligence: From Networked Control to Trust in MANET

1. Swarm Collaborative Intelligence: From Networked Control to Trust in MANET John S. Baras Institute for Systems Research Department of Electrical and Computer Engineering And Department of Computer Science University of Maryland, College Park, MD 20742 Workshop on Swarming in Natural and Engineered Systems Napa Valley, California August 3-4, 2005

2. Thanks to • Collaborators: Tao Jiang, George Theodorakopoulos, Xiaobo Tan, Wei Xi, Pedram Hovareshti • Funding sources: ARL (CTA on C&N), ARO, ARO CIP URI (Wireless Network Security), ARO MURI (Networked Control Systems), DARPA (Dynamic Coalitions)

3. Outline • Autonomous collaborating vehicles • A stochastic approach based on MRF • Analysis • Convergence study of a simple case • Convergence speed analysis • Simulation results • A hybrid scheme to improve the performance • Distributed trust in MANET • Trust (and Mistrust) spreading and dynamics • Effects of topology on convergence • Spin glasses and cooperative games • Collaboration via trust schemes • Conclusions and future work

4. A Battlefield Scenario • Mission • Autonomous, distributed maneuvering of a vehicle group to reach and cover a target area • Constraints • Desired inter-vehicle distance • Obstacles avoidance • Threats (stationary or moving) avoidance • Requirement • Using only local or static information

5. Review of Deterministic Gradient-Flow Approach • Dilemma of the Deterministic gradient-flow approach • Potentials-based approach can accommodate multiple objectives and constraints in a distributed and computationally effective way • The system dynamics could be trapped by the local minima • Weighted sum of potential functions: • Target (attraction) potential Jg • Neighbor (avoidance) potential Jn • Obstacle potential Jo • Potential Js due to stationary threats • Potential Jm due to moving threats Gradient flow:

6. Being Trapped by Local Minima Different initial conditions may cause vehicles to be trapped by local minimum

7. Markov Random Fields • Markov Random Fields (MRF) • A collection of random variables X={Xs},s∈ S with discrete values in phase spaces • A neighborhood system on S is a family N = {Ns}, s∈ S, where Ns ⊂ S, and r ∈ Ns ↔ s ∈ Nr • The marginal probability depends only on neighbor’s phase value • Gibbs Field (GF) • A clique is a subset c ⊂ S, such at for alls,r ∈ c, r ∈ Ns • The potential energy of a configuration x={xs} is defined as a sum of all clique potentials

8. Gibbs Sampler and Gibbs Distribution • Gibbs distribution (global description) • The marginal probability is function of local potentials • Gibbs distribution is function of global potentials • Hammersley-Clifford theorem: A MRF on a graph is equivalent to a GF • Gibbs sampler • Gibbs sampler (MCMC method)defines a Markov chain on a Gibbs Field • The stationary distribution of the MC is the Gibbs distribution • Using simulated annealing algorithm, final configuration converges to global minimum with probability 1

9. target Obstacle Agents Modeling a Swarm as a GF • Agent s can communicate with neighboring agents in Ns which stay within the interaction range Rs • An agent can go at most Rmwithin one move, which defines the phase space s • Gibbs potential is designed to reflect global objective • 2D mission space on discrete lattice cells • Difficulties in applying classical results • Non-stationary neighborhood system • Time-varying and state-dependent phase space

10. Gibbs Sampler Based Algorithm • Algorithm for single vehicle • Step1. Pick a cooling schedule T(n) and the total number N of annealing steps • Step2. At each annealing step n, conduct a location update for the vehicle by performing the following: • Determine the set L of candidate locations for the next move • For each l ∈ L, evaluate • Update locationby sampling above distribution • Step 3. Let n = n+1. If n = N, stop; otherwise go to step 2

11. Convergence Study • Single vehicle with limited sensing and moving range • Fixed temperature • Assume accessible area is connected • Unique stationary distribution where • From any distribution v, • Simulated annealing • Cooling schedule • Let Qn = (PT(n))τ where

12. Convergence Rate Study • Convergence rate of a single vehicle case • For the single-vehicle case, the convergence rate is characterized by where • Using convergence rate bound as a design indicator • Design λg*to maximize the convergence rate • Potential function • Empirical distributiondistance

13. Parallel Sampling • Problems with sequential sampling • Global indexing is difficult in practice • Long time for one sweep • Parallel sampling • Agents update locations in parallel by sampling local characteristics • Conflicts could be solved by coin-toss. • Simulation showed the MAS achieve global objective with only local strategies.

14. Stochastic Path Planning Simulation Parallel stochastic path exploration based on MRF can get around the local minima • Potential function • Target (attraction) potential Jg • Neighbor (avoidance) potential Jn • Obstacle potential Jo

15. Simulation : Gathering • Potential function • The first term attracts nodes close to z0 • The second term tends to cluster nodes

16. Simulation: Gathering • 200 nodes on 50 by 50 grid；1= 0.05 , 2 =1,  =103 • Rm=22, Rs=62； T(n)=1/(4log(400+n)) specified center Z0=(25,25) unspecified center

17. Simulation: Line Formation • Potential function •  is scaling factor •  is a penalization for node with no neighbor • mkis the number of neighboring nodes of node k • k,k’is the desired angle of the line segment • dk,k’ /Rs puts more weight on farther neighbors, which encourages the formation of long lines

18. 200 nodes on 50 by 50 grid =10 , =5 Rm=2 2 Rs=102, 62, 42 T(n)=1/(4log(400+n)) Simulation: Line Formation One line Three lines Two lines

19. A Hybrid Control Scheme • Deterministic potential approach • Pro: Save traveling time • Con: May be get trapped by some obstacles • Stochastic approach based on MRF • Pro: Trouble free. Converge to global minimum for sure. • Con: Waste time for path exploration • Hybrid control scheme combines both advantages and may strike the right balance

20. Hybrid Scheme Algorithm • Step 1. Each vehicle (node) starts with the deterministic gradient-flow method and goes to Step2 • Step 2. If a vehicle stops moving for dconsecutive time instants and its location is not within the target area, then it switches to the simulated annealing method with a predetermined cooling schedule • Step 3. After performing simulated annealing for Ntime instants, the vehicle switches to the gradient method and goes to Step 2

21. Impact of Memory • Hybrid scheme with memory • Experience can help vehicle to learn the complex environments better and thus change its behavior to achieve better performance. • Implementation: when a vehicle determines it is trapped , it increases the risk level R of that spot, and does local sampling as follows

22. Impact of Memory (cont.) • Hybrid scheme with memory

23. Autonomic Wireless Networks • Wireless networks, such as mobile ad hoc networks (MANET) and sensor networks: • No trusted centralized authority • Resource (power, bandwidth, computation etc.) constraints • Rapidly and dynamically changing topology and connectivity • Uncertainty & incompleteness of trust evidence: trust values in [-1, 1] • Distributed trust computation and locality of trust information exchanges • Unique properties • Each node is its own authority and it is selfish • Networking functions (route discovery, packet forwarding and etc. ) rely on cooperation between nodes • Cooperation utilizes local information and local interactions (between neighbors)

24. Cooperation and Games • In distributed wireless networks • Cooperation is restricted to only local interactions • Decision is made by each node individually • Nodes are self-interested • Explain and analyze emergent properties • Game theoretic methods • Provide a framework for modeling individual interactions • Understand complex global structures and dynamics of a system composed of a large number of agents with simple local interactions • Guide for analytical approach • Examples: Ising model, prisoner’s dilemma • Goal: how to encourage nodes to collaborate in games? • Incentive: trust systems to promote cooperation and circumvent misbehaving nodes.

25. A Simple Distributed Trust Computation Policy • Based on simple voting methods • Voters: • Nodes that qualified as legitimate voters by certificates signed by offline servers (have trust evidence about node i) • Assume uniformly distributed in the network • Policy: decision based on threshold • is the total number of votes node i received (signed sum) • is the decision threshold • is the number of i’s neighbors

26. Trusted nodes Neutral nodes Positive votes Negative votes Simple Voting Scheme • Number of positive votes on node i: Vp,i = 3 • Number of negative votes on node i: Vn,i = 1 • Effective votes: Vi = Vp,i - Vn,i= 2 • Given η = 0.3, Vi > η|Ni| = 1.8, node i is designated “trusted”

27. Trust revocation: Changes in topology, membership, secure paths Referees of a node may change, trust evidence for a node may change Votes timeout or negative votes Trust spreads Trust-connected network Trust Dynamics • Trust spreading Initial “islands” of trusts

28. Trust Graph • Trust graph: GT(VT, ET) • Induced subgraph of G(V, E) by VT • VTis the set of nodes which are designated “trusted” by the trust computation algorithm • ET= {e | e in E and both ends of e are in VT} • Trust metric Psp: percentage of trusted pairs that are connected by one or more secure paths, which are composed of trusted nodes • NPsecure is the number of trusted pairs that are connected by one or more secure paths. • It is dependent of the cluster size and connectivity of GT

29. Random Graph Model Erdos and Renyi random graphs (ER model) • Whenηis small • Most of nodes are considered to be trusted • Psp is dominated by the edge present probability p in ER random graphs • Zero-one law in random graph theory is present • Increasing the threshold ηresults in • Reducing the number of trusted nodes • Increasing critical values • Smaller Psp Simulation results of Psp as function of decision threshold η

30. Small-world Networks Psp vs. ηafter one iteration Psp vs. ηin steady states • Number of trusted paths increases as trust spreads with each iteration • Different curves are with different rewiring probability Prw • Prw= 0 represents a regular lattice • Prw = 1 converges to a random graph • Observe the transition from lattices to random graphs • With a relative small portion of shortcuts, small-world networks facilitate the formation of secure paths • The effects of topology are obvious, so any distributed trust computation model should take into account the topology properties

31. A Trust Revoke Trust Revocation • The trust revocation process is initiated: • when topology, membership or secure paths change • when referees or trust evidence for a node changes • when positive votes are timeout or new negative votes are received • Decision policy of the revocation process • Revocation on a specific node, say B, usually starts from few nodes that have negative observations on B; • A node A accepts the revocation on B, if it finds that more than a threshold fraction Φ of its neighbors revoke node B; • Question: can a revocation be accepted by a large fraction of nodes in the network?

32. Phase Transition of Revocation • Revocation is launched from a randomly chosen node in an Erdös-Rényi random graph with average degree set as the Y-axis. • Global cascade: area that lie inside of the contour represents the percentage of nodes, which accept the revocation, is greater than the value corresponding to the contour (level surfaces of histogram) • Phase transitions happen suddenly: the steep of the contours is very sharp, which represents phase transitions

33. Previous Work • Decentralized path-inference protocols • Combination of trust along and across paths (Beth,1994) • Probability of finding a trust path from source to target (Maurer, 1996) • Local interaction • EigenTrust (Kamvar, 2003) • PeerTrust (Xiong, 2004) • Bayesian methods (Buchegger, 2003) • Our work is similar with EigenTrust and PeerTrust, which provided promising results. • However, results of EigenTrust and PeerTrust are all based on simulations. • We analyze our local interaction rule using graph theory. • We also provide a theoretical justification for network management that facilitates trust propagation.

34. Voting Scheme • Voting rule: • is the trust value of node i • is the voting value of node j about node i • Local voting rule • Function f should satisfy the following properties: • The range of f is [-1,1]. • Votes from neighbors with higher trust value are more credible, so they should carry larger weights. • Policy: threshold rule for trustworthiness of the target agent where is the threshold, which is a constant

35. Simple Voting Rule • We use the weighted average as the voting rule, where weights are trust values of voters • is the degree of node i • n represents discrete time • Assume is a constant, i.e. it doesn’t change with time, which is true when considering the steady state • The voting rule can be written in system equationwhere D = diag[d1 ,d2 ,…, dN], T is a vector representing trust values of all nodes and V is the matrix for votes

36. Convergence of Simple Voting Rule • Voting without uncertainty • For each pair (i, j) , if i and j are neighbors, then vij = 1. • V = A, where A is the adjacency matrix of graph G, and D-1A is a stochastic matrix with the largest eigenvalue being 1. • Let be the right eigenvector of D-1A corresponding to eigenvalue 1. then • If , all nodes are trusted, and none is trusted otherwise. The initial trust values are very crucial. • Voting with uncertainty • vij≤ 1, D-1A is a semi-stochastic matrix. • We proved , so T0. Trust cannotbe established at all!!!

37. Voting with Headers • We have shown that using the simple voting scheme, trust can only be established under certain strict conditions: • All votes value are 1 and the initial configuration must satisfy • A single vote with value less than 1 will result in failure of trust establishment. • We introduce the notion of headers • Headers are pre-trusted agents and only vote for nodes that they fully trust. • If node i is trusted with bi headers, it will get bi more votes with value 1. Let B = diag[b1 , b2 ,…, bN ]. • The system equation changes to

38. Convergence of Voting with Headers • Voting without uncertainty • V = A, and define . The system equation changes to • If there is at least one node i such that bi > 0, (D+B)-1A goes to 0. Therefore T(n)  1 and all nodes are trusted. • Voting with uncertainty • Using the same technique as above, let . We are able to find the condition such that • If we let , then all nodes are trusted. • Theorem: Given the threshold is η, the number of headers for each node must satisfy • This theorem proves, as well as provides, a network design method to establish a fully trusted network by introducing headers

39. Spreading Speed and Topology • The time for updating rule to reach the steady state, i.e., how fast the trust values converge. • Perron-Frobenius Theorem in algebraic graph theory: For a stochastic matrix A • isthe largest eigenvalue of A, which is 1 and isthe second largest eigenvalue of A. • The convergence rate of An is of order • Normalized adjacency matrices are stochastic matrices, therefore those with smaller converge faster. • What kind of networks or which network topology has smaller second largest eigenvalue

40. Spreading Speed and Topology (cont’) • We consider the small-world model proposed by Watts and Strogatz in 1998 • High clustering coefficient and small average graphical distance between any pair. • We use Φ-model, which is modeled by adding small number of new edges into a regular lattice. • Adding just 1% more edges, spreading finishes in 10 times less rounds.

41. Ising and Spin Glass Models • Statistical Physics models for magnetization • Orientation of each particle’s spin depends on its neighbors • Ising Model: behavior of simple magnets • Spin Glass Model: complex materials • Math interpretation: • s = {s1, s2,…, sn} is a configuration of n particle spins, where sj = 1 or -1 , spin j is up or down • Hamiltonian, or Energy for configuration s • Ising Model: Jij = J for all i, j • Spin Glass Model: Jij depend on i,j and can be random processes

42. Ising/SG Models and Games • Ising and Spin Glass models can be interpreted as dynamic (repeated) games: each particle selects its own spin to maximize its own payoff • Ising model (Jij = J) : align its spin with the majority of neighbors spin • High T, conservative agents, not willing to change, small payoffs • Low T, aggressive agents, larger payoffs • Collection of local decisions reduces the total energy of the interacting particles • Statistical Mechanics primary object of interest • Recent excitement: computation of ground state, partition functionZ, NP - complete, Replica Method • Application to: turbocodes, image restoration, neural networks, learning, associative memory, SAT, knapsack, SA, number parttioning, graph partitioning, CDMA, MIMO, … • Inspires an approach where trust is used as an incentive for cooperation • si represents whether node icooperates or not with neighbors • Jij can be interpreted as the worth of player j to player i • Cooperate or not based on benefit from cooperation and trust values of neighbors

43. Spin Glass Cooperative Game • Spin Glass model as a cooperative game (spin glass game) • In , the weights wijfrustrate the system • Both positive and negative local feedback (e.g. wij{-1, 1}) • Interaction topology (i.e. the matrix J = [Jij] ) moderates effects pos. and neg. fback • S N = {1, 2, …, N} is a coalition, in which all nodes cooperate • v(S) : value of characteristic function of the game , v: 2NR; maximum payoff S can get without cooperation from other nodes N /S. • Model can be used to find what form or policy for Jij can induce all (or most) nodes to cooperate: maximize the coalition Γ=(N, v) 2 6 J21 J12 3 Subset S={1,2,3,4} v(S)=J12+J21+J14+J41+J43+J34 -J36 -J15 1 5 J34 J14 J41 4 J43

44. Cooperative Games and Dynamic Coalitions • Have a number of players, some can be coalitions themselves • How do they negotiate an “acceptable” DC security policies set? • What are the properties of the final result: “the negotiated policy set”? • Is there an efficient scheme that gets us there? • Cooperative games allow us to set up different types of games between the players, examine different concepts of solutions and values • Can prove mathematically properties of the solution and value: e.g. minimizes maximum dissatisfaction, is anonymous, is stable • Can get iterative methods to get to solution (negotiation schema), can use all kinds of constraints, invariance to aV + b scaling (preferences) • Working on extensions to partial information, learning, robustness to uncertainties

45. Spin Glass Cooperative Game Properties • Spin Glass gameis a convex and superadditive game iff (net pos. effects) • Shapley value : in the core • Not well understood in the regime of both negative and positive net effects • Effects of interaction matrix structure (sparsity, neighborhood structure, range of interactions, strength of interactions) not well understood; Topology effects in network analog • Oriented Spin Glass Game Γ(N,v) where v now depends on both an interaction matrixJ and a preference vectorL ; a pair of char. fcns • Replica method can be used to analyze various problems under various models and constraints on J and L

46. Cooperative Games with Negotiation • ConsiderΓ= (N, v), N as before but with • Γ= (N, v) convex, superadditive, if • Theorem : Γ= (N, v) has a nonempty core. The payoff allocation to node i , is in the core. Compute as follows • This payoff allocation indicates a way to encourage cooperation • Players with positive gain can negotiate with their neighbors by sacrificing certain gain (offering their partial gain ijxji)

47. Trust as Mechanism to Induce Collaboration • Trust is an incentive for collaboration • Nodes who refrain from cooperation get lower trust values • They will be eventually penalized because other nodes tend to only cooperate with highly trusted ones. • Assume, for node i, that the loss for not cooperating with node j is a nondecreasing function of xji as f (xji), and the new characteristic function is • Theorem : if , the core is nonempty and is a feasible payoff allocation in the core. • By introducing a trust mechanism, all nodes are induced to collaborate without any negotiation

48. Dynamics of Cooperation • System model • The network is modeled as a discrete-time system Two linked dynamics • Trust propagation • Game evolution j all neighbors of i vij trust value node i votes for node j

49. Game Evolution • Strategy of node i: • γij= 1(= 0) represents that i cooperates (does not cooperate) with its neighbor j • Payoff for node i when interacting with j: xij = Jij γij γji • xij > 0 (< 0) positive link (negative link) • Node selfishness cooperate with neighbors on positive links • Strategy updates: node i chooses γij= 1 only if all of the following are satisfied: • Neighbor j has not been revoked • Neighbor j is cooperative • xij > 0, or the cumulative payoff of i is less than the case when it unconditionally conducts γij= 1. • Trust propagation: • The threshold is chosen to ensure global revocation propagation • Reestablishing periodτ : once a node is revoked, in order to reestablish trust the revocation has to be nullified for τ consecutive time steps

50. Results of Game Evolution • Theorem: , there exists τ0, such that for a reestablishing period τ > τ0 • The iterated game converges to Nash equilibrium; • In the Nash equilibrium, all nodes cooperate with all their neighbors. • Comparison of games with (without) trust mechanism, strategy update: Percentage of cooperating pairs vs negative links Average payoffs vs negative links