Trading agent competition performance evaluation
This presentation is the property of its rightful owner.
Sponsored Links
1 / 31

Trading Agent Competition: Performance Evaluation PowerPoint PPT Presentation


  • 83 Views
  • Uploaded on
  • Presentation posted in: General

Trading Agent Competition: Performance Evaluation. Presented by Brett Borghetti [email protected] 22 March 2006. Think about this. You own a small business You make a bunch of strategic decisions/plans/policies Your 1 st quarter net profit is $100,000 Which choices helped?

Download Presentation

Trading Agent Competition: Performance Evaluation

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Trading agent competition performance evaluation

Trading Agent Competition: Performance Evaluation

Presented by Brett Borghetti

[email protected]

22 March 2006


Think about this

Think about this

  • You own a small business

  • You make a bunch of strategic decisions/plans/policies

  • Your 1st quarter net profit is $100,000

    • Which choices helped?

    • Which choices hurt?

    • Can your decisions be examined independently?

    • How do you improve next quarter?


The situation

The Situation

  • We sometimes have to make our plans and policies before their execution

  • We don’t know fully what the market will do next quarter (uncertainty)

  • We are in competition with other businesses/entities who may act to thwart our plans


A solution

A Solution

  • Repeat (until good enough):

    • Predict the effects of our choices offline

    • Adjust our choices to optimize outcome

  • Execute our plans

  • Measure the effectiveness of our choices online


Presentation overview

Presentation Overview

  • TAC-SCM Overview

  • Current analysis methods

  • New methods

  • Future Research


What is tac scm

What is TAC-SCM?

  • Simulation of a market supply chain

    • Agent is the computer manufacturer

    • Buys parts from suppliers in auction

    • Manage assembly line/production schedule

    • Reverse Auction to sell computers

    • Ship computers to customers

  • Six agents compete: maximize profit

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH


Tac scm interaction

TAC-SCM Interaction

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH


Game flow diagram

Game Flow Diagram


Tac why is it interesting

Complexity: Beyond human-in-the-loop capability

Compete with 5 other agents selling computers

Real time: 15 sec/day x 220 days

Auctions (normal and reverse for all transactions)

8 parts suppliers with production capacity changing daily

16 different computer types to build in 3 price classes

100s of Customers with varying demand and reserve prices

Price probing, future purchase decisions . . . . .

Small market: Agents have large impact on each other

Explicit Competition – PROFIT!

Learning other’s habits & patterns and out-thinking them

Information denial / Decision perturbation

TAC - Why is it Interesting?

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH


Umn minnetac design

UMN MinneTAC Design

  • Component-based architecture

    • Procurement – Purchase parts from suppliers

    • Production – Manages the production line

    • Sales – Interacts with customers to make sales

    • Shipping – plans customer shipping schedule

    • Repository – centralized data storage / accesors

    • Oracle – decision assistance evaluators

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH


Design pros and cons

Design pros and cons

  • Lower module coupling = good design

    • More simultaneous developers

    • Easier to maintain

  • Self interest vs. Common good

  • Causality – which components responsible for a good or bad decision?

  • How do we analyze and improve our global performance?

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH


Current analysis methods

Current Analysis Methods

  • Run offline simulations and tweak components to optimize profit

    • CPU intensive (1 hour per game)

    • Statistical significance => many games

    • Competition is limited

    • Causal analysis is complicated

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH


New analysis methods

New Analysis Methods

  • What if we could measure performance of components inside of the agent?

    • We could directly compare performance between two components of the same type against the same TAC market dataset

    • We could reduce the number of games required to show correlations / relative performance

    • We could more rapidly determine which ‘tweaks’ actually have an effect on game outcome

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH


Challenges of measuring

Challenges of Measuring

  • Which metrics are actually correlated with profit?

  • How do we assign sharing of credit or blame?

  • How do we account for the varying market conditions while taking measurements over multiple games?

  • How do we simulate various competitive environments offline?

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH


Methodology overview

Methodology - Overview

  • Controlling the market conditions

    • Control Randomness

    • Control market supply / demand situation

  • Measuring component performance

    • Create metrics

    • Determine if metric is correlated with profit

    • Assign component responsibility

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH


Controlling randomness

Controlling Randomness

  • Re-design server to allow deterministic / replayable games

  • Three types of random processes:

    • Server variables (customer/supplier)

    • Agent-dependent variables

    • Dummy agent variables

  • Each process gets its own seed

    • Eliminates race conditions in replays

    • Allows some process true randomness while others replay

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH


Market manipulation agents

Market Manipulation Agents

  • Goal – develop a way of manipulating supply and demand conditions during a simulation to observe how competitive agents respond

  • Method – Build TAC agents that are not concerned with their own profit, but rather with absorbing/releasing market share

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH


Market manipulation agents1

Market Manipulation Agents

  • Market Relief Agent

    • Accepts and fulfils no customer RFQs

    • Purchases no parts from suppliers

    • Result: Reduces demand on suppliers and reduces supply to customers

  • Market Pressure Agent

    • Makes more promises to customers than regular agent could handle

    • Buys more parts from suppliers than regular agent should

    • Result: Increases demand on suppliers and causes customer demand to go down

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH


Measuring component performance

Measuring Component Performance

  • Create suite of metrics to measure:

    • Replacement costs when a part is sold

    • Storage costs of parts/computers

    • Late penalties

    • Wasted production cycles

    • Remaining inventory at end of game

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH


Measuring causality

Measuring Causality

  • How do we assign responsibility?

  • For example: Why was the item late?

    • Didn’t ship the product?

    • Didn’t make the product?

    • Didn’t have the parts?

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH


Implementing metrics

Implementing Metrics

  • Allow for easy creation of new metrics

    • Serialize game information

    • Evaluations can then be made offline

    • Enables us to experiment in finding metrics that are correlated with profit.

  • But how do we even know if a metric is correlated with profit?

    • Large amount of variability in each game

    • Need a large sample size, which takes time

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH


Results to date

Results to date

  • We have some preliminary data regarding how the manipulation agents cause the other agents to behave under various market conditions

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH


Performance results market relief agent vs dummy agents

Performance Results: Market Relief Agent vs Dummy Agents

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

Note the scale of this graph


Performance results market relief agent vs dummy agents1

Performance Results: Market Relief Agent vs Dummy Agents

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

  • Unexpected benefits!

    • MRAs can reveal undesireable traits/logic flaws in an agent


Performance results market pressure agent vs minnetac

Performance Results: Market Pressure Agent vs MinneTAC

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH


Performance results market pressure agent vs minnetac1

Performance Results: Market Pressure Agent vs MinneTAC

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH


Performance results market pressure agent vs minnetac2

Performance Results: Market Pressure Agent vs MinneTAC

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH


Performance results market pressure agent vs competition

Performance Results: Market Pressure Agent vs Competition

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH


Conclusions

Conclusions

  • We’ve created some new tools for measuring offline performance

    • Replayable games

    • Market Condition Manipulation

    • Embedded Metrics Collection

  • Started choosing what metrics contain information allowing profit prediction

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH


Future work

Future Work

  • Improve Market Manipulation agents

    • Make competition modeling more realistic

  • Find additional metrics that have a better correlation to overall profit

    • Better off-line prediction of on-line performance

  • Use metrics to guide development of better components

    • Leads to better profit performance [build to the metric]

  • Use on-line metrics to make live strategic decisions

    • Live ‘tuning’ of components if they begin to underperform

    • Selection of ‘pinch-hitter’ components in certain market conditions

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH


Acknowledgement info

Acknowledgement / Info

Special thanks to:

  • Eric Sodomka

  • Dr. Maria Gini

  • Dr. John Collins

  • UMN TAC team

    More Info at

  • MinneTAC website

    • www.cs.umn.edu/tac

  • SICS website

    • www.sics.se/tac


  • Login