trading agent competition performance evaluation
Download
Skip this Video
Download Presentation
Trading Agent Competition: Performance Evaluation

Loading in 2 Seconds...

play fullscreen
1 / 31

Trading Agent Competition: Performance Evaluation - PowerPoint PPT Presentation


  • 107 Views
  • Uploaded on

Trading Agent Competition: Performance Evaluation. Presented by Brett Borghetti [email protected] 22 March 2006. Think about this. You own a small business You make a bunch of strategic decisions/plans/policies Your 1 st quarter net profit is $100,000 Which choices helped?

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Trading Agent Competition: Performance Evaluation' - bradley-albert


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
think about this
Think about this
  • You own a small business
  • You make a bunch of strategic decisions/plans/policies
  • Your 1st quarter net profit is $100,000
    • Which choices helped?
    • Which choices hurt?
    • Can your decisions be examined independently?
    • How do you improve next quarter?
the situation
The Situation
  • We sometimes have to make our plans and policies before their execution
  • We don’t know fully what the market will do next quarter (uncertainty)
  • We are in competition with other businesses/entities who may act to thwart our plans
a solution
A Solution
  • Repeat (until good enough):
    • Predict the effects of our choices offline
    • Adjust our choices to optimize outcome
  • Execute our plans
  • Measure the effectiveness of our choices online
presentation overview
Presentation Overview
  • TAC-SCM Overview
  • Current analysis methods
  • New methods
  • Future Research
what is tac scm
What is TAC-SCM?
  • Simulation of a market supply chain
    • Agent is the computer manufacturer
    • Buys parts from suppliers in auction
    • Manage assembly line/production schedule
    • Reverse Auction to sell computers
    • Ship computers to customers
  • Six agents compete: maximize profit

TAC-SCMCURRENT ANALYSIS NEW METHODS FUTURE RESEARCH

tac scm interaction
TAC-SCM Interaction

TAC-SCMCURRENT ANALYSIS NEW METHODS FUTURE RESEARCH

tac why is it interesting
Complexity: Beyond human-in-the-loop capability

Compete with 5 other agents selling computers

Real time: 15 sec/day x 220 days

Auctions (normal and reverse for all transactions)

8 parts suppliers with production capacity changing daily

16 different computer types to build in 3 price classes

100s of Customers with varying demand and reserve prices

Price probing, future purchase decisions . . . . .

Small market: Agents have large impact on each other

Explicit Competition – PROFIT!

Learning other’s habits & patterns and out-thinking them

Information denial / Decision perturbation

TAC - Why is it Interesting?

TAC-SCMCURRENT ANALYSIS NEW METHODS FUTURE RESEARCH

umn minnetac design
UMN MinneTAC Design
  • Component-based architecture
    • Procurement – Purchase parts from suppliers
    • Production – Manages the production line
    • Sales – Interacts with customers to make sales
    • Shipping – plans customer shipping schedule
    • Repository – centralized data storage / accesors
    • Oracle – decision assistance evaluators

TAC-SCMCURRENT ANALYSIS NEW METHODS FUTURE RESEARCH

design pros and cons
Design pros and cons
  • Lower module coupling = good design
    • More simultaneous developers
    • Easier to maintain
  • Self interest vs. Common good
  • Causality – which components responsible for a good or bad decision?
  • How do we analyze and improve our global performance?

TAC-SCMCURRENT ANALYSIS NEW METHODS FUTURE RESEARCH

current analysis methods
Current Analysis Methods
  • Run offline simulations and tweak components to optimize profit
    • CPU intensive (1 hour per game)
    • Statistical significance => many games
    • Competition is limited
    • Causal analysis is complicated

TAC-SCMCURRENT ANALYSIS NEW METHODS FUTURE RESEARCH

new analysis methods
New Analysis Methods
  • What if we could measure performance of components inside of the agent?
    • We could directly compare performance between two components of the same type against the same TAC market dataset
    • We could reduce the number of games required to show correlations / relative performance
    • We could more rapidly determine which ‘tweaks’ actually have an effect on game outcome

TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

challenges of measuring
Challenges of Measuring
  • Which metrics are actually correlated with profit?
  • How do we assign sharing of credit or blame?
  • How do we account for the varying market conditions while taking measurements over multiple games?
  • How do we simulate various competitive environments offline?

TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

methodology overview
Methodology - Overview
  • Controlling the market conditions
    • Control Randomness
    • Control market supply / demand situation
  • Measuring component performance
    • Create metrics
    • Determine if metric is correlated with profit
    • Assign component responsibility

TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

controlling randomness
Controlling Randomness
  • Re-design server to allow deterministic / replayable games
  • Three types of random processes:
    • Server variables (customer/supplier)
    • Agent-dependent variables
    • Dummy agent variables
  • Each process gets its own seed
    • Eliminates race conditions in replays
    • Allows some process true randomness while others replay

TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

market manipulation agents
Market Manipulation Agents
  • Goal – develop a way of manipulating supply and demand conditions during a simulation to observe how competitive agents respond
  • Method – Build TAC agents that are not concerned with their own profit, but rather with absorbing/releasing market share

TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

market manipulation agents1
Market Manipulation Agents
  • Market Relief Agent
    • Accepts and fulfils no customer RFQs
    • Purchases no parts from suppliers
    • Result: Reduces demand on suppliers and reduces supply to customers
  • Market Pressure Agent
    • Makes more promises to customers than regular agent could handle
    • Buys more parts from suppliers than regular agent should
    • Result: Increases demand on suppliers and causes customer demand to go down

TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

measuring component performance
Measuring Component Performance
  • Create suite of metrics to measure:
    • Replacement costs when a part is sold
    • Storage costs of parts/computers
    • Late penalties
    • Wasted production cycles
    • Remaining inventory at end of game

TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

measuring causality
Measuring Causality
  • How do we assign responsibility?
  • For example: Why was the item late?
      • Didn’t ship the product?
      • Didn’t make the product?
      • Didn’t have the parts?

TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

implementing metrics
Implementing Metrics
  • Allow for easy creation of new metrics
    • Serialize game information
    • Evaluations can then be made offline
    • Enables us to experiment in finding metrics that are correlated with profit.
  • But how do we even know if a metric is correlated with profit?
    • Large amount of variability in each game
    • Need a large sample size, which takes time

TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

results to date
Results to date
  • We have some preliminary data regarding how the manipulation agents cause the other agents to behave under various market conditions

TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

performance results market relief agent vs dummy agents
Performance Results: Market Relief Agent vs Dummy Agents

TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

Note the scale of this graph

performance results market relief agent vs dummy agents1
Performance Results: Market Relief Agent vs Dummy Agents

TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

  • Unexpected benefits!
    • MRAs can reveal undesireable traits/logic flaws in an agent
performance results market pressure agent vs minnetac
Performance Results: Market Pressure Agent vs MinneTAC

TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

performance results market pressure agent vs minnetac1
Performance Results: Market Pressure Agent vs MinneTAC

TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

performance results market pressure agent vs minnetac2
Performance Results: Market Pressure Agent vs MinneTAC

TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

performance results market pressure agent vs competition
Performance Results: Market Pressure Agent vs Competition

TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

conclusions
Conclusions
  • We’ve created some new tools for measuring offline performance
    • Replayable games
    • Market Condition Manipulation
    • Embedded Metrics Collection
  • Started choosing what metrics contain information allowing profit prediction

TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

future work
Future Work
  • Improve Market Manipulation agents
    • Make competition modeling more realistic
  • Find additional metrics that have a better correlation to overall profit
    • Better off-line prediction of on-line performance
  • Use metrics to guide development of better components
    • Leads to better profit performance [build to the metric]
  • Use on-line metrics to make live strategic decisions
    • Live ‘tuning’ of components if they begin to underperform
    • Selection of ‘pinch-hitter’ components in certain market conditions

TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

acknowledgement info
Acknowledgement / Info

Special thanks to:

  • Eric Sodomka
  • Dr. Maria Gini
  • Dr. John Collins
  • UMN TAC team

More Info at

  • MinneTAC website
    • www.cs.umn.edu/tac
  • SICS website
    • www.sics.se/tac
ad