slide1
Download
Skip this Video
Download Presentation
Armed Bandits, Machine Learning and Fast Java: Practical Advice for Real-Time APIs

Loading in 2 Seconds...

play fullscreen
1 / 28

Armed Bandits, Machine Learning and Fast Java: Practical Advice for Real-Time APIs - PowerPoint PPT Presentation


  • 125 Views
  • Uploaded on

Armed Bandits, Machine Learning and Fast Java: Practical Advice for Real-Time APIs Breandan Considine OneSpot, Inc. Why Real-Time?. The world is full of hard problems Types of real time applications Hard (nuclear reactor control) Firm (auction bidding) Soft (train scheduling)

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Armed Bandits, Machine Learning and Fast Java: Practical Advice for Real-Time APIs' - evelyn-savage


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1
Armed Bandits, Machine Learning and Fast Java: Practical Advice for Real-Time APIs

Breandan Considine

OneSpot, Inc.

why real time
Why Real-Time?
  • The world is full of hard problems
  • Types of real time applications
  • Hard (nuclear reactor control)
  • Firm (auction bidding)
  • Soft (train scheduling)
  • Real-time is a good thing
  • Real world applications
  • Performance over scalability
benefits of real time processing
Benefits of Real-Time Processing
  • Forces us to narrow our priorities
  • Focus on constant, stable solutions rather than time-varying, exact solutions
  • Abundance of data but scarce processing power
  • Lifespan of actionable data extremely short
  • Tradeoff between optimality and throughput
  • Speed and parallelism will come over time
  • Upfront investment with long-term benefits
real time interactive tasks rits
Real-time Interactive Tasks (RITs)
  • Online auctions: DSPs, SSPs
  • Multivariate testing
  • Inventory control, SCM
  • Scheduling, navigation, routing
  • Recommendation systems
  • High frequency trading
  • Fraud prevention
common thread
Common Thread
  • Agent offered a context and set of choices
  • Each choice has a unknown payoff distribution
  • Choose an option, measure the outcome
  • Goal: Maximize cumulative payoff
  • Many instances
  • Time sensitive
  • Nontrivial features
challenges
Challenges
  • Impractical to test every action in context
  • Computationally intractable to consider
  • Cost of full survey outweighs benefit
  • Exploration-Exploitation Tradeoff
  • Opportunity cost for suboptimal choices
  • Local extrema conceal optimal solutions
  • Latency comes at the cost of throughput
  • Every clock cycle must count
  • Firm real-time characteristics
dis advantages
Dis/advantages
  • Starts from scratch, training is expensive
  • Credit assignment problem & reward structure
  • Issues with non-stationary systems
  • Continuously integrates feedback
  • Adapts to real-time decisions
  • No assumptions about data
  • Follows signal on-line
  • Similar to how we learn
non blocking algorithms
Non-blocking Algorithms
  • Critical for high performance I/O
  • Relatively difficult to implement correctly
  • Offers large speedup over lock-based variants
  • Types of non-blocking guarantees
  • Wait-freedom
  • Lock-freedom
  • Obstruction-freedom
lock freedom
Lock-Freedom
  • Guarantees progress for at least one thread
  • Does not guarantee starvation-freedom
  • May be slower overall, see Amdahl\'s law
java memory model
Java Memory Model
  • happens-before relation
  • Threaded operations follow a partial order
  • Ensures JVM does not reorder ops arbitrarily
  • Sequential consistency is guaranteed for race-free programs
  • Does not prevent threads from having different visibility on operations, unless explicitly declared
the volatile keyword
Thevolatilekeyword
  • Mechanics governed by two simple rules
  • Each action within a thread happens in program order
  • volatile writes happen before all subsequent reads on that same field
  • Reads from and writes to main memory
  • Syntactic shorthand for lock on read, unlock on write – incurs similar performance toll
java concurrency
Java Concurrency
  • ConcurrentHashMap, ConcurrentLinkedQueue
  • Need to carefully benchmark
  • Can be significantly slower depending on implementation
  • Avoid using default hash map constructor
  • Faster implementations exist, lock-free
  • Java 8 improvements in the pipeline
  • Prone to atomicity violations
slide15
ConcurrentHashMap<String, Data> map;

Data updateAndGet(String key) {

Data d = map.get(key);

if(d == null) { // Atomic violation

d = new Data();

map.put(key, d);

}

return d;

}

java atomics
Java Atomics
  • Guarantees lock-free thread safety
  • Uses CAS primitives to ensure atomic execution
  • Better performance than volatile under low to moderate contention, must be tested in production setting
slide17
private T current;

public synchronized <T> T compareAndSet(T expected, T new) {

T previous = current;

if(current == expected)

current = new;

return previous;

}

aba problem
ABA Problem
  • Direct equality testing is not sufficient
  • Full A-B-A transaction can execute immediately before execution of CAS primitive, causing unintended equality when structure has changed
  • Solution: generate a unique tag whenever value changes, then CAS against value-tag pair
false sharing
False Sharing
  • Can be prevented by padding out fields
  • Java 8 addresses this problem with @Contended
multi armed bandit problems
Multi-Armed Bandit Problems
  • N choices, each with hidden payoff distributions
  • What strategy maximizes cumulative payoff?
  • Observation: Choose randomly from a distribution representing observed probability, return ARGMAX
bayesian bandits
Bayesian Bandits

*http://camdp.com/blogs/multi-armed-bandits

adaptive control problems
Adaptive Control Problems
  • Parameter estimation for real time processes
  • Uses continuous feedback to adjust output
counting filtering problems
Counting/Filtering Problems
  • Large domain of inputs (IPs, emails, strings)
  • Need to maintain online, streaming aggregates
  • See Hadoop libraries for good implementations
  • Observation: Fast hashing is key.
bloom filters
Bloom Filters
  • Fast probabilistic membership testing
  • Guarantees no false negatives, low space overhead
references
References

http://mechanical-sympathy.blogspot.ie/

http://camdp.com/blogs/multi-armed-bandits

http://blog.locut.us/2011/09/22/proportionate-ab-testing

http://blog.locut.us/2008/01/12/a-decent-stand-alone-java-bloom-filter-implementation/

http://www.cl.cam.ac.uk/research/srg/netos/lock-free/

https://github.com/edwardw/high-scale-java-lib

M. Michael, et al. Simple, Fast, and Practical Non-Blocking and Blocking Concurrent Queue Algorithms. [PDF]

P. Tsigas, et al. Wait-free queue algorithms for the real-time java specification. [PDF]

ad