Armed Bandits, Machine Learning and Fast Java: Practical Advice for Real-Time APIs
1 / 28

Armed Bandits, Machine Learning and Fast Java: Practical Advice for Real-Time APIs - PowerPoint PPT Presentation

  • Uploaded on

Armed Bandits, Machine Learning and Fast Java: Practical Advice for Real-Time APIs Breandan Considine OneSpot, Inc. Why Real-Time?. The world is full of hard problems Types of real time applications Hard (nuclear reactor control) Firm (auction bidding) Soft (train scheduling)

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Armed Bandits, Machine Learning and Fast Java: Practical Advice for Real-Time APIs' - evelyn-savage

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Armed Bandits, Machine Learning and Fast Java: Practical Advice for Real-Time APIs

Breandan Considine

OneSpot, Inc.

Why real time
Why Real-Time? Advice for Real-Time APIs

  • The world is full of hard problems

  • Types of real time applications

  • Hard (nuclear reactor control)

  • Firm (auction bidding)

  • Soft (train scheduling)

  • Real-time is a good thing

  • Real world applications

  • Performance over scalability

Benefits of real time processing
Benefits of Real-Time Processing Advice for Real-Time APIs

  • Forces us to narrow our priorities

  • Focus on constant, stable solutions rather than time-varying, exact solutions

  • Abundance of data but scarce processing power

  • Lifespan of actionable data extremely short

  • Tradeoff between optimality and throughput

  • Speed and parallelism will come over time

  • Upfront investment with long-term benefits

Real time interactive tasks rits
Real-time Interactive Tasks (RITs) Advice for Real-Time APIs

  • Online auctions: DSPs, SSPs

  • Multivariate testing

  • Inventory control, SCM

  • Scheduling, navigation, routing

  • Recommendation systems

  • High frequency trading

  • Fraud prevention

Common thread
Common Thread Advice for Real-Time APIs

  • Agent offered a context and set of choices

  • Each choice has a unknown payoff distribution

  • Choose an option, measure the outcome

  • Goal: Maximize cumulative payoff

  • Many instances

  • Time sensitive

  • Nontrivial features

Challenges Advice for Real-Time APIs

  • Impractical to test every action in context

  • Computationally intractable to consider

  • Cost of full survey outweighs benefit

  • Exploration-Exploitation Tradeoff

  • Opportunity cost for suboptimal choices

  • Local extrema conceal optimal solutions

  • Latency comes at the cost of throughput

  • Every clock cycle must count

  • Firm real-time characteristics

Traditional supervised learning cycle
Traditional Supervised Learning Cycle Advice for Real-Time APIs

Reinforcement learning rl
Reinforcement Learning (RL) Advice for Real-Time APIs

Dis advantages
Dis/advantages Advice for Real-Time APIs

  • Starts from scratch, training is expensive

  • Credit assignment problem & reward structure

  • Issues with non-stationary systems

  • Continuously integrates feedback

  • Adapts to real-time decisions

  • No assumptions about data

  • Follows signal on-line

  • Similar to how we learn

Non blocking algorithms
Non-blocking Algorithms Advice for Real-Time APIs

  • Critical for high performance I/O

  • Relatively difficult to implement correctly

  • Offers large speedup over lock-based variants

  • Types of non-blocking guarantees

  • Wait-freedom

  • Lock-freedom

  • Obstruction-freedom

Lock freedom
Lock-Freedom Advice for Real-Time APIs

  • Guarantees progress for at least one thread

  • Does not guarantee starvation-freedom

  • May be slower overall, see Amdahl's law

Java memory model
Java Memory Model Advice for Real-Time APIs

  • happens-before relation

  • Threaded operations follow a partial order

  • Ensures JVM does not reorder ops arbitrarily

  • Sequential consistency is guaranteed for race-free programs

  • Does not prevent threads from having different visibility on operations, unless explicitly declared

The volatile keyword
The Advice for Real-Time APIsvolatilekeyword

  • Mechanics governed by two simple rules

  • Each action within a thread happens in program order

  • volatile writes happen before all subsequent reads on that same field

  • Reads from and writes to main memory

  • Syntactic shorthand for lock on read, unlock on write – incurs similar performance toll

Java concurrency
Java Concurrency Advice for Real-Time APIs

  • ConcurrentHashMap, ConcurrentLinkedQueue

  • Need to carefully benchmark

  • Can be significantly slower depending on implementation

  • Avoid using default hash map constructor

  • Faster implementations exist, lock-free

  • Java 8 improvements in the pipeline

  • Prone to atomicity violations

ConcurrentHashMap<String, Data> map; Advice for Real-Time APIs

Data updateAndGet(String key) {

Data d = map.get(key);

if(d == null) { // Atomic violation

d = new Data();

map.put(key, d);


return d;


Java atomics
Java Atomics Advice for Real-Time APIs

  • Guarantees lock-free thread safety

  • Uses CAS primitives to ensure atomic execution

  • Better performance than volatile under low to moderate contention, must be tested in production setting

private T current; Advice for Real-Time APIs

public synchronized <T> T compareAndSet(T expected, T new) {

T previous = current;

if(current == expected)

current = new;

return previous;


Aba problem
ABA Problem Advice for Real-Time APIs

  • Direct equality testing is not sufficient

  • Full A-B-A transaction can execute immediately before execution of CAS primitive, causing unintended equality when structure has changed

  • Solution: generate a unique tag whenever value changes, then CAS against value-tag pair

False sharing
False Sharing Advice for Real-Time APIs

  • Can be prevented by padding out fields

  • Java 8 addresses this problem with @Contended

Multi armed bandit problems
Multi-Armed Bandit Problems Advice for Real-Time APIs

  • N choices, each with hidden payoff distributions

  • What strategy maximizes cumulative payoff?

  • Observation: Choose randomly from a distribution representing observed probability, return ARGMAX

Bayesian bandits
Bayesian Bandits Advice for Real-Time APIs


Adaptive control problems
Adaptive Control Problems Advice for Real-Time APIs

  • Parameter estimation for real time processes

  • Uses continuous feedback to adjust output

Pacing techniques
Pacing Techniques Advice for Real-Time APIs

Pid controller
PID Controller Advice for Real-Time APIs

Counting filtering problems
Counting/Filtering Problems Advice for Real-Time APIs

  • Large domain of inputs (IPs, emails, strings)

  • Need to maintain online, streaming aggregates

  • See Hadoop libraries for good implementations

  • Observation: Fast hashing is key.

Bloom filters
Bloom Filters Advice for Real-Time APIs

  • Fast probabilistic membership testing

  • Guarantees no false negatives, low space overhead

Special thanks to ian clarke matt cohen
Special thanks to Advice for Real-Time APIsIan ClarkeMatt Cohen

References Advice for Real-Time APIs

M. Michael, et al. Simple, Fast, and Practical Non-Blocking and Blocking Concurrent Queue Algorithms. [PDF]

P. Tsigas, et al. Wait-free queue algorithms for the real-time java specification. [PDF]