Applying control theory to stream processing systems
This presentation is the property of its rightful owner.
Sponsored Links
1 / 23

Applying Control Theory to Stream Processing Systems PowerPoint PPT Presentation


  • 50 Views
  • Uploaded on
  • Presentation posted in: General

Applying Control Theory to Stream Processing Systems. Wei Xu ( [email protected] ) Bill Kramer ( [email protected] ) Joe Hellerstein ( [email protected] ). TCQ drops tuples silently if result queue is full. Description of the system. TCQ Complex internal structure. Data Source.

Download Presentation

Applying Control Theory to Stream Processing Systems

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Applying control theory to stream processing systems

Applying Control Theory to Stream Processing Systems

Wei Xu ([email protected])

Bill Kramer ([email protected])

Joe Hellerstein ( [email protected] )


Description of the system

TCQ drops tuples silently if result queue is full

Description of the system

TCQ

Complex internal structure

Data Source

Input Buffer


Why do we need control

Why do we need control?

  • Data source does not provide accurate data rate


Why do we need control1

Why do we need control?

  • TCQ node drops tuples when result queue fill up

Source

Buffer

TCQ

Result Q


Control problems

Control Problems

  • Providing an accurate data source

    • Get the actual data rate

  • Regulate queue length on TCQ node

    • Prevent dropping tuples

    • Maximize throughput (and adapts when disturbance happens)


System with control

2

Queue Length Monitor

System with Control

Controlled

Data Source

Output Rate

Controller


The control architecture

PI Controller

The Control Architecture

P Controller


Result an accurate data source

Result – An accurate data source

P Controller with Pre-compensation

PI Controller


Result regulating queue length

Result – regulating queue length

Source

Buffer

TCQ

Result Q


Result under cpu contention

Result – Under CPU Contention

Source

Buffer

TCQ

Result Q


Why theory is useful

Why theory is useful?

  • One of my implementations .. What happened?

Source

Buffer

TCQ

Result Q


What is going on

What is going on?

Controlled

Output Thread(Code Reuse)

Queue Length

Controller

Desired

Queue length

Data Rate to TCQ

Actual Queue Length


Theory meets reality

Output Y from simulation

Theory meets reality

Queue length

Time


Tricky part of parameter estimation

Tricky part of parameter estimation

Model evaluation – Making the system operate in desired range

Data rate vs free space

Free Space

Non-Linear range

Easy for data source, but queue length ..


Settling time and overshoot matters

Settling Time and Overshoot matters

A lot of small disturbance in a Java program

Incremental garbage collection

P Controller

PI Controller


Conclusion

Conclusion

  • Advantages of feedback control

    • Make system more robust under disturbance

    • Treat complex systems as black boxes

      • Cope with the system characteristics instead of having to change it

    • Encourage reporting system statistics

    • Implementation is easy and has theoretical guarantees


Future work

Future Work

  • Load balancer

  • Smaller sample time to reduce disturbance caused by Java GC?

  • Controller on scheduling of system shared by multiple streams


Backup slides

Backup Slides


Outline

Outline

  • Problems and Motivation

  • Controller design

  • Result

  • Discussion


Description of the system1

Description of the System

Tuples

TCQ Node

Tuple

Blocks

Routing

Logic

Input Buffer

Data

Source

TCQ Node

Load Splitter

Tuples

Queue length

  • Operation of Load Splitter

  • Arriving blocks wait in Input Buffer

  • Tuples are routed to balance TCQ queue lengths

  • Stop routing if queue length is too large to avoid tuple discards

Revised


Compare to open loop control

Compare to Open Loop Control

We know

Y(k) , and we know what we want y(k+1) to be.. Use transfer function to solve for u(k)…

(Expected result – accuracy and disturbance ) -- do be done


Estimation of the transfer function

Estimation of the transfer function

y(k+1)=ay(k)+bu(k)

Regression


Tricky part of parameter estimation1

Tricky part of parameter estimation

Model evaluation – A data rate that make it operate in linear range


  • Login