fly by wire l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Fly-by-Wire PowerPoint Presentation
Download Presentation
Fly-by-Wire

Loading in 2 Seconds...

play fullscreen
1 / 61

Fly-by-Wire - PowerPoint PPT Presentation


  • 525 Views
  • Uploaded on

Fly-by-Wire Instrument Mediated Performance Tuning Kirk Pepperdine kirk@kodewerk.com Fly-by-Wire A fly-by-wire system literally replaces physical control of the aircraft with an electrical interface. Wikipedia Computer mediated controls between the operator and the machine

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Fly-by-Wire' - Audrey


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
fly by wire

Fly-by-Wire

Instrument Mediated Performance Tuning

Kirk Pepperdine

kirk@kodewerk.com

fly by wire2
Fly-by-Wire
  • A fly-by-wire system literally replaces physical control of the aircraft with an electrical interface. Wikipedia
  • Computer mediated controls between the operator and the machine
  • First attempted in test flight of F-8 (from NASA)
    • Pilot incapable of flying without assistance
fly by wire3
Fly-by-Wire

F-8 (from NASA galleries)

reality vs senses
Reality vs. Senses
  • We learn to fly using sight
    • Licensed under Visual Flight Rules (VFR)
  • What happens when we can’t see?
    • Human senses are ill equipped for flight
  • Even when we can see we don’t always see things correctly
    • In order to see anything our brain fills in details
reality vs senses5
Reality vs. Senses
  • Conflict
    • All of our senses tell us one thing
      • You believe you are straight and level
    • Instrumentation tells us something else
      • Your attitude is sending you toward the ground
fly by wire6
Fly-by-Wire

My name is Bob and today I’ll be your pilot

who i am
Who I am
  • Kirk Pepperdine
    • Engaged around the world to solve Java performance problems
    • Co-author of http://www.javaperformancetuning.com
    • Associate Editor http://www.theserverside.com
    • Member of Sun Java Champions program
our typical story
Our Typical Story
  • Customer JoGoSlo Ltd calls us in desperation
    • Users complain about poor performance
      • Customers consider abandoning the project
    • 1 Million € invested
  • Management applying pressure
  • Developers
    • Can’t reproduce the problem in development
    • Recoding parts of the system hasn’t produced results
    • Common comment, “maybe we should do this”
    • Working under stress
effects of stress
Effects of Stress
  • Learning and problem solving capabilities are impaired
    • During a stressful event, catecholamines suppress activity in areas at the front of the brain inhibiting the ability to handle complex social or intellectual tasks and behaviors
    • Why Zebras don’t get Ulcers (Robert M. Sapolsky)
  • In organizations you see “tribal” behaviours
effects of stress11
Effects of Stress
  • Skills needed are impaired by the situation
  • Tend to rely on what you already know
    • Pilots may revert to VFR
      • Risk of crashing
    • Developers look at code
      • Risk missing the real problem
senses14
Senses

public static boolean checkInteger(String testInteger) {

try {

//fails if not a number

Integer theInteger = new Integer(testInteger);

return

(theInteger.toString() != "") && //not empty

(theInteger.intValue() > 10) &&

((theInteger.intValue() >= 2) &&

(theInteger.intValue() <= 100000)) &&

(theInteger.toString().charAt(0) == '3');

} catch (NumberFormatException err) {

return false;

}

}

senses15
Senses

public static void testDataset(String dataset)

throws IOException {

DataInputStream rdr = new DataInputStream(dataset);

long starttime = System.currentTimeMillis();

int truecount = 0;

String s;

try {

while ((s = rdr.readUTF()) != null)

if (checkInteger(s)) truecount++;

} catch (EOFException e) {}

rdr.close();

System.out.println(truecount + " (count); time

+ (System.currentTimeMillis() - starttime));

}

reality vs senses17
Reality vs Senses
  • Why isn’t code enough?
    • Developers work in a static medium
    • Performance tuners work in a dynamic medium that includes
      • Users
      • Application
      • Hardware
  • Need to consider dynamic nature of systems
      • Users
static view
Static View

public static int[][] mul( int[][] a, int[][] b) {

int rows = a.length;

int columns = (a[ 0]).length;

int[][] result = new int[rows][columns];

for ( int i = 0; i < rows; i++)

for ( int j = 0; j < columns; j++)

result[i][j] = a[i][j] * b[j][i];

return result;

}

dynamic view
Dynamic View

public static void main(String[] args) {

int[][] a = { {1, 1}, {1, 1}, {1, 1}, {1, 1}, {1, 1}, {1, 1}, {1, 1}, {1, 1}};

int[][] b = { {1, 2, 3, 4, 5, 6, 7, 8}, {8, 7, 6, 5, 4, 3, 2, 1}};

long start = System.currentTimeMillis();

for ( int i = 0; i < 50000; i++)

mul( b, a);

System.out.println(System.currentTimeMillis() - start);

start = System.currentTimeMillis();

for ( int i = 0; i < 50000; i++)

mul( a, b);

System.out.println(System.currentTimeMillis() - start);

}

dynamic observation21
Dynamic Observation
  • Response time is king
    • Defines the user experience
  • 130ms vs 300ms
    • Same code
    • Same amount of data
  • Why the difference?
  • What are the elements of a system?
elements of a system23
Elements of a System
  • Each layer contributes resources
  • Each layer defines whole classes of performance problems
  • Change a layer, change the problem
    • This include people
hardware
Hardware
  • Simply may not have enough
    • Buy more
    • May not be able to utilize more may
  • Utilization measured using
    • Unix : vmstat, iostat, mpstat, netstat, top
    • Windows : perfmon, taskmanager
hardware cpu
Hardware - CPU
  • CPU consumption reaches 100% for extended periods of time
    • Real values are often 40-60%
  • Measure using
    • vmstat in Unix
    • Taskmanager in Windows
hardware cpu26
Hardware - CPU
  • Two different types of consumption
    • Application (user)
    • Operating System (system)
  • Need to be considered separately
    • Directs tuning efforts
hardware cpu27
Hardware – CPU
  • High system utilization
    • Tune OS
    • Application usage of system calls
      • Locks, frequent I/O, excessive context switching
  • High user utilization
    • Focus on the application (or JVM)
  • Not necessarily bad
    • Use response times as a guide
    • Consider peaks in utilization (capacity)
hardware memory
Hardware - Memory
  • System starved of memory
    • Will cause high system utilization
    • May appear to be an disk I/O bottleneck
  • Measured using
    • vmstat, swap, and scan rate columns
    • perfmon to report on counters
  • Best solution is buy more memory
hardware i o
Hardware – I/O
  • Symptoms
    • Low CPU utilization
    • Poor response times
    • Cannot boost CPU utilization
  • Can be measured with
    • iostat or netstat
    • Perfmon (network or disk counters)
  • Reported as absolute numbers
    • Need to know hardware supported rates
hardware30
Hardware
  • If CPU, memory, disk and network I/O are all normal move to the JVM
slide31
JVM
  • Java memory is contained in Java heap
    • Managed by separate threads
    • Java process starved for memory will appear to be CPU bound
    • Large heap can cause lots of OS page swapping
  • Measure using –verbose:gc flag
  • Analyize output with GcViewer (Tagtram)
jvm memory
JVM Memory
  • Garbage collector (GC) manages Java heap memory
  • Look for GC efficiency
    • Defined as the portion of time spent managing memory over the run time duration
    • Values > 10% imply GC bottleneck
  • Tune generational spaces
  • Reduce memory utilization
heaps of heap
Heaps of Heap

1.330: [GC 1232K->755K(1984K), 0.0016326 secs]

1.377: [GC 1264K->808K(1984K), 0.0017882 secs]

1.448: [GC 1319K->905K(1984K), 0.0020653 secs]

1.595: [GC 1417K->919K(1984K), 0.0019377 secs]

1.682: [GC 1431K->1027K(1984K), 0.0029697 secs]

2.457: [GC 1538K->1104K(1984K), 0.0033163 secs]

2.583: [GC 1616K->1248K(1984K), 0.0034569 secs]

2.745: [GC 1759K->1355K(1984K), 0.0020179 secs]

2.861: [GC 1866K->1462K(1984K), 0.0021721 secs]

2.964: [GC 1963K->1488K(2112K), 0.0019511 secs]

2.967: [Full GC 1488K->1190K(2112K), 0.0472200 secs]

heaps of heap34
Heaps of Heap

1.330: [GC 1232K->755K(1984K), 0.0016326 secs]

1.377: [GC 1264K->808K(1984K), 0.0017882 secs]

1.448: [GC 1319K->905K(1984K), 0.0020653 secs]

1.595: [Full GC 1488K->1190K(2112K), 0.0472200 secs]

  • Runtime = 1595 – 1330 = 265
  • GC Time = 1.6326 + 1.7882 + 2.0653 + 47.22 = 52.7061
  • GC efficiency = 52.7061 / 265 = 19.8%
jvm threads
JVM Threads
  • Too many threads
    • High system utilizations
    • High cpu utilization with slow progression
    • Threads also stress memory
  • Too few threads and processes will starve for CPU
  • Measuring using ^c in the console window
    • Dumps state of all threads
webserver thread dump
Webserver Thread Dump

Full thread dump Java HotSpot(TM) Server VM (1.4.2_08-b03 mixed mode):

"RMI ConnectionExpiration-[192.168.0.15:34113]" daemon prio=1 tid=0x0892f658 nid=0x2d7a waiting on condition [5b86f000..5b86f494]

at java.lang.Thread.sleep(Native Method)

at sun.rmi.transport.tcp.TCPChannel$Reaper.run(TCPChannel.java:447)

at java.lang.Thread.run(Thread.java:534)

"RMI TCP Connection(902)-192.168.0.15" daemon prio=1 tid=0x41e112b8 nid=0x2d7a runnable [5ccff000..5ccff414]

at java.net.SocketInputStream.socketRead0(Native Method)

at java.net.SocketInputStream.read(SocketInputStream.java:129)

at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)

at java.io.BufferedInputStream.read(BufferedInputStream.java:201)

- locked <0x49978800> (a java.io.BufferedInputStream)

at java.io.FilterInputStream.read(FilterInputStream.java:66)

at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java)

at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport)

at java.lang.Thread.run(Thread.java:534)

webserver thread dump38
Webserver Thread Dump

"PingThread-8692809" daemon prio=1 tid=0x081a3058 nid=0x2d7a waiting on condition [5bda9000..5bda9294]

at java.lang.Thread.sleep(Native Method)

at org.exolab.jms.client.rmi.RmiJmsConnectionStub$PingThread.run(RmiJmsConnectionStub.java:249)

"EventManagerThread" daemon prio=1 tid=0x083101f8 nid=0x2d7a in Object.wait() [5caa9000..5caa9514]

at java.lang.Object.wait(Native Method)

- waiting on <0x47aa2800> (a java.lang.Object)

at java.lang.Object.wait(Object.java:429)

at org.exolab.jms.events.BasicEventManager.run(BasicEventManager.java)

- locked <0x47aa2800> (a java.lang.Object)

at java.lang.Thread.run(Thread.java:534)

"PingThread-18183604" daemon prio=1 tid=0x08136dc8 nid=0x2d7a waiting on condition [5cc7f000..5cc7f494]

at java.lang.Thread.sleep(Native Method)

at org.exolab.jms.client.rmi.RmiJmsConnectionStub$PingThread.run(

RmiJmsConnectionStub.java:249)

webserver thread dump39
Webserver Thread Dump

"RMI RenewClean-[192.168.0.15:34113]" daemon prio=1 tid=0x081a2c68 nid=0x2d7a in Object.wait() [5ba10000..5ba10594]

at java.lang.Object.wait(Native Method)

- waiting on <0x4858a940> (a java.lang.ref.ReferenceQueue$Lock)

at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)

- locked <0x4858a940> (a java.lang.ref.ReferenceQueue$Lock)

at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(

DGCClient.java:500)

at java.lang.Thread.run(Thread.java:534)

"BoundedThreadPool0-33" prio=1 tid=0x41ec8710 nid=0x2d7a in Object.wait() [5ca29000..5ca29594]

at java.lang.Object.wait(Native Method)

- waiting on <0x47a5> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool)

- locked <0x47a520b8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

webserver thread dump40
Webserver Thread Dump

"BoundedThreadPool0-32" prio=1 tid=0x5a159ed0 nid=0x2d7a runnable [5c9a9000..5c9a9614]

at java.net.SocketInputStream.socketRead0(Native Method)

at java.net.SocketInputStream.read(SocketInputStream.java:129)

at org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:99)

at org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector)

at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:257)

at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:192)

at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:293)

at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector)

at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool)

- locked <0x47a521f8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

webserver thread dump41
Webserver Thread Dump

"BoundedThreadPool0-31" prio=1 tid=0x5ad60d78 nid=0x2d7a runnable [5c929000..5c929694]

at org.mortbay.jetty.HttpGenerator.prepareBuffers(HttpGenerator.java:878)

at org.mortbay.jetty.HttpGenerator.flushBuffers(HttpGenerator.java:681)

at org.mortbay.jetty.HttpGenerator.complete(HttpGenerator.java:671)

at org.mortbay.jetty.HttpConnection.doHandler(HttpConnection.java:388)

at org.mortbay.jetty.HttpConnection.access$1500(HttpConnection.java:38)

at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(

HttpConnection.java:598)

at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:487)

at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:196)

at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:293)

at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector)

at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool)

- locked <0x47a52158> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

webserver thread dump42
Webserver Thread Dump

"BoundedThreadPool0-30" prio=1 tid=0x41e2f878 nid=0x2d7a in Object.wait() [5c8a9000..5c8a9714]

at java.lang.Object.wait(Native Method)

- waiting on <0x47a5> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool)

- locked <0x47a52298> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"BoundedThreadPool0-29" prio=1 tid=0x5a4c5650 nid=0x2d7a in Object.wait() [5c828000..5c828794]

at java.lang.Object.wait(Native Method)

- waiting on <0x47a5> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool)

- locked <0x47a52108> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"BoundedThreadPool0-28" prio=1 tid=0x5a4c53f8 nid=0x2d7a runnable [5c7a8000..5c7a8814]

at java.net.SocketInputStream.socketRead0(Native Method)

at java.net.SocketInputStream.read(SocketInputStream.java:129)

at org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:99)

at org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector)

at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:257)

at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:192)

at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:293)

at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector)

webserver thread dump43
Webserver Thread Dump

"BoundedThreadPool0-27" prio=1 tid=0x41e6a640 nid=0x2d7a in Object.wait() [5c728000..5c728894]

at java.lang.Object.wait(Native Method)

- waiting on <0x47a5> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool)

- locked <0x47a52338> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"BoundedThreadPool0-26" prio=1 tid=0x41e693f8 nid=0x2d7a in Object.wait() [5c6a8000..5c6a8914]

at java.lang.Object.wait(Native Method)

- waiting on <0x47a5> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool)

- locked <0x47a52248> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"BoundedThreadPool0-25" prio=1 tid=0x086b1c50 nid=0x2d7a in Object.wait() [5c628000..5c628994]

at java.lang.Object.wait(Native Method)

- waiting on <0x47a4> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool)

- locked <0x47a4e180> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

webserver thread dump44
Webserver Thread Dump
  • According to client, system was idle
    • Did not accept any more connection requests
  • Let’s go back a few slides…
    • Why was HttpGenerator.prepareBuffers() being called?
webserver thread dump45
Webserver Thread Dump

"BoundedThreadPool0-31" prio=1 tid=0x5ad60d78 nid=0x2d7a runnable [5c929000..5c929694]

at org.mortbay.jetty.HttpGenerator.prepareBuffers(HttpGenerator.java:878)

at org.mortbay.jetty.HttpGenerator.flushBuffers(HttpGenerator.java:681)

at org.mortbay.jetty.HttpGenerator.complete(HttpGenerator.java:671)

at org.mortbay.jetty.HttpConnection.doHandler(HttpConnection.java:388)

at org.mortbay.jetty.HttpConnection.access$1500(HttpConnection.java:38)

at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(

HttpConnection.java:598)

at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:487)

at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:196)

at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:293)

at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector)

at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool)

- locked <0x47a52158> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

jvm threads46
JVM Threads
  • Use thread pooling to control over-utilization
    • Acts to calm traffic
    • Smoothen traffic reduces thrashing
    • Reduced thrashing increases throughput
      • Should reduce response times
application
Application
  • Applications
    • Interact with external systems
    • Add locks to protect software resources
  • The only things to consider are lock contention and interactions with external systems
application48
Application
  • Measure using ^c in the console window
  • Lock contention
    • Threads waiting on monitors
  • Waits for external systems
    • Almost all communications with external systems occur via sockets
    • Threads waiting on socket read/write
application49
Application
  • Most problematic external system
    • Relational databases
  • Monitor with intercepting technology
    • P6spy acts like a jdbc driver
    • Logs interactions
    • Logs can be viewed with Ironeye (source forge)
users
Users
  • If at the end of the analysis you’ve eliminated hardware, JVM, and the Application, the only thing left are the users
users53
Users
  • Users follow specific patterns
  • Users will work a specific rates
  • Essential that these patterns and rates are
    • Understood
    • Well simulated
  • Examine logs
  • Visit the floor
users54
Users
  • Need this data to setup realistic testing conditions
  • Use test harness such as Apache JMeter
    • Visit how to do this in the demo
fly by wire55
Fly-by-Wire
  • Investigative process of a dynamic system
    • Code is only one element
  • Quickly eliminate problems
  • Start with performance requirements to setup test harness
  • Measure utilization of resource offered by each layer
fly by wire56
Fly-by-Wire

Let Measurements lead the way

joegoslow why
JoeGoSlow Why?
  • Discussions with client suggested database caching
    • Database O/R mapping code described as ugly
  • Suspected memory leak
    • Still was unsure of this
    • Users claimed application sometimes recovered
  • Question: is this from the database interaction or a memory leak in a long-term temporary object?
joegoslow why58
JoeGoSlow Why?

Session timed out

joegoslow why59
JoeGoSlow Why?
  • HTTPSession timeout correlated strongly with decrease in memory
    • Confirmed with commercial memory profiler
      • Output from memory profilers is often very confusing for large systems
      • This additional information helped us filter the memory profiler
joegoslow no more
JoeGoSlow No More
  • Performance degradation was due to memory leak
    • Confirmed initial hypothesis
  • Memory leak was due to HTTPSession timeout
    • Complete surprise to everyone
    • Refocused efforts in the right direction
    • Solution was implemented in minutes
    • Solution was validated in minutes
fly by wire61
Fly-by-Wire

Questions?

kirk@kodewerk.com