Lecture 10: Just-in-Time COMPILATION & PEEPHOLE OPTIMIZATIONS

CSC 313 – Advanced Programming Topics Lecture 10:Just-in-Time COMPILATION &PEEPHOLE OPTIMIZATIONS

Problem We Face • Want to optimize code, but face problems: Programmers are stupid • Code awful: often ignores simple optimizations • Methods merge cases that optimized differently • Cannot tell system about likely values or cases

Problem We Face • Want to optimize code, but face problems: Programmers are stupid • Code awful: often ignores simple optimizations • Methods merge cases that optimized differently • Cannot tell system about likely values or cases Wait, why does this matter?

Data Centers & Server Farms • Major expense for companies • Akamai at 95,000 servers (12/2011) • Dual core/dual processor in each • Each one had 16GB RAM • 250+ TB of storage total in system • Powering all of this demanded 53% of Niagara Falls output

Data Centers & Server Farms • At least 25 data centers for Google • Centers held ~900,000 servers • Greater diversity in machines • Generally < 2 years old • 4 – 8 GB RAM each • MS had set goal of 800,000 servers (2011)

Optimization Matters • Lots of money at stake • Estimates assume server counts double in 18 months • At least 3 more centers being build by Google • Each center costs $450 - $600 million • 1% improvement is huge for these companies • For Google that is 4,500+ servers • $72 million saved in power worldwide (2005)

Traditional Approach • Entire program compiled and optimized once • Target single processor (e.g., Xeon, Power) • Creates static machine language executable • Repeatedly analyze code during optimization • After each step reanalyze new result to improve • Does anything to get last 1% improvement

Compilation Is Slow

Optimizer Limited • Compiler must examine code as it is written • Cannot take advantage of many situations while (true) { String line = scanner.readLine();inti; if (line.compareTo(“EXIT”) != 0) {i = 1; } else {i = 0; } // What is the value of i?

JIT to the Rescue • Add Just-In-Time compiler (JIT) • As it runs, system will spy on methods • Look for patterns that can yield optimization • When good pattern found, recompile

JIT to the Rescue • Add Just-In-Time compiler (JIT) • As it runs, system will spy on methods • Look for patterns that can yield optimization • When good pattern found, recompile Execution Time Original 1.00

JIT to the Rescue • Add Just-In-Time compiler (JIT) • As it runs, system will spy on methods • Look for patterns that can yield optimization • When good pattern found, recompile Execution Time Original 1.00 W/Speedup - 0.20

JIT to the Rescue • Add Just-In-Time compiler (JIT) • As it runs, system will spy on methods • Look for patterns that can yield optimization • When good pattern found, recompile Execution Time Original 1.00 W/Speedup - 0.20 Total 1.50

What Happened? (1) • Program must examine methods as they run

What Happened? (1) • Program must examine methods as they run • Cannot just execute on computer at top speed

What Happened? (1) • Program must examine methods as they run • Cannot just execute on computer at top speed • Will need to evaluate & interpret instructions

What Happened? (1) • Program must examine methods as they run • Cannot just execute on computer at top speed • Will need to evaluate & interpret instructions • 10 times slower to interpret instructions

What Happened? (1) • Program must examine methods as they run • Cannot just execute on computer at top speed • Will need to evaluate & interpret instructions • 10 times slower to interpret instructions • Analyze results to see if optimization found

What Happened? (2)

What Happened? (2) • Compiling done once, if we were not using JIT • Will take forever (~10 minutes in coder speak) • Reducing even 1% still more than employees salary • As program runs, JIT compiles & recompiles • Each recompilation better & faster to run program • But compilation now added to total execution time

What Went Wrong You're bound to be unhappy if you optimize everything.

What Went Wrong You're bound to be unhappy if you optimize everything. Execution Time Original 1.00 W/Overhead 0.70 W/Speedup - 0.20 Total 1.50

Can We Reduce Costs? • Not worth doing this for certain methods • main() • String.offsetByCodePoints() • Most constructors • Anything that your partner wrote • Ignore cold code (code not run often/long) • Code not run enough to justify JIT costs • Code not important enough to justify JIT costs

Some Like It Hot • JIT examines only hottest methods • Executed most often or for the longest time • Hot methods are where program's time spent • Need data to analyze; only these provide it • Also these methods only where impact felt • When profitable, recompile hot methods • So the JIT will do this only if costs < time saved

Should We Recompile? • When we need to make this decision… • Time to compile code must be known • Need to find out how often method run • Improvement percentage also needed • Naturally, there is no way to know any of this • What can we do? • Do best, hope & pray that future resembles past • Make wild- scientific guess about how to do

Does It Work? • Dynamo runs programs 10% faster • 45,000+ servers would not be needed by Google • Would have saved US $270 million in 2005 power bills

Does It Work? • Dynamo runs programs 10% faster • 45,000+ servers would not be needed by Google • Would have saved US $270 million in 2005 power bills Execution Time Original 1.00 W/Overhead 0.10 W/Speedup - 0.20 Total 0.90

What Does It All Mean? • Optimizations can make a huge difference • Without our knowledge this will already be done • Always check "optimizations" do no harm • Compilers optimize a lot, but only if can prove safe • Know compiler details if you want fast code • After all, this is only way to write optimizable code

Instances of Class Class • Objects representing classes used by program • What fields is has & how fields laid out in memory • Starting address for methods to call them in code • This is structure (literally) extended by subclasses • Superclass code expects members to not move • Overriden method has starting address changed • Add fields to end to save older field definition • But, we must ask, what about optimizations?

Normal Method Call Class Super foo() StartAddress1bar() StartAddress2 public class Super {private void foo() {System.err.print(“Foo”); }public void bar() {foo();System.err.println(“bar”);} } Super s = new Super(); s.bar();

Polymorphic Method Call Class Super foo() StartAddress1bar() StartAddress2 public class Super {private void foo() { … }public void bar() {foo();System.err.println(“bar”);} } public class Sub extends Super {private void foo() {System.err.print(“Notfoo”); } } Super s = new Sub(); s.bar() Class Sub foo() StartAddress3bar() StartAddress2

What Is Done For Us • Good news! JIT compilers are very smart • If class has subclasses, will be recorded & used • Examines what the types of instances used are • If one method definition can ever be used… • JIT compiler can skip step to look up address • If more possible, but only one actually used… • Slightly slower, but guess address & add failpath • SOL if many used, since lookup is necessary

Help Compiler Help Us • Use final keyword to let compiler know: • Specify that a variable keeps initial value • Method not overriden& starting address constant • Compiler can do as much as possible w/o checks!

Compiler Helping Us • With final compiler can do many things • Getter & setter method calls become direct access • Addresses hardcoded & lookups eliminated • Use keyhole optimizations relying on actual values

final Method Calls public class Super {private final void foo() {System.err.print(“Foo”); }public void bar() {foo();System.err.println(“bar”);} } Super s = new Super(); s.bar();

final Field Uses public class GoodField {private int LENGTH = 8;public long recombine(byte[] arr){ long sum = 0; for (inti = 0; i < LENGTH; i++){ sum = (sum * LENGTH) + arr[i]; } return sum;} }

final Field Uses public class GoodField {private finalint LENGTH = 8;public long recombine(byte[] arr){ long sum = 0; for (inti = 0; i < LENGTH; i++){ sum = (sum * LENGTH) + arr[i]; } return sum;} }

final Field Uses public class GoodField {private final int LENGTH = 8;public long recombine(byte[] arr){ long sum = 0; for (inti = 0; i < 8; i++){ sum = (sum * 8) + arr[i]; } return sum;} }

final Field Uses public class GoodField {private final int LENGTH = 8;public long recombine(byte[] arr){ long sum = 0; for (inti = 0; i < 8; i++){ sum = (sum << 3) + arr[i]; } return sum;} }

Rules of Optimization • NO! • Not Yet • For experts only

Problems With final • Remember that final is a negative option • It removes options and limits future possibilities • Code becomes static and its reuse may be hurt • Each savings is small (remember Amdahl's Law) • Use when ABSOLUTELYNECESSARY ONLY

Problems With final • Remember that final is a negative option • It removes options and limits future possibilities • Code becomes static and its reuse may be hurt • Each savings is small (remember Amdahl's Law) • Use when ABSOLUTELYNECESSARY ONLY Isolate code that changes from code that stays the same

For Next Lecture • Read pages 79 – 94 in the book • What is the Decorator Pattern? • Can we learn design from coffee shops? • They are LOUSY coders who mess up great systems • What do the following have in common: • Dijkstra’s algorithm • Household finance program • E-mail program

Lecture 10: Just-in-Time COMPILATION & PEEPHOLE OPTIMIZATIONS