Analyzing TZ Data

1. Analyzing TZ Data James A. Rome TGA August 2010

2. What are we trying to do? • I was hired by NASA to analyze air traffic in ZOB48 • I had access to ETMS data • The most useful data were TZ messages • But as we saw, they are highly quantized • 1–4 minute time intervals (no seconds) -> 6 nm • Multiple TZs from one flight at a given time (due to multiple radar reports) • Situation is far from ideal. What do we do? • First step is to look at the data and see how bad it is, and what we can (or cannot) do with it. • We will need to calculate things like GCD

3. Handling data is non-trivial • ACID,DEPT_APRT,ARR_APRT,POSIT_TIME,CUR_LAT,CUR_LON,GROUNDSPEED,ALTITUDE • COA497,CLE,MCO,8/7/99 0:01,2487,4907,472,290 • UAL1517,CLE,DEN,8/7/99 0:05,2487,4909,502,310 • Why do we need the airports? • Because the flight number applies to several legs of a flight • The big question is how should we arrange these data? • This depends on what we want to do with the data • Do any planes “collide” (<1000 ft or <5 nm)? • What path does the flight take through the sector? • When does the controller do something to the flight? • How many flights are there at any time? • . . . Initially, the data arearranged by time There are 3900 TZs

4. How good are the data? • The TZ messages have inherent error bars: • Lat/Lons are in minutes • 1 minute = 1 nm of latitude, and somewhat less in longitude. So positions might be off by a mile or so. • Time is in minutes, so if seconds are rounded down, may be up to a minute earlier than actual • This leads to distance inaccuracy of (speed/60) nm • Speed is in knots (nm/h) and is accurate enough • Altitude is in 100s of feet, so may be off by 100 ft • The distances can be off by 6–8 miles. 5 miles is the required separation distance between planes.

5. How should we handle the data? • You could not do this with a TI calculator • Maybe you could use Excel if the number of flights is small, but you would quickly find that you would be unable to answer all the above questions • We will have to write a computer program to make any progress • Every scientist or engineer who wants to solve a new problem will get to this point. If you cannot write a computer program, you cannot succeed!

6. It is time to write a program • We are going to calculate the great circle distance (GCD) between two Lat/Lon points • This will be part of a utility class because we will need these tools to do more interesting things. • The methods (things it can do) should be • Available to other programs (public) • Because there is no need for any changing data in the class as a whole, the methods can be static • This implies that multiple methods can call the static method at the same time without conflict (thread safe) • Static methods can be called without instantiating the class via ClassName.method • No new operator is needed

7. Make ZOB48 Project in NetBeans Select File, New Project

8. Packages • In a language as large as Java, it is important to organize your files to avoid name conflicts • Your method “openFile” might conflict with a method of the same name in some other part of Java • Packages create a unique way of naming your files • by convention, the package name is the reverse of your URL: gov.ornl.my_project_name • Packages also denote accessibility boundaries for methods and fields • private is in my class • protected is in my package • public is open access • You can move things around later via refactoring

9. Make a new package Note the Directory Tree this creates Package name = arc.airtraffic.util (arc is left-over from my course for ARC)

10. What is an object? • An object is a thing that has properties (attributes or fields) and can do stuff (methods). • In Java, almost everything is an object • An object is defined by a template called a class (the computer code that is used to create the object). • An object does not exist until it is instantiated. • What type of objects might we want to create for our ZOB48 analysis? • A TZ object, a Flight object, a Utility object, …

11. What is a class? • A class is the template for an object. • Unless a class has methods that are declared public static, you must instantiate the object from the class template before using it. new does this.Vector vec = new Vector(); // create vec • To call a method of a class, you use the variable name (e.g., vec above):vec.add("This is a String"); • If the method is static, you use the class name:double sinpi = Math.sin(3.14159265); double pi = Math.PI; // a static field

12. You could have created the package in this dialog. Make a new class • NetBeans will create a new Java file for you. • The name of the file must match the name of the class. Capitalization counts! • This file is LatLon.java • It is placed in the bottom of the package tree.

13. Special JavaDocs fields LatLon utility class Write JavaDoc comments as you go. This is a JavaDocs comment A static method can be called without instantiating an instance of its class End-of-line comment We will need the radius of the earth. final = a constant PI is a static field of the Java Math class

14. How do you know about Math.PI? • JavaDocs documents all Java classes • If you fill in the data, you can make JavaDocs of your own code too. • NetBeans can give you the JavaDocs for any method if you configure it. Package names Class names

15. NetBeans tip: How to change JRE

16. Add method to calculate angle Calling static methods I usually break up long formulas using temporary variables (trm1, trm2) to aid in debugging

17. Method to calculate the distance We never really want the angle between the points at the center of the Earth, so make a method we can call directly to get the great-circle-distance.

18. Make the JavaDocs

19. Here are your JavaDocs • As you type your code, NetBeanswill display your choices, and theirJavaDocs. • You can double-click a choice toinsert it into your code. • NetBeans will even guess at thenames of your variables to use asarguments!

20. Time to test the code • Although NetBeans finds typos and bad usage for you, it does not guarantee that your math is correct. • So let us make a test program to find the distance between two lat/lon points. • We will make a new class called “Runner” in the arc.airtraffic package. • I decided to calculate the distance from (approximately) Toledo to Cleveland

21. Google is great Toledo Cleveland Note, I eyeballed the lat/lons

22. The Runner class array of Strings main signature always NetBeans matches braces Be careful about parentheses, which determine theorder of computation. I always use more than needed.

23. Click Run, Run main project . . . You must tell NetBeans where to find main() You can have severaal main() methods in a project

24. It works!

25. TZ object • What might a TZ object consist of? • Attributes for one flight at one time • Time, lat, lon, altitude, speed, and a combination of the flight number, departure and destination airports. • E.g., NWA527_CLV_MSP • Actions it can do • Store a new (or updated) value of an attribute • Give you (another part of your program) any of these values • Objects generally do not allow other objects to access their attributes directly. Encapsulation

26. TZ pseudocode public methods and attributes accessible to all other objects Pseudocode lets you concentrate on what your code does, rather than how to write it, although this example is so simple, it is real code. public class TZData { // Class attributes private int altitude; // in hundreds of feet? private double latitude; // in radians private double longitude; // in radians private String acid; // the aircraft ID private double speed; // knots private long time; // time in seconds past 1/1/70 // Class methods // next slide content goes here } everything after // is a comment capital S means it is a class accessible only to TZData objects

27. TZData methods public TZData() {;} // default constructor public TZData(int alt, double lat, double lon, double speed, String acid, long time) { altitude = alt; latitude = lat; longitude = lon; this.speed = speed; this.acid = acid; // Only can use = for Strings this.time = time; } // These are the setters. Note the capitalization public intgetAltitude() {return altitude;} public double getLatitude() {return latitude;} public double getLongitude() {return longitude;} public double getSpeed() {return speed;} public String getAcid() {return acid;} public long getTime() {return time;} These are the type of thing returned to the caller

28. TZData methods (continued) • // the setter methods • public void setSpeed(double spd) {speed = spd;} • public void setLatitude(double lat) {latitude = lat;} • public void setLongitude(double lon) {longitude = lon;} • // This time we will use a constructor for String • public void setAcid(String id) {acid = new String(id);} • public void setAltitude(int alt) {altitude = alt;} • public void setTime(long time) {this.time = time;} • } // the end of the class (repeated from slide 1} • Notice how we used lowercase set and get for each of these methods to prepend the variable name which we capitalized. This makes this class into a Java Bean. • Java Beans can be added as elements into NetBeans void means that nothing is returned to caller no “return” needed for voids

29. What have we done? • NetBeans has done a lot of organizing for you • In the ZOB48 project directory is a src tree • The package name was translated into a directory tree • The class file is placed in the directory at the bottom of the tree Note: The name of the .java file must always be the same as the class name

30. Time to enter the class code NetBeans can do the grunge work of typing in the getters and setters

31. Getters and setters Right-click the TZDate.java file name, select Refactor, Encapsulate Fields

32. Our TZData class is complete • We could write a main method to test our class, but TZData is an important, but rather uninteresting class, so we will skip that for now.

33. How to read in data • We have a multi-pronged problem to solve • How to actually read data from a file • What to read the data into • Clearly we will read each row of data into a TZData object. (That's why we made them!) • But there are 3900 lines of data. We need a way (or several ways) of organizing these 3900 objects into things we can access easily. • How to convert the strings we read into the right types of data.

34. Look before you leap!!!!!!!!! • I cannot emphasize how important it is to look hard at all the issues before you start to code. • This is actually the fun part of writing a program • I do woodworking. I will spend a month figuring out how to make a jig to do something (e.g., a wide finger joint which needs 1/64" accuracy). • We are going to go though my thought process on how to read and store the data. • Many times . . .

35. Organizing TZData objects • Assume we have read in the data. How do we organize it? • Depends on the problem . . . • Can we tell if any flights are too close? • Need positions of all flights at a given time for each time Kind of a 2-D array. But the rows have different lengths. Hard to search.

36. Java has Collections

37. We will put the TZData at a given time in a Vector <E> is the type of Object in the Vector

38. We will organize the Vectors using a TreeMap I originally chose a HashMap, but decided that it would be useful to have the data ordered by time. Using the two is essentially the same. • A TreeMap allows you to store Key-Value pairs, sorted by the key. • Key = Long time • Value = Vector of TZData • A TreeMap is an ordered HashMap. The Hash allows very fast searching through the list of keys. Example: • 24 is a hash of "cat" • "cat" = 3 + 1 + 20 = 24 • It is a bad hash: • "act" = 24 also – a collision

39. Data organization The Keys are actually Long objects Long key = new Long(4356780L); Vectors of TZData Objects at given times These arrows are actually the memory locations of the TZData Vectors. These are called pointers. TreeMap

40. Pseudocode for loading Data • // open the data file • // while there are more lines { • // read a line from the file • // convert the data • // insert it into a TZData • // get the time • // if(the time is not in keys) { • // make a new TZ Vector • // add TZData to Vector • // put it into TreeMap • // } • // else { • // get Vector for this time • // see if this flight is there • // if (it is there) toss it • // else add it to the Vector • // } • // } // end of while • There are many ways to do this • We will assume the data are not in order • Notice that we have now introduced the new concepts of • looping (can you find 2?) • logical branching (if, else, else if) • But they naturally express how you have to think about the problem

41. Thinking more on the problem . . . • If we are going to decide whether any planes "hit" each other, we need to know all the flights in the air at once. • But the TZs seem to occur at different intervals for different flights, and they are not consistent at all! • We cannot implement the previous data arrangement because we are going to have to "manufacture" the missing data • Arrange the data by flight, ordered by time • Use interpolation to create new TZData objects at every minute • Then rearrange the data as needed • How????

42. Data organization redux 2 • We do not need the flight names sorted. So use a TreeSet with flight name as key. • We do need the times sorted. Implies that the times can be compared. TreeSet of times (must be Longs) These arrows are actually the memory locations of the TreeSets HashMap

43. TreeSet

44. Data organization redux 3 • But in an object-oriented world, we should really create a Flight object to store its data and to act upon it! • Attributes: TreeSet of TZData (sortedTZs) • Methods:interpolateTimesaddTZData // add a new TZData pointgetStart // get the entrance TZData for the sectorgetEnd // get the exit TZData for the sectorgetSortedTZs and setSortedTZs • This is just a start, I would think there will be more

45. We need a Comparator • Compare is an Interface • We have to tell the Sorted Tree how to compare TZData objects • Because Comparator is an interface, we need to implement its methods in a new class • package arc.airtraffic.data; • import java.util.Comparator; • class TZDataCompare implements Comparator<TZData> { • public int compare(TZData tzd1, TZData tzd2) { • long t1 = tzd1.getTime(); • long t2 = tzd2.getTime(); • // return -1, 0, +1 if tzd1 <, =, > tzd2 • return (int)(t1-t2); • } • } Click the lightbulb

46. Flight class We still need a method that interpolates any missing TZs to 1-minute intervals

47. File i/o (page 452) • A File object represents an existing File • File f = new File("full_path_to_file"); • You always want to buffer i/o. Disk access is slow, so read a big chunk at a time into a buffer. • import java.io.*; // need this • class FileReader { • public boolean readFile(String filename) { • try { // this may fail, so check for errors! • File f = new File(filename); • FileReader fileReader = new FileReader(f); • BufferedReader reader = new BufferedReader(fileReader); • // the file is now open and ready for reading

48. File i/o (continued) • // Make a place to put a line you read • String line = null; // always initialize variables • while((line = reader.readLine()) != null) { • System.out.println(line); // do something with it • } • reader.close(); // always close the file when done! • } catch (Exception ex) { • ex.printStackTrace(); • return false; // it failed • } • return true; // It succeeded • } // End of method readFile • } // End of class These parentheses make sure that the string is put into line before the test for null

49. Some String methods • ACID,DEPT_APRT,ARR_APRT,POSIT_TIME,CUR_LAT,CUR_LON,GROUNDSPEED,ALTITUDE • COA497,CLE,MCO,8/7/99 0:01,2487,4907,472,290 • UAL1517,CLE,DEN,8/7/99 0:05,2487,4909,502,310 • These are the lines we read. • Data are comma delimited. We must split them. • Must convert first 3 fields to UAL517_CLE_DEN • Must convert the date/time to seconds past 1/1/70 • Times are always a pain • Must store the data in the TZData

50. Splitting the String • String.split • public String[] split(String regex) • Splits this string around matches of the given regular expression. • // Assume that String line contains our line of data • String[] elements = line.split(","); // an array of Strings • int n = elements.length; // Java arrays know their length • String flightName = elements[0] + "_" + elements[1] + "_" • + elements[2]; // Array indices start at 0 in Java • int altitude = Integer.parseInt(elements[7]); • int speed = Integer.parseInt(elements[6]); • int lon = -Integer.parseInt(elements[5]); • int lat = Integer.parseInt(elements[4]); • // We still need to get the time…