1 / 5

Transfer Learning Evaluation Workshop 2-3 May 2006 Evaluation Data Format & Analysis

Transfer Learning Evaluation Workshop 2-3 May 2006 Evaluation Data Format & Analysis Clayton T. Morrison, Yu-Han Chang, Paul R. Cohen USC Information Sciences Institute {clayton,ychang,cohen}@isi.edu. Data Format. ( list of IDs and variable settings for each

lanai
Download Presentation

Transfer Learning Evaluation Workshop 2-3 May 2006 Evaluation Data Format & Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Transfer Learning Evaluation Workshop 2-3 May 2006 Evaluation Data Format & Analysis Clayton T. Morrison, Yu-Han Chang, Paul R. Cohen USC Information Sciences Institute {clayton,ychang,cohen}@isi.edu

  2. Data Format ( list of IDs and variable settings for each ((ID repl#) (list of points on curve)) . . . ((ID repl#) (list of points on curve)) ) ( ( ( :<condition-name> (:<variable_1> <value_1>) . . . (:<variable_n> <value_n>) ) . . . ) ( ( ( :<condition-name> <replication-number> ) ( (x1 y1) (x2 y2) . . . (xn yn) ) ) . . . ) ) • We’ll expect x1=0 so we can measure jump-start. • We prefer that x1, x2, … for each curve to be consistent but will interpolate if necessary

  3. An example ( ( ( :exp1-b (:near-far 0.5) (:mtn NIL) (:train NIL) ) ( :exp1-ab (:near-far 0.5) (:mtn NIL) (:train T) ) ) ( ( ( :exp1-b 1 ) ( (0 15.4)(20 28.3)(40 33.1)(60 54.2)(80 56.1)(100 56.2) ) ) ( ( :exp1-b 2 ) ( (0 11.5)(20 23.9)(40 31.9)(60 54.0)(80 55.5)(100 55.9) ) ) . . . ( ( :exp1-b 7 ) ( (0 13.5)(20 28.9)(40 39.4)(60 56.0)(80 55.9)(100 56.2) ) ) ( ( :exp1-ab 1 ) ( (0 35.5)(20 40.4)(40 60.3)(60 62.4)(80 66.7)(100 66.9) ) ) . . . ( ( :exp1-ab 7 ) ( (0 40.1)(20 56.2)(40 59.1)(60 66.4)(80 66.9)(100 66.8) ) ) ) )

  4. Provide TL Analysis Functions • If you want to test any statistics other than the standard ones (transfer ratio, jump-start, etc), simply provide a function that operates on a pair of learning curves: (defun your-stat-fn (curve1 curve2) <your code to compute a statistic f>) where curve1 looks like ((0 10.3) (20 12.8) … (100 34.1)) • We’ll provide code for calculating the standard statistics. • You can provide replacement functions for the statistic (e.g. transfer-ratio), summarization (e.g. average), and asymptote (e.g. max-over-all-data) • We’ll also place all of this code online so you can use it. • To compare your curves, you’ll do something like (process-data :exp1-b :exp1-ab 7 #’transfer-ratio)

  5. End Example of good curve. * bonus points for group that provides us with one

More Related