Venugopal, Basu, and Foufoula-Georgiou, 2005: New metric for comparing precipitation patterns…

1. Venugopal, Basu, and Foufoula-Georgiou, 2005: New metric for comparing precipitation patterns… Verification methods reading group April 4, 2008 D. Ahijevych

2. Forecast Quality Index • useful for ensembles • uses “surrogate fields” • accounts for “close” forecasts • One number

3. Outline • Paper overview • universal image quality index (UIQI) and modified UIQI • components of forecast quality index (FQI) • Geometric examples (from Sukanta and Efi) • Perturbed “fake” examples (also from S and E) • Cases from SPC Spring 2005 • surrogates • traditional skill scores • expert rankings

4. Paper overview – forecast ensembles • filter out similar members, and keep just enough to characterize the probability structure of forecast • find “best” member and propagate it forward • single measure (like RMSE and EqTh) but has important additional information

5. Paper overview - UIQI • R1 and R2 are fields being compared • 3 terms: • covariance • means • standard deviations • 3 properties: • correlation • brightness (bias) • distortion (variability)

6. Paper overview – UIQI, Hausdorff • UIQI • entirely amplitude-based measure • not efficient at telling difference between displaced patterns and amplitude error • Distance-based measures • Hausdorff distance

7. Paper Overview - Hausdorff A h(A,B) forward distance B

8. Paper Overview - Hausdorff A B h(B,A) backward distance

9. Paper Overview - Hausdorff A h(B,A) backward distance B h(A,B) forward distance

10. Paper Overview - Hausdorff A H(A,B) B

11. Paper Overview – partial Hausdorff A h(A,B) B ?

12. A B Paper overview - Hausdorff a1 b1 h(A,B) forward distance a3 b2 a2

13. Paper overview - FQI

14. Paper overview - FQI

15. Paper Overview - surrogates

16. 2 1 0 Paper overview – illustrative example

18. Geometric examples CSI = 0 for first 4; CSI > 0 for the 5th

19. mod. UIQI, including zero pixels PHD75 mod. UIQI

24. when I did 10 surrogates <HS> = 271 +/-27

25. Perturbed fake cases • 3 pts right, -5 pts up • 6 pts right, -10 pts up • 12 pts right, -20 pts up • 24 pts right, -40 pts up • 48 pts right, -80 pts up • 12 pts right, -20 pts up, times 1.5 • 12 pts right, -20 pts up, minus 0.05”

34. Spring 2005 SPC cases • surrogates • pictures • example of distribution of forward and backward Hausdorff distances • comparison to traditional methods • comparison to expert scores

36. 75th percentile 100 surrogates – distribution of Hausdorff distance, solid/forward, dash/backward count Hausdorff distance (in grid spacing units)

37. surrogate mean PHD75 standard error mod. UIQI FQI: 0.47-0.49 PHD75

38. 0.26-0.28 0.25-0.27 0.34-0.37

39. 0.21-0.23 0.22-0.23 0.30-0.31

40. 0.21 0.25 0.24

41. 0.30 0.31 0.19

42. 0.51 0.69 0.42

43. 0.27 0.30 0.37

44. 0.37 0.40 0.33

45. 0.34 0.33 0.49

46. 0.42 0.48 0.54

47. r = w/o 1st case first case really bad; experts start out too generous?

48. expert scores vs grid stats grid stats agree: first case was bad

49. Pearson correlation coefficient and Spearman rank correlation coefficient

50. FQI Discussion • application to ensembles • adding to MET • . . .