Ground based evaluation of cloud forecasts. Robin Hogan Ewan O’Connor, Anthony Illingworth University of Reading, UK Clouds radar collaboration meeting 17 Nov 09. Project. Aim: to retrieve and evaluate the crucial cloud variables in forecast and climate models
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
Robin Hogan
Ewan O’Connor, Anthony Illingworth
University of Reading, UK
Clouds radar collaboration meeting 17 Nov 09
joint with EUSAAR (trace gases) + Earlinet (lidar/aerosol)
ACTRIS: Aerosol Clouds and Trace gases Research Infrastructure Network.
Ice
Liquid
Rain
Aerosol
L2a IWC on radar/lidar grid
L2b Cloud fraction on model grid
Chilbolton
Observations
Met Office
Mesoscale Model
ECMWF
Global Model
MeteoFrance
ARPEGE Model
KNMI
RACMO Model
Swedish RCA model
07 km
Illingworth et al. (BAMS 2007)
3 months data. 2009.
Ideally global and 1.5km model as well.
Compare 12km with 4km and also 4km averaged 3x3 boxes.
Is the performance any better at 4km?
Can make more overcast skies?
Any improvement on mid level cloud?
What about low level clouds?
What about getting the right cloud in the right place at
the right time  skill scores?
Mean fraction too low
Equitable threat score:
Falls with fraction
threshold and age
of forecast.
Barrett, Hogan & O’Connor (GRL 2009)
Radar and lidar provide cloud boundaries and cloud properties above site
MeteoFrance:
Local mixing scheme: too little entrainment
SMHI:
Prognostic TKE scheme: no diurnal evolution
All other models have a nonlocal mixing scheme in unstable conditions and an explicit formulation for entrainment at cloud top: better performance over the diurnal cycle
Model cloud
Model clearsky
Observed cloud Observed clearsky
For given set of observed events, only 2 degrees of freedom in all possible forecasts (e.g. a & b), because 2 quantities fixed:
 Number of events that occurred n =a +b +c +d
 Base rate (observed frequency of occurrence) p =(a +c)/n
Extreme dependency score
Hogan, O’Connor and Illingworth (2009 QJRMS)
SEDS
EDS
LBSS
HSS
LOR
Hogan, Ferro, Jolliffe and Stephenson (WAF, in press)
CLOUDNET
Clouds in the 4km v 12km NAE.
Diurnal cycle of BL clouds in various models.
Problems with the ETS (now GSS) – use SEDS
Now DRIZZLE!
BL clouds in models drizzle all the time.
New observations from CloudSat/Calipso
compared with FWD model from ECMWF.
22dBZ 0.4g/m3 or 0.001mm/hr
(1mm per month: 0.6 W/m2).
ECMWF FWD MODEL:
LWP 100 g/m2 0dBZ
160 times too much drizzle!
Drizzle rate 0.03mm/hr.
{20 W/m2, 300m layer cools 0.3/hr}
OBSERVED
Z
OBSERVATIONS: Z  LWP.
LWP 100 g/m2 22dBZ
LWP
MODEL
ECMWF rain flux parameterisation
Autoconversion of cloud mixing ratio qcl to rain mixing ration qr
=K qcl
Threshold term: turns off autoconversion for value below qcl,crit = 0.3 g kg1
Without threshold term: dqr /dt q cl
LWP of 1000g/m2 0.6 mm/hr
LWP of 100gm2 0.06 mm/hr
Add threshold assume adiabatic 0.03mm/hr (0dBZ)
So why not increase qcl.crit to stop all the drizzle forming?
NO!
This will increase the lwp of all water clouds,
make them too bright and destroy the global radiation balance.
26
Evidence that the clouds in ECMWF are more adiabatic than observed?
F
Cloud amount
>80%
Observed 25% adiabatic? Modelled 50% adiabatic?
MODEL AUTOCONVERSION: for LWP 100g/m2
100% adiabatic 0.03mm/hr 0dBZ 300m deep/ max LWC 0.6gm3
50% adiabatic 0.02mm/hr 450m deep/max LWC 0.45g/m3
25% adiabatic 0.01mm/hr 8dBZ 700m deep/max LWC 0.3g/m3 CSAT gate 500m.
27
1 year from Murgtal
DWD COSMO model
Joint PDFs of cloud fractionb
a
d
c
…or use a simple contingency table
Reality (n=16, p=1/4)
Forecast
Underprediction No bias Overprediction
Best possible forecast

Positive
skill
Random
forecast
Negative
skill
Random unbiased forecast
Constant forecast of occurrence
Constant forecast of
nonoccurrence
Worst possible forecast
H=0.5
H=0.75
H=1
Defined by Gandin and Murphy (1992):
HSS = [xE(x)] / [nE(x)]; x = a+d ETS = [aE(a)] / [a+b+cE(a)]
E(a) = (a+b)(a+c)/nis the expected value of a for an unbiased random forecasting system
Simple attempts to hedge will fail for all these measures
LOR = ln[ad/bc] ORSS = [ad/bc – 1] / [ad/bc + 1]
Random and constant forecasts all score zero, so these measures are all equitable, right?
HSS
LOR
SEDS
HSS
LOR
S[E(a), E(b), E(c), E(d)] = 0
But expected score may not be zero!
E[S(a,b,c,d)] = S P(a,b,c,d)S(a,b,c,d)
Width of random probability distribution decreases for larger sample size n
A measure is only equitable if positive and negative scores cancel
Which measures are equitable?ETS & ORSS are asymmetric
n = 16
n = 80
Report confidence intervals and “pvalues” (the probability of a score being achieved by chance)
2007
2004
Met Office DWD
2007
2004
3.0 d
2.7 days
2.6 days
3.2 d
3.1 days
2.9 days
4.0 days
2.7 days
3.1 d
2.9 days
2.4 days
4.3 days
2.9 days
4.3 days
2.7 days
Lidar apparent backscatter coefficient (m1 sr1)
Latitude
Optically thick liquid cloud obscures view of any clouds beneath
Solution: forwardmodel the measurements (including attenuation) using the ECMWF variables
ECMWF raw cloud fraction
ECMWF processed cloud fraction
IceSAT cloud fraction
Wilkinson, Hogan, Illingworth and Benedetti (MWR 2008)
Tropical skill appears to peak at midlevels but cloud very infrequent here
Clearly need to apply SEDS to cloud estimated from lidar & radar!
Highest skill in north midlatitude and polar upper troposphere
Unreliable region
Is some of reduction of skill at low levels because of lidar attenuation?
Lowest skill: tropical boundarylayer clouds
Wilkinson, Hogan, Illingworth and Benedetti (MWR 2008)
Perfect
No resolution
No skill
Jakob et al. (2004)
ECMWF cloud fraction after processing
IceSAT cloud fraction
ECMWF raw cloud fraction
Simulate lidar backscatter:
Error due to uncertain extinctiontobackscatter ratio
Reduction in model due to lidar attenuation
4. ECMWF rain flux parameterisation
Autoconversion of cloud mixing ratio qcl to rain mixing ration qr
=K qcl
Threshold term: turns off autoconversion for value below qcl,crit = 0.3 g kg1
Without threshold term: dqr /dt q cl
LWP of 1000g/m2 0.6 mm/hr
LWP of 100gm2 0.06 mm/hr
Add threshold assume adiabatic 0.03mm/hr (0dBZ)
So why not increase qcl.crit to stop all the drizzle forming?
NO!
This will increase the lwp of all water clouds,
make them too bright and destroy the global radiation balance.
68