uncertainty analysis using gem sa l.
Skip this Video
Loading SlideShow in 5 Seconds..
Uncertainty Analysis Using GEM-SA PowerPoint Presentation
Download Presentation
Uncertainty Analysis Using GEM-SA

Loading in 2 Seconds...

play fullscreen
1 / 31

Uncertainty Analysis Using GEM-SA - PowerPoint PPT Presentation

  • Uploaded on

Uncertainty Analysis Using GEM-SA. Outline. Setting up the project Running a simple analysis Exercise More complex analyses. Setting up the project. Create a new project. Select Project -> New, or click toolbar icon. Project dialog appears We’ll specify the data files first. Files.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Uncertainty Analysis Using GEM-SA' - Rita

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
  • Setting up the project
  • Running a simple analysis
  • Exercise
  • More complex analyses

GEM-SA course - session 4

create a new project
Create a new project
  • Select Project -> New, or click toolbar icon
  • Project dialog appears
  • We’ll specify the data files first

GEM-SA course - session 4

  • Using “Browse” buttons, select input and output files
  • The “Inputs” file contains one column for each parameter and one row for each model training run (the design)
  • The “Outputs” file contains the outputs from those runs (one column, in this example)

GEM-SA course - session 4

our example
Our example
  • We’ll use the example “model1” in the GEM-SA DEMO DATA directory
  • This example is based on a vegetation model with 7 inputs
  • The model has 16 outputs, but for the present we will consider output 4
    • June monthly GPP

GEM-SA course - session 4

number of inputs
Number of inputs
  • Click on Options tab
  • Select number of inputs using
  • Or click “From Inputs File”

GEM-SA course - session 4

define input names
Define input names
  • Click on “Names …”
  • The “Input parameter names” dialog opens
  • Enter parameter names
  • Click “OK”

GEM-SA course - session 4

complete the project
Complete the project
  • We will leave all other settings at their default values for now
  • Click “OK”
  • The Input Parameter Ranges window appears

GEM-SA course - session 4

close and save project
Close and save project
  • Click “Defaults from input ranges” button
  • Click “OK”
  • Select Project -> Save
    • Or click toolbar icon
  • Choose a name and click “Save”

GEM-SA course - session 4

build the emulator
Build the emulator
  • Click to build the emulator
  • A lot of things now start to happen!
    • The log window at the bottom starts to record various bits of information
    • A little window appears showing progress of minimisation of the roughness parameter estimation criterion
    • The “Main Effects” tab is selected, in which several graphs are drawn
      • Progress bar at the bottom

GEM-SA course - session 4

focus on the log window
Focus on the log window
  • The “Main Effects” and “Sensitivity Analysis” tabs are concerned with SA, and will be considered in the next session
    • We are interested just now simply in Uncertainty Analysis (UA)
  • The “Output Summary” tab contains all we need and more
  • But the key things can be seen more simply in the log window at the bottom
    • Diagnostics of the emulator build
    • The basic uncertainty analysis results

GEM-SA course - session 4

emulation diagnostics
Emulation diagnostics
  • Note where the log window reports …
  • The first line says roughness parameters have been estimated by the simplest method
  • The values of these indicate how non-linear the effect of each input parameter is
    • Note the high value for input 4 (MO)

Estimating emulator parameters by maximising probability distribution...

maximised posterior for emulator parameters: precision = sigma-squared = 0.342826, roughness = 0.217456 0.0699709 0.191557 16.9933 0.599439 0.459675 1.01559

GEM-SA course - session 4

uncertainty analysis mean
Uncertainty analysis – mean
  • Below this, the log reports
  • So the best estimate of the output (June GPP) is 24.1 (mol C/m2)
    • This is averaged over the uncertainty in the 7 inputs
      • Better than just fixing inputs at best estimates
    • There is an emulation standard error of 0.062 in this figure

Estimate of mean output is 24.145, with variance 0.00388252

GEM-SA course - session 4

uncertainty analysis variance
Uncertainty analysis – variance
  • The final line of the log is
  • This shows the uncertainty in the model output that is induced by input uncertainties
    • The variance is 73.9
    • Equal to a standard deviation of 8.6
    • So although the best estimate of the output is 24.1, the uncertainty in inputs means it could easily be as low as 16 or as high as 33

Estimate of total output variance = 73.9033

GEM-SA course - session 4

a small change
A small change
  • Run the same model with Output 11 instead of Output 4
  • Calculate the coefficient of variation (CV) for this output
    • NB: the CV is defined as the standard deviation divided by the mean

GEM-SA course - session 4

input distributions
Input distributions
  • A normal (gaussian) distribution is generally a more realistic representation of uncertainty
    • Range unbounded
    • More probability in the middle
  • Default is to assume the uncertainty in each input is represented by a uniform distribution
    • Range determined by the range of values found in the input file or separately input

GEM-SA course - session 4

changing input distributions
Changing input distributions
  • Reopen Project dialog by Project -> Edit … or clicking on
  • Select Options tab
  • Click All unknown, product normal
  • Then OK
  • A new dialog opens to specify means and variances

GEM-SA course - session 4

model 1 example











































Model 1 example
  • Uniform distributions from input ranges
  • Normal distributions to match
    • Range about 4 std deviations
  • Except for MO
    • Narrower distribution

GEM-SA course - session 4

effect on ua
Effect on UA
  • After running the revised model, we see:
    • It runs faster, with no need to rebuild the emulator
    • The mean is changed a little and variance is halved

The emulator fit is unchanged

Estimate of mean output is 26.2698, with variance 0.00784475

Estimate of total output variance = 38.1319

GEM-SA course - session 4

reducing mo uncertainty further
Reducing MO uncertainty further
  • If we reduce the variance of MO even more, to 49:
    • UA mean changes a little more and variance reduces again
    • Notice also how the emulation uncertainty has increased (0.004 for uniform)
    • This is because the design points cover the new ranges less thoroughly

Estimate of mean output is 26.3899, with variance 0.0108792

Estimate of total output variance = 27.1335

GEM-SA course - session 4

a homework exercise
A homework exercise
  • What happens if we reduce the uncertainty in MO to zero?
  • Two ways to do this
    • Literally set variance to zero
    • Select “Some known, rest product normal” on Project dialog, check the tick box for MO in the mean and variance dialog
  • What changes do you see in the UA?

GEM-SA course - session 4

cross validation
  • Reopen the Project dialog and select the Options tab
  • Look at the bottom menu box, labelled “Cross-validation”
  • There are 3 options
    • None
    • Leave-one-out
    • Leave final 20% out
  • CV is a way of checking the emulator fit
    • Default is None because CV takes time

GEM-SA course - session 4

leave one out cv

Close to 1

Leave-one-out CV
  • After estimating roughness and other parameters, GEM predicts each training run point using only the remaining n-1 points
  • Results appear in log window

Cross Validation Root Mean-Squared Error = 0.907869

Cross Validation Root Mean-Squared Relative Error = 4.34773 percent

Cross Validation Root Mean-Squared Standardised Error = 1.15273

Largest standardised error is 4.32425 for data point 61

Cross Validation variances range from 0.18814 to 3.92191

Written cross-validation means to file cvpredmeans.txt

Written cross-validation variances to file cvpredvars.txt

(Model 1, output 4, uniform inputs)

GEM-SA course - session 4

leave final 20 out cv
Leave final 20% out CV
  • This is an even better check, because it tests the emulator on data that have not been used in any way to build it
  • Emulator is built on first 80% of data and used to predict last 20%
  • Standardised error a bit bigger
    • But not bad for just 24 runs predicted

Cross Validation Root Mean-Squared Error = 1.46954

Cross Validation Root Mean-Squared Relative Error = 7.4922 percent

Cross Validation Root Mean-Squared Standardised Error = 1.73675

Largest standardised error is 5.05527 for data point 22

Cross Validation variances range from 0.277304 to 4.886

GEM-SA course - session 4

output summary tab
Output Summary tab
  • The “Output Summary” tab presents all of the key results in a single list
  • Tidier than searching for the details in the log window
    • Although the log window actually has more information
    • Can print using

GEM-SA course - session 4

other options
Other options
  • There are various other options associated with the emulator building that we have not dealt with
    • See built in help facility for explanations
    • Also slides at the end of session 3
  • But we’ve done the main things that should be considered in practice
  • And it’s enough to be going on with!

GEM-SA course - session 4

when it all goes wrong
When it all goes wrong
  • How do we know when the emulator is not working?
    • Large roughness parameters
      • Especially ones hitting the limit of 99
    • Large emulation variance on UA mean
    • Poor CV standardised prediction error
      • Especially when some are extremely large
  • In such cases, see if a larger training set helps
    • Other ideas like transforming output scale
  • A suite of diagnostics is being developed in MUCM
    • See Bastos and O’Hagan on my website
      • http://tonyohagan.co.uk/academic/pub.html
    • Not implemented in GEM-SA yet

GEM-SA course - session 4