an empirical study of the prediction performance of space filling designs l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
An Empirical Study of the Prediction Performance of Space-filling Designs PowerPoint Presentation
Download Presentation
An Empirical Study of the Prediction Performance of Space-filling Designs

Loading in 2 Seconds...

play fullscreen
1 / 21

An Empirical Study of the Prediction Performance of Space-filling Designs - PowerPoint PPT Presentation


  • 217 Views
  • Uploaded on

An Empirical Study of the Prediction Performance of Space-filling Designs. Rachel T. Johnson Douglas C. Montgomery Bradley Jones. Computer Models. Widely used in engineering design Use continues to grow May have lots of variables, many responses Can have long run times

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'An Empirical Study of the Prediction Performance of Space-filling Designs' - meryl


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
an empirical study of the prediction performance of space filling designs

An Empirical Study of the Prediction Performance of Space-filling Designs

Rachel T. Johnson

Douglas C. Montgomery

Bradley Jones

computer models
Computer Models
  • Widely used in engineering design
  • Use continues to grow
  • May have lots of variables, many responses
  • Can have long run times
  • Complex output results
  • Need efficient methods for designing the experiment and analyzing the results

Johnson QPRC 2009

computer simulation types
Computer Simulation Types

Computer Simulation Models

Stochastic Simulation Models

Deterministic Simulation Models

Computational Fluid Dynamics

(CFD)

Discrete Event Simulation

(DES)

Johnson QPRC 2009

comparing designs
Comparing Designs
  • Space-filling designs compared
    • Sphere packing (SP)
    • Latin Hypercube (LH)
    • Uniform (U)
    • Gaussian Process Integrated Mean Square Error (GP IMSE)
  • Assumed surrogate model
    • Gaussian Process model
  • Comparisons based on prediction

Johnson QPRC 2009

designs
Designs

Sphere Packing

Latin Hypercube

  • Sphere Packing:
  • Johnson et al. (1990)
  • Maximizes the minimum distance between pairs of design points
  • Latin Hypercube:
  • McKay et al. (1979)
  • A random n x s matrix, in which columns are a random permutation of {1, . . ., n}
  • Uniform:
  • Feng (1980)
  • A set of n points uniformly scattered within the design space
  • GP IMSE:
  • Sacks et al. (1989)
  • Minimizes the integrated mean square error of the Gaussian Process model
  • I - Optimal:
  • Box and Draper (1963)
  • Minimizes the average prediction variance (of a linear regression model) over a design region
  • Maximum Entropy:
  • Shewry and Wynn (1987)
  • Maximizes the information contained in the distribution of a data set

Uniform

GASP IMSE

Maximum Entropy

I - Optimal

Johnson QPRC 2009

model fitting techniques
Model Fitting Techniques
  • Gaussian Process Model
    • Assumes a normal distribution
    • Interpolator based on correlation between points
    • Prediction variance calculated as

Johnson QPRC 2009

evaluation of designs for the gp model
Evaluation of Designs for the GP Model
  • Recall the prediction variance:
  • Prediction variance dependent on:
    • Design
    • Value of unknown θ
    • Sample size
    • Dimension of x
  • Design of Experiments can be used to evaluate the designs

Johnson QPRC 2009

example fds plots
Example FDS Plots

Uniform

Sphere Packing

Maximum Entropy

GP IMSE

Latin Hypercube

Johnson QPRC 2009

research questions
Research Questions
  • Does it matter in practice what design you choose?
    • Is there a dominating experimental design that performs better in terms of model fitting and prediction?
  • What is the role of sample size in experimental designs used to fit the GASP model?
    • At what point does prediction error variance and other measures of prediction performance become reasonably small with respect to N, the sample size chosen?

Johnson QPRC 2009

empirical comparison
Empirical Comparison
  • Used test functions to act as surrogates to simulation code
  • Evaluated designs based on RMSE and AAPE
  • Interested in effect of:
    • Design type
    • Sample size
    • Dimension of design
    • *Gaussian Correlation Function

Johnson QPRC 2009

comparison procedure
Comparison Procedure
  • Step 1: Choose a test function
  • Step 2: Choose a sample size and space-filling design
  • Step 3: Create the design with the number of factors equal to the number in the test function chosen in step 1) and the specifications set in step 2)
  • Step 4: Using the test function in 1) find the values that correspond to each row in the design
  • Step 5: Fit the GASP model
  • Step 6: Generate a set of 40,000 uniformly random selected points in the design space and compare the predicted value (generated by the fitted GASP model) to the actual value (generated by the test function) at these points. Compute the Root mean square error (RMSE).

Johnson QPRC 2009

test functions
Test Functions

Test Function #1

Test Function #2

Test Function #4

Test Function #3

Johnson QPRC 2009

test function 1 results
Test Function #1 Results

Johnson QPRC 2009

test function 2 results
Test Function #2 Results

Johnson QPRC 2009

test function 3 results
Test Function #3 Results

Johnson QPRC 2009

test function 2 results16
Test Function #2 Results

Johnson QPRC 2009

anova analysis
ANOVA Analysis

Johnson QPRC 2009

jackknife plots
Jackknife Plots

Test Function #2 – LHD with a sample size of 20

Test Function #2 – LHD with a sample size of 50

Johnson QPRC 2009

conclusions
Conclusions
  • No one design type is better than the others
  • Increasing sample size decreases RMSE
  • There is a strong interaction between sample size and test function type – the more complex the function the more runs required
  • Jackknife plot is an excellent indicator of a “good fit”

Johnson QPRC 2009

references
References
  • Fang, K.T. (1980). “The Uniform Design: Application of Number-Theoretic Methods in Experimental Design,” Acta Math. Appl. Sinica.3, pp. 363 – 372.
  • Johnson, M.E., Moore, L.M. and Ylvisaker, D. (1990). “Minimax and maxmin distance design,” Journal for Statistical Planning and Inference26, pp. 131 – 148.
  • McKay, N. D., Conover, W. J., Beckman, R. J. (1979). “A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code,” Technometrics21, pp. 239 – 245.
  • Sacks, J., Welch, W. J., Mitchell, T. J., and Wynn, H. P. (1989). “Design and Analysis of Computer Experiments,” Statistical Science4(4), pp. 409 – 423.
  • Shewry, M.C. and Wynn, H.P. (1987). “Maximum entropy sampling,” Journal of Applied Statistics14, pp. 898 – 914.

Johnson QPRC 2009

correlation function
Correlation Function

Gaussian Correlation Function

Cubic Correlation Function

Johnson QPRC 2009