Advanced residual analysis techniques for model selection
Download
1 / 30

Advanced Residual Analysis Techniques for Model Selection - PowerPoint PPT Presentation


  • 81 Views
  • Uploaded on

3. 2. 5. 1. Advanced Residual Analysis Techniques for Model Selection. 4 University of Rome “ Tor Vergata ”. A.Murari 1 , D.Mazon 2 , J.Vega 3 , P.Gaudio 4 , M.Gelfusa 4 , A.Grognu 5 , I.Lupelli 4 , M.Odstrcil 5. The Scientific Method and Models.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Advanced Residual Analysis Techniques for Model Selection' - dawn


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Advanced residual analysis techniques for model selection

3

2

5

1

Advanced Residual Analysis Techniques for Model Selection

4 University of Rome

“Tor Vergata”

A.Murari1, D.Mazon2, J.Vega3, P.Gaudio4, M.Gelfusa4, A.Grognu5, I.Lupelli4, M.Odstrcil5


The scientific method and models
The Scientific Method and Models

  • Model building is a innate faculty of human beings because it allows handling information in a much more economic way.

Model

  • Model validation: process of assessing the quality of your model

  • Model selection: process of selecting the best model, among many, to interpret the available data.

A “subject value” approach is advocated: both model selection and model validation are “utility” based


Advanced residual analysis techniques for model selection

Model selection: introduction

  • Model selection =No established and universal methodology available

  • Model Falsification Criterion (MFC) :

    • Estimates the most appropriate model among a set of competing and independent ones

    • Based both on the accuracy and the robustness of the candidate models

    • Implements a form of falsification principle more than the ‘Occam razor’

    • A model is not penalised for its complexity but on the basis of its lack of robustness


Advanced residual analysis techniques for model selection

MFC : THE BASIC PHYLOSOPHY

  • ROBUSTNESS: a model is not penalised for its complexity but more for its lack of robustness, i.e. the fact that its estimates degrade if errors in the parameters are made

  • small errors introduced on each model parameter

  • study of the repercussions on the global estimates

The repercussions of the parameter errors are quantified with some sort of information theoretic quantity (Shannon entropy) calculated for the residuals

Details in the paper A.Murari et al “Preliminary discussion on a new Model Selection Criterion, based on the statistics of the residuals and the falsification principle” Conference FDT2 (Frontiers in Diagnostic Technologies)


The correlation tests method
The correlation tests method

Hypothesis: the noise is random and additive

Consequence: the residuals of a perfect model should be randomly distributed

The model with the distribution of the residuals closer to a random one is to be preferred

A random distribution of numbers (residuals) maximizes the Shannon entropy

Hypotheses


Advanced residual analysis techniques for model selection

MFC : Matematics an example

  • Mathematical expression of the Model Falsification Criterion :

MFC1

MFC2

  • rithe absolute value of the i-th residual

  • pirthe quantised probability of the i-th residual

  • rpar,i calculated after varying each parameter one at time (± 10 %)

  • ppar,i the quantised probability of this new residual

  • npar the number of model parameters

  • 1<i<n where n is the total number of experimental points


Advanced residual analysis techniques for model selection

MFC : MATHEMATICS

  • Mathematical expression of the Model Falsification Criterion :

A better model = smaller change in case of small errors introduced on the various parameters

A better model = smaller sum of residuals + higher entropy of residuals

The best among the candidate models is the one which presents the lowest value of the MFC indicator


Advanced residual analysis techniques for model selection

NUMERICAL TESTS

  • A purely numerical equation :

  • exact solution + random noise of ± 10% = synthetic experimental data

  • Seven models created to fit the data









Advanced residual analysis techniques for model selection

NUMERICAL TESTS

  • INTUITIVE CLASSIFICATION n Model 1

    • Model 7

    • Model 4

    • Model 2

    • Model 6

    • Model 3

    • Model 5

Error of ±10% introduced on each parameter one at time

1

Model classification obtained (classified in order of increasing MFC value)

MFC evaluated for each model


Advanced residual analysis techniques for model selection

NUMERICAL TESTS: Results

  • MFC AIC BIC B

    • Model 1 41 -105 -549

    • Model 7 106 959372

    • Model 4 120 889 435

    • Model 2 130 955362

    • Model 6 309 1071570

    • Model 3 493 1231 700

    • Model 5 5043 1634 1129

  • Various forms of MFC criteria seem to outperform traditional criteria in particular for extrapolation and for high levels of noise


Advanced residual analysis techniques for model selection

ScalingLaws: Numerical tests

  • Electron temperature required to access the H-mode of confinement in tokamak plasmas :

  • Variables scanned over their respective interval (using 500 values)

  • Synthetic experimental data generated by adding a random noise of ±10%

  • Five models considered to test the indicator

  • Bt R a n q N

    • Min 2 0.8 0.2 1 2

    • Max 8 2 0.7 10 8


Advanced residual analysis techniques for model selection

SCALING LAWS

  • INTUITIVE CLASSIFICATION n

    • Model 1

    • Model 2

    • Model 5

    • Model 3

    • Model 4

Error of ±10% introduced on each parameter one at time

MFC evaluated for each model

Model classification obtained (classified in order of increasing MFC value)


Advanced residual analysis techniques for model selection

MFC Classification

  • Results of the MFC criterion :

  • INTUITIVE CLASSIFICATION MFC value n

    • Model 1 Model 2 = 2.40*106

    • Model 2Model 1 = 3.21*106

    • Model 5 Model 5 = 4.81*106

    • Model 3 Model 3 = 1.12*107

    • Model 4Model 4 = 1.22*107

  • Results of the MFC criterion: since the exponent of ne is very low, at realistic noise levels the MFC realises that the models containing this quantity are prone to overfitting and that models without this parameter are more robust.


Advanced residual analysis techniques for model selection

Residuals

VS

VS

  • small dependence from the density

  • when affected by an error bigger MFC value

  • not a fundamental variable

  • The MFC criterion also automatically penalises the major and minor radii from the scaling laws of individual devices because they do not vary over a significant range


Advanced residual analysis techniques for model selection

ITPA DATABASE

theoretical models variables used models generated n

ChankinBt, q, Rmodel 1

Kernel collisionless

Kernel collisional Bt, q, R, nmodel 2

Rogister

ScottBt, nmodel 3

Shaing & Crume q, R, a, nmodel 4

none Bt, q, R, a, n model 5

none Bt, q, n model 6

linear regression

  • errors introduced = bounds of the 85% confidence interval

lower

upper

  • lower bound central value upper bound n

  • 3.05 3.25 3.44


Advanced residual analysis techniques for model selection

ITPA DATABASE

  • JET : Linear regression VS well-known theoretical models


Advanced residual analysis techniques for model selection

ITPA DATABASE

  • JET : 469 shots

  • MFC n

    • Model 6 4.49*104

    • Model 3 4.49*104

    • Model 5 2.08*105

    • Model 2 2.18*105

    • Model 1 3.27*105

    • Model 4 7.88*105


Advanced residual analysis techniques for model selection

ITPA DATABASE

  • ASDEX : 48 shots

  • MFC n

    • Model 6 4265

    • Model 3 4664

    • Model 5 1.83*105

    • Model 1 2.22*105

    • Model 2 4.11*105

    • Model 4 1.05*106


Advanced residual analysis techniques for model selection

ITPA DATABASE

  • CMOD : 98 shots

  • MFC n

    • Model 6 2.13*105

    • Model 3 2.21*105

    • Model 2 1.06*109

    • Model 1 1.24*109

    • Model 4 1.17*1010

    • Model 5 1.49*1010


Advanced residual analysis techniques for model selection

ITPA DATABASE

  • JET + ASDEX + CMOD : 615 shots

  • MFC n

    • Model 4 2.20*105

    • Model 3 2.25*105

    • Model 6 2.37*105

    • Model 5 2.68*105

    • Model 2 2.72*105

    • Model 1 2.93*105


Advanced residual analysis techniques for model selection

ITPA DATABASE: Summary

  • Summary of the best results obtained :

  • JET :

  • ASDEX :

  • CMOD :

  • All the database :

The MFC determines that Te depends only on Bt, n and q but with not the same exponents at all

Results are very different from the ones obtained with each independent tokamak


Advanced residual analysis techniques for model selection

ITPA: Comparison Exponents

  • Analyse of the best results obtained :

  • Plasma radius, a and R, not evaluated as fundamental variables

  • Not the same exponents at all

JET

ASDEX

CMOD


Summary and future developments
Summary and Future developments

  • The MFC criterion has some potential advantages compared to traditional criteria particularly in the case of scaling laws and extrapolation

  • The application to the ITPA database has given some interesting results (See also talk by I.Lupelli)

  • Model selection: various alternative MFC criteria are being applied to the power threshold to reach the H mode of confinement