1 / 30

# Advanced Residual Analysis Techniques for Model Selection - PowerPoint PPT Presentation

3. 2. 5. 1. Advanced Residual Analysis Techniques for Model Selection. 4 University of Rome “ Tor Vergata ”. A.Murari 1 , D.Mazon 2 , J.Vega 3 , P.Gaudio 4 , M.Gelfusa 4 , A.Grognu 5 , I.Lupelli 4 , M.Odstrcil 5. The Scientific Method and Models.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

## PowerPoint Slideshow about ' Advanced Residual Analysis Techniques for Model Selection' - dawn

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

2

5

1

Advanced Residual Analysis Techniques for Model Selection

4 University of Rome

“Tor Vergata”

A.Murari1, D.Mazon2, J.Vega3, P.Gaudio4, M.Gelfusa4, A.Grognu5, I.Lupelli4, M.Odstrcil5

• Model building is a innate faculty of human beings because it allows handling information in a much more economic way.

Model

• Model validation: process of assessing the quality of your model

• Model selection: process of selecting the best model, among many, to interpret the available data.

A “subject value” approach is advocated: both model selection and model validation are “utility” based

Model selection: introduction

• Model selection =No established and universal methodology available

• Model Falsification Criterion (MFC) :

• Estimates the most appropriate model among a set of competing and independent ones

• Based both on the accuracy and the robustness of the candidate models

• Implements a form of falsification principle more than the ‘Occam razor’

• A model is not penalised for its complexity but on the basis of its lack of robustness

MFC : THE BASIC PHYLOSOPHY

• ROBUSTNESS: a model is not penalised for its complexity but more for its lack of robustness, i.e. the fact that its estimates degrade if errors in the parameters are made

• small errors introduced on each model parameter

• study of the repercussions on the global estimates

The repercussions of the parameter errors are quantified with some sort of information theoretic quantity (Shannon entropy) calculated for the residuals

Details in the paper A.Murari et al “Preliminary discussion on a new Model Selection Criterion, based on the statistics of the residuals and the falsification principle” Conference FDT2 (Frontiers in Diagnostic Technologies)

Hypothesis: the noise is random and additive

Consequence: the residuals of a perfect model should be randomly distributed

The model with the distribution of the residuals closer to a random one is to be preferred

A random distribution of numbers (residuals) maximizes the Shannon entropy

Hypotheses

MFC : Matematics an example

• Mathematical expression of the Model Falsification Criterion :

MFC1

MFC2

• rithe absolute value of the i-th residual

• pirthe quantised probability of the i-th residual

• rpar,i calculated after varying each parameter one at time (± 10 %)

• ppar,i the quantised probability of this new residual

• npar the number of model parameters

• 1<i<n where n is the total number of experimental points

MFC : MATHEMATICS

• Mathematical expression of the Model Falsification Criterion :

A better model = smaller change in case of small errors introduced on the various parameters

A better model = smaller sum of residuals + higher entropy of residuals

The best among the candidate models is the one which presents the lowest value of the MFC indicator

• A purely numerical equation :

• exact solution + random noise of ± 10% = synthetic experimental data

• Seven models created to fit the data

• INTUITIVE CLASSIFICATION n Model 1

• Model 7

• Model 4

• Model 2

• Model 6

• Model 3

• Model 5

Error of ±10% introduced on each parameter one at time

1

Model classification obtained (classified in order of increasing MFC value)

MFC evaluated for each model

NUMERICAL TESTS: Results

• MFC AIC BIC B

• Model 1 41 -105 -549

• Model 7 106 959372

• Model 4 120 889 435

• Model 2 130 955362

• Model 6 309 1071570

• Model 3 493 1231 700

• Model 5 5043 1634 1129

• Various forms of MFC criteria seem to outperform traditional criteria in particular for extrapolation and for high levels of noise

ScalingLaws: Numerical tests

• Electron temperature required to access the H-mode of confinement in tokamak plasmas :

• Variables scanned over their respective interval (using 500 values)

• Synthetic experimental data generated by adding a random noise of ±10%

• Five models considered to test the indicator

• Bt R a n q N

• Min 2 0.8 0.2 1 2

• Max 8 2 0.7 10 8

• INTUITIVE CLASSIFICATION n

• Model 1

• Model 2

• Model 5

• Model 3

• Model 4

Error of ±10% introduced on each parameter one at time

MFC evaluated for each model

Model classification obtained (classified in order of increasing MFC value)

• Results of the MFC criterion :

• INTUITIVE CLASSIFICATION MFC value n

• Model 1 Model 2 = 2.40*106

• Model 2Model 1 = 3.21*106

• Model 5 Model 5 = 4.81*106

• Model 3 Model 3 = 1.12*107

• Model 4Model 4 = 1.22*107

• Results of the MFC criterion: since the exponent of ne is very low, at realistic noise levels the MFC realises that the models containing this quantity are prone to overfitting and that models without this parameter are more robust.

VS

VS

• small dependence from the density

• when affected by an error bigger MFC value

• not a fundamental variable

• The MFC criterion also automatically penalises the major and minor radii from the scaling laws of individual devices because they do not vary over a significant range

theoretical models variables used models generated n

ChankinBt, q, Rmodel 1

Kernel collisionless

Kernel collisional Bt, q, R, nmodel 2

Rogister

ScottBt, nmodel 3

Shaing & Crume q, R, a, nmodel 4

none Bt, q, R, a, n model 5

none Bt, q, n model 6

linear regression

• errors introduced = bounds of the 85% confidence interval

lower

upper

• lower bound central value upper bound n

• 3.05 3.25 3.44

• JET : Linear regression VS well-known theoretical models

• JET : 469 shots

• MFC n

• Model 6 4.49*104

• Model 3 4.49*104

• Model 5 2.08*105

• Model 2 2.18*105

• Model 1 3.27*105

• Model 4 7.88*105

• ASDEX : 48 shots

• MFC n

• Model 6 4265

• Model 3 4664

• Model 5 1.83*105

• Model 1 2.22*105

• Model 2 4.11*105

• Model 4 1.05*106

• CMOD : 98 shots

• MFC n

• Model 6 2.13*105

• Model 3 2.21*105

• Model 2 1.06*109

• Model 1 1.24*109

• Model 4 1.17*1010

• Model 5 1.49*1010

• JET + ASDEX + CMOD : 615 shots

• MFC n

• Model 4 2.20*105

• Model 3 2.25*105

• Model 6 2.37*105

• Model 5 2.68*105

• Model 2 2.72*105

• Model 1 2.93*105

• Summary of the best results obtained :

• JET :

• ASDEX :

• CMOD :

• All the database :

The MFC determines that Te depends only on Bt, n and q but with not the same exponents at all

Results are very different from the ones obtained with each independent tokamak

• Analyse of the best results obtained :

• Plasma radius, a and R, not evaluated as fundamental variables

• Not the same exponents at all

JET

ASDEX

CMOD

• The MFC criterion has some potential advantages compared to traditional criteria particularly in the case of scaling laws and extrapolation

• The application to the ITPA database has given some interesting results (See also talk by I.Lupelli)

• Model selection: various alternative MFC criteria are being applied to the power threshold to reach the H mode of confinement