137 Views

Download Presentation
## A Core Course on Modeling

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -

**A Core Course on Modeling**Week 7-A working model – and then? Contents • The Need for Interpretation • Approach • Criteria for Modeling • Genericity • Scalability • Specialization • Population • Convincingness • Distinctiveness • Surprise • Impact • Criteria for modeling and purposes • Summary • References to lecture notes + book • References to quiz-questions and homework assignments (lecture notes)**A Core Course on Modeling**Week 7-A working model – and then? The Need for Interpretation 2 A model outcome in itself cannot solve a problem. Interpretation is necessary final step. How to achieve a good-quality interpretation? Model makers in 19th century London predicted 3 meter rising of horse manure levels by 1950 Model makers in 21st century predict 0.5 meter rising of sea levels by 2050**A Core Course on Modeling**Week 7-A working model – and then? Approach 3 We want to say that one model (including the interpretation of the outcomes) is better than another. ‘Better’ means: with respect to fulfilling the model’s purpose. Therefore we need criteria to compare. The first step to quality assessment is the ability to compare things that are utterly different. Comparison invariably needs criteria.**A Core Course on Modeling**Week 7-A working model – and then? Approach 4 So: we will try to formulate criteria for comparing problem solutions. Problem solution = model outcome + interpretation As yet, we have no means to compare ‘quality’ of interpretations can be assessed (compared) with the tools of Chapter 6 (sensitivity analysis, uncertainty analysis)**A Core Course on Modeling**Week 7-A working model – and then? Criteria for Modeling 5 When formulating criteria for comparing problem solutions, the main problem is the immense variety of problems asking for modeling. Our criteria need to apply to all of them. The large variety of possible modeling processes is a complication for finding a single generic set of criteria.**A Core Course on Modeling**Week 7-A working model – and then? Criteria for Modeling 6 The standard approach to finding a complete set of ‘things’, is to construct a complete taxonomy. A complete taxonomy requires enumerable properties: properties where all possible values can be enumerated. The quarto-game is based on 4 enumerable properties: color(black,white), size(tall,short), shape(round,square), interior(hollow,solid). Rows of 4 must be formed of pieces which all have a common value for one property**A Core Course on Modeling**Week 7-A working model – and then? Criteria for Modeling 7 Property 1: Does the criterion regard the begin (definition stage of the modeling process) or the end (conclusion stage)? • Example of criterion related to definition: what is the scale of the problem? (scalability) • Example of criterion related to conclusion: how much depends on it? (impact) The input-side of a problem relates to stage 1 (problem definition); the output-side relates to stage 5 (conclusion: presentation and interpretation). This photo seems to suggest that input and output are relative notions …**A Core Course on Modeling**the modeled system: part of the world, including the model and some stakeholders the model: the present painting Week 7-A working model – and then? Criteria for Modeling 8 stakeholder2: the modeler Property 2: Does the criterion regard the modeled system or the (stakeholders + context) ? stakeholder1: the customer • Example of criterion related to modeled system: what is the scale of the problem? (scalability) • Example of criterion related to stakeholders: how much knowledge do the stakeholders need? (specialization) In this famous Velasquez painting, the stakeholders and the modeled system are deliberately confused. In modeling practice, the distinction between subject (the modeled system) and object (the stakeholder(s)) are clearly distinct. Hopefully.**A Core Course on Modeling**Week 7-A working model – and then? Criteria for Modeling 9 Property 3: Does the criterion regard qualitative or quantitative aspects ? • Example of criterion related to quantitative aspect: what is the scale of the problem? (scalability) • Example of criterion related to qualitative aspect: how different can modeled systems be? (genericity) The distinction quantitative – qualitative is sometimes associated to the closed – open distinction. ‘Closed’ means that all possible outcomes can be foreseen or enumerated; ‘open’ means that this is not possible.**define/ conclude**modeled system / stakeholders qualitative / quantitative criterion define modeled system qualitative define modeled system quantitative define stakeholders qualitative define stakeholders quantitative conclude modeled system qualitative conclude modeled system quantitative conclude stakeholders qualitative conclude stakeholders quantitative A Core Course on Modeling Week 7-A working model – and then? Criteria for Modeling 10 Combinations genericity scalability speciali-zation population convin-cingness The 3 distinctions together form a imaginary cube, each corner corresponds to one of the criteria. distinctive-ness surprise impact**A Core Course on Modeling**Week 7-A working model – and then? Genericity 11 Genericity: to which extent is the approach capable to handle various types of modeled systems / purposes? Examples: Physics: involve more effects, principles and interactions (flow, continuum mechanics) Mathematics: abstraction Computer Science: generic tools (graphs, databases, compilers, XML, ...)**A Core Course on Modeling**Week 7-A working model – and then? Genericity 12 Genericity: example Given an object of some shape and density ; we want to know its mass. • If it is a cube with side p: m=a3 • if it is a rectangular block with height h: m=h*Areatop • if it is a truncated pyramid: m= h*(Areatop + (Areatop*Areabottom) + Areabottom)/3 (works for all these shapes!) Above models are increasingly generic; each can deal with the simpler cases as well. How heavy is the Borobodur? (E.g., in the context of risks for landslides etc.: the Borobodur is one of the heaviest temples in the world, restauration in the 1980-ies required partial reinforcement of the hill on which it stands to prevent landslides). Only formula (‘model’) 3 is sufficiently generic**A Core Course on Modeling**Week 7-A working model – and then? Scalability 13 Scalability: to which extent can some characteristic dimensions of the problem increase, where the model still functions? Scale: n (=number of elements, states, records, time steps, …) Performance: O(1) – solving effort (SE) does not depend on n O(log n) – SE hardly depends on n O(n) – SE proportional to n O(n log n) – SE slightly worse than proportional O(np) – SE polynomial O(2n), O(n!), … – SE worse than polynomial Many problems contain some characteristic dimension (e.g., n=number of boxes). If n grows, there will come a point where the model fails when trying to solving the problem. The larger n can be, the better the scalability of the solution.**A Core Course on Modeling**Week 7-A working model – and then? Scalability 14 Scalability: to which extent can some characteristic dimensions of the problem increase, where the model still functions? Scale: n (=number of elements, states, records, time steps, …) Performance: O(1) – solving effort (SE) does not depend on n O(log n) – SE hardly depends on n O(n) – SE proportional to n O(n log n) – SE slightly worse than proportional O(np) – SE polynomial O(2n), O(n!), … – SE worse than polynomial O(1) n The size of Google’s data repository, according to http://www. worldwide websize.com/ , varied between 15 and 55 billion webpages in less than 4 months. Still, the response time stays near constant due to sophisticated indexing strategies**A Core Course on Modeling**Week 7-A working model – and then? Scalability 15 Scalability: to which extent can some characteristic dimensions of the problem increase, where the model still functions? Scale: n (=number of elements, states, records, time steps, …) Performance: O(1) – solving effort (SE) does not depend on n O(log n) – SE hardly depends on n O(n) – SE proportional to n O(n log n) – SE slightly worse than proportional O(np) – SE polynomial O(2n), O(n!), … – SE worse than polynomial O(1) n To tell if there is a needle in the haystack is easy, provided the hay is sorted: check the halfway element. If this is larger than the needle, discard the upper half, otherwise discard the lower half – and checking the halfway element in the remaining part until either found or not found.**A Core Course on Modeling**Week 7-A working model – and then? Scalability 16 Scalability: to which extent can some characteristic dimensions of the problem increase, where the model still functions? Scale: n (=number of elements, states, records, time steps, …) Performance: O(1) – solving effort (SE) does not depend on n O(log n) – SE hardly depends on n O(n) – SE proportional to n O(n log n) – SE slightly worse than proportional O(np) – SE polynomial O(2n), O(n!), … – SE worse than polynomial O(1) n For many tasks, the effort of doing n things is proportional to n. If one chicken lays three eggs per week, 5 chicken lay 15 eggs per week. This does not hold for everything, though … climbing n flights of stairs may take more effort than n the effort of one. But computers don’t get tired, and many computer tasks scale similarly to chicken laying eggs.**A Core Course on Modeling**Week 7-A working model – and then? Scalability 17 Scalability: to which extent can some characteristic dimensions of the problem increase, where the model still functions? Scale: n (=number of elements, states, records, time steps, …) Performance: O(1) – solving effort (SE) does not depend on n O(log n) – SE hardly depends on n O(n) – SE proportional to n O(n log n) – SE slightly worse than proportional O(np) – SE polynomial O(2n), O(n!), … – SE worse than polynomial O(1) n Sorting an array of unsorted things can be done in O(n log n). Military go faster, but they work in parallel.**A Core Course on Modeling**Week 7-A working model – and then? Scalability 18 Scalability: to which extent can some characteristic dimensions of the problem increase, where the model still functions? Scale: n (=number of elements, states, records, time steps, …) Performance: O(1) – solving effort (SE) does not depend on n O(log n) – SE hardly depends on n O(n) – SE proportional to n O(n log n) – SE slightly worse than proportional O(np) – SE polynomial O(2n), O(n!), … – SE worse than polynomial O(1) n Straightfowardly sweeping a matrix takes O(n3), but faster methods exist that bring this down to something like O(n2.5). The difference is noticable only for n sufficiently large.**A Core Course on Modeling**Week 7-A working model – and then? Scalability 19 Scalability: to which extent can some characteristic dimensions of the problem increase, where the model still functions? Scale: n (=number of elements, states, records, time steps, …) Performance: O(1) – solving effort (SE) does not depend on n O(log n) – SE hardly depends on n O(n) – SE proportional to n O(n log n) – SE slightly worse than proportional O(np) – SE polynomial O(2n), O(n!), … – SE worse than polynomial O(1) n The travelling salesman problem asks for the shortest route connecting n places. Exact solution is believed impossible in O(np) but good approximations sometimes take only O(np)**A Core Course on Modeling**Week 7-A working model – and then? Specialization 20 Specialization: to which extent does the model / model outcomes requires specialized knowledge on behalf of the problem owner? How much control should the modeler give the problem owner over • validity of assumptions, • regime of application, • use of results? A model, intended for specialists has to face the challenge of withstanding expert scrutiny. On the other hand, a model intended for laymen has to be robust against unintended misuse.**A Core Course on Modeling**Week 7-A working model – and then? Specialization 21 Specialization: to which extent does the model / model outcomes requires specialized knowledge on behalf of the problem owner? • Invest in presentation • graph style (line plot, scatterplot, polar plot, barchart, pie chart, 3D surface, …) • labels, captions, scales (zero, lin/log, units, major / minor scale lines …) • how to represent uncertainty? • which plots to combine in a single graph? The appropriate visualization style is the answer to the question: what message should this graph convey?**A Core Course on Modeling**Week 7-A working model – and then? Specialization 22 Specialization: to which extent does the model / model outcomes requires specialized knowledge on behalf of the problem owner? • When in doubt, be conservative A modeler resembles a blindfolded torreador: without full knowledge, consequences of the modeled system being mischievous may be quite harmful. For good reasons, the jackass adagium reads: ‘don’t try this at home!’**A Core Course on Modeling**Week 7-A working model – and then? Specialization 23 Specialization: to which extent does the model / model outcomes requires specialized knowledge on behalf of the problem owner? • Be warned for biased users • give a full and detailed account of as much as possible assumptions and estimates • insist on the report being indivisible • refrain from easy-to-misapprehend results • Get a second opinion Users having personal interests in model outcomes may not always take a balanced stand.**A Core Course on Modeling**Week 7-A working model – and then? Specialization 24 Specialization: to which extent does the model / model outcomes requires specialized knowledge on behalf of the problem owner? • Take care for self-fulfilling and self-denying prophecies • wicked problems: the problem solution post-hoc modifies the problem statement Both legendary prophet Michael Nostradamus (glass bowl?) and present-day opinion investigators (black ballot box?) relate to the working principle of self-fulfilling and self-denying prophecies.**A Core Course on Modeling**Week 7-A working model – and then? Population 25 Population: what size of intended audience does the model address? • Large size: low level of specialization • Large size: no option for bi-directional communication • Large size: consider interactive model (example: ‘stemwijzer’) In the 1999, Wachowski-brothers’ movie ‘the Matrix’, a simulation’s intended audience size is stretched so as to encompass the entire world’s population – with the extra that nearly everybody is ignorant of the fact.**A Core Course on Modeling**Week 7-A working model – and then? Convincingness 26 Convincingness: a model is more convincing if it contains fewer and/or less implausible assumptions. • Are assumptions logically deducible from other, less problematic assumptions? • If not, are there any first-principle ‘laws’ to back up assumptions (example: if this component behaves as a lever, we can use the physical laws of torque in a lever)? • If not, can we construct a simplified model system (example: tidal motions as a lumped system)with outcomes that can be empirically verified? • If not, can we get support from comparison to empirical observations on an existing system (example: behavior of visitors in shopping malls to predict buying behavior)? • If not, is there at least any argument for consistency of the assumptions? (example: price elasticity, ‘fun’ in the taxi model) In the early days of photography, a photo (as prototypical empirical evidence) counted as proof. As this no longer holds, we need to re-think what constitutes ‘plausible evidence’.**A Core Course on Modeling**Week 7-A working model – and then? Convincingness 27 Convincingness: a model is more convincing if it contains fewer implausible assumptions. • Convincingness in the absence of ‘laws’: • conceive a plausible formal model system to which laws do apply (e.g., replace the world + oceans by a homogenous ball covered by a sheet of water). The measure of correspondence between these two is left out of the discussion; • convincingness relies on the agreement with empirical data. Tides are caused by subtle interplay between gravity, water (viscosity vs. inertia), and the complex geometry of the earth’s sea floors. A (glass box) model to predict tidal motions contains none of these, but a hypothetical ‘sheet of water on a perfect sphere’ instead.**A Core Course on Modeling**Week 7-A working model – and then? Convincingness 28 Convincingness: a model is more convincing if it contains fewer implausible assumptions. • Convincingness in the absence of a formal model system and/or no empirical validation: • look for an empirical model system • physical: water tank, wind tunnel • economical, social: comparative populations, historical survey data • biological: the guinea pig! Scale models are physical replica of the modeled system, usually at a smaller scale. Applications include windtunnel (flow, drag) and hydromechanical investigations, but also scale models as used in architecture and urban studies. These are examples of empirical model systems. BTW: J. Swift’s story seems to relate to a scale model; it is actually a social model.**A Core Course on Modeling**Week 7-A working model – and then? Distinctiveness 29 Distinctiveness: a model has higher distinctiveness if it allows distinction between alternatives that are more similar. • Most purposes ask for some form of distinctiveness. Examples: • prediction 1: something is correctly predicted to happen at time T1 rather than T2; how close can T1 and T2 be? • explanation: Q1 causes P1 and not P2; Q2 causes P2 and not P1. How close can Q1 vs. Q2, or P1 vs. P2 be so that the explanation holds? • optimization: input P1 yields output Q1, and input P2 yields output Q2, where Q1 correctly is found to dominate Q2. How close can P1 and P2 be? Many modeling purposes boil down to the ability of distinguishing between two alternatives that are as similar as possible. This relates to accuracy and precision, but also to non-numerical purposes.**A Core Course on Modeling**Week 7-A working model – and then? Distinctiveness 30 Distinctiveness: a model has higher distinctiveness if it allows distinction between alternatives that are more similar. This hinges on the notion of ‘distance’, explained in week 6. The quantitative measure for distinctiveness is usually a similarity measure: the more similar A and A’ whereas the model can successfully distinguish them, the better its distinctiveness.**A Core Course on Modeling**Week 7-A working model – and then? Distinctiveness 31 Distinctiveness: a model has higher distinctiveness if it allows distinction between alternatives that are more similar. Distinctiveness relates to two common types of errors: • false positive (conclude X where there is no X) • false negative (don’t conclude X where there is X) Successful spreading of false money is an example of false negative: the counterfeit goes unnoticed. Coin operated vending machines have to safeguard against both false positives and false negatives; the consequences of the two types of errors being very different. This is true for most models where distinctiveness plays a role.**A Core Course on Modeling**Week 7-A working model – and then? Surprise 32 The potential for surprise of a model outcome is the extent to which it may bring unforeseen new ideas. There are Open and closed spaces of outcomes. Examples of closed outcomes: • A model computing the probability of X can only produce a number between 0 and 1. • A model verifying Y can only produce ´true´ or ´false´. A widespread opinion is, that computations cannot ‘output more than you put in’. This underestimates computers’ scope of computation, or overestimates humans’ scope of anticipation.**A Core Course on Modeling**Week 7-A working model – and then? Surprise 33 The potential for surprise of a model outcome is the extent to which it may bring unforeseen new ideas. Examples of open outcomes: • result of using ontologies • outcome of evolutionary algorithms • outcome of PCA, abstraction, … Common feature: output space has size O(2n) whereas input space has size O(n) Evolutionary programming, applied to shape design. Given a comprehensible genotype representation, the space of phenotypes is virtually infinity. Xai Xu et. al., Special Issue of SIGGRAPH, Vol. 31, No. 4, pp. 57:1-57:10, 2012**A Core Course on Modeling**Week 7-A working model – and then? Impact 34 The potential for impact of a model is the extent to which the model outcome can affect the stakeholders (either beneficially or adversely). Two perspectives: • prestige and profit: the more impact the better • risk and responsability: the less impact the better ‘Impact’ is a double-edged sword: some modelers attempt to increase the impact of their models whereas other modellers prefer to play safe and seek modeling with less impact.**A Core Course on Modeling**Week 7-A working model – and then? Impact 35 The potential for impact of a model is the extent to which the model outcome can affect the stakeholders. Intended impact can be capitalized: r1=revenues in absence of model outcome; r2=revenues with model outcome present; c1=cost of ownership in absence of model outcome; c2=cost of ownership with model outcome present; =((r2-r1)-(c2-c1))/(|r2-r1|+|c2-c1|) • 0 ||1: magnitude of impact • If 0 1: positive impact (e.g., models for optimization); • if -1 0 negative impact (e.g., risk prediction or analysis) To assign a quantitative value to a model outcome we must assume some market mechanism where we identify revenues and costs of ownership, both in the absence of the model outcomes and in their presence.**A Core Course on Modeling**Week 7-A working model – and then? Impact 36 The potential for impact of a model is the extent to which the model outcome can affect the stakeholders. Adverted impact can be quantified or capitalized: C=the estimated chance (per time interval) of an incident (=an erroneous model outcome, e.g. a type-I or type-II error) V=the estimated value loss per incident Both C and V can serve to quantify the negative impact of a model. Their product CV has dimension of money / time and can be added to c2 in the formula for .**A Core Course on Modeling**Week 7-A working model – and then? Impact 37 The potential for impact of a model is the extent to which the model outcome can affect the stakeholders. The impact of a model should balance with its reliability.**A Core Course on Modeling**Week 7-A working model – and then? Criteria for modeling and purposes 38 Not each of the eight criteria is equally relevant for all purposes. Given the purpose of the model, the modeler should identify the most relevant criteria to seek improvement when necessary. For instance: … (see next sheets)**A Core Course on Modeling**Week 7-A working model – and then? Criteria for modeling and purposes 39 Not each of the eight criteria is equally relevant for all purposes. prediction (1, 2): convincingness, distinctiveness, impact An almanac is a model to serve the purpose of predictions (both 1 and 2). Its impact is rather large (given the long historical record of almanac-use), but the distinctiveness is small, and its convincingness decreases over time.**A Core Course on Modeling**Week 7-A working model – and then? Criteria for modeling and purposes 40 Not each of the eight criteria is equally relevant for all purposes. compression: scalability, population, distinctiveness Printing is a form of compression (transferring written text into printed text) with huge population and distinctiveness; by present day’s standards, the scalability is not impressive**A Core Course on Modeling**Week 7-A working model – and then? Criteria for modeling and purposes 41 Not each of the eight criteria is equally relevant for all purposes. inspiration: surprise All the world’s religious traditions acknowledge holy scriptures of some sort as primary source of inspiration. Their potential for surprise is sometimes rather limited.**A Core Course on Modeling**Week 7-A working model – and then? Criteria for modeling and purposes 42 Not each of the eight criteria is equally relevant for all purposes. unification: genericity, convincingness, surprise In alchemy, the pre-scientific forerunner of both chemistry and psychology, unification of conflicting properties was one of the leading purposes. In modern times, scores on all three criteria are low – when seen through modern eyes.**A Core Course on Modeling**Week 7-A working model – and then? Criteria for modeling and purposes 43 Not each of the eight criteria is equally relevant for all purposes. abstraction: distinctiveness, convincingness, surprise The term ‘abstraction’ in modern art no longer defines a relation between a modeled system and a model; rather, the absence of any modeled system is the defining feature for true abstract art. The potential for surprise, however, sometimes still applies.**A Core Course on Modeling**Week 7-A working model – and then? Criteria for modeling and purposes 44 Not each of the eight criteria is equally relevant for all purposes. verification: scalability, convincingness, impact Biting a golden coin is the proverbial way to verify its authenticity. Scalability and convincingness are probably low, but if the coin contains chocolate, or if the biter has metal inlays, the impact may be quite considerable.**A Core Course on Modeling**Week 7-A working model – and then? Criteria for modeling and purposes 45 Not each of the eight criteria is equally relevant for all purposes. exploration: genericity, surprise Few models for exploration will have a larger potential of surprise than the Mars exploration robot.**A Core Course on Modeling**Week 7-A working model – and then? Criteria for modeling and purposes 46 Not each of the eight criteria is equally relevant for all purposes. decision: convincingness, distinctiveness, impact The Delphi oracle was a decision-making model that derived its convincingness from its deliberately low distinctiveness – thereby being known for its major impact.**A Core Course on Modeling**Week 7-A working model – and then? Criteria for modeling and purposes 47 Not each of the eight criteria is equally relevant for all purposes. optimization: genericity, scalability, convincingness, impact Of all generic optimization models for continuous systems, hill climbing is perhaps one of the most generic ones. It often suffers from scalability, though – even when implemented by modern, automated devices.**A Core Course on Modeling**Week 7-A working model – and then? Criteria for modeling and purposes 48 Not each of the eight criteria is equally relevant for all purposes. specification: genericity, distinctiveness Sometimes the distinction between a specification model (blueprint) and a realization is somewhat blurred.**A Core Course on Modeling**Week 7-A working model – and then? Criteria for modeling and purposes 49 Not each of the eight criteria is equally relevant for all purposes. realization: genericity, distinctiveness, impact Realization …**A Core Course on Modeling**Week 7-A working model – and then? Criteria for modeling and purposes 50 Not each of the eight criteria is equally relevant for all purposes. steering & control: distinctiveness The prototypical stick-and-carrot steering, with the stick tied to the donkey’s back is an example of open-loop control without a set-point.