Qualitative reverse engineering
Download
1 / 41

Qualitative - PowerPoint PPT Presentation


  • 358 Views
  • Updated On :

Qualitative Reverse Engineering. Do ri an Šuc and Ivan Bratko Artificial Intelligence Laboratory Faculty of Computer and Information Science University of Ljubljana, Slovenia. Overview. Qualitative reverse engineering: generating explanation of engineering designs in qualitative terms

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Qualitative ' - liam


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Qualitative reverse engineering

Qualitative Reverse Engineering

DorianŠuc and Ivan Bratko

Artificial Intelligence Laboratory

Faculty of Computer and Information Science

University of Ljubljana, Slovenia


Overview
Overview

  • Qualitative reverse engineering:

    • generating explanation of engineering designs

    • in qualitative terms

  • Case study: industrial crane controller

    • regression, model trees

    • qualitative trees

  • Discussion


Motivation
Motivation

  • But: incomplete documentation

    • Why did designer move from one model to another?

    • Why was a design abandoned in favor of another?

    • How a designed system works?

  • Difficulties in re-use of design libraries:

    • intuitions behind designs are not explained

Accumulated knowledge in companies: libraries of designs and corresponding simulation models


Context
Context

  • Automated explanation generation

    • knowledge sharing: making tacit knowledge available to other people

    • model re-use: for model to be used by another designer,generate explanation how model works

  • Clockwork European project:

    • computer aided engineering design

    • industrial partners: car and heating industry

    • libraries of simulation models


The problem
The problem

  • Given designed system

  • Explain:

    • how the system works?

    • ideas, intuitions behind the design

  • Approach:

    • use ML on traces of system’s behavior

    • similar to behavioural cloning (Michie et al., 90; Sammut et al., 92)


Case studies
Case studies

  • Reverse engineering of industrial crane controller, presented in this paper

  • Reverse engineering of car suspension system


Case study anti sway crane
Case study: anti-sway crane

For transportation of slabs

Designed by Czech Technical Univ.

In daily use

Crane parameters:

travel 100m

height 15m,

width 30m

80-120 tons


Anti sway crane
Anti-sway crane

Control force: F

State: cX,cV, lX,lV

?

Anti-sway controller

  • control force F

  • achieves cVDes

  • minimizes swing

desired carriage velocity cVDes




Learning problem
Learning problem

controller = ?

Given simulation model + controller explain how controller works



Learning problem2
Learning problem

cVDesF cX cV lX lV

2.0 0 0.0 0.0 0.0 0.0

2.0 24914 0.10.3 0.10.3

2.0 40188 0.4 2.1 0.4 2.1

2.0 -425 1.6 3.1 1.5 2.8

… ….. … … … …

ML


Ml methods used
ML methods used

  • Linear regression

  • Regression, model trees

  • Qualitative trees (QUIN)


Qualitative induction quin
QUalitative INduction (QUIN)

  • Learning of qualitative trees

  • Qualitative tree defines a partition of the state space into areas with common qualitative behavior

  • Qualitatively constrained functions (QCFs) in leaves:

    • generalization of monotonicity constraint M+

    • similar to qualitative proportionality predicates (QPT-Forbus, 84)


Example problem for quin
Example problem for QUIN

  • Noisy examples generated from z=x2-y2


Qualitative patterns in data
Qualitative patterns in data

x > 0 & y > 0 =>

z = M+,-(x,y)


Induced qualitative tree for function z x 2 y 2
Induced qualitative tree for functionz=x2-y2

z is monotonically increasing in its dependence on x and monotonically decreasing in its dependence on y

z is positively related to x and negatively related to y


Quin algorithm
QUIN algorithm

  • MDL to learn “most consistent” QCF

  • Learning qualitative tree (similar to ID3)

    • for every possible split, split the examples into two subsets, find the “most consistent” QCF for both subsets and select the split minimizing tree-error cost (based on MDL)

  • QUIN is heuristic improvement of this idea


Related work in learning qualitative models
Related work in learning qualitative models

  • Previous work includes:

    • Mozetič, 87 (KARDIO),

    • Coiera, 89; Hau, Coiera, 97

    • Bratko, Muggleton, Varšek, 91

    • Richards, Kraan, Kuipers, 92

    • Say, Kuru, 96

    • Kay, Rinner, Kuipers, 2000

  • Most of these systems induce QDE models, hard to apply to our problem


Reverse engineering of anti sway crane controller
Reverse engineering of anti-sway crane controller

ML from examples of controller’s execution:

  • Linear regression, regression and model trees

    (Weka-Witten&Frank,2000; M5-Quinlan, 92)

  • QUIN


Reverse engineering of anti sway crane controller1
Reverse engineering of anti-sway crane controller

ML from examples of controller’s execution:

  • Linear regression, regression and model trees

    (Weka-Witten&Frank,2000; M5-Quinlan, 92)

  • QUIN

  • Given:cVDes=2

  • an execution tracecVDes=0

  • of anti-sway controller (follow cVDes,minimize swing)

  • learn control force F = F(lX, lV, cV, cVDes)


Evaluation criteria
Evaluation criteria

Comparing the induced models:

  • suitability as explanation

  • control performance

    • criterion of controller’s success: mean abs. velocity error and mean abs. swing error bellow errors of anti-sway controller + 10%


Regression and model trees
Regression and model trees

Learning F=F (lX, lV, cV, cVDes):

  • pruning: smallest non-single leaf tree has 34 leaves

  • control performance:

    • cannot work: cVDes does not appear in any of induced trees

  • Helping the learning system:

  • derived attribute: relative carriage velocity cVRel=cV-cVDes

  • pruning: smallest non-single leaf tree has 17 leaves

  • control performance:

    • does not move(F0 when lX0), or huge swing

  • Summary:

  • complex trees, not even close to achieving the task

  • not suitable as explanation


Linear regression
Linear regression

Learning F=F (lX, lV, cV, cVDes):

  • F= 23218 cVDes -18578 cV -20319 lX + 12410 lV -4800

  • bad fit (R=0.73, RRE=69.12%)

  • unsatisfactory control performance

  • Helping the learning system (using cVRel):

  • Similar, even slightly worse results

  • Summary:

  • simple models

  • but unsatisfactory control performance and bad fit


Experiments with quin
Experiments with QUIN

Using cVRel:

accelerate

yes

no

F=M-(lX), lX decreas.

=> increase F

Small swing

=> follow cVDes

(since cVRel=cV-cVDes)

Large swing

=> reduce swing


Experiments with quin cont
Experiments with QUIN, cont.

F = F(lX, lV, cV, cVDes):

yes

no

Equivalent to previous qualitative tree!

(given the particular crane dynamics)


Control with qua li titative trees
Control with qualititative trees

  • By transforming QCFs in leaves into real-valued functions satisfying qualitative constraints

M+(X)

Randomly generated real-valued increasing functions

f+(X)

Arbitrary increasing fn. of X

  • Testing 100 transformed qual. trees for control:

    • 43 successful (according to criterion of success)



R everse engineering of car suspension system
Reverse engineering ofcar suspension system

  • Given Quarter Car Model, reverse engineer controller for semi-active car suspension

  • Matlab/Simulink model provided by CTU and Intec


Car suspension model
Car suspension model

The original controller

Black Box

The shock absorber

characteristics


Reverse engineering of car suspension system

  • Qualitative controller was induced with QUIN

  • Induced controller applied to control car model

  • Performance compared with original controller


P erformance comparison
Performance comparison

Original controller:

Qualitative controller:

Comparatively better

Comparatively worse


S ummary of approach
Summary of approach

  • Reverse engineering in qualitative terms, useful to grasp the intuition behind the design

  • Induction of qualitative trees (QUIN)

  • Qualitative-to-quantitative transformation of qualitative control strategies (involves optimisation)


Q ualitative v s quantitative models
Qualitative vs.quantitative models

  • Induced qualitative models superior as explanations (unsurprisingly)

  • Induced qualitative controllers* also perform better (surprisingly!)

    *“Qualitative controller” means:

    qualitative control strategy induced by QUIN, applied as controller after qualitative-to-quantitative transformation


W hy numerical learning fails
Why numerical learning fails?

1. Linear regression alone insufficient -

easy to see

2. But, why model trees also fail?

3. Why qualitative trees succeed when model

trees fail?


E xample from crane
Example from crane

  • M5 model tree contains leaf

    - 0.0055 < lX =< 0.0055

    Corresponding example set is very awkward for linear regression, see visualisation

  • QUIN finds better split:

    - 0.223 < lX =< 0.163

    This example set exhibits clearer qualitative pattern


E xample from crane cont
Example from crane, cont.


E xample from crane cont1
Example from crane, cont.


T entative explanation
Tentative explanation

  • QUIN can detect subtle qualitative patterns, although they are obscured by large quantitative changes

  • Regression tree learning (M5) does not find good splits, resulting in regression-unfriendly leaves


Comparison
Comparison

Qualitative trees more successful:

  • suitability as explanation

  • control performance

    Why regression trees fail?

  • Important patterns overshadowed by numerically dominant attributes

    • cVDes does not appear in regression trees!

  • Still qualitatively visible


  • ad