qualitative reverse engineering n.
Skip this Video
Loading SlideShow in 5 Seconds..
Qualitative Reverse Engineering PowerPoint Presentation
Download Presentation
Qualitative Reverse Engineering

Loading in 2 Seconds...

play fullscreen
1 / 41

Qualitative Reverse Engineering - PowerPoint PPT Presentation

  • Uploaded on

Qualitative Reverse Engineering. Do ri an Å uc and Ivan Bratko Artificial Intelligence Laboratory Faculty of Computer and Information Science University of Ljubljana, Slovenia. Overview. Qualitative reverse engineering: generating explanation of engineering designs in qualitative terms

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Qualitative Reverse Engineering' - liam

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
qualitative reverse engineering

Qualitative Reverse Engineering

DorianŠuc and Ivan Bratko

Artificial Intelligence Laboratory

Faculty of Computer and Information Science

University of Ljubljana, Slovenia

  • Qualitative reverse engineering:
    • generating explanation of engineering designs
    • in qualitative terms
  • Case study: industrial crane controller
    • regression, model trees
    • qualitative trees
  • Discussion
  • But: incomplete documentation
    • Why did designer move from one model to another?
    • Why was a design abandoned in favor of another?
    • How a designed system works?
  • Difficulties in re-use of design libraries:
    • intuitions behind designs are not explained

Accumulated knowledge in companies: libraries of designs and corresponding simulation models

  • Automated explanation generation
    • knowledge sharing: making tacit knowledge available to other people
    • model re-use: for model to be used by another designer,generate explanation how model works
  • Clockwork European project:
    • computer aided engineering design
    • industrial partners: car and heating industry
    • libraries of simulation models
the problem
The problem
  • Given designed system
  • Explain:
    • how the system works?
    • ideas, intuitions behind the design
  • Approach:
    • use ML on traces of system’s behavior
    • similar to behavioural cloning (Michie et al., 90; Sammut et al., 92)
case studies
Case studies
  • Reverse engineering of industrial crane controller, presented in this paper
  • Reverse engineering of car suspension system
case study anti sway crane
Case study: anti-sway crane

For transportation of slabs

Designed by Czech Technical Univ.

In daily use

Crane parameters:

travel 100m

height 15m,

width 30m

80-120 tons

anti sway crane
Anti-sway crane

Control force: F

State: cX,cV, lX,lV


Anti-sway controller

  • control force F
  • achieves cVDes
  • minimizes swing

desired carriage velocity cVDes

learning problem
Learning problem

controller = ?

Given simulation model + controller explain how controller works

learning problem2
Learning problem

cVDesF cX cV lX lV

2.0 0 0.0 0.0 0.0 0.0

2.0 24914 0.10.3 0.10.3

2.0 40188 0.4 2.1 0.4 2.1

2.0 -425 1.6 3.1 1.5 2.8

… ….. … … … …


ml methods used
ML methods used
  • Linear regression
  • Regression, model trees
  • Qualitative trees (QUIN)
qualitative induction quin
QUalitative INduction (QUIN)
  • Learning of qualitative trees
  • Qualitative tree defines a partition of the state space into areas with common qualitative behavior
  • Qualitatively constrained functions (QCFs) in leaves:
    • generalization of monotonicity constraint M+
    • similar to qualitative proportionality predicates (QPT-Forbus, 84)
example problem for quin
Example problem for QUIN
  • Noisy examples generated from z=x2-y2
qualitative patterns in data
Qualitative patterns in data

x > 0 & y > 0 =>

z = M+,-(x,y)

induced qualitative tree for function z x 2 y 2
Induced qualitative tree for functionz=x2-y2

z is monotonically increasing in its dependence on x and monotonically decreasing in its dependence on y

z is positively related to x and negatively related to y

quin algorithm
QUIN algorithm
  • MDL to learn “most consistent” QCF
  • Learning qualitative tree (similar to ID3)
    • for every possible split, split the examples into two subsets, find the “most consistent” QCF for both subsets and select the split minimizing tree-error cost (based on MDL)
  • QUIN is heuristic improvement of this idea
related work in learning qualitative models
Related work in learning qualitative models
  • Previous work includes:
    • Mozetič, 87 (KARDIO),
    • Coiera, 89; Hau, Coiera, 97
    • Bratko, Muggleton, Varšek, 91
    • Richards, Kraan, Kuipers, 92
    • Say, Kuru, 96
    • Kay, Rinner, Kuipers, 2000
  • Most of these systems induce QDE models, hard to apply to our problem
reverse engineering of anti sway crane controller
Reverse engineering of anti-sway crane controller

ML from examples of controller’s execution:

  • Linear regression, regression and model trees

(Weka-Witten&Frank,2000; M5-Quinlan, 92)

  • QUIN
reverse engineering of anti sway crane controller1
Reverse engineering of anti-sway crane controller

ML from examples of controller’s execution:

  • Linear regression, regression and model trees

(Weka-Witten&Frank,2000; M5-Quinlan, 92)

  • QUIN
  • Given:cVDes=2
  • an execution tracecVDes=0
  • of anti-sway controller (follow cVDes,minimize swing)
  • learn control force F = F(lX, lV, cV, cVDes)
evaluation criteria
Evaluation criteria

Comparing the induced models:

  • suitability as explanation
  • control performance
    • criterion of controller’s success: mean abs. velocity error and mean abs. swing error bellow errors of anti-sway controller + 10%
regression and model trees
Regression and model trees

Learning F=F (lX, lV, cV, cVDes):

  • pruning: smallest non-single leaf tree has 34 leaves
  • control performance:
    • cannot work: cVDes does not appear in any of induced trees
  • Helping the learning system:
  • derived attribute: relative carriage velocity cVRel=cV-cVDes
  • pruning: smallest non-single leaf tree has 17 leaves
  • control performance:
    • does not move(F0 when lX0), or huge swing
  • Summary:
  • complex trees, not even close to achieving the task
  • not suitable as explanation
linear regression
Linear regression

Learning F=F (lX, lV, cV, cVDes):

  • F= 23218 cVDes -18578 cV -20319 lX + 12410 lV -4800
  • bad fit (R=0.73, RRE=69.12%)
  • unsatisfactory control performance
  • Helping the learning system (using cVRel):
  • Similar, even slightly worse results
  • Summary:
  • simple models
  • but unsatisfactory control performance and bad fit
experiments with quin
Experiments with QUIN

Using cVRel:




F=M-(lX), lX decreas.

=> increase F

Small swing

=> follow cVDes

(since cVRel=cV-cVDes)

Large swing

=> reduce swing

experiments with quin cont
Experiments with QUIN, cont.

F = F(lX, lV, cV, cVDes):



Equivalent to previous qualitative tree!

(given the particular crane dynamics)

control with qua li titative trees
Control with qualititative trees
  • By transforming QCFs in leaves into real-valued functions satisfying qualitative constraints


Randomly generated real-valued increasing functions


Arbitrary increasing fn. of X

  • Testing 100 transformed qual. trees for control:
    • 43 successful (according to criterion of success)
r everse engineering of car suspension system
Reverse engineering ofcar suspension system
  • Given Quarter Car Model, reverse engineer controller for semi-active car suspension
  • Matlab/Simulink model provided by CTU and Intec
car suspension model
Car suspension model

The original controller

Black Box

The shock absorber



Reverse engineering of car suspension system

  • Qualitative controller was induced with QUIN
  • Induced controller applied to control car model
  • Performance compared with original controller
p erformance comparison
Performance comparison

Original controller:

Qualitative controller:

Comparatively better

Comparatively worse

s ummary of approach
Summary of approach
  • Reverse engineering in qualitative terms, useful to grasp the intuition behind the design
  • Induction of qualitative trees (QUIN)
  • Qualitative-to-quantitative transformation of qualitative control strategies (involves optimisation)
q ualitative v s quantitative models
Qualitative vs.quantitative models
  • Induced qualitative models superior as explanations (unsurprisingly)
  • Induced qualitative controllers* also perform better (surprisingly!)

*“Qualitative controller” means:

qualitative control strategy induced by QUIN, applied as controller after qualitative-to-quantitative transformation

w hy numerical learning fails
Why numerical learning fails?

1. Linear regression alone insufficient -

easy to see

2. But, why model trees also fail?

3. Why qualitative trees succeed when model

trees fail?

e xample from crane
Example from crane
  • M5 model tree contains leaf

- 0.0055 < lX =< 0.0055

Corresponding example set is very awkward for linear regression, see visualisation

  • QUIN finds better split:

- 0.223 < lX =< 0.163

This example set exhibits clearer qualitative pattern

t entative explanation
Tentative explanation
  • QUIN can detect subtle qualitative patterns, although they are obscured by large quantitative changes
  • Regression tree learning (M5) does not find good splits, resulting in regression-unfriendly leaves

Qualitative trees more successful:

    • suitability as explanation
    • control performance

Why regression trees fail?

  • Important patterns overshadowed by numerically dominant attributes
    • cVDes does not appear in regression trees!
  • Still qualitatively visible