- By
**liam** - Follow User

- 383 Views
- Uploaded on

Download Presentation
## Qualitative Reverse Engineering

**An Image/Link below is provided (as is) to download presentation**

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

### Qualitative Reverse Engineering

DorianŠuc and Ivan Bratko

Artificial Intelligence Laboratory

Faculty of Computer and Information Science

University of Ljubljana, Slovenia

Overview

- Qualitative reverse engineering:
- generating explanation of engineering designs
- in qualitative terms
- Case study: industrial crane controller
- regression, model trees
- qualitative trees
- Discussion

Motivation

- But: incomplete documentation
- Why did designer move from one model to another?
- Why was a design abandoned in favor of another?
- How a designed system works?
- Difficulties in re-use of design libraries:
- intuitions behind designs are not explained

Accumulated knowledge in companies: libraries of designs and corresponding simulation models

Context

- Automated explanation generation
- knowledge sharing: making tacit knowledge available to other people
- model re-use: for model to be used by another designer,generate explanation how model works

- Clockwork European project:
- computer aided engineering design
- industrial partners: car and heating industry
- libraries of simulation models

The problem

- Given designed system
- Explain:
- how the system works?
- ideas, intuitions behind the design

- Approach:
- use ML on traces of system’s behavior
- similar to behavioural cloning (Michie et al., 90; Sammut et al., 92)

Case studies

- Reverse engineering of industrial crane controller, presented in this paper
- Reverse engineering of car suspension system

Case study: anti-sway crane

For transportation of slabs

Designed by Czech Technical Univ.

In daily use

Crane parameters:

travel 100m

height 15m,

width 30m

80-120 tons

Anti-sway crane

Control force: F

State: cX,cV, lX,lV

?

Anti-sway controller

- control force F
- achieves cVDes
- minimizes swing

desired carriage velocity cVDes

Matlab model of crane

controller

Learning problem

cVDesF cX cV lX lV

2.0 0 0.0 0.0 0.0 0.0

2.0 24914 0.10.3 0.10.3

2.0 40188 0.4 2.1 0.4 2.1

2.0 -425 1.6 3.1 1.5 2.8

… ….. … … … …

ML

ML methods used

- Linear regression
- Regression, model trees
- Qualitative trees (QUIN)

QUalitative INduction (QUIN)

- Learning of qualitative trees
- Qualitative tree defines a partition of the state space into areas with common qualitative behavior

- Qualitatively constrained functions (QCFs) in leaves:
- generalization of monotonicity constraint M+
- similar to qualitative proportionality predicates (QPT-Forbus, 84)

Example problem for QUIN

- Noisy examples generated from z=x2-y2

Induced qualitative tree for functionz=x2-y2

z is monotonically increasing in its dependence on x and monotonically decreasing in its dependence on y

z is positively related to x and negatively related to y

QUIN algorithm

- MDL to learn “most consistent” QCF
- Learning qualitative tree (similar to ID3)
- for every possible split, split the examples into two subsets, find the “most consistent” QCF for both subsets and select the split minimizing tree-error cost (based on MDL)

- QUIN is heuristic improvement of this idea

Related work in learning qualitative models

- Previous work includes:
- Mozetič, 87 (KARDIO),
- Coiera, 89; Hau, Coiera, 97
- Bratko, Muggleton, Varšek, 91
- Richards, Kraan, Kuipers, 92
- Say, Kuru, 96
- Kay, Rinner, Kuipers, 2000
- Most of these systems induce QDE models, hard to apply to our problem

Reverse engineering of anti-sway crane controller

ML from examples of controller’s execution:

- Linear regression, regression and model trees

(Weka-Witten&Frank,2000; M5-Quinlan, 92)

- QUIN

Reverse engineering of anti-sway crane controller

ML from examples of controller’s execution:

- Linear regression, regression and model trees

(Weka-Witten&Frank,2000; M5-Quinlan, 92)

- QUIN

- Given:cVDes=2
- an execution tracecVDes=0
- of anti-sway controller (follow cVDes,minimize swing)
- learn control force F = F(lX, lV, cV, cVDes)

Evaluation criteria

Comparing the induced models:

- suitability as explanation
- control performance
- criterion of controller’s success: mean abs. velocity error and mean abs. swing error bellow errors of anti-sway controller + 10%

Regression and model trees

Learning F=F (lX, lV, cV, cVDes):

- pruning: smallest non-single leaf tree has 34 leaves
- control performance:
- cannot work: cVDes does not appear in any of induced trees

- Helping the learning system:
- derived attribute: relative carriage velocity cVRel=cV-cVDes
- pruning: smallest non-single leaf tree has 17 leaves
- control performance:
- does not move(F0 when lX0), or huge swing

- Summary:
- complex trees, not even close to achieving the task
- not suitable as explanation

Linear regression

Learning F=F (lX, lV, cV, cVDes):

- F= 23218 cVDes -18578 cV -20319 lX + 12410 lV -4800
- bad fit (R=0.73, RRE=69.12%)
- unsatisfactory control performance

- Helping the learning system (using cVRel):
- Similar, even slightly worse results
- Summary:
- simple models
- but unsatisfactory control performance and bad fit

Experiments with QUIN

Using cVRel:

accelerate

yes

no

F=M-(lX), lX decreas.

=> increase F

Small swing

=> follow cVDes

(since cVRel=cV-cVDes)

Large swing

=> reduce swing

Experiments with QUIN, cont.

F = F(lX, lV, cV, cVDes):

yes

no

Equivalent to previous qualitative tree!

(given the particular crane dynamics)

Control with qualititative trees

- By transforming QCFs in leaves into real-valued functions satisfying qualitative constraints

M+(X)

Randomly generated real-valued increasing functions

f+(X)

Arbitrary increasing fn. of X

- Testing 100 transformed qual. trees for control:
- 43 successful (according to criterion of success)

Reverse engineering ofcar suspension system

- Given Quarter Car Model, reverse engineer controller for semi-active car suspension
- Matlab/Simulink model provided by CTU and Intec

Reverse engineering of car suspension system

- Qualitative controller was induced with QUIN
- Induced controller applied to control car model
- Performance compared with original controller

Performance comparison

Original controller:

Qualitative controller:

Comparatively better

Comparatively worse

Summary of approach

- Reverse engineering in qualitative terms, useful to grasp the intuition behind the design
- Induction of qualitative trees (QUIN)
- Qualitative-to-quantitative transformation of qualitative control strategies (involves optimisation)

Qualitative vs.quantitative models

- Induced qualitative models superior as explanations (unsurprisingly)
- Induced qualitative controllers* also perform better (surprisingly!)

*“Qualitative controller” means:

qualitative control strategy induced by QUIN, applied as controller after qualitative-to-quantitative transformation

Why numerical learning fails?

1. Linear regression alone insufficient -

easy to see

2. But, why model trees also fail?

3. Why qualitative trees succeed when model

trees fail?

Example from crane

- M5 model tree contains leaf

- 0.0055 < lX =< 0.0055

Corresponding example set is very awkward for linear regression, see visualisation

- QUIN finds better split:

- 0.223 < lX =< 0.163

This example set exhibits clearer qualitative pattern

Tentative explanation

- QUIN can detect subtle qualitative patterns, although they are obscured by large quantitative changes
- Regression tree learning (M5) does not find good splits, resulting in regression-unfriendly leaves

Comparison

Qualitative trees more successful:

- suitability as explanation
- control performance

Why regression trees fail?

- Important patterns overshadowed by numerically dominant attributes
- cVDes does not appear in regression trees!
- Still qualitatively visible

Download Presentation

Connecting to Server..