Qualitative Reverse Engineering

400 Views

Download Presentation
## Qualitative Reverse Engineering

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -

**Qualitative Reverse Engineering**DorianŠuc and Ivan Bratko Artificial Intelligence Laboratory Faculty of Computer and Information Science University of Ljubljana, Slovenia**Overview**• Qualitative reverse engineering: • generating explanation of engineering designs • in qualitative terms • Case study: industrial crane controller • regression, model trees • qualitative trees • Discussion**Motivation**• But: incomplete documentation • Why did designer move from one model to another? • Why was a design abandoned in favor of another? • How a designed system works? • Difficulties in re-use of design libraries: • intuitions behind designs are not explained Accumulated knowledge in companies: libraries of designs and corresponding simulation models**Context**• Automated explanation generation • knowledge sharing: making tacit knowledge available to other people • model re-use: for model to be used by another designer,generate explanation how model works • Clockwork European project: • computer aided engineering design • industrial partners: car and heating industry • libraries of simulation models**The problem**• Given designed system • Explain: • how the system works? • ideas, intuitions behind the design • Approach: • use ML on traces of system’s behavior • similar to behavioural cloning (Michie et al., 90; Sammut et al., 92)**Case studies**• Reverse engineering of industrial crane controller, presented in this paper • Reverse engineering of car suspension system**Case study: anti-sway crane**For transportation of slabs Designed by Czech Technical Univ. In daily use Crane parameters: travel 100m height 15m, width 30m 80-120 tons**Anti-sway crane**Control force: F State: cX,cV, lX,lV ? Anti-sway controller • control force F • achieves cVDes • minimizes swing desired carriage velocity cVDes**Matlab model of crane**controller**Learning problem**controller = ? Given simulation model + controller explain how controller works**Learning problem**cVDesF cX cV lX lV 2.0 0 0.0 0.0 0.0 0.0 2.0 24914 0.10.3 0.10.3 2.0 40188 0.4 2.1 0.4 2.1 2.0 -425 1.6 3.1 1.5 2.8 … ….. … … … … ML**ML methods used**• Linear regression • Regression, model trees • Qualitative trees (QUIN)**QUalitative INduction (QUIN)**• Learning of qualitative trees • Qualitative tree defines a partition of the state space into areas with common qualitative behavior • Qualitatively constrained functions (QCFs) in leaves: • generalization of monotonicity constraint M+ • similar to qualitative proportionality predicates (QPT-Forbus, 84)**Example problem for QUIN**• Noisy examples generated from z=x2-y2**Qualitative patterns in data**x > 0 & y > 0 => z = M+,-(x,y)**Induced qualitative tree for functionz=x2-y2**z is monotonically increasing in its dependence on x and monotonically decreasing in its dependence on y z is positively related to x and negatively related to y**QUIN algorithm**• MDL to learn “most consistent” QCF • Learning qualitative tree (similar to ID3) • for every possible split, split the examples into two subsets, find the “most consistent” QCF for both subsets and select the split minimizing tree-error cost (based on MDL) • QUIN is heuristic improvement of this idea**Related work in learning qualitative models**• Previous work includes: • Mozetič, 87 (KARDIO), • Coiera, 89; Hau, Coiera, 97 • Bratko, Muggleton, Varšek, 91 • Richards, Kraan, Kuipers, 92 • Say, Kuru, 96 • Kay, Rinner, Kuipers, 2000 • Most of these systems induce QDE models, hard to apply to our problem**Reverse engineering of anti-sway crane controller**ML from examples of controller’s execution: • Linear regression, regression and model trees (Weka-Witten&Frank,2000; M5-Quinlan, 92) • QUIN**Reverse engineering of anti-sway crane controller**ML from examples of controller’s execution: • Linear regression, regression and model trees (Weka-Witten&Frank,2000; M5-Quinlan, 92) • QUIN • Given:cVDes=2 • an execution tracecVDes=0 • of anti-sway controller (follow cVDes,minimize swing) • learn control force F = F(lX, lV, cV, cVDes)**Evaluation criteria**Comparing the induced models: • suitability as explanation • control performance • criterion of controller’s success: mean abs. velocity error and mean abs. swing error bellow errors of anti-sway controller + 10%**Regression and model trees**Learning F=F (lX, lV, cV, cVDes): • pruning: smallest non-single leaf tree has 34 leaves • control performance: • cannot work: cVDes does not appear in any of induced trees • Helping the learning system: • derived attribute: relative carriage velocity cVRel=cV-cVDes • pruning: smallest non-single leaf tree has 17 leaves • control performance: • does not move(F0 when lX0), or huge swing • Summary: • complex trees, not even close to achieving the task • not suitable as explanation**Linear regression**Learning F=F (lX, lV, cV, cVDes): • F= 23218 cVDes -18578 cV -20319 lX + 12410 lV -4800 • bad fit (R=0.73, RRE=69.12%) • unsatisfactory control performance • Helping the learning system (using cVRel): • Similar, even slightly worse results • Summary: • simple models • but unsatisfactory control performance and bad fit**Experiments with QUIN**Using cVRel: accelerate yes no F=M-(lX), lX decreas. => increase F Small swing => follow cVDes (since cVRel=cV-cVDes) Large swing => reduce swing**Experiments with QUIN, cont.**F = F(lX, lV, cV, cVDes): yes no Equivalent to previous qualitative tree! (given the particular crane dynamics)**Control with qualititative trees**• By transforming QCFs in leaves into real-valued functions satisfying qualitative constraints M+(X) Randomly generated real-valued increasing functions f+(X) Arbitrary increasing fn. of X • Testing 100 transformed qual. trees for control: • 43 successful (according to criterion of success)**Reverse engineering ofcar suspension system**• Given Quarter Car Model, reverse engineer controller for semi-active car suspension • Matlab/Simulink model provided by CTU and Intec**Car suspension model**The original controller Black Box The shock absorber characteristics**Reverse engineering of car suspension system**• Qualitative controller was induced with QUIN • Induced controller applied to control car model • Performance compared with original controller**Performance comparison**Original controller: Qualitative controller: Comparatively better Comparatively worse**Summary of approach**• Reverse engineering in qualitative terms, useful to grasp the intuition behind the design • Induction of qualitative trees (QUIN) • Qualitative-to-quantitative transformation of qualitative control strategies (involves optimisation)**Qualitative vs.quantitative models**• Induced qualitative models superior as explanations (unsurprisingly) • Induced qualitative controllers* also perform better (surprisingly!) *“Qualitative controller” means: qualitative control strategy induced by QUIN, applied as controller after qualitative-to-quantitative transformation**Why numerical learning fails?**1. Linear regression alone insufficient - easy to see 2. But, why model trees also fail? 3. Why qualitative trees succeed when model trees fail?**Example from crane**• M5 model tree contains leaf - 0.0055 < lX =< 0.0055 Corresponding example set is very awkward for linear regression, see visualisation • QUIN finds better split: - 0.223 < lX =< 0.163 This example set exhibits clearer qualitative pattern**Tentative explanation**• QUIN can detect subtle qualitative patterns, although they are obscured by large quantitative changes • Regression tree learning (M5) does not find good splits, resulting in regression-unfriendly leaves**Comparison**Qualitative trees more successful: • suitability as explanation • control performance Why regression trees fail? • Important patterns overshadowed by numerically dominant attributes • cVDes does not appear in regression trees! • Still qualitatively visible