1 / 41

Qualitative Reverse Engineering

Qualitative Reverse Engineering. Do ri an Å uc and Ivan Bratko Artificial Intelligence Laboratory Faculty of Computer and Information Science University of Ljubljana, Slovenia. Overview. Qualitative reverse engineering: generating explanation of engineering designs in qualitative terms

liam
Download Presentation

Qualitative Reverse Engineering

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Qualitative Reverse Engineering DorianŠuc and Ivan Bratko Artificial Intelligence Laboratory Faculty of Computer and Information Science University of Ljubljana, Slovenia

  2. Overview • Qualitative reverse engineering: • generating explanation of engineering designs • in qualitative terms • Case study: industrial crane controller • regression, model trees • qualitative trees • Discussion

  3. Motivation • But: incomplete documentation • Why did designer move from one model to another? • Why was a design abandoned in favor of another? • How a designed system works? • Difficulties in re-use of design libraries: • intuitions behind designs are not explained Accumulated knowledge in companies: libraries of designs and corresponding simulation models

  4. Context • Automated explanation generation • knowledge sharing: making tacit knowledge available to other people • model re-use: for model to be used by another designer,generate explanation how model works • Clockwork European project: • computer aided engineering design • industrial partners: car and heating industry • libraries of simulation models

  5. The problem • Given designed system • Explain: • how the system works? • ideas, intuitions behind the design • Approach: • use ML on traces of system’s behavior • similar to behavioural cloning (Michie et al., 90; Sammut et al., 92)

  6. Case studies • Reverse engineering of industrial crane controller, presented in this paper • Reverse engineering of car suspension system

  7. Case study: anti-sway crane For transportation of slabs Designed by Czech Technical Univ. In daily use Crane parameters: travel 100m height 15m, width 30m 80-120 tons

  8. Anti-sway crane Control force: F State: cX,cV, lX,lV ? Anti-sway controller • control force F • achieves cVDes • minimizes swing desired carriage velocity cVDes

  9. Anti-sway crane

  10. Matlab model of crane controller

  11. Learning problem controller = ? Given simulation model + controller explain how controller works

  12. Learning problem

  13. Learning problem cVDesF cX cV lX lV 2.0 0 0.0 0.0 0.0 0.0 2.0 24914 0.10.3 0.10.3 2.0 40188 0.4 2.1 0.4 2.1 2.0 -425 1.6 3.1 1.5 2.8 … ….. … … … … ML

  14. ML methods used • Linear regression • Regression, model trees • Qualitative trees (QUIN)

  15. QUalitative INduction (QUIN) • Learning of qualitative trees • Qualitative tree defines a partition of the state space into areas with common qualitative behavior • Qualitatively constrained functions (QCFs) in leaves: • generalization of monotonicity constraint M+ • similar to qualitative proportionality predicates (QPT-Forbus, 84)

  16. Example problem for QUIN • Noisy examples generated from z=x2-y2

  17. Qualitative patterns in data x > 0 & y > 0 => z = M+,-(x,y)

  18. Induced qualitative tree for functionz=x2-y2 z is monotonically increasing in its dependence on x and monotonically decreasing in its dependence on y z is positively related to x and negatively related to y

  19. QUIN algorithm • MDL to learn “most consistent” QCF • Learning qualitative tree (similar to ID3) • for every possible split, split the examples into two subsets, find the “most consistent” QCF for both subsets and select the split minimizing tree-error cost (based on MDL) • QUIN is heuristic improvement of this idea

  20. Related work in learning qualitative models • Previous work includes: • Mozetič, 87 (KARDIO), • Coiera, 89; Hau, Coiera, 97 • Bratko, Muggleton, Varšek, 91 • Richards, Kraan, Kuipers, 92 • Say, Kuru, 96 • Kay, Rinner, Kuipers, 2000 • Most of these systems induce QDE models, hard to apply to our problem

  21. Reverse engineering of anti-sway crane controller ML from examples of controller’s execution: • Linear regression, regression and model trees (Weka-Witten&Frank,2000; M5-Quinlan, 92) • QUIN

  22. Reverse engineering of anti-sway crane controller ML from examples of controller’s execution: • Linear regression, regression and model trees (Weka-Witten&Frank,2000; M5-Quinlan, 92) • QUIN • Given:cVDes=2 • an execution tracecVDes=0 • of anti-sway controller (follow cVDes,minimize swing) • learn control force F = F(lX, lV, cV, cVDes)

  23. Evaluation criteria Comparing the induced models: • suitability as explanation • control performance • criterion of controller’s success: mean abs. velocity error and mean abs. swing error bellow errors of anti-sway controller + 10%

  24. Regression and model trees Learning F=F (lX, lV, cV, cVDes): • pruning: smallest non-single leaf tree has 34 leaves • control performance: • cannot work: cVDes does not appear in any of induced trees • Helping the learning system: • derived attribute: relative carriage velocity cVRel=cV-cVDes • pruning: smallest non-single leaf tree has 17 leaves • control performance: • does not move(F0 when lX0), or huge swing • Summary: • complex trees, not even close to achieving the task • not suitable as explanation

  25. Linear regression Learning F=F (lX, lV, cV, cVDes): • F= 23218 cVDes -18578 cV -20319 lX + 12410 lV -4800 • bad fit (R=0.73, RRE=69.12%) • unsatisfactory control performance • Helping the learning system (using cVRel): • Similar, even slightly worse results • Summary: • simple models • but unsatisfactory control performance and bad fit

  26. Experiments with QUIN Using cVRel: accelerate yes no F=M-(lX), lX decreas. => increase F Small swing => follow cVDes (since cVRel=cV-cVDes) Large swing => reduce swing

  27. Experiments with QUIN, cont. F = F(lX, lV, cV, cVDes): yes no Equivalent to previous qualitative tree! (given the particular crane dynamics)

  28. Control with qualititative trees • By transforming QCFs in leaves into real-valued functions satisfying qualitative constraints M+(X) Randomly generated real-valued increasing functions f+(X) Arbitrary increasing fn. of X • Testing 100 transformed qual. trees for control: • 43 successful (according to criterion of success)

  29. Anti-sway clone

  30. Reverse engineering ofcar suspension system • Given Quarter Car Model, reverse engineer controller for semi-active car suspension • Matlab/Simulink model provided by CTU and Intec

  31. Car suspension model The original controller Black Box The shock absorber characteristics

  32. Reverse engineering of car suspension system • Qualitative controller was induced with QUIN • Induced controller applied to control car model • Performance compared with original controller

  33. Performance comparison Original controller: Qualitative controller: Comparatively better Comparatively worse

  34. Summary of approach • Reverse engineering in qualitative terms, useful to grasp the intuition behind the design • Induction of qualitative trees (QUIN) • Qualitative-to-quantitative transformation of qualitative control strategies (involves optimisation)

  35. Qualitative vs.quantitative models • Induced qualitative models superior as explanations (unsurprisingly) • Induced qualitative controllers* also perform better (surprisingly!) *“Qualitative controller” means: qualitative control strategy induced by QUIN, applied as controller after qualitative-to-quantitative transformation

  36. Why numerical learning fails? 1. Linear regression alone insufficient - easy to see 2. But, why model trees also fail? 3. Why qualitative trees succeed when model trees fail?

  37. Example from crane • M5 model tree contains leaf - 0.0055 < lX =< 0.0055 Corresponding example set is very awkward for linear regression, see visualisation • QUIN finds better split: - 0.223 < lX =< 0.163 This example set exhibits clearer qualitative pattern

  38. Example from crane, cont.

  39. Example from crane, cont.

  40. Tentative explanation • QUIN can detect subtle qualitative patterns, although they are obscured by large quantitative changes • Regression tree learning (M5) does not find good splits, resulting in regression-unfriendly leaves

  41. Comparison Qualitative trees more successful: • suitability as explanation • control performance Why regression trees fail? • Important patterns overshadowed by numerically dominant attributes • cVDes does not appear in regression trees! • Still qualitatively visible

More Related