Remarks on Inductive Bias

1. Remarks on Inductive Bias Shiliang Sun (sunsl02@mails.tsinghua.edu.cn) PhD Student Pattern Recognition and Intelligent Systems Department of Automation Tsinghua University

2. Outline • What is inductive bias? • Should inductive bias be avoided? • Principles reflecting the idea of inductive bias

3. What is inductive bias? • Consider the general setting in which an arbitrary learning algorithm L is provided an arbitrary set of training data Dc={<x,c(x)>} of some arbitrary target concept c. • After training, L is asked to classify a new instance xi. Let L(xi,Dc) denote the classification that L assigns to xi after learning from the training data Dc. • The inductive inference step performed by L is as follows: (Dc^xi)induce L(xi,Dc).

4. What is inductive bias? • Because L is an inductive learning algorithm, the result L(xi,Dc) it infers will not in general be provably correct. • Additional assumptions should be added to Dc^xi so that L(xi,Dc) would follow deductively and correctly. • The inductive bias is defined to be the set of assumptions B, s.t. for all new instances xi, (B^Dc^xi) deduce L(xi,Dc).

5. Should inductive bias be avoided • Rote-Learner is an unbiased inductive learner: learning corresponds simply to storing each observed training example in memory. (Is it the only unbiased learner?) • Has no generalization capability. • Conclusion: inductive bias should not be avoided.

6. Principles reflecting the idea of inductive bias • Philosophically, Occam's razor: prefer the simplest hypothesis that fits the data. • Computationally, • Kolmogorov Complexity. Minimum Description Length (MDL). • VC dimension. Structure Risk Minimum (SRM). Algorithm: Support Vector Machine.

7. Thanks!