This presentation is the property of its rightful owner.
1 / 9

Feature Selection and Extraction PowerPoint PPT Presentation

Feature Selection and Extraction. ☺ Given a set of features S={v 1 , v 2 , …, v D }, find a subset of S’ with |S’|=d < D such that J(S’) ≥J(T) for any subset T of S, |T|=d.

Feature Selection and Extraction

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Feature Selection and Extraction

☺ Given a set of features S={v1, v2, …, vD}, find a subset of S’ with |S’|=d < D such that J(S’)≥J(T) for any subset T of S, |T|=d.

☺ Given a set of features S={v1, v2, …, vD}, find a set S’ with |S’|=d < D derived from S such that J(S’)≥J(T) for any set T, with |T|=d, derived from S.

Feature Selection

To exhaustively select d optimal features out

of D needs to evaluate D!/[(D-d)!d!] feature

sets which is not practical even for small D

and d, e.g., when D=20, d=10, 184756

feature sets would have to be considered.

Two ways to overcome this problem.

• Whitney’s method (1971) 1101-1103.

(2)Branch and Bound (1977) 917- 922.

Whitney’s Nonparametric Method

1. Use 1-nn decision and leave-out-out error

2. First feature selected should have the

smallest error

3. Next feature selected is the one joined

with the previously selected features has

the smallest error

4. Continue step 3 until d features are

selected.

Results of Whitney’s on iris, imox, 8OX Data Sets (in # of errors)

Feature ordering

3 4 1 2

Iris18 7 7 6 /150

Imox 6 8 7 1 5 2 4 3

81 37 13 8 7 5 4 10 /192

8OX 7 6 3 5 1 2 4 8

17 7 2 1 1 0 0 3 /45

Branch and Bound (1/3)

Let the number of features in the original set be n.

We have to select a subset of features so that the

value of a criterion is optimized over all subsets of

size m < n. Let (Z1,Z2,….,Zk) be the k=n-m features

to be discarded to obtain an m feature subset.

Each variable Zi can take on values in {1,2,….,n}

but the order of Zi’s is immaterial, hence we

consider only sequences of Zi’s, such that

Z1 < Z2 < …. <Zk

Branch and Bound (2/3)

The feature selection criterion, Jm(Z1,Z2,...,Zk), is a function

of the m (=n-k) features obtained by discarding Z1,Z2,...,Zk

from the n feature set. The feature subset selection

problem is to find the optimum subset {Z1*, Z2*,…, Zk* }

such that

Jm(Z1*, Z2*,…, Zk* ) = max {Jm(Z1,Z2,...,Zk)}

where the criterion J must satisfy the monotonicity, which is

defined by

J(Z1)≧J(Z1,Z2) ≧…≧ Jm(Z1,Z2,...,Zk)

Branch and Bound (3/3)

Let B be a lower bound on the optimum

(maximum) value of the criterion

Jm(Z1*, Z2*,…, Zk* ), i.e., B≦Jm(Z1*,Z2*,…,Zk* )

If J(Z1,Z2,...,Zh) (h<k=n-m) were less than B,

then

Jm(Z1,Z2,...,Zh,Zh+1,…,Zk)<B

for all possible {Zh+1,…,Zk}