This presentation is the property of its rightful owner.
Sponsored Links
1 / 149

第 四 章 集成学习与弱可学习理论 PowerPoint PPT Presentation


  • 107 Views
  • Uploaded on
  • Presentation posted in: General

第 四 章 集成学习与弱可学习理论. 1. 引言. 集成学习的根本思路是对同一问题使用一系列学习器进行学习,并使用一定的策略把各个不同的学习结果进行整合从而获得比单个学习器更好的学习效果。 主要任务 : 集成学习并非力求得到单一的最优分类器,而是通过一组由多个假设组合而成的集成得到更优的假设。. 泛化能力 ( generalization ability )是指机器学习算法对新鲜样本的适应能力。学习的目的是学到隐含在数据对背后的规律,对具有同一规律的学习集以外的数据,经过训练的网络也能给出合适的输出,该能力称为泛化能力。. 集成学习的发展和现状. 2.

Download Presentation

第 四 章 集成学习与弱可学习理论

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


3669242


3669242

1


3669242


3669242

  • generalization ability


3669242

2


3669242

  • 1990HansonSalamon50%


3669242

  • Valiant Kearns 1/2,;

  • ,Valiant Kearns PAC, , ? , ,1990, Schapire , , Boosting


3669242

  • 1995FreundSchapire BoostingSchapire Freund


3669242

  • 1996FreundSchapireAdaBoost(Adaptive Boost)FreundBoosting

  • Adaboost()()


3669242

  • 1996Breiman(Bootstrap Sampling)BaggingBaggingBreimanP&C(Perturb and Combine)BaggingBaggingk


3669242

  • (SVM)SVM(SE-SVM)

  • 2002C4.5Rule-PANENec4.5


3669242

  • BoostingBagging


3669242

  • AdaBoostArc-x4MultiBoost fBoostMiniBoost

  • BaggingSEQUELWaggingP-BaggingGASEN


3669242

  • Perrone(overfiting)


3669242

  • quinlanBreiman


3669242

  • 2001


3669242

  • ELRCByung


3669242


3669242


3669242

3


3669242


3669242

  • :

    1.:


3669242

2.:()

3.


3669242

  • :

    1.

    2.


3669242

3.


3669242

  • 1


3669242

2

3

BP


3669242

4


3669242

  • 4.1 PAC(Probably Approximately Correct)

    ValiantPAC1PAC


Pac probably approximately correct

PAC(Probably Approximately Correct)

  • PAC

  • X:

  • c: XX{0,1}

  • C: X D: X h: c

    (Probably)(Approximately)c


3669242

  • PAC :

    CPACcD0<<1/20<<1/2(1-)hP[h(x)!=c(x)] <=1/,1/


3669242

  • (PAC)PAC(1-)1PACcHc


3669242

  • 4.2

  • SN(x1,y1)...(xn,yn)xnD(x)yn=f(xn)fFD0,1/2P[h(x)f(x)]h1-1/1/


3669242

  • P[h(x)f(x)]h(1/2)+,01/2

  • 50%


3669242

  • 1990Schapire


3669242

  • BoostingBoosting


3669242

  • 4.3

  • (),,,,


3669242


3669242

  • (multi-classifier system)(mixture of experts)(committee-based learning)


3669242

4.1


3669242

  • 4.1,Dm,Dii=1,2,...,mhii=1,2,...,m,h*=F(h1,h2,...,hm)


3669242

  • 4.4

  • :

    1.

    BaggingBoosting

    2.


3669242

  • 4.4.1

  • ,,

  • Lam,


3669242

  • 1.

  • 1)Bootstrap Sampling

    BaggingBootstrap Sampling(),Bootstrap Sampling,63.2%BreimanBagging,,,,kBagging,


3669242

  • 2)Boosting

  • BoostingSchapire,:,,,,

  • Boosting,,,,,


3669242

  • 3)

  • K,K,,K-1,K,,,,,Bagging,Bagging,,,


3669242

  • 2

  • ,,,,


3669242

  • OpitzGuerra-Salcedo[45],[46],,,,1997Dietterich


3669242

  • 3.

  • ,,,[48],


3669242

  • 4.

  • ,Dietterich,,AdaBoost,


3669242

  • 5.

  • DietterichECOC,L,,LL,L0A,1BA,BLL,,,


3669242

  • 4.4.2

  • ,:

  • 1.

    ,,,Bagging


3669242

  • 2.

    ,,,Boosting

  • 3.

    ,


3669242

  • 4.

    :,


3669242

  • 4.4.3

  • 1.:

  • 2.:

  • 3.:()

  • 4.:ECOC()


3669242

  • :

    TJt-thj-thht,j=l0Pt(wj|X)


3669242

  • 1

  • :


3669242

  • :

  • :

  • :

    wt

  • :

  • :

  • :

  • :

  • :


3669242

  • 2


3669242

  • 3D-S

  • D-S


3669242

  • 4

  • LL


3669242

  • 5


3669242

  • 6


3669242

  • 4.4.4

  • Bias-varianceGeman

  • Bias-variance:

    (bias):

    (variance):

    :


3669242

  • Yu:


Adaboost

AdaBoost

5


3669242

  • Boosting,,,


3669242

  • AdaBoostSchapireFreund1995,Boosting,,


3669242

4.1 AdaBoost

  • ----------------------------------------------------------------------------

  • D = {(x1; y1),(x2; y2),...,(xm; ym)}

  • LT

  • D1(i) = 1/N ,1/N

  • For t = 1 to T

  • Dt

  • httht=L(D,Dt)

  • htt=Dt(i)[ht(xi)yi]

  • t=1/2ln((1-t)/t) ht

  • Zt


3669242

4.2 AdaBoost


3669242

  • XYY={+1-1}D = {(x1; y1),(x2; y2),...,(xm; ym)}, xi X and yiY (i=1;...;m)

  • tDtDtht : XYht, Dt+1Dt+1AdaBoostT,,T


3669242

  • 5.1 AdaBoost

  • AdaBoosthtt1/2-t1/2()t


3669242

  • FreundschapireH(x):

  • (4.1)

    Kullback-Leibler

    SehapiresingerFreundsehapire

    :

    (4.2)


3669242

  • (4-2)


5 2 t

5.2 T

  • AdaBoostTT:

  • t(4.1)

    H(x):


3669242

  • TTT(Cross-validation)H(x)T


3669242

5.3


3669242

  • FreundschaPireNVCdBoostingTBaumHaussferlzll:


3669242

  • TBoosting


3669242

  • AdaBoostsehapireFreundmargin()margin(x,y)margin:

    [-1,l]


3669242

  • H(x,y)marginmarginmarginmarginmarginmargin


3669242

  • schapiremargin


3669242

5.4

  • AdaBoost

    (4.8)

  • AdaBoosthtat(4.8)


Adaboost m1

AdaBoost.M1

6


3669242

  • Adaboost,,AdaBoostFreundSchapire1997AdaBoost.M1

  • ,AdaBoost


3669242

4.3 AdaBoost.M1


3669242

  • AdaBoost.M1

  • D = {(x1; y1),(x2; y2),...,(xm; ym)}AdaBoostyiY={1,...,k}

    LT

  • D1(i) = 1/N ,1/N

  • For t = 1 to T

    Dt

    ht

    htt=Dt(i)[ht(xi)yi]t>1/2T=t-1

    t=t / (1-t)


3669242

  • AdaBoost.M11/21-1/k,k,k=2,,k>2, AdaBoost.M1AdaBoostAdaBoost,50%, AdaBoost.M1,,


Adaboost m2

AdaBoost.M2

7


3669242

  • AdaBoostBoostingAdaBoost.M101Yyy10BoostingBoostingAdaBoost.M2


3669242

4.4 AdaBoost.M2


3669242

AdaBoost.M2

D = {(x1; y1),(x2; y2),...,(xm; ym)}yiY={1,...,k}

LT

B={(i,y): i{1,...,m},yyi}

D1(i,y)=1/|B| (i,y)B

For t = 1 to T

Dt

ht0-1

ht

t=t / (1-t)


Bagging

Bagging

8


3669242

  • BaggingBreimanBoostingBaggingn0.632


3669242

  • BaggingBoostingBaggingBoostingBaggingBoostingBaggingBoostingBagging


3669242

  • Bagging :

  • D = {(x1; y1),(x2; y2),...,(xm; ym)}

    LT

    For t= 1,...,T

    DDt=Bootstrap(D);

    ht

    10


3669242

  • BreimanBagging

    Bagging

    C(orde correct)CC.


3669242

  • ,Bagging,,Bagging


Stacking

Stacking

9


3669242

  • StackingWolpert1992


3669242

  • BoostingBagging,,,meta-learner(),


3669242

  • ,,,,,


3669242

  • :

    For t = 1,..., T

    ht=Lt(D)

    End

    D =

    For i = 1,...,m:

    For t = 1,...,T

    htxizit = ht(xi) %

    End

    End

    h=L(D)

    H(x)=h(h1(x),...,ht(x))


3669242

4.6


3669242

  • meta-learner


3669242

10


3669242

10.1

  • ?


3669242


3669242

  • [86]:(selective Ensemble)


3669242


3669242

10.2

  • Many could Be Better Than All


3669242


3669242

  • 1.

  • Nf:RmRnf1,f2,,fNwl,w2,,wNN(4.16)(4.17):

    0wi1 (4.16)

    (4.17)

    i:

    (4.18)

    filil


3669242

  • f:RmRnxRmp(x)d(x)fifi(x):


3669242

  • xfi:

    (4.20)

    :

    (4.21)

    ifip(x):

    (4.22)

    (4.23)


3669242

  • fifj(correlation):

  • (4.28)


3669242

  • wi=1/N(i=12N)(4.28):

  • k


3669242

  • :


3669242

2.

f: RmN

f1f2,fN

fRm{-1+1}


3669242

  • mD=[d1,d2,,dm]Tdjjfiifi=[fi1,fi2,...,fim]TfijijDfidj{-1+1}(j=12m)fij{-1l}(i=12N; j=12m)ijfijdj=+1 ,fijdj=-1im:


3669242

  • Error(x):

    j:

  • Sgn()


3669242

  • j

    +1-1 (105+15-1:


3669242

  • kj:

  • :


3669242

  • k


3669242

  • :


3669242


10 3 gasen

10.3 GASEN

  • GASEN(Genetic Algorithm based Selecte Ensemble)


3669242

  • GASENbootstrap()


Gasen

GASEN

D = {(x1; y1),(x2; y2),...,(xm; ym)}

LT

For t= 1,...,T

DDt=Bootstrap(D);

Nt = L(Dt)

(w)

w*

N*


3669242

4.7 GASEN


3669242

10.4

  • GASENwuGASENe-GasEN;Castro;[90]simAnnGASEN;[91]ACOSEN;(PSO)(ACO)(BPSO)


3669242

  • SEME(selective Ensemble of Multiple Eigenspaces)+SEMESEMESEME[93][94]


3669242

11


3669242

  • Drucker(USPS)12000NIST22000045000450003, 17%43%[96]FreundSchapireUSPS1200097092007306.4%8.6%26%


3669242

  • Hansen20-25%KroghSollichAdaBoost2001.4%UCI2%

  • MaoBoosting,BaggingOCRBoostingBaggingBagging


3669242

  • Gutta(divide and conquer)(gating)Huang


3669242

  • CherkauerNASAJARTOOLGaussMagellan

  • ShimshoniItntrator


3669242

  • SchapireSingerBoosting(text categorization)BoostingRocchioWeissemailTorstenBaggingbagged-CTREE


3669242


3669242

  • WAS (Weighted appearance specific)


3669242

  • AT&T, , )[115]


3669242

  • LCDSBoosting,


3669242

  • HoffmannBoostingMerlerBoostingBaggingx, TieuMorenoFurlanello OnodaBoostingRBF,


3669242


3669242

12


3669242

  • ()


3669242


3669242


3669242


  • Login