Additive Models and Trees. Lecture Notes for CMPUT 466/551 Nilanjan Ray. Principal Source: Department of Statistics, CMU. Topics to cover. GAM: Generalized Additive Models CART: Classification and Regression Trees MARS: Multiple Adaptive Regression Splines. Generalized Additive Models.
Additive Models and Trees
Lecture Notes for CMPUT 466/551
Principal Source: Department of Statistics, CMU
What is GAM?
The functions fjare smoothing functions in general, such as splines, kernel
functions, linear functions, and so on…
Each function could be different, e.g., f1 can be linear, f2 can be a natural
Compare GAM with Linear Basis Expansions (Ch. 5 of [HTF])
Any similarity (in principle) with Naïve Bayes model?
Until the functions change less than a prespecified threshold
How to choose fitting functions?
Additive Logistic Regression: Backfitting
Fitting logistic regression (P99)
Fitting additive logistic regression (P262)
Using weighted least squares to fit a linear model to zi with weights wi, give new estimates
c. Using weighted backfitting algorithm to fit an additive model to zi with weights wi, give new estimates
3. Continue step 2 until converge
3.Continue step 2 until converge
Sensitivity: Probability of predicting spam given true state is spam =
Specificity: Probability of predicting email given true state is email =
Note that this is still an additive model
Penalty on the complexity/size of the tree
Cost: sum of squared errors
Node impurity measures versus class proportion for 2-class problem
Construct B number of trees from B bootstrap samples– bootstrap trees
is computed from the bth bootstrap sample
in this case a tree
Bagging reduces the variance of the original tree by aggregation
|C| = 2 * N * p
The final model M typically overfits the data
=>Need to reduce the model size (# of terms)
Backward deletion procedure
Choose the model size with minimum GCV.
e.g. (Xj - t1) * (Xj - t1) is not considered
MARS forward procedure
= CART tree growing procedure