- 93 Views
- Uploaded on
- Presentation posted in: General

Machine Learning

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

SCE 5820: Machine Learning

Instructor: Jinbo Bi

Computer Science and Engineering Dept.

- Instructor: Dr. Jinbo Bi
- Office: ITEB 233
- Phone: 860-486-1458
- Email:jinbo@engr.uconn.edu
- Web: http://www.engr.uconn.edu/~jinbo/
- Time: Tue / Thur. 2:00pm – 3:15pm
- Location: BCH 302
- Office hours: Thur. 3:15-4:15pm

- HuskyCT
- http://learn.uconn.edu
- Login with your NetID and password
- Illustration

- Ph.D in Mathematics
- Research interests: machine learning, data mining, optimization, biomedical informatics, bioinformatics

subtyping

GWAS

Color of flowers

Cancer,

Psychiatric disorders, …

http://labhealthinfo.uconn.edu/EasyBreathing

- Prerequisite: Basics of linear algebra, calculus, optimization and basics of programming
- Course textbook (not required):
- Introduction to Data Mining(2005) by Pang-Ning Tan, Michael Steinbach, Vipin Kumar
- Pattern Recognition and Machine Learning (2006) Christopher M. Bishop
- Pattern Classification (2nd edition, 2000) Richard O. Duda, Peter E. Hart and David G. Stork

- Additional class notes and copied materials will be given
- Reading material links will be provided

- Objectives:
- Introduce students knowledge about the basic concepts of machine learning and the state-of-the-art machine learning algorithms
- Focus on some high-demanding application domains with hands-on experience of applying data mining/ machine learning techniques

- Format:
- Lectures, Micro teaching assignment, Quizzes, A term project

- Micro teaching assignment (1): 20%
- In-class/In-lab open-book open notes quizzes (4-5): 40%
- Term Project (1): 30%
- Participation:10%
- Term Project is one for each term. A term can consist of one or two students. Each student in the team needs to specify his/her roles in the project.
- Term projects can be chosen from a list of pre-defined projects

- Computers
- Participation in micro-teaching sessions is very important, and itself accounts for 50% of the credits for micro-teaching assignment
- Quizzes are graded by the instructor
- Final term projects will be graded by the instructor
- If you miss two quizzes, there will be a take-home quiz to make up the credits (missing one may be ok for your final grade.)

- Students in our class need to form THREE roughly-even study groups
- The instructor will help to balance off the study groups
- Each study group will be responsible of teaching one specific topic chosen from the following:
- Support Vector Machines
- Spectral Clustering
- Boosting (PAC learning model)

- Each team needs to give two presentations: a progress or preparation presentation (10-15min); a final presentation in the last week (15-20min)
- Each team needs to submit a project report
- Definition of the problem
- Data mining approaches used to solve the problem
- Computational results
- Conclusion (success or failure)

- Data mining (sometimes called data or knowledge discovery) is the process of analyzing data from different perspectives and summarizing it into useful information
- http://www.kdd.org/kdd2013/ ACM SIGKDD conference

- The ultimate goal of machine learning is the creation and understanding of machine intelligence
- http://icml.cc/2013/ ICML conference

- The main goal of statistical learning theory is to provide a framework for studying the problem of inference, that is of gaining knowledge, making predictions, and decisions from a set of data.
- http://nips.cc/Conferences/2012/ NIPS conference

- Fuzzy set and fuzzy logic
- Fuzzy if-then rules

- Evolutionary computation
- Genetic algorithms
- Evolutionary strategies

- Artificial neural networks
- Back propagation network (supervised learning)
- Self-organization network (unsupervised learning, will not be covered)

- Lack theoretical analysis about the behavior of the algorithms
- Traditional Techniquesmay be unsuitable due to
- Enormity of data
- High dimensionality of data
- Heterogeneous, distributed nature of data

Statistics/AI

Machine Learning/

Pattern Recognition

Soft Computing

- Supervised learning such as classification and regression
- Support vector machines
- Regularized least squares
- Fisher discriminant analysis (LDA)
- Graphical models (Bayesian nets)
- Boosting algorithms
Draw from Machine Learning domains

- Unsupervised learning such as clustering
- K-means
- Gaussian mixture models
- Hierarchical clustering
- Graph based clustering (spectral clustering)

- Dimension reduction
- Feature selection
- Compact feature space into low-dimensional space (principal component analysis)

- Many perspectives to analyze how the algorithm handles uncertainty
- Simple examples:
- Consistency analysis
- Learning bounds (upper bound on test error of the constructed model or solution)

- “Statistical” not “deterministic”
- With probability p, the upper bound holds
P( > p) <= Upper_bound

- With probability p, the upper bound holds

- Prediction tasks (supervised problem)
- Use some variables to predict unknown or future values of other variables.

- Description tasks (unsupervised problem)
- Find human-interpretable patterns that describe the data.

From [Fayyad, et.al.] Advances in Knowledge Discovery and Data Mining, 1996

- Given a collection of examples (training set )
- Each example contains a set of attributes, one of the attributes is the class.

- Find a model for class attribute as a function of the values of other attributes.
- Goal: previously unseen examples should be assigned a class as accurately as possible.
- A test set is used to determine the accuracy of the model. Usually, the given data set is divided into training and test sets, with training set used to build the model and test set used to validate it.

Test

Set

Model

categorical

categorical

continuous

class

Learn

Classifier

Training

Set

- High Risky Patient Detection
- Goal: Predict if a patient will suffer major complication after a surgery procedure
- Approach:
- Use patients vital signs before and after surgical operation.
- Heart Rate, Respiratory Rate, etc.

- Monitor patients by expert medical professionals to label which patient has complication, which has not.
- Learn a model for the class of the after-surgery risk.
- Use this model to detect potential high-risk patients for a particular surgical procedure

- Use patients vital signs before and after surgical operation.

- Face recognition
- Goal: Predict the identity of a face image
- Approach:
- Align all images to derive the features
- Model the class (identity) based on these features

- Cancer Detection
- Goal: To predict class (cancer or normal) of a sample (person), based on the microarray gene expression data
- Approach:
- Use expression levels of all genes as the features
- Label each example as cancer or normal
- Learn a model for the class of all samples

- Alzheimer's Disease Detection
- Goal: To predict class (AD or normal) of a sample (person), based on neuroimaging data such as MRI and PET
- Approach:
- Extract features from neuroimages
- Label each example as AD or normal
- Learn a model for the class of all samples

Reduced gray matter volume (colored areas) detected by MRI voxel-based

morphometry in AD patients compared to normal healthy controls.

- Predict a value of a real-valued variable based on the values of other variables, assuming a linear or nonlinear model of dependency.
- Extensively studied in statistics, neural network fields.
- Find a model to predict the dependent variable as a function of the values of independent variables.
- Goal: previously unseen examples should be predicted as accurately as possible.
- A test set is used to determine the accuracy of the model. Usually, the given data set is divided into training and test sets, with training set used to build the model and test set used to validate it.

Refund

Marital

Taxable

Tid

Loss

Status

Income

100

1

Yes

Single

125K

120

2

No

Married

100K

-200

3

No

Single

70K

-300

4

Yes

Married

120K

-400

5

No

Divorced

95K

-500

6

No

Married

60K

-190

7

Yes

Divorced

220K

300

8

No

Single

85K

Test

Set

-240

9

No

Married

75K

Model

90

10

No

Single

90K

10

Continuous target

categorical

categorical

continuous

Current data, want to use the model to predict

Learn

Regressor

Training

Set

Past transaction records, label them

goals: Predict the possible loss from a customer

- Examples:
- Predicting sales amounts of new product based on advertising expenditure.
- Predicting wind velocities as a function of temperature, humidity, air pressure, etc.
- Time series prediction of stock market indices.

- Given a set of data points, each having a set of attributes, and a similarity measure among them, find clusters such that
- Data points in one cluster are more similar to one another.
- Data points in separate clusters are less similar to one another.

- Similarity Measures:
- Euclidean Distance if attributes are continuous.
- Other Problem-specific Measures

- Euclidean Distance Based Clustering in 3-D space.

Intracluster distances

are minimized

Intercluster distances

are maximized

- High Risky Patient Detection
- Goal: Predict if a patient will suffer major complication after a surgery procedure
- Approach:
- Use patients vital signs before and after surgical operation.
- Heart Rate, Respiratory Rate, etc.

- Find patients whose symptoms are dissimilar from most of other patients.

- Use patients vital signs before and after surgical operation.

- Document Clustering:
- Goal: To find groups of documents that are similar to each other based on the important terms appearing in them.
- Approach: To identify frequently occurring terms in each document. Form a similarity measure based on the frequencies of different terms. Use it to cluster.
- Gain: Information Retrieval can utilize the clusters to relate a new document or search term to clustered documents.

- Clustering Points: 3204 Articles of Los Angeles Times.
- Similarity Measure: How many words are common in these documents (after some word filtering).

- K-Nearest-Neighbor classifiers
- Naïve Bayes classifier
- Neural Networks
- Linear Discriminant Analysis (LDA)
- Support Vector Machines (SVM)
- Decision Trees
- Logistic Regression
- Graphical models

- Linear Regression
- Ridge Regression
- LASSO – Least Absolute Shrinkage and Selection Operator
- Neural Networks

- K-Means
- Hierarchical clustering
- Graph-based clustering (Spectral clustering)
- Semi-supervised clustering
- Others

- Scalability
- Dimensionality
- Complex and Heterogeneous Data
- Data Quality
- Data Ownership and Distribution
- Privacy Preservation

- An experiment (random variable) is a well-defined process with observable outcomes.
- The set or collection of all outcomes of an experiment is called the sample space, S.
- An event E is any subset of outcomes from S.
- Probability of an event, P(E) is P(E) = number of outcomes in E / number of outcomes in S.

Apples and Oranges

X: identity of the fruit

Y: identity of the box

Assume P(Y=r) = 40%, P(Y=b) = 60% (prior)

P(X=a|Y=r) = 2/8 = 25%

P(X=o|Y=r) = 6/8 = 75%

P(X=a|Y=b) = 3/4 = 75%

P(X=o|Y=b) = 1/4 = 25%

Marginal

P(X=a) = 11/20, P(X=o) = 9/20

Posterior

P(Y=r|X=o) = 2/3

P(Y=b|X=o) = 1/3

- Marginal Probability
- Conditional Probability

Joint Probability

- Sum Rule

The marginal prob of X equals the sum of

the joint prob of x and y with respect to y

- Product Rule

The joint prob of X and Y equals the product of the conditional prob of Y

given X and the prob of X

p(X,Y)

p(Y)

Y=2

Y=1

p(X|Y=1)

p(X)

- Sum Rule
- Product Rule
- Bayes’ Rule

= p(X|Y)p(Y)

posterior likelihood × prior

p(X=a) = p(X=a,Y=r) + p(X=a,Y=b)

= p(X=a|Y=r)p(Y=r) + p(X=a|Y=b)p(Y=b) P(X=o) = 9/20

=0.25*0.4 + 0.75*0.6 = 11/20

p(Y=r|X=o) = p(Y=r,X=o)/p(X=o)

= p(X=o|Y=r)p(Y=r)/p(X=o)

= 0.75*0.4 / (9/20) = 2/3

Assume P(Y=r) = 40%, P(Y=b) = 60%

P(X=a|Y=r) = 2/8 = 25%

P(X=o|Y=r) = 6/8 = 75%

P(X=a|Y=b) = 3/4 = 75%

P(X=o|Y=b) = 1/4 = 25%

p(X=a) = p(X=a,Y=r) + p(X=a,Y=b)

= p(X=a|Y=r)p(Y=r) + p(X=a|Y=b)p(Y=b) P(X=o) = 9/20

=0.25*0.4 + 0.75*0.6 = 11/20

p(Y=r|X=o) = p(Y=r,X=o)/p(X=o)

= p(X=o|Y=r)p(Y=r)/p(X=o)

= 0.75*0.4 / (9/20) = 2/3

Assume P(Y=r) = 40%, P(Y=b) = 60%

P(X=a|Y=r) = 2/8 = 25%

P(X=o|Y=r) = 6/8 = 75%

P(X=a|Y=b) = 3/4 = 75%

P(X=o|Y=b) = 1/4 = 25%

- The mean of a random variable X is the average value X takes.
- The variance of X is a measure of how dispersed the values that X takes are.
- The standard deviation is simply the square root of the variance.

- X= {1, 2} with P(X=1) = 0.8 and P(X=2) = 0.2
- Mean
- 0.8 X 1 + 0.2 X 2 = 1.2

- Variance
- 0.8 X (1 – 1.2) X (1 – 1.2) + 0.2 X (2 – 1.2) X (2-1.2)

y

x

- SC_prob_basics1.pdf (necessary)
- SC_prob_basic2.pdf
Loaded to HuskyCT

- The product of two matrices
- Special case: vector-vector product, matrix-vector product

C

A

B

B

C

A

1

.

1

.

.

1

where

eigen-decomposition of A

Frobenius norm

orthogonal

diagonal

orthogonal

- SC_linearAlg_basics.pdf (necessary)
- SVD_basics.pdf
loaded to HuskyCT

- This is the end of the FIRST chapter of this course
- Next Class
Cluster analysis

- General topics
- K-means

- Slides after this one are backup slides, you can also check them to learn more