Knowledge-based systems

Lecture 4 Knowledge-based systems SanaullahManzoor CS&IT, Lahore Leads University sanaullahmanzoor1988@gmail.com https://sites.google.com/site/engrsanaullahmanzoor/home

Quiz 1 Time 15 min • Q1: What is data pre-processing, write and briefly explain number of steps involved in data pre-processing before classification? 12marks • Q2: Write formula of Bayesian classifier. 3 marks

Overview • Classification • Bayesian Classifier • Matlab Introduction • Classification using Matalb • Bayesian classifier implementation 3

Bayesian Classifier

Classification!!!! Lets start with simple classification form (General Problem) Problem statement: Given features X1,X2,…,Xn Predict a label Y

Classification!!!! Classifier 5 Example :Digit Recognition Features : X1,…,Xn {0,1} (Black vs. White pixels) Lables : Y  {5,6} (predict whether a digit is a 5 or a 6)

Classification!!!! Classifier 5 Example :Digit Recognition Our Problem can be stated as : “what is the probability that the image represents a 5 given its pixels?”

Bayesian Classification!!!! Lets Solve our Problem with “Bayesian Rule” (Thomas Bayes 1702 – 1761) Bayesian classification is Probabilistic approach .

Bayesian Classification!!!! Likelihood Prior Bayesian rules : Normalization Constant

Bayesian Classification!!!! Three components of Bayesian rule are: 1-Likelihood 2-Prior 3-Normalization Constants or Evidence Likelihood : It is probability of features in a given class Like in our case what is probability of features() for given Y={class 5 or class 6}.

Bayesian Classification!!!! Three components of Bayesian rule are: 1-Likelihood 2-Prior 3-Normalization Constants or Evidence Prior : It is “probability of occurrence of a class” If class is 5. If class is 6.

Bayesian Classification!!!! Three components of Bayesian rule are: 1-Likelihood 2-Prior 3-Normalization Constants or Evidence Normalization Constants or Evidence : It is Probability of occurrence of a feature

Bayesian Classification!!!! Solution of our Example : If class is 5. If class is 6. Which one class is with greater probability that’s our solution 

Bayesian Classification!!!! Sunny=1 Overcast=2 Rain=3 Hot=1 Mild=2 Cool=3 High=1 Normal=2 Example 2: Weak=1 Strong=2 No=1 Yes=2

Bayesian Classification!!!! 1.Calaculate total Yes and No Probability Solution:

2. Calculate Yes and No probability in the feature “Outlook” Bayesian Classification!!!!

Bayesian Classification!!!! 3. Calculate Yes and No probability in the feature “Temperature”

Bayesian Classification!!!! 4. Calculate Yes and No probability in the feature “Humidity and Wind”

Bayesian Classification!!!! Solution:

Bayesian Classification!!!! Given a new instance (Testing phase) x’=(Outlook=Sunny, Temperature=Cool, Humidity=High, Wind=Strong) Using our calculations:

Bayesian Classification!!!! x’=(Outlook=Sunny, Temperature=Cool, Humidity=High, Wind=Strong) P(Outlook=Sunny|Play=No) = 3/5 P(Temperature=Cool|Play==No) = 1/5 P(Huminity=High|Play=No) = 4/5 P(Wind=Strong|Play=No) = 3/5 P(Play=No) = 5/14

Bayesian Classification!!!! x’=(Outlook=Sunny, Temperature=Cool, Humidity=High, Wind=Strong) P(Outlook=Sunny|Play=Yes) = 2/9 P(Temperature=Cool|Play=Yes) = 3/9 P(Huminity=High|Play=Yes) = 3/9 P(Wind=Strong|Play=Yes) = 3/9 P(Play=Yes) = 9/14

Hands-on Example # Import Bayes.csv from class webpage # Select training data traindata <- Bayes[1:14,] # Select test data testdata <- Bayes[15,]

Fit Classifier function nb = NaiveBayes.fit(training, class)Description nb = NaiveBayes.fit(training, class) builds a NaiveBayes classifier object nb. training is an N-by-D numeric matrix of training data. Rows of training correspond to observations; columns correspond to features. class is a classing variable for trainingK distinct levels. Each element of class defines which class the corresponding row of training belongs to. training and class must have the same number of rows. nb = NaiveBayes.fit(..., 'param1',val1, 'param2',val2, ...) specifies one or more of the following name/value pairs: • 'Distribution' – a string or a 1-by-D cell vector of strings, • specifying which distributions fit uses to model the data. • If the value is a string, fit models all the features using one type of distribution. • fit can also model different features using different types of distributions. • If the value is a cell vector, its jth element specifies the distribution fit uses for the jth feature. • The available types of distributions are:

Predict Function Predict class label for test data Syntax cpre = predict(nb,test) Description cpre = predict(nb,test) classifies each row of data in test into one of the classes according to the NaiveBayes classifier nb, and returns the predicted class level cpre. test is an N-by-nb.ndims matrix, where N is the number of observations in the test data. Rows of test correspond to points, columns of test correspond to features. cpre is an N-by-1 vector of the same type as nb.CLevels, and it indicates the class to which each row of test has been assigned. cpre = predict(...,'HandleMissing',val) specifies how predict treats NaN (missing values). val can be one of the following:

Construct Naïve Bayes Classifier the hard way # Calculate the Prior for Play Pplay <- table(traindata$Play) Pplay <- Pplay/sum(Pplay) # Calculate P(Sunny | Play) sunny <- table(traindata[,c("Play", "Sunny")]) sunny <- sunny/rowSums(sunny)

# Calculate P(Hot | Play) hot <- table(traindata[,c("Play", "Hot")]) hot <- hot/rowSums(hot) # and Calculate P(Windy | Play) windy <- table(traindata[,c("Play", "Windy")]) windy <- windy/rowSums(windy)

# Evaluate testdata Pyes <- sunny["Yes","Yes"] * hot["Yes","No"] * windy["Yes","Yes"] Pno <- sunny["No","Yes"] * hot["No","No"] * windy["No","Yes"] # Do we play or not? Max(Pyes, Pno)

Introduction to Matlab

MATLAB stands for Matrix Laboratory. Matlab had many functions and toolboxes to help in various applications It allows you to solve many technical computing problems, especially those with matrix and vector formulas, in a fraction of the time it would take to write a program in a scalar non-interactive language such as C or Fortran. Introduction

Introduction The MATLAB System MATLAB system consists of these main parts: • Desktop Tools and Development Environment • Includes the MATLAB desktop and Command Window, an editor and debugger, a code analyzer, browsers for viewing help, the workspace, files, and other tools • Mathematical Function Library • vast collection of computational algorithms ranging from elementary functions, like sine, cosine, and complex arithmetic, to more sophisticated functions like matrix inverse, matrix eigenvalues, Bessel functions, and fast Fourier transforms.

Introduction • The Language • The MATLAB language is a high-level matrix/array language with control flow statements, functions, data structures, input/output, and object-oriented programming features. • Graphics • MATLAB has extensive facilities for displaying vectors and matrices as graphs, as well as editing and printing these graphs. It also includes functions that allow you to customize the appearance of graphics as well as build complete graphical user interfaces on your MATLAB applications. • External Interfaces • The external interfaces library allows you to write C and Fortran programs that interact with MATLAB.

Main Matlab Window

Working with Matrices and Arrays • Since Matlab makes extensive use of matrices, the best way for you to get started with MATLAB is to learn how to handle matrices. • Separate the elements of a row with blanks or commas. • Use a semicolon ; to indicate the end of each row. • Surround the entire list of elements with square brackets, [ ]. A = [16 3 2 13; 5 10 11 8; 9 6 7 12; 4 15 14 1]

Working with Matrices and Arrays • MATLAB displays the matrix you just entered: A = 16 3 2 13 5 10 11 8 9 6 7 12 4 15 14 1 • Once you have entered the matrix, it is automatically remembered in the MATLAB workspace. You can simply refer to it as A. • Keep in mind, variable names are case-sensitive

Introduction • When you do not specify an output variable, MATLAB uses the variable ans, short for answer, to store the results of a calculation. • Subscripts The element in row i and column j of A is given by A(i,j). So to compute the sum of the elements in the fourth column of A, we have: A(1,4) + A(2,4) + A(3,4) + A(4,4) Which produces: ans = 34

Numbers • The Colon Operator • For example: 1:10 is a row vector containing the integers from 1 to 10: 1 2 3 4 5 6 7 8 9 10 • To obtain non-unit spacing, specify an increment. For example: 100:-7:50 will give you 100 93 86 79 72 65 58 51 • Subscript expressions involving colons refer to portions of a matrix. For example: A(1:k,j) refers to the first k elements of the jth column of A.

Numbers MATLAB uses conventional decimal notation, with an optional decimal point and leading plus or minus sign, for numbers. Scientific notation uses the letter e to specify the power. Imaginary numbers use either i or j as a suffix. Examples of legal numbers are: 3 -99 0.0001 9.6397238 1.60210e-20 6.02252e23 1i -3.14159j 3e5i MATLAB software stores the real and imaginary parts of a complex number.

The Load Function and M-Files The load function reads binary files containing matrices generated by earlier MATLAB sessions, or reads text files containing numeric data. • M-Files You can create your own programs using M-files, which are plain text files containing MATLAB code. Use the MATLAB Editor or another text editor to create a file containing the same statements you would type at the MATLAB command line. Save the file under a name that ends in .m

Arrays Arithmetic operations on arrays are done element by element. This means that addition and subtraction are the same for arrays and matrices, but that multiplicative operations are different. MATLAB uses a dot, or decimal point, as part of the notation for multiplicative array operations. Example: A.*A the result is an array containing the squares of the integers ans = 256 9 4 169 25 100 121 64 81 36 49 144 16 225 196 1

Multivariate Data MATLAB uses column-oriented analysis for multivariate statistical data. Each column in a data set represents a variable and each row an observation. The (i,j)th element is the ith observation of the jth variable. As an example, consider a data set with three variables: • Heart rate • Weight • Hours exercise per week For five observations, the resulting array might look like • D = [ 72 134 3.2 81 201 3.5 69 156 7.1 82 148 2.4 75 170 1.2 ]

Numbers • Now you can apply MATLAB analysis functions to this data set. For example, to obtain the mean and standard deviation of each column, use mu = mean(D), sigma = std(D) mu = 75.8 161.8 3.48 sigma = 5.6303 25.499 2.2107 • Entering Long Statements If a statement does not fit on one line, use an ellipsis (three periods), ... , followed by Return or Enter to indicate that the statement continues on the next line. For example, s = 1 -1/2 + 1/3 -1/4 + 1/5 - 1/6 + 1/7 ... - 1/8 + 1/9 - 1/10 + 1/11 - 1/12;

Graphics • MATLAB provides a variety of techniques to display data graphically. • Interactive tools enable you to manipulate graphs to achieve results that reveal the most information about your data. • You can also edit and print graphs for presentations, or export graphs to standard graphics formats for presentation in Web browsers or other media.

Basic Plotting Functions • The plot function has different forms, depending on the input arguments. • If y is a vector, plot(y) produces a piecewise graph of the elements of (y) versus the index of the elements of (y). • If you specify two vectors as arguments, plot(x,y) produces a graph of y versus x. • You can also label the axes and add a title, using the ‘xlabel’, ‘ylabel’, and ‘title’ functions. Example: xlabel('x = 0:2\pi') ylabel('Sine of x') title('Plot of the Sine Function','FontSize',12)

Plotting Multiple Data Sets in One Graph • Multiple x-y pair arguments create multiple graphs with a single call to plot. For example: x = 0:pi/100:2*pi; y = sin(x); y2 = sin(x-.25); y3 = sin(x-.5); plot(x,y,x,y2,x,y3)

Displaying Multiple Plots in One Figure subplot(m,n,p) This splits the figure window into an m-by-n matrix of small subplots and selects the pth subplot for the current plot. • Example: t = 0:pi/10:2*pi; [X,Y,Z] = cylinder(4*cos(t)); subplot(2,2,1); mesh(X) subplot(2,2,2); mesh(Y) subplot(2,2,3); mesh(Z) subplot(2,2,4); mesh(X,Y,Z)

Controlling the Axes • Setting Axis Limits & Grids The axis command lets you to specify your own limits: axis([xminxmaxyminymax]) You can use the axis command to make the axes visible or invisible: axis on / axis off The grid command toggles grid lines on and off: grid on / grid off

Knowledge-based systems