Linear classifier

2025-10-17T08:56:36Z

2001:720:101C:504:3850:56D9:23DD:DCB5:

{{short description|Statistical classification in machine learning}}
In [[machine learning]], a '''linear classifier''' makes a [[Statistical classification|classification]] decision for each object based on a [[linear combination]] of its [[Feature (machine learning)|features]]. A simpler definition is to say that a linear classifier is one whose decision boundaries are ''linear''. Such classifiers work well for practical problems such as [[document classification]], and more generally for problems with many variables ([[feature vector|features]]), reaching accuracy levels comparable to non-linear classifiers while taking less time to train and use.{{r|ieee}}

== Definition ==
[[Image:Svm separating hyperplanes.png|thumb|right|In this case, the solid and empty dots can be correctly classified by any number of linear classifiers. H1 (blue) classifies them correctly, as does H2 (red). H2 could be considered "better" in the sense that it is also furthest from both groups.
H3 (green) fails to correctly classify the dots.]]

If the input feature vector to the classifier is a [[real number|real]] vector <math>\vec x</math>, then the output score is

:<math>y = f(\vec{w}\cdot\vec{x}) = f\left(\sum_j w_j x_j\right),</math>

where <math>\vec w </math> is a real vector of weights and ''f'' is a function that converts the [[dot product]] of the two vectors into the desired output. (In other words, <math>\vec{w}</math> is a [[one-form]] or [[linear functional]] mapping <math>\vec x</math> onto '''R'''.) The weight vector <math>\vec w</math> is learned from a set of labeled training samples. Often ''f'' is a '''threshold function''', which maps all values of <math>\vec{w}\cdot\vec{x}</math> above a certain threshold to the first class and all other values to the second class; e.g.,

:<math>
f(\mathbf{x}) = \begin{cases}1 & \text{if }\ \mathbf{w}^T \cdot \mathbf{x} > \theta,\\0 & \text{otherwise}\end{cases}
</math>

The superscript T indicates the transpose and <math> \theta </math> is a scalar threshold. A more complex ''f'' might give the probability that an item belongs to a certain class.

For a two-class classification problem, one can visualize the operation of a linear classifier as splitting a [[High-dimensional space|high-dimensional]] input space with a [[hyperplane]]: all points on one side of the hyperplane are classified as "yes", while the others are classified as "no".

A linear classifier is often used in situations where the speed of classification is an issue, since it is often the fastest classifier, especially when <math>\vec x</math> is sparse. Also, linear classifiers often work very well when the number of dimensions in <math>\vec x</math> is large, as in [[document classification]], where each element in <math>\vec x</math> is typically the number of occurrences of a word in a document (see [[document-term matrix]]). In such cases, the classifier should be well-[[regularization (machine learning)|regularized]].

==Generative models vs. discriminative models==
There are two broad classes of methods for determining the parameters of a linear classifier <math>\vec w</math>. They can be [[Generative model|generative]] and [[Discriminative model|discriminative]] models.<ref>T. Mitchell, [https://www.cs.cmu.edu/~tom/mlbook/NBayesLogReg.pdf Generative and Discriminative Classifiers: Naive Bayes and Logistic Regression.] Draft Version, 2005</ref><ref>A. Y. Ng and M. I. Jordan. [http://www.cs.berkeley.edu/~jordan/papers/ng-jordan-nips01.ps On Discriminative vs. Generative Classifiers: A comparison of logistic regression and Naive Bayes.] in NIPS 14, 2002.</ref> Methods of the former model [[joint probability distribution]], whereas methods of the latter model [[Conditional probability distribution|conditional density functions]] <math>P({\rm class}|\vec x)</math>. Examples of such algorithms include:
* [[linear discriminant analysis|Linear Discriminant Analysis]] (LDA)—assumes [[normal distribution|Gaussian]] conditional density models
* [[Naive Bayes classifier]] with multinomial or multivariate Bernoulli event models.

The second set of methods includes [[discriminative model]]s, which attempt to maximize the quality of the output on a [[training set]]. Additional terms in the training cost function can easily perform [[regularization (machine learning)|regularization]] of the final model. Examples of discriminative training of linear classifiers include:
* [[Logistic regression]]—maximum likelihood estimation of <math>\vec w</math> assuming that the observed training set was generated by a binomial model that depends on the output of the classifier.
* [[Perceptron]]—an algorithm that attempts to fix all errors encountered in the training set
* Fisher's Linear Discriminant Analysis—an algorithm (different than "LDA") that maximizes the ratio of between-class scatter to within-class scatter, without any other assumptions. It is in essence a method of dimensionality reduction for binary classification.<ref>R.O. Duda, P.E. Hart, D.G. Stork, "Pattern Classification", Wiley, (2001). {{ISBN|0-471-05669-3}}</ref>
* [[Support vector machine]]—an algorithm that maximizes the [[Margin (machine learning)|margin]] between the decision hyperplane and the examples in the training set.

'''Note:''' Despite its name, LDA does not belong to the class of discriminative models in this taxonomy. However, its name makes sense when we compare LDA to the other main linear [[dimensionality reduction]] algorithm: [[principal components analysis]] (PCA). LDA is a [[supervised learning]] algorithm that utilizes the labels of the data, while PCA is an [[unsupervised learning]] algorithm that ignores the labels. To summarize, the name is a historical artifact.<ref>{{Cite book |last1=Duda |first1=Richard O. |title=Pattern classification |last2=Hart |first2=Peter E. |last3=Stork |first3=David G. |date=2001 |publisher=John Wiley & Sons, Inc |isbn=978-0-471-05669-0 |edition=Second |series=A Wiley-Interscience publication |location=New York Chichester Weinheim Brisbane Singapore Toronto |pages=117}}</ref>

Discriminative training often yields higher accuracy than modeling the conditional density functions{{Citation needed|date=October 2017}}. However, handling missing data is often easier with conditional density models{{Citation needed|date=October 2017}}.

All of the linear classifier algorithms listed above can be converted into non-linear algorithms operating on a different input space <math>\varphi(\vec x)</math>, using the [[kernel trick]].

===Discriminative training===
Discriminative training of linear classifiers usually proceeds in a [[supervised learning|supervised]] way, by means of an [[optimization algorithm]] that is given a training set with desired outputs and a [[loss function]] that measures the discrepancy between the classifier's outputs and the desired outputs. Thus, the learning algorithm solves an optimization problem of the form<ref name="ieee">{{cite journal |author1=Guo-Xun Yuan |author2=Chia-Hua Ho |author3=Chih-Jen Lin |title=Recent Advances of Large-Scale Linear Classification |journal=Proc. IEEE |volume=100 |issue=9 |year=2012|url=http://dmkd.cs.vt.edu/TUTORIAL/Bigdata/Papers/IEEE12.pdf |archive-url=https://web.archive.org/web/20170610105707/http://dmkd.cs.vt.edu/TUTORIAL/Bigdata/Papers/IEEE12.pdf |archive-date=2017-06-10 |url-status=live}}</ref>

:<math>\underset{\mathbf{w}}{\arg\min} \;R(\mathbf{w}) + C \sum_{i=1}^N L(y_i, \mathbf{w}^\mathsf{T} \mathbf{x}_i)</math>

where

* {{math|'''w'''}} is a vector of classifier parameters,
* {{math|''L''(''yi'', '''w'''T'''x'''''i'')}} is a loss function that measures the discrepancy between the classifier's prediction and the true output {{mvar|yi}} for the {{mvar|i}}'th training example,
* {{math|''R''('''w''')}} is a [[Regularization (mathematics)|regularization]] function that prevents the parameters from getting too large (causing [[overfitting]]), and
* {{mvar|C}} is a scalar constant (set by the user of the learning algorithm) that controls the balance between the regularization and the loss function.

Popular loss functions include the [[hinge loss]] (for linear SVMs) and the [[log loss]] (for linear logistic regression). If the regularization function {{mvar|R}} is [[convex function|convex]], then the above is a [[convex optimization|convex problem]].{{r|ieee}} Many algorithms exist for solving such problems; popular ones for linear classification include ([[Stochastic gradient descent|stochastic]]) [[gradient descent]], [[L-BFGS]], [[coordinate descent]] and [[Newton method]]s.

== See also ==
* [[Backpropagation]]
* [[Linear regression]]
* [[Perceptron]]
* [[Quadratic classifier]]
* [[Support vector machines]]
* [[Winnow (algorithm)]]

== Notes ==
<references/>

== Further reading ==
# Y. Yang, X. Liu, "A re-examination of text categorization", Proc. ACM SIGIR Conference, pp. 42–49, (1999). [http://citeseer.ist.psu.edu/yang99reexamination.html paper @ citeseer]
# R. Herbrich, "Learning Kernel Classifiers: Theory and Algorithms," MIT Press, (2001). {{ISBN|0-262-08306-X}}

{{DEFAULTSORT:Linear Classifier}}
[[Category:Classification algorithms]]
[[Category:Statistical classification]]

Vero - Wikipedia - User contributions [en]

Linear classifier