Logistic Regression is a function approximation algorithm that uses training data to directly estimate P(Y|X). In this sense, Logistic Regression is often referred to as a discriminative classifier because we can view the distribution P(Y|X) as directly discriminating the value of the target value Y for any given instance X.
We estimate probabilities for Y=0 and Y=1 given X with the following functions:
or, by removing the logistic function:
from the equation, we can use:
We might also consider how logistic regression equations are linear in the logit scale:
We'll let Y=1 indicate spam.
The Bias term, B0, encodes the prior probability of each class -- the bigger the term, the more likely we expect to see a Y=1 (i.e. spam).
If we have an example of X={Mother,Work,Viagra,Mother} then we use the bias term and add the weights of each word, mutiplying each feature by its frequency if it appears more than once:
So how do we find the weights of the coefficients that maximize the conditional likelihood? Take derivitives and gradients and use gradient descents of beta that maximize conditional likelihood.
CS229 has a detailed explanation of this in its notes
Here's an excerpt (but read the link above for full details):
For our purposes, just consider that higher weights mean that the feature is more important for a specific class.
Another motivating example from mathematicalmonk's great video
shows that the feature fector includes the weights of each feature that may lead to death by cholestoral.
We need to pass the equation through a sigmoid function to keep it between 0 and 1:
We use the standard logistic function here.
- An exponential function makes big things small, a logarithmic function makes small things big
- The logistic function is just one over 1+ the exponential function.
- It looks like an S and allows us to keep the answer always between a 0 and a 1:
Logistic Regression is a linear classifier over X. The linear classifiers produced by Logistic Regression and Gaussian Naive Bayes are identical in the limit as the number of training examples approaches infinity, provided the Naive Bayes assumptions hold. However, if these assumptions do not hold, the Naive Bayes bias will cause it to perform less accurately than Logistic Regression, in the limit. Put another way, Naive Bayes is a learning algorithm with greater bias, but lower variance, than Logistic Regression. If this bias is appropriate given the actual data, Naive Bayes will be preferred. Otherwise, Logistic Regression will be preferred. (source: Tom Mitchell)
Note: In this class, a brief overview of the theoretical underpinnings is provided, along with links to the complete derivations and detailed mathematical concepts underlying the technique. This "NTK" (Need-To-Know) philosophy will extend to additional machine learning techniques. In applied data science, we want to know a) how to apply a technique, b) how to evaluate the efficacy of a technique and c) how a technique 'works'. The 'why' it works, as well as the details and mathematics, can be left as references to seek out on an as-needed basis.
| Title | Author | Type | Length | Difficulty | Description | Rating (1 to 4 Stars) | ----- | ----- | ---- | ----- | ------ | --- | --- | --- | | Digging Into Data - Logistic Regression ( Video and Slides )| Jordan Boyd-Graber | Video, PDF | 30 mins | Easy | Worth a skim to get an easy to understand overview | +++ | Logistic Regression | Mathematical Monk | Video | 10 mins | Easy | Very quick Kahn-style intro to logistic with a good motivating example | +++ | Logistic Regression Chapter | C Shalizi, CMU | PDF | 8 Pages | Advanced | Mathematical foundation for classification by logistic regression including maximizing likelihood discussion and derivation. Good source for mathematical foundation | ++++ | Logistic Regression with Python | YHat | Exercise/Tutorial | Estimated 1.5 hours | Medium | A good example to work through using statsmodels | ++++