Gradient descent linear regression python
|
In this tutorial you can learn how the gradient descent algorithm works and implement it from scratch in python. First we look at what linear regression is, then we define the loss function. We learn how the gradient descent algorithm works and finally we will implement it on a given data set and make predictions. Show  This is the written version of this video. Watch it if you prefer that! Linear RegressionIn statistics, linear regression is a linear approach to modelling the relationship between a dependent variable and one or more independent variables. Let X be the independent variable and Y be the dependent variable. We will define a linear relationship between these two variables as follows: This is the equation for a line that you studied in high school. m is the slope of the line and c is the y intercept. Today we will use this equation to train our model with a given dataset and predict the value of Y for any given value of X. Our challenge today is to determine the value of m and c, such that the line corresponding to those values is the best fitting line or gives the minimum error. Loss FunctionThe loss is the error in our predicted value of m and c. Our goal is to minimize this error to obtain the most accurate value of m and c.
Here yᵢ is the actual value and ȳᵢ is the predicted value. Lets substitute the value of ȳᵢ: So we square the error and find the mean. hence the name Mean Squared Error. Now that we have defined the loss function, lets get into the interesting part — minimizing it and finding m and c. The Gradient Descent AlgorithmGradient descent is an iterative optimization algorithm to find the minimum of a function. Here that function is our Loss Function. Understanding Gradient Descent Imagine a valley and a person with no sense of direction who wants to get to the bottom of the valley. He goes down the slope and takes large
steps when the slope is steep and small steps when the slope is less steep. He decides his next position based on his current position and stops when he gets to the bottom of the valley which was his goal.
Dₘ is the value of the partial derivative with respect to m. Similarly lets find the partial derivative with respect to c, Dc : 3. Now we update the current value of m and c using the following equation: 4. We repeat this process until our loss function is a very small value or ideally 0 (which means 0 error or 100% accuracy). The value of m and c that we are left with now will be the optimum values. Now going back to our analogy, m can be considered the current position of the person. D is equivalent to the
steepness of the slope and L can be the speed with which he moves. Now the new value of m that we calculate using the above equation will be his next position, and L×D will be the size of the steps he will take. When the slope is more steep (D is more) he takes longer steps and when it is less steep (D is less), he takes smaller steps. Finally he arrives at the bottom of the valley which corresponds to our loss
= 0. Implementing the ModelNow let’s convert everything above into code and see our model in action ! 1.4796491688889395 0.10148121494753726Can we use gradient descent for linear regression?Gradient Descent Algorithm gives optimum values of m and c of the linear regression equation. With these values of m and c, we will get the equation of the best-fit line and ready to make predictions.
Does Sklearn use gradient descent for linear regression?A Linear Regression model converging to optimum solution using Gradient Descent. However, the sklearn Linear Regression doesn't use gradient descent.
How do you use gradient descent in Python?To find the w at which this function attains a minimum, gradient descent uses the following steps:. Choose an initial random value of w.. Choose the number of maximum iterations T.. Choose a value for the learning rate η∈[a,b]. Repeat following two steps until f does not change or iterations exceed T. a.Compute: Δw=−η∇wf(w) b.. Is gradient descent a regression model?Gradient descent is an algorithm that approaches the least squared regression line via minimizing sum of squared errors through multiple iterations. So far, I've talked about simple linear regression, where you only have 1 independent variable (i.e. one set of x values).
|
