Different Types of Regression Analysis – A Basic Guide

Introduction

The term regression is used to indicate the estimation or prediction of the average value of one variable for a specified value of another variable. And Regression Analysis is a statistical tool used to estimate the relationship between a dependent variable and an independent variable. For example, If a Manger of a firm wants the exact relationship between advertisement expenditure and sales for future planning then the regression technique will be most suitable for him. 

There are different types of regression analysis, let’s talk about it in more details:- 

1. Linear Regression

Linear regression is a type of model where the relationship between an independent variable and a dependent variable is assumed to be linear. The estimate of variable “y” is obtained from an equation, y’- y_bar = byx(x-x_bar)……(1) and estimate of variable “x” is obtained through the equation x’-x_bar = bxy(y-y_bar)…..(2). The graphical representation of linear equations on (1) & (2) is known as Regression lines. These lines are obtained through the Method of Least Squares. 

There are two kinds of Linear Regression Model:-

• Simple Linear Regression: A linear regression model with one independent and one dependent variable.
• Multiple Linear Regression: A linear regression model with more than one independent variable and one dependent variable.

Assumptions of Linear Regression

• Sample size : n = 20 (cases per independent variable)
• Heteroscedasticity is absent —
• Linear Relationships exist between the variables.
• Independent Sample observations.
• No multicollinearity & auto-correlation
• Independent Sample observations.

2. Polynomial Regression

It is a type of Regression analysis that models the relationship of values of the Dependent variable “x” and Independent variables “y’’ as non-linear. It is a special case of Multiple Linear Regression even though it fits a non-linear model to data. It is because data may be correlated but the relationship between two variables might not look linear. 

3. Logistic Regression

Logistic Regression is a method that was used first in the field of Biology in the 20th century. It is used to estimate the probability of certain events that are mutually exclusive, for example, happy/sad, normal/abnormal, or pass/fail. The value of probability strictly ranges between 0 and 1.

4. Quantile Regression

Quantile Regression is an econometric technique that is used when the necessary conditions to use Linear Regression are not duly met. It is an extension of Linear Regression analysis i.e., we can use it when outliers are present in data as its estimates strong against outliers as compared to linear regression.

5. Ridge Regression

To understand Ridge Regression we first need to get through the concept of Regularization. 

Regularization: There are two types of Regularization, L1 regularization & L2 regularization. L1 regularization adds an L1 penalty equal to the value of coefficients to restrict the size of coefficients, which leads to the removal of some coefficients. On the other hand, L2 regularization adds a penalty L2 which is equal to the square of coefficients. 

Using the above method Regularization solves the problem of a scenario where the model performs well on training data but underperforms on validation data.

6. Lasso Regression 

LASSO (Least Absolute Shrinkage and Selection Operator) is a regression technique that was introduced first in geophysics. The term “Lasso” was coined by Professor Robert Tibshirani. Just like Ridge Regression, it uses regularization to estimate the results. Plus it also uses variable selection to make the model more efficient.

7. Elastic Net Regression 

Elastic net regression is favoured over ridge and lasso regression when one has to deal with exceedingly correlated independent variables.

8. Principle components regression (PCR)

Principle components regression technique which is broadly used when one has various independent variables. The technique is used for assuming the unknown regression coefficient in a standard linear regression model. The technique is divided into two steps, 

1. Obtaining the principal components

2. Go through the regression Analysis on Principle components.

9. Partial least regression (PCR)

It is a substitute technique of principal components regression when one has a widely correlated independent variable. The technique is helpful when one has many independent variables. Partial least regression is widely used in the chemical, drug, food, and plastic industry.

10. Support Vector Regression

Support vector regression can be used to solve both linear and nonlinear models. Support vector regression has been determined to be productive to be an effective real-value function estimation.

11. Ordinal Regression 

Ordinal regression is used to foreshow ranked values. The technique is useful when the dependent variable is ordinal. Two examples of Ordinal regression are Ordered Logit and ordered probit.

12. Poisson Regression 

Poisson Regression is used to foreshow the number of calls related to a particular product on customer care. Poisson regression is used when the dependent variable has a calculation. Poisson regression is also known as the log-linear model when it is used to model contingency tablets. Its dependent variable y has Poisson distribution.

13. Negative Binomial Regression

Similar to Poisson regression, negative Binomial regression also accord with count data, the only difference is that the Negative Binomial regression does not predict the distribution of count that has variance equal to its mean.

14. Quasi Poisson Regression

Quasi Poisson Regression is a substitute for negative Binomial regression. The technique can be used for overdispersed count data.

15. Cox Regression 

Cox Regression is useful for obtaining time-to-event data. It shows the effect of variables on time for a specific period. Cox Regression is also known as proportional Hazards Regression.

16. Tobit Regression

Tobit Regression is used to Evaluate linear relationships between variables when censoring ( observing independent variable for all observation) exists in the dependent variable. The value of the dependent is reported as a single value.

APPLICATION OF REGRESSION ANALYSIS

• Forecasting: Different types of regression analysis can be used to forecast future opportunities and threats for a business. For instance, a customer’s likely purchase volume can be predicted using a demand analysis.

However, when it comes to business, demand isn’t the only variable that affects profitability. 

• Comparison with competition: A company’s financial performance can be compared to that of a specific competitor using this tool. Also, it can be used to determine the correlation between the stock prices of two different companies within the same industry or different industries. 

When compared to a rival company, it can help identify which factors are influencing its sales. It can help small businesses achieve rapid success in a short term.

• Problem Identification: In addition to providing factual evidence, a regression can be used to identify and correct judgment errors. For example, a retail shop owner may believe that extending the hours of operation will result in a significant increase in sales. However, regression analysis shows that the monetary gains as a result of increasing the working hours are not enough to offset the increase in operational costs that comes along with it.

Regression analysis may provide the business owners with quantitative support for their decisions and prevent them from making mistakes because of their intuition.

• Decision Making: Regression analysis (and other types of statistical analysis) are now being used by many businesses and their top executives to make better business decisions and reduce guesswork and intuition.

Scientific management is made possible by regression. Data overload is a problem for both small and large organizations. To make the best decisions possible, managers can use regression analysis to sort through data and select relevant factors.

Conclusion

The types of regression analysis are listed above but choosing a correct regression model is a tough grind. It requires vast knowledge about statistical tools and their application. The correct method was chosen based on the nature of the variable, data, and the model itself. Overall the different types of Regression Analysis have calculated discrete and distinct data very easily in the recent, not only in the field of mathematics/statistics but it has many applications in the real world as well. Hence, Regression analysis is a boon for mankind.

If you are interested in making a career in the Data Science domain, our 11-month in-person Post Graduation in Data Science course can help you immensely in becoming a successful Data Science professional. 

ALSO, READ

• What is Data Visualization In 2020?