guideofcasinos.com

Home

   Casino game
   List of casinos
   Sports book
   Baccarat
   Blackjack
   Numbers game
   Slot machine
   Straperlo
   Totalisator
   Video Lottery Terminal
   Video poker
   Golden Palace Poker
   Bet exchange
   Roulette
   Russian roulette
   Croupier
   Casino Night
   Casinos
   Lottery machine

Links

Sponsored Links
Regression analysis	Regression Analyses	Analysis

Regression analysis

Regression analysis is any statistical method where the mean of one or more random variables is predicted conditioned on other (measured) random variables. In particular, there are linear regression, logistic regression, Poisson regression and supervised learning. Regression analysis is more than curve fitting (choosing a curve that best fits given data points): it involves fitting a model with both deterministic and stochastic components. The deterministic component is called the predictor and the stochastic component is called the error term.

Sometimes there are only two variables, one of which is called X and can be regarded as constant, i.e., non-random, because it can be measured without substantial error and its values can even be chosen at will. For this reason it is called the independent or controlled variable. The other variable called Y, is a random variable called the dependent variable, because its values depend on X. In regression we are interested in the variation of Y on X.

Typical examples are the dependence of the blood pressure Y on the age X of a person, or the dependence of the weight Y of certain animals on their daily ration of food X. This dependence is called the regression of Y on X.

Regression is usually posed as an optimization problem as we are attempting to find a solution where the error is at a minimum. The most common error measure that is used is the least squares: this corresponds to a Gaussian likelihood of generating observed data given the (hidden) random variable. In a certain sense, least squares is an optimal estimator: see the Gauss-Markov theorem.

The optimization problem in regression is typically solved by algorithms such as the gradient descent algorithm, the Gauss-Newton algorithm, and the Levenberg-Marquardt algorithm. Probabilistic algorithms such as RANSAC can be used to find a good fit for a sample set, given a parametrized model of the curve function.

Regression can be expressed as a maximum likelihood method of estimating the parameters of a model. However, for small amounts of data, this estimate can have high variance. Bayesian methods can also be used to estimate regression models. A prior is placed over the parameters, which incorporates everything known about the parameters. (For example, if one parameter is known to be non-negative a non-negative distribution can be assigned to it.) A posterior distribution is then obtained for the parameter vector. Bayesian methods have the advantages that they use all the information that is available and they are exact, not asymptotic, and thus work well for small data sets. Some practitioners use maximum a posteriori (MAP) methods, a simpler method than full Bayesian analysis, in which the parameters are chosen that maximize the posterior. MAP methods are related to Occam's Razor: there is a preference for simplicity among a family of regression models (curves) just as there is a preference for simplicity among competing theories.

1 General formulation
2 Linear regression
3 The Gauss-Markov linear model
4 Least-squares estimation of the coefficients
5 Example
6 See also
7 References
8 External links

General formulation

We want to predict the values of a random variable Y conditioned on other random variables called factors. Let $p\in\mathbb{N}^*$ be the number of factors used for this prediction.

$(\Omega,\mathcal{A}, P)$ will denote a probability space and $(Γ, S)$ will be a measure space such that $(Γ, + ,.)$ is an ordered field (e.g. $\Gamma = \mathbb{R}^n$ and $S=\mathcal{B}_n$ with $n\in\mathbb{N}^*$ ). We can now define the dependent variable $Y:(\Omega,\mathcal{A})\rightarrow(\Gamma, S)$ and $\forall i\in \{1,\cdots,p\}, X_i:(\Omega,\mathcal{A})\rightarrow(\Gamma, S)$ . Now, let $F$ be a set of functions defined on $Ω$ with values in $Γ$ such that $Y,X_1,\cdots,X_p\in F$ and $d$ be a metric such that $(F, d)$ is a complete metric space.

We are looking for a measurable function $f:(\Gamma^p,S^p)\rightarrow(\Gamma,S)$ such that $d(\omega\mapsto Y(\omega),\omega\mapsto f(X_1(\omega),\cdots,X_p(\omega))$ is minimal.

Linear regression

Linear regression is the most common case in practice. We suppose that the function $f$ depends linearly on the covariates $X_1,\cdots,X_p$ so we are really just looking for the right coefficients.

Let $Θ$ be a set of coefficients. The hypothesis of the linear regression is:

$\exists (\beta^0,\cdots,\beta^p)\in\theta^{p+1}:$

$\mathbb{E}(Y|X_1,\cdots,X_p)=\beta^0 + \sum_{j=1}^p \beta^j X_j$

and the metric used is:

$\forall f,g\in F, d(f,g) = \mathbb{E}[(f-g)^2]$

We therefore want to minimize $\mathbb{E}[(Y-f(X_1,\cdots,X_p))^2]$ , which means that

$f(X_1,\cdots,X_p)=\mathbb{E}(Y|X_1,\cdots,X_p) = \beta^0 + \sum_{j=1}^p \beta^j X_j$ .

Hence, we only need to find $\beta^0,\cdots,\beta^p$ .

In order to solve this problem efficiently, several methods exist. The most common one is the Gauss-Markov method, but it requires extra hypotheses.

The Gauss-Markov linear model

Under assumptions which are met relatively often, there exists an optimal solution to the linear regression problem. These assumptions (called Gauss-Markov hypothesis) are:

We use the linear regression model, $n\in\mathbb{N}^*, n\ge p$ $Y=(Y_1,\cdots,Y_n)$ and $\forall j\in\{1,\cdots,p\},X_j=(X_{1j},\cdots,X_{nj})$ $\Gamma = \mathbb{R}^n, S=\mathcal{B}_n$ We then define the error $\varepsilon := Y - \mathbb{E}(Y|(X_1,\cdots,X_p))\sim\mathcal{N}(0,\sigma^2 I)$ independent where $\sigma^2<+\infty$ and $I$ is the $n\times n$ identity matrix. $(\beta^0,\cdots,\beta^p)\in\Theta^{p+1}$

Least-squares estimation of the coefficients

We want an estimate of $\beta=(\beta^0,\cdots,\beta^p)$ . Under the Gauss-Markov assumptions, there exists an optimal solution. We can see the unknown function $f = \mathbb{E}(Y|(X_1,\cdots,X_p))$ as the projection of $Y$ on the subspace of $F$ generated by $(X_1,\cdots,X_p)$ . Let $\widehat{Y}:= X\beta$ , where $X$ is the matrix whose columns are $(X_1,\cdots,X_p)$ .

If we define the scalar product $\langle\bullet,\bullet\rangle$ by $\langle u,v\rangle:=u^t v$ and write $\|\bullet\|$ for the induced norm, the metric d can be written $d(f,g) = \mathbb{E}[\|f-g\|^2]$ . Minimizing this norm is equivalent to projecting orthogonally $Y$ on the subspace induced by $(X_1,\cdots,X_p)$ .

$X^t (Y-\widehat{Y}) = 0$ because the projection is orthogonal, therefore, an estimate of the unknown coefficients $\beta=(\beta^0,\cdots,\beta^p)$ is

$\widehat{\beta}:=(X^t X)^{-1} X^t Y$ . This is called the least-squares estimate of the linear regression coefficients.

How good is this estimate? Under the Gauss-Markov assumptions, the Gauss-Markov theorem states that the least-square estimation of the linear regression coefficients are the best we can do. More precisely, under the Gauss-Markov assumptions, of all unbiased estimators of the linear regression coefficients, the least-square ones are the most efficient ones.

Things look great, but no matter how attractive, this method lacks robustness: departure from the normality assumptions will corrupt the results. However, this method is the most widely used in practice, and because of the central limit theorem, for large values of $n$ , the Gauss-Markov assumptions are often met.

If the Gauss-Markov hypotheses are not met, a variety of techniques are available.

If the error term is not normal but forms an exponential family one can use generalized linear models. Other techniques include the use of weighted least squares or transforming the dependent variable using the Box-Cox transformation.

If outliers are present the normal distribution can be replaced by a t-distribution or, alternatively, robust regression methods may be used.

If the predictor is not linear a nonparametric regression or semiparametric regression or nonlinear regression may be used.

Example

The simplest example of regression is in the one dimensional case. We are given a vector of x values and another vector of y values and we are attempting to find a function such that $f (x i) = y i$ .

let $\vec{x} = \begin{pmatrix} -2 \\ -1 \\ 0 \\ 1 \\ 2 \\ \end{pmatrix}, \vec{y} = \begin{pmatrix} 5 \\ 2 \\ 1 \\ 2 \\ 5 \\ \end{pmatrix}$

Let's assume that our solution is in the family of functions defined by a 3rd degree Fourier expansion written in the form:

f (x) = a 0 / 2 + a 1 cos(x) + b 1 sin(x) + a 2 cos(2 x) + b 2 sin(2 x) + a 3 cos(3 x) + b 3 sin(3 x)

where $a i, b i$ are real numbers. This problem can be represented in matrix notation as:

$\begin{pmatrix} 1/2, & \cos(x), & \sin(x), & \cos(2x), & \sin(2x), & \cos(3x), & \sin(3x), \\ \end{pmatrix} \begin{pmatrix} a_{0} \\ a_{1} \\ b_{1} \\ a_{2} \\ b_{2} \\ a_{3} \\ b_{3} \\ \end{pmatrix} = \vec{y}$

filling this form in with our given values yields a problem in the form Xw = y

$\begin{pmatrix} 1/2 & \cos(-2) & \sin(-2) & \cos(-4) & \sin(-4) & \cos(-6) & \sin(-6)\\ 1/2 & \cos(-1) & \sin(-1) & \cos(-2) & \sin(-2) & \cos(-3) & \sin(-3)\\ 1/2 & 1 & 0 & 1 & 0 & 1 & 0\\ 1/2 & \cos(1) & \sin(1) & \cos(2) & \sin(2) & \cos(3) & \sin(3)\\ 1/2 & \cos(2) & \sin(2) & \cos(4) & \sin(4) & \cos(6) & \sin(6)\\ \end{pmatrix} . \begin{pmatrix} a_{0} \\ a_{1} \\ b_{1} \\ a_{2} \\ b_{2} \\ a_{3} \\ b_{3} \\ \end{pmatrix} = \begin{pmatrix} 5 \\ 2 \\ 1 \\ 2 \\ 5 \\ \end{pmatrix}$

This problem can now be posed as an optimization problem to find the minimum sum of squared errors.

3rd degree Fourier function

$\min_{\vec{w}} \sum_{i=1}^{n} (\vec{x_{i}}\vec{w} - y_{i})^2$

$\min_{\vec{w}} \|X\vec{w} - \vec{y}\|^2.$

solving this with least squares yields:

$\vec{w} = \begin{pmatrix} 0 \\ 4.25 \\ 0 \\ -6.13 \\ 0 \\ 2.88 \\ 0 \\ \end{pmatrix}$

thus the 3rd-degree Fourier function that fits the data best is given by:

f (x) = 4.25cos(x) - 6.13cos(2 x) + 2.88cos(3 x).

References

Audi, R., Ed. (1996) The Cambridge Dictionary of Philosophy. Cambridge, Cambridge University Press. curve fitting problem p.172-173.
David Birkes and Yadolah Dodge, Alternative Methods of Regression (1993), ISBN 0-471-56881-3
W. Hardle, Applied Nonparametric Regression (1990), ISBN 0-521-42950-1
J. Fox, Applied Regression Analysis, Linear Models and Related Methods. (1997), Sage