{"id":214,"date":"2017-10-04T10:46:49","date_gmt":"2017-10-04T10:46:49","guid":{"rendered":"http:\/\/www.nullplug.org\/ML-Blog\/?p=214"},"modified":"2017-11-02T07:37:22","modified_gmt":"2017-11-02T07:37:22","slug":"linear-regression","status":"publish","type":"post","link":"https:\/\/www.nullplug.org\/ML-Blog\/2017\/10\/04\/linear-regression\/","title":{"rendered":"Linear Regression"},"content":{"rendered":"

\n Prediction is very difficult, especially about the future. – Niels Bohr<\/a>\n<\/p><\/blockquote>\n

The problem<\/h2>\n

Suppose we have a list of vectors (which we can think of as samples) $x_1, \\cdots, x_m\\in \\Bbb R^n$ and a corresponding list of output scalars $y_1, \\cdots, y_m \\in \\Bbb R$ (which we can regard as a vector $y\\in \\Bbb R^m$). We want to find a vector $\\beta=(\\beta_1,\\cdots \\beta_n)\\in \\Bbb R^n$ such that the sum of squared error<\/em> $$ \\sum_{i=1}^m (x_i \\cdot \\beta – y_i)^2 $$ is as small as possible. Alternatively, if we suppose that our pairs $(x_i, y_i)$ arise from a function $f(x_i)=y_i$, we want to find the closest approximation to $f$ by a linear function of the form $\\widehat{f}(a)=\\sum_{i=1}^n \\beta_i \\pi_i(a)$, where $\\pi_i \\colon \\Bbb R^n\\to \\Bbb R$ is the projection to the $i$th coordinate.<\/p>\n

Let’s arrange the input vectors $x_1,\\cdots, x_m$ into an $m\\times n$-matrix $A$, where $x_i$ appears as the $i$th row. Then we have a matrix equation of the form $ A \\beta = \\widehat{y}$ and we want to find $\\beta$ such that $||\\widehat{y}-y||^2$ is as small as possible1<\/a><\/sup>.<\/p>\n

<\/p>\n

Let’s try to use calculus. Here we are trying to minimize:
\n$$ SSE(\\beta) = \\lVert\\widehat{y}-y\\rVert^2=(A\\beta -y)^T(A\\beta -y).$$ Differentiating with respect to $\\beta$ (see Tensor Calculus<\/a>) yields
\n\\begin{align}
\n \\frac{dSSE(\\beta)}{d\\beta} & = 2(A\\beta -y)^TA \\label{eq:der}.
\n\\end{align}
\nDifferentiating with respect to $\\beta$ again yields $2A^TA$ which will be a positive semidefinite matrix. This matrix will be positive definite if and only if $\\ker A =0$2<\/a><\/sup>.<\/p>\n

Now $\\ker A \\neq 0$ if and only if one of the columns of $A$ can be expressed as a linear combination of the others. In terms of our original problem, this means that one of the components of our input vectors is in fact expressible as a linear combination of the others, so there is a hidden dependency amongst the variables.<\/p>\n

Exercise<\/h4>\n

Perform the above calculation yourself while being careful about what are vector valued quantities.<\/p>\n

Let’s suppose $\\ker A\\neq 0$ (we can arrange this anyway by dropping dependent columns). So our matrix is positive definite and we see that any critical value of $SSE(\\beta)$ in \\eqref{eq:der} is a minimum. So by setting \\eqref{eq:der} to 0 and solving for $\\beta$ (while observing that $A^TA$ is non-singular, given our assumptions) we get:
\n\\begin{equation}
\n\\beta=(A^TA)^{-1}A^Ty.\\label{eq:solv}
\n\\end{equation}
\nThis value of $\\beta$ is uniquely determined and hence yields the global minimum of $SSE(\\beta)$.<\/p>\n

Note that when $A$ is an invertible square matrix, then \\eqref{eq:solv} reduces to $\\beta = A^{-1} y$, which will have $SSE(\\beta)=0$; just as we should expect from linear algebra.<\/p>\n

Implementation concerns<\/h3>\n

Using a naive implementation of Cramer’s rule<\/a> to invert an $n\\times n$ matrix will take $O(n!)$ time and is totally impractical for problems of interest. While using Gaussian elimination to invert an $n\\times n$ matrix takes $O(n^3)$-time (by being clever<\/a> one can cut the runtime in half for inverting the symmetric matrix $A^TA$). The remaining multiplications require $O(n^2m)$ time and since $m\\geq n$ (otherwise $\\ker A\\neq 0$), the dominant term will be the multiplications so the algorithm runs in $O(n^2m)$-time.<\/p>\n

We may also be concerned about errors in the data. Suppose that the “correct” value of $y$ is, in fact $y+e$. For simplicity suppose that $A$ is invertible. We might hope that if $\\lVert e\\rVert$ is very small relative to $\\lVert y\\rVert$ then the error between $A^{-1}y$ and $A^{-1}(y+e)$ will be small relative to $A^{-1}y$.
\nWe can calculate
\n\\begin{align}
\n\\frac{\\lVert A^{-1} e \\rVert}{\\lVert A^{-1} y \\rVert}\/\\frac{\\lVert e\\rVert}{\\lVert y\\rVert} & = \\frac{\\lVert A^{-1} e\\rVert}{\\lVert e\\rVert }\\cdot \\frac{\\lVert y\\rVert }{\\lVert A^{-1}y\\rVert } \\label{eq:cond}
\n\\end{align}
\nAs we try to maximize this expression over all $e$ and $y$ (so we can replace $y$ with $Ay$), we obtain the product of operator norms<\/a> $\\lVert A^{-1}\\rVert \\cdot \\lVert A \\rVert$. While we won’t go into operator norms here, we will point out that when $A$ is diagonal, then $\\lVert A\\rVert$ is the maximum of the absolute values on the diagonal entries, so that \\eqref{eq:cond} becomes $|M|\/|m|$ where $|M|$ is the maximum of the absolute values of the diagonal entries and $|m|$ is the minimum of the absolute values of these entries. In particular, if $A$ is the 2×2 diagonal matrix with diagonal entries $10$ and $0.1$, inversion can multiply relative errors by a factor of a 100.<\/p>\n

Typically the standard thing to do is not to invert the matrix, but rather construct a matrix decomposition from which it is easy to solve the system of equations by back substitution. Not that this would not help the numerical stability problem with the diagonal matrix.<\/p>\n

Beyond simple linear functions<\/h3>\n

At this point we have seen how to find an approximation to a function $f(x)$ of the form $\\widehat{f}(x)=\\sum_{i=1}^n \\beta_i \\pi_i(x)$ that minimizes the sum of squared errors.<\/p>\n

What happens if our function is just the constant function $f(x)=10$? In this case, our approximation will do rather poorly. In particular, $\\widehat{f}(0)=0$, so this will never be a very good approximation about this point.<\/p>\n

What we could do in this case is assume that our function instead has the form: $$\\widehat{f}(x)=\\sum_{i=1}^n \\beta_i \\pi_i(x)+\\beta_{n+1}\\cdot 1.$$ We can now solve this new approximation problem by just changing the vectors $x_i=(\\pi_1(x_i),\\cdots,\\pi_n(x_i))$ into new $n+1$-dimensional vectors $(\\pi_1(x_i),\\cdots,\\pi_n(x_i),1)$ and applying linear regression as before. For non-degenerate collections of vectors ${x_i}$, we will end up with the desired function $\\widehat{f}(x)=10$. This is sometimes called the bias trick<\/em>.<\/p>\n

Here is an example of such a linear regression being fit to more and more data points. For this example, we used python’s scikit-learn package<\/a> and rendered the images using matplotlib<\/a> (see the code<\/a>).
\n

\"\"
Fitting to more and more data points<\/figcaption><\/figure> In this case, the mean squared error approaches the Bayes error rate<\/a>.<\/p>\n

More generally if we think our $f(x)$ can be well approximated by a function of the form $\\widehat{f}(x)=\\sum_{i=1}^N \\beta_i f_i(x)$ for any<\/em> fixed set of $N$ functions $\{f_i\}$, we can just construct a new sample matrix whose $i$th row is the $N$-vector $(f_1(x_i),\\cdots,f_N(x))$ and apply linear regression.<\/p>\n

For example, if we are looking for a function $\\widehat{f}\\colon \\Bbb R\\to \\Bbb R$ of the form $\\widehat{f}(x)=\\beta_1 x^2+\\beta_2 x +\\beta_3$, we just have to replace the samples $x_1,\\cdots, x_m$ with the vectors $(x_1^2, x_1, 1), \\cdots, (x_m^2, x_m, 1)$ and apply linear regression as before. This is an example of polynomial regression<\/em>. For example, we have the following (see the code<\/a>):
\n

\"\"
High degree polynomials fit the data better<\/figcaption><\/figure> These examples were constructed as above, but we first transform the data following the above discussion using PolynomialFeatures<\/a>. Alternatively, it is very easy to just code this transformation yourself.<\/p>\n

We could similarly try to find Fourier approximations<\/a> for a function which we expect has periodic behavior. We could also use wavelets<\/a> or splines<\/a>. Clearly, this list goes on and on.<\/p>\n

Warning<\/h3>\n

If we choose a suitably expressive class of functions to define our generalized linear regression problem, then there is a good chance that the linear regression problem will fit the data perfectly. For example, if we have any function $f\\colon \\Bbb R \\to \\Bbb R$ and we know its values on $n$ distinct points, then we can perfectly approximate this function by a polynomial of degree at least $n-1$ (e.g., 2 points determine a line, 3 points determine a quadratic curve, etc.<\/a>). While this will perfectly fit our given data, it becomes quite unlikely that it will match any new data points. This is a classic example of overfitting<\/em>. We can see this phenomenon in the following example:

\"\"
This estimator has high variance<\/figcaption><\/figure> Here the degree 10 polynomial will fit any 30 points relatively well, but will drastically fail to generalize well to the entire dataset.<\/p>\n

A Maximal Likelihood Justification for using SSE<\/h3>\n

While the above exposition is elementary, one might wonder where the choice to minimize the sum of squared errors comes from.<\/p>\n

Let us suppose that that we have a function of the form $f(x)=g(x)+\\epsilon$ where $\\epsilon$ is an $\\Bbb R$ valued random variable3<\/a><\/sup> $\\epsilon \\sim N(0, \\sigma^2)$ and $g(x)$ is a deterministic function. We could mentally justify this by saying that the output variables are not recorded exactly, but there is some normally distributed measurement error or there is some additional non-linear random factors contributing to the value of $f(x)$. Then the conditional density function of $f(x)$ is $$p(y|x)=\\frac{e^{-(g(x) -y)^2\/2\\sigma^2}}{\\sqrt{2\\pi \\sigma^2}}.$$<\/p>\n

The maximal likelihood estimate<\/em> (MLE) (see here<\/a> for more details) for a vector of values $y=(y_1,\\cdots,y_m)$ associated to a sequence of independent samples $x=(x_1,\\cdots,x_m)$ is then
\n\\begin{align}
\n
\n\\end{align}<\/p>\n

So in this case, the MLE for $y$ is the choice of $y$ which minimizes the sum of squared errors.<\/p>\n

A frequentist description<\/h3>\n

If we suppose that our data samples $(x,y)$ are being drawn from a fixed distribution, e.g., they are the values of a vector valued random variable $X$ and a random variable $Y$. Moreover, suppose the joint pdf of $X$ and $Y$ is $p(x,y)$. Then we can regard linear regression as attempting to find
\n\\begin{align}
\n
\n\\end{align}<\/p>\n

Bayesian approach<\/h2>\n

Here we will consider a Bayesian approach to the polynomial regression problem. First we consider the case, where we approximate data coming from a degree 3 polynomial using a Gaussian model that expects a degree 3 polynomial and is trained on all 200 samples. In other words, we are assuming a conditional distribution of the form
\n$$p(y,x|\\theta)=1\/\\sqrt{2\\pi \\sigma^2}\\cdot e^{\\frac{(y-\\sum_{i=0}^3 \\beta_i x^i)^2}{2\\sigma^2}},$$ where $\\theta$ is the vector of parameters in this model. For our prior, we suppose that the parameters are independent random variables where $\\beta_i~N(0, 100)$ and $\\sigma$ is given by a half normal distribution<\/a> with standard deviation 1 (since this can only take non-negative values). To calculate the posterior $p(\\theta | D)$ we apply Bayes rule. Unfortunately Bayes rule has this nasty integral over the parameter space in the denominator:
\n$$ \\int_\\theta p(D|\\theta)p(\\theta)d\\theta.$$ Rather than try and calculate this integral directly, we will calculate this using Monte Carlo integration<\/a>, which is sampled using an algorithm which is completely NUTS<\/a>. All of this is implemented using the package PyMC3<\/a>. This is overkill for this situation, but it is nice to see how the general algorithm is implemented (see here<\/a> for the code).<\/p>\n

In effect what we are doing is using a special technique to generate samples $S$ of $\\theta$ according to the prior distribution $p(\\theta)$. For each $\\theta\\in S$, we calculate $z_\\theta =p(D|\\theta)$ and see that $$p(\\theta|D)\\approx \\frac{z_\\theta}{\\sum_{\\theta’ \\in S} z_{\\theta’}}.$$ As the number of samples goes up, the accuracy of this estimate increases.<\/p>\n

We can look at the output here:
\n

\"\"
A degree 3 approximation<\/figcaption><\/figure>
\nBy taking posterior samples of the parameters we obtain the curves in grey. The mean of these samples generates the black curve in the middle. Alternatively, we can sample from $p(y|x)$ (that is we average over all of the choices of parameters). Taking the mean values for each $y$ given an $x$ we obtain the red curve. We have marked a band of radius the standard deviation about these samples.<\/p>\n

The summary statistics for our parameter values are listed here. Note that the ‘true value’ of our function is $$f(x)=
\n7 x^3 – 9.1 x^2 + 3.5 x – 0.392,$$ with a standard deviation of 0.3. We note that none of the true coefficients fall into the 95% credibility intervals, which is not very promising.<\/p>\n\n\n\n\n\n\n\n\n\n
<\/th>\nmean<\/th>\nsd<\/th>\nmc_error<\/th>\nhpd_2.5<\/th>\nhpd_97.5<\/th>\n<\/tr>\n<\/thead>\n
coeff__0<\/th>\n-0.209607<\/td>\n0.080981<\/td>\n0.001872<\/td>\n-0.370318<\/td>\n-0.056474<\/td>\n<\/tr>\n
coeff__1<\/th>\n1.914993<\/td>\n0.704315<\/td>\n0.020752<\/td>\n0.562936<\/td>\n3.260617<\/td>\n<\/tr>\n
coeff__2<\/th>\n-5.556357<\/td>\n1.651007<\/td>\n0.050628<\/td>\n-8.678463<\/td>\n-2.410120<\/td>\n<\/tr>\n
coeff__3<\/th>\n4.794343<\/td>\n1.099424<\/td>\n0.033513<\/td>\n2.710534<\/td>\n6.909714<\/td>\n<\/tr>\n
sigma<\/th>\n0.291662<\/td>\n0.015164<\/td>\n0.000186<\/td>\n0.263677<\/td>\n0.321718<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

We can gain further information by studying the pdfs of the resulting variables:<\/p>\n

\"\" <\/p>\n

Now we examine how the Bayesian method does with respect to our overfitting problem above. Here we try to fit a degree 10 polynomial to only 30 data points. We then obtain the following:

\"\"
Again there is a lot of variance in the additional models.<\/figcaption><\/figure><\/p>\n

Here we see the samples for the polynomial curves again have wild variance, but our posterior predictions are actually quite reasonable. Looking at the summary statistics in the table below , we note that the model has very low confidence in almost all of the parameter values and huge credibility intervals that mostly contain the true values. By averaging over all of these parameters we reduce the variance in our predictions and obtain a much better model.<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
<\/th>\nmean<\/th>\nsd<\/th>\nmc_error<\/th>\nhpd_2.5<\/th>\nhpd_97.5<\/th>\n<\/tr>\n<\/thead>\n
coeff__0<\/th>\n-0.053526<\/td>\n0.200482<\/td>\n0.002303<\/td>\n-0.469330<\/td>\n0.317603<\/td>\n<\/tr>\n
coeff__1<\/th>\n0.658498<\/td>\n1.898380<\/td>\n0.022927<\/td>\n-3.246255<\/td>\n4.222251<\/td>\n<\/tr>\n
coeff__2<\/th>\n-2.835666<\/td>\n5.895880<\/td>\n0.084508<\/td>\n-13.723288<\/td>\n9.131853<\/td>\n<\/tr>\n
coeff__3<\/th>\n2.316211<\/td>\n8.093137<\/td>\n0.126718<\/td>\n-13.588722<\/td>\n18.071517<\/td>\n<\/tr>\n
coeff__4<\/th>\n1.342521<\/td>\n8.552999<\/td>\n0.104256<\/td>\n-15.235386<\/td>\n18.123408<\/td>\n<\/tr>\n
coeff__5<\/th>\n-0.630845<\/td>\n8.596230<\/td>\n0.099357<\/td>\n-16.968628<\/td>\n16.577238<\/td>\n<\/tr>\n
coeff__6<\/th>\n-1.251081<\/td>\n8.598991<\/td>\n0.103051<\/td>\n-17.388627<\/td>\n15.975167<\/td>\n<\/tr>\n
coeff__7<\/th>\n-0.712717<\/td>\n8.881522<\/td>\n0.102497<\/td>\n-18.123559<\/td>\n16.398549<\/td>\n<\/tr>\n
coeff__8<\/th>\n0.373434<\/td>\n8.985303<\/td>\n0.104398<\/td>\n-16.930835<\/td>\n18.813871<\/td>\n<\/tr>\n
coeff__9<\/th>\n0.753457<\/td>\n8.428446<\/td>\n0.094453<\/td>\n-16.404371<\/td>\n16.786885<\/td>\n<\/tr>\n
coeff__10<\/th>\n0.835288<\/td>\n7.222892<\/td>\n0.089429<\/td>\n-13.213493<\/td>\n15.239204<\/td>\n<\/tr>\n
sigma<\/th>\n0.276957<\/td>\n0.041169<\/td>\n0.000551<\/td>\n0.205731<\/td>\n0.360664<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Exercise<\/h4>\n

How did our choice of prior bias the predictions of the coefficients? What would be some other reasonable choices of prior?<\/p>\n

\n
\n
    \n
  1. \nNote that minimizing the sum is equivalent to minimizing the average. ↩<\/a>\n<\/li>\n
  2. \nRecall that on $\\Bbb R^m$ we have a scalar product $\\langle v,w\\rangle=v^Tw$ (here we view vectors as column vectors). This formula tells us that $$\\langle v,Ax\\rangle =v^TAx=x^TA^Tv=\\langle x,A^Tv\\rangle =\\langle A^Tv,x\\rangle $$ (note that we used that every 1×1 matrix is symmetric). This implies that $$\\langle x, A^TA x\\rangle =\\langle Ax, Ax \\rangle \\geq 0$$ for all $x$. Moreover, this quantity is zero if and only if $Ax=0$. ↩<\/a>\n<\/li>\n
  3. \nSo our function $f$ is of the form $f\\colon \\Bbb R^n\\times \\Omega \\to \\Bbb R$ where $f(x,-)\\colon \\Omega \\to \\Bbb R$ is a measurable function on some probability which is distributed normally with mean $g(x)$ and variance $\\sigma^2$.
    \n     ↩<\/a>\n<\/li>\n<\/ol>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

    Prediction is very difficult, especially about the future. – Niels Bohr The problem Suppose we have a list of vectors (which we can think of as samples) $x_1, \\cdots, x_m\\in \\Bbb R^n$ and a corresponding list of output scalars $y_1, \\cdots, y_m \\in \\Bbb R$ (which we can regard as a vector $y\\in \\Bbb R^m$). … Continue reading “Linear Regression”<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"advanced_seo_description":"","jetpack_seo_html_title":"","jetpack_seo_noindex":false,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[6,3],"tags":[],"class_list":["post-214","post","type-post","status-publish","format-standard","hentry","category-regression","category-supervised-learning"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p9dIpN-3s","jetpack_likes_enabled":true,"jetpack-related-posts":[{"id":405,"url":"https:\/\/www.nullplug.org\/ML-Blog\/2017\/10\/18\/tensor-calculus\/","url_meta":{"origin":214,"position":0},"title":"Tensor Calculus","author":"Justin Noel","date":"October 18, 2017","format":false,"excerpt":"Introduction I will assume that you have seen some calculus, including multivariable calculus. That is you know how to differentiate a differentiable function $f\\colon \\Bbb R \\to \\Bbb R$, to obtain a new function $$\\frac{\\partial f}{\\partial x} \\colon \\Bbb R \\to \\Bbb R.$$ You also know how to differentiate a\u2026","rel":"","context":"In "Supplementary material"","block_context":{"text":"Supplementary material","link":"https:\/\/www.nullplug.org\/ML-Blog\/category\/supplementary-material\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":508,"url":"https:\/\/www.nullplug.org\/ML-Blog\/2017\/11\/09\/problem-set-4\/","url_meta":{"origin":214,"position":1},"title":"Problem Set 4","author":"Justin Noel","date":"November 9, 2017","format":false,"excerpt":"Problem Set 4 This is to be completed by November 16th, 2017. Exercises Datacamp Complete the lessons: a. Supervised Learning in R: Regression b. Supervised Learning in R: Classification c. Exploratory Data Analysis (If you did not already do so) Let $\\lambda\\geq 0$, $X\\in \\Bbb R^n\\otimes \\Bbb R^m$, $Y\\in \\Bbb\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/www.nullplug.org\/ML-Blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":486,"url":"https:\/\/www.nullplug.org\/ML-Blog\/2017\/11\/03\/problem-set-3\/","url_meta":{"origin":214,"position":2},"title":"Problem Set 3","author":"Justin Noel","date":"November 3, 2017","format":false,"excerpt":"Problem Set 3 This is to be completed by November 9th, 2017. Exercises [Datacamp](https:\/\/www.datacamp.com\/home Complete the lesson \"Introduction to Machine Learning\". This should have also included \"Exploratory Data Analysis\". This has been added to the next week's assignment. MLE for the uniform distribution. (Source: Kaelbling\/Murphy) Consider a uniform distribution centered\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/www.nullplug.org\/ML-Blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":344,"url":"https:\/\/www.nullplug.org\/ML-Blog\/2017\/10\/10\/parameter-estimation\/","url_meta":{"origin":214,"position":3},"title":"Parameter Estimation","author":"Justin Noel","date":"October 10, 2017","format":false,"excerpt":"\u2026the statistician knows\u2026that in nature there never was a normal distribution, there never was a straight line, yet with normal and linear assumptions, known to be false, he can often derive results which match, to a useful approximation, those found in the real world. - George Box (JASA, 1976, Vol.\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/www.nullplug.org\/ML-Blog\/category\/general\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.nullplug.org\/ML-Blog\/wp-content\/uploads\/2017\/10\/compressed_polyreg_normal.gif?resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.nullplug.org\/ML-Blog\/wp-content\/uploads\/2017\/10\/compressed_polyreg_normal.gif?resize=350%2C200 1x, https:\/\/i0.wp.com\/www.nullplug.org\/ML-Blog\/wp-content\/uploads\/2017\/10\/compressed_polyreg_normal.gif?resize=525%2C300 1.5x"},"classes":[]},{"id":35,"url":"https:\/\/www.nullplug.org\/ML-Blog\/2017\/09\/26\/supervised-learning\/","url_meta":{"origin":214,"position":4},"title":"Supervised Learning","author":"Justin Noel","date":"September 26, 2017","format":false,"excerpt":"A big computer, a complex algorithm, and a long time does not equal science. - Robert Gentleman Examples Before getting into what supervised learning precisely is, let's look at some examples of supervised learning tasks: Identifying breast cancer. A sample study. Image classification. List of last year's ILSVRC Winners Threat\u2026","rel":"","context":"In "Supervised Learning"","block_context":{"text":"Supervised Learning","link":"https:\/\/www.nullplug.org\/ML-Blog\/category\/supervised-learning\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":33,"url":"https:\/\/www.nullplug.org\/ML-Blog\/2017\/09\/26\/machine-learning-overview\/","url_meta":{"origin":214,"position":5},"title":"Machine Learning Overview","author":"Justin Noel","date":"September 26, 2017","format":false,"excerpt":"Science is knowledge which we understand so well that we can teach it to a computer; and if we don't fully understand something, it is an art to deal with it. Donald Knuth Introduction First Attempt at a Definition One says that an algorithm learns if its performance improves with\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/www.nullplug.org\/ML-Blog\/category\/general\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/web.stanford.edu\/class\/cs234\/images\/header2.png?resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/web.stanford.edu\/class\/cs234\/images\/header2.png?resize=350%2C200 1x, https:\/\/i0.wp.com\/web.stanford.edu\/class\/cs234\/images\/header2.png?resize=525%2C300 1.5x, https:\/\/i0.wp.com\/web.stanford.edu\/class\/cs234\/images\/header2.png?resize=700%2C400 2x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/www.nullplug.org\/ML-Blog\/wp-json\/wp\/v2\/posts\/214","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.nullplug.org\/ML-Blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.nullplug.org\/ML-Blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.nullplug.org\/ML-Blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.nullplug.org\/ML-Blog\/wp-json\/wp\/v2\/comments?post=214"}],"version-history":[{"count":10,"href":"https:\/\/www.nullplug.org\/ML-Blog\/wp-json\/wp\/v2\/posts\/214\/revisions"}],"predecessor-version":[{"id":479,"href":"https:\/\/www.nullplug.org\/ML-Blog\/wp-json\/wp\/v2\/posts\/214\/revisions\/479"}],"wp:attachment":[{"href":"https:\/\/www.nullplug.org\/ML-Blog\/wp-json\/wp\/v2\/media?parent=214"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.nullplug.org\/ML-Blog\/wp-json\/wp\/v2\/categories?post=214"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.nullplug.org\/ML-Blog\/wp-json\/wp\/v2\/tags?post=214"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}