{"id":344,"date":"2017-10-10T08:59:07","date_gmt":"2017-10-10T08:59:07","guid":{"rendered":"http:\/\/www.nullplug.org\/ML-Blog\/?p=344"},"modified":"2017-11-02T08:10:36","modified_gmt":"2017-11-02T08:10:36","slug":"parameter-estimation","status":"publish","type":"post","link":"http:\/\/www.nullplug.org\/ML-Blog\/2017\/10\/10\/parameter-estimation\/","title":{"rendered":"Parameter Estimation"},"content":{"rendered":"

\n \u2026the statistician knows\u2026that in nature there never was a normal distribution, there never was a straight line, yet with normal and linear assumptions, known to be false, he can often derive results which match, to a useful approximation, those found in the real world. – George Box (JASA, 1976, Vol. 71, 791-799)\n<\/p><\/blockquote>\n

Parameter estimation<\/h2>\n

Suppose that we are given some data $D=(x_1,\\cdots, x_n)\\in T^n$. Our task is to find a parametric probability distribution $m(\\hat{\\theta})$ on $T$ such that if we were to independently sample from it $n$-times, we would could reasonably1<\/a><\/sup> expect to obtain $D$. We first make some choice2<\/a><\/sup> of possible distributions $m(-)$, which leaves us with the task of identifying the $\\hat{\\theta}$.<\/p>\n

Likelihood approach <\/a><\/h3>\n

One approach is the maximal likelihood estimate<\/em> or MLE. For this we set $$\\hat{\\theta}=\\theta_{MLE} = \\mathrm{argmax}_\\theta p(D | \\theta).$$ That is we choose $\\theta$ so that the probability density of $D$ from the given model is maximized.<\/p>\n

There are a few comments that we should make here:<\/p>\n

    \n
  1. There may not be a maximum for a general distribution. Indeed the models typically considered for logistic regression do not have a maximum.<\/li>\n
  2. If there is a maximum, it may not be unique. <\/li>\n
  3. This is an optimization problem and when this problem is not convex<\/a>, we may not have any method that is guaranteed to identify find a maximum even when it exists. <\/li>\n<\/ol>\n

    We do not actually address these problems, instead we just say that optimization techniques will give us a parameter value that is “good enough”.<\/p>\n

    Example<\/h4>\n

    Suppose that $D=(x_1,\\cdots, x_n)\\in \\Bbb R^n$ and we want to find a normal distribution $N(\\mu, \\sigma^2)$ which models the data under the assumption that these were independent samples.<\/p>\n

    First we construct the MLE of $\\mu$:<\/p>\n

    \\begin{align}
    \n
    \n\\end{align}<\/p>\n

    We differentiate the last expression with respect to $\\mu$ and set this to 0 and obtain
    \n\\begin{align}
    \n -\\sum_{i=1}^n (x_i -\\mu)&=0 \\
    \n \\mu_{MLE} &= \\frac{\\sum_{i=1}^n x_i}{n}.
    \n\\end{align}
    \nTaking second derivatives shows that this is in fact a minimum. In other words, the MLE of $\\mu$ is the sample mean<\/em>.<\/p>\n

    Exercise<\/h4>\n

    Show that MLE of $\\sigma^2$ is the (biased) sample variance<\/em>. In other words, $$\\sigma_{MLE}^2 = \\frac{\\sum_{i=1}^n (x_i-\\mu_{MLE})^2}{n}.$$<\/p>\n

    Frequentist evaluation <\/a><\/h3>\n

    The frequentist approach gives us a method to evaluate the above estimates of the parameters. Suppose that our data is drawn from a true distribution $m(\\hat{\\theta})$ and set $\\overline{\\theta} = E(\\theta_{MLE}(D))$, where the expectation is given by integrating over $D$ given the “true” model. Define the bias<\/em> of $\\theta_{MLE}(-)$ by $$\\mathrm{bias}(\\theta_{MLE})=\\overline{\\theta}-\\hat{\\theta}.$$<\/p>\n

    Let $\\mu_S(D)$ be the sample mean of data drawn from some distribution $m$ with finite mean $\\widehat{\\mu}$. Then we see
    \n\\begin{align}
    \n E(\\mu_S) & = E\\left(\\frac{\\sum_{i=1}^n X_i}{n}\\right) \\
    \n & = \\frac{\\sum_{i=1}^n E(X_i)}{n} \\
    \n & = n\\widehat{\\mu}\/n = \\widehat{\\mu}
    \n\\end{align}
    \nso this is an unbiased estimate.<\/p>\n

    For a visualization of an unbiased point estimate from the frequentist point of view look here<\/a>.<\/p>\n

    Exercise<\/h4>\n
      \n
    1. Show that the expected value of the sample variance $\\sigma^2_S$ of data drawn from some distribution $m$ with finite mean $\\widehat{\\mu}$ and finite variance $\\widehat{\\sigma^2}$ is $$ E(\\sigma^2_S)= \\frac{n \\widehat{\\sigma^2}}{n-1}.$$
      \nShow that it follows that the bias of $\\sigma^2_S$ is $\\widehat{\\sigma^2}\/(n-1)$. <\/li>\n
    2. Show that the unbiased sample variance<\/em> $\\sigma^2_U(X_1,\\cdots,X_n) = \\frac{\\sum_{i=1}^n(X_i-\\mu_S(X_1,\\cdots X_n))^2}{n-1}$ is, in fact, an unbiased estimate.<\/li>\n
    3. Construct a symmetric 95% confidence interval<\/em> (see here<\/a> for some additional discussion) for our estimate $\\mu_S$ of $\\widehat{\\mu}$ which is centered about $\\mu_S$, when $n=10$, $\\mu_S=15$ and $\\sigma^2_U = 2$. Hint: the random variable $\\frac{\\mu_S-\\widehat{\\mu}}{(\\sigma_U \/\\sqrt{n})}$ has the same distribution as the Student’s $t$-distribution<\/a> whose inverse cdf values can be calculated by statistics packages. <\/li>\n<\/ol>\n

      Bias-Variance tradeoff<\/h3>\n

      Suppose that we have an estimator $\\theta=\\delta(D)$ of $\\hat{\\theta}$ and the expected value of the estimate is $E(\\delta(D))=\\overline{\\theta}$. We obtain the following expression for our expected squared error
      \n\\begin{align}
      \n E((\\theta-\\hat{\\theta})^2) &= E(\\left((\\theta-\\overline{\\theta})-(\\hat{\\theta}-\\overline{\\theta})\\right)^2) \\
      \n &= E((\\theta-\\overline{\\theta})^2)-E(\\theta-\\overline{\\theta})\\cdot 2(\\hat{\\theta}-\\overline{\\theta}) + (\\hat{\\theta}-\\overline{\\theta})^2 \\
      \n &= \\mathrm{var}(\\theta)+\\mathrm{bias}^2(\\theta)
      \n\\end{align}<\/p>\n

      This equation indicates that while it may be nice to have an unbiased estimator (i.e., one that will, on average, give precisely the correct parameter), if this comes with the cost of creating an estimate that varies wildly with the data then we will still expect a large amount of error and indicates why we would like estimators with low-variance.<\/p>\n

      A typical example of this is seen in polynomial regression (here we are viewing the problem as trying to estimate $p(y|x)$). As we vary the degrees of the polynomial being used, we find that very large degree polynomials can fit the data well:

      \"\"
      High degree polynomials fit the data better<\/figcaption><\/figure><\/p>\n

      However, the fit for high degree polynomials varies highly depending on the choice of data points:

      \"\"
      This estimator has high variance<\/figcaption><\/figure><\/p>\n

      The MAP Estimate <\/a><\/h3>\n

      One Bayesian approach to parameter estimation is called the MAP estimate<\/em> or maximum a posteriori estimate<\/em>. Here we are given data $D$, which we want to say is modeled by a distribution $m(\\theta)$ and we construct the MAP estimate of $\\theta$ as
      \n\\begin{equation}
      \n
      \n\\end{equation}
      \nIn other words, we choose the mode of the posterior distribution. Note that if our prior $p(\\theta)$ is constant independent of $\\theta$, then the only part of the right hand side that depends on $\\theta$ is $p(D|\\theta)$ and the MAP estimate of $\\theta$ is the same as the MLE of $\\theta$.<\/p>\n

      Example<\/h4>\n

      Suppose that we have a coin whose bias $\\theta$ we would like to determine (so $\\theta$ is the probability of the coin coming up heads). We flip the coin twice and obtain some data $D=(H,H)$. Under the assumption that the coin flips are independent we are trying to model a Bernoulli distribution<\/a>. Now we calculate the conditional probability:
      \n\\begin{equation}
      \n p(D|\\theta) = \\theta^2(1-\\theta)^0
      \n\\end{equation}
      \nand quickly see the MLE for $\\theta$ (given that $\\theta \\in [0,1]$) is what we would expect: $\\theta = 1,$ as mentioned in the hypothesis testing       example<\/a> of an overconfident estimator.<\/p>\n

      As already mentioned, the MLE and MAP estimate agree in the case of the uniform prior, but the Bayesian approach allows us to calculate $p(\\theta | D)$, which gives us a measure of our confidence in this estimate. In this case we see:
      \n\\begin{align}
      \n p(\\theta | D) &= \\frac{p(D|\\theta)p(\\theta)}{\\int_{\\theta’} p(D|\\theta’)p(\\theta’)d\\theta’} \\
      \n&= \\frac{\\theta^2}{\\int_{\\theta’} (\\theta’)^2d\\theta’} \\
      \n&= \\frac{\\theta^2}{1\/3} \\
      \n&= 3\\theta^2.
      \n\\end{align}<\/p>\n

      If we want to find a 99.9% credible interval for the estimate $\\theta_*\\in [a,1]$, then we just integrate
      \n\\begin{align}
      \nP(\\theta \\in [a,1] | D) &=\\int_{\\theta =a}^1 p(\\theta | D)d\\theta \\
      \n &=\\int_{\\theta =a}^1 3\\theta^2 d\\theta \\
      \n &=1-a^3.
      \n\\end{align}
      \nSetting this to be equal to $1-10^{-3}$ we see that $a=0.1$. This interval is much more conservative than the confidence interval obtained by frequentist methods.<\/p>\n

      Now what if we do not have a uniform prior. Instead we suppose we have some prior information3<\/a><\/sup> that makes us believe that the most likely value of $\\theta$ is actually 0.25 and that it has variance 0.05. How do we incorporate this information into a prior? Well the mode and the variance are not sufficient to determine a distribution over the unit interval, so let’s assume that $p(\\theta)$ has some convenient form that fits these values.<\/p>\n

      It would simplify matters if $p(\\theta)$ had the same form as $p(D|\\theta)=\\theta^{|H|}(1-\\theta)^{|T|}$. So, supposing $p(\\theta)=C\\theta^a(1-\\theta)^b$ for some constant $C$ (i.e., $p(\\theta)\\propto \\theta^a (1-\\theta)^b$), we would see that
      \n\\begin{align}
      \n p(\\theta | D) &\\propto p(D|\\theta)p(\\theta) \\
      \n &\\propto \\theta^{|H|}(1-\\theta)^{|T|} \\theta^a(1-\\theta)^b\\
      \n& = \\theta^{|H|+a}(1-\\theta^{|T|+b}).
      \n\\end{align}
      \nThis has the same form as the prior! Such a prior is called a conjugate prior to the given likelihood function. Having such a prior is super convenient for computation.<\/p>\n

      In this case, the conjugate prior to the Bernoulli distribution is the Beta distribution<\/a> which has the form $p(\\theta | \\alpha, \\beta)\\propto \\theta^{\\alpha-1}(1-\\theta)^{\\beta-1}$. One can calculate the mode and variance of this distribution (or just look it up) and get
      \n\\begin{align}
      \n\\textrm{mode} &= \\frac{\\alpha -1}{\\alpha+\\beta-2} \\
      \n\\sigma^2 &= \\frac{\\alpha \\beta}{(\\alpha+\\beta)^2(\\alpha+\\beta+1)}
      \n\\end{align}
      \nPlugging this into       Wolfram Alpha<\/a> gives us some approximate parameter values: $\\alpha \\approx 1.4$ and $\\beta \\approx 2.3$.<\/p>\n

      Using this prior, the new posterior is
      \n$$ p(\\theta | {HH})\\propto \\theta^{2.4}(1-\\theta)^{1.3}.$$ We can form the MAP estimate by taking the $\\log$ of this expression (which is fine away from the points $\\theta = 0, 1$, which we know can not yield the maximum), differentiating with respect to $\\theta$, setting this to 0, and finally checking that the second derivative at this point is positive.<\/p>\n

      Exercise<\/h4>\n

      Show the MAP estimate for this problem is $2.4\/3.7\\approx 0.65$ by calculating the mode of the $\\beta$ density function.<\/p>\n

      We see that our estimate of $\\theta$ shifts drastically as we obtain (even just a little bit) more information. Now lets see how the posterior changes as we see even longer strings of heads:
      \n\"\"
      \nFrom the image we see that as posterior gradually becomes highly confident that the the true value of $\\theta$ is very large and assigns very small probability densities to small values of $\\theta.$<\/p>\n

      To see how much our choice of prior affects the posterior in the presence of the same data, we can look at the analogous chart starting from a uniform prior, i.e., when the $\\beta$ parameters are $\\alpha=\\beta=1$.
      \n\"\"
      \nOr we can look at a side by side comparison:
      \n\"\"
      \nFrom looking at the graphs, we can see that while the two priors are very different (e.g., the non-uniform prior assigns very small densities to large values of $\\theta$), the posteriors rapidly become close to each other. How the posterior depends on the prior[\/caption]Since they never agree, our methods still depend on our choice of prior, but we can also see that the both methods are approaching the same estimate.<\/p>\n

      To finally drive this point home, let’s consider a severely biased prior $\\beta(1.01,10.0)$, whose mode puts the probability of heads at approximately 0 and with near 0 variance in the binomial model. Then we can see how the posterior changes with additional evidence:
      \n

      \"\"
      How the posterior depends on the prior<\/figcaption><\/figure>
      \nThe severely biased prior is much slower to approach the same consensus as the other priors.<\/p>\n

      Exercise<\/h4>\n

      The severely biased prior is very close to what we would obtain as a posterior from a uniform prior after witnessing 9 tails<\/em> in a row. If we were to witness 9 tails in a row followed by 20 heads in a row, how would you change your modeling assumptions to better reflect the data?<\/p>\n

      Further point estimates<\/h4>\n

      As discussed above, one way to estimate a parameter is the MAP estimate which is just the mode of the posterior distribution $p(\\theta | D)$. This is a very common approach to parameter estimation because it transforms the task into an optimization problem and we have a number of tools for such tasks.<\/p>\n

      However, the mode of a distribution can be far from a typical point. The mode of a distribution is also not invariant under reparametrization. One alternative is to select the mean<\/em> or expected value of $\\theta$ from the posterior distribution. Another alternative is to take the median<\/em> of the posterior distribution.<\/p>\n

      \n
      \n
        \n
      1. \nI know this is not a very precise statement, but I want to be flexible at this point. ↩<\/a>\n<\/li>\n
      2. \nHow we make this choice is the model selection problem<\/em> which will be discussed later. ↩<\/a>\n<\/li>\n
      3. \nI apologize that this example is so contrived. ↩<\/a>\n<\/li>\n<\/ol>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

        \u2026the statistician knows\u2026that in nature there never was a normal distribution, there never was a straight line, yet with normal and linear assumptions, known to be false, he can often derive results which match, to a useful approximation, those found in the real world. – George Box (JASA, 1976, Vol. 71, 791-799) Parameter estimation Suppose … Continue reading “Parameter Estimation”<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"advanced_seo_description":"","jetpack_seo_html_title":"","jetpack_seo_noindex":false,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[1],"tags":[],"class_list":["post-344","post","type-post","status-publish","format-standard","hentry","category-general"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p9dIpN-5y","jetpack_likes_enabled":true,"jetpack-related-posts":[{"id":486,"url":"http:\/\/www.nullplug.org\/ML-Blog\/2017\/11\/03\/problem-set-3\/","url_meta":{"origin":344,"position":0},"title":"Problem Set 3","author":"Justin Noel","date":"November 3, 2017","format":false,"excerpt":"Problem Set 3 This is to be completed by November 9th, 2017. Exercises [Datacamp](https:\/\/www.datacamp.com\/home Complete the lesson \"Introduction to Machine Learning\". This should have also included \"Exploratory Data Analysis\". This has been added to the next week's assignment. MLE for the uniform distribution. (Source: Kaelbling\/Murphy) Consider a uniform distribution centered\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"http:\/\/www.nullplug.org\/ML-Blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":286,"url":"http:\/\/www.nullplug.org\/ML-Blog\/2017\/10\/05\/statistical-inference-2\/","url_meta":{"origin":344,"position":1},"title":"Statistical Inference","author":"Justin Noel","date":"October 5, 2017","format":false,"excerpt":"All models are wrong, but some are useful. - George Box Introduction The general setup for statistical inference is that we are given some data $D$ which we assume arise as the values of a random variable that we assume is distributed according to some parametric model $m(\\theta)$. The goal\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"http:\/\/www.nullplug.org\/ML-Blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":214,"url":"http:\/\/www.nullplug.org\/ML-Blog\/2017\/10\/04\/linear-regression\/","url_meta":{"origin":344,"position":2},"title":"Linear Regression","author":"Justin Noel","date":"October 4, 2017","format":false,"excerpt":"Prediction is very difficult, especially about the future. - Niels Bohr The problem Suppose we have a list of vectors (which we can think of as samples) $x_1, \\cdots, x_m\\in \\Bbb R^n$ and a corresponding list of output scalars $y_1, \\cdots, y_m \\in \\Bbb R$ (which we can regard as\u2026","rel":"","context":"In "Regression"","block_context":{"text":"Regression","link":"http:\/\/www.nullplug.org\/ML-Blog\/category\/regression\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.nullplug.org\/ML-Blog\/wp-content\/uploads\/2017\/10\/trace.png?resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.nullplug.org\/ML-Blog\/wp-content\/uploads\/2017\/10\/trace.png?resize=350%2C200 1x, https:\/\/i0.wp.com\/www.nullplug.org\/ML-Blog\/wp-content\/uploads\/2017\/10\/trace.png?resize=525%2C300 1.5x"},"classes":[]},{"id":508,"url":"http:\/\/www.nullplug.org\/ML-Blog\/2017\/11\/09\/problem-set-4\/","url_meta":{"origin":344,"position":3},"title":"Problem Set 4","author":"Justin Noel","date":"November 9, 2017","format":false,"excerpt":"Problem Set 4 This is to be completed by November 16th, 2017. Exercises Datacamp Complete the lessons: a. Supervised Learning in R: Regression b. Supervised Learning in R: Classification c. Exploratory Data Analysis (If you did not already do so) Let $\\lambda\\geq 0$, $X\\in \\Bbb R^n\\otimes \\Bbb R^m$, $Y\\in \\Bbb\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"http:\/\/www.nullplug.org\/ML-Blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":33,"url":"http:\/\/www.nullplug.org\/ML-Blog\/2017\/09\/26\/machine-learning-overview\/","url_meta":{"origin":344,"position":4},"title":"Machine Learning Overview","author":"Justin Noel","date":"September 26, 2017","format":false,"excerpt":"Science is knowledge which we understand so well that we can teach it to a computer; and if we don't fully understand something, it is an art to deal with it. Donald Knuth Introduction First Attempt at a Definition One says that an algorithm learns if its performance improves with\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"http:\/\/www.nullplug.org\/ML-Blog\/category\/general\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/web.stanford.edu\/class\/cs234\/images\/header2.png?resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/web.stanford.edu\/class\/cs234\/images\/header2.png?resize=350%2C200 1x, https:\/\/i0.wp.com\/web.stanford.edu\/class\/cs234\/images\/header2.png?resize=525%2C300 1.5x, https:\/\/i0.wp.com\/web.stanford.edu\/class\/cs234\/images\/header2.png?resize=700%2C400 2x"},"classes":[]},{"id":61,"url":"http:\/\/www.nullplug.org\/ML-Blog\/2017\/09\/26\/probability-and-statistics-background\/","url_meta":{"origin":344,"position":5},"title":"Probability and Statistics Background","author":"Justin Noel","date":"September 26, 2017","format":false,"excerpt":"Statistics - A subject which most statisticians find difficult, but in which nearly all physicians are expert. - Stephen S. Senn Introduction For us, we will regard probability theory as a way of logically reasoning about uncertainty. I realize that this is not a precise mathematical definition, but neither is\u2026","rel":"","context":"In "Supplementary material"","block_context":{"text":"Supplementary material","link":"http:\/\/www.nullplug.org\/ML-Blog\/category\/supplementary-material\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"http:\/\/www.nullplug.org\/ML-Blog\/wp-json\/wp\/v2\/posts\/344","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.nullplug.org\/ML-Blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.nullplug.org\/ML-Blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.nullplug.org\/ML-Blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/www.nullplug.org\/ML-Blog\/wp-json\/wp\/v2\/comments?post=344"}],"version-history":[{"count":10,"href":"http:\/\/www.nullplug.org\/ML-Blog\/wp-json\/wp\/v2\/posts\/344\/revisions"}],"predecessor-version":[{"id":347,"href":"http:\/\/www.nullplug.org\/ML-Blog\/wp-json\/wp\/v2\/posts\/344\/revisions\/347"}],"wp:attachment":[{"href":"http:\/\/www.nullplug.org\/ML-Blog\/wp-json\/wp\/v2\/media?parent=344"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.nullplug.org\/ML-Blog\/wp-json\/wp\/v2\/categories?post=344"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.nullplug.org\/ML-Blog\/wp-json\/wp\/v2\/tags?post=344"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}