HW10.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:82c22c0a95eac977d8c442c66ea893e5d67e9ac90b75e6cee0b09200c54c4d7e"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "**AM 207**: Homework 10"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "_ _ _ _ _\n",
      "\n",
      "Pavlos Protopapas <br>\n",
      "Handed out: Monday, April 28, 2014<br>\n",
      "Due: Monday, May 5, 2014\n",
      "\n",
      "**Instructions**:\n",
      "\n",
      "+ Upload your answers in an ipython notebook to the dropbox.\n",
      "\n",
      "+ Your individual submissions use the following filenames: AM207_YOURNAME_HM10.ipynb\n",
      "\n",
      "+ Your code should be in code cells as part of your ipython notebook. Do not use a different language (or format) unless you get permission from the TFs. If you use any special libraries you must include them with your code (program should run as is). \n",
      "\n",
      "+ If you have multiple files (e.g. you've added code files and images) create a tarball for all files in a single file and name it: AM207_YOURNAME_HM1.tar or AM207_YOURNAME_HM1.zip\n",
      "\n",
      "_ _ _ _ _"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Question 1: Can I get your digits?"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "One of the most famous publically available datasets around is MNIST, a collection of 70000 images of hand-written single-digit numbers (from 0 to 9) created by Corinna Cortes and Yann LeCun. The dataset originally appeared for the paper LeCun, Yann; L\u00e9on Bottou, Yoshua Bengio, Patrick Haffner (1998). \"Gradient-Based Learning Applied to Document Recognition\". It comprises a collection of images of handwritten digits from Census employees and high-school students widely used for testing machine learning and computer vision algorithms.  (See the image at bottom of this question for a visualization of the digits from the Andreas Mueller's Peekaboo blog http://peekaboo-vision.blogspot.com)\n",
      "\n",
      "\n",
      "Write an algorithm to classify the images in the training dataset.  Feel free to use whatever algorithm you think will be best.  We've covered some likely candidates in class (ask a teaching fellow for guidance), but feel free to implement something from outside of class (although it should be somewhat related to topics we've covered, and the algorithm should be your own implementation).\n",
      "\n",
      "You can find the data set and all instructions for participating in the competition and submitting your results at [Driven Data](http://hidden-fjord-7729.herokuapp.com/).  You will have to register and create an account. Please use the same email account for which you recieve homework feedback.  This Kaggle-like site  will test your algorithm's performance on a test handwritten image dataset and compare that performance with other submissions from your fellow students. In addition to having the satisfaction of submitting an extra credit homework, you will get a $100 restaurant gift certificate if you win!  \n",
      "\n",
      "Note that in addition to any submission requirements for the [Driven Data site](http://hidden-fjord-7729.herokuapp.com/), you'll be required to submit an ipython notebook with your full solution to this problem set (including code and writeup for this problem) to the HW Dropbox as usual.\n",
      "\n",
      "Two of your classmates, Peter Bull and Isaac Slavitt, are working with Pavlos as an independent study to build a website that hosts data science competitions. They've been kind enough to allow us to use their platform to host our HW10 competition. As with any beta test, there may be a few nagging bugs. We really appreciate your thoughts, comments, and feedback on what works and what doesn\u2019t. Please reach out if you encounter any issues: bull@fas.harvard.edu; slavitt@fas.harvard.edu.\n",
      "\n",
      "\n",
      "<div class=\"figure\">\n",
      "<img alt=\"schematic\" src=\"./files/mnist_originals.png\"/>\n",
      "</div>"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Question 2: GP Regression"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "\n",
      "In the file [survey.csv](./files/survey.csv), from the 2004 National Annenberg Election Survey, we have, in 3 columns `age`, the age of the respondents, `numr`, the number of responses, and `knowgay`, the number of people at that age who know at-least one gay person. \n",
      "\n",
      "Use the `scikit-learn`, `GPy`, `pymc`,  `infpy` or your own implementation to perform a 1-D Gaussian Process regression  on this data, with the fraction of people who know atleast one gay person as the dependent variable, and `age` as the covariate. Use a squared exponential covariance function (with parameter $\\theta_0$ in scikit-learn, or amplitude and length scale in other implementations). \n",
      "\n",
      "The likelihood can be considered to be the product of binomials,  one binomial distribution per age group. \n",
      "\n",
      "$$\\cal{L} = \\prod_{a} y | p(a)$$\n",
      "\n",
      "where $y$ is the observed fraction of respondents who know a gay person, $a$ is age, and $ y | p(a) \\sim Binom \\left(N=numr[a], k=knowgay[a], p(a) \\right)$.\n",
      "\n",
      "That is, for each age group, the Binomial is described by the total number of trials, here  `numr`, the number of  'successes'  given by `knowgay` and the probability of success function $p(a)$. Use the Gaussian approximation  (http://en.wikipedia.org/wiki/Binomial_distribution#Normal_approximation) to approximate the Binomial since `numr` is large. Therefore you can simply use a GP posterior with the error for each measurement to be given using this approximation. \n",
      "\n",
      "Find the best parameters by the usual maximization of the marginal log-likelihood. Plot the mean predictions for ages 0-100, with a 2-sigma envelope, as we did in the lecture notes. (Dont worry about splitting into test and training sets for this problem)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}