notebooks/Issue_Labeling_Model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Background\n",
    "---\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/machine-learning-apps/Issue-Label-Bot/blob/master/notebooks/Issue_Labeling_Model.ipynb)\n",
    "\n",
    "The goal of the MLapp project is to provide the following:\n",
    "\n",
    "1. Illustrate how to build machine learning powered developer tools using the [GitHub Api](https://developer.github.com/v3/) and Flask.  We would like to show data scientists how to build exciting data products using machine learning on the GitHub marketplace, that developers can use.  Specifically, we will build an illustrative data product that will automatically label issues.  \n",
    "\n",
    "2. Gather feedback and iterate \n",
    "\n",
    "\n",
    "The scope of this notebook is to addresses part of goal #1, by illustrating how we can acquire a dataset of GitHub issue labels and train a classifier.  \n",
    "\n",
    "The top issues on GitHub by count are illustrated in [this spreadsheet](https://docs.google.com/spreadsheets/d/1NPacnVsyZMBneeewvPGhCx512A1RPYf8ktDN_RpKeS4/edit?usp=sharing).  To keep things simple, we will build a model to classify an issue as a `bug`, `feature` or `question`.  We use hueristics to collapse a set of issue labels into these three categories, which can be viewed [in this query](https://console.cloud.google.com/bigquery?sq=123474043329:01abf8866144486f932c756730ddaff1).  \n",
    "\n",
    "The heueristic for these class labels are contained within the below case statement:\n",
    "\n",
    "```{sql}\n",
    "  CASE when labels like '%bug%' and labels not like '%not bug%' then True else False end as Bug_Flag,\n",
    "  CASE when labels like '%feature%' or labels like '%enhancement%' or labels like '%improvement%' or labels like '%request%' then True else False end as Feature_Flag,\n",
    "  CASE when labels like '%question%' or labels like '%discussion%' then True else False end as Question_Flag,\n",
    "```\n",
    "    the above case statement is located within [this query](https://console.cloud.google.com/bigquery?sq=123474043329:01abf8866144486f932c756730ddaff1)\n",
    "    \n",
    "\n",
    "The following alternative projects were tried before this task that we did not pursue further:\n",
    " - Transfer learning using the [GitHub Issue Summarizer](https://github.com/hamelsmu/Seq2Seq_Tutorial) to enable the prediction of custom labels on existing repos.  Found that this did not work well as there is a considerable amount of noise with regards to custom labels in repositories and often not enough data to adequately predict this.  \n",
    " - Tried to classify more than the above three classes, however the human-labeled issues are very subjective and it is not clear what is a question vs. a bug.  \n",
    " - Tried multi-label classification since labels can co-occur.  There is very little overlap between `bug`, `feature` and `question` labels, so we decided to simplify things and make this a multi-class classificaiton problem instead.  \n",
    "\n",
    "\n",
    "Note: the code in this notebook was executed on a [p3.8xlarge](https://aws.amazon.com/ec2/instance-types/p3/) instance on AWS.\n",
    "\n",
    "## Outline \n",
    "\n",
    "This notebook will follow these steps:\n",
    "\n",
    "1. Download and partition dataset\n",
    "2. Pre-process dataset\n",
    "2. Build model architecture & Train Model\n",
    "3. Evaluate Model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Download and Partition Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "pd.set_option('max_colwidth', 1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train: 2,698,578 rows 10 columns\n",
      "Test: 476,220 rows 10 columns\n"
     ]
    }
   ],
   "source": [
    "df = pd.concat([pd.read_csv(f'https://storage.googleapis.com/codenet/issue_labels/00000000000{i}.csv.gz')\n",
    "                for i in range(10)])\n",
    "\n",
    "#split data into train/test\n",
    "traindf, testdf = train_test_split(df, test_size=.15)\n",
    "\n",
    "traindf.to_pickle('traindf.pkl')\n",
    "testdf.to_pickle('testdf.pkl')\n",
    "\n",
    "#print out stats about shape of data\n",
    "print(f'Train: {traindf.shape[0]:,} rows {traindf.shape[1]:,} columns')\n",
    "print(f'Test: {testdf.shape[0]:,} rows {testdf.shape[1]:,} columns')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion of the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>repo</th>\n",
       "      <th>title</th>\n",
       "      <th>body</th>\n",
       "      <th>num_labels</th>\n",
       "      <th>labels</th>\n",
       "      <th>c_bug</th>\n",
       "      <th>c_feature</th>\n",
       "      <th>c_question</th>\n",
       "      <th>class_int</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>112473</th>\n",
       "      <td>\"https://github.com/DockStation/dockstation/issues/6\"</td>\n",
       "      <td>DockStation/dockstation</td>\n",
       "      <td>feature request: image layers window hide</td>\n",
       "      <td>some feedback for this window:\\r \\r ! image  https://cloud.githubusercontent.com/assets/9369080/25952392/3769f842-3669-11e7-8a91-4ed2de640246.png \\r \\r \\r it's really cool, but blocking.  i think users would prefer to be able hide it  the \\ ok\\  was not hiding it in my case - don't know if it should?</td>\n",
       "      <td>2</td>\n",
       "      <td>[\"enhancement\", \"wish\"]</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33329</th>\n",
       "      <td>\"https://github.com/julianschritt/secreth_telegrambot/issues/2\"</td>\n",
       "      <td>julianschritt/secreth_telegrambot</td>\n",
       "      <td>grant group admins permissions to /cancelgame and /startgame</td>\n",
       "      <td>ending a game requires the person who started it to end it. even kicking and re-adding the bot doesn't close an existing game. so if the person who started a game is afk for a long time the bot becomes useless for that group.</td>\n",
       "      <td>1</td>\n",
       "      <td>[\"enhancement\", \"enhancement\"]</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95978</th>\n",
       "      <td>\"https://github.com/remotestorage/remotestorage-bookmarks-chrome/issues/3\"</td>\n",
       "      <td>remotestorage/remotestorage-bookmarks-chrome</td>\n",
       "      <td>deprecate in favor of memm?</td>\n",
       "      <td>this thing is super old and nobody's actively maintaining it. i don't know if someone's actively using it, but i think it would make sense to direct people coming here to https://github.com/lesion/memm, which works and is maintained by @lesion, who's also a core contributor to rs.js.</td>\n",
       "      <td>1</td>\n",
       "      <td>[\"question\"]</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                               url  \\\n",
       "112473                       \"https://github.com/DockStation/dockstation/issues/6\"   \n",
       "33329              \"https://github.com/julianschritt/secreth_telegrambot/issues/2\"   \n",
       "95978   \"https://github.com/remotestorage/remotestorage-bookmarks-chrome/issues/3\"   \n",
       "\n",
       "                                                repo  \\\n",
       "112473                       DockStation/dockstation   \n",
       "33329              julianschritt/secreth_telegrambot   \n",
       "95978   remotestorage/remotestorage-bookmarks-chrome   \n",
       "\n",
       "                                                               title  \\\n",
       "112473                     feature request: image layers window hide   \n",
       "33329   grant group admins permissions to /cancelgame and /startgame   \n",
       "95978                                    deprecate in favor of memm?   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                 body  \\\n",
       "112473  some feedback for this window:\\r \\r ! image  https://cloud.githubusercontent.com/assets/9369080/25952392/3769f842-3669-11e7-8a91-4ed2de640246.png \\r \\r \\r it's really cool, but blocking.  i think users would prefer to be able hide it  the \\ ok\\  was not hiding it in my case - don't know if it should?   \n",
       "33329                                                                               ending a game requires the person who started it to end it. even kicking and re-adding the bot doesn't close an existing game. so if the person who started a game is afk for a long time the bot becomes useless for that group.   \n",
       "95978                    this thing is super old and nobody's actively maintaining it. i don't know if someone's actively using it, but i think it would make sense to direct people coming here to https://github.com/lesion/memm, which works and is maintained by @lesion, who's also a core contributor to rs.js.   \n",
       "\n",
       "        num_labels                          labels  c_bug  c_feature  \\\n",
       "112473           2         [\"enhancement\", \"wish\"]  False       True   \n",
       "33329            1  [\"enhancement\", \"enhancement\"]  False       True   \n",
       "95978            1                    [\"question\"]  False      False   \n",
       "\n",
       "        c_question  class_int  \n",
       "112473       False          1  \n",
       "33329        False          1  \n",
       "95978         True          2  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# preview data\n",
    "traindf.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Discussion of the data:  \n",
    "\n",
    "- url:        url where you can find this issue\n",
    "- repo:       owner/repo name\n",
    "- title:      title of the issue\n",
    "- body:       body of the issue, not including comments\n",
    "- num_labels: number of issue labels\n",
    "- labels:     an array of labels applied a user manually applied to the issue (represented as a string)\n",
    "- c_bug:      boolean flag that indicates if the issue label corresponds to a bug\n",
    "- c_feature:  boolean flag that indicates if the issue label corresponds to a feature\n",
    "- c_question: boolean flag that indicates if the issue label corresponds to a question\n",
    "- class_int:  integer between 0 - 2 that corresponds to the class label.  **0=bug, 1=feature, 2=question**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Class frequency **0=bug, 1=feature, 2=question**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "class_int\n",
       "0    1211335\n",
       "1    1231499\n",
       "2     255744\n",
       "dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "traindf.groupby('class_int').size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "number of unique repos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Avg # of issues per repo: 7.6\n",
      " Avg # of issues per org: 8.7\n"
     ]
    }
   ],
   "source": [
    "print(f' Avg # of issues per repo: {len(traindf) / traindf.repo.nunique():.1f}')\n",
    "print(f\" Avg # of issues per org: {len(traindf) / traindf.repo.apply(lambda x: x.split('/')[-1]).nunique():.1f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most popular repos by # of issues:\n",
    "\n",
    " - `pcnt` = percent of total issues in the dataset\n",
    " - `count` = number of issues in the dataset for that repo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pcnt</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>repo</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Microsoft/vscode</th>\n",
       "      <td>0.005128</td>\n",
       "      <td>16281</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rancher/rancher</th>\n",
       "      <td>0.002450</td>\n",
       "      <td>7779</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MicrosoftDocs/azure-docs</th>\n",
       "      <td>0.001963</td>\n",
       "      <td>6233</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>godotengine/godot</th>\n",
       "      <td>0.001952</td>\n",
       "      <td>6198</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ansible/ansible</th>\n",
       "      <td>0.001950</td>\n",
       "      <td>6192</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hashicorp/terraform</th>\n",
       "      <td>0.001649</td>\n",
       "      <td>5235</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>kubernetes/kubernetes</th>\n",
       "      <td>0.001621</td>\n",
       "      <td>5147</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lionheart/openradar-mirror</th>\n",
       "      <td>0.001330</td>\n",
       "      <td>4221</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>elastic/kibana</th>\n",
       "      <td>0.001201</td>\n",
       "      <td>3813</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>magento/magento2</th>\n",
       "      <td>0.001129</td>\n",
       "      <td>3583</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dart-lang/sdk</th>\n",
       "      <td>0.001127</td>\n",
       "      <td>3579</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dotnet/corefx</th>\n",
       "      <td>0.001121</td>\n",
       "      <td>3558</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>eclipse/che</th>\n",
       "      <td>0.001114</td>\n",
       "      <td>3538</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>brave/browser-laptop</th>\n",
       "      <td>0.001067</td>\n",
       "      <td>3389</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>kbower/tickettest1</th>\n",
       "      <td>0.000920</td>\n",
       "      <td>2921</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Kademi/kademi-dev</th>\n",
       "      <td>0.000887</td>\n",
       "      <td>2815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>owncloud/core</th>\n",
       "      <td>0.000819</td>\n",
       "      <td>2600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>elastic/elasticsearch</th>\n",
       "      <td>0.000809</td>\n",
       "      <td>2568</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>eslint/eslint</th>\n",
       "      <td>0.000765</td>\n",
       "      <td>2428</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>angular/angular</th>\n",
       "      <td>0.000719</td>\n",
       "      <td>2283</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>openshift/origin</th>\n",
       "      <td>0.000717</td>\n",
       "      <td>2277</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CartoDB/cartodb</th>\n",
       "      <td>0.000685</td>\n",
       "      <td>2174</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cockroachdb/cockroach</th>\n",
       "      <td>0.000655</td>\n",
       "      <td>2081</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RocketChat/Rocket.Chat</th>\n",
       "      <td>0.000650</td>\n",
       "      <td>2065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Microsoft/AL</th>\n",
       "      <td>0.000647</td>\n",
       "      <td>2055</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                pcnt  count\n",
       "repo                                       \n",
       "Microsoft/vscode            0.005128  16281\n",
       "rancher/rancher             0.002450   7779\n",
       "MicrosoftDocs/azure-docs    0.001963   6233\n",
       "godotengine/godot           0.001952   6198\n",
       "ansible/ansible             0.001950   6192\n",
       "hashicorp/terraform         0.001649   5235\n",
       "kubernetes/kubernetes       0.001621   5147\n",
       "lionheart/openradar-mirror  0.001330   4221\n",
       "elastic/kibana              0.001201   3813\n",
       "magento/magento2            0.001129   3583\n",
       "dart-lang/sdk               0.001127   3579\n",
       "dotnet/corefx               0.001121   3558\n",
       "eclipse/che                 0.001114   3538\n",
       "brave/browser-laptop        0.001067   3389\n",
       "kbower/tickettest1          0.000920   2921\n",
       "Kademi/kademi-dev           0.000887   2815\n",
       "owncloud/core               0.000819   2600\n",
       "elastic/elasticsearch       0.000809   2568\n",
       "eslint/eslint               0.000765   2428\n",
       "angular/angular             0.000719   2283\n",
       "openshift/origin            0.000717   2277\n",
       "CartoDB/cartodb             0.000685   2174\n",
       "cockroachdb/cockroach       0.000655   2081\n",
       "RocketChat/Rocket.Chat      0.000650   2065\n",
       "Microsoft/AL                0.000647   2055"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pareto_df = pd.DataFrame({'pcnt': df.groupby('repo').size() / len(df), 'count': df.groupby('repo').size()})\n",
    "pareto_df.sort_values('pcnt', ascending=False).head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pre-Process Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To process the raw text data, we will use [ktext](https://github.com/hamelsmu/ktext)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ktext.preprocess import processor\n",
    "import dill as dpickle\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clean, tokenize, and apply padding / truncating such that each document length = 75th percentile for the dataset.\n",
    "Retain only the top keep_n words in the vocabulary and set the remaining words to 1 which will become common index for rare words.\n",
    "\n",
    "**Warning:** the below block of code can take a long time to execute.\n",
    "\n",
    "#### Learn the vocabulary from the training dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:....tokenizing data\n",
      "WARNING:root:Setting maximum document length to 105 based upon hueristic of 0.75 percentile.\n",
      " See full histogram by insepecting the `document_length_stats` attribute.\n",
      "WARNING:root:(1/2) done. 855 sec\n",
      "WARNING:root:....building corpus\n",
      "WARNING:root:(2/2) done. 145 sec\n",
      "WARNING:root:Finished parsing 2,698,578 documents.\n",
      "WARNING:root:...fit is finished, beginning transform\n",
      "WARNING:root:...padding data\n",
      "WARNING:root:done. 102 sec\n",
      "WARNING:root:....tokenizing data\n",
      "WARNING:root:Setting maximum document length to 10 based upon hueristic of 0.75 percentile.\n",
      " See full histogram by insepecting the `document_length_stats` attribute.\n",
      "WARNING:root:(1/2) done. 66 sec\n",
      "WARNING:root:....building corpus\n",
      "WARNING:root:(2/2) done. 17 sec\n",
      "WARNING:root:Finished parsing 2,698,578 documents.\n",
      "WARNING:root:...fit is finished, beginning transform\n",
      "WARNING:root:...padding data\n",
      "WARNING:root:done. 23 sec\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 8min 8s, sys: 32.3 s, total: 8min 40s\n",
      "Wall time: 20min 24s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "train_body_raw = traindf.body.tolist()\n",
    "train_title_raw = traindf.title.tolist()\n",
    "\n",
    "# Clean, tokenize, and apply padding / truncating such that each document length = 75th percentile for the dataset.\n",
    "#  also, retain only the top keep_n words in the vocabulary and set the remaining words\n",
    "#  to 1 which will become common index for rare words \n",
    "\n",
    "# process the issue body data\n",
    "body_pp = processor(heuristic_pct_padding=.75, keep_n=8000)\n",
    "train_body_vecs = body_pp.fit_transform(train_body_raw)\n",
    "\n",
    "# process the title data\n",
    "title_pp = processor(heuristic_pct_padding=.75, keep_n=4500)\n",
    "train_title_vecs = title_pp.fit_transform(train_title_raw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Apply transformations to Test Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:...tokenizing data\n",
      "WARNING:root:...indexing data\n",
      "WARNING:root:...padding data\n",
      "WARNING:root:...tokenizing data\n",
      "WARNING:root:...indexing data\n",
      "WARNING:root:...padding data\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 57.5 s, sys: 31 s, total: 1min 28s\n",
      "Wall time: 4min 16s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "test_body_raw = testdf.body.tolist()\n",
    "test_title_raw = testdf.title.tolist()\n",
    "\n",
    "test_body_vecs = body_pp.transform_parallel(test_body_raw)\n",
    "test_title_vecs = title_pp.transform_parallel(test_title_raw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Extract Labels\n",
    "\n",
    "Add an additional dimension to the end to facilitate compatibility with Keras."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_labels = np.expand_dims(traindf.class_int.values, -1)\n",
    "test_labels = np.expand_dims(testdf.class_int.values, -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Check shapes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the number of rows in data for the body, title and labels should be the same for both train and test partitions\n",
    "assert train_body_vecs.shape[0] == train_title_vecs.shape[0] == train_labels.shape[0]\n",
    "assert test_body_vecs.shape[0] == test_title_vecs.shape[0] == test_labels.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save pre-processors and data to disk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the preprocessor\n",
    "with open('body_pp.dpkl', 'wb') as f:\n",
    "    dpickle.dump(body_pp, f)\n",
    "\n",
    "with open('title_pp.dpkl', 'wb') as f:\n",
    "    dpickle.dump(title_pp, f)\n",
    "\n",
    "# Save the processed data\n",
    "np.save('train_title_vecs.npy', train_title_vecs)\n",
    "np.save('train_body_vecs.npy', train_body_vecs)\n",
    "np.save('test_body_vecs.npy', test_body_vecs)\n",
    "np.save('test_title_vecs.npy', test_title_vecs)\n",
    "np.save('train_labels.npy', train_labels)\n",
    "np.save('test_labels.npy', test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build Architecture & Train Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.keras.utils import multi_gpu_model\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.layers import Input, GRU, Dense, Embedding, BatchNormalization, Concatenate\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "import numpy as np\n",
    "import dill as dpickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.12.0\n"
     ]
    }
   ],
   "source": [
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the data and shape information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "with open('title_pp.dpkl', 'rb') as f:\n",
    "    title_pp = dpickle.load(f)\n",
    "\n",
    "with open('body_pp.dpkl', 'rb') as f:\n",
    "    body_pp = dpickle.load(f)\n",
    "    \n",
    "#load the training data and labels\n",
    "train_body_vecs = np.load('train_body_vecs.npy')\n",
    "train_title_vecs = np.load('train_title_vecs.npy')\n",
    "train_labels = np.load('train_labels.npy')\n",
    "\n",
    "#load the test data and labels\n",
    "test_body_vecs = np.load('test_body_vecs.npy')\n",
    "test_title_vecs = np.load('test_title_vecs.npy')\n",
    "test_labels = np.load('test_labels.npy')\n",
    "\n",
    "\n",
    "issue_body_doc_length = train_body_vecs.shape[1]\n",
    "issue_title_doc_length = train_title_vecs.shape[1]\n",
    "\n",
    "body_vocab_size = body_pp.n_tokens\n",
    "title_vocab_size = title_pp.n_tokens\n",
    "\n",
    "num_classes = len(set(train_labels[:, 0]))\n",
    "assert num_classes == 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Build Model Architecture\n",
    "\n",
    "We did very little hyperparameter tuning.  Keeping model simple as possible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "body_input = Input(shape=(issue_body_doc_length,), name='Body-Input')\n",
    "title_input = Input(shape=(issue_title_doc_length,), name='Title-Input')\n",
    "\n",
    "body = Embedding(body_vocab_size, 50, name='Body-Embedding')(body_input)\n",
    "title = Embedding(title_vocab_size, 50, name='Title-Embedding')(title_input)\n",
    "\n",
    "body = BatchNormalization()(body)\n",
    "body = GRU(100, name='Body-Encoder')(body)\n",
    "\n",
    "title = BatchNormalization()(title)\n",
    "title = GRU(75, name='Title-Encoder')(title)\n",
    "\n",
    "x = Concatenate(name='Concat')([body, title])\n",
    "x = BatchNormalization()(x)\n",
    "out = Dense(num_classes, activation='softmax')(x)\n",
    "\n",
    "model = Model([body_input, title_input], out)\n",
    "\n",
    "model.compile(optimizer=Adam(lr=0.001), \n",
    "              loss='sparse_categorical_crossentropy', \n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "Body-Input (InputLayer)         (None, 105)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "Title-Input (InputLayer)        (None, 10)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "Body-Embedding (Embedding)      (None, 105, 50)      400050      Body-Input[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "Title-Embedding (Embedding)     (None, 10, 50)       225050      Title-Input[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_15 (BatchNo (None, 105, 50)      200         Body-Embedding[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_16 (BatchNo (None, 10, 50)       200         Title-Embedding[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "Body-Encoder (GRU)              (None, 100)          45300       batch_normalization_15[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "Title-Encoder (GRU)             (None, 75)           28350       batch_normalization_16[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "Concat (Concatenate)            (None, 175)          0           Body-Encoder[0][0]               \n",
      "                                                                 Title-Encoder[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_17 (BatchNo (None, 175)          700         Concat[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "dense_5 (Dense)                 (None, 3)            528         batch_normalization_17[0][0]     \n",
      "==================================================================================================\n",
      "Total params: 700,378\n",
      "Trainable params: 699,828\n",
      "Non-trainable params: 550\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.callbacks import CSVLogger, ModelCheckpoint\n",
    "\n",
    "script_name_base = 'Issue_Label_v1'\n",
    "csv_logger = CSVLogger('{:}.log'.format(script_name_base))\n",
    "model_checkpoint = ModelCheckpoint('{:}_best_model.hdf5'.format(script_name_base),\n",
    "                                   save_best_only=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 2698578 samples, validate on 476220 samples\n",
      "Epoch 1/4\n",
      "2698578/2698578 [==============================] - 603s 223us/step - loss: 0.5008 - acc: 0.8086 - val_loss: 0.4699 - val_acc: 0.8209\n",
      "Epoch 2/4\n",
      "2698578/2698578 [==============================] - 599s 222us/step - loss: 0.4558 - acc: 0.8265 - val_loss: 0.4611 - val_acc: 0.8248\n",
      "Epoch 3/4\n",
      "2698578/2698578 [==============================] - 598s 221us/step - loss: 0.4380 - acc: 0.8332 - val_loss: 0.4593 - val_acc: 0.8250\n",
      "Epoch 4/4\n",
      "2698578/2698578 [==============================] - 597s 221us/step - loss: 0.4230 - acc: 0.8393 - val_loss: 0.4616 - val_acc: 0.8243\n"
     ]
    }
   ],
   "source": [
    "batch_size = 900\n",
    "epochs = 4\n",
    "history = model.fit(x=[train_body_vecs, train_title_vecs], \n",
    "                    y=train_labels,\n",
    "                    batch_size=batch_size,\n",
    "                    epochs=epochs,\n",
    "                    validation_data=[(test_body_vecs, test_title_vecs), test_labels], \n",
    "                    callbacks=[csv_logger, model_checkpoint])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluate Model\n",
    "\n",
    "Compute a confusion matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_impl.py:112: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.models import load_model\n",
    "from utils import plot_confusion_matrix\n",
    "\n",
    "\n",
    "#load the test data and labels\n",
    "test_body_vecs = np.load('test_body_vecs.npy')\n",
    "test_title_vecs = np.load('test_title_vecs.npy')\n",
    "test_labels = np.load('test_labels.npy')\n",
    "\n",
    "#load the best model\n",
    "best_model = load_model('Issue_Label_v1_best_model.hdf5')\n",
    "\n",
    "#get predictions\n",
    "y_pred = np.argmax(best_model.predict(x=[test_body_vecs, test_title_vecs],\n",
    "                                      batch_size=15000),\n",
    "                   axis=1)\n",
    "\n",
    "# get labels\n",
    "y_test = test_labels[:, 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normalized confusion matrix\n",
      "[[0.88173203 0.09765211 0.02061586]\n",
      " [0.1303451  0.83997974 0.02967516]\n",
      " [0.27873486 0.23896011 0.48230502]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f487d90af60>"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUIAAAEYCAYAAAApuP8NAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xd8FNX6x/HPN4RQRAEBKUFAEESaVDuKiohS7Ii9t2v36rVgu167V0UFr/2HXVFAAVEUFSsKCFgAFRGQ3kFBWsLz+2MmuAkhu5DN7ib7vH3ty92ZszNndsmzp805MjOccy6dZSQ7A845l2weCJ1zac8DoXMu7XkgdM6lPQ+Ezrm054HQOZf2PBCmAUl3SHo5fN5A0hpJ5eJ8jtmSusbzmNtx7rskLZO0qBjHKJHPJdEk3Szp2WTno7TxQBgHYRBYImmniG0XSBqbxGwVysx+N7MqZpabyPNK2lfSKEmrJK2QNF7SuXE4bgPgn0ALM6uzo8cpyc9FkoX/PjIjtpUPt8U0kFdSF0nzoqUzs3vM7ILi5DcdeSCMn3LAVcU9iAJl6nuRdADwMfApsCdQA7gUODoOh28ALDezJXE4VklaSf7rPTrcFjeRgdZtnzL1B5dkDwLXSapW2E5JB0qaIGl1+P8DI/aNlXS3pC+Bv4DG4ba7JH0VVtlGSKoh6RVJf4THaBRxjEclzQ33fSup8zby0SgsoWRKOiA8dt5jvaTZYboMSTdKmilpuaTBknaNOM6ZkuaE+/rF8Nm8YGb3m9kyC3xrZn0ijnehpF/D0uJwSfUi9pmkSyTNCEuUA8MfjK7Ah0C9MP+DCis5RVbbw5LpxPBzWizp4YKfS/i6XpiPFWG+Low43h3h5/GipD8lTZXUMcpn8BJwVsTrs4AXC+TzXEnTw2P+JunicPtOwHsR17kmzN8dkt6S9LKkP4BzlL8Z5BRJsyTtEr4+WtIiSbWi5DX9mJk/ivkAZgNdgaHAXeG2C4Cx4fNdCX79zwQygVPD1zXC/WOB34GW4f7y4bZfgSZAVWAa8Et4nkyCP6L/i8jDGQQlrUyCquIioGK47w7g5fB5I8CAzALXUJ6gxHZv+Poq4GugPlABeAp4LdzXAlgDHBLuexjIAboW8tlUBnKBw4r4/A4HlgHtw+M9DnwWsd+AkUA1ghLgUqB7uK8LMC8ibb7Xkd9P+HwccGb4vAqwf2GfC/AZ8ARQEWgbnvPwiM9zPXAMQU3gXuDrIq7PgFbA4vAaqofPWwEWka5H+H0LOJTgR7F9Edd1B7AJOI6gUFMp8rsO07wCDAr/bSwAeib77yUVH14ijK/bgCsK+cXtAcwws5fMLMfMXgN+AnpFpBlkZlPD/ZvCbf9nZjPNbDVBiWCmmY0xsxzgTaBd3pvN7GUzWx6+/yGCgLLXduT9MeBPIK90dwnQz8zmmdkGgj+wk8IS00nASDP7LNx3K7B5G8etTvBHurCIc58OPG9mk8Lj3QQcEFniBe4zs1Vm9jvwCUFw2hGbgD0l1TSzNWb2dcEEknYHDgJuMLP1ZjYFeJb8JbovzGyUBW2KLwH7RDnvemAEcEr4GB5u28LM3g2/bzOzT4EPgEJL9hHGmdnbZrbZzNYVsv8ygh+ascAIMxsZ5XhpyQNhHJnZjwQllxsL7KoHzCmwbQ6QHfF6biGHXBzxfF0hr6vkvZB0XVitWi1pFUEpsmYs+Q6rYF2A08wsL6A1BIaFVdFVwHSCkl3t8Hq25NfM1gLLt3H4lQRBsm4RWcj3+ZjZmvB4kZ9PZI/wX0Rc+3Y6H2gG/BQ2L/TcRn5WmNmfEdsKfl8F81Mxhja6FwmC6VbVYthSdf06rI6vIihxRvsOC/t3s4WZrSL40WwFPBTlWGnLA2H83Q5cSP4/mgUEgSVSA2B+xOsdngYobA/8F9AHqG5m1YDVBFWsWN77H+BYM/sjYtdc4GgzqxbxqGhm8wlKd7tHHKMyQdVrK2b2F0F19MQispHv8wnbxGqQ//OJ1VqC6njescoBW0roZjbDzE4FdgPuB95SRG9/RH52lbRzxLaC39eO+JzgB6E28EXkDkkVgCHAf4Ha4Xc4ir+/w239+yjy342ktsB5wGsEpX5XCA+EcWZmvwJvAFdGbB4FNJN0WthJcQpBO1u8qik7E7TRLQUyJd0G7BLtTWEVcDBwlpn9UmD3k8DdkhqGaWtJOjbc9xbQU9LBkrKAOyn639K/CBryr5dUIzzePpJeD/e/BpwrqW0YEO4BvjGz2VGvfGu/EJTOekgqD9xC0EyQd81nSKoVlnxXhZvzVevNbC7wFXCvpIqS2hCUJF/egfxEHtcImkN6h88jZYX5XArkSDoa6BaxfzFQQ1LVWM8nqWKY55uBc4FsSf8oxiWUWR4IS8adwJZShpktB3oSdGIsJwgMPc1sWZzONxp4nyAIzCFoeyqyyhQ6gqB08lZEb+TUcN+jBO1YH0j6k6DjZL/weqYStD29SlA6XAlsc4ybmX1F0E51OPCbpBXA0wQ/EJjZGIJ2xiHh8ZoAfWO9+ALnWg38g6BNbz5BCTEyb92BqZLWhNfYdxtta6cSdKAsAIYBt4f5LJawHXhqIdv/JPjxHEzweZ5G8Pnn7f+J4Afjt7C5ol7BYxTiXmCumf0vbHs9A7hLUtPiXkdZo61/mJxzLr14idA5l/Y8EDrn0p4HQudc2vNA6JxLe36TdhGUWcmUtXP0hGVU6712j56oDMvMiDoMs0ybNOnbZWYWt/uSy+3S0CynsA76/Gzd0tFm1j1e542FB8IiKGtnKuzVJ3rCMuq9Tx5OdhaSatcqWcnOQlJVKq+Cd0MVi+Wsi+nvaf2UgTHdERVPHgidc4khQUZqznvrgdA5lzgpOtWmB0LnXOIoNdtdPRA65xLEq8bOuXQnvGrsnEt38qqxc8551dg5l+bkVWPnXJoTXjV2zqU7QUZqhpzUzJVzrmxK0fu3PRA65xLDh88451zqDqhOzfDsnCubpOiPqIdQd0k/S/pVUsE1xJHUQNInkiZL+l7SMdGO6YHQOZc4yoj+KOrtwTrVA4GjCZbEPVVSiwLJbgEGm1k7gtUQn4iWLa8aO+cSIz7TcO0L/GpmvwWH1OvAscC0iDTG3+t6VyVYkrVIHgidc4lT/HGE2eRfs3se4XrbEe4gWI/7CoL1xbtGO6hXjZ1zCaJYq8Y1JU2MeFy0nSc6FRhkZvWBY4CXpKLr3F4idM4lhoi1arzMzDpuY998IHIxnfrhtkjnA90BzGycpIpATWDJtk7oJULnXILEXCIsygSgqaQ9JGURdIYML5Dmd+AIAEl7AxWBpUUd1EuEzrnEKWYboZnlSLocGA2UA543s6mS7gQmmtlw4J/AM5KuIeg4OcfMrKjjeiB0ziVOHAZUm9koYFSBbbdFPJ8GHLQ9x/RA6JxLDPk0XM4559NwOefSm4CMDC8ROufSmcJHCkrN8JwGjjxwb74bdis/vnM715175Fb7d69TnfefvpJxr93A+Ddu4qiDg9spMzMzeObOM5kw+GYmD7mF687rluisx8UnY0bTuVMrDmq/NwMeeXCr/V9/+TlHHbofDWpWZuQ7Q/PtG/zaSxzUoQUHdWjB4NdeSlSW4+qD0e/TpuVetGy+Jw8+cN9W+zds2MAZp51Cy+Z70vnA/ZgzezYAH435kAP37UDHtq05cN8OjP3k4wTnvDiEFP2RDKU2EEpqJOnHZOdjR2RkiP439uHYy5+g3Yl3cXL3DjRvXCdfmhsu6M6QDydxwKn3c9ZN/8ejN50CwIld21MhK5NOfe7hwNPv54ITD6JB3V2TcRk7LDc3l37XX8XLbw7nk6+/4+0hb/DLT9PzpcnefXceGfgsx53UN9/2lStX8Mj9dzFyzBe8+9GXPHL/XaxatTKR2S+23Nxcrr7yMt4Z8R6Tv5/Gm6+/xvRp0/KlGfT8c1SvVp2pP/3KFVddQ7+bbwCgRo2avPX2CCZO+YFnnn+B8845MxmXsMMyMjKiPpKSr6ScNc11atWImXOXMXv+cjbl5PLm6En07NImXxozY5edKgJQtUolFi5dHWzHqFwxi3LlMqhUIYuNm3L5c+36hF9DcUz+dgKNGjehYaPGZGVlcewJfRg9akS+NLs3aESLVq23+sP49KMP6dzlCKpX35Vq1arTucsRjB3zQSKzX2wTxo+nSZM92aNxcP0nn9KXkSPeyZdm5Ih3OP3MswE44cSTGPvxR5gZbdu1o169egC0aNmS9evWsWHDhoRfw47yEmHJyJT0iqTpkt6SVFnSbEk1ASR1lDQ2fF5L0oeSpkp6VtKcvHSJVm+3qsxb/HcpZv7ilWTXqpovzd1PjaLvMfvy6/v/Ydjjl3Lt/W8CMHTMZP5av5FZH97NL+/dSf8XP2LlH38lNP/FtWjhAupl/32XVN162SxaWPAuqW29dz716ke8N7t+zO9NFQsWzKd+xDVkZ9dn/vz5W6fZPUiTmZnJLlWrsnz58nxphg0dQtt27alQoULJZzoeFOMjCUp7INwLeMLM9gb+AP5RRNrbgY/NrCXwFtCgsESSLsq72dty1sU9w7Hq070jL4/4mj2738rxV/yP5+46C0l0atmI3NzNNO7Wj7173M5VZx5Oo+waScunS45pU6dyy803MOCJp5KdlZgJedW4hMw1sy/D5y8DBxeR9mDgdQAzex8otGHJzJ42s45m1lGZleKa2TwLlqymfu3qW15n167O/LDqm+fs4w5gyAeTAPjm+1lUzCpPzWo70efojnzw1TRycjazdOUaxk35jQ4tCo3pKatO3XosmP/3TEoLF8ynTt3sGN+bzYJ5Ee+dPy/m96aKevWymRdxDfPnzyM7O3vrNHODNDk5OfyxejU1agQ/ePPmzeOUk4/n2edfpHGTJonLeBx41bhkFLx/0IAc/r6uionNTmwmTp3Dng1q0bBeDcpnluPko9rz7tjv86WZu2gFXfbdC4C99qhNxQrlWbpyDfMWraBLp2B75YpZ7NumET/PXpzwayiOtu07Mmvmr/w+ZxYbN27knaGD6XZ0z5jee+gRR/LZJ2NYtWolq1at5LNPxnDoEVv3uqeyjp068euvM5g9K7j+N994nR49e+dL06Nnb1556QUAhg55i0MPOxxJrFq1ihN69+A/d9/HgQdt111kKcEDYcloIOmA8PlpwBfAbKBDuO3EiLRfAn0AJHUDqpMkubmbueb+wYx44jKmDL2FIR9MZvpvi7j10h70OLQ1ADc+PIzzTjiQb964kRfuPZcLbwuGiTz5xmdUqZzFt2/144tXrueld77mxxlRJ+BNKZmZmdz1QH9OO7EnXfZrQ6/jTmKvvVvw4D3/5oOw02TKpIl0aNmYke8M4YZrLuOwA9oCUL36rlx9/c30OPxAehx+INf8qx/Vq5euXvPMzEweeXQAvXocRdvWe3PiyX1o0bIld95xGyNHBBOpnHPe+SxfsZyWzffksf4Pc9fdwRCbJ58YwMyZv3LvXXeyX4e27NehLUuWbHN2qdQiUIaiPpKStSiTMqQsSY2A94GJBIFvGnBm+Pw5gjbDsUBHM+siaTfgNaA2MA7oCTQys212uWVU3s0q7NWn5C4ixc385OFkZyGpdq2SlewsJFWl8vq2iHkBt1v5mk2sWq97oqZbNqhvXM8bi1J7Z4mZzQaaF7Lrc6BZIdtXA0eF0/gcAHQqKgg65+IvWVXfaEptINwBDYDB4ZTdG4ELk5wf59JLWDUu9mGk7sCjBPMRPmtm9xXY/whwWPiyMrCbmVUr6phpEwjNbAbQLtn5cC6dFbdEGLGc55EECzdNkDQ8nIMQADO7JiL9FcTwd1/aO0ucc6VIHHqNtyznaWYbCYbEHVtE+lMJ+gaKlDYlQudccomYe4VrSpoY8fppM3s6fB7Lcp7B+aSGwB5A1JkpPBA65xJDMVeNi1rFbnv0Bd4ys9xoCT0QOucSJg69xrEs55mnL3BZLAf1QOicS5g49BpvWc6TIAD2JbiZIv95pOYEN02Mi+Wg3lninEuY4naWmFkOkLec53RgcN5ynpIi71PsC7webRnPPF4idM4lRLzuJY62nGf4+o7tOaYHQudcwvjiTc45l5p32HkgdM4ljt9r7JxLa1KwcFkq8kDonEuQ5E28Go0HQudcwqRoHPRA6JxLEK8aO+fSnfBA6JxzXjV2zqU5rxo759Kd8HGEzrm058NnnHMuZavGqXkHtHOu7FHQWRLtEfUwUndJP0v6VdKN20jTR9I0SVMlvRrtmF4idM4lRDzaCGNZxU5SU+Am4CAzWylpt2jH9UDonEuYOFSNt6xiByApbxW7aRFpLgQGmtlKADNbEjVfxc2Vc87FKsaqcU1JEyMeF0UcorBV7LILnKYZ0EzSl5K+DheEL5KXCIvQsll93vngwWRnI2ma9Lor2VlIquUf/TvZWShbEreKXSbQFOhCsLjTZ5Jam9mqot7gnHMlTigeVeNYVrGbB3xjZpuAWZJ+IQiME7Z1UK8aO+cSJg69xltWsZOURbBI0/ACad4mKA0iqSZBVfm3og7qJULnXMIUt9fYzHIk5a1iVw54Pm8VO2CimQ0P93WTNA3IBa43s+VFHdcDoXMuIeI1Q3W0VezCJTyvDR8x8UDonEsYv8XOOZf2UjQOeiB0ziWIT8PlnEt3Ko2zz0japag3mtkf8c+Oc64sS9E4WGSJcCpg5F+bPu+1AQ1KMF/OuTKoXGmrGpvZ7tva55xz20ux32KXcDHdWSKpr6Sbw+f1JXUo2Ww558qiDEV/JCVf0RJIGgAcBpwZbvoLeLIkM+WcK5syMhT1kQyx9BofaGbtJU0GMLMV4T1+zjkXMxH0HKeiWALhJkkZBB0kSKoBbC7RXDnnyqQU7SuJKRAOBIYAtST9G+gD+ERtzrnto+RVfaOJGgjN7EVJ3wJdw00nm9mPJZst51xZIyCjNPcaE0x3swnYuB3vcc65fBKxip2kcyQtlTQlfFwQ7Zix9Br3A14D6hHMBvuqpJuiZ9c55/6WNw1XcXqNI1axOxpoAZwqqUUhSd8ws7bh49loeYuljfAsoJ2Z/RVm5G5gMnBvDO91zrkt4lA1jmUVu+3PVwxpFpI/YGaG25xzbrsohgfFX8UO4ERJ30t6S1LUu+SKmnThEYIhMyuAqZJGh6+7UcQiKM45VxgR873GxV3FbgTwmpltkHQx8AJweFFvKKpqnNczPBV4N2L718XIoHMuXSku03BFXcWuwPokzwIPRDtoUZMuPLedGXTOuSLFYfTMllXsCAJgX+C0/OdQXTPLa77rDUyPdtConSWSmgB3E/TQVMzbbmbNYs66cy7tbUfVeJtiXMXuSkm9gRyCpr1zoh03ll7jQcBdwH8JuqzPJbzdzjnntkc8puGKYRW7m4DtGuIXS69xZTMbHZ5gppndQhAQnXNuu8TYa5xwsQTCDeGkCzMlXSKpF7BzCeerzPv04w/oesA+HLZvK5587L9b7R8/7gt6H3EAzeruzHsjhm3ZPn/u7/Q+4gB6HrYf3Tt34NVBzyQy23Fz5L578t0rV/Lja1dx3emdt9q/+25Vef/Rcxn33KWMH/QPjtq/6Vb7l47ux9V9D0pUluPqg9Hv07ZVc1rv3ZT/PnjfVvs3bNjAWaf3pfXeTTn04P2ZM3s2ABMnjGf/Tu3Yv1M79uvYluHvDNvqvalKCqrG0R7JEEsgvAbYCbgSOAi4EDivOCeVdKWk6ZJe2c73NZJ0WvSUqS03N5c7briG5197m9FfTGLE0DeZ8XP+9tx62bvzwGNP0+uEU/Jtr1W7Dm+OGsvIT75hyHuf8uTjD7F40YJEZr/YMjJE/2t7cux1L9HuzAGc3LU1zRvVypfmhrMPZcgnP3LA+f/jrDve5NFre+bbf/8V3fngmxmJzHbc5Obmcu1VlzNs+Ci+/W4qb77xOtOn5x8P/ML/PUe1atX4YfoMLr/yam7tF9xJ1qJlK74YN4GvJ0zm7RHvccVll5CTk5OMy9ghCnuOi3okQ9RAaGbfmNmfZva7mZ1pZr3N7MtinvcfwJFmdvp2vq8RBXqIYhHelpMyvps0kYZ7NKFBoz3Iysqi5/EnMeb9kfnS1G/QkOYtW5ORkf8rysrKokKFCgBs3LiBzZtL34xonfauz8z5K5i9cCWbcnJ586Mf6Hlw83xpzIxdKgfXWXWniixc9ueWfb06N2f2wpVMm7U0ofmOl4kTxtO4yZ7s0bgxWVlZnNTnFEaOeCdfmpEjhnP6mWcDcPwJJzH2k48wMypXrkxmZtC0v2H9+pSd+n5b4nGvcUnYZiCUNEzS0G09dvSEkp4EGgPvSeon6XlJ4yVNlnRsmKaRpM8lTQofB4Zvvw/oHN5IfU14c/WAiGOPlNQlfL5G0kOSvgMOkNRB0qeSvpU0WlLdHb2G4lq8aAF1s/8eDF+nbjaLF8Zeqlswfx7HHLovB7drxsWXX0vtOvVKIpslpl6tnZm3ZPWW1/OX/kF2zfyLJt79f5/Qt9s+/Drknwx78Ayu7R8MZd2pUhb/PK0zd//f2ERmOa4WLJhP/d3rb3mdnV2fhfPnb52mfjBcLjMzk112qcry5cHwuAnjv6Fj21bs26ENjw3435bAmOqk6NXiZFWNi/oEBxSxb4eZ2SWSuhNM/38t8LGZnSepGjBe0hhgCUGJcb2kpgSTPnQEbgSuM7OeEMwyUcSpdgK+MbN/SioPfAoca2ZLJZ1CMCRoqyp+eDvPRQD16qfm+lX1susz6tPxLF60gEvOPoWjex1Pzd1qJztbcdWnaxtefm8yj77xFfu13J3nbj2RDmcN5JZzD+PxwV+xdt3GZGcxaTrtux8Tp/zIT9Onc9EF59DtqKOpWLFi9DemgFQtwRY1oPqjBJy/G9Bb0nXh64oEy4QuAAZIagvkAjsyZjGXYEJZgL2AVsCH4RdRjm3cL21mTwNPA7Ru275EhgnVrlMvXwlg0cL51K67/aW62nXq0ax5CyZ88xVH9zo+nlksUQuW/kn93apueZ1daxfmL8u/TPbZPdpz7HUvAvDN1LlUzMqkZtXKdGpRn+O7tODuS7tRtUpFNpuxfuMmnhw6PqHXUBz16mUzb+68La/nz5+Xr4awJc28uWTXr09OTg5//LGaGjVq5EvTfO+92alKFaZN/ZH2HYpzR1ripOocfsnOl4ATI6bLaWBm0wk6aBYD+xCUBLe1RkoO+a8h8mdxvZnlRpxnasR5WptZt/heSuzatOvA7N9+Ze6c2WzcuJGRw97iiKN6xPTehQvmsX7dOgBWr1rJxG/G0bhJ0yjvSi0Tf5rPnvV3pWHdapTPLMfJR7Tm3S9+ypdm7uLVdOnQGIC9GtakYlYmS1etpevlz9G8zyM07/MIA978mgdf+rxUBUGADh07MfPXGcyeNYuNGzfy1uA36NGzd740PXr24pWXXgBg2NC3OLTL4Uhi9qxZWzpHfp8zh19+/okGDRsl+hJ2SN6A6tJWNU6E0cAVkq4wM5PUzswmA1WBeWa2WdLZBCU4gD/JP3RnNvCPcHhPNsEUPYX5mWCpgQPMbFxYVW5mZlNL4qKiyczM5Pb7HuacU3qzOTeXk047i2bNW/DIfXfSum17unbvyfeTJ3LpOX1ZvXoVH38wikcfuIv3P/+Wmb/8zD2334QkzIwL/nEVe7VolYzL2GG5uZu55pF3GfHQWZTLyOCFdycxffZSbj3/cCb9NJ93v/yZGwe+zxP/OpYr+hyImXHhPaVnmEg0mZmZPNT/cY7t2Z3c3FzOOudcWrRoyX/+fRvt23ekR6/enH3u+Vxw7lm03rsp1XfdlRdeeg2Ar776gocfvJ/M8uXJyMig/6MDqVmzZpKvKHYpOlM/Mout9iepgpltiMtJpdkEJb21QH/gQIKS3Swz6xm2Cw4huIPlfeAyM6sSBrDRQA2CO176Ay8DHQjuJ6wO3GFmYyWtMbMqEedsCzxGEGQzgf5mVuQgvNZt29s7Hxa3g7z0anlCek85ufyj9F6aZ6cKGd8WcxaYfOo0bWWnPzwkarqHezeP63ljEcu9xvsCzxEEkAaS9gEuMLMrdvSkZtYo4uXFheyfAbSJ2HRDuH0TW0+nU+gQnMggGL6eAhyyA9l1zsVJuWQ3xm1DLNl6DOgJLAcws+8Ienydcy5meYs3RXskQyxthBlmNqdAt3futhI759y2pGiBMKZ8zQ2rxyapnKSrgV9KOF/OuTImXgOqo61iF5HuREkmKWp7YyyB8FKCgc8NCIa07B9uc8657VLcW+xiXcVO0s7AVcA3seQrlgXelxDMAuucc8USh+Ezsa5i9x/gfuD6WA4aS6/xMxQyEauZXVRIcuecK1Q8Zqim8FXs9st3Hqk9sLuZvSspPoEQGBPxvCJwfIGMOOdcdIq5RFhT0sSI10+Ht75GP0Vwc8XDxDA9f6RYqsZvFDjRS8AX23MS55wDUGxzUBe1nGe0Vex2JphXYGw40qUOMFxSbzOLDK757MgtdnsAZWuqE+dciROQWfzxM0WuYmdmq4Et9xxKGkswY9U2gyDE1ka4kr/bCDMIVoXaZpe1c85tS3Gn4YpxFbvtVmQgVJDrffi76LnZYr052TnnIgR3lhT/ONFWsSuwvUssxywyEIYzwowys9I1vYlzLvUoLr3GJSKWGvsUSe1KPCfOuTItr0QY7ZEM2ywRSso0sxygHTBB0kyCabNEUFhsn6A8OufKBFGutE3VD4wH2gO9i0jjnHMxEclbpS6aogKhAMxsZoLy4pwry5JY9Y2mqEBYS9K129ppZg+XQH6cc2VUnG6xKxFFBcJyQBWIbSi4c85Fk6yJV6MpKhAuNLM7E5YT51yZl6JxMHoboXPOxYNEqew1PiJhuXDOpYXUDINFBEIzW5HIjDjnyra8xZtSUbIXeHfOpZEU7TT2QOicSxQVe/aZkuKB0DmXEKJ0L+fpnHNxEY8F3qMt5ynpEkk/SJoi6YvCVrkryEuERVi/aTM/L/kz2dlImo+evybZWUiqS9/6IdlZKFtU/IlZI5bzPJJg4aYJkoabWeQqdq+a2ZNh+t4Ea5h0L+q4XiJ0ziVEXtU42iOKLct5mtlGIG85zy3M7I+IlztRyCqcBXmJ0DmXMDEOnylqFbuoy3kCSLoMuBbIAg6PdkIaqqGeAAAWy0lEQVQPhM65hImxZlzUKnYxMbOBwEBJpwG3AGcXld6rxs65hAiqxor6iCLacp4FvQ4cF+2gHgidcwkSvcc4hqrzluU8JWURLOeZb+U6SU0jXvYAZkQ7qFeNnXMJU9zx1DEu53m5pK7AJmAlUarF4IHQOZcgeVXj4oq2nKeZXbW9x/RA6JxLDEFGijbGeSB0ziWMUnQiLg+EzrmEyFvXOBV5IHTOJYzPR+icS3teNXbOpTWvGjvnXIzTbCWDB0LnXMKkZhj0QOicSxBfvMk55yidC7w751xcea+xcy7teYnQOZf2UjUQpugt0M65skYEVeNo/0U9TvRV7K6VNE3S95I+ktQw2jE9EDrnEkNBiTDao8hD/L2K3dFAC+DUQpbrnAx0NLM2wFvAA9Gy5oHQOZcwxQ2ExLaK3Sdm9lf48muC6fyL5IHQOZcgsVSMBeEqdhGPiyIOUtgqdtlFnPR84L1oOfPOEudcwiRqFbvgXDoD6AgcGi2tB0LnXEKIuPQax7SKXbhmST/gUDPbEO2gXjV2ziVMHHqNY1nFrh3wFNDbzJbEki8vESbJxC8+5qn7bmFzbi5HnXg6fS64Mt/+oS88yeghr1CuXDmq7lqDq//Tn9r1gh/C5x66kwmfjcE2b6bdAYdw8U13o1QdoLUNX382hv5338zm3Fx6nXwmZ158db79rz8/kBFvvkS5zEyqVa/Jzfc+Tp3svwsCa9f8welHH0Dnrj345+1ROwVTTuu6VTi9QzYZgk9nruDdaUsLTddx9124onMjbn9/BrNXrKOc4Lz96tNw10qUk/hy1kpGbuO9qShBq9g9CFQB3gz/Ln43s95FHbfUlggl3Vzg9VfJysv2ys3N5Ym7buTO/73Kk8M/59NRw/h95s/50jTZuxWPvjGaJ4aN5eAje/H8Q3cCMG3yBKZNHs/AoZ/wxNuf8svUKfwwodRcOhBc/0P//hcPPTOYV0aNY8zIIcz69ad8aZq2aMNzQz/mxRFfcFj33gx84PZ8+5/pfw9tOx2YyGzHjQRndczmoU9mcdO7v7B/w2rU26XCVukqZmbQba+a/Lps7ZZtnRpUI7NcBreMmsHt78+gy541qLlT+URmf8fFYfgMBKvYmVkzM2tiZneH224LgyBm1tXMaptZ2/BRZBCEUhwIgXyB0MxKzV/FLz9Mol6DPai7eyPKl8/ikKOPY9zH7+dLs8++B1OxUmUAmu/TgWWLFwLBP5RNGzeQs2lj+P8cqtWolfBrKI7p339L/YZ7kN2gEeWzsjiixwl8PiZ/x16H/Ttvuf6WbTuydPGCLft++nEKK5YtpdPBhyU03/HSuEZlFq/ZyNK1G8ndbHwzZxXt6++yVboT2tTm3WlL2ZRr+bZXyMwgQ1C+XAa5m411mzYnKuvFFo8B1SWhRAOhpH6SfpH0haTXJF0naaykjuH+mpJmh8/LSXpQ0oRwRPjF4fa6kj6TNEXSj5I6S7oPqBRueyVMtyb8v8Lj/CjpB0mnhNu7hOd+S9JPkl5RkuqTy5csomadelte16xdj+VLFm0z/eihr9Kx8+EA7N22E206HcQZh7XhjMPa0OGgLjRo0qzE8xxPSxcvZLc6f4942K1OPZaGgb4wI958mf0P6QrA5s2bGXDfrVx+450lns+SUr1SeVas3bTl9Yq/NlG9cv5SXcPqldi1chbfLfgz3/YJv69iQ85mHj2+BY8ctzfvTV/K2o25Ccl3ceV1lhS3RFgSSqyNUFIHgobMtuF5JgHfFvGW84HVZtZJUgXgS0kfACcAo83s7nBUeWUz+1zS5WbWtpDjnBCecx+gJjBB0mfhvnZAS2AB8CVwEPBFgXxfBFwEsFvdqOMwS9zHI95ixtQpPDDobQAW/D6Lub/N4MWPpgDQ78KT+fHbr2nVYf9kZrPEjH5nMD/9OJmBr4wEYOgrz3HAoUfmC6RljYBT29fl2a/nbrWvcY3KbDbj6mHTqJxVjn5H7snURWtYunZj4jO6A1K1KbskO0s6A8PyRnhLGh4lfTegjaSTwtdVgaYEvUTPSyoPvG1mU6Ic52DgNTPLBRZL+hToBPwBjDezeWF+pgCNKBAIzexp4GmApi3b5q+TxEmN3eqwbNHfVb1lixdQY7c6W6WbPO5T3ni6P/cPGkb5rKAN6asxo9hrnw5UqrwTAB0PPoLp300sVYGwVu26LFn094iHJYsWUKt23a3STfhyLC/87yEGvjKSrPD6f5wyge8njmPoq8+xbu1aNm3aSOXKO3Hp9bdv9f5UtXLdJnaNaNfbtXJ5Vv71dwmxYvkM6letyI1HNAGgaqVMrj6kEf0/m83+jarxw4I/yTX4c0MuM5auZY8alUpPIEzRabiS0UaYE3HeihHbBVwR0cC5h5l9YGafAYcQjBUaJOmsYpw7cjxRLknqNW/Wqh0Lfv+NRfPmsGnTRj577232P+yofGlmTv+Bx/99PbcNeDFfG2Ctutn8OPErcnNyyNm0iR8mfkWDxk0TfQnF0rx1e+bN/o0Fc+ewaeNGPnp3KAcf0T1fml+mfc8Dt13L/U++SvWI67/joacZ+ukPDPnkOy678U66H9e3VAVBgFnL/6L2zlnU3Kk85TLEfg2rMXn+H1v2r9u0mcuHTuO64T9x3fCfmLnsL/p/NpvZK9axfO0mWtSuAkBWOdGkZmUW/hF1mFzKSLuqMfAZQeC6NzxPL4KxPbOBDsB44KSI9KOBSyV9bGabJDUjCH41gXlm9kxYZW4PvAhsklTezDaR3+fAxZJeAHYlCKLXA81L6Dq3W7nMTC69+V5uubgvm3Nz6Xb8qTTcszkvDbifpi33Yf/DuvPcQ/9m/V9ruffaC4AgAN4+4CUO7taL78d/wT+O7wISHQ4+jP26HFX0CVNMZmYm19z2ANeefxK5ubn0POl0Gjfdm2cevYfmrdrR+YijGXj/7az7ay23XHkuALXr1eeBJ19Ncs7jY7PBSxMXcP1hjckQfPbbSuav3sDxrWsze8W6fEGxoI9mLOeC/etzzzHNQPD5byuZu2p9AnNfPKlaNZZZidT+goNL/YCzgSXA7wTthCOBwQQlsneBM8yskaQM4C6CgClgKXBc+Lge2ASsAc4ys1mS7gd6A5PM7HRJa8ysStgB8gDB7BQG3GVmb0jqAlxnZj3DvA0gGHc0aFv5b9qyrT02+IO4fialSdWsUjIso4Q8NWHrNrp08uLp+3wbj1vd8rTep70N/eDLqOma1akc1/PGokQDYb4TSXcAa8zsvwk5YRx4IPRAmM7iHgjbtrdhMQTCprUTHwj9zhLnXMKkatU4YYHQzO5I1Lmcc6koeQOmo/ESoXMuYdK+ROicS29xmoarRHggdM4lTKpWjUvzpAvOuVImHgOqY1jF7hBJkyTlRNypViQPhM65xBBkxPAo8hCxrWL3O3AOEPMIfK8aO+cSqNhV4y2r2AFIylvFblpeAjObHe6LeX4yLxE65xJiO6bhiucqdjHxEqFzLmGiVX1DcVnFbnt4IHTOJUwceo1jWsVue3nV2DmXOIrhUbSoq9jtCA+EzrmEUBx6jc0sB8hbxW46MDhvFTtJvYPzqJOkecDJwFOSpkbLm1eNnXMJE48B1WY2ChhVYNttEc8nEFSZY+aB0DmXOKl5Y4kHQudc4sTYa5xwHgidcwni03A559Kczz7jnHN4IHTOOa8aO+fSXBLXLY7GA6FzLiG8jdA55/CqsXPOeYnQOec8EDrn0l6qVo1lZsnOQ8qStBSYk8Qs1ASWJfH8yebXn9zrb2hmteJ1MEnvE1xTNMvMrHu8zhsLD4QpTNLERM/Um0r8+tP7+hPJ5yN0zqU9D4TOubTngTC1PZ3sDCSZX79LCG8jdM6lPS8ROufSngdC51za80DonEt7Hgidc2nPA2EKk7RnsvOQaFLhd6NKSqt/q9v6HFzJ8HuNU4wkmZlJ2gd4V9KnZnZ6svOVKBYOY5B0AVCLYBq7Z81sSVIzliB53z+wM/BHsvOTLtLqV7Y0CIPgMcD9wGNAZ0mDkpurxJJ0FdAX+AY4CTg/uTlKnPD7PxoYJul+Se3TrTScDP4BpxBJ5SRVBC4FXjGzB4AmQGtJr0SkK1PVpsjrkZQJNACOBNoDC4EHJFWWVCFJWUwYSfsCVxL8CFYDTgeO8GBYsvzDTS1ZZrYe+AlYB2Bmm4AzgWMl3RtuKzOj4CVlRFSHexME/obAh8D+wPFmlgucRhAcyyxJ9YAHge/N7B3gX8BqoDtwlAfDkuMfbIqQ1ISgOlQPGA9cE9FZUgEYDJwg6cRk5bEkmNlmAEmHANeb2c/AQGAP4A0z2yjpbOCfwLTk5TQh/gI+IfieO5vZauBhYCNwNFA9mZkry7yzJHWsBqYAA4CzgN2AJyX9DhwOHENQUlyftByWkLBN7A3g4nDTROB64L+SegCtgJPM7LckZbFERHSMtSH4vmcB/YH5wPWSNpvZl5LuBuqa2fJk5rcs83uNk0zSnmb2a/i8BkH7UFuCtqFaQDawAGhE8Edykpn9kpzcxp+kima2XtJYYGcz6xCxLxswIKes9RpLKmdmueGPwECCkmAHgu94EnAAcApwu5l9nrycpgevGieBpHLh/zOAtyXdDhD+4g8gKPW9BWw2sy8IgsGdwBllLAh2BfpLqmFmXYB14SzGeRaa2YKyFAQl1QYIg2BV4ELgbDM7H7gZOBioBwwC3iFsK3YlywNhAknaGbb8ERxC0Pj/KHCOpMvCfUuBCcBaghIhZjYLOMbMvk9KxkvODwTtXjdJ2tXMDgbKSfoK/m4/LCvC3vGnJTUFCNsA/wJahK9HETQLXAfkAAPNbGKSsptWPBAmiKTKBAOkT5TUHHiKoC2wCcEQkVsk3RB2DORVibb8EZhZmRlcK6mzpNZmthi4gqB97A5JVczsSGCZpAbJzWX8hb3jJwJIyptr8GOggaT9wtffAEuATDPLSXwu05O3ESaQpOOBGwlKe7eY2Vdhz3B3YD+COwl2BoaZ2bDk5TS+wgC3JnzenCDQ1wEeM7PpkmoBHwEzgIvNrEwt2CRpJ2CDmeVIagwsIlgU7CkzuyXsDGkC5AJtCH4EhyYvx+nHe40TyMyGSfoTGELQE/wVwR/E7wTDRa4L01nErValWjhAvJekdUBFgs6fDwmaBS6S9KyZTZX0MnAIQXtoWXMgcJmk5wm+427AXsC3Yc9wv/AHYh/gYTP7tqx8/6WFB8IEM7Mxks4BHpQ008xek7SaIAjUApaG6crKH8EmgmE/rwOVgeZmtlbSJoKS8IOSvicIFmeXxSEiZvahpOuAocBR4aD59ZLaA+Ml7WZmlxB8TnnvKSvff6nggTAJwpJhDvCCpL4EvcR3lrHeUVkgNywFryIo+Z4AvGRmEyQtBH4mKB1fEnYKlRkR4wSrAKOANQRjIw8ys7/MbKWkTgTBsDUwtax1EJUW3kaYRJJOIBgWc6GZjSsr1aHI65BUzcxWhc8PJGgjHW1mAyV1BGaa2cokZrdERATBY4HDgHvMbEk4gca+ZtZCwQxD2cB7ZeF7L808ECZZOGxkRbLzURIkXQscStA2+LSZDQkHEF9FMGykFnCKmS1IYjZLjKQjCWYRuszMxkVsfxI4iGDUxr/M7N0kZdGFvGqcZGU4CF4K9AJ6As8DgyWdb2aDwtsGzwH6ldUgGDoEeAmYI+k0gs/jWzO7JBxMvtjMfkhqDh3ggdCVgHDg8FqCOQUvJugJ7k4wjjLDzJ4nuJe4TImoDpcPZw2aQtAmejbwGsGtcw3CWsCYZObV5edVY1csYdBTxCwyWWa2MXxeF3gRuMDM5kgaCbQD9gbWlMWOAUndCUqCiwgGS28gGEP4u6S2BCXEE8xsRhKz6QrwEqErrp0iBktfDTSRVBPoBywDZgP7hrPI/AZcVJbukokkqQvwEMEdQ28D9QnuH5akI4AngGs9CKYev8XO7TAFE6k+Gj4/A+gN3EDQEZAX8KYRlJAuBZ4pi22CkjLDCTSOIOgIKk9QInwsvE2uIsEYygu9YyQ1edXY7ZBwyrA3gMsJeoCvBV4muFWwJ3CcmW2ISF81nGSgzJG0UzhI/GyCNsG6wMlhc8CZBBPrPudDZFKXlwjdjtpIMEPK7QRz6K0F7iEoFfU2sw2Sbpd0W5i+rFaH9wAmhzPKTCC4h7o/MD8cJ/gvYL4HwdTmgdDtEDP7k6AzoBcwjuCPvwHBbWQ1wztmjgfeDNOXyUAQ3g3zCsF1rgL+Q3Av8SiCuSVvMbP3wk4ll6K8aux2mKSGQFOCP/g7gbkEVWUDqhKsQfJj8nJYcsKS4DozWxS+volwgSkzWxROI6awelwm7hgqyzwQumKT1IGgvfBWgkWmMoDKZbhNsAnwHPAZweSpi8PtgwhmmD7GytBM4unAq8au2MzsW4IJRx8nmE9wU1kLgnlV27Dd7zyCZoF9gXMl1QmTjSaYRKJOoQdxKctLhC5uJLUiqC7OTHZeSoKkXgTzCVYCfgWmEwwN+oags+hk4MxwfkWvDpciHgidi4Gk3Qg6gi4ys2mSLgdqEgTAcsAuwHgzezuJ2XQ7yKvGzsVmE8GdWDXC108R3DnSlWDw9H/M7G2FKxS60sUDoXMxCOdMHAwcJqlVOKnCmwQlwlYEKxFWMrPcZObT7RgPhM7FbjCQBTwcLrj0OPAkwW2EDcN9rhTyNkLntoOCtakPICgFTjazT8Lq8M55M3G70scDoXPFIKmcV4dLPw+Ezrm0522Ezrm054HQOZf2PBA659KeB0LnXNrzQOi2kJQraYqkHyW9KalyMY7VJVysCUm9Jd1YRNpqkv6xA+e4Q9J1sW4vkGaQpJO241yNJJXJKcWcB0KX3zoza2tmrQhmoL4kcqcC2/1vxsyGm9l9RSSpBmx3IHQuXjwQum35HNgzLAn9LOlF4Edgd0ndJI2TNCksOVaBYClLST9JmkSwdgfh9nMkDQif15Y0TNJ34eNA4D6C1e+mSHowTHe9pAmSvpf074hj9ZP0i6QvgL2iXYSkC8PjfCdpSIFSbldJE8Pj9QzTl5P0YMS5Ly7uB+lSnwdCtxVJmcDRwA/hpqbAE2bWkuDe2luArmbWHpgIXCupIvAMwdT9Hdj2nHyPAZ+a2T5Ae2AqcCMwMyyNXi+pW3jOfYG2QAdJh4QTwPYNtx0DdIrhcoaaWafwfNOB8yP2NQrP0QN4MryG84HVZtYpPP6F4WzUrgzzdY1dpEqSpoTPPyeYhbkeMMfMvg637w+0AL4M5yrNIlizpDkwK2/NXkkvAxcVco7DCdb9JbwjY7Wk6gXSdAsfk8PXVQgC487AMDP7KzzH8BiuqZWkuwiq31UIJk/NMzhcZH6GpN/Ca+gGtIloP6wanttnnC7DPBC6SOvMrG3khjDYrY3cBHxoZqcWSJfvfcUk4F4ze6rAOa7egWMNIlha9DtJ5wBdIvYVvK3KwnNfYWaRARNJjXbg3K6U8Kqx215fAwdJ2hOCNX0lNQN+AhqF63kAnLqN939EsNh7XntcVeBPgtJentHAeRFtj9nhxKifAcdJqhROftArhvzuDCyUVB44vcC+kyVlhHluTDDN/mjg0jA9kppJ2imG87hSzEuEbruY2dKwZPWapArh5lvM7BdJFwHvSvqLoGq9cyGHuAp4WtL5QC5wqZmNk/RlODzlvbCdcG9gXFgiXQOcYWaTJL0BfAcsIVhHOJpbCabSXxr+PzJPvwPjCWaXvsTM1kt6lqDtcJKCky8Fjovt03GllU+64JxLe141ds6lPQ+Ezrm054HQOZf2PBA659KeB0LnXNrzQOicS3seCJ1zae//AdKn4cHQlu/cAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_confusion_matrix(y_test, \n",
    "                      y_pred, \n",
    "                      classes=np.array(['bug', 'feature', 'question']), \n",
    "                      normalize=True, \n",
    "                      title='Normalized Confusion Matrix')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Make Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import IssueLabeler\n",
    "from tensorflow.keras.models import load_model\n",
    "import dill as dpickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "#load the best model\n",
    "best_model = load_model('Issue_Label_v1_best_model.hdf5')\n",
    "\n",
    "#load the pre-processors\n",
    "with open('title_pp.dpkl', 'rb') as f:\n",
    "    title_pp = dpickle.load(f)\n",
    "\n",
    "with open('body_pp.dpkl', 'rb') as f:\n",
    "    body_pp = dpickle.load(f)\n",
    "    \n",
    "# instantiate the IssueLabeler object\n",
    "issue_labeler = IssueLabeler(body_text_preprocessor=body_pp,\n",
    "                             title_text_preprocessor=title_pp,\n",
    "                             model=best_model)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bug': 0.12618249654769897,\n",
       " 'feature': 0.1929263472557068,\n",
       " 'question': 0.6808911561965942}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "issue_labeler.get_probabilities(body='Can someone please help me?', \n",
    "                                title='random stuff')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bug': 0.019261939451098442,\n",
       " 'feature': 0.9305700659751892,\n",
       " 'question': 0.05016808584332466}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "issue_labeler.get_probabilities(body='It would be great to add a new button', \n",
    "                                title='requesting a button')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bug': 0.9065071940422058,\n",
       " 'feature': 0.03202613815665245,\n",
       " 'question': 0.06146678701043129}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "issue_labeler.get_probabilities(body='It does` not work, I get bad errors', \n",
    "                                title='nothing works')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}