ULMFiT Sentiment by Dmitriy Milevich.ipynb

{"cells":[{"metadata":{"_uuid":"8f4ba61032210e7c6c38d1e10b2945d3f7ff5f6d"},"cell_type":"markdown","source":"# Twitter US Airlines Sentiment with ULMFiT"},{"metadata":{"_uuid":"002138cc3d30e60595f6fdd0249b5274f69645d4"},"cell_type":"markdown","source":"## Introduction"},{"metadata":{"_uuid":"db8eaed4bba95cbf0d6e51e4af3fab888dec6867"},"cell_type":"markdown","source":"This notebook implements training the [AWD-LSTM](https://arxiv.org/pdf/1708.02182.pdf) architecure according to [ULMFiT](https://arxiv.org/pdf/1801.06146.pdf) paper on the [Twitter US Airline Sentiment Dataset](https://www.kaggle.com/crowdflower/twitter-airline-sentiment) using fastai library. The aim of the model is to determine sentiments of the tweets."},{"metadata":{},"cell_type":"markdown","source":"First, needed libraries are imported."},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd \nimport matplotlib.pyplot as plt\nimport seaborn as sn\n\nfrom fastai.text import *\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix, accuracy_score\n\n%matplotlib inline ","execution_count":272,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We read the file containing tweets into a DataFrame object and have a look at the first few records."},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.read_csv('../input/Tweets.csv')\ndf.head()","execution_count":273,"outputs":[{"output_type":"execute_result","execution_count":273,"data":{"text/plain":"             tweet_id             ...                           user_timezone\n0  570306133677760513             ...              Eastern Time (US & Canada)\n1  570301130888122368             ...              Pacific Time (US & Canada)\n2  570301083672813571             ...              Central Time (US & Canada)\n3  570301031407624196             ...              Pacific Time (US & Canada)\n4  570300817074462722             ...              Pacific Time (US & Canada)\n\n[5 rows x 15 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>tweet_id</th>\n      <th>airline_sentiment</th>\n      <th>airline_sentiment_confidence</th>\n      <th>negativereason</th>\n      <th>negativereason_confidence</th>\n      <th>airline</th>\n      <th>airline_sentiment_gold</th>\n      <th>name</th>\n      <th>negativereason_gold</th>\n      <th>retweet_count</th>\n      <th>text</th>\n      <th>tweet_coord</th>\n      <th>tweet_created</th>\n      <th>tweet_location</th>\n      <th>user_timezone</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>570306133677760513</td>\n      <td>neutral</td>\n      <td>1.0000</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>Virgin America</td>\n      <td>NaN</td>\n      <td>cairdin</td>\n      <td>NaN</td>\n      <td>0</td>\n      <td>@VirginAmerica What @dhepburn said.</td>\n      <td>NaN</td>\n      <td>2015-02-24 11:35:52 -0800</td>\n      <td>NaN</td>\n      <td>Eastern Time (US &amp; Canada)</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>570301130888122368</td>\n      <td>positive</td>\n      <td>0.3486</td>\n      <td>NaN</td>\n      <td>0.0000</td>\n      <td>Virgin America</td>\n      <td>NaN</td>\n      <td>jnardino</td>\n      <td>NaN</td>\n      <td>0</td>\n      <td>@VirginAmerica plus you've added commercials to the experience... tacky.</td>\n      <td>NaN</td>\n      <td>2015-02-24 11:15:59 -0800</td>\n      <td>NaN</td>\n      <td>Pacific Time (US &amp; Canada)</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>570301083672813571</td>\n      <td>neutral</td>\n      <td>0.6837</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>Virgin America</td>\n      <td>NaN</td>\n      <td>yvonnalynn</td>\n      <td>NaN</td>\n      <td>0</td>\n      <td>@VirginAmerica I didn't today... Must mean I need to take another trip!</td>\n      <td>NaN</td>\n      <td>2015-02-24 11:15:48 -0800</td>\n      <td>Lets Play</td>\n      <td>Central Time (US &amp; Canada)</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>570301031407624196</td>\n      <td>negative</td>\n      <td>1.0000</td>\n      <td>Bad Flight</td>\n      <td>0.7033</td>\n      <td>Virgin America</td>\n      <td>NaN</td>\n      <td>jnardino</td>\n      <td>NaN</td>\n      <td>0</td>\n      <td>@VirginAmerica it's really aggressive to blast obnoxious \"entertainment\" in your guests' faces &amp;amp; they have little recourse</td>\n      <td>NaN</td>\n      <td>2015-02-24 11:15:36 -0800</td>\n      <td>NaN</td>\n      <td>Pacific Time (US &amp; Canada)</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>570300817074462722</td>\n      <td>negative</td>\n      <td>1.0000</td>\n      <td>Can't Tell</td>\n      <td>1.0000</td>\n      <td>Virgin America</td>\n      <td>NaN</td>\n      <td>jnardino</td>\n      <td>NaN</td>\n      <td>0</td>\n      <td>@VirginAmerica and it's a really big bad thing about it</td>\n      <td>NaN</td>\n      <td>2015-02-24 11:14:45 -0800</td>\n      <td>NaN</td>\n      <td>Pacific Time (US &amp; Canada)</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"We see that our DataFrame contains numerous columns that provide information about the corresponding tweet. In our context of training a language model the most important columns are 'airline_sentiment' and 'text'. Let's check if there're no missing values in them.\nNext we visualise the counts of tweets by sentiment category."},{"metadata":{"trusted":true},"cell_type":"code","source":"df[['airline_sentiment', 'text']].isnull().sum()","execution_count":274,"outputs":[{"output_type":"execute_result","execution_count":274,"data":{"text/plain":"airline_sentiment    0\ntext                 0\ndtype: int64"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"There're no missing values so we can proceed. Let's visualise counts of tweets by sentiment category."},{"metadata":{"trusted":true},"cell_type":"code","source":"df['airline_sentiment'].value_counts().plot(kind='bar')","execution_count":275,"outputs":[{"output_type":"execute_result","execution_count":275,"data":{"text/plain":"<matplotlib.axes._subplots.AxesSubplot at 0x7f5b4cab6160>"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"<Figure size 432x288 with 1 Axes>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAYAAAAEeCAYAAACDq8KMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEpxJREFUeJzt3X+wZ3Vdx/HnS1ZUNPkRq9mC7pabRprBbIA50w8pfmS2VIqY1OrQ7ExR2Y/JsJowk9ImNW3SZAQHjUKiDCrTIUQbNdHlh/Irhg1FWFFWlx+bprHw7o/zWbzu3PV+77J7zr1+no+Z79xzPufz/X7fZ+7c+/qez/mc801VIUnqzyOmLkCSNA0DQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOrVi6gK+mUMPPbRWr149dRmStKxcddVVX6yqlQv1W9IBsHr1ajZt2jR1GZK0rCS5bZZ+DgFJUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOrWkLwQb2+oz/3XqEvapz7z2eVOXIGkJ8QhAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1aqYASPKbSW5Icn2Sv0vy6CRrklyZZHOSdyfZv/V9VFvf3LavnvM6r2ztNyc5Yd/skiRpFgsGQJJVwK8D66rqGcB+wKnA64A3VtVTgbuB09tTTgfubu1vbP1IckR73vcBJwJvSbLf3t0dSdKsZh0CWgE8JskK4ADgTuC5wMVt+/nAyW15fVunbT8uSVr7hVX1tar6NLAZOPrh74IkaU8sGABVtQX4c+CzDP/47wWuAu6pqh2t2x3Aqra8Cri9PXdH6//tc9vneY4kaWSzDAEdzPDpfQ3wncBjGYZw9okkG5NsSrJp69at++ptJKl7swwB/Tjw6araWlX3A/8IPAc4qA0JARwGbGnLW4DDAdr2A4EvzW2f5zkPqapzqmpdVa1buXLlHuySJGkWswTAZ4FjkxzQxvKPA24ErgBe0PpsAC5py5e2ddr2D1RVtfZT2yyhNcBa4ON7ZzckSYu1YqEOVXVlkouBq4EdwDXAOcC/AhcmeU1rO7c95VzgXUk2A9sYZv5QVTckuYghPHYAZ1TVA3t5fyRJM1owAACq6izgrF2ab2WeWTxV9VXghbt5nbOBsxdZoyRpH/BKYEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1aqYASHJQkouT/FeSm5I8O8khSS5Lckv7eXDrmyRvTrI5yaeSHDXndTa0/rck2bCvdkqStLBZjwDeBLyvqp4OPAu4CTgTuLyq1gKXt3WAk4C17bEReCtAkkOAs4BjgKOBs3aGhiRpfAsGQJIDgR8GzgWoqv+rqnuA9cD5rdv5wMlteT3wzhp8DDgoyZOAE4DLqmpbVd0NXAacuFf3RpI0s1mOANYAW4F3JLkmyduTPBZ4YlXd2fp8HnhiW14F3D7n+Xe0tt21S5ImMEsArACOAt5aVUcCX+brwz0AVFUBtTcKSrIxyaYkm7Zu3bo3XlKSNI9ZAuAO4I6qurKtX8wQCF9oQzu0n3e17VuAw+c8/7DWtrv2b1BV51TVuqpat3LlysXsiyRpERYMgKr6PHB7kqe1puOAG4FLgZ0zeTYAl7TlS4FfbLOBjgXubUNF7weOT3JwO/l7fGuTJE1gxYz9fg24IMn+wK3AyxjC46IkpwO3Aae0vu8FfhLYDHyl9aWqtiX5Y+ATrd+rq2rbXtkLSdKizRQAVXUtsG6eTcfN07eAM3bzOucB5y2mQEnSvuGVwJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnq1MwBkGS/JNck+Ze2vibJlUk2J3l3kv1b+6Pa+ua2ffWc13hla785yQl7e2ckSbNbzBHAy4Gb5qy/DnhjVT0VuBs4vbWfDtzd2t/Y+pHkCOBU4PuAE4G3JNnv4ZUvSdpTMwVAksOA5wFvb+sBngtc3LqcD5zclte3ddr241r/9cCFVfW1qvo0sBk4em/shCRp8WY9AvgL4BXAg23924F7qmpHW78DWNWWVwG3A7Tt97b+D7XP8xxJ0sgWDIAkPwXcVVVXjVAPSTYm2ZRk09atW8d4S0nq0ixHAM8BfjrJZ4ALGYZ+3gQclGRF63MYsKUtbwEOB2jbDwS+NLd9nuc8pKrOqap1VbVu5cqVi94hSdJsFgyAqnplVR1WVasZTuJ+oKpeAlwBvKB12wBc0pYvbeu07R+oqmrtp7ZZQmuAtcDH99qeSJIWZcXCXXbrd4ELk7wGuAY4t7WfC7wryWZgG0NoUFU3JLkIuBHYAZxRVQ88jPeXJD0MiwqAqvog8MG2fCvzzOKpqq8CL9zN888Gzl5skZKkvc8rgSWpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASerUw/lCGGlpedWBU1ewb73q3qkr0LcYjwAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ3yC2EkTe6Z5z9z6hL2qes2XDd1CfPyCECSOmUASFKnDABJ6tSCAZDk8CRXJLkxyQ1JXt7aD0lyWZJb2s+DW3uSvDnJ5iSfSnLUnNfa0PrfkmTDvtstSdJCZjkC2AH8dlUdARwLnJHkCOBM4PKqWgtc3tYBTgLWtsdG4K0wBAZwFnAMcDRw1s7QkCSNb8EAqKo7q+rqtrwduAlYBawHzm/dzgdObsvrgXfW4GPAQUmeBJwAXFZV26rqbuAy4MS9ujeSpJkt6hxAktXAkcCVwBOr6s626fPAE9vyKuD2OU+7o7Xtrl2SNIGZAyDJ44B/AH6jqu6bu62qCqi9UVCSjUk2Jdm0devWvfGSkqR5zBQASR7J8M//gqr6x9b8hTa0Q/t5V2vfAhw+5+mHtbbdtX+DqjqnqtZV1bqVK1cuZl8kSYswyyygAOcCN1XVG+ZsuhTYOZNnA3DJnPZfbLOBjgXubUNF7weOT3JwO/l7fGuTJE1glltBPAf4BeC6JNe2tt8DXgtclOR04DbglLbtvcBPApuBrwAvA6iqbUn+GPhE6/fqqtq2V/ZCkrRoCwZAVX0YyG42HzdP/wLO2M1rnQect5gCJUn7hlcCS1KnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTowdAkhOT3Jxkc5Izx35/SdJg1ABIsh/wV8BJwBHAi5McMWYNkqTB2EcARwObq+rWqvo/4EJg/cg1SJIYPwBWAbfPWb+jtUmSRrZi6gJ2lWQjsLGt/k+Sm6esZx87FPjiWG+W1431Tt0Y9ffHH2W0t+rAuH97Lx39d/eUWTqNHQBbgMPnrB/W2h5SVecA54xZ1FSSbKqqdVPXoT3j72/58nc3GHsI6BPA2iRrkuwPnApcOnINkiRGPgKoqh1JfhV4P7AfcF5V3TBmDZKkwejnAKrqvcB7x37fJaqLoa5vYf7+li9/d0CqauoaJEkT8FYQktQpA0CSOmUASFKnDIAJJHlMkqdNXYfUkwxOS/KHbf3JSY6euq4pGQAjS/J84FrgfW39B5J4LcQSl2R7kvvmeWxPct/U9WkmbwGeDby4rW9nuDllt5bcrSA68CqGm+J9EKCqrk2yZsqCtLCq+rapa9DDdkxVHZXkGoCqurtdkNotA2B891fVvck33BvEubjLTJInAI/euV5Vn52wHM3m/nZL+gJIshJ4cNqSpuUQ0PhuSPLzwH5J1ib5S+CjUxel2ST56SS3AJ8GPgR8Bvi3SYvSrN4MvAd4QpKzgQ8DfzJtSdPyQrCRJTkA+H3g+Nb0fuA1VfXV6arSrJJ8Engu8O9VdWSSHwNOq6rTJy5NM0jydOA4IMDlVXXTxCVNygAYWZKjqurqqevQntl5F8kWBEdW1YNJPllVz5q6Nn1zSd4MXFhVHnE3ngMY3+uTfAdwMfDuqrp+6oK0KPckeRzwH8AFSe4CvjxxTZrNVcAftCnY72EIg00T1zQpjwAm0ALgFOBFwOMZguA101alWSR5LPC/DOfPXgIcCFxQVV+atDDNLMkhwM8x3I7+yVW1duKSJmMATCjJM4FXAC+qqq6noy0HbQbJv1fVj01di/Zcu/jrRQzfR35TVT1/4pIm4yygkSX53iSvSnIdsHMG0GETl6UZVNUDwINJDpy6Fi1ekj9rM7heDVwPrOv5nz94DmAK5wHvBk6oqs9NXYwW7X+A65Jcxpyx/6r69elK0oz+G3h2VY33Pc5LnENA0iIk2TBPc1XVO0cvRjNJ8vSq+q8kR823vedZeR4BjCTJRVV1Shv6mZu6YfgH8v0TlabFOaiq3jS3IcnLpypGM/ktYCPw+nm2FcN1HV3yCGAkSZ5UVXcmecp826vqtrFr0uIlubqqjtql7ZqqOnKqmjSbJI/e9YLL+dp64kngkVTVnW3xV6rqtrkP4FemrE0LS/LiJP8MrEly6ZzHFcC2qevTTOa7AKzri8IcAhrfTwC/u0vbSfO0aWn5KHAncCjfOJSwHfjUJBVpJu26m1XAY5IcyTDsCsM1OAdMVtgSYACMJMkvM3zS/64kc/9hfBvwkWmq0qzakdptDPeT1/JyAvBShunWb5jTvh34vSkKWio8BzCSNnf8YOBPgTPnbNpeVQ4hLBNJtvP1k/j7A48EvlxVj5+uKs0iyc9V1T9MXcdSYgBMxPvJL38ZvtRhPXBsVZ25UH9NI8lpVfU3SX6beb57o6reMM/TuuBJ4JEleb73k//WUIN/Yhhi0NL12PbzcQxDrrs+uuURwMi8n/zyluRn56w+AlgH/EhVeW5Ay45HAOO7v9058hFJHlFVVzD8E9Hy8Pw5jxMYTiSun7QizaTdC+jxSR6Z5PIkW5OcNnVdU3IW0Pi8n/wyVlUvm7oG7bHjq+oVSX6GYej1Zxn+Dv9m0qom5BHA+NYz3E/+N4H3Mdygqus7Ei4nSb6nfXq8vq1/f5I/mLouzWTnB97nAX9fVfdOWcxS4DkAaRGSfAj4HeBtO2//kOT6qnrGtJVpIUleC5zM8AHsaOAg4F+q6phJC5uQRwAjS7I9yX27PG5P8p4k3zV1fVrQAVX18V3adkxSiRalTdX9IYbvAbifYei16/M3ngMY318AdwB/y3BJ+qnAdwNXM3xXwI9OVplm8cUk302bT57kBQy3iNASl+SRwGnADw+XcPAh4K8nLWpiDgGNLMknq+pZu7RdW1U/MN82LS3tKO0chk+SdzNcz/ES7+a69CV5O8OV2+e3pl8AHqiqX5quqml5BDC+ryQ5Bbi4rb8A2Hk7WtN46dsCvAO4AjgEuA/YwPA1g1rafnCXD1gfaNfldMtzAON7CcMnj7uAL7Tl05I8BvjVKQvTTC5hmLV1P/A5hq+IdBrv8vBAG74DHjqae2DCeibnEJC0CM74Wb6SHMdw9HZra1oNvKxdjNkljwBG5jzyZe+jSZ45dRHaIx8B3gY8yPAlPm8D/nPSiibmEcDInEe+vCW5EXgqw8nfr+F3Oi8bSS5iOGdzQWv6eYbveH7hdFVNy5PA4zugqj7epqHt5Dzy5eOkqQvQHntGVR0xZ/2KFujdMgDG5zzyZczpnsva1UmOraqPASQ5Btg0cU2TcghoZM4jl6aR5CbgacDOL196MnAzwxF4l8N4BsDIkjyKYe7/ar4+j7yqynnk0j6U5CnfbHuPH8IcAhrfJcA9DLd++NzEtUjd6PEf/EI8AhiZM34kLRVeBzA+55FLWhI8AhiZ88glLRUGwMh2dyLK8UlJYzMAJKlTngOQpE4ZAJLUKQNAkjplAEhSpwwASerU/wO0LhiD019FkgAAAABJRU5ErkJggg==\n"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"We see that neutral and positive tweets are underepresented compared to negative ones. We will not balance this dataset and evaluate the perfomance of the model on the test set with the same proportion of sentiments.\n\nNext let's investugate the relationship between tweet length and its sentiment."},{"metadata":{"trusted":true},"cell_type":"code","source":"df['tweet_length'] = df['text'].apply(len)\ndf.groupby(['tweet_length', 'airline_sentiment']).size().unstack().plot(kind='line')","execution_count":276,"outputs":[{"output_type":"execute_result","execution_count":276,"data":{"text/plain":"<matplotlib.axes._subplots.AxesSubplot at 0x7f5b4444b748>"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"<Figure size 432x288 with 1 Axes>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAXoAAAELCAYAAADX3k30AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xd4lFX2wPHvnfTeEyABktACoYQqiEpTwAooKCwiimVd6+paWNvPuuqKgtgVFAsKLopiV5ogTYN0CJBAIAkJaaRnkszM/f0xkyGBQAZIzAycz/PwZOadd965k5AzN+c977lKa40QQoizl6GlByCEEKJ5SaAXQoiznAR6IYQ4y0mgF0KIs5wEeiGEOMtJoBdCiLOcBHohhDjLSaAXQoiznAR6IYQ4y7m39AAAwsPDdWxsbEsPQwghXMrGjRvztdYRje3nFIE+NjaW5OTklh6GEEK4FKXUAUf2k9SNEEKc5STQCyHEWU4CvRBCnOWcIkffkJqaGjIzMzEajS09lHOKt7c3MTExeHh4tPRQhBBNxGkDfWZmJgEBAcTGxqKUaunhnBO01hQUFJCZmUlcXFxLD0cI0UScNnVjNBoJCwuTIP8XUkoRFhYmf0UJcZZx2kAPSJBvAfI9F+Ls49SBXgjhvPLLqtiSUdTSwxAOkEAvhDgt763ax7R5f7T0MIQDzopAf9lll1FU1PDMIjY2lvz8fADOP//8v3JYDvvPf/5T735zj7OoqIg333yzWV9DnP1KjDVU1phbehjCAWdFoP/+++8JDg6ut01rjcViqbdt7dq1f+WwHHZsoG/ucUqgF02hstqMReuWHoZwgMsF+rFjx9K3b18SExN59913gaOz9vT0dLp06cINN9xA9+7dycjIqPdcf39/AFauXMnQoUMZP348CQkJTJ48GW37D7tx40aGDBlC3759GTVqFNnZ2Sccy+zZs+nWrRs9e/Zk4sSJAJSXlzNt2jQGDBhA7969+frrrwGYN28eV199NaNHj6ZTp0489NBDAEyfPp3KykqSkpKYPHnyceMcMmQIY8aMIT4+nunTpzN//nwGDBhAjx49SEtLAyAvL49rrrmG/v37079/f9asWQPAk08+ybRp0xg6dCjx8fHMnj3b/pppaWkkJSXx4IMPnuFPRJyrjDUWjplLCWeltW7xf3379tXH2rlz53HbtNa6oKBAa611RUWFTkxM1Pn5+bp9+/Y6Ly9P79+/Xyul9Lp16+z71z6mtdZ+fn5aa61XrFihAwMDdUZGhjabzXrgwIF69erVurq6Wg8aNEjn5uZqrbVesGCBvummmxoch9Zat27dWhuNRq211keOHNFaa/3vf/9bf/zxx/ZtnTp10mVlZfqDDz7QcXFxuqioSFdWVup27drpgwcP1htXrbrjDAoK0ocOHdJGo1G3adNGP/HEE1prrWfNmqXvvfderbXWkyZN0qtXr9Zaa33gwAGdkJCgtdb6//7v//SgQYO00WjUeXl5OjQ0VFdXV+v9+/frxMTEE76vE33vhajrhrkbdId/f9fSwzinAcnagRjrtBdMncjs2bNZvHgxABkZGezdu7fe4+3bt2fgwIGNHmfAgAHExMQAkJSURHp6OsHBwWzfvp1LLrkEALPZTOvWrU94jJ49ezJ58mTGjh3L2LFjAfj5559ZsmQJM2bMAKzXAxw8eBCAESNGEBQUBEC3bt04cOAAbdu2Pek4+/fvbx9Dhw4dGDlyJAA9evRgxYoVACxdupSdO3fan1NSUkJZWRkAl19+OV5eXnh5eREZGcnhw4cb/d4I4YjKGjNmSd24BJcK9CtXrmTp0qWsW7cOX19fhg4detzFPX5+fg4dy8vLy37bzc0Nk8mE1prExETWrVvn0DG+++47Vq1axTfffMNzzz3Htm3b0FrzxRdf0KVLl3r7btiwocHXPJVxGgwG+32DwWB/vsViYf369Xh7ezv0PoVoClU1ZrS2ZgXk+gvn5lI5+uLiYkJCQvD19SUlJYX169c36fG7dOlCXl6ePdDX1NSwY8eOBve1WCxkZGQwbNgwXnzxRYqLiykrK2PUqFG89tpr9pz/pk2bGn1dDw8PampqTnvcI0eO5LXXXrPf37x580n3DwgIoLS09LRfTwjAXnFjkUm903OpQD969GhMJhNdu3Zl+vTpDqVoToWnpyeLFi3i4YcfplevXiQlJZ2wAsZsNnP99dfTo0cPevfuzT333ENwcDCPP/44NTU19OzZk8TERB5//PFGX/e2226zp4FOx+zZs0lOTqZnz55069aNt99++6T7h4WFMXjwYLp37y4nY8VpOxroJdI7O6Wd4IfUr18/fewKU7t27aJr164tNKJzm3zvhSP6P7eUvNIqUp4ZjbeHW0sP55yklNqote7X2H4uNaMXQjgPY7V1Ru8Ec0XRCJc6GdtS7rzzTntteq17772Xm266qYVGJETLq03dSOWN85NA74A33nijpYcghFOpMVsw2c7CSo7e+TmUulFKBSulFimlUpRSu5RSg5RSoUqpX5RSe21fQ2z7KqXUbKVUqlJqq1KqT/O+BSHEX81Yp8eNRcpunJ6jOfpXgR+11glAL2AXMB1YprXuBCyz3Qe4FOhk+3cb8FaTjlgI0eLqNjOTOO/8Gg30Sqkg4CJgLoDWulprXQSMAT607fYhMNZ2ewzwke0K3fVAsFLqxJeXCiFcTlXN0SY3Zon0Ts+RGX0ckAd8oJTapJSao5TyA6K01rUdv3KAKNvtaKBuN7FM27Z6lFK3KaWSlVLJeXl5p/8OzmLHdpk8dOgQ48ePb8ERCWFVd0bvDCXa4uQcCfTuQB/gLa11b6Cco2kaAGzNdU7pp621fldr3U9r3S8iIuJUnnrOODbQt2nThkWLFrXgiISwqqyW1I0rcSTQZwKZWusNtvuLsAb+w7UpGdvXXNvjWUDdTl0xtm1nnfT0dLp27cqtt95KYmIiI0eOpLKykrS0NEaPHk3fvn258MILSUlJASAtLY2BAwfSo0cPHnvsMXs74rKyMkaMGEGfPn3o0aOHvbXxse2E09PT6d69OwADBw6s155h6NChJCcnn7BNshBNqe7JWCmvdH6NlldqrXOUUhlKqS5a693ACGCn7d9U4AXb19qIsgS4Sym1ADgPKK6T4mkWT32zg52HSpr0mN3aBPJ/VyY2ut/evXv57LPPeO+997j22mv54osv+OCDD3j77bfp1KkTGzZs4I477mD58uXce++93HvvvUyaNKlemwJvb28WL15MYGAg+fn5DBw4kKuuuooXXniB7du323vXpKen259z3XXX8fnnn/PUU0+RnZ1NdnY2/fr145FHHmH48OG8//77FBUVMWDAAC6++GKHm70J4YhKqbpxKY7W0d8NzFdKeQL7gJuw/jXwuVLqZuAAcK1t3++By4BUoMK271krLi6OpKQkAPr27Ut6ejpr165lwoQJ9n2qqqoAWLduHV999RUAf/vb33jggQcAa47zkUceYdWqVRgMBrKyshptJ3zttdcycuRInnrqKT7//HN77v5EbZKlpYFoSvXKK2VG7/QcCvRa681AQ/0URjSwrwbuPMNxnRJHZt7N5dg2wIcPHyY4OLjRDpJ1zZ8/n7y8PDZu3IiHhwexsbHHtV8+VnR0NGFhYWzdupWFCxfa/0I4UZtkIZqSsU7VjUzonZ/0umligYGBxMXF8b///Q+wBt4tW7YA1rz6F198AcCCBQvszykuLiYyMhIPDw9WrFjBgQMHgMbbCV933XX897//pbi4mJ49ewKcVptkIU5V3dSNlFc6Pwn0zWD+/PnMnTuXXr16kZiYaD8hOmvWLF555RV69uxJamqqfbWpyZMnk5ycTI8ePfjoo49ISEgAGm8nPH78eBYsWMC1115r33Y6bZKFOFV1q26kvNL5Sa+bMxAbG8v27dvt92tz7gA//vjjcftHR0ezfv16lFIsWLCA3bt3AxAeHn7CVa0+/fTTevfrvl5UVNRxK0b5+PjwzjvvnPqbEeIUGE1SdeNKJND/hTZu3Mhdd92F1prg4GDef//9lh6SEKfFWLeO3nKSHYVTkED/F7rwwgvt+XohXFmlVN24FMnRCyFOWf2qGwn0zk4CvRDilEnVjWuRQC+EOGXSpti1SKAXQpyyKsnRuxQJ9E4gPT39uDJKR9U2RhPiryS9blyLBHoncLJAf2ydvBDOoLLajIebAqSO3hVIoD8Dp9qm+MYbb6zXT752Nj59+nRWr15NUlISM2fOZN68eVx11VUMHz6cESNGnLCNsRAtxVhjwdfTWp0tcd75nR119D9Mh5xtTXvMVj3g0hca3e1U2hSfyAsvvMCMGTP49ttvAZg3bx5//vknW7duJTQ0FJPJ1GAbY6VUk71dIU6FscaMv5c7xZU1UnXjAs6OQN+CTqVN8am45JJLCA0NBU7cxrhVq1ZN8yaEOEWVtkAPcjLWFZwdgd6BmXdzOZU2xe7u7lhs14tbLBaqq6tPeNy6C4WcThtjIZqTscZMZKA3IIHeFUiOvomdrE1xbGwsGzduBGDJkiXU1NQAjbcjPlEbYyFaSmWNGT9PN0B63bgCCfTN4ERtim+99VZ+/fVXevXqxbp16+yz9p49e+Lm5kavXr2YOXPmccc7URtjIVqC1rreyVipunF+yhl6Sffr108nJyfX27Zr1y5Z/q6FyPdenIyxxkzC4z8yJqkNX28+xNvX92F099YtPaxzklJqo9a6odX/6pEZvRDilNQuOuJnOxlrltSN05NAL4Q4JbVXxfp62HL0TpAVECcngV4IcUqMtYFeyitdhkOBXimVrpTappTarJRKtm0LVUr9opTaa/saYtuulFKzlVKpSqmtSqk+zfkGhBB/rdoZvb+XzOhdxanM6IdprZPqJP6nA8u01p2AZbb7AJcCnWz/bgPeaqrBCiFann1G7yk5eldxJqmbMcCHttsfAmPrbP9IW60HgpVSckpeiLNE7epSfjKjdxmOBnoN/KyU2qiUus22LUprnW27nQNE2W5HAxl1nptp2yYa8Pbbb/PRRx8B1h43hw4dsj92yy23sHPnzpYamhANqq26qZ3RS5ti5+doC4QLtNZZSqlI4BelVErdB7XWWil1Sj9t2wfGbQDt2rU7laeeVW6//Xb77Xnz5tG9e3fatGkDwJw5c1pqWEKcUHm1tXV2gHftydiWHI1whEMzeq11lu1rLrAYGAAcrk3J2L7m2nbPAtrWeXqMbduxx3xXa91Pa90vIiLi9N9BC0pPTychIYHJkyfTtWtXxo8fT0VFBcuWLaN379706NGDadOm2ZuaTZ8+nW7dutGzZ08eeOABAJ588klmzJjBokWLSE5OZvLkySQlJVFZWcnQoUNJTk7m7bff5sEHH7S/7rx587jrrrsA+OSTTxgwYABJSUn8/e9/x2w2Hz9QIZpQWZU10Ad6ewByZawraHRGr5TyAwxa61Lb7ZHA08ASYCrwgu1rbZP0JcBdSqkFwHlAcZ0UT7N48fcXSSlMaXzHU5AQmsDDAx5udL/du3czd+5cBg8ezLRp03jllVd45513WLZsGZ07d+aGG27grbfeYsqUKSxevJiUlBSUUhQVFdU7zvjx43n99deZMWMG/frVv9DtmmuuYdCgQbz00ksALFy4kEcffZRdu3axcOFC1qxZg4eHB3fccQfz58/nhhtuaLpvhBDHKD8m0DvD1fXi5ByZ0UcBvymltgC/A99prX/EGuAvUUrtBS623Qf4HtgHpALvAXc0+aidSNu2bRk8eDAA119/PcuWLSMuLo7OnTsDMHXqVFatWkVQUBDe3t7cfPPNfPnll/j6+jr8GhEREcTHx7N+/XoKCgpISUlh8ODBLFu2jI0bN9K/f3+SkpJYtmwZ+/bta5b3KUStsipbeaV3bdWNBHpn1+iMXmu9D+jVwPYCYEQD2zVwZ5OMzkGOzLyby7GLfwQHB1NQUHDcfu7u7vz+++8sW7aMRYsW8frrr590MZJjTZw4kc8//5yEhATGjRuHUgqtNVOnTuX5558/4/chhKPKq0z4e7njZrD+35c47/zkytgzdPDgQdatWwfAp59+Sr9+/UhPTyc1NRWAjz/+mCFDhlBWVkZxcTGXXXYZM2fOtLcurutk7YrHjRvH119/zWeffcbEiRMBGDFiBIsWLSI313p6pLCwUFoYi2ZXZjTh5+WGLc5L1Y0LODsWHmlBXbp04Y033mDatGl069aN2bNnM3DgQCZMmIDJZKJ///7cfvvtFBYWMmbMGIxGI1prXnnlleOOdeONN3L77bfj4+Nj//CoFRISQteuXdm5cycDBgwAoFu3bjz77LOMHDkSi8WCh4cHb7zxBu3bt/9L3rs4N5VVm/CrN6OXQO/spE3xGUhPT+eKK65g+/btLT2UJuUK33vRcm784HcKy6v5/O+DSHj8Rx4a3YU7hnZs6WGdk6RNsRCiWZRXmfDzdMdgOz/lBHNF0QgJ9GcgNjb2rJvNC9GYsioz/t7u9hy9VN04P6cO9M6QVjrXyPdcNKasquaYqhv5P+PsnDbQe3t7U1BQIIHnL6S1pqCgAG9v75YeinBi5VVm/Lzc7KXFUnXj/Jy26iYmJobMzEzy8vJaeijnFG9vb2JiYlp6GMKJlVWZ7MsIuhmU1NG7AKcN9B4eHsTFxbX0MIQQddSYLVSbLPjbOlcalPS6cQVOm7oRQjif2j43tTN6g1KSo3cBEuiFEA4rNVoDfW2fG4NSkqN3ARLohRAOq+1F7y85epcigV4I4bBjUzdKSR29K5BAL4RwmL1FsW29WDeDkhJoFyCBXgjhsIZOxkrVjfOTQC+EcFiZsX6O3lp105IjEo6QQC+EcFjterFHA720zXAFEuiFEA47NnXjZlByMtYFSKAXQjisrNqEp7sBDzdr6JDUjWuQQC+EcFjterG1DAZpauYKJNALIRxWu15sLWmB4BocDvRKKTel1Cal1Le2+3FKqQ1KqVSl1EKllKdtu5ftfqrt8djmGboQ4q9WVmXG38vDft9NKcwS553eqczo7wV21bn/IjBTa90ROALcbNt+M3DEtn2mbT8hxFnAmro5OqNXShYecQUOBXqlVAxwOTDHdl8Bw4FFtl0+BMbabo+x3cf2+AhVu0KBEMKllVcf7UUPtl43kqN3eo7O6GcBDwEW2/0woEhrbbLdzwSibbejgQwA2+PFtv2FEC6u7qIjIDl6V9FooFdKXQHkaq03NuULK6VuU0olK6WSZRUpIVxDmdFkX3QEbC0QLCd5gnAKjszoBwNXKaXSgQVYUzavAsFKqdqfeAyQZbudBbQFsD0eBBQce1Ct9bta635a634RERFn9CaEEH+N8iqTvRc9WMsr5cpY59dooNda/1trHaO1jgUmAsu11pOBFcB4225Tga9tt5fY7mN7fLmW/wlCuDyLRVNeba6fo5emZi7hTOroHwbuV0qlYs3Bz7VtnwuE2bbfD0w/syEKIZzB0UVH6lbdyJWxruCUFgfXWq8EVtpu7wMGNLCPEZjQBGMTQjiRvNIqACICvOzbpOrGNciVsUIIh+SUGAGICvC2bzNIHb1LkEAvhHDI4dpAH1Q30Ev3SlcggV4I4ZCcYmvqplVg/UAvE3rnJ4FeCOGQwyVGArzcj7syVqpunJ8EeiGEQ3KKjfXSNiC9blyFBHohhEMOlxrrpW1Aqm5chQR6IYRDDhcbiQz0qrdNVphyDRLohRCNslg0uaVVx83operGNUigF0I0Kr+8CpNF0yro2EAvOXpXIIFeCNGow7bSyqiGcvQS6J2eBHohRKNqr4ptKHUjmRvnJ4FeCNGo2qtij0vdSNWNS5BAL4Ro1OESIwYF4f7HVt1Ijt4VSKAXQjQqp9hIRIAXbob6yz8bpB+9S5BAL4RoVE7J8RdLgS1HL0sJOj0J9EKIRuWWVBHZYKCX1I0rkEAvhGhUQXnVcfl5kPJKVyGBXghxUmaLprC8mgh/z+MeU0phltSN05NAL4Q4qSMV1Vg0hDU4owctM3qnJ4FeCHFSBWXVwPGllSBVN65CAr0Q4qQKyqztD8IaSN1Yq24k0Ds7CfRCiJPKswX68BMFeonzTq/RQK+U8lZK/a6U2qKU2qGUesq2PU4ptUEplaqUWqiU8rRt97LdT7U9Htu8b0EI0ZxOlrpxM0h5pStwZEZfBQzXWvcCkoDRSqmBwIvATK11R+AIcLNt/5uBI7btM237CSFcVH5ZFe4GRaC3x3GPST9619BooNdWZba7HrZ/GhgOLLJt/xAYa7s9xnYf2+MjlFL1r5sWQriMgrJqwvw9MRiO/zU2GBQyoXd+DuXolVJuSqnNQC7wC5AGFGmtTbZdMoFo2+1oIAPA9ngxENbAMW9TSiUrpZLz8vLO7F0IIZpNQXkVYX7Hp23AemWsVN04P4cCvdbarLVOAmKAAUDCmb6w1vpdrXU/rXW/iIiIMz2cEKKZ5Nlm9A1xU3JlrCs4paobrXURsAIYBAQrpdxtD8UAWbbbWUBbANvjQUBBk4xWCPGXKyirIqKBE7FgvTJWa7loytk5UnUToZQKtt32AS4BdmEN+ONtu00FvrbdXmK7j+3x5Vr+FwjhsgpONqO35e3lfKxzc298F1oDHyql3LB+MHyutf5WKbUTWKCUehbYBMy17T8X+FgplQoUAhObYdxCiL9AeZWJyhpzg6WVYM3Rg7UfzrG96oXzaDTQa623Ar0b2L4Pa77+2O1GYEKTjE4I0aJqa+gb6nMD2CtxJE/v3OTKWCHECeWXn7j9AVjr6EECvbOTQC+EOKH8UmugP9HJWDclOXpXIIFeCHFCBeW1qZuGZ/SqTo5eOC8J9EKIE6qd0Yf6nbzqRgrrnJsEeiHECWWXGAn29cDL3a3Bx2tz9DKjd24S6IUQJ7Q7p5TOUQEnfNwgdfQuQQK9EKJBWmt255SS0OokgV4d3Vc4Lwn0QogGZR6ppKzKREKrwBPuU1t1I43NnJsEeiFEg3bnlALQ5aQzeknduAIJ9EKIBqXklACNBPraHL1EeqcmgV4I0aCUnFLahvrg73XiTim1OXq5Mta5SaAXQjQoJaeULlEnzs/D0Tp6Ka90bhLohRDHMdaY2Z9fTtfWJ07bgLUfPUiO3tlJoBdCHCc1twyzRZ80Pw91e91IpHdmEuiFEMeprbg5WWklSI7eVUigF0IcJyWnBE93A7FhvifdzyA5epcggV4IcZyUnFI6R/nj7nbyEFFbRy8TeucmgV4IcRxHKm4Aaj8HZEbv3CTQCyHqKSyvJq+06qQ9bmopORnrEiTQCyHqqb0iNqGR0kqQqhtXIYFeCFFPSnbjPW5qSa8b19BooFdKtVVKrVBK7VRK7VBK3WvbHqqU+kUptdf2NcS2XSmlZiulUpVSW5VSfZr7TQghms7unFLC/DxPuE5sXQbJ0bsER2b0JuBfWutuwEDgTqVUN2A6sExr3QlYZrsPcCnQyfbvNuCtJh+1EKLZpOSU0KVVgD3/fjIGSd24hEYDvdY6W2v9p+12KbALiAbGAB/advsQGGu7PQb4SFutB4KVUq2bfORCiCZnsWj2HC5r9EKpWm727pXNOSpxpk4pR6+UigV6AxuAKK11tu2hHCDKdjsayKjztEzbNiGEk8s8UklljZnOUf4O7S9XxroGhwO9Usof+AL4p9a6pO5j2rqO2Cn9pJVStymlkpVSyXl5eafyVCFEM0nLLwOgY6SjgV5WmHIFDgV6pZQH1iA/X2v9pW3z4dqUjO1rrm17FtC2ztNjbNvq0Vq/q7Xup7XuFxERcbrjF0I0obRca6DvEHFqgV7WjHVujlTdKGAusEtr/Uqdh5YAU223pwJf19l+g636ZiBQXCfFI4RwYml55YT6eRLi5+nQ/kf70TfnqMSZOvHSMUcNBqYA25RSm23bHgFeAD5XSt0MHACutT32PXAZkApUADc16YiFEM0mLa+M+HA/h/dXkqN3CY0Geq31b8CJ6qxGNLC/Bu48w3EJIVrAvrwyRiRENb6jjZusGesS5MpYIQQARRXV5JdV0yHS8Rm9XBnrGhxJ3QghzmLGGjMV1dalA8HxE7EgVTeuQgK9EOcorTWf/Z7Bq8v2UFFt5tYL44FTDfRHjyWcl6RuhDhH/ZaazyOLt9Em2Icas4VXl+3F081ATIiPw8dwkxWmXIIEeiHOUfvyrKmad6f04/5LOmO2aGLDfRtdVaouydG7BkndCHGOOlRciYebIszPk2mD41i6M5euDvSgr8teXimR3qlJoBfiHJVTbKRVkDcGg8KAYuHfBzrUsbIue3ml5OidmqRuhDhHZRcbaR14NB9/qkEepOrGVUigF+IclV1cSetg7zM6huToXYMEeiHOQRaL5nBxFa2CzjTQHz2ecF4S6IU4BxVWVFNtttAmyPFSyoZIjt41SKAX4hxwqKiS77dl22fe2UVGgDOe0dfm9aWO3rlJ1Y0Q54A3V6byyfqDDIwP5ZVrk8gurgSg9RkG+toZvUzonZvM6IU4B6TllhPu78W2zGIe/mIrOSXWGX3rM0zd1OboperGucmMXohzQHpBOUM6RxAV6MU7q/YRE+Jrv1jqTBytupFA78xkRi/EWa6i2kR2sZH4CD8u7haF2aL5Zssh+8VSZ8Ie6CVH79Qk0AtxlkvPrwAgNsyPpJhgwv09Kasy1btY6nQdrbo540OJZiSBXoizXG2f+bhwPwwGxfCESIAzvlgK6uToJdI7NQn0Qpzl9ueXARAb7gvAxV2tSwWeaWklWMsrlZJ+9M5OAr0QZ7l9+eW0DvLG19Nae3Fhpwi6RAUwIDa0SY5vUEqqbpycBHohXFSVycywGSv5dMPBk+63P7+cuPCj68D6eLrx030XMaKr44uAn4ybUpKjd3KNBnql1PtKqVyl1PY620KVUr8opfbavobYtiul1GylVKpSaqtSqk9zDl6Ic9n+/HL255fz7Hc7ySisOOl+seGOL/h9qpSSqhtn58iMfh4w+pht04FlWutOwDLbfYBLgU62f7cBbzXNMIUQAKm5pXyxMdN225p7N9aYefSr7cflyffllZGeX05RRQ3xzRjo3QxK6uidXKOBXmu9Cig8ZvMY4EPb7Q+BsXW2f6St1gPBSqnWTTVYIc51ry9P5YHA5CzGAAAgAElEQVRFWyirMpGaW4ZS8PDoBFbtyWP6F9uoMVsoqzLx7y+3MvzlXxn28kqAeqmbpmZQCrOl2Q4vmsDpXhkbpbXOtt3OAWqTfdFARp39Mm3bshFCnJZluw7Tt30Iwb6eJB84gtawPauY1NwyYkJ8uO2ieMqqTLy2PJUVu3M5UlGNyaK55YI4lIIdh0ro2z6k2cZnUPWvjDVbNBv2FQBwfsfwRp+/IiWXPu1DCPLxaLYxnuvOuAWC1lorpU757zal1G1Y0zu0a9fuTIchxFkpt8TIzR8mc9PgWG4f0oHMI9ZmZFszi0jNLaNjhD9KKf41sgsdIvz5YXs2ceH+jEyMok+75gvudRnqpG4yCiu49p11ZBcb8fYwsOOp0XUan2kyj1TSNtS33vu7ad4fPHFFN6ZdEPeXjPdcdLpVN4drUzK2r7m27VlA2zr7xdi2HUdr/a7Wup/Wul9ERMRpDkOIs9uWzGIAft5xmD/SrRlUd4Nic0YR+/LL6Rjpb993bO9o3pnSj+mXJvxlQR5qq26sgf7H7TlkFxsZ1zsaY42FQ0WV9v1W7s7jopdW2M8twNHzDIXl1X/ZeM9FpxvolwBTbbenAl/X2X6DrfpmIFBcJ8UjhDhFWzOLAMgqqmTemnS8PQwMS4hkRUoe1SZLvUD/lyvOhP2rUXXKK7dlFdMq0JtJA6x/paflHQ3qyQcK0Rp2ZZfYt6XZrtotrqz568Z9DnKkvPIzYB3QRSmVqZS6GXgBuEQptRe42HYf4HtgH5AKvAfc0SyjFuIcsSWzmJgQH9wMiuQDR+gVE0yfdiFU1pgBWi7QH0mHOZfAgr/hZjhaXrn9UDHdo4OIj7Ce/N2XV25/yvYsa4A/UHB02z7bB4EE+ubVaI5eaz3pBA+NaGBfDdx5poMSwpVprdlxqITu0UEOP2fnoRIMBogP98fT3WA/ztbMIkZ1a0VmUQVrUgvoFxtCr5ijx+0YEdDk429UWR58eCWUHgLAy9OERWvKqkzszy9nTK9owvw8CfR2Z1/+0Rn9jkPWQL8//2jNf+0HgQT65iX96IVoYp/+fpBHF2/nf7cPor8DbQbS8sq44rXV9a4uTWwTyCvXJlFUUUPPtkF0jwmyBvr2oXS3Bfpwfy+CfFugUuWPOVCUAUmTYfN8glQ5ZgvsyCpGa+gRE4hSivgIf3sgzy0xkl9WBVh749eq/SCQQN+8JNAL0YSMNWZeXboXgLWpBfSPDeXdVWlkHqnk9iEdaBN8fGvgN1ek4elu4Jkx3TlUZKTUWMOc3/Zz38LNAPSKCaZjpD/uBsVFnSNwMyg6RvoTFej1l743ACwW9Ob55MQNJjx+CB6b5xOsytBasy3LeuK49i+Z+HA/1qZZyyxrZ/PxEX6k2/LyxhqzvYqoRAJ9s5JALwTWxTl+3Z3H7sOlDIgNdaj+uyEfrUsnt7SKIB8P/kgvxFhjZtbSvVRUm1nwRwaTz2vHHUM7EhFgDdIZhRV8tTmLGwa1Z0K/owVruaVVLNlyCE83A52jAvB0N9hPcAK88bc+eLm3QKuq/StZW53H7SjcN/+HiyLDCS4pw6w127OKiQr0IjLA2hUzPsKPLzdlUV5lYrvtQ+Cy7q15fUUqJcYacoqNaA2B3u4yo29m0tRMnHMeXbyNZ77dWW/brKV7+cf8P5m1dC/Pfb/L4WMt2pjJ6FmrWJdWwIqUXF5bnsqQzhGM6x3NxgNHWL03n4pqM8+N687YpDZ8tO4AQ19awcYD1lLJ2cv2YlBw20Xx9Y47/dIEvD0MdG0TaM/Z19WlVUCz9q85oU2fsMY/CE+DJ31DEljp60MAZVi0teKmR53zEvER1hPF+/PL2XGohPZhvvbZfnp+uf1EbFK7EIora6TVcTOSQC/OKcYaM4s2ZrLwjwyqTUev21+1J48BsaHceH4sew6X1nusrk0Hj/DPBZv48+ARAD5Zf4CUnFImvbeem+b9QZsgH54ek0j/2FAqa8zMXrYXX083rukTw3/H9+KX+y4iIsCLv3+8kee+28n/NmZy0+C44xbpbhPswztT+vHEFd2a75txqsryYNe3/BkcQc+IngxrPRiLUnipQmvgzi+nR3SwfXd75U1+OTuyi+neJsjeimF/fjlptvx9UkwQJoumotr817+nc4SkbsQ55Y/0QqpMFqpMFpIPFHJ+h3Dyy6pIySnlwVFdaBvqy7y16ezNLSWxTRCV1WaWp+TyW2o+O7NL2JJhrWs/UlHDSxN6sjmjiH8M7UCNyYK3hxt3De+It4cbPp5ugHWWOyoxCm8P6/34CH/mTO3PuDfW8N7q/YzrHc3DoxMaHOuQzk5wIWHebsj6E5Imwfo3KLfUsMtUxq1RfQnxbQWAh9sRtmUVE+TjwbX9Y+xPjQ3zQyl4e2UaGYWVTD6vPe3DrFfFpudXcLCwgqhAL1rbzlsUV9bg5yUhqTnId1WclbTWKHX8wter9uTh6WZAo1m5O4/zO4TbTxgO7hhOgLf1V2LHoRK83N24+s01lBhNBHq707V1IA+O6sKR8mrmrtnPJ+utfeCv6tWGrq0D671OZIA3ceF+7M8vt6/oVKtjpD9zb+zPmtR87h7e0d4iwCmtfB52LIbSbPh9Dlu6DMNStZc+UX2obXxiUNb8+4vX9Kz3l4m3hxsxIT7szC7hql5tmDKwPd4ebrQJ8mZndjF7D5cRH+5v73FTXFnT4MlqceYk0AuXYawxs3pvPsO6RODuduKso9aaf3zyJxrNW5P7YqgTSFfvzad/nLU9wMrduTxyWVfWpuYT4O1Oj+ggFODr6cbOQyWk5ZVRUW3mk5vPY1CHMHtAziisYM5v+3lzRSrRwT4ktGq4ln1AbCjpBeX2NVrrPRYXyoC4plnhqdlYLLB/FSg3WPYUAMnRibjt30dSRBIZpdb+hcH+VdzdoyOju7c67hDPju1BtcnCJd2OftjFhvvx047DGJT1XETtLF5OyDYfydELp1ZcWUNGYQX5ZVX87b313PpRMvPWpp/0Ob+l5vPjjhx+2nGYd1fvs28/XGIkJaeUCztFMKxLJHsOl5F5pII1afkMjLcGcoNB0bV1IDsOFfPT9hwGdQjjgk7h9WbdbUN9Ob9DGCaL5uKukQ3+5QBw78WdeH9qf8L8W6AMsink7oCKAhj9PLTqAd3Hs7HsIF1Du+Lr4UuIt/UDMyTQxL9GdmnwEEM6R9QL8nD0at6XxvdiZGKrejN60TxkRi9azOaMIlbtyWNI5wh6xgQdFzCNNWaufnMNaXnlKAWebgYSWgXw6tK9jO0dTXgDAVRrzUs/7SY62Ifu0YHM+Gk3fduH0D82lKW7DgNwYadwvNzdePa7Xdz4wR9kFFZy8+CjnRMT2wQyf8NBzBbNrcdUw9SaNKAda9MKGJV4/Cy2VptgH9dORez71fo14Qrofyu5FblsX3w5kxKsF8uHeFkDfWFN6Skd9u7hnbiqVxv62S4mk0Df/CTQi7+ExaLRYJ8Zf7PlEP/6fAvVZguv/LKH9mG+XNmzDQPjw0hoHUC4vxevL08lLa+ce0Z0ospk5rLurfH3dmfUzFXct3AzrYO8USg6twrgok7hxIb78d7qfWzNLOal8T0Z2a0VY974jSlzNzB1UCwfrEknoVUAXVsFohTcf0ln1qblA/5cUidgd28ThNmiUYrjZqO1rujZmrhwv1Nqc+By9v/Kxsh4lu7+hAmdJ/DYmsdwM7gxrtM4ADzcPAjAwBHTiZcxbEhEgJf9OgKAQFugl4ummo8EetHsNh44wr8+30xEgBef3jqQ5Sm53P3ZJvrHhjBjQi827CtkyZZDvLkylddXpAKQ1DaY7VnFXNMnhvsv6VzveDdfEMc7q/YR7u+J1rAw2ZorDvByp7TKxIWdwhnXOxp3NwOL/nE+t3yYzDur9jEgNpR3bzias79nRCfuGdHpuPF2a2M9sdqvfYj94p9jKaXO7iBvqkanr+HF9rHs2vUJn+z6BIBZw2bRIbiDfbdQ5ckRc9UZvVSAlztKyYy+OUmgF40yWzQHCyvQWtsvgqnr681ZzFq6F5PFwoiEKP7vym72NMz/kjN4+IuthPp5kl5QwRNfb+eH7Tn0iA7i45vPw9vDjfZhflzbvy1HyqvZlV3Cpowivt2aTVSgN49d3vW413t4dAJ3DOto/5M/p9jIj9uz+fNgEeN6RzO0S4T99cP9vfjM9uFycbdIvNzdGn2/naMCiA3zZWL/c3hBnANr2K5q2GUu5fZet1NWXUZ8cDwj2tXvZRji5s0RXXaCgzjGYFAEentIoG9GEujPERaLZnNmEUkxwfWqUOyMxXB4J6T/RtXWL6gIjCfw+o/5aP1BXv55D2VVJjzdDKx4cCjRdfLOew+X8vAXW4kN86N9kLUGPdzfk7uGd2Lv4VIe+2o7A+PDeGdKX575dief/Z6Br6cbsyf1tteW1wrx8+T8juGc3zGcO4d1POF7MRhUvWXnWgV5c+PgOG4c3PD+Pp5uXN7T8aWLPd0NrHxwmMP7n3VM1fDTIywMi8TH3Yep3abi79lwO+QQDz8yVa61Qsdw+rUdQT4S6JuTBPpzxOsrUnnllz08fkU3bj52ybbUpfDZ38D2J/hB3ZZOBd/x4jP38ZZxFBd1juDirpE8/c1O3lu1jyevSgTgSHk1d3+2CT9Pdz66eQAR/l7ct3AzM37eQ2puGduyignwdmfWxCQCvD148qpEiitrGJsU3ayLVTeZGiOkfAuJ48DQ+F8CTcJYAt6Bje/X1GoqYe8v1q85WynOT+HH2FjGxF95wiAPEOoRyDaDG1QVg8/pr2olgb55SaBvCSWHwDcM3E9edmcyWyisqD5hnthRGw8U8uqyvXi6G3h16R7G9Y4m1M/T+mBZLiy+ncqA9qyJu5sXNntR6RXOwqDZ3J/3Kb0uvoJRI/qjlGJbZjGf/X6QqefH8tWmLOb+tp+KahNzp/a3j/GFa3qilGJtWj75ZVV8PD6GSEsBEI2vpzvvTOl3Ru+FtBWw9XOoLoPxH4DbGfwXLkiD8jzwDoJIW4po6ZOgLTDiSfjp35D8Pnj4QsJlZzZuR+z7FT4eB7cugza9m//1am2cBz89ykw/N/wtmluLS/ik03lUmbK5LuG6kz41xDuEI24GLBUFGCTQOy0J9E2tLA8sJgg8Qaqg6CC8cR70vxlGPnvCwxhrzNzyYTIbDxSy8sFhRAV6k5ZXRnSwz3EpD8wmyEqGqETwqnPxTuE+jqx+j+DN3/Kibw86/G0Gd81ZxpIvPubGq0Zal4Jb9jSmyhLGFD7Anpww2of58tm084jxToK3BjE69WkYNgySP+CZvPlsskxmxMsrsWiY2rmGR4uewHNNa8g8H4ZOx9vDi5kDSsHzC/Tu71FLiqwX3FwxE/raVp8szoTfZkFNBYTEwUUPQG1pZXEmrH+LHyNiWJSXzOzE2/ENaA1BMbD7B/hsIrj7gKnSesVmzwnHf/O0hq/usAbsMa+DmwdUl1uD+OGdcP0iKM2BNweC2bZW6egXrGP5bab1/qHNsN9WXpi23Broi7NAGY7/2VaVQfJca7sAgzsMugsi6p9AdsjmT0GbYdP8pg30FrP1gqec7ei/fY6q++GYugy+vY9DsQP5wLa8c8fz7uaj/YsY2X4knUNO/j5CfMIwK0VpaRZBYSdOtzUmyMeDQ8WVje8oTosE+qZUXQ5zhlsD793J4GlNT2itqbZU4+XmBb88YQ1wWxZaZ422XzrT7+/Cwd9xj+pGdderueObXAanv8YrbqtZ9+3DdBsxhcvfm0uXaCM3dzJx+YD78PD0gw3vYln5InnVR/hvq0TW+loYHvIIPjkV/OvQ/fibyjlIB8abvoP/rWONZ6F1scdZ1iGb3byYXnUjHRL78+GV3QjzdyP5cDKVKowul82Az6fA/26ElO/wBr71SWN25NNcevk19EiZBRmZENQafnsFDq6H0DjYPB+8glAJl7E2OIL1B3/ln9/cgyFjAxsi42i/YS6tyo+wPCCYtRlV/LNNT/w7jYLsLTD/WkxlOcyKaUOWhzuv7V7Kw4TCnb/DhrchMAbLXb9T/d5wvFe/DJ0uhuXPWoNZqx7QY4L19bd8an2DJiPEXgDr34RC28VTa1+HglTrB9DEz2DTx/DTI+ATChFdocNwWP8GtE6ypiPSlllz0B9eAeX5cN3HED/Ueqwdi+G7B6AiH1NgNDM9qxhyaB0DblkD7p6O/9+pMZK19wdmR4TRN+1rRlVMJ8i3TqvkikLrB0/b8yCwTcP/9zz9rKmX32ZBVQl0vRLMNdbv2+7vMQN3LJlAUFgnXky6n6rti1i34RUujOzKom4jULs+Jsw7jH+mzgfgzt6NLxYX4mu96rewJIszqUEK9PGQ8spmdM4E+uKqYvw9/HEzuGG2mDlSdYRwn6O/SDnlOTy7/lnGdx7PgFYDmL1pNmaLmQe73oinX4Rjv7TLn7PO2MH6yzb8UQBm/TmLT3d9yn1xY5i4YzGGtudBxgbYtxJj7DAOfPs0T+V+RrG7O2+mLCZyxUtcXdODK9w3UOwWypg9/2bhoeV4xuwhXWkeT4PyrGTG9bobnx8e4mf3BJ6I9qPGcARztQc/7n+LX/K2UqTdmR7wBs/cNAaK/4D1b2GM6sW/1nnRxSOPYb06ctMGD8Ijo1k0oQdfpi5gzrY5HKk6QoBnAF9e9SWtul4Ju74hvXUinydcRGrq93SvfpnEqEnwxZcQPwSmLIZti6j4+g7I3IDvRQ/ChQ+wNvdP7lp+FzXUEN/rSlrv/YZbioMJCHZnZM+JfJm5Ao0nm9Y+wj8qs8le8QxD3L1JueI/ZO14mwTlzfwguPRQJj2XPUXZ/l95p9covv/6KkzBsCRtN0GvD4DKQvD0h40fwPJnoLqcPzpeRFlkZ4asnYNh51cQmQhTv6Fkw1vM3v4OMVVV3HjebdaZetxFVM+9hJL83YRPWgAx/SC6D8aYfnyz4WVGpq8kaOP71g8Kn1D45BroeyN4BVo/4KL7waQFLCpP46MNz/G5pYI3l/2b/oOnQ/oq64dBVHcYOh2tNXmVeUT61m+LULPnJx4M9ma7txffA+98NYYlE37Gt+Qw36x9noE7fyKyshhQEJFg/SslcRxceD9sWQCL/279K6C6HPL3UOPmRebGd2lXY8JNucHoF/hs01usLUmFklS6bVrEOk8Da0P9uCy6D+v3fcNFMRdxbedruWPZHYzpMIb4oIYvFqsr1N/6182R8hziGtn3ZGpTN7U9itbvK2Dv4VK6tAqkb/sQ5+4H5AKUM/SA7tevn05OTm624+fuX8nYX+8hwbcVs8d+ySO/PcKqzFU8PvBxrul8DQAP/foQP6T/AECoNnBEaTSafsZqLlZ+ZHa4iDYWRVuzxtBtDNGB7egQ3AGjycjunD8J3bWJ6DWP8YPnKKJ9a+hV9hul01bz9J+b+KngWSLdfMg1V9LerCnVffh70Wpa0ZOtJVFkRS7lJz8/LBYf3IAHcyxcX70X+tzAxsRHSf7gfowRv/F+UBALs3L4v/BICtwtfHkwnxwdxuT2nTB45TI/O4cfAiJ416eKhYVVdPzbEjwiu9S74rSipoJFW7byxKICDF45+Me9gbeHB9H+bUgtSmVwm8GMjhvN8xuep2tYV+ae/x82Ln+cf1bspNpSQ7RnMPsqDzPJK4Z/p6xFXfU69JnCzoKd3PXL3wn2CGDB2K/YW7SXG3+4kfaB7fEweJBdno2vuzeYjIT7tWZTwXYuaX8JV1TBI4d+otxWsRHg4U+YTzgmi4mFVy7k6q+vRpfn8XL2IWaGhrDFx4dBbc5nTdYabjLCfYVF7L78eQ6HtMWnOJO+m75gf8l+JgYZMJqr6OoXwzNJ99Cl46XsyN/BP5ffRU5lPu5a8/VlnxIT0Z0v937J25vfpLiqmHmXfURimPVk87tb3+W1Ta8RaTJxS6mRPE8vggbdw8i032md8gNGcxWzO/Zja3AUd/W5mwd+fYD4oHiK83aRba7gqrJyriorp6f2gupS8i/9D09l/czK0n280OMOLu/zD7LLstlbtJdVvz7JQlMeMy74D77f3M8dId78y+RP+4L93BMVwUhDEC+f/4x1gnBos/XcQlYyXPQQrHsDQuNJcbPwvqGcvUGRpFfmYdJmrm01mMcveJYsXcW4L6+gb3kpbgYPfrU1bxvedjjLM5YD8NbFb3FB9AVsyt1EQmgCPu6NX9W7K3MN1y67nVnRlzPi4hesqcvNn5CdeBWZVYX0b9Xfod/Rt1am8eKPKex8ehR/Hihi6ge/Y7atrTikcwRvTO6Dv3S2PI5SaqPWutETX2d/oN84j0fXP8P3fj6YgSA3L4os1XQ0aVLdFZM6T2CgIYB7U97n5vhxsHk+a73cebi0hsPKwmOhAdQo8LJYqDqmfCzK7EWpqqLCANeUljEp35/HWv2LA7lZ/Mpr/OHlyYPhobSyVLHgUDY/djyfd0x+ZNRkY/CwdvwzaI1FKW7peieB9GP29ofRbkd4LvEmBidOxsvdh0nvrifb537OM5bwUnUwo9UU8oPncn9hCVs6XMqyI2t48cIXuSxzJ6UrnmVU+7b0jOzD/YMeJS4oDg+DtRTxj5w/eOy3xzhUfohu3tdxwLgGT68Khrcbxtb8rUzpOoWxHceilGJJ2hIe/e1RDMqARVuID4rnzYvfpI1va17+4Dw+dDdyUYWRGy59i60l+3lv23v4uPtQaCzkpsSbWJ6xHKPJyMIrFpJdns2k76yXzc8dOZe+UX3ZXrCdHuE9MFRXcHh2D7It1fiOeZ0HU+axr3gf0wdMZ3LXyewu3M0dP91CbnURBuDFi15idNxopq+ezrIDS7m+wzjm7Flg/5mc1+o8CqsKKags4M6kO3l7y9sopZg5dCZ3L78bH3cfpkdfwoN75zO8/SX4uPvwxd4v6BnRk9yKXLTWLLhiAb7uvoz6YhTtAtpRengL+90UbijMWH9fYvzaoM3VZBnzCfQMpKS6BIMy8PkVnxNqUfx32d2srMrFqE0Mbj2IqJwd/GIqokopWpvN5Lu58a8uf2PGngVU2NpAXuPVhicn/gQ/PsKt6Z+z19sXXw9/Mk3WOvWvxn5FgEcAP+z/gdQje+i5by3X7NuI8g6m4tZlXL3yLkqrS+kT2YeOIR3JLs/mu33f8VD/h1i4eyF5FXl8lZWNl9nEfd3O54qu1zGh8wTmbJvDlrwtvDrsVQzq1Eokc0oyuGTxZTwRNpAJl70Dn4zDvG8l0zr2YI/BzE/jfyLQM9CaWjtJ5dKnGw7yyOJt/Hd8T579dietg3x46/o+rNydx3Pf76JdqC+do/xpHeTDLRfGERPie0rjPFud04H+UNkhKk2VULCPjC9v5O7IUKZ1nkj79HU8U5XOvSWVTDaEMMOSz2eB/miliDKZWVJkwddUBZM+gyX3gDaz98r3MXh5EJu+inWWSNb+voTRxu/Y4enJUj8/VE0ApZYAdgYXEBfYgf0laQAosyfarZpIk4l/HA7n4/KJeLXrw+/7C7lrWAcubbeD1T//i+q2A2jbfSJjO41DKUWRsYh7V9zLn7l/AuDt5k2fyH6szf6Nd+IncX7iJHRILBO/ncjOQusqSVO6TeGh/g9ZywG3fMYcQxmvbnsXgEjfSKZ1n8aWvC38uP9H2ga0JT4onpWZK4Gjs7hjaa35dt+37C/ej6+HL9d2udb6CwvonUuY99MdvBcaRinWBTrOa3Uez1/4PDOSZ/D9/u9RKOaOmmuf0b239T2UUtzS45bjf2AHNwAa2g2kuKqYXw78wlUdrsLTzZouO1x+mOd+voPRncZwWfcbrE8pOciYr8Zg0iYujb2U67tdT0phCi8nv0yFqYI3RrzBRTEXsbtwN1N+mILRZMTfw59PLv+E+KB4Zm2cxdztcwG4pcct3NP7HnYf2c0NP9xAlG8U/Vv15397/sfHl35Mt3VzyNq+kJh/rCfH3ZOfD/zMrsJdFFQWcGuPW0kMT2TmxpnEBMTUe3/lNeX8b/f/mLN9DjXmakZYvJjWbjS+MQMYv+qflBoMJNRYmN5qKL4ePiScdy8quC2YqvgjfSnT1kwH4D8X/Ien1z1NUmQSaUVp5FXm4e/hT1lNGde5hXF/vwd5s3QXH+78kA9GfUC/Vtbf+xpzDdf/cD07C3YS7BXM7OGz6V1RAe7eENP3NH6zjldtrqbvJ325ya8jOUrT7UAypcExvOtWznOxV3PVkKcgbw98cCkMnQ4Dbm3wOD/tyOHvH28EINzfk8V3DKZtqDWYr9qTx8u/7KGqxsy+vHI0mpsviOe+Szo5dAHc2axFA71SajTwKuAGzNFav3Cy/U870JtrrP88fcFioXz9m/w343u+NGbU2y3cO5Rvr/4eT+3OwXWf0r7Plbj7BGJZM5ulezfyYmU5w0qqua9mM6t6PE/3YdexYuchvtuaxYaD1tmUn6cb5dVmAr3dmd99Iz38y2DwPWSaAtmWWcTWik/5et+XTOk2hbb+sbz1+zf0jkrksUGTKa7xZs7qfXy7NZv2Yb58cvN51ja7pqoGSyyrzdV8t+87KkwVbM3byg/7f6BdYDuWjF1in3Ftzt3Moj2LuL7b9SSE1l+4QmtNWlEaKUdSWJCygC15W/Bx92Fy18nc2uNWvN29eX/7+/Ztp8xigZ/+TXHnkaxRVfSK7EW0fzQAhcZCpnw/hXGdxjUc1JvQwpSFlJvKuTHxRvv3JassiwMlBzi/zfn2/ZYdWMbT65/muQues3+oFVcVM+m7SVzc/mLu63OfPb31R84fTF89ndyKXAa0GsDcUXOtJah5uyHuwtMaZ425Bo22f3ABrP39NVbs+55/jnobv5D2xz1Ha80/lv4Df09/ZgyZwYu/v8gnuz6hlV8rXh/+Op1COvHqn6/y/vb37c+Z0HkCTwx6ot5xMkoyeHvr29zW8zbaBx7/Ok1h4LzuGAELoG3fx8vMXryQfQh19Tvw64vonPNh4f8AAAtgSURBVG3k+gSRes1b9Iu72FqYUEeVycz327JpFWhtRhfw/+3de3BU5RnH8e+PDSGwQUi4QyEJKAhUBawKrThjiwpaRWunBayXep+WWmptx0vt2Bmdqq11dOyU6tSK9Vqndcq0lqqtUyo1qCB3RVRQwRgwhhBKwJA8/eO8kU3YDRGz7Mn6fGZ2cvbdc84+eTb75Oy757xvUff9nwh4b3sDdz7zOk8s2/zxEX5VXQMjSpMcX1HCnBPKPlNdPDkr9JISwOvAKcBm4CVgtpmty7TNwRb6XUvn023xHRRN/SHLNy7i+v+tp6ogwQX1uxi/OzpVa+Wgc3mz4GwSzSVUvlVD7a5GRg1IMv3zg3l23VbWV9czbexAJpWV8N8NH/B8mIQC4MjBvTnjqCEkEmJzbQOTR/Zj2tiB9CpM/4eUabKLT2tT3SYKE4UMLU5ztsUBmBlra9YyODm41ZfP2ZStPHwa6WLKFGfdnjoWrF3AGSPPaDWuSy7V7q7lwXUPMufIOQzotW/mqRerXmTFthXUNNQwd+JcehemHxs/m2Y8MIHNauLGPhOoOPYyFle/xBUjz6b4iW9D1Uo+SHRjZvkodjRHF+Q9dtoDjG/uBmufZPf2d1ncXMfYYV9k+Lhzo2tM6quiHSf7Q//R1Hy4gZc3PkvTsEl071HMKWWn8K/Xqrn5b69SVJBgaN8imreup2ftel7oeRJnHTOUUQOL2Va/h/frGijrl+Toz/Vhysh+7c5j0BXlstBPAW4ys9PC/esAzOznmbY52EL/s7/cwJ9qFzJ8byPvFhTQq7Enu2ou5fBeR1BRu4TkR9t4qPlUhpcmSYRBqI4tK+GhyrfZsHUnE0f05fzJZZwzcdi+I7pNH7Ls7VpOHjOQMRkmlHDO7XPX0lvppgTfO/5HrR/YsxOeuobmkgpu7dlExZY1HPH6s4zf8xFFZtzUvz+LinuxSzC3djtXbN/RavPqRIIbB5SytKiI5vD+PKzwMJbMXrJvpY2LYdH1UL2apu5JrhzyR57fuJOGxia6CUqThXywM7pWojRZyJhBvemsY5DDiroz//zO6QI7WB0t9Nn4jDMMSO072Qyc0HYlSZcDlwOMGHFwg0cNGDCN0m1N7LV1HNnUh6OHXU1D/wLe+XAX3YfNZPSwPlSOH7TflaXfmlxG/e5G+vba/5TJ48pLOa485jP/OBcj3z/h2vQP9CiGc+bTDbgeonP8Vz4Ke+ohOYC9219hemEvZpRP57huxdF1GCUV0Hc4IErr3mHHsp9zSclYvjzydJKlo+lW0KZLp0dv6F4E028lMf4c7us9mOZmo2rHbvolCynqnqB+dyP/fbOGv62qoqoTL8pqbEo/gXwcZeOI/uvAdDO7NNw/HzjBzOZm2ibbp1c651w+6ugRfTY6rLYAw1Pufy60Oeecy4FsFPqXgCMkVUgqBGYBC7PwPM455zqg0/vozWyvpLnAP4hOr7zfzNZ29vM455zrmKyccGpmTwFPZWPfzjnnPpn8OqnUOefcfrzQO+dcnvNC75xzec4LvXPO5blYjF4paRvwdq7jSKM/8EGug/gEulK8XSlW8HizrSvFG6dYy8xswIFWikWhjytJL3fkqrO46ErxdqVYwePNtq4Ub1eKtYV33TjnXJ7zQu+cc3nOC3377s11AJ9QV4q3K8UKHm+2daV4u1KsgPfRO+dc3vMjeuecy3Ne6AFJwyU9J2mdpLWSvh/ab5K0RdKKcDs917G2kLRJ0uoQ18uhrVTSM5I2hJ8luY4TQNKYlByukLRD0rw45VfS/ZK2SlqT0pY2n4rcLekNSaskTYpJvL+Q9FqI6UlJfUN7uaSGlDzPj0GsGV97SdeF3K6XdNqhjLWdeB9PiXWTpBWhPae57TAz+8zfgCHApLDcm2jO23HATcA1uY4vQ8ybgP5t2m4Hrg3L1wK35TrONHEngPeBsjjlFzgJmASsOVA+gdOBvwMCJgNLYxLvqUBBWL4tJd7y1PViEmva1z6871YCPYAK4E0gket42zx+B/DTOOS2ozc/ogfMrMrMlofleuBVoikRu5qZwIKwvAA4O4exZPIV4E0zi9UFcma2GPiwTXOmfM4EHrRIJdBX0pBDE2kkXbxm9rSZ7Q13K4km/cm5DLnNZCbwmJntMbONwBvA8VkLLo324lU0ufQ3gEcPZUyflhf6NiSVAxOBpaFpbvgofH9cukICA56WtCzMvwswyMyqwvL7wKDchNauWbR+k8Q1v5A5n+nmRY7bgcHFRJ86WlRIekXSvyVNzVVQbaR77eOe26lAtZltSGmLY25b8UKfQlIx8CdgnpntAH4DjAImAFVEH9ni4kQzmwTMAL4r6aTUBy36XBmrU6rCjGNnAU+Epjjnt5U45jMTSTcAe4GHQ1MVMMLMJgJXA49IOixX8QVd5rVvYzatD1TimNv9eKEPJHUnKvIPm9mfAcys2syazKwZuI9D/BGyPWa2JfzcCjxJFFt1SxdC+Lk1dxGmNQNYbmbVEO/8BpnyGdt5kSVdBHwVOC/8cyJ0g9SE5WVE/d6jcxYk7b72cc5tAfA14PGWtjjmNh0v9Hzc7/Y74FUz+1VKe2q/6znAmrbb5oKkpKTeLctEX8KtIZqb98Kw2oXAX3ITYUatjobimt8UmfK5ELggnH0zGahL6eLJGUnTgR8DZ5nZrpT2AZISYXkkcATwVm6i/DimTK/9QmCWpB6SKohiffFQx5fBNOA1M9vc0hDH3KaV62+D43ADTiT6WL4KWBFupwN/AFaH9oXAkFzHGuIdSXRmwkpgLXBDaO8H/BPYADwLlOY61pSYk0AN0CelLTb5JfoHVAU0EvULX5Ipn0Rn2/ya6OhtNfCFmMT7BlH/dsvf8Pyw7rnh72QFsBw4MwaxZnztgRtCbtcDM+KQ29D+AHBlm3VzmtuO3vzKWOecy3PedeOcc3nOC71zzuU5L/TOOZfnvNA751ye80LvnHN5zgu9c87lOS/0LtYk9ZX0nSzuf56kXgdYZ5Ok/p38vOWS5qTcv0jSPZ35HM618ELv4q4vkLVCD8wD2i30WVIOzDnQSs51Bi/0Lu5uBUaFSR1+L+ksgDCxxv1h+WJJt4Tlb0l6Maz/25TL00+V9IKk5ZKekFQs6SpgKPCcpOc6Ekw7+98p6RZJKyVVShoU2keF+6sl3SxpZ8rvNTXs5wehbaikRYomOrm9c9LnnBd6F3/XEo1fPwH4B9EwsRANXTsuLE8FFksaC3wT+FJYvwk4L3S7/ASYZtGIny8DV5vZ3cB7wMlmdvKBAsm0//BwEqg0s2OAxcBlof0u4C4zO4rocvrU3+s/ZjbBzO4MbRPC/o8CvikpdXAv5w5aQa4DcO4T+A8wT9I4YB1QEgbHmgJcRTTw2LHAS9E4dfQkGnFyMtE/hSWhvRB44SCe/ysZ9g/wEfDXsLwMOCUsT2HfhCWPAL9sZ///NLM6AEnriGbhered9Z3rEC/0rsswsy2K5kGdTnTUXEo0289OM6sPo5AuMLPrUreTdCbwjJnN/pQhpN1/0Gj7Bo5q4uDeW3tSlg92H87tx7tuXNzVE83j26KS6AvUxURH+NeEnxCNNPl1SQPh48m9y8I2X5J0eGhPShqdYf/tybT/9lQSjXAI0examX4v57LGC72LNYsmdVgiaY2kXxAV9QIze4NoWNjS0IaZrSPqi39a0irgGaLhb7cBFwGPhvYXgCPDU9wLLOrIl7GZ9n+AzeYBV4f1DwfqQvsqoCl8efuDjFs71wl8mGLnsiico99gZiZpFjDbzGbmOi732eJ9gM5l17HAPeH7g+1Ek3Y7d0j5Eb1zgaSlQI82zeeb2epcxONcZ/FC75xzec6/jHXOuTznhd455/KcF3rnnMtzXuidcy7PeaF3zrk8938zSS3fvtQtrgAAAABJRU5ErkJggg==\n"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"We see that for negative sentiment distribution is higly skewed towards longer tweets. \n\nWe split our dataset into train and test parts. We don't show the test part to our model until it is trained and use it for evaluation purposes. Then we save them as .csv files for later purporses."},{"metadata":{"trusted":true},"cell_type":"code","source":"rnd_state = 111\ndf_train, df_test = train_test_split(df, test_size=0.15, random_state = rnd_state)\ndf_train[['airline_sentiment', 'text']].to_csv('Tweets_train.csv', index=False, encoding='utf-8')\ndf_test[['airline_sentiment', 'text']].to_csv('Tweets_test.csv', index=False, encoding='utf-8')\n\n","execution_count":277,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we create TextDataBunch objects for language model and classifier and save them so in future iterations we can load them straight away and skip running previous steps. \n\nWe specify 15% our training data for validation purposes so that we can experiment with hypoparameters and adjust them based on perfomance on the validation data."},{"metadata":{"trusted":true},"cell_type":"code","source":"data_lm = TextLMDataBunch.from_csv('.', 'Tweets_train.csv', valid_pct=0.15)\ndata_clas = TextClasDataBunch.from_csv('.', 'Tweets_train.csv', valid_pct=0.15, vocab=data_lm.train_ds.vocab, bs=32)\ndata_lm.save('data_lm_export.pkl')\ndata_clas.save('data_clas_export.pkl')","execution_count":278,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Load previosly created data for language model and classifier."},{"metadata":{"trusted":true},"cell_type":"code","source":"data_lm = load_data('.', 'data_lm_export.pkl')\ndata_clas = load_data('.', 'data_clas_export.pkl', bs=32)","execution_count":279,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we have a look at what our TextDataBunch objects contain. The texts of tweets has gone through automatic tokenization stage that includes but not limited to:\n* Separation according to spaces and punctuation.\n* Transforming text to lower case.\n* Introducing special tokens that encode spicific information, e.g. indication of the beginning of a text.\nAfter that tokens are numericalised meaning substituting words by their values in the vocabulary. The resulting numerical sequences will serve as inputs for the language model.\nWe print an example preprocessed text with the encoding it was transformed to."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Preprocessed text:\", data_lm.x[0])\nprint(\"\\n\")\nprint(\"Corresponding numerical sequence:\", data_lm.x[0].data)\n","execution_count":280,"outputs":[{"output_type":"stream","text":"Preprocessed text: xxbos @united xxmaj what does \" xxmaj cabin xxmaj functionality \" mean when i 'm taking a customer satisfaction survey ?\n\n\nCorresponding numerical sequence: [   2   21    5   71  133   65    5  859    5 3858   65  545   86   11   80  450   15   73 3859 2601   14]\n","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"## 1. Language model"},{"metadata":{},"cell_type":"markdown","source":"Now we implement our language model. The UlMFit apporach consists of three main steps:\n1. Pretraining the language model. We will download the model pretrained on a large corpus of English text. \n2. Fine-tuning the language model. This is necessary to adjust the language model to the specificities of the dataset we are going to work with.\n3. Using our language model as an ecoder for the classifier which will infer the sentiment of the tweets.\n\nNow we are ready to create the learner for our model. It will come with pretrained [AWD_LSTM](https://arxiv.org/pdf/1708.02182.pdf) architecure which will serve as our laguage model. "},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = language_model_learner(data_lm, AWD_LSTM, drop_mult=0.5)\n","execution_count":281,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next we explore the space of possible learning rates."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.lr_find()\nlearn.recorder.plot()","execution_count":282,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.HTML object>","text/html":""},"metadata":{}},{"output_type":"stream","text":"LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"<Figure size 432x288 with 1 Axes>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8XHW9//HXZ2ayNXubtE3TJd0LbemWFoGyiQrIvVgEveDGqtQF9F6vyL3eq3IV3K6/K6iAFawKqFcKCggCArJDS0pLF7rvabokbdLs28z398dMQshN29DkzJlJ3s/HYx7MnDkz590wyWe+3+8536855xAREQEI+B1AREQSh4qCiIh0UlEQEZFOKgoiItJJRUFERDqpKIiISCcVBRER6aSiICIinVQURESkU8jvAO9VQUGBKykp8TuGiEhSWblyZZVzrvB4+yVdUSgpKaGsrMzvGCIiScXMdvVmP3UfiYhIJxUFERHppKIgIiKdVBRERKSTioKIiHRSURARkU4qCiIi0klFQUQkCfzkmc28tKXS8+OoKIiIJDjnHD99bivLtx/2/FgqCiIiCa65LUI44shK934SChUFEZEEV9fSBkBWmoqCiMigV9/cDkC2WgoiIlLfEi0KaimIiEhnS0FFQUREqOtoKaj7SEREOscU0lI8P5aKgohIgqtXS0FERDp0FIXMtKDnx1JREBFJcHXN7aSGAqSFVBRERAa9+pY2suNw5hGoKIiIJLz65va4jCeAioKISMKrb2mPyzUKoKIgIpLw6ppVFEREJEYtBRER6VTfojEFERGJqVf3kYiIdKhTS0FERABa2sO0tkd0nYKIiEBDSxiIz7TZoKIgIpLQOtdSSPd+hlRQURARSWjxXHUNVBRERBJaR1GIx/rMoKIgIpLQ6lvaALUURESE6BQXEJ8FdsDDomBmU81sdZdbrZl9pds+ZmZ3mNlWM1tjZnO9yiMikow6u4/i1FLw7CjOuU3AbAAzCwJ7gT912+1CYHLsdipwV+y/IiJC17OPkryl0M15wDbn3K5u2z8C/NZFvQ7kmVlRnDKJiCS8+pZ2AgYZKd6vugbxKwqXA7/vYXsxsKfL4/LYtncxs8+ZWZmZlVVWVnoUUUQk8XRMm21mcTme50XBzFKBi4EHT/Q9nHNLnHOlzrnSwsLC/gsnIpLg6lvayY7ThWvg4ZhCFxcCbzrnDvTw3F5gTJfHo2Pb5ARUN7TywPJdvLSlipNH5XDq+KHMLxnKsKw0v6OJyAmK5wypEJ+icAU9dx0BPAp8ycz+QHSA+Yhzbl8cMiWl+pZ2Xt92iJe2VFJZ30LJsEwmFGYxKi+dp9bt549l5TS1hZk2MpvfLd/N0ld2AjAyJ52C7FQKstIYmplKe9jR0NJOfUs7Da3tNLaGaWwJ09jaTjjiMDMMCAWNScOzmD4ql+mjchhfkElGapD0lOgtaIbDAWAYaaEA6SlB0kIBHNDUFn3PxpYwhxpaqKxrobK+lebWMCNz0xmVl8GovHRa2iLsrWmivLqRqvpWivMymFiYxYTCTDLj+MsgkojiuZYCeFwUzCwT+CBwfZdtiwGcc3cDTwAfBrYCjcDVXuZJNPuONPHmrhrWlNewek8Nuw834hyYQcCMlKCRnhIkIzVIJOJYX1FLe8SRnhJgZE46T68/QHsk+kc5JWh8ZHYx1505nmkjc2htj7B27xFW7DjM9sp6qupbqKxvYfP+OlJDATLTQmSlhRienc6Q1GDsFiIUMBzgHDS3h9m8v44/lu2hsTXsy89oVG46E4dndRaJxtYwe6ub2FvTxKH6FoIBIyUYIDUUoC0coa65nbrmdhpa2gkEjJSAkRIKkJuRwsziXGaNyWPOmDwmFGYRDMSnj1akL+pa2snLiF/3kTnn4naw/lBaWurKysr8jtEn6/Ye4ed/38qT6/fjHKQGA5xUlM3E4Vmxb9/RP8pt4QjNbWGa2sKEI45ZY/I4c1IB80rySQsFaQtH2HO4kV2HGzm5KIcROeme5A1HHDuqGiivbqS5LUJLe5im1jAdnxwDIi46xW/H8wBDUoNkpIYYkhJkWFYqhdlpFGankRYKsv9IMxU10T/uaaEAo/OHMDo/g4KsNMqrG9lWWc+2yga2HqyP3j9YT0OsMOVmpDAqL4PC7DScc7S2R2gLRwgGjOz0FLLTQ2SmhXAO2sPR5yrrW1iz5wh1sXO+01MCTBmRzdQR2UwrymHO2Dymj8ohLRSfMzxEeuu8Hz/PtKIcfv6Jvl3GZWYrnXOlx9tPbfN+dqi+herGNtrCEdrDjpb2MA2t4Wh3TXM7T6zbx/ObKslOC/GFcyZy/vSRTB2ZfUJ/jFKCASYUZjGhMMuDf8k7goFoN9Kk4f13nNyMFKaOzO7xuckjspk84t3POeeorGshIzV4woNukYhje1UDq/fUsGFfLRv31/L3TQd5cGU5AKmhAKcU5zJvXD5zxuYzd1wew7PTO49f3dhGa3uEkbneFF+RntS3tMftwjVQUeiz6oZW3txdzctbq3hlaxWbD9Qfc/+hmal87fypfPq0ceTE8YyCZGdmDO9jSyhwlOJ2oLaZVburWbkrelv6yk5+8eJ2AIrzMgCorGuhNRwBYNrIbP5x1igumllESUFmnzKJHM9AHGhOSKv31LC3uomTirIZNyyz1/3LB2qbue+1XayrOMKGfbUcqG0BIC0UYMH4oVwyZzTF+RmkBqN93SnBaP99ZlqQzNQQw3PS1EWRYEbkpHPBjCIumBG9brKlPcy6vbWs2l3N6j01pIYCjMhJZ0R2Gu0Rx1/X7edHT23iR09tYszQDEqGZUZvBZmMHTqEMUMzGJM/RIPk0mfhiKOhNTxwBpoT0fqKI/z46c08t/Fg57aMlCDTirKZMyaf+SX5zCvJ7+w26NDY2s6SF7fzixe20xqOMHl4FmdMLGBaUTYzinOZOzaf9DhdcSjeSgsFmTcun3nj8nt8/rozJ7C3pom/rt3HmvIj7DzUwJ9X7+2cuKxDcV4G/zhrFB+dW8yUET13lYkcS0NrfNdSgEFUFHZUNfDfT2/i8TX7yM1I4aYLpnLmpEI27q/l7X21rN9bywPLd/GrV3YA0V/o4rwMRuamMywrlSfW7uNAbQsfnjmSr18wjXHD1G0wmBXnZXDdmRM6H3eMOew53Mjuw43sqW5kxY7D/PKl7dz9wjamj8rholOKeP+04UwdkR23q1MluXXMexSvtRRgEBWFnVUN/H3jQW54/ySuO3MCubFTvGaOzu3cp7U9wvqKI5TtrGZ9xRH2HWnmrfIa9h9p5uRR0dH/0pKhfv0TJIGZGUMzUxmamcqsMXkAfOGc6FjEY29V8OfVe/nhk5v44ZObKM7L4LyThnPNGeM1JiHH9M6qazol9ahO9JRU5xw1jW3kZ6ae0Gv1zU766kBtM3/feJDnNh7khc2VtEccHy8dw43nTaIoN8PveJKAVu6q5tK7XuU31yzg7Cl9m+JHp6R2Y2YnVBA6XivSVyNy0rl8wVguXzCWg3XN3Pn3bTywfBcPvVnOFfPH8JnTS5jo8enFklzivT4zDKKiIJJIhmen8+2Lp3PtwvHc8ewWfrdiN795bRcLJxXw6dPGcd604YSCWhhxsNOYgsggM2boEH70sVncdME0/veN3TywfDfX37eSwuw0PjqnmI+VjmbScJ25NFjFe31mUFEQSQiF2Wl86f2TWXz2RJ7beJA/lpVzz8s7+MWL25k7No/PnTWBD508koDmaxpUOk5zjuc1LyoKIgkkFAzwoekj+dD0kVTWtfDnVXu5f/kuFt//JhMKM1l89kQWzS4mNaSupcHAjzEFfbJEElRhdhqfPWsCz331HH56xRzSQ0FuWraGRT9/hQO1zX7Hkziob25nSGowrjP6qiiIJLhgwPjHWaN4/MaF3PXJuew61MAlP3+FTfvr/I4mHqtvie+8R6CiIJI0zIwLZxbxv9efRnvEcdndr/Lq1iq/Y4mH6uK8wA6oKIgknRnFufzpi2dQlJvOlUtX8NvXdpJsF6FK79Q3x3fabFBREElKxXkZPLj4dBZOKuCbj6xn8f0rOdLY5ncs6WfxXooTVBREklZuRgr3Xjmf/7joJJ7beJAP3/ESK3dV+x1L+lG811IAFQWRpBYIGNedOYEHF59OIABXLHmdVzTOMGBEB5rjuxiXioLIADB7TB6PfnEh4wsy+exvy1i1Wy2GgaCuuS2uU1yAioLIgJGfmcp91y6gICuNq5a+wcb9tX5Hkj5wzumUVBHpm+E56Txw3alkpAT51D0r2FnV4HckOUHNbREiDg00i0jfjBk6hPuvW0DEOT7xy9fZc7jR70hyAup8mAwPVBREBqRJw7O579oFNLSGueKXr1NercKQbPyYNhtUFEQGrOmjcrn/2lM50tTGJ365nH1HmvyOJO+BH5PhgYqCyIA2c3Qu9117KtUNrVyx5HUON7T6HUl6qaOloKIgIv1q9pg8fn3NAipqmrlp2RpNiZEk6jpaCuo+EpH+Nm9cPjdfOI1nNhzg/uW7/Y4jvdA5pqCL10TEC1efUcLZUwr57l/eZvMBTbud6OrVUhARL5kZ//2xWWSnh7jx96tobgv7HUmOoaMoZKYF43pcFQWRQaQwO40fXTaLjfvr+N4TG/yOI8dQ19xOaihAWmgAFQUzyzOzZWa20cw2mNlp3Z7PNbPHzOwtM1tvZld7mUdE4Nxpw7l24Xh+89ou/rSq3O84chR1zW1xP/MIvG8p3A486ZybBswCun81+SLwtnNuFnAO8GMzS/U4k8igd/OF0zh1/FBufmgt6/Ye8TuO9KCmqY28IfEdZAYPi4KZ5QJnAfcCOOdanXM13XZzQLaZGZAFHAbavcokIlEpwQA//+RchmWmcv19KzlU3+J3JOmmuqGV/CHx/47sZUthPFAJLDWzVWZ2j5lldtvnZ8BJQAWwFviycy7iYSYRiSnISuPuT8+jsr6FL/1uFe1h/eolkurGNvIHUksBCAFzgbucc3OABuDmbvucD6wGRgGzgZ+ZWU73NzKzz5lZmZmVVVZWehhZZHA5ZXQe37tkJq9tP8Ttz27xO450UdPYSt4AaymUA+XOueWxx8uIFomurgYedlFbgR3AtO5v5Jxb4pwrdc6VFhYWehhZZPC5dN5oLp07mjuf38b6Co0vJIrqxlaGZg6gouCc2w/sMbOpsU3nAW932213bDtmNgKYCmz3KpOI9Ow//+Ek8oekctOyNbSpG8l3Ta1hmtsiA2ugOeYG4AEzW0O0e+g2M1tsZotjz38HON3M1gLPAl93zmmBWZE4yxuSyncXTWd9RS1LXtT3Mr9VN0YnLvRjoNnTk2Cdc6uB0m6b7+7yfAXwIS8ziEjvXDCjiItmFnH7M1s4f/oIJg3P9jvSoOVnUdAVzSLS6dsXT2dIWpCvLVtDOKLZVP1S0xhddW2gnX0kIkmmMDuNWy6ezqrdNfzsua1+xxm0Ota9yB9IA80ikpw+MruYS+YUc/uzm1mx47DfcQalmlj30UAcaBaRJPSdRTMYO3QIX/nDqs4/UBI/1Z3dR2opiEgCyEoLcccVczhY18LND63Vam1xVt3YSnZaiJRg/P9EqyiISI9OGZ3HTRdM5cn1+/ndCq3WFk/VDa3kZca/6whUFETkGK5bOIGFkwr43hMb2Xekye84g0Z03iN/JoxWURCRowoEjNsumUlbOMItj3afkEC8UtPozwypoKIgIscxdtgQbjxvMk+u388zbx/wO86g4NcMqaCiICK98NkzJzBlRBbfenQ9ja1a8sRr1Q3+zJAKKgoi0gupoQC3XTKTvTVN/OQZTbHtpbZwhLqWdnUfiUhiKy0ZyhULxnDvyzvYsK/W7zgDVscUF0N19pGIJLqvXzCN3IwUvvXIel274JHqzquZ1VIQkQSXNySVr50/lRU7D/PoWxV+xxmQqhv8myEVVBRE5D36eOkYThmdy21PbKC+RYPO/a1jigs/5j2CXhYFM5toZmmx++eY2Y1mludtNBFJRMGAccvF0zlQ28JPn9Ogc3/rmGvKj6U4ofcthYeAsJlNApYAY4DfeZZKRBLanLH5XDZvNL96eQfbKuv9jjOgHPZxgR3ofVGIOOfagUuAnzrnvgYUeRdLRBLd1y+YRnooyC2Pva1B535U09hGWihARmrQl+P3tii0mdkVwJXAX2Lb/OnwEpGEUJidxlc+OIUXN1fytK507jfVDf5NcQG9LwpXA6cBtzrndpjZeOA+72KJSDK48rRxTB2RzX899jZNrWG/4wwI1Y1tvqy41qFXRcE597Zz7kbn3O/NLB/Ids79wONsIpLgQsEAt3xkOntrmrjreS3f2R+qG1t9m/cIen/20fNmlmNmQ4E3gV+a2f/zNpqIJIP3TRjGxbNGcfeL29l1qMHvOEmv2scZUqH33Ue5zrla4KPAb51zpwIf8C6WiCSTb1x0EikB45bHNL12X9U0tpHv0xQX0PuiEDKzIuDjvDPQLCICwIicdL78gck8t/Egz27QoPOJikScr2spQO+Lwn8BTwHbnHNvmNkEQFetiEinq88Yz4SCTH745CYiEZ2ieiJqm9uIOP/mPYLeDzQ/6Jw7xTn3+djj7c65S72NJiLJJCUY4MsfmMymA3X8dd1+v+MkpY4pLpJhoHm0mf3JzA7Gbg+Z2Wivw4lIcvmHU0YxaXgWtz+7Wa2FE9AxQ2rCn5IKLAUeBUbFbo/FtomIdAoGjBvPm8zmA/U8sW6f33GSTo3PU1xA74tCoXNuqXOuPXb7NVDoYS4RSVIXzSyKthae2aLWwnt0uCFJuo+AQ2b2KTMLxm6fAg55GUxEklMwYHz5vMlsOVjP42vVWngvanxeYAd6XxSuIXo66n5gH3AZcJVHmUQkyV00s4gpI7K449kthNVa6LXqxlaCASMnPeRbht6efbTLOXexc67QOTfcObcI0NlHItKjQGxsYcvBep7UmUi9Vt3YRv6QFMzMtwx9WXntX463g5nlmdkyM9toZhvM7LQe9jnHzFab2Xoze6EPeUQkgVw4o4gJBZnc/cI2Ta3dS9UNrb52HUHfikJvStntwJPOuWnALGDDu94gunrbncDFzrnpwMf6kEdEEkgwYFx/9gTW7j3Cy1ur/I6TFPyeDA/6VhSOWfrNLBc4C7gXwDnX6pyr6bbbJ4CHnXO7Y/sc7EMeEUkwi+YUMyInjbue3+Z3lKRQ09jm6+mocJyiYGZ1Zlbbw62O6PUKxzIeqASWmtkqM7vHzDK77TMFyI/NwrrSzD5zlByfM7MyMyurrKzs7b9NRHyWFgpy3cIJvLrtEKv3dP9OKN35PUMqHKcoOOeynXM5PdyynXPHGx4PAXOBu5xzc4AG4OYe9pkHXAScD/ynmU3pIccS51ypc660sFCXR4gkkytOHUtuRorWWzgO5xzVDW3k+ThDKvSt++h4yoFy59zy2ONlRItE932ecs41OOeqgBeJjj2IyACRlRbiytPG8dT6A2w9WOd3nITV2BqmNRxJ7JZCXzjn9gN7zGxqbNN5QPfJ1h8BFppZyMyGAKfSbTBaRJLflaeXkJ4S4O4XtvsdJWF1zHs0dKAWhZgbgAfMbA0wG7jNzBab2WIA59wG4ElgDbACuMc5t87jTCISZ8Oy0rh8/lgeWb2X/Uea/Y6TkKpjU1zkJfHZR8flnFsdGws4xTm3yDlX7Zy72zl3d5d9fuScO9k5N8M59xMv84iIf65dOJ6Ig6Wv7PA7SkLaXxstliNy0n3N4XVLQUQEgDFDh/DhmUU8sHw3tc1tfsdJOBU1TQCMysvwNYeKgojEzfVnTaC+pZ3fLd/td5SEU1HTRGoowDAf11IAFQURiaMZxbksnFTAr17eQUt72O84CWVvTROjctMJBPyb9whUFEQkzq4/ewIH61p4ZHWF31ESSkVNk+9dR6CiICJxtnBSAScX5bDkxe1ahKeLippmFQURGXzMohPlbT1Yz3MbNd0ZQFs4woE6FQURGaQumlnEqNx07n1Zp6cCHKhtxjkozvP3dFRQURARH4SCAa48vYTXth/i7Ypav+P4rqImeo2CWgoiMmhdPn8sGSlBXcxG4lyjACoKIuKT3CEpXDZvNI+srqCqvsXvOL7a21EUclUURGQQu+qMElrDER54fXBfzFZR08TQzFQyUoN+R1FREBH/TCzM4typhdz3+q5BfTFb9BoF/weZQUVBRHx2zcLxVNW38Je39vkdxTcVNc0J0XUEKgoi4rOFkwqYMiKLe1/egXOD82K2RLmaGVQURMRnZsbVZ4zn7X21rNhx2O84cVfb3EZdS7u6j0REOiyaXUzekBR+NQhPT02k01FBRUFEEkBGapBPLBjL394+wJ7DjX7HiSsVBRGRHnz6tHGYGb99baffUeJqb+xq5mIVBRGRdxTlZnDhjJH84Y09NLS0+x0nbipqmkgJGoVZaX5HAVQURCSBXH3GeOqa23n4zXK/o8RNRU0TIxNgcZ0OKgoikjDmjs1j1pg8lr6yc9CstVBR05Qw1yiAioKIJBAz45ozSthe1cALWyr9jhMXFTXNCTOeACoKIpJgLpxRxIicNO59aeCfntoejrC/NjEW1+mgoiAiCSU1FODqM8bz8tYq1u094nccTx2sayEccSoKIiLH8olTx5KdFuKuF7b5HcVT71yjkBhXM4OKgogkoJz0FD512jj+unYfO6oa/I7jmY51FDSmICJyHFefUUIoGGDJi9v9juKZjmU4i1QURESObXh2OpfNG81DK8s5WNvsdxxPVNQ0kZuRQlZayO8onVQURCRhfe7MCbRHIvzqlZ1+R/FEIk2Z3UFFQUQSVklBJh+eWcQDr++itrnN7zj9bm9NE6NyE2eQGVQURCTBLT57InUt7QNuHedwxLG9qoEJhZl+R3kXT4uCmeWZ2TIz22hmG8zstKPsN9/M2s3sMi/ziEjymVGcyxmThvHrV3fQ2h7xO06/2XWogdb2CJNHZPsd5V28bincDjzpnJsGzAI2dN/BzILAD4CnPc4iIknqs2dO4EBtC39ZU+F3lH6z+UA9AFMGS1Ews1zgLOBeAOdcq3OupoddbwAeAg56lUVEktvZUwqZOiKbJS9uHzDrOG85UAfA5OFZPid5Ny9bCuOBSmCpma0ys3vM7F2dZ2ZWDFwC3OVhDhFJcmbGdWeOZ+P+Ol7eWuV3nH6x6UAdo/MzyEyg01HB26IQAuYCdznn5gANwM3d9vkJ8HXn3DE7Cs3sc2ZWZmZllZWDY+ZEEXm3i2ePYnh22oC5mG3LgfqE6zoCb4tCOVDunFsee7yMaJHoqhT4g5ntBC4D7jSzRd3fyDm3xDlX6pwrLSws9DCyiCSqtFCQq84o4aUtVWzYV+t3nD5pC0fYXjXIioJzbj+wx8ymxjadB7zdbZ/xzrkS51wJ0aLxBefcn73KJCLJ7ZMLxjEkNcgvX0ru1sLOqgbawo4pIxJrPAG8P/voBuABM1sDzAZuM7PFZrbY4+OKyACUOySFf5o/hkdXV7DncKPfcU5Yop55BB4XBefc6li3zynOuUXOuWrn3N3Oubt72Pcq59wyL/OISPK7/qyJBMy48/mtfkc5YZsP1BEwmJRgZx6BrmgWkSQzMjedyxeM4cGy8qRtLWw5WMfYoUNITwn6HeX/UFEQkaTz+XM6WgvJuQjPpv11Cdl1BCoKIpKEinIz+Kf5Y3iwbA/l1cnVWmhpD7PzUKOKgohIf0rW1sKOqgbCEcfkBDzzCFQURCRJjcrL4OPzR/Ng2Z7OZS2Twab90ektpo5US0FEpF994ZxJGMbP/548ZyJtOVBPMGCML0isKbM7qCiISNLq2lpIljORNh+oo2TYENJCiXfmEagoiEiS++K5ydVa2HygLmG7jkBFQUSSXFFuBlcsGMOyleXsPpTYrYXmtjC7DjcyebiKgoiIZ75w7iQCAeOnz23xO8oxbT1Yj3OJO8gMKgoiMgCMyEnnk6eO5eFVe9lZ1eB3nKPaHFtYJxEnwuugoiAiA8Lnz55IKGDckcCthbV7j5CeEmDcsMQ88whUFERkgBiek86n3zeOP6/a2/mNPNGU7axmzph8UoKJ+6c3cZOJiLxHnz9nItnpKXzrkfUJt5ZzfUs76yuOML8k3+8ox6SiICIDxrCsNL52/lRe236IR9+q8DvOu6zeXUPEQWnJUL+jHJOKgogMKFcsGMvM4lxufXwDdc1tfsfp9MbOwwQM5ozN8zvKMakoiMiAEgwY31k0g8r6Fm5/JnEGnct2Heakohyy01P8jnJMKgoiMuDMHpPH5fPHsPTVnZ0T0PmpLRxh1e4a5id41xGoKIjIAHXT+dPITg/xzUfW+T7ovGFfLY2tYUoTfJAZVBREZIDKz0zlpvOnsXzHYR5fu8/XLG/srAagdJxaCiIivvmn+WOYPiqH2x7fQGNru285ynYeZszQDEbmpvuWobdUFERkwAoGjG9fPJ2KI83c7dMKbc453thZzfwkaCWAioKIDHDzS4bykdmjuPvF7b6subDrUCNV9S0Jf31CBxUFERnw/u3CkwgFjO8+/nbcj/3GzsMACX8lcwcVBREZ8EbmpvPFcyfx1PoDvLSlMq7HLttZTW5GChMLE3dm1K5UFERkULh24XhKhg3hP/+8jua2cNyO+8auw5SOyycQsLgdsy9UFERkUEhPCfLdRTPZeaiRO+O0dGd5dSPbKxuSZjwBVBREZBBZOLmAS+YUc9cL29h60Psrne97bRfBgHHx7FGeH6u/qCiIyKDyjYtOYkhqiH9/eB2RiHdXOje0tPP7Fbu5YPpIivMyPDtOf1NREJFBpSArjX//8DRW7DzMspXlnh3n4TfLqW1u55qFJZ4dwwsqCiIy6Hxs3hgWlAzl1ic2sPtQ/1+7EIk4lr6yk1mjc5k7NjlORe2goiAig04gYPzwslMwg6t/vYIjjf277sILmyvZXtXANQvHY5YcZx118LQomFmemS0zs41mtsHMTuv2/CfNbI2ZrTWzV81slpd5REQ6lBRk8otPzWP34UYW37+S1vZIv733r17ZwYicNC6cUdRv7xkvXrcUbgeedM5NA2YBG7o9vwM42zk3E/gOsMTjPCIinU6dMIzvf/QUXtt+iG/8aW2/TLGDCmQGAAAKrklEQVS9+UAdL22p4jOnlZAaSr7OmJBXb2xmucBZwFUAzrlWoLXrPs65V7s8fB0Y7VUeEZGeXDpvNLsON3LHs1sYlZfBP39wSp/e786/byUtFOCKBWP7KWF8eVYUgPFAJbA01i20Eviyc67hKPtfC/zVwzwiIj365w9MpqKmiduf3UJbOMLXzp96QmMBf1ixmz+vruBL505iaGaqB0m952XbJgTMBe5yzs0BGoCbe9rRzM4lWhS+fpTnP2dmZWZWVlkZ33lLRGTgMzN+eOkpfOLUsdz5/DZueezt99yVtGp3Nd98ZD1nTi7oc2vDT162FMqBcufc8tjjZfRQFMzsFOAe4ELn3KGe3sg5t4TYeENpaam/6+qJyIAUCBi3LppBRkqQe1/eQXNbmO8umkEoePzvzpV1LXz+/jcZnpPGHZfPIZgk8xz1xLOi4Jzbb2Z7zGyqc24TcB7wrnlrzWws8DDwaefcZq+yiIj0hpnxHxedxJDUID99bivrK2r5/qUzmT4q96ivaW2P8MXfvUlNUysPff508pO026iD10PjNwAPmNkaYDZwm5ktNrPFsee/CQwD7jSz1WZW5nEeEZFjMjO++qGp/PwTc9l3pImLf/YKP3hyY48zq27aX8dH73qFFTsO84NLTzlm8UgW1h+nYMVTaWmpKytT7RAR79U0tnLr4xt4cGU5w7PTOH/6SD548ggWjB/KvS/v4PZntpCdHuLWS2ZwQYJfk2BmK51zpcfdT0VBROTYXt1axa9f3clLW6poagsTChjtEceFM0by3UUzGJaV5nfE4+ptUfByoFlEZEA4fVIBp08qoLktzMtbqnh5axWlJflcNLMo6aaxOB4VBRGRXkpPCfKBk0fwgZNH+B3FM8l3DbaIiHhGRUFERDqpKIiISCcVBRER6aSiICIinVQURESkk4qCiIh0UlEQEZFOSTfNhZlVAjXAkW5P5R5n2/Hud/y3AKg6gWg9Hb83z3fffqzH3bN23XYiueOZuet9P37W+nzo83Gs55Px8/FeMgNMds4df8Y+51zS3YAl73Xb8e53+W9Zf2XqzfPdtx/rcfesfc0dz8x+/6z1+dDnY6B9Pt5L5t4co+OWrN1Hj53AtuPd7+n1fc3Um+e7bz/W456y9iV3PDN3ve/Hz1qfj/dOn4/e30/0zL05BpCE3UdeM7My14uZBBNNMuZW5vhJxtzK7I9kbSl4aYnfAU5QMuZW5vhJxtzK7AO1FEREpJNaCiIi0mlAFwUz+5WZHTSzdSfw2nlmttbMtprZHdZlJQ0zu8HMNprZejP7Yf+m9ia3mX3bzPbG1sJebWYfTvTMXZ7/qpk5Myvov8Se/Zy/Y2ZrYj/jp81sVBJk/lHs87zGzP5kZnn9mdnD3B+L/Q5GzKzf+vH7kvUo73elmW2J3a7ssv2Yn3vfnMjpU8lyA84C5gLrTuC1K4D3AQb8Fbgwtv1c4BkgLfZ4eJLk/jbwr8n0s449NwZ4CtgFFCR6ZiCnyz43AncnQeYPAaHY/R8AP0iGzwdwEjAVeB4o9TtrLEdJt21Dge2x/+bH7ucf69/l921AtxSccy8Ch7tuM7OJZvakma00s5fMbFr315lZEdFf7tdd9P/eb4FFsac/D3zfOdcSO8bBJMntKQ8z/w9wE9Dvg19eZHbO1XbZNbO/c3uU+WnnXHts19eB0f2Z2cPcG5xzmxIl61GcD/zNOXfYOVcN/A24wM/f1eMZ0EXhKJYANzjn5gH/CtzZwz7FQHmXx+WxbQBTgDPNbLmZvWBm8z1N+46+5gb4UqyL4Fdmlu9d1E59ymxmHwH2Oufe8jpoF33+OZvZrWa2B/gk8E0Ps3boj89Gh2uIfmuNh/7M7bXeZO1JMbCny+OO/Iny7/o/BtUazWaWBZwOPNil+y7tPb5NiGhT8H3AfOCPZjYhVu090U+57wK+Q/Sb63eAHxP9A+CJvmY2syHAvxPt2oiLfvo545z7BvANM/s34EvAt/otZDf9lTn2Xt8A2oEH+ifdMY/Vb7m9dqysZnY18OXYtknAE2bWCuxwzl0S76z9YVAVBaItoxrn3OyuG80sCKyMPXyU6B/Qrk3o0cDe2P1y4OFYEVhhZhGi851UJnJu59yBLq/7JfAXD/NC3zNPBMYDb8V+EUcDb5rZAufc/gTN3N0DwBN4WBTop8xmdhXwD8B5Xn7B6aK/f9Ze6jErgHNuKbAUwMyeB65yzu3ssste4Jwuj0cTHXvYi///rp75Pajh9Q0oocuAEfAq8LHYfQNmHeV13QeBPhzbvhj4r9j9KUSbhpYEuYu67PPPwB8SPXO3fXbSzwPNHv2cJ3fZ5wZgWRJkvgB4Gyjs76zx+HzQzwPNJ5qVow807yA6yJwfuz+0t597P26+B/D0Hwe/B/YBbUS/4V9L9Nvnk8BbsV+Ebx7ltaXAOmAb8DPeudAvFbg/9tybwPuTJPd9wFpgDdFvYEWJnrnbPjvp/7OPvPg5PxTbvoboXDPFSZB5K9EvN6tjt349Y8rD3JfE3qsFOAA85WdWeigKse3XxH7GW4Gr38vn3o+brmgWEZFOg/HsIxEROQoVBRER6aSiICIinVQURESkk4qCiIh0UlGQAcHM6uN8vHvM7OR+eq+wRWdVXWdmjx1vllIzyzOzL/THsUW60ympMiCYWb1zLqsf3y/k3pkkzlNds5vZb4DNzrlbj7F/CfAX59yMeOSTwUUtBRmwzKzQzB4yszditzNi2xeY2WtmtsrMXjWzqbHtV5nZo2b2HPCsmZ1jZs+b2TKLrjfwQMec97HtpbH79bFJ8N4ys9fNbERs+8TY47Vm9t1etmZe450JAbPM7FkzezP2Hh+J7fN9YGKsdfGj2L5fi/0b15jZLf34Y5RBRkVBBrLbgf9xzs0HLgXuiW3fCJzpnJtDdBbT27q8Zi5wmXPu7NjjOcBXgJOBCcAZPRwnE3jdOTcLeBH4bJfj3+6cm8m7Z8TsUWzen/OIXnEO0Axc4pybS3Qdjx/HitLNwDbn3Gzn3NfM7EPAZGABMBuYZ2ZnHe94Ij0ZbBPiyeDyAeDkLjNb5sRmvMwFfmNmk4nOGpvS5TV/c851nUt/hXOuHMDMVhOdE+flbsdp5Z0JBlcCH4zdP4135sj/HfDfR8mZEXvvYmAD0Tn3ITonzm2xP/CR2PMjenj9h2K3VbHHWUSLxItHOZ7IUakoyEAWAN7nnGvuutHMfgb83Tl3Sax//vkuTzd0e4+WLvfD9Pw70+beGZw72j7H0uScmx2bLvwp4IvAHUTXYygE5jnn2sxsJ5Dew+sN+J5z7hfv8bgi/4e6j2Qge5roTKUAmFnH1Me5vDNN8VUeHv91ot1WAJcfb2fnXCPRJTy/amYhojkPxgrCucC42K51QHaXlz4FXBNrBWFmxWY2vJ/+DTLIqCjIQDHEzMq73P6F6B/Y0tjg69tEpz0H+CHwPTNbhbet5a8A/2Jma4guwHLkeC9wzq0iOsPqFUTXYyg1s7XAZ4iOheCcOwS8EjuF9UfOuaeJdk+9Ftt3Ge8uGiK9plNSRTwS6w5qcs45M7scuMI595HjvU7ETxpTEPHOPOBnsTOGavBw+VOR/qKWgoiIdNKYgoiIdFJREBGRTioKIiLSSUVBREQ6qSiIiEgnFQUREen0/wGUolAaM9IvSQAAAABJRU5ErkJggg==\n"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"Next we fit the last layer of our model for 1 epoch using 1 cycle policy. This approach was described to work best in the original paper. We use the learning rate of one order of magnitude lower than the one corresponding to the lowest loss on the previous plot as recommended. We use values for the cyclic momentum and other parameters as they were found to work well in guidelines of fastai course."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.fit_one_cycle(1, 1e-02, moms=(0.8, 0.7))","execution_count":283,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.HTML object>","text/html":"<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: left;\">\n      <th>epoch</th>\n      <th>train_loss</th>\n      <th>valid_loss</th>\n      <th>accuracy</th>\n      <th>time</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <td>0</td>\n      <td>5.306254</td>\n      <td>4.332844</td>\n      <td>0.221632</td>\n      <td>00:06</td>\n    </tr>\n  </tbody>\n</table>"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"Next we unfreeze the whole model and fine-tune it for 10 epochs."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.unfreeze()\nlearn.fit_one_cycle(10,1e-03, moms=(0.8, 0.7))","execution_count":284,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.HTML object>","text/html":"<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: left;\">\n      <th>epoch</th>\n      <th>train_loss</th>\n      <th>valid_loss</th>\n      <th>accuracy</th>\n      <th>time</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <td>0</td>\n      <td>4.429353</td>\n      <td>4.095856</td>\n      <td>0.249817</td>\n      <td>00:08</td>\n    </tr>\n    <tr>\n      <td>1</td>\n      <td>4.241044</td>\n      <td>3.932308</td>\n      <td>0.271489</td>\n      <td>00:07</td>\n    </tr>\n    <tr>\n      <td>2</td>\n      <td>4.079092</td>\n      <td>3.818842</td>\n      <td>0.280093</td>\n      <td>00:08</td>\n    </tr>\n    <tr>\n      <td>3</td>\n      <td>3.922644</td>\n      <td>3.750963</td>\n      <td>0.288332</td>\n      <td>00:07</td>\n    </tr>\n    <tr>\n      <td>4</td>\n      <td>3.779513</td>\n      <td>3.705878</td>\n      <td>0.294257</td>\n      <td>00:07</td>\n    </tr>\n    <tr>\n      <td>5</td>\n      <td>3.654723</td>\n      <td>3.679228</td>\n      <td>0.296429</td>\n      <td>00:07</td>\n    </tr>\n    <tr>\n      <td>6</td>\n      <td>3.550940</td>\n      <td>3.666553</td>\n      <td>0.296956</td>\n      <td>00:07</td>\n    </tr>\n    <tr>\n      <td>7</td>\n      <td>3.471123</td>\n      <td>3.665824</td>\n      <td>0.297463</td>\n      <td>00:07</td>\n    </tr>\n    <tr>\n      <td>8</td>\n      <td>3.415020</td>\n      <td>3.664080</td>\n      <td>0.297200</td>\n      <td>00:07</td>\n    </tr>\n    <tr>\n      <td>9</td>\n      <td>3.381201</td>\n      <td>3.664337</td>\n      <td>0.297220</td>\n      <td>00:08</td>\n    </tr>\n  </tbody>\n</table>"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"In the last few epochs we see that the accuracy on the validation set stagnates which means that training further will only lead to overfitting.\n\nNext we ask our trained language model to finish a phrase."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.predict(\"My experience was\", n_words=10)","execution_count":285,"outputs":[{"output_type":"execute_result","execution_count":285,"data":{"text/plain":"'My experience was ok with Cancelled Flightled flight and a few'"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"We see that our model follows basic rules of grammar. \n\nFinally we save our trained model to use it as an encoder for the classifier."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.save_encoder('ft_enc')","execution_count":286,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2. Classifier"},{"metadata":{},"cell_type":"markdown","source":"Now it's time to implement the final stage of ULMFit - Classifier. Fot this we create an appropriate learner and import our fine-tuned language model as an encoder."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5)\nlearn.load_encoder('ft_enc')","execution_count":287,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Again, let's explore different values for the learning rate."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.lr_find()\nlearn.recorder.plot()","execution_count":288,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.HTML object>","text/html":""},"metadata":{}},{"output_type":"stream","text":"LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"<Figure size 432x288 with 1 Axes>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAYwAAAEKCAYAAAAB0GKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd8VGXa//HPlU4SEohJ6L1IEWkBQawrYll7W3vbldVdHnXr73F7c9XVXZ917e5asK666uKKYlmxIKgBpIP0ElogpJBJMin374+ZYMCQDCQzZyb5vl+veTFzzplzrpvJ5Mpdzn2bcw4REZHmxHkdgIiIxAYlDBERCYkShoiIhEQJQ0REQqKEISIiIVHCEBGRkChhiIhISJQwREQkJEoYIiISkgSvA2hN2dnZrm/fvl6HISISM+bPn7/LOZcTyrFtKmH07duX/Px8r8MQEYkZZrYx1GPVJCUiIiFRwhARkZAoYYiISEiUMEREJCRKGCIiEhIlDBERCYkShoiIhEQJQ0Qkhr2zfAcPf7A2ItdSwhARiWHvLt/BE3PWR+RaShgiIjGs3F9DWnJkJu1QwhARiWE+fy1pSUoYIiLSjL1VNaQmxUfkWkoYIiIxzOevIV1NUiIi0hxfVS2pShgiItKcvVU1pCerSUpERJrh89eSGuud3mb2uJntNLOlB9k/xMzmmlmVmf34gH0bzGyJmX1hZloRSUSkEc65wLDaNtDp/SRwehP7i4CbgXsOsv9k59wo51xeawcmItIWVFTX4hyxfx+Gc+5DAknhYPt3Ouc+B6rDFYOISFtWXlUL0O47vR3wtpnNN7OpXgcjIhKNyqtqACLWJBWZtHTojnPOFZhZLvCOma0M1li+JphQpgL07t07kjGKiHiq3B9MGO25huGcKwj+uxN4FRjfxLGPOufynHN5OTk5kQpRRMRz9U1S7XZqEDNLM7OO9c+BKUCjI61ERNqzr2oYMd4kZWbPAycB2Wa2Bfg1kAjgnHvYzLoC+UAGUGdmtwLDgGzgVTOrj+8559xb4YpTRCRW+eprGBFqkgrbVZxzlzWzfzvQs5FdpcDIsAQlItKG1Hd6a/JBERFpUn2TlCYfFBGRJvn8wfsw2munt4iIhGZvVQ2J8UZSQmR+lSthiIjEKF9V5JZnBSUMEZGYtbcqcsuzghKGiEjM8vlrInYPBihhiIjErPIIroUBShgiIjGrvEo1DBERCUF5VY36MEREpHk+f61GSYmISPPKq2oiNi0IKGGIiMSscn9NxKYFASUMEZGYVFNbR2V1nUZJiYhI03zV9VObq0lKRESaEOm1MEAJ45BV19bxzvIdVAazu4iIF/ZGeC0MUMI4ZLe/sYIbpufzrUfmsq2kwutwRKSd8kV4LQxQwjgkbyzexpOfbODkI3NYs3MvZ//tYz5bX+R1WCLSDpVXRXYtDFDCCNm6wr38v38tZnTvTjxyVR7/njaJjJRELn9sHtPnbsA553WIItKO1C/Pqk7vKFNZXcv3nl1AQrxx/+VjSEqIY2BuR16bNokTB+fwq38v49tP5bO9pNLrUEWknahfnlWd3lGkts7xy9eWsnJ7Gfd+axQ9OnXYty8jJZHHrs7jV2cN45O1uzj13g94KX+zahsiEnb1TVKaSyoKlFVW84+P13PSPe/z0vwtTDt5ICcfmfu14+LijOuP68dbt5zA0K4Z/OTlxXznqXxKK6s9iFpE2gufP/JNUpFLTVHsyTnr8VXXUumvpbKmjqJyP28t3c7eqhrG9e3MbWcM5YyjujZ5jr7ZabwwdQJPfrKBP85cwbcemcdT140jNyMlQqUQkfbEi05vJQzgT7NW4fMH/vNTEuNITUpgyrAuXDepHyN6ZoZ8nvraxsDcdG58Zj4XPPQJ068fT/+c9MOKa3tJJR+uLmRkz04c2bXjYZ1DRNqmcn8NKYlxxMdZxK6phAHMve0UkhPiSE6Iw6zl//knDM7h+RsmcP2Tn3PRw3N54tpxjOzVKaT3bi2u4PVFW3lr2XYWbioGIDHeuHXyYL57Qn8S4tWKKCKBUVKRvAcD1IcBQGaHRFIS41slWdQb2asTL990LGnJ8Vz+2DyWFpQ0+54Zi7Yy+S8fcMebK6murePHUwbz2vcnMWV4V+6etYqLH5nL+l3lrRZjJM1bt5vLHp3HawsLNChApBUEpjaPbMKwtvTlzcvLc/n5+V6HsZ/tJZVc8OAcquscr9x0LL2yUr92TFVNLbe/sYLpczeS16cz91w8kr7ZafsdM2PRVn7x6hKqax1TT+jPdZP60ik1KVLFOGx1dY4HZ6/hL+98SWJ8HFU1dZw4OIc/nHdUo/8XIhKaG6bns7nIx1u3ntCi85jZfOdcXijHqoYRZl0zU3jq+vFUVddyzROfsafcv9/+Tbt9XPLIPKbP3cgNx/fj+akTvpYsAM4Z2Z23f3AiJwzO5q/vrWbSnf/ljjdXUFhWFamiHLLde6u45onPuOftL/nm0d357OeT+c3Zw8jfUMSUez/kodlrWVpQsm9OHBEJnS/Ca2GAahgR89n6Iq78x6cc1T2Dv18zjg++3MkrCwr4eM0u0pMSuPviozn9qG4hnWvl9lIeeH8t/1m8lcS4OLLTk+iQFE+HpHgyOyRy7bH9mDw095Cb2Op/Fhp7X1VNLQs2FpPTMYmBuU13wK/eUcYrCwt4KX8zpZU1/PrsYVw+vve+824truCXry3lvZU7970np2Myw7tn8NPThjCse8YhxS3SHp37wBwyOyQy/frxLTrPodQwlDAi6M0l2/jecwswoM5Bz84duGBMTy4d14vuDW4IDNW6wr388/PN7C73UxEcFrymcC8bd/s4flA2vz57WKO/3HftreLd5Tt4a9l2lhaUUFVdR1VtHdW1daQlJTCsewYjemQyokcmPn8t76/ayZw1u/aNJDtlSC43njSAcX2zgMBCLiu2lTF33S5mLNrK0oJS4uOM4wdl85PTjmR496+PNHPOsWbnXtbs3Mv63eWsLyzn/VU7KfZVc9NJA5j2jYEkJwTGl1fV1PLJ2t0s31pKaUU1pZXVlFbU0LNzB64/rh9dNHRZ2qFT//IBg7qk8+AVY1t0HiWMKPav+VvI37iHc0d1Z3zfLOJaeUhcdW0dT8/dyL3vfkmFv5aL83qS2SGJCn8N5f5aNhX5yN9QRJ2DXlkdOLZ/NqnJ8SQlxJEUH0exr5olBSWs2FZKVU0dAD06deDkITmcMCiHldvLePKTDRSV+xnbpzPpyQnM37hnX7PSUT0yOH90T84Z2Z2cjsmHFPuecj+/f2M5rywoYGBuOtcc25d563bzwarCfedPTogjo0MiHVMS2LjbR3yccdm4Xtx40gC6ZXagqqaWjbt9rCssp6TCT1VNHVXVdfhr6+iakcKw7hkMzE0nUaPNJMZNuvO/TBxwBPdcPLJF51HCEHbvreKet1fxYv4W4s3okBRPalI8R6Qn8Y0jczntqK4M65Zx0Garmto61hTuJSEujgE5afsdV+Gv5cX8zTz1yQbi44zx/bL2PbplHnpN6UCzV+3k568upaC4guz0JE4d1oUpw7pyTP+s/UaFbNrt48HZa3h5/hbizOiamcKWPT7qmvmRToqPY1CXdK6b1I8Lx/Ro1dFxIpEy8rdvc96o7vz23KNadB4lDNmnrs61ei0mEnz+Gjbu9jG4S8dmb0zaXOTj7x+tY1e5nwHZafTPSad/ThpHpCeTnBBHSmI8CXHG5iIfy7eVsnxbKXPW7GJpQSnHDjiCP54/otGBBiLRbNDPZ3LD8f356elDWnSeQ0kYunGvjYvFZAGB6Q6Gdgut87tXVmpIf2UN6tKRQV06cu6oHtTVOZ77bBN3vbmS0/7vQ24+ZRBTT+ivpiqJCVU1tVTXuojOVAsaVivtVFycceWEPrz7oxM5ZWgud89axTn3z2HxlmKvQxNplm/fPFKRm3gQlDCkneuSkcKDV4zlsavzKCqv4rwH5nDHzBVU+LVmu0QvL9bCADVJiQBw6rAuHNM/iztmruSRD9fx+qKtDO2WQVZaEllpSfQ+IpWLx/YiKUF/Y4n3vFgLA5QwRPbJSEnkjgtGcM7I7jzy4Vq2lVSyfFspu8v9+GvqeOGzzdx32Wj6qYNcPFbuwVoYEMaEYWaPA2cBO51zX+uRNLMhwBPAGODnzrl7Guw7HfgrEA/83Tl3Z7jiFDnQxAFHMHHAEfteO+d4e/kOfvryYs667yN+f95RXDCmp4cRSntX34fRljq9nwROb2J/EXAzcE/DjWYWDzwAnAEMAy4zs2FhilGkWWbGacO78uYtxzO8RyY/fHERt7ywkJ1lWsNdvFF/I2ub6fR2zn1IICkcbP9O59znwIFrmY4H1jjn1jnn/MALwLnhilMkVN07deD5Gybwg8mDmblkGyfdPZv73lutDnKJuPrlWbUeBvQANjd4vSW4rVFmNtXM8s0sv7CwMOzBSfsWH2fcMnkQ7/zgRE4YlMNf3vmSk++ZzRuLt3kdmrQj5ftqGEoYh8Q596hzLs85l5eTk+N1ONJO9M1O4+GrxvLidyeS0zGZac8vYM6aXV6HJe1EebBWqxoGFAC9GrzuGdwmEnXG98vin9+dwICcdG554Qv1a0hE+KpqMIOUxMj+Co/GhPE5MMjM+plZEnApMMPjmEQOKjUpgQevGMPeqmpuef4Lapub/VCkhfZW1ZKWlBDxiTPDljDM7HlgLnCkmW0xs2+b2Y1mdmNwf1cz2wL8EPhF8JgM51wNMA2YBawAXnTOLQtXnCKtYXCXjvz+3KOYu24397232utwpI3z+Wsifg8GhPE+DOfcZc3s306guamxfTOBmeGISyRcLs7rFUgY/13N+H5ZTBqY7XVI0kaV+2sjfpc3RGeTlEjM+sN5RzEgJ52p0/N5c4lGTkl4lFfVkOpBDUMJQ6QVpSYl8PS3xzOoS0duenYBd7y5gpraOq/DkjamvKpGNQyRtqBbZgf++d0JXH5Mbx75YB1XP/4ZW/b4aEuLlYm3yv01EZ8WBDT5oEhYJCfE88fzRzCqVyd+8dpSjrvrfTJSEgKLOOWm861xvRjdu7PXYUqM8lXVkpathCHSplyS14uxfTrz0ZeFrCncy+ode5m5ZBszFm3l6W8fw9g+Shpy6Mr9NaRFeB4pUMIQCbsBOekMyEnf93pnWSXfemQe1z7xGS9MncDw7pkeRiexqLyqNuLTgoD6MEQiLrdjCs985xg6Jidw9T8+Y83OvV6HJDHEOUe5v4Z0jZISaR96dOrAszdMwMy48u+fsmFXudchSYyorK7DOUj1oNNbCUPEI/2y03jmO+OprKnl7L99rBlvJST1a2F40YehhCHioSFdM3h92nEMyE3n+88t4GevLqGyWutryMH59i3PqhqGSLvTKyuVl26cyI0nDuC5Tzdxzv0fs2m3z+uwJErt9WgtDFDCEIkKifFx/O8ZQ5h+/Xh2lFZxw/T8fX9JijTk82gtDFDCEIkqJwzO4W+XjebLnWXc9soS3R0uX7NvtT2NkhKREwbn8KNTB/PvL7by1CcbvA5Hokx5VaCGobmkRASA7500kMlDu/CHN1bw+YYir8ORKFK+r9NbNQwRAeLijD9fMpKenTvwvWcXaOlX2ad837Ba1TBEJCizQyIPXzWWkopqfv1vLTopAfWd3urDEJH9DOmawa2TB/Hm0u28tVQ39gmUVlSTnBBHcoIShogc4Ibj+zO8ewa//PcySnzVXocjHtvj89MpNdGTaythiES5xPg47rrwaIrK/dw+c7nX4YjHin3VdE5N8uTaShgiMeCoHplMPaE/L+Zv4ePVu7wORzxU7Ksms4NqGCLShFtOGUT/7DRue3XxvukhpP0prvCrhiEiTUtJjOeui45ma3El1z/5ORV+TVLYHu3xVasPQ0SaN65vFvd+axT5G4qY+nS+ZrZtZ5xzlPiq6aQahoiE4pyR3fnTRSP5aPUuvv/sAvw1dV6HJBHi89fir61TDUNEQnfR2J784byjeG/lTm7950Jq6zRJYXtQXBEYVt05mhOGmQ0ws+Tg85PM7GYz6xTe0ESkKVdO6MMvvjmUmUu28+e3V3kdjkTAnnI/AJkdortJ6l9ArZkNBB4FegHPhS0qEQnJd47vz2Xje/Pg7LXMXKI7wdu6klioYQB1zrka4Hzgb865nwDdwheWiITqN+cMY3TvTvz4pUV8uaPM63AkjPb4AjWMaO/0rjazy4BrgP8Et3mT4kRkP8kJ8Tx85VjSkhOYOj1f04e0YcW+2KhhXAdMBG53zq03s37A0+ELS0QORZeMFB66Ygxb9lRw6z8XUqdO8DapOFjDyIzmhOGcW+6cu9k597yZdQY6OufuCnNsInII8vpm8auzh/H+qkKe0Ep9bVKxr5rUpHhPZqqF0EdJzTazDDPLAhYAj5nZX8Ibmogcqqsm9GHy0FzuemslK7eXeh2OtLI9vmo6eTSPFITeJJXpnCsFLgCmO+eOASaHLywRORxmxp0XHk1GSgK3vvCF7gRvY0oq/J51eEPoCSPBzLoBl/BVp7eIRKHs9GTuvmgkK7eXcc+s/e/PqKqpVRKJYV7OIwUQ6qKwvwNmAXOcc5+bWX9gdfjCEpGWOHlILldN6MPfP17P0G4ZlFRU8+HqQuat201CXBxXTezDt4/rR3Z6stehyiEo9vkZ0jXDs+uHlDCccy8BLzV4vQ64MFxBiUjL/ezMocxZu4sfvbQIgP7ZaXwrrxe7yv08/MFanpiznsvH9+HGk/qT2zHF42glFMW+as9GSEGICcPMegJ/AyYFN30E3OKc29LEex4HzgJ2OueOamS/AX8FzgR8wLXOuQXBfbXAkuChm5xz54RWHBGp1yEpnieuHcen64uY2P8IemWl7tu3ZudeHpq9lqfmbuDt5duZMe04stK8axuX5jnnKK6o9uweDAi9D+MJYAbQPfh4PbitKU8Cpzex/wxgUPAxFXiowb4K59yo4EPJQuQw9TkijUvyeu2XLAAG5qbz50tG8vKNE9lZVsW05xZQU6tZb6NZWVUNtXWOTh7NIwWhJ4wc59wTzrma4ONJIKepNzjnPgSKmjjkXAIjrpxzbh7QKdixLiIRMrp3Z/54/gg+WbubP85c6XU40oT6O/i97PQONWHsNrMrzSw++LgS2N3Ca/cANjd4vSW4DSDFzPLNbJ6ZndfC64hIEy4a25PrJvXl8Tnr+df8g7Yyi8e8nkcKQk8Y1xMYUrsd2AZcBFwbppgA+jjn8oDLgf8zswEHO9DMpgaTS35hYWEYQxJpu3525lAm9j+C215dwvyNe7wORxrh9TxSEPrUIBudc+c453Kcc7nOufNo+SipAgLTpNfrGdyGc67+33XAbGB0E7E96pzLc87l5eQ02UomIgeRGB/HA1eMIbdjMpc+Opf/e/dLreQXZb6qYUR5wjiIH7bw2jOAqy1gAlDinNtmZp0bLNaUTWBk1vIWXktEmpGVlsSMacdx5ohu/N+7qznn/o9ZvKXY67AkqH4tjFhokmqMNbnT7HlgLnCkmW0xs2+b2Y1mdmPwkJnAOmAN8BjwveD2oUC+mS0C3gfudM4pYYhEQFZaEn+9dDR/vzqPPT4/5z0wh9cWFngdlgB7ygMJI9PDuaRCvdO7MU3On+ycu6yZ/Q74fiPbPwFGtCAuEWmhycO6MK5fFlf941P+/M4qzh7Znfi4Jv9GlDArrvDTMTmBxPiW/J3fMk1e2czKzKy0kUcZgfsxRKSNyuyQyI0nDmBzUQXvrdjhdTjtntd3eUMzCcM519E5l9HIo6NzriW1ExGJAVOGdaFHpw48MWeD16G0e8U+P5097L+AlvVhiEgblxAfmKhw7rrdWl/DY17PVAtKGCLSjEvH9SIlMY4nVcvwVElFtacjpEAJQ0Sa0Sk1ifNH9+TVhQUUlfu9Dqfd2uPze7raHihhiEgIrpvUl6qaOp7/bJPXobRLdXWOEo9nqgUlDBEJweAuHTluYDZPz91ItWa1jbjSymqcg0w1SYlILLhuUl+2l1bqRj4PRMM8UqCEISIhOvnIXEb37sTvXl/Opt0+r8NpV6JhHilQwhCREMXFGfddOhozmPb8Aqpqar0Oqd0ojoJ5pEAJQ0QOQa+sVO6+eCSLt5Rw55tacClS9i2epFFSIhJLThvelWuP7csTczYwa9l2r8NpF+qbpHSnt4jEnNvOHMKIHpn85KVFFBRXeB1Om1fsq8YMMlTDEJFYk5wQz/2Xj6aiupZ/fLTe63DavGKfn4yURM9nDFbCEJHD0ueINE4b3pWX52+mwq8O8HAqrvB+HilQwhCRFrhqQh9KK2t4fdFWr0Np0wITD3rbfwFKGCLSAuP7ZTG4SzpPz9vodShtWkkUzCMFShgi0gJmxpUT+rCkoIRFm7X+d7js8Xk/jxQoYYhIC50/ugepSfGqZYRRsc+vJikRiX0dUxI5b3QPXl+0lWKfpj9vbTW1dZRW1qjTW0TahiuP6UNVTR0vz9/idShtTmllDeD9Xd6ghCEirWBY9wzy+nTmmXkbqatzXofTpuy7yztNTVIi0kZcNbEPG3b7eH2xhti2pvqpzTNVwxCRtuKbI7oxuncnfvnaUraVaLqQ1lIcJfNIgRKGiLSShPg47r1kFNW1jh+/tEhNU61kT/1Mter0FpG2pG92Gr88axhz1uzmqbkbvA6nTVi/ay8JcUaXjBSvQ1HCEJHWddn4XpwyJJc731zJ6h1lXocT85ZtLWVgbjopifFeh6KEISKty8y488KjSUtO4NZ/fqGV+Vpo2dZShnXP8DoMQAlDRMIgp2Myd114NMu2lvLjlxarP+Mw7SyrpLCsiuHdM70OBVDCEJEwOXVYF356+pG8vmgrd7+9yutwYtLyraUADI+SGkaC1wGISNt104kD2LKngodmr6VX51QuP6a31yHFlGXBhBEtTVJKGCISNmbG784ZzrbiCn7576V0y0zh5CG5XocVM5ZvLaV3VioZKd4PqQU1SYlImCXEx3H/5WMY2q0j055bQGFZldchxYxlW0sY1i06aheghCEiEZCWnMB9l47GV13Lk59oDfBQlFVWs2G3L2r6L0AJQ0QipH9OOmcc1ZXpczdSVlntdThRb+X2wD0sw3soYYhIO3TTiQMpq6zh2U83eR1K1FtWUAIQNUNqQQlDRCJoRM9Mjh+UzT8+Xk9ltW7oa8qyraUckZZEbsdkr0PZRwlDRCLqphMHUFhWxb8WaLGlptTf4W1mXoeyT1gThpk9bmY7zWzpQfabmd1nZmvMbLGZjWmw7xozWx18XBPOOEUkciYOOIKRPTN55IN11NTWeR1OVPLX1LF6Z1lUNUdB+GsYTwKnN7H/DGBQ8DEVeAjAzLKAXwPHAOOBX5tZ57BGKiIRYWbcdNJANhX5mLl0u9fhRKUvd5RRXeuiaoQUhDlhOOc+BIqaOORcYLoLmAd0MrNuwGnAO865IufcHuAdmk48IhJDpgzrQv+cNB58f43mmWrE8m3RNSVIPa/7MHoAmxu83hLcdrDtItIGxMUZ//ONgazcXqYlXRuxfGspqUnx9D0izetQ9uN1wmgxM5tqZvlmll9YWOh1OCISonNH9mBYtwz+9NYqjZg6wLKtJQztlkFcXPR0eIP3CaMA6NXgdc/gtoNt/xrn3KPOuTznXF5OTk7YAhWR1hUXZ/zszKEUFFfw9NyNXocTNerqHMu3lkZdcxR4nzBmAFcHR0tNAEqcc9uAWcAUM+sc7OyeEtwmIm3IcYOyOWFwDn/772qKfX6vw4kKG4t8lPtr21/CMLPngbnAkWa2xcy+bWY3mtmNwUNmAuuANcBjwPcAnHNFwO+Bz4OP3wW3iUgbc9sZQyirquGB99d4HUpUWBns8B4aRZMO1gvr9ObOucua2e+A7x9k3+PA4+GIS0Six9BuGVw4pidPfbKRqyf2pVdWqtcheWrdrnIgMPdWtPG6SUpEhB9NGYwZ/GmWVuZbv6uc3I7JpCdH33JFShgi4rlumR347okDeH3RVmYta983863fVU6/7OgaTltPCUNEosK0kwcyvHsGt72yhJ1llV6H45n1u8rpn6OEISJyUEkJcfz10lGUV9Xw05cXE+jibF9KfNUUlftVwxARac7A3I7cdsYQZq8q5Jl2uGbG+t2BDu9+2dHX4Q1KGCISZa6e2JfjB2Vz+xvLWVu41+twImr9rkB5VcMQEQlBXJxxz8UjSUmM5zczlnkdTkStLywnzqB3lA4tVsIQkajTJSOFK47pzSdrd1Piaz/rf6/bVU6vrFSSEqLzV3N0RiUi7d4pQ7tQW+eY/eVOr0OJmGgeUgtKGCISpUb17ER2ehLvLN/hdSj78flruOmZ+Vzz+GeUVrZe7cc5p4QhInI44uKMbwzJ5YMvC/HXRMdSrkXlfi577FNmLdvOnDW7uPSReRSWVbXKuXeWVeHz19JfCUNE5NBNHtqFssoaPt/g/dyjm4t8XPTQJ6zcVsrDV47lH9eOY/2uci55ZC5b9vhafP51hdE9pBaUMEQkih03KJvkhDjeXeFts9Sq7WVc+NAn7NpbxTPfOYYpw7ty4uAcnvnOeHbvreKih+ayZmdZi66xPjjpYL8ovcsblDBEJIqlJiUwaWA2767Y8bU7v19ZsIVV21v2SzoUJb5qrn/yc8zg5ZuOZVzfrH37xvbJ4p/fnUh1bR0/e3Vpi66zftdekhPi6JaR0tKQw0YJQ0Si2uShXdhcVMHqnV/dxPf6oq388MVFTH06P6zLuzrn+H//WsyO0koeunIsg7t0/NoxQ7tlcPXEvny+oYjtJYc/B1Z9h3e0LcvakBKGiES1U4bmAuwbLbW5yMfPXllC76xUNu728eiH68J27afnbeStZdv5yWlHMqZ354Med9bIbjgHM5dsO+xrrYvyEVKghCEiUa5LRgpH98zkvRU7qKmt45YXFgLw7HeO4ZsjuvHA+2vYXNTyTucDLdtawh/+s4KTjszhhuP7N3nsgJx0hnbL4D+Ltx7WtWpq69i026eEISLSUpOHdmHh5mJ+NWMZCzYV88cLRtArK5VfnDWU+Djjt68vb9Xr7a2qYdpzC+mclsifLx4ZUjPRWUd3Y8GmYgqKKw75elv2VFBT55QwRERa6pShuThKNlf0AAAOZ0lEQVQHz326iUvyenL2yO5AYOGlm08ZxLsrdvBeK42kcs7xs1eWsHF3OX+9dDRHpCeH9L5vjugGwMzFh94stX7fsqxKGCIiLTKsWwY9O3egf3Yavzln+H77rp/UjwE5afzm9WWt0gH+zLyNzFi0lR9MHsyE/keE/L6+2Wkc1SOD/xxGP0b9Ot7RfA8GKGGISAwwM56/YQL//O5EUpP2X+s6KSGO3517FJuLKpg+d0OLrvPF5mJ+95/lnHRkDt8/eeAhv/+so7uzaHPxIfeprN+1l8wOiXROTTzka0aSEoaIxIReWankdGy8eWjSwGyO6ZfF0/M2Uld3eCv17Sn38/1nF5DbMYV7Lxl1WMNb65ul3jjEWkb9kFqz6B1SC0oYItJGXDGhD5uLKvhwdeEhv7euzvHDF79gZ1klD1wxhs5pSYcVQ6+sVEb26nTIo6XWF5ZH9RxS9ZQwRKRNOH14V45IS+LZw1jadfrcDby/qpBfnjWMUb06tSiOs4/uxtKCUjYE+yWaU+GvZWtJZdSPkAIlDBFpI5IS4rhkXC/eW7GDrYcwtLWo3M9f3vmS4wZmc9WEPi2O48xgs9QrC7Y0e6xzbl+/S/+c6O7wBiUMEWlDLh/fGwe88PnmkN/zl3dWUe6v5VdnD2uVPoTunTpw5oiuPPTBWhZu2nPQ48oqq5n23ELueHMlk4fm7rujPZopYYhIm9ErK5WTBufwwmebqK5tfg2NldtLee7TTVw1oU+j80QdrjvOP5ouGSlMe24he8r9jV733Pvn8Nay7fzvGUN49Ko8UhLjW+364aKEISJtyhXH9GFnWVWzN/I55/jtjOVkdEjk1smDWjWGzNREHrxiDIVlVfzwxS/2jdzy19TxwPtrOPf+OZRV1fDcd47hxhMHRPWEgw0pYYhIm3LykFy6Z6bwzLymO79nLdvB3HW7+dGpg+mUenijoppydM9O/PKsoby/qpBHPlzH3LW7OfO+j7h71iq+MSSXN24+jmMO4cbAaJDQ/CEiIrEjPs64bHxv/vzOlwddI7uyupbbZy7nyC4duWx877DFcuWEPny6vog/zVqJc9ArqwNPXDuOk4dEf39FY1TDEJE251vjehEfZ7yU33jn96sLC9hcVMEvzxpGQnz4fg2aGXdeeDQnDc5h2skDefvWE2M2WYBqGCLSBuVmpHD8oGxeW1jAj6cc+bU+gpfnb2FQbjqTBoa/SSg9OYEnrhsf9utEgmoYItImXTCmJ1tLKpm3fvd+29fvKmf+xj1cOLZn1E/FEW2UMESkTZoyrAvpyQm8uqBgv+2vLNhCnMH5o3t4FFnsUsIQkTYpJTGeM0d0ZeaSbVT4A9Oe19U5XllQwHGDcuiSkeJxhLFHCUNE2qzzR/ek3F/L28u3AzBv3W4Kiiu4aGxPjyOLTUoYItJmHdMvix6dOvBKsFnq5QVb6JiSwJRhXTyOLDYpYYhImxUXZ5w3ujsfrS5kw65y3lyynbOO7hYT03BEo7AmDDM73cxWmdkaM/vfRvb3MbP3zGyxmc02s54N9tWa2RfBx4xwxikibdf5o3tS5+DmFxZSUV2r5qgWCNt9GGYWDzwAnApsAT43sxnOueUNDrsHmO6ce8rMvgHcAVwV3FfhnBsVrvhEpH0YmJvOyJ6ZLNpSQr/sNMb07ux1SDErnDWM8cAa59w655wfeAE494BjhgH/DT5/v5H9IiItVj+E9oLRPXTvRQuEM2H0ABrel78luK2hRcAFwefnAx3NrP7WyxQzyzezeWZ2XhjjFJE27sKxPbluUl+uaIUFktozr6cG+TFwv5ldC3wIFAC1wX19nHMFZtYf+K+ZLXHOrT3wBGY2FZgK0Lt3+CYRE5HY1TElkV+fPdzrMGJeOGsYBUCvBq97Brft45zb6py7wDk3Gvh5cFtx8N+C4L/rgNnA6MYu4px71DmX55zLy8nJafVCiIhIQDgTxufAIDPrZ2ZJwKXAfqOdzCzbzOpjuA14PLi9s5kl1x8DTAIadpaLiEiEhS1hOOdqgGnALGAF8KJzbpmZ/c7MzgkedhKwysy+BLoAtwe3DwXyzWwRgc7wOw8YXSUiIhFmzjmvY2g1eXl5Lj8/3+swRERihpnNd87lhXKs7vQWEZGQKGGIiEhIlDBERCQkShgiIhKSNtXpbWaFwMYDNmcCJc1sa/i6secNt2UDuw4jvMbiCPWY1izD4cbfVHyhHNNUvE29bqws4SxDOD+Dhs9jtQz6LjQdXyjHNFeGSH8X+jjnQruJzTnXph/Ao81ta/i6secHbMtvrThCPaY1y3C48bd2GUJ9fZCyhK0M4fwM2kIZ9F0Ifxmi5bvQ2KM9NEm9HsK215t53tg5WiOOUI9pi2UI9fXBynW4mjtHOD+DUK4fCi/LEG0/R41ti/UyRMt34WvaVJNUJJhZvgtxzHI0ivX4QWWIFrFehliPHyJfhvZQw2htj3odQAvFevygMkSLWC9DrMcPES6DahgiIhIS1TBERCQk7TZhmNnjZrbTzJYexnvHmtmS4Frl91mDJbzM7H/MbKWZLTOzP7Vu1F+Lo9XLYGa/MbOCBuupn9n6ke8XR1g+h+D+H5mZC854HDZh+hx+H1zr/gsze9vMurd+5PtiCEf8dwe/B4vN7FUz69T6ke8XRzjKcHHwe1xnZmHrJ2hJ7Ac53zVmtjr4uKbB9ia/LyE53CFZsf4ATgDGAEsP472fARMAA94EzghuPxl4F0gOvs6NwTL8BvhxLH8OwX29CMyUvBHIjrUyABkNjrkZeDjG4p8CJASf3wXcFYOfwVDgSALr8eRFW+zBuPoesC0LWBf8t3Pweeemynkoj3Zbw3DOfQgUNdxmZgPM7C0zm29mH5nZkAPfZ2bdCHyZ57nApzAdqF9C9iYCU7FXBa+xMwbLEFFhLMO9wE+BsHfShaMMzrnSBoemEcZyhCn+t11giQOAeQQWUAubMJVhhXNuVTjjbknsB3Ea8I5zrsg5twd4Bzi9tb7z7TZhHMSjwP8458YSWD72wUaO6UFgffJ6DdcqHwwcb2afmtkHZjYurNE2rqVlAJgWbEp43Mw6hy/Ug2pRGczsXKDAObco3IE2ocWfg5ndbmabgSuAX4Ux1sa0xs9RvesJ/EUbaa1ZhkgLJfbG9AA2N3hdX55WKafXa3pHDTNLB44FXmrQtJd8iKdJIFAVnACMA140s/7BjB52rVSGh4DfE/iL9vfAnwl84SOipWUws1TgZwSaRDzRSp8DzrmfAz83s9sILEb261YLsgmtFX/wXD8HaoBnWye6kK/bamWItKZiN7PrgFuC2wYCM83MD6x3zp0f7tiUML4SBxQ750Y13Ghm8cD84MsZBH6hNqxeN1yrfAvwSjBBfGZmdQTmeikMZ+ANtLgMzrkdDd73GPCfcAbciJaWYQDQD1gU/LL1BBaY2Xjn3PYwx16vNX6WGnoWmEmEEgatFL+ZXQucBZwSqT+aGmjtzyCSGo0dwDn3BPAEgJnNBq51zm1ocEgBgZVM6/Uk0NdRQGuUM1wdObHwAPrSoKMJ+AS4OPjcgJEHed+BnUdnBrffCPwu+HwwgaqhxVgZujU45gfAC7H2ORxwzAbC3Okdps9hUINj/gd4OcbiPx1YDuSE+/8+3D9HhLnT+3Bj5+Cd3usJdHh3Dj7PCqWcIcUZqQ8z2h7A88A2oJpAzeDbBP4yfQtYFPxh/9VB3psHLAXWAvfz1Q2QScAzwX0LgG/EYBmeBpYAiwn8BdYt1spwwDEbCP8oqXB8Dv8Kbl9MYM6fHjEW/xoCfzB9EXyEbZRXGMtwfvBcVcAOYFY0xU4jCSO4/frg//8a4LpD+b4099Cd3iIiEhKNkhIRkZAoYYiISEiUMEREJCRKGCIiEhIlDBERCYkShrRpZrY3wtf7u5kNa6Vz1VpgttqlZvZ6czO+mlknM/tea1xbpDEaVittmpntdc6lt+L5EtxXk+qFVcPYzewp4Evn3O1NHN8X+I9z7qhIxCftj2oY0u6YWY6Z/cvMPg8+JgW3jzezuWa20Mw+MbMjg9uvNbMZZvZf4D0zO8nMZpvZyxZY8+HZ+rUFgtvzgs/3BicQXGRm88ysS3D7gODrJWb2hxBrQXP5anLFdDN7z8wWBM9xbvCYO4EBwVrJ3cFjfxIs42Iz+20r/jdKO6SEIe3RX4F7nXPjgAuBvwe3rwSOd86NJjA77B8bvGcMcJFz7sTg69HArcAwoD8wqZHrpAHznHMjgQ+BGxpc/6/OuRHsP4Noo4LzH51C4M57gErgfOfcGAJrsPw5mLD+F1jrnBvlnPuJmU0BBgHjgVHAWDM7obnriRyMJh+U9mgyMKzBTKAZwRlCM4GnzGwQgdl6Exu85x3nXMM1Cz5zzm0BMLMvCMwF9PEB1/Hz1eSN84FTg88n8tVaBM8B9xwkzg7Bc/cAVhBY2wACcwH9MfjLvy64v0sj758SfCwMvk4nkEA+PMj1RJqkhCHtURwwwTlX2XCjmd0PvO+cOz/YHzC7we7yA85R1eB5LY1/l6rdV52EBzumKRXOuVHBKdtnAd8H7iOwPkYOMNY5V21mG4CURt5vwB3OuUcO8boijVKTlLRHbxOYARYAM6ufRjqTr6Z8vjaM159HoCkM4NLmDnbO+Qgs0/ojM0sgEOfOYLI4GegTPLQM6NjgrbOA64O1J8ysh5nltlIZpB1SwpC2LtXMtjR4/JDAL9+8YEfwcgLT0gP8CbjDzBYS3tr3rcAPzWwxgUVwSpp7g3NuIYGZay8jsD5GnpktAa4m0PeCc243MCc4DPdu59zbBJq85gaPfZn9E4rIIdGwWpEICzYxVTjnnJldClzmnDu3ufeJeE19GCKRNxa4PziyqZgILoEr0hKqYYiISEjUhyEiIiFRwhARkZAoYYiISEiUMEREJCRKGCIiEhIlDBERCcn/BzQmBYQW/w1tAAAAAElFTkSuQmCC\n"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"Now we will train our classifier gradually unfreezing layers from the top. Training all layers straight way may result in loss of information achieved through fine-tuning of language model. "},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.fit_one_cycle(8, 1e-2, moms=(0.8, 0.7))","execution_count":290,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.HTML object>","text/html":"<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: left;\">\n      <th>epoch</th>\n      <th>train_loss</th>\n      <th>valid_loss</th>\n      <th>accuracy</th>\n      <th>time</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <td>0</td>\n      <td>0.682942</td>\n      <td>0.537429</td>\n      <td>0.786824</td>\n      <td>00:06</td>\n    </tr>\n    <tr>\n      <td>1</td>\n      <td>0.655243</td>\n      <td>0.532780</td>\n      <td>0.772898</td>\n      <td>00:06</td>\n    </tr>\n    <tr>\n      <td>2</td>\n      <td>0.646463</td>\n      <td>0.528764</td>\n      <td>0.795394</td>\n      <td>00:06</td>\n    </tr>\n    <tr>\n      <td>3</td>\n      <td>0.619592</td>\n      <td>0.503237</td>\n      <td>0.797001</td>\n      <td>00:06</td>\n    </tr>\n    <tr>\n      <td>4</td>\n      <td>0.611944</td>\n      <td>0.511356</td>\n      <td>0.795394</td>\n      <td>00:06</td>\n    </tr>\n    <tr>\n      <td>5</td>\n      <td>0.570965</td>\n      <td>0.483966</td>\n      <td>0.803964</td>\n      <td>00:06</td>\n    </tr>\n    <tr>\n      <td>6</td>\n      <td>0.574974</td>\n      <td>0.482108</td>\n      <td>0.808249</td>\n      <td>00:06</td>\n    </tr>\n    <tr>\n      <td>7</td>\n      <td>0.567295</td>\n      <td>0.481574</td>\n      <td>0.806642</td>\n      <td>00:06</td>\n    </tr>\n  </tbody>\n</table>"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.freeze_to(-2)\nlearn.fit_one_cycle(5, slice(1e-2/(2.6**4),1e-2), moms=(0.8, 0.7))","execution_count":291,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.HTML object>","text/html":"<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: left;\">\n      <th>epoch</th>\n      <th>train_loss</th>\n      <th>valid_loss</th>\n      <th>accuracy</th>\n      <th>time</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <td>0</td>\n      <td>0.599458</td>\n      <td>0.501742</td>\n      <td>0.798072</td>\n      <td>00:07</td>\n    </tr>\n    <tr>\n      <td>1</td>\n      <td>0.578998</td>\n      <td>0.488006</td>\n      <td>0.807177</td>\n      <td>00:06</td>\n    </tr>\n    <tr>\n      <td>2</td>\n      <td>0.533419</td>\n      <td>0.465607</td>\n      <td>0.820568</td>\n      <td>00:07</td>\n    </tr>\n    <tr>\n      <td>3</td>\n      <td>0.514364</td>\n      <td>0.455950</td>\n      <td>0.827531</td>\n      <td>00:07</td>\n    </tr>\n    <tr>\n      <td>4</td>\n      <td>0.457032</td>\n      <td>0.459975</td>\n      <td>0.823781</td>\n      <td>00:07</td>\n    </tr>\n  </tbody>\n</table>"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.unfreeze()\nlearn.fit_one_cycle(5, slice(5e-3/(2.6**4),5e-3), moms=(0.8, 0.7))","execution_count":292,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.HTML object>","text/html":"<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: left;\">\n      <th>epoch</th>\n      <th>train_loss</th>\n      <th>valid_loss</th>\n      <th>accuracy</th>\n      <th>time</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <td>0</td>\n      <td>0.440196</td>\n      <td>0.475264</td>\n      <td>0.825388</td>\n      <td>00:13</td>\n    </tr>\n    <tr>\n      <td>1</td>\n      <td>0.426092</td>\n      <td>0.453305</td>\n      <td>0.831280</td>\n      <td>00:14</td>\n    </tr>\n    <tr>\n      <td>2</td>\n      <td>0.380461</td>\n      <td>0.461150</td>\n      <td>0.833958</td>\n      <td>00:13</td>\n    </tr>\n    <tr>\n      <td>3</td>\n      <td>0.330918</td>\n      <td>0.486886</td>\n      <td>0.830745</td>\n      <td>00:13</td>\n    </tr>\n    <tr>\n      <td>4</td>\n      <td>0.263258</td>\n      <td>0.495219</td>\n      <td>0.831280</td>\n      <td>00:14</td>\n    </tr>\n  </tbody>\n</table>"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"## Evaluating the model"},{"metadata":{},"cell_type":"markdown","source":"Now we will evalute the perfomance. For this let's first read the previously saved test part of our dataset. Then make a prediction of each tweet."},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.read_csv(\"Tweets_test.csv\", encoding=\"utf-8\")\ntest_df['pred_sentiment'] = test_df['text'].apply(lambda row: str(learn.predict(row)[0]))\ntest_df['airline_sentiment'].value_counts().plot(kind='bar')","execution_count":293,"outputs":[{"output_type":"execute_result","execution_count":293,"data":{"text/plain":"<matplotlib.axes._subplots.AxesSubplot at 0x7f5b44c2d710>"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"<Figure size 432x288 with 1 Axes>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAYAAAAEeCAYAAACDq8KMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAFTNJREFUeJzt3X+0ZWV93/H3RxAVjfyQK7Ez4BCdaqk/AmsKWNdKjbT80OjQRBEicbR0zWrFxgZXFU1WsSam2jYa6UqoEyHCChEpiWWaEilB1JUY1AGVH6JliiIzolwFgWiNoN/+cZ6Jx+uduXfuuXP2vT7v11pn3b2f/ZxzvnfNmvs5z7OfvU+qCklSfx41dAGSpGEYAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1Kn9hy5gTw477LBat27d0GVI0qpy4403fqOqZhbqt6IDYN26dWzbtm3oMiRpVUly12L6OQUkSZ0yACSpUwsGQJKLk9yb5NZ5jr0hSSU5rO0nyQVJtie5OcmxY303JbmjPTYt768hSdpbixkBvB84ZW5jkiOAk4CvjDWfCqxvj83Aha3vocD5wPHAccD5SQ6ZpHBJ0mQWDICq+jhw3zyH3g28ERj/RpmNwKU1cgNwcJKnACcD11bVfVV1P3At84SKJGl6lnQOIMlGYGdVfW7OoTXA3WP7O1rb7tolSQPZ62WgSQ4E3sJo+mfZJdnMaPqII488cl+8hSSJpY0AngYcBXwuyZeBtcBNSX4a2AkcMdZ3bWvbXfuPqaotVbWhqjbMzCx4HYMkaYn2egRQVbcAT96130JgQ1V9I8lW4HVJLmd0wveBqronyTXAb4+d+D0JePPE1S+zdef9r6FL2Ke+/I4XD12CpBVkMctAPwD8NfCMJDuSnL2H7lcDdwLbgT8AXgtQVfcBvwl8uj3e1tokSQNZcARQVWcucHzd2HYB5+ym38XAxXtZnyRpH/FKYEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROLRgASS5Ocm+SW8fa/nOSLyS5OcmHkhw8duzNSbYn+WKSk8faT2lt25Oct/y/iiRpbyxmBPB+4JQ5bdcCz6qq5wD/B3gzQJKjgTOAf9ie8/tJ9kuyH/B7wKnA0cCZra8kaSALBkBVfRy4b07b/66qR9ruDcDatr0RuLyq/raqvgRsB45rj+1VdWdVfQ+4vPWVJA1kOc4B/Avgz9v2GuDusWM7Wtvu2n9Mks1JtiXZNjs7uwzlSZLmM1EAJPl14BHgsuUpB6pqS1VtqKoNMzMzy/WykqQ59l/qE5O8GvgF4MSqqta8EzhirNva1sYe2iVJA1jSCCDJKcAbgZdW1XfGDm0FzkjymCRHAeuBTwGfBtYnOSrJAYxOFG+drHRJ0iQWHAEk+QDwAuCwJDuA8xmt+nkMcG0SgBuq6l9V1W1JrgA+z2hq6Jyq+n57ndcB1wD7ARdX1W374PeRJC3SggFQVWfO03zRHvq/HXj7PO1XA1fvVXWSpH3GK4ElqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOrVgACS5OMm9SW4dazs0ybVJ7mg/D2ntSXJBku1Jbk5y7NhzNrX+dyTZtG9+HUnSYi1mBPB+4JQ5becB11XVeuC6tg9wKrC+PTYDF8IoMIDzgeOB44Dzd4WGJGkYCwZAVX0cuG9O80bgkrZ9CXDaWPulNXIDcHCSpwAnA9dW1X1VdT9wLT8eKpKkKVrqOYDDq+qetv014PC2vQa4e6zfjta2u/Yfk2Rzkm1Jts3Ozi6xPEnSQiY+CVxVBdQy1LLr9bZU1Yaq2jAzM7NcLytJmmOpAfD1NrVD+3lva98JHDHWb21r2127JGkgSw2ArcCulTybgKvG2l/VVgOdADzQpoquAU5Kckg7+XtSa5MkDWT/hTok+QDwAuCwJDsYreZ5B3BFkrOBu4DTW/ergRcB24HvAK8BqKr7kvwm8OnW721VNffEsiRpihYMgKo6czeHTpynbwHn7OZ1LgYu3qvqJEn7jFcCS1KnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUxMFQJJfS3JbkluTfCDJY5McleSTSbYn+WCSA1rfx7T97e34uuX4BSRJS7PkAEiyBvhVYENVPQvYDzgDeCfw7qp6OnA/cHZ7ytnA/a393a2fJGkgk04B7Q88Lsn+wIHAPcALgSvb8UuA09r2xrZPO35ikkz4/pKkJVpyAFTVTuC/AF9h9If/AeBG4FtV9UjrtgNY07bXAHe35z7S+j9pqe8vSZrMJFNAhzD6VH8U8PeAxwOnTFpQks1JtiXZNjs7O+nLSZJ2Y5IpoH8KfKmqZqvqYeBPgecDB7cpIYC1wM62vRM4AqAdPwj45twXraotVbWhqjbMzMxMUJ4kaU8mCYCvACckObDN5Z8IfB64HnhZ67MJuKptb237tOMfqaqa4P0lSROY5BzAJxmdzL0JuKW91hbgTcC5SbYzmuO/qD3lIuBJrf1c4LwJ6pYkTWj/hbvsXlWdD5w/p/lO4Lh5+n4XePkk7ydJWj5eCSxJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUqYkCIMnBSa5M8oUktyd5XpJDk1yb5I7285DWN0kuSLI9yc1Jjl2eX0GStBSTjgDeA3y4qp4JPBe4HTgPuK6q1gPXtX2AU4H17bEZuHDC95YkTWDJAZDkIODngIsAqup7VfUtYCNwSet2CXBa294IXFojNwAHJ3nKkiuXJE1kkhHAUcAs8IdJPpPkfUkeDxxeVfe0Pl8DDm/ba4C7x56/o7VJkgYwSQDsDxwLXFhVxwDf5ofTPQBUVQG1Ny+aZHOSbUm2zc7OTlCeJGlPJgmAHcCOqvpk27+SUSB8fdfUTvt5bzu+Ezhi7PlrW9uPqKotVbWhqjbMzMxMUJ4kaU+WHABV9TXg7iTPaE0nAp8HtgKbWtsm4Kq2vRV4VVsNdALwwNhUkSRpyvaf8Pn/BrgsyQHAncBrGIXKFUnOBu4CTm99rwZeBGwHvtP6SpIGMlEAVNVngQ3zHDpxnr4FnDPJ+0mSlo9XAktSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktSpSb8QRlo53nrQ0BXsW299YOgK9BPGEYAkdcoAkKROGQCS1CkDQJI6ZQBIUqcmDoAk+yX5TJI/a/tHJflkku1JPpjkgNb+mLa/vR1fN+l7S5KWbjlGAK8Hbh/bfyfw7qp6OnA/cHZrPxu4v7W/u/WTJA1kogBIshZ4MfC+th/ghcCVrcslwGlte2Pbpx0/sfWXJA1g0hHA7wJvBH7Q9p8EfKuqHmn7O4A1bXsNcDdAO/5A6y9JGsCSAyDJLwD3VtWNy1gPSTYn2ZZk2+zs7HK+tCRpzCQjgOcDL03yZeByRlM/7wEOTrLrFhNrgZ1teydwBEA7fhDwzbkvWlVbqmpDVW2YmZmZoDxJ0p4sOQCq6s1Vtbaq1gFnAB+pqlcC1wMva902AVe17a1tn3b8I1VVS31/SdJk9sV1AG8Czk2yndEc/0Wt/SLgSa39XOC8ffDekqRFWpa7gVbVR4GPtu07gePm6fNd4OXL8X6SpMl5JbAkdcoAkKROGQCS1Cm/EUzSivDsS549dAn7zC2bbhm6hHk5ApCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROLTkAkhyR5Pokn09yW5LXt/ZDk1yb5I7285DWniQXJNme5OYkxy7XLyFJ2nuTjAAeAd5QVUcDJwDnJDkaOA+4rqrWA9e1fYBTgfXtsRm4cIL3liRNaMkBUFX3VNVNbfsh4HZgDbARuKR1uwQ4rW1vBC6tkRuAg5M8ZcmVS5ImsiznAJKsA44BPgkcXlX3tENfAw5v22uAu8eetqO1SZIGMHEAJHkC8CfAv62qB8ePVVUBtZevtznJtiTbZmdnJy1PkrQbEwVAkkcz+uN/WVX9aWv++q6pnfbz3ta+Ezhi7OlrW9uPqKotVbWhqjbMzMxMUp4kaQ8mWQUU4CLg9qp619ihrcCmtr0JuGqs/VVtNdAJwANjU0WSpCnbf4LnPh/4FeCWJJ9tbW8B3gFckeRs4C7g9HbsauBFwHbgO8BrJnhvSdKElhwAVfWXQHZz+MR5+hdwzlLfT5K0vLwSWJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTUw+AJKck+WKS7UnOm/b7S5JGphoASfYDfg84FTgaODPJ0dOsQZI0Mu0RwHHA9qq6s6q+B1wObJxyDZIkph8Aa4C7x/Z3tDZJ0pTtP3QBcyXZDGxuu3+T5ItD1rOPHQZ8Y1pvlndO6526MdV/P/5DpvZWHZju/71XT/3f7qmL6TTtANgJHDG2v7a1/Z2q2gJsmWZRQ0myrao2DF2HlsZ/v9XLf7uRaU8BfRpYn+SoJAcAZwBbp1yDJIkpjwCq6pEkrwOuAfYDLq6q26ZZgyRpZOrnAKrqauDqab/vCtXFVNdPMP/9Vi//7YBU1dA1SJIG4K0gJKlTBoAkdcoAkKROGQADSPK4JM8Yug6pJxk5K8m/b/tHJjlu6LqGZABMWZKXAJ8FPtz2fzaJ10KscEkeSvLgPI+Hkjw4dH1alN8Hngec2fYfYnRzym6tuFtBdOCtjG6K91GAqvpskqOGLEgLq6qfGroGTez4qjo2yWcAqur+dkFqtwyA6Xu4qh5IfuTeIK7FXWWSPBl47K79qvrKgOVocR5ut6QvgCQzwA+GLWlYTgFN321JfhnYL8n6JP8V+MTQRWlxkrw0yR3Al4CPAV8G/nzQorRYFwAfAp6c5O3AXwK/PWxJw/JCsClLciDw68BJreka4Leq6rvDVaXFSvI54IXAX1TVMUl+Hjirqs4euDQtQpJnAicCAa6rqtsHLmlQBsCUJTm2qm4aug4tza67SLYgOKaqfpDkc1X13KFr054luQC4vKoccTeeA5i+30ny08CVwAer6tahC9Je+VaSJwAfBy5Lci/w7YFr0uLcCPxGW4L9IUZhsG3gmgblCGAALQBOB14BPJFREPzWsFVpMZI8Hvh/jM6fvRI4CLisqr45aGFatCSHAr/E6Hb0R1bV+oFLGowBMKAkzwbeCLyiqrpejrYatBUkf1FVPz90LVq6dvHXKxh9H/ntVfWSgUsajKuApizJP0jy1iS3ALtWAK0duCwtQlV9H/hBkoOGrkV7L8l/aiu43gbcCmzo+Y8/eA5gCBcDHwROrqqvDl2M9trfALckuZaxuf+q+tXhStIi/V/geVU1ve9xXuGcApL2QpJN8zRXVV069WK0KEmeWVVfSHLsfMd7XpXnCGBKklxRVae3qZ/x1A2jPyDPGag07Z2Dq+o94w1JXj9UMVqUc4HNwO/Mc6wYXdfRJUcAU5LkKVV1T5Knzne8qu6adk3ae0luqqpj57R9pqqOGaomLU6Sx8694HK+tp54EnhKquqetvnaqrpr/AG8dsjatLAkZyb5n8BRSbaOPa4H7hu6Pi3KfBeAdX1RmFNA0/fPgDfNaTt1njatLJ8A7gEO40enEh4Cbh6kIi1Ku+5mDfC4JMcwmnaF0TU4Bw5W2ApgAExJkn/N6JP+zyQZ/4PxU8BfDVOVFquN1O5idD95rS4nA69mtNz6XWPtDwFvGaKglcJzAFPS1o4fAvxH4LyxQw9VlVMIq0SSh/jhSfwDgEcD366qJw5XlRYjyS9V1Z8MXcdKYgAMxPvJr34ZfanDRuCEqjpvof4aRpKzquqPkryBeb57o6reNc/TuuBJ4ClL8hLvJ/+ToUb+B6MpBq1cj28/n8BoynXuo1uOAKbM+8mvbkl+cWz3UcAG4J9UlecGtOo4Api+h9udIx+V5FFVdT2jPyJaHV4y9jiZ0YnEjYNWpEVp9wJ6YpJHJ7kuyWySs4aua0iuApo+7ye/ilXVa4auQUt2UlW9Mck/ZzT1+ouM/h/+0aBVDcgRwPRtZHQ/+V8DPszoBlVd35FwNUny99unx1vb/nOS/MbQdWlRdn3gfTHw36vqgSGLWQk8ByDthSQfA/4d8N5dt39IcmtVPWvYyrSQJO8ATmP0Aew44GDgz6rq+EELG5AjgClL8lCSB+c87k7yoSQ/M3R9WtCBVfWpOW2PDFKJ9kpbqvuPGX0PwMOMpl67Pn/jOYDp+11gB/DHjC5JPwN4GnATo+8KeMFglWkxvpHkabT15ElexugWEVrhkjwaOAv4udElHHwM+G+DFjUwp4CmLMnnquq5c9o+W1U/O98xrSxtlLaF0SfJ+xldz/FK7+a68iV5H6Mrty9pTb8CfL+q/uVwVQ3LEcD0fSfJ6cCVbf9lwK7b0ZrGK99O4A+B64FDgQeBTYy+ZlAr2z+a8wHrI+26nG55DmD6Xsnok8e9wNfb9llJHge8bsjCtChXMVq19TDwVUZfEeky3tXh+236Dvi70dz3B6xncE4BSXvBFT+rV5ITGY3e7mxN64DXtIsxu+QIYMpcR77qfSLJs4cuQkvyV8B7gR8w+hKf9wJ/PWhFA3MEMGWuI1/dknweeDqjk79/i9/pvGokuYLROZvLWtMvM/qO55cPV9WwPAk8fQdW1afaMrRdXEe+epw6dAFasmdV1dFj+9e3QO+WATB9riNfxVzuuardlOSEqroBIMnxwLaBaxqUU0BT5jpyaRhJbgeeAez68qUjgS8yGoF3OY1nAExZkscwWvu/jh+uI6+qch25tA8leeqejvf4IcwpoOm7CvgWo1s/fHXgWqRu9PgHfiGOAKbMFT+SVgqvA5g+15FLWhEcAUyZ68glrRQGwJTt7kSU85OSps0AkKROeQ5AkjplAEhSpwwASeqUASBJnTIAJKlT/x9Rb/mv944a5QAAAABJRU5ErkJggg==\n"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"We see that test set contains same proportion of sentiments as train set.\nFinally we calculate the accuracy on the test set. "},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Test Accuracy: \", accuracy_score(test_df['airline_sentiment'], test_df['pred_sentiment']))","execution_count":294,"outputs":[{"output_type":"stream","text":"Test Accuracy:  0.8324225865209471\n","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"Now let's ivestigate mistakes that has been made by our model using confusion matrix."},{"metadata":{"trusted":true},"cell_type":"code","source":"conf_matrix = confusion_matrix(y_true=test_df['airline_sentiment'].values, y_pred=test_df['pred_sentiment'].values, labels=['negative', 'neutral', 'positive'])\nlabels = ['negative', 'neutral', 'positive']\nsn.heatmap(conf_matrix, annot=True, fmt='g', xticklabels=labels, yticklabels=labels)","execution_count":295,"outputs":[{"output_type":"execute_result","execution_count":295,"data":{"text/plain":"<matplotlib.axes._subplots.AxesSubplot at 0x7f5b44c34160>"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"<Figure size 432x288 with 2 Axes>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAWYAAAD8CAYAAABErA6HAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XucVfP+x/HXZ2ZS0b1IcomE34nUMSEc6pSiE/GTW4lwdA5xyOWcEyHK+bmFI9cicj3lEp0joYtbSSVJSbrQSSLpqpRm5vP7Y6+yS01r9syevfaa97PHesza373XXp+9ZvrMZ77ru77L3B0REYmOnEwHICIiW1NiFhGJGCVmEZGIUWIWEYkYJWYRkYhRYhYRiRglZhGRiFFiFhGJGCVmEZGIyUv3DjYtX6hLC9OsUZNTMh1C7H23blWmQ6gQCn5eYqV9j5LknEr1Dij1/tJBFbOISMSkvWIWESlXRYWZjqDUlJhFJF4KCzIdQakpMYtIrLgXZTqEUlNiFpF4KVJiFhGJFlXMIiIRo5N/IiIRo4pZRCRaXKMyREQiRif/REQiRl0ZIiIRo5N/IiIRo4pZRCRidPJPRCRidPJPRCRa3NXHLCISLepjFhGJGHVliIhEjCpmEZGIKdyU6QhKTYlZROJFXRkiIhGjrgwRkYhRxSwiEjFKzCIi0eI6+SciEjHqYxYRiRh1ZYiIRIwqZhGRiFHFLCISMaqYRUQipkAT5WeFvv+4h3cnTqFO7Vq88swjANz9wGO8M/FD8irlsU/DBgy4/mpqVK/Gp5/Npd8d9wPgOJdd1I12JxwLwNMjXuGlUWNwd7qcehLdzz49Y58p6gYO6k+7DiewfPkK2h5zGgCdOrfn6r/1osnBB/CHtucwc8ZsAJr/9jDuvK8fAGbGwNsfZMxr4zIVelaqXLkyb49/iV0qVyYvL5eXX36NW24dyFPDBnHEEYezadMmpk6dwaWX/Y2CGCSuYsWgYs7JdADl4bSOJ/LIPQO2amvVsgUjn36EkU89TKN9GvLY08MBOPCA/Rj++P28NOxBHh04gFvvHERBQSHzFn7FS6PG8Pxj9/HSsId4Z9IU/vv1N5n4OFlhxPOv0K3Ln7Zq+3zOfC45/0omT5q2Tfs8Tm5zFu2PP4NuXXpyx703k5ubW57hZr2NGzfSrv1ZHJF/Ikfkt6dD+9YcdeRvef75kTQ99Hiat2hL1apVuPiirpkONf2KisIvEVUhEnN+88OoWaP6Vm3HHnUEeXmJ//zNmh7Cd8uWA1C1SpUt7Rt//hnMAFj41WIOa3rwlufzmx/G2HcmluOnyC4fTvqIVStXb9U2/4uFLJj/1a9eu+GnDRQWJu46UblyZdy9PEKMnXXr1gNQqVIeeZUq4e68Pmb8luenTp3B3ns3yFR45ceLwi8RFToxm1lVMzs4ncFkysjX3uS4Vi23PJ45+3M6d/sTp59/KTdddzl5ebkceMB+TP9kNqtWr+GnDRt474OpfPvd9xmMOl5aHHEY4ye9yriJr/D3q2/dkqglvJycHKZNfZOlS2Yybty7TJn68Zbn8vLy6NbtDN54Y0IGIywnFaViNrNTgBnAmOBxczMblc7Aysujw54nNzeXTu3bbGlr1vQQXn32Uf712D957OkRbNz4M40b7ctF3c6kZ+8b+PPVN3JwkwPIyakQf3CUi48/+pTfH9OZjm3P5vLel1C58i6ZDinrFBUVkd+yPfvtn0/L/BY0bfpLHfXAoH/w3nsf8v7EKRmMsJxUoIq5H3AksArA3WcA++/oxWbW08ymmdm0x556vtRBpssrr73FuxOncMfNf8WCLotkjRvty65VqzJv4VcAnHFKB0YMHcSwh+6iRvXqNNp373KOOP7mf7GQ9evWc/D/NMl0KFlr9eo1vP3ORDq0bw3AjX17s/vudbn2un4ZjavcFBSEXyIqbGLe5O6rt2nbYUeguw9293x3z//j+eemHl0avT95GkOfe4FBd9xM1SpVtrR//c23FBQk/oz+5tvv+HLRYho2qA/ADytXAbD022WMe2ciHU9sXe5xx9E++zbccrKv4T4NaNxkfxb/d0mGo8ou9erVoWbNGgBUqVKFdm2PZ+7cBVx04bm0P7E13c7rVXH67t3DLxEVdrjcbDPrCuSaWRPgL8Ck9IVVtq67+XamfjyTVavW0Pa087js4u489vRwft60iUuuugFIdF/c/NcrmD5zNo8/PYK8vDxycoy+1/aidq2aAPS+fgCr1qwhLy+PG665jBrVq2XyY0Xag4/dRatjW1Knbi2mzRrH3bc/yKqVqxlwx/XUqVeHp4Y/xOxP59KtS0+ObPVbel35RwoKCigqKuL6a/uzcsWqTH+ErNKgQX2GPn4fubk55OTk8OKL/+a10WPZsH4RixZ9zfvvJXoeX3llNANuuy/D0aZZhPuOw7Iwv0XNbFfgBqB90PQGMMDdN+xs203LF0b311JMNGpySqZDiL3v1ukXRXko+HnJr/sUS+inZ28MnXOqdutf7P7MbCjQCVjm7ocGbXWA4UAj4CvgLHdfaYn+0H8CHYH1QA93nx5scwHQN3jbAe4+rLj9hu3KOMTdb3D3lsHSN0xSFhEpd2V78u9J4KRt2v4OjHP3JsC44DHAyUCTYOkJPAxbEvnNwFEkztXdbGa1i9tp2MQ80MzmmFl/Mzs05DYiIuWvsDD8shPu/i6wYpvmzsDmincYcFpS+1OeMBmoZWYNgA7AW+6+wt1XAm/x62S/lVCJ2d3bAG2A74FHzexTM+u7k81ERMpfCcYxJ48gC5aeIfZQ392XBuvfAvWD9YbA4qTXfR207ah9h0LPleHu3wL3m9kE4K/ATcCA4rcSESlnJTj55+6DgcGp7srd3czK/Dxa2AtM/sfM+pnZp8AgEiMyNIhXRKIn/ReYfBd0URB8XRa0LwH2SXrd3kHbjtp3KGwf81ASF5d0cPfW7v6wuy/b2UYiIuXNizz0kqJRwAXB+gXAq0nt51vC0cDqoMvjDaC9mdUOTvq1D9p2KFRXhru3SiV6EZFyV4bjmM3seaA1UM/MviYxuuJ2YISZXQwsAs4KXj6axFC5+SSGy10I4O4rzKw/MDV43a3uvu0Jxa0Um5jNbIS7nxV0YST/erHE/rxZ+I8oIlIOynACLHff0aXLbbfzWgd67eB9hpLoeQhlZxXzlcHXTmHfUEQko2Jw5V+xfcxJQ0Iuc/dFyQtwWfrDExEpoYoy7Sdw4nbaTi7LQEREykTcJzEys0tJVMYHmNnMpKeqA7p9h4hET4Qr4bB21sf8HPA68H/8cj04wNqdnVUUEcmI1IfBRUaxiTmYg3k1cC6Ame0BVAGqmVk1d/9v+kMUESmBGNyWLNQ45uDWUvcAe5G4ymU/YA7QNH2hiYiUnMegKyPsyb8BwNHAF+6+P4kxfJPTFpWISKqKPPwSUSW5tdQPQI6Z5bj7BCA/jXGJiKQmBjdjDTu73Cozqwa8CzxrZsuAdekLS0QkRRGuhMMKm5g7AxuA3kA3oCZwa7qCEhFJWUEFOfnn7snVcbH3qhIRyagId1GEFXZUxlq2nsQIEsPopgHXuPvCsg5MRCQlFagr4z4St0N5jsTMcucAjYHpJGZMap2O4ERESioOw+XCJuZT3f3wpMeDzWyGu//NzK5PR2AiIimJQcUcdrjcejM7y8xyguUsEicD4dddHCIimRODccxhK+ZuwD+Bh0gk4snAeWZWFbg8TbGJiJRcRbkkOzi5d8oOnn6/7MIRESmdUtzLLzLC3iX7IDMbZ2azgsfNzKxvekMTEUlBDLoywvYxDwH6AJsA3H0miZEZIiLREoM7mITtY97V3aeYWXJbQRriEREpnQhXwmGFTczLzawxwQgMM+sCLC1+ExGRDKhAibkXMBg4xMyWAF+SGKkhIhIpXhjdLoqwwibmJcATwASgDrAGuIAQExnt3bhjysFJOCfV1v0K0u25dZp+PGtUoIr5VWAViUuwv0lfOCIipROH4XJhE/Pe7n5SWiMRESkLMUjMYYfLTTKzw9IaiYhIWSgqwRJRYSvm44AeZvYlsJHEDHPu7s3SFpmISAq8IMIZN6SwifnktEYhIlJWsj8vh54rY1G6AxERKQsV6eSfiEh2qCgVs4hItlDFLCISNaqYRUSixWMwvZoSs4jEiqtiFhGJmBgk5rBX/omIZAUvCr8Ux8wONrMZScsaM7vKzPqZ2ZKk9o5J2/Qxs/lmNtfMOqT6GVQxi0islFVXhrvPBZoDmFkuiVk2RwIXAve6+93Jrzez35C4s1NTYC9grJkd5O4lvjusKmYRiRUvtNBLCbQFFuzkYrvOwL/cfaO7fwnMB45M5TMoMYtIrJSkK8PMeprZtKSl5w7e9hzg+aTHl5vZTDMbama1g7aGwOKk13wdtJWYErOIxIoXWfjFfbC75yctg7d9PzPbBTgVeCFoehhoTKKbYykwsKw/g/qYRSRW0jBc7mRgurt/B7D5K4CZDQH+EzxcAuyTtN3eQVuJqWIWkVhxt9BLSOeS1I1hZg2SnjsdmBWsjwLOMbPKZrY/0ASYkspnUMUsIrFSlhWzme0GnAj8Kan5TjNrDjjw1ebn3H22mY0APgMKgF6pjMgAJWYRiZmiko22KJa7rwPqbtPWvZjX3wbcVtr9KjGLSKx4Udkl5kxRYhaRWFFiFhGJGM/+6ZiVmEUkXlQxi4hETAmGwUWWErOIxEphGY7KyBQlZhGJFVXMIiIRoz5mEZGI0agMEZGIUcWche574DZOPKk1y7//gRNanQpArdo1GfzEPeyzb0MW/3cJl/TozepVazjjzE5cftUlmBk//riOv17dj89mzc3wJ4i+SpUrcf3w/uRVrkRubi5TX/+AkfcO50/3Xcn+hzWmsKCQhZ/M48nrH6WwoJCTe3am1Wm/AyA3N5e9DmzI5b+9iHWrf8zwJ8kelStXZsL4l6hcuTK5ebm8/PJr3HrrQCaMf5nq1asBsPvudZk6bQZdulyc4WjTq7Ao++dmM09z3V+/5iGR+sPi6GPyWbduPQ88cvuWxHzjrdeyauVqBt07hCt6X0LNWjUYcPNA8o9swbwvFrB61Rp+3+53XNfnck5ue3aGP8GvnVS7aaZD+JXKu1Zh4/oN5OblcsOLA3j2lqHsVrM6M9+eDsCl9/dm7pTPGP/MG1tt17xtPh0u7sQdXftlIOode+6byZkOYad2221X1q1bT15eHu+8PZKrr76ZD6dM3/L88OGD+fe/3+SZZ17MYJTF2/TzklKXuzMbnRI65zT76t+RLK+z/1dLCU2eNI1VK1dv1XZSx7YMf+4VAIY/9won/6EdANOmfMzqVWsA+GjaJzTYa8/yDTaLbVy/AYDcvFxy8/JwZ0tSBlj4yTxq71n3V9sdfepxTB71frnFGSfr1q0HoFKlPCpVqkRy0VW9ejXatD6WV18dk6nwyk2RW+glqipcYt6e3Xevy7Lvvgdg2Xffs/vuv04YXbt3YfzYd8s7tKxlOTncOvpuBn00lNnvf8LCGfO2PJebl8sxp5/Ap+98vNU2u1TZhcNOaM6016NfnUZRTk4O06a+yTdLZjJ23LtMmfrL8e3c+STGT5jI2rXx7x5Kw3zM5a7YxGxma4Nbdm+7rDWzNeUVZHlztv5L6NjfHUXX7mfQ/6Yyv4NMbHlRETd1vJberXpywOFNaHjQLzd2OL//Jcyd8hlfTJ2z1TbN2+Uzb9pc9S2nqKioiPyW7Wm0fz4t81vQtOnBW547+6zODB/+SgajKz/u4ZeoKjYxu3t1d6+xnaW6u9fY0XbJNzj86edVZR91Gfv++x/Yo/7uAOxRf3eWf79iy3O/aXoQ9wzqzwXn9mLlyuh/lqhZv2Y9cz6YRbMTWgBw2pVnUr1uTZ7v/+SvXnv0KccxedR75Rxh/KxevYa335lI+/atAahbtzYtW7Zg9OhxmQ2snFS4rgwz28PM9t287Oh1yTc4rLpLrdJHmWZvvD6es7ueBsDZXU9jTPAD3HDvBgx9ZhC9ev6NhQu+ymCE2aV6nRrsWmNXACpV3oWmxzXjmwVLOOHsthx6fHMevuJetj3pXLX6rhx81G+Y/tbUTISc9erVq0PNmolaqUqVKrRrezxz5y4A4Iz/7cTo0WPZuHFjJkMsN4VFOaGXqAo1XM7MTiVxJ9i9gGXAfsAcIHrDAXbikccHcsxxLalTtzYff/Y2d/3fIAbdM4Qhw+6la/cz+HrxN1zSozcA1/ztMmrXqcUdA28CoKCwkA6tu2Qy/KxQa4/aXDLwcnJycrEcY8prk/hk/EcMnT+CH5Z8z40j/wHAR2M+5NX7EzcePqLDUcx67xN+/qliJI+y1qBBfYY+fh+5uTlYTg4vvvhvRo8eC8BZZ53KnXc9mOEIy0+EeyhCCzVczsw+AX4PjHX3FmbWBjjP3Xc6IDJqw+XiKIrD5eImG4bLxUFZDJeb1OCM0DnnmKUvRbI/I2wtv8ndfwByzCzH3ScA+WmMS0QkJXEYlRH2yr9VZlYNeBd41syWAevSF5aISGrK8CbZGRO2Yu4MrAd6A2OABcAp6QpKRCRVjoVeomqnFbOZ5QL/cfc2JH4ZDUt7VCIiKSqIcBdFWDutmN29ECgys5rlEI+ISKlUiIo58CPwqZm9RVLfsrv/JS1RiYikKA59zGET88vBkkzD4EQkcqJcCYcVNjHXcvd/JjeY2ZVpiEdEpFTiUDGHHZVxwXbaepRhHCIiZaIQC71EVbEVs5mdC3QF9jezUUlPVQdWbH8rEZHMicGdpXbalTEJWArUIzFXxmZrgZnpCkpEJFVFEa6Ewyo2Mbv7ImAR0Kp8whERKZ04jEoIO7vcWn75vLsAlYB1xc3JLCKSCXE4+RcqMbt79c3rZmYkLtE+Ol1BiYikqsiyvyujxDNFe8IrQIc0xCMiUiqFJViiKmxXxv8mPcwhMeXnhrREJCJSChVhVMZmyTPJFQBfkejOEBGJlNiPytjM3S9MdyAiImWhLEdlmNlXJIYHFwIF7p5vZnWA4UAjEkXqWe6+Mjj/9k+gI4lpknu4+/RU9huqj9nMDjKzcWY2K3jczMz6prJDEZF0KrLwS0ht3L25u2++a9PfgXHu3gQYFzwGOBloEiw9gYdT/QxhT/4NAfoAmwDcfSZwTqo7FRFJl6ISLCnqzC/z0g8DTktqfyoYIDEZqGVmDVLZQdjEvKu7T9mmrSCVHYqIpFOhhV/MrKeZTUtaem7zdg68aWYfJT1X392XBuvfAvWD9YbA4qRtvw7aSizsyb/lZtY4CBIz60LiUm0RkUgpSSXs7oOBwcW85Dh3X2JmewBvmdnn22zvZlbmFxuGTcy9SAR/iJktAb4EupV1MCIipVWWV/65+5Lg6zIzGwkcCXxnZg3cfWnQVbEsePkSYJ+kzfcO2kosbFfGEuAJ4DbgX8BbbH8qUBGRjHILvxTHzHYzs+qb14H2wCxgFL/kvwuAV4P1UcD5lnA0sDqpy6NEwlbMrwKrgOnAN6nsSESkPJRhxVwfGJkYBUce8Jy7jzGzqcAIM7uYxCRvZwWvH01iqNx8EsPlUh5mHDYx7+3uJ6W6ExGR8lJWl1q7+0Lg8O20/wC03U67k+j2LbWwXRmTzOywstihiEg6pWEcc7kLWzEfB/Qwsy+BjYCR+AXRLG2RiYikoMJM+0niihYRkcirMIk5uJOJiEjkVZg7mIiIZIso9x2HpcQsIrES5Qnww0p7Yl654cd076LCe+6byZkOIfba1dd57mxRFIPODFXMIhIrFebkn4hItsj+elmJWURiRhWziEjEFJT9LJzlTolZRGIl+9OyErOIxIy6MkREIkbD5UREIib707ISs4jEjLoyREQipjAGNbMSs4jEiipmEZGIcVXMIiLRoopZRCRiNFxORCRisj8tKzGLSMwUxCA1KzGLSKzo5J+ISMTo5J+ISMSoYhYRiRhVzCIiEVPoqphFRCJF45hFRCJGfcwiIhGjPmYRkYhRV4aISMSoK0NEJGI0KkNEJGLUlSEiEjFxOPmXk+kARETKkpfgX3HMbB8zm2Bmn5nZbDO7MmjvZ2ZLzGxGsHRM2qaPmc03s7lm1iHVz6CKWURipQy7MgqAa9x9uplVBz4ys7eC5+5197uTX2xmvwHOAZoCewFjzewgdy8s6Y6VmIGcnBwmfzCaJd98y+mn9+DRR+7miCOaYWbMm7eQi//Ym3Xr1mc6zKyWk5PDh5NfZ8mSbznt9Au47NIeXHHFHznwwP3Zs8Gh/PDDykyHmHXqNajHtfddS+16tXF3Xn/udV4d+ioAp/Y4lU4XdKKosIgp46cw9B9DafG7Flz49wvJ2yWPgp8LePy2x/lk0icZ/hRlz8vo5J+7LwWWButrzWwO0LCYTToD/3L3jcCXZjYfOBL4oKT7VmIGrrjiYj7/fD7Va1QD4Nrr+rF27Y8A3HnnTVx26YXcdfeDmQwx6/3lij8y5/N51KheHYBJH0zltdFjGfvWixmOLHsVFhYypP8QFsxaQNXdqnL/6Pv5+L2PqVWvFke3P5peHXqx6edN1KxbE4A1K9bQ76J+rPhuBfsdvB8DnhlA95bdM/wpyl5hGk7+mVkjoAXwIXAscLmZnQ9MI1FVrySRtCcnbfY1xSfyHarwfcwNGzbg5JPbMvSJ57a0bU7KAFWrVimz38AV1ZZjPPT5LW0zZsxm0aKvMxhV9lu5bCULZi0A4Kd1P7F4/mLq7lmXP3T/AyMeGsGmnzcBsPqH1QAsmL2AFd+tAGDR3EVUrlKZSrtUykzwaVSEh17MrKeZTUtaem77fmZWDXgJuMrd1wAPA42B5iQq6oFl/RlCJWZLOM/Mbgoe72tmR5Z1MJkw8O5+9OlzG0VFWyffIYMHsvi/H3PwQQfy4ENDMxRdPAwceAt9+gygqCgO58ujaY+996Bx08bM/XguDQ9oyKFHHsq9o+7lzhfu5KDDD/rV64/reBzzP52/JXnHibuXZBns7vlJy+Dk9zKzSiSS8rPu/nLw/t+5e6G7FwFDSHRXACwB9knafO+grcTCVswPAa2Ac4PHa4Gs/9u+Y8e2LPt+OR9//Omvnruk5zXs1+gIPp87jzPPPDUD0cVDx47t+H7ZcqZv5xhL2aiyaxX6PtqXR/s9yvof15Obl0v1WtXpfWpvHrvtMfo81Ger1+970L5cdP1FDOozKEMRp1dJKubimJkBjwNz3P2epPYGSS87HZgVrI8CzjGzyma2P9AEmJLKZwibmI9y917ABoCgP2WXHb04+c+DosJ1qcRVLo5p1ZJOf2jPF3M/4JmnH6RN62N58on7tzxfVFTEiBGjOP30jsW8ixTnmGPy6dSpPfO+mMyzzzxEmzbHMuzJ+3e+oYSSm5dL38F9mfDKBCaNmQTA8qXLmfj6RAC+mPEF7k7NOol+5np71uPGITdy91V3s3TR0ozFnU5lNVyORF9yd+D32wyNu9PMPjWzmUAboDeAu88GRgCfAWOAXqmMyIDwiXmTmeUS3BnczHanmHHcyX8e5OTulkpc5aLvjbdzQOOWHHRwK87r3osJb0+kx4V/oXHjRlte06nTicydOz9zQWa5vn1vZ/8D8mly0NF0O+8yJkyYyAU9/pLpsGLjqruuYvG8xYwcMnJL2wdvfMDhxxwOQMP9G5JXKY/VK1azW43duGXYLTzxf0/w2bTPMhVy2hW6h16K4+7vu7u5ezN3bx4so929u7sfFrSfGoze2LzNbe7e2N0PdvfXU/0MYUdl3A+MBPYws9uALkDfVHcaZWbG44/dS40a1TGDmTPncPkVfXa+oZTI5b0u4pprLmPPPXdn+kdjGTNmPH/683WZDiurNG3ZlHZd2vHlnC95YMwDAAy7YxhvDn+T3nf35uGxD1PwcwEDeyfOTZ3S4xT2arQXXa/qSterugJwQ7cbtpwcjIs4XJJtYUccmNkhQFvAgHHuPifMdrtU3jv7j1LEadRI+rWr3yzTIVQIry9+3Ur7Hq0atgn9H+KDJRNKvb90CFUxm9n9JAZOZ/0JPxGJtzgUKmH7mD8C+prZAjO728zy0xmUiEiqympURiaFSszuPszdOwItgbnAHWY2L62RiYikoAxHZWRMSS/JPhA4BNgPCNXHLCJSngo9+y9kCtvHfCeJgdQLgOFAf3dflc7ARERSEYc+5rAV8wKglbsvT2cwIiKlFeW+47CKTcxmdoi7fw5MBfY1s32Tn3f36ekMTkSkpKLcdxzWzirmq4GebH/2JAd+X+YRiYiUQlHcuzLcffMUeCe7+4bk58ysStqiEhFJURwq5rDjmCeFbBMRyahCLwq9RNXO+pj3JDEDf1Uza0HicmyAGsCuaY5NRKTEYt+VAXQAepCY8PmepPa1wPVpiklEJGVx6MrYWR/zMGCYmZ3h7i+VU0wiIimLfcVsZue5+zNAIzO7etvnk2f1FxGJgthXzMDmWe6rpTsQEZGyUJjaTUMiZWddGY8GX28pn3BEREonDpdkh71L9p1mVsPMKpnZODP73szOS3dwIiIlVWGm/QTau/saoBPwFYlZ5nQfIBGJHHcPvURV2EmMNr/uD8AL7r46cWdvEZFoif2ojCT/MbPPgZ+AS4O7ZG/YyTYiIuWuIozKAMDd/x7Mybza3QvNbB3QOb2hiYiUXJQvtQ4r7ET5lYDzgOODLox3gEfSGJeISEqi3HccVtiujIeBSsBDwePuQdsf0xGUiEiqKlIfc0t3Pzzp8Xgz+yQdAYmIlEYcKuaww+UKzazx5gdmdgCQ/ZfXiEjsxGEcc9iK+TpggpktDB43Ai5MS0QiIqUQh4o5bGKeCDwKtAVWAW8AH6QrKBGRVFWYURnAU8AaoH/wuCvwNHBmOoISEUlVRTr5d6i7/ybp8QQz+ywdAYmIlEYcujLCnvybbmZHb35gZkcB09ITkohI6rwE/6IqbMV8BDDJzP4bPN4XmGtmnwLu7s3SEp2ISAnFoWIOm5hPSmsUIiJlJA59zBaH3y5lzcx6uvvgTMcRZzrG6adjnL3C9jFXND0zHUAFoGOcfjrGWUqJWUQkYpSYRUQiRol5+9Qvl346xumnY5yldPJPRCRiVDGLiESMEvNOmFktM7ss6fFeZvZiJmOKCzNrZGZdU9z2x7KOJ07M7M9mdn6+VXK9AAAEaklEQVSw3sPM9kp67jEz+82Ot5ZMU1fGTphZI+A/7n5ohkOJHTNrDVzr7p2281yeuxcUs+2P7l4tnfHFhZm9TeI4axqFLJH1FXNQdc0xsyFmNtvM3jSzqmbW2MzGmNlHZvaemR0SvL6xmU02s0/NbMDmysvMqpnZODObHjy3+WaztwONzWyGmd0V7G9WsM1kM2uaFMvbZpZvZruZ2VAzm2JmHye9VyykcMyfNLMuSdtvrnZvB34XHNveQWU3yszGA+OK+Z7EWnB8PzezZ4Pj/KKZ7WpmbYOfp0+Dn6/KwetvN7PPzGymmd0dtPUzs2uD454PPBsc56pJP6d/NrO7kvbbw8weCNbPC35+Z5jZo2aWm4ljUWG5e1YvJCbtLwCaB49HkLhx7DigSdB2FDA+WP8PcG6w/mfgx2A9D6gRrNcD5gMWvP+sbfY3K1jvDdwSrDcA5gbr/wDOC9ZrAV8Au2X6WGXwmD8JdEnafvMxb03ir5HN7T2Ar4E6xX1Pkt8jjktwfB04Nng8FOgLLAYOCtqeAq4C6gJzk45LreBrPxJVMsDbQH7S+79NIlnvDsxPan8dOA74H+DfQKWg/SHg/Ewfl4q0ZH3FHPjS3WcE6x+R+ME+BnjBzGaQmOS/QfB8K+CFYP25pPcw4B9mNhMYCzQE6u9kvyOAzZXgWcDmvuf2wN+Dfb8NVCEx8VOclOSYl8Rb7r4iWE/lexIXi919YrD+DImbVHzp7l8EbcOA44HVwAbgcTP7X2B92B24+/fAQjM72szqAoeQuClGWxITl00NvpdtgQPK4DNJSGEnMYq6jUnrhST+865y9+YleI9uJCqII9x9k5l9RSKh7pC7LzGzH8ysGXA2iQocEgnlDHefW4L9Z5uSHPMCgm4zM8sBdinmfdclrZf4exIj2578WUWiOt76Re4FZnYkieTZBbgc+H0J9vMvEkXF58BId3czM2CYu/dJKXIptbhUzNtaA3xpZmcCWMLmu3xPBs4I1s9J2qYmsCxIAG2A/YL2tUD1YvY1HPgrUNPdZwZtbwBXBD/gmFmL0n6gLFDcMf+KRAUGcCpQKVjf2bHd0fekItjXzFoF611JzH/eyMwODNq6A++YWTUSP3ujSXStHf7rtyr2OI8EOgPnkkjSkOiS6mJmewCYWR0zq0jHPuPimpghUW1dbGafALNJ/PBBol/u6uDP4wNJ/CkI8CyQb4k5ps8nUUHg7j8AE81sVvKJkiQvkkjwI5La+pNIPjPNbDa/3JIr7nZ0zIcAJwTtrfilKp5J4g7sn5hZ7+2833a/JxXEXKCXmc0BagP3krgB8gvB8SgCHiGRcP8T/Dy/D1y9nfd6Enhk88m/5CfcfSUwB9jP3acEbZ+R6NN+M3jft0itW0pSVOGGy5nZrsBPwZ9s55A4EVghzvZLdjAN0azw4tLHXBJHAA8E3QyrgIsyHI+IyFYqXMUsIhJ1ce5jFhHJSkrMIiIRo8QsIhIxSswiIhGjxCwiEjFKzCIiEfP/K3MVbkmgytcAAAAASUVORK5CYII=\n"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"We see that our model classified significant amount of neutral tweets as negatives which is reasonable because even as humans it sometimes not clear. What is more interesting is to look at positive tweets that were classified as negatives."},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.set_option('display.max_colwidth', -1)\ntest_df.loc[(test_df['airline_sentiment'] == 'positive') & (test_df['pred_sentiment'] == 'negative')].head()","execution_count":296,"outputs":[{"output_type":"execute_result","execution_count":296,"data":{"text/plain":"    airline_sentiment      ...       pred_sentiment\n54   positive              ...        negative     \n63   positive              ...        negative     \n128  positive              ...        negative     \n129  positive              ...        negative     \n152  positive              ...        negative     \n\n[5 rows x 3 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>airline_sentiment</th>\n      <th>text</th>\n      <th>pred_sentiment</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>54</th>\n      <td>positive</td>\n      <td>@united good job at CLE .. TPA on schedule ... 4 to 5 inches of snow ! http://t.co/9tbsJquw41</td>\n      <td>negative</td>\n    </tr>\n    <tr>\n      <th>63</th>\n      <td>positive</td>\n      <td>@AmericanAir yes called your UK number on skype…well worth the $.50.  i recommend others do the same.</td>\n      <td>negative</td>\n    </tr>\n    <tr>\n      <th>128</th>\n      <td>positive</td>\n      <td>@united you suck. @SouthwestAir you're the best.</td>\n      <td>negative</td>\n    </tr>\n    <tr>\n      <th>129</th>\n      <td>positive</td>\n      <td>@USAirways your ticket agents at gate 4 in Providence airport rocked tonight, especially Kristy, sorry if that is not the correct spelling.</td>\n      <td>negative</td>\n    </tr>\n    <tr>\n      <th>152</th>\n      <td>positive</td>\n      <td>@united This is probably the least dependable airline in the Western Hemisphere. @united does not belong in Star Alliance, but SkyTeam</td>\n      <td>negative</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"From here we see that these tweets are indeed confusing as often they are responses to other users' tweets."},{"metadata":{},"cell_type":"markdown","source":"## Coclusion\nIn this notebook we explored the application of ULMFiT strategy for detecting sentiments of tweets and achieved level of accuracy similar to human. Improvement in perfomance might be obtained by tweaking the parameters of the model as well as taking into account additional information such as relation between particular tweets."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}