TrabalhoFinal_AR_REV02.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0XkYIDM8FfXM"
   },
   "source": [
    "# Trabalho Final de NLP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hwBguZ4W5PIE"
   },
   "source": [
    "## TURMA: FIAP-8IA \n",
    "### Alunos\n",
    "* ALEX DIAS DE ARAUJO - RM 334983\n",
    "* ANDRE LUIZ SILVA RODRIGUES - RM 334911\n",
    "* JOSÉ ANTONIO WESLEY CRUZ - RM 334341\n",
    "* PAULO ARAUJO BARBOSA - RM 335284"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 121
    },
    "colab_type": "code",
    "id": "gxXlweBISUeW",
    "outputId": "e4a4ce4b-48e1-4ebf-ba8c-3d453d184414"
   },
   "outputs": [],
   "source": [
    "# from google.colab import drive\n",
    "# drive.mount('/content/gdrive', force_remount=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HHdyMggUGKXx"
   },
   "source": [
    "## 1) Importando o dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import nltk\n",
    "import seaborn as sb\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn import svm\n",
    "from sklearn import metrics\n",
    "\n",
    "# Importe as bibliotecas de Pipelines e Pré-processadores\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.preprocessing import MaxAbsScaler\n",
    "from sklearn.model_selection import cross_val_predict\n",
    "\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 7717,
     "status": "ok",
     "timestamp": 1575929611456,
     "user": {
      "displayName": "André Rodrigues",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mC4VcoLAlPEtIAJVPWTppZJ9S3ou2QH75zgDkJrgn0=s64",
      "userId": "01975921740661022202"
     },
     "user_tz": 180
    },
    "id": "oKqFzqTMVc-k",
    "outputId": "78006b8c-835e-4756-b4bc-17f41b5c1021"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(49459, 4)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('https://s3.amazonaws.com/aulas-fiap/imdb-reviews-pt-br.csv')\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### a) Fazendo o embaralhamento dos dados, pois o dataset está ordenado."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.utils import shuffle\n",
    "df = shuffle(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 624,
     "status": "ok",
     "timestamp": 1575929651734,
     "user": {
      "displayName": "André Rodrigues",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mC4VcoLAlPEtIAJVPWTppZJ9S3ou2QH75zgDkJrgn0=s64",
      "userId": "01975921740661022202"
     },
     "user_tz": 180
    },
    "id": "RuOeB3bhugBu",
    "outputId": "fd1707ae-1c05-4a6b-e601-1fa3f88c31dd"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text_en</th>\n",
       "      <th>text_pt</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>24689</th>\n",
       "      <td>24691</td>\n",
       "      <td>What can I say? The little kid inside has alwa...</td>\n",
       "      <td>O que posso dizer? A criança dentro sempre tev...</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22529</th>\n",
       "      <td>22531</td>\n",
       "      <td>The best Batman movie of the 90s no doubt abou...</td>\n",
       "      <td>O melhor filme de Batman dos anos 90, sem dúvi...</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29351</th>\n",
       "      <td>29353</td>\n",
       "      <td>By my \"Kool-Aid drinkers\" remark, I mean that ...</td>\n",
       "      <td>Pelo meu comentário \"Kool-Aid bebedores\", quer...</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28473</th>\n",
       "      <td>28475</td>\n",
       "      <td>Assassin Hausers John Cusak mission is to whac...</td>\n",
       "      <td>Assassin Hausers A missão de John Cusak é atac...</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39594</th>\n",
       "      <td>39596</td>\n",
       "      <td>Ive seen this movie after watching Paltrows ve...</td>\n",
       "      <td>Eu vi esse filme depois de assistir a versão d...</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17273</th>\n",
       "      <td>17275</td>\n",
       "      <td>Usually, Alan Alda plays characters that are t...</td>\n",
       "      <td>Normalmente, Alan Alda interpreta personagens ...</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36066</th>\n",
       "      <td>36068</td>\n",
       "      <td>Now for the truth, its very very weak storylin...</td>\n",
       "      <td>Agora, para a verdade, seu enredo muito fraco ...</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16292</th>\n",
       "      <td>16294</td>\n",
       "      <td>In 1895, in a small village in Japan, the wife...</td>\n",
       "      <td>Em 1895, em uma pequena aldeia no Japão, a esp...</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7210</th>\n",
       "      <td>7212</td>\n",
       "      <td>Dr. Chopper starts shortly after teenager Nich...</td>\n",
       "      <td>Dr. Chopper começa logo depois que o adolescen...</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>726</th>\n",
       "      <td>727</td>\n",
       "      <td>The characterization in this movie is among th...</td>\n",
       "      <td>A caracterização deste filme está entre as pio...</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17906</th>\n",
       "      <td>17908</td>\n",
       "      <td>Pretty funny stuff. Charlie was still working ...</td>\n",
       "      <td>Coisas muito engraçadas. Charlie ainda estava ...</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35841</th>\n",
       "      <td>35843</td>\n",
       "      <td>Here goes the perfect example of what not to d...</td>\n",
       "      <td>Aqui vai o exemplo perfeito do que não fazer q...</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20842</th>\n",
       "      <td>20844</td>\n",
       "      <td>Clearly patterned after the first gangster mov...</td>\n",
       "      <td>Claramente padronizado após os primeiros filme...</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27451</th>\n",
       "      <td>27453</td>\n",
       "      <td>Absolutely the worst film yet by Burton, who s...</td>\n",
       "      <td>Absolutamente o pior filme de Burton, que pare...</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22896</th>\n",
       "      <td>22898</td>\n",
       "      <td>r#88 Brilliant, very entertaining show with sp...</td>\n",
       "      <td>r # 88 Show brilhante, muito divertido, com ef...</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          id                                            text_en  \\\n",
       "24689  24691  What can I say? The little kid inside has alwa...   \n",
       "22529  22531  The best Batman movie of the 90s no doubt abou...   \n",
       "29351  29353  By my \"Kool-Aid drinkers\" remark, I mean that ...   \n",
       "28473  28475  Assassin Hausers John Cusak mission is to whac...   \n",
       "39594  39596  Ive seen this movie after watching Paltrows ve...   \n",
       "17273  17275  Usually, Alan Alda plays characters that are t...   \n",
       "36066  36068  Now for the truth, its very very weak storylin...   \n",
       "16292  16294  In 1895, in a small village in Japan, the wife...   \n",
       "7210    7212  Dr. Chopper starts shortly after teenager Nich...   \n",
       "726      727  The characterization in this movie is among th...   \n",
       "17906  17908  Pretty funny stuff. Charlie was still working ...   \n",
       "35841  35843  Here goes the perfect example of what not to d...   \n",
       "20842  20844  Clearly patterned after the first gangster mov...   \n",
       "27451  27453  Absolutely the worst film yet by Burton, who s...   \n",
       "22896  22898  r#88 Brilliant, very entertaining show with sp...   \n",
       "\n",
       "                                                 text_pt sentiment  \n",
       "24689  O que posso dizer? A criança dentro sempre tev...       pos  \n",
       "22529  O melhor filme de Batman dos anos 90, sem dúvi...       pos  \n",
       "29351  Pelo meu comentário \"Kool-Aid bebedores\", quer...       neg  \n",
       "28473  Assassin Hausers A missão de John Cusak é atac...       neg  \n",
       "39594  Eu vi esse filme depois de assistir a versão d...       pos  \n",
       "17273  Normalmente, Alan Alda interpreta personagens ...       pos  \n",
       "36066  Agora, para a verdade, seu enredo muito fraco ...       neg  \n",
       "16292  Em 1895, em uma pequena aldeia no Japão, a esp...       pos  \n",
       "7210   Dr. Chopper começa logo depois que o adolescen...       neg  \n",
       "726    A caracterização deste filme está entre as pio...       neg  \n",
       "17906  Coisas muito engraçadas. Charlie ainda estava ...       pos  \n",
       "35841  Aqui vai o exemplo perfeito do que não fazer q...       neg  \n",
       "20842  Claramente padronizado após os primeiros filme...       pos  \n",
       "27451  Absolutamente o pior filme de Burton, que pare...       neg  \n",
       "22896  r # 88 Show brilhante, muito divertido, com ef...       pos  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2) Checking for missing values\n",
    "**É fácil checar missing values usando método isnull() com o método sum(), o número retornado condiz com a quantidade True para o teste, ou seja, quantidade de valores nulos nas variaveis**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id           0\n",
       "text_en      0\n",
       "text_pt      0\n",
       "sentiment    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 170
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 615,
     "status": "ok",
     "timestamp": 1575929614871,
     "user": {
      "displayName": "André Rodrigues",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mC4VcoLAlPEtIAJVPWTppZJ9S3ou2QH75zgDkJrgn0=s64",
      "userId": "01975921740661022202"
     },
     "user_tz": 180
    },
    "id": "oz0V7StwdK8s",
    "outputId": "f827162a-a36e-408e-867d-5c9b0f67c5c0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 49459 entries, 24689 to 2133\n",
      "Data columns (total 4 columns):\n",
      "id           49459 non-null int64\n",
      "text_en      49459 non-null object\n",
      "text_pt      49459 non-null object\n",
      "sentiment    49459 non-null object\n",
      "dtypes: int64(1), object(3)\n",
      "memory usage: 1.9+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### a) Como estamos construindo um modelo para classificar o sentimento, nosso alvo será a variável \"sentiment\" do dataframe 'df'.\n",
    "**Para ter certeza de que é uma variável binária, vamos usar a função countplot () do Seaborn.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 606,
     "status": "ok",
     "timestamp": 1575929624541,
     "user": {
      "displayName": "André Rodrigues",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mC4VcoLAlPEtIAJVPWTppZJ9S3ou2QH75zgDkJrgn0=s64",
      "userId": "01975921740661022202"
     },
     "user_tz": 180
    },
    "id": "ArKQ3to-dQFd",
    "outputId": "0b89ff15-03ce-4a6e-de34-7b5f75e0f696"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "neg    24765\n",
       "pos    24694\n",
       "Name: sentiment, dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sentiment.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEWCAYAAACnlKo3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAbj0lEQVR4nO3de7hddX3n8ffHBBBRapBIkSCxGKugFTUCXtqiWG5jG3W8QCsEZAbbB2pt7VNp7QheGHVGZdAqijUC1YrUG4FBMTKIVy6hUjCiJQWESIBAQKBYLfidP9bvyPZkJzmsZJ+Tw3m/nmc/e63fuv3WPvvsz16/39prpaqQJKmPR0x1BSRJ05chIknqzRCRJPVmiEiSejNEJEm9GSKSpN4MEW1WSX47yQ8Hxn8zyXeT3JPkDUk+kuR/THKdjkzyzS19nZMhyQ1JXjLV9VifJE9Mcm+SWVNdF03M7KmugKZOkguAS6vqrePKFwEfBeZV1f0PZZ1V9Q3gNweK/gr4WlU9a1Prq4efJDcA/62qvgpQVTcCj56iuhSwoKpWTsX2pyuPRGa204HDk2Rc+eHApx5qgCQZ9qVkN2BFv+pJ2tIZIjPbF4EdgN8eK0gyB3gpcGYb3ybJe5PcmOTW1hy1bZu2X5JVSd6c5BbgE2Nlbfr/A14E/F1ronhKktOTvHNge4uSXJnk7iT/luSgVn5UkmtaM9h1SV4/sMzYdt+U5LYkq5McNTD9cUmWtnVeBuw+uNNJnp/k8iQ/ac/PX98LlGTXJJ9PsibJHUn+bj3znZLkprbNK5IMvqZ7J1nept2a5P2t/JFJPtnWe1ery05t2q8l+Xjbtx8needYE0+SJye5uNX/9iSf2UD9D0/yo7aNt4yb9ogkx7fX/Y4kZyfZYT3r2THJea2ea5N8I8kj2rQnJPlce42uT/KGgeVObOs9s/0tVyRZ2Kb9A/BE4Nz2/virJPOT1NgXkiRfa/v+7TbPue3v+6n2el6eZP7A9p6aZFmr4w+TvHpg2ulJPpTk/7a6XJpk9zbt6222f2nbec36XlONU1U+ZvAD+Bjw9wPjrweuHBj/P8BSurB5DHAu8K42bT/gfuA9wDbAtq1s1cDyX6NrrhgbPx14ZxveG/gJ8Ht0X2h2AZ7apv0Xug//AL8L3Ac8e9x23w5sBRzSps9p088Czga2A54O/Bj4Zpu2A3An3dHWbOCwNv64Ia/NLOBfgJPbuh4JvLBNO3JsnW38tcDj2jrfBNwCPLJN+w5weBt+NLDvwGt9LvCotq3nANu3aV+ka1LcDng8cBnw+jbt08Bb2mv2yzoNqf8ewL3A77S/z/vb6/aSNv2NwCXAvDb9o8Cn17OudwEfaa/3VnRfPNLqcAXwVmBr4DeA64AD23InAv/R/kaz2nouGVjvDWP1aePzgQJmD7x/Vrb3wq8B3wf+FXhJe63PBD7R5t0OuAk4qk17NnA7sOfAe28t3ftuNvAp4KyBbRfw5Kn+n5xujymvgI8pfgPAC+k+yLdt498C/rwNB/h3YPeB+Z8HXN+G9wN+PvZhOVA20RD5KHDyBOv5ReDPBrbx07EPmlZ2G7Bv+6D6T1oYtWn/kwdD5HDgsnHr/g5w5JBtPg9YM7idgWlHMhAiQ6bfCTyzDX8deBuw47h5Xgd8G/itceU7AT8b+5u0ssOAi9rwmcBpdH1WG3rN3jruQ3K79vcaC5FrgP0Hpu/cXrth+/t24JzxH7LAPsCN48r+mgc/2E8EvjowbQ/gpwPjN7DxEHnLwPT3AV8aGP992pce4DXAN8bV5aPACQPvvcEvTIcAPxgYN0R6PGzOmuGq6pt0H5SLkvwG8FzgH9vkuXTfkq9ozRh3AV9u5WPWVNV/9Nz8rsC/DZuQ5OAkl7Rmibvo/uF3HJjljvrVPpv76L7lz6X7lnnTwLQfDQw/Ydz42PRd1lO/H9UE+oZa09o1rYnpLrpvzWP1PRp4CvCD1vzy0lb+D8AFwFlJbk7yv5JsRdePtBWweuB1/yjdEQl0JysEuKw1D71uPdV6AgOvQ1X9O3DHwPTdgC8MbOMa4AG6EBvvf9MdEXwlXfPi8QPreMLYOtp6/mbcOm4ZGL4PeGSG95+tz60Dwz8dMj7WEb8bsM+4uvwR8OsbqMuUdOI/nHh2lqD7ZnsE3VlVX6mqsX/S2+n+Sfesqh+vZ9lNuQz0TYzrr4CuHwb4XKvTOVX1n0m+SPfBuTFr6JpsdgV+0MqeODD9ZroPm0FPpAvHYfV7YpLZGwqS1v/xZmB/YEVV/SLJnWP1raprgcNaH8IrgM8meVz7UH8b8LbWrn8+8MP2/DO6I5d1tltVtwD/vW37hcBXk3y91j2raDXwtIF6PoquyW1w/15XVd9a374NbPMeuma6NyXZE7goyeVtHddX1YKNrWN9q+653DA3ARdX1e9txnVqIzwSEXQh8hK6D6Yzxgqr6hd0fSYnJ3k8QJJdkhy4mbb7ceCoJPu3Tt5dkjyVrm19G1ogJDkYOGAiK6yqB4DPAycmeVSSPYDFA7OcDzwlyR8mmd06UPcAzhuyusvoPojfnWS7dB3hLxgy32PogmsNMDvJW4HtxyYmeW2Sue31vKsVP5DkRUmeka7D/G66pqQHqmo18BXgfUm2b6/N7kl+t63vVUnmtfXcSfdB/MCQen0WeGmSFybZmq5JavB//iPASUl2a+udm+707nUkeWm6Dv20uj7QHpcBd6c7uWLbJLOSPD3Jc4etZ4hb6fpRNofz6P62hyfZqj2em+RpG11y89dlxjBERFXdQNc2vx1dJ/qgN9M1Y1yS5G7gq/zq70A2ZbuX0XWCnkzXL3MxsFv71vsGus7xO4E/HFKvDTmOrpniFrp28E8MbPMOurPP3kTXtPNXwEur6vYh9XuArs39ycCNwCq6dvfxLgC+RNfh+yO6juTB5rSDgBVJ7gVOAQ5tTYC/TvdBfzddU9LFwCfbMkfQhen322vwWbo+C+iaHC9t61tK11d0/ZD6rwCOpWueXN3Ws2pgllPa8l9Jcg9dJ/s+Q/YPYAHd3/5euj6kD1fV1wZeo72A6+mOXv+erjlvIt4F/G1rfvrLCS4zVHvfHAAcSnfEeQsPnvQxEScCZ7S6vHpjM6uT1qEkSdJD5pGIJKk3Q0SS1JshIknqzRCRJPU2434nsuOOO9b8+fOnuhqSNK1cccUVt1fV3PHlMy5E5s+fz/Lly6e6GpI0rSQZf6UHwOYsSdImMEQkSb0ZIpKk3gwRSVJvIwuRdHeEu6hdHntFkj9r5Semu1Pble1xyMAyf51kZbsj2YED5Qe1spUDl6AmyZPS3Z3s2iSfaReZkyRNklEeidwPvKmqnkZ3s6Bj2xVVobsR0V7tcT5Am3YosCfdBes+3K4IOgv4EHAw3dVWDxtYz3vauhbQXVzu6BHujyRpnJGFSFWtrqp/bsP30F2ldNiNf8YsorsL28/aFUlX0t3Gcm9gZVVdV1U/p7v16aJ2SeoX013dFLpLmL9sNHsjSRpmUvpE2g13ngVc2oqOS3JVkiVJ5rSyXfjVy2evamXrK38ccNfATXvGyodt/5gky5MsX7NmzWbYI0kSTEKIJHk03V3q3lhVdwOn0t3Nbi+6exy8b2zWIYtXj/J1C6tOq6qFVbVw7tx1fnApSepppL9Yb/eL/hzwqar6PMDArVdJ8jEevKPcKrpbmo6ZR3djGdZTfjvw2IFblw7OPzLL3/DHo96EpqGFH/jIVFcBgD/+tldj0Lo+8vyFI1v3KM/OCt3tT6+pqvcPlO88MNvLge+14aXAoUm2SfIkujupXQZcDixoZ2JtTdf5vrS6u2ldBLyyLb8YOGdU+yNJWtcoj0ReABwOXJ3kylb2N3RnV+1F1/R0A/B66G7lmeRsutuB3g8c2269SZLj6G5BOgtY0m77Cd2tW89K8k7gu3ShJUmaJCMLkar6JsP7Lc7fwDInAScNKT9/2HJVdR3d2VuSpCngL9YlSb0ZIpKk3gwRSVJvhogkqTdDRJLUmyEiSerNEJEk9WaISJJ6M0QkSb0ZIpKk3gwRSVJvhogkqTdDRJLUmyEiSerNEJEk9WaISJJ6M0QkSb0ZIpKk3gwRSVJvhogkqTdDRJLUmyEiSerNEJEk9WaISJJ6M0QkSb0ZIpKk3gwRSVJvhogkqTdDRJLUmyEiSerNEJEk9WaISJJ6G1mIJNk1yUVJrkmyIsmftfIdkixLcm17ntPKk+QDSVYmuSrJswfWtbjNf22SxQPlz0lydVvmA0kyqv2RJK1rlEci9wNvqqqnAfsCxybZAzgeuLCqFgAXtnGAg4EF7XEMcCp0oQOcAOwD7A2cMBY8bZ5jBpY7aIT7I0kaZ2QhUlWrq+qf2/A9wDXALsAi4Iw22xnAy9rwIuDM6lwCPDbJzsCBwLKqWltVdwLLgIPatO2r6jtVVcCZA+uSJE2CSekTSTIfeBZwKbBTVa2GLmiAx7fZdgFuGlhsVSvbUPmqIeWSpEky8hBJ8mjgc8Abq+ruDc06pKx6lA+rwzFJlidZvmbNmo1VWZI0QSMNkSRb0QXIp6rq86341tYURXu+rZWvAnYdWHwecPNGyucNKV9HVZ1WVQurauHcuXM3backSb80yrOzAnwcuKaq3j8waSkwdobVYuCcgfIj2lla+wI/ac1dFwAHJJnTOtQPAC5o0+5Jsm/b1hED65IkTYLZI1z3C4DDgauTXNnK/gZ4N3B2kqOBG4FXtWnnA4cAK4H7gKMAqmptkncAl7f53l5Va9vwnwCnA9sCX2oPSdIkGVmIVNU3Gd5vAbD/kPkLOHY961oCLBlSvhx4+iZUU5K0CfzFuiSpN0NEktSbISJJ6s0QkST1ZohIknozRCRJvRkikqTeDBFJUm+GiCSpN0NEktSbISJJ6s0QkST1ZohIknozRCRJvRkikqTeDBFJUm+GiCSpN0NEktSbISJJ6s0QkST1ZohIknozRCRJvRkikqTeDBFJUm+GiCSpN0NEktSbISJJ6s0QkST1ZohIknozRCRJvRkikqTeDBFJUm+GiCSpt5GFSJIlSW5L8r2BshOT/DjJle1xyMC0v06yMskPkxw4UH5QK1uZ5PiB8icluTTJtUk+k2TrUe2LJGm4UR6JnA4cNKT85Kraqz3OB0iyB3AosGdb5sNJZiWZBXwIOBjYAziszQvwnrauBcCdwNEj3BdJ0hAjC5Gq+jqwdoKzLwLOqqqfVdX1wEpg7/ZYWVXXVdXPgbOARUkCvBj4bFv+DOBlm3UHJEkbNRV9Iscluao1d81pZbsANw3Ms6qVra/8ccBdVXX/uPKhkhyTZHmS5WvWrNlc+yFJM95kh8ipwO7AXsBq4H2tPEPmrR7lQ1XVaVW1sKoWzp0796HVWJK0XrMnc2NVdevYcJKPAee10VXArgOzzgNubsPDym8HHptkdjsaGZxfkjRJJvVIJMnOA6MvB8bO3FoKHJpkmyRPAhYAlwGXAwvamVhb03W+L62qAi4CXtmWXwycMxn7IEl60MiORJJ8GtgP2DHJKuAEYL8ke9E1Pd0AvB6gqlYkORv4PnA/cGxVPdDWcxxwATALWFJVK9om3gycleSdwHeBj49qXyRJw40sRKrqsCHF6/2gr6qTgJOGlJ8PnD+k/Dq6s7ckSVNkQs1ZSS6cSJkkaWbZ4JFIkkcCj6JrkprDg2dFbQ88YcR1kyRt4TbWnPV64I10gXEFD4bI3XS/JJckzWAbDJGqOgU4JcmfVtUHJ6lOkqRpYkId61X1wSTPB+YPLlNVZ46oXpKkaWBCIZLkH+h+aX4l8EArLsAQkaQZbKKn+C4E9mg/8pMkCZj4L9a/B/z6KCsiSZp+JnoksiPw/SSXAT8bK6yqPxhJrSRJ08JEQ+TEUVZCkjQ9TfTsrItHXRFJ0vQz0bOz7uHB+3VsDWwF/HtVbT+qikmStnwTPRJ5zOB4kpfhxQ8lacbrdT+Rqvoi3T3OJUkz2ESbs14xMPoIut+N+JsRSZrhJnp21u8PDN9Pd0OpRZu9NpKkaWWifSJHjboikqTpZ6I3pZqX5AtJbktya5LPJZk36spJkrZsE+1Y/wSwlO6+IrsA57YySdIMNtEQmVtVn6iq+9vjdGDuCOslSZoGJhoityd5bZJZ7fFa4I5RVkyStOWbaIi8Dng1cAuwGnglYGe7JM1wEz3F9x3A4qq6EyDJDsB76cJFkjRDTfRI5LfGAgSgqtYCzxpNlSRJ08VEQ+QRSeaMjbQjkYkexUiSHqYmGgTvA76d5LN0lzt5NXDSyGolSZoWJvqL9TOTLKe76GKAV1TV90daM0nSFm/CTVItNAwOSdIv9boUvCRJYIhIkjaBISJJ6s0QkST1ZohIknobWYgkWdLuP/K9gbIdkixLcm17ntPKk+QDSVYmuSrJsweWWdzmvzbJ4oHy5yS5ui3zgSQZ1b5IkoYb5ZHI6cBB48qOBy6sqgXAhW0c4GBgQXscA5wKv/xl/AnAPsDewAkDv5w/tc07ttz4bUmSRmxkIVJVXwfWjiteBJzRhs8AXjZQfmZ1LgEem2Rn4EBgWVWtbdfuWgYc1KZtX1XfqaoCzhxYlyRpkkx2n8hOVbUaoD0/vpXvAtw0MN+qVrah8lVDyodKckyS5UmWr1mzZpN3QpLU2VI61of1Z1SP8qGq6rSqWlhVC+fO9YaMkrS5THaI3NqaomjPt7XyVcCuA/PNA27eSPm8IeWSpEk02SGyFBg7w2oxcM5A+RHtLK19gZ+05q4LgAOSzGkd6gcAF7Rp9yTZt52VdcTAuiRJk2Rk9wRJ8mlgP2DHJKvozrJ6N3B2kqOBG4FXtdnPBw4BVgL30W69W1Vrk7wDuLzN9/Z2QyyAP6E7A2xb4EvtIUmaRCMLkao6bD2T9h8ybwHHrmc9S4AlQ8qXA0/flDpKkjbNltKxLkmahgwRSVJvhogkqTdDRJLUmyEiSerNEJEk9WaISJJ6M0QkSb0ZIpKk3gwRSVJvhogkqTdDRJLUmyEiSerNEJEk9WaISJJ6M0QkSb0ZIpKk3gwRSVJvhogkqTdDRJLUmyEiSerNEJEk9WaISJJ6M0QkSb0ZIpKk3gwRSVJvhogkqTdDRJLUmyEiSerNEJEk9WaISJJ6M0QkSb0ZIpKk3qYkRJLckOTqJFcmWd7KdkiyLMm17XlOK0+SDyRZmeSqJM8eWM/iNv+1SRZPxb5I0kw2lUciL6qqvapqYRs/HriwqhYAF7ZxgIOBBe1xDHAqdKEDnADsA+wNnDAWPJKkybElNWctAs5ow2cALxsoP7M6lwCPTbIzcCCwrKrWVtWdwDLgoMmutCTNZFMVIgV8JckVSY5pZTtV1WqA9vz4Vr4LcNPAsqta2frK15HkmCTLkyxfs2bNZtwNSZrZZk/Rdl9QVTcneTywLMkPNjBvhpTVBsrXLaw6DTgNYOHChUPnkSQ9dFNyJFJVN7fn24Av0PVp3NqaqWjPt7XZVwG7Diw+D7h5A+WSpEky6SGSZLskjxkbBg4AvgcsBcbOsFoMnNOGlwJHtLO09gV+0pq7LgAOSDKndagf0MokSZNkKpqzdgK+kGRs+/9YVV9OcjlwdpKjgRuBV7X5zwcOAVYC9wFHAVTV2iTvAC5v8729qtZO3m5IkiY9RKrqOuCZQ8rvAPYfUl7AsetZ1xJgyeauoyRpYrakU3wlSdOMISJJ6s0QkST1ZohIknozRCRJvRkikqTeDBFJUm+GiCSpN0NEktSbISJJ6s0QkST1ZohIknozRCRJvRkikqTeDBFJUm+GiCSpN0NEktSbISJJ6s0QkST1ZohIknozRCRJvRkikqTeDBFJUm+GiCSpN0NEktSbISJJ6s0QkST1ZohIknozRCRJvRkikqTeDBFJUm+GiCSpN0NEktTbtA+RJAcl+WGSlUmOn+r6SNJMMq1DJMks4EPAwcAewGFJ9pjaWknSzDGtQwTYG1hZVddV1c+Bs4BFU1wnSZoxZk91BTbRLsBNA+OrgH3Gz5TkGOCYNnpvkh9OQt1mgh2B26e6EluED350qmugdfn+bDbTu3O3YYXTPUQypKzWKag6DTht9NWZWZIsr6qFU10PaRjfn5NjujdnrQJ2HRifB9w8RXWRpBlnuofI5cCCJE9KsjVwKLB0iuskSTPGtG7Oqqr7kxwHXADMApZU1YoprtZMYhOhtmS+PydBqtbpQpAkaUKme3OWJGkKGSKSpN4MEUlSb4aIJKk3Q0QblGR+kh8kOSPJVUk+m+RRSfZP8t0kVydZkmSbNv+7k3y/zfveqa6/Hp7a+/KaJB9LsiLJV5Jsm2T3JF9OckWSbyR5apt/9ySXJLk8yduT3DvV+/BwYYhoIn4TOK2qfgu4G/gL4HTgNVX1DLpTxf8kyQ7Ay4E927zvnKL6amZYAHyoqvYE7gL+K91pvX9aVc8B/hL4cJv3FOCUqnou/iB5szJENBE3VdW32vAngf2B66vqX1vZGcDv0AXMfwB/n+QVwH2TXlPNJNdX1ZVt+ApgPvB84J+SXEl3yaid2/TnAf/Uhv9xMiv5cDetf2yoSTOhHxO1H3/uTRcyhwLHAS8eZcU0o/1sYPgBYCfgrqraa4rqMyN5JKKJeGKS57Xhw4CvAvOTPLmVHQ5cnOTRwK9V1fnAGwH/mTWZ7gauT/IqgHSe2aZdQtfcBd0XHG0mhogm4hpgcZKrgB2Ak4Gj6JoNrgZ+AXwEeAxwXpvvYuDPp6i+mrn+CDg6yb8AK3jw/kJvBP4iyWV0TVw/maL6Pex42RNtUJL5wHlV9fQprorUW5JHAT+tqkpyKHBYVXkDu83APhFJM8FzgL9LErozuV43xfV52PBIRJLUm30ikqTeDBFJUm+GiCSpN0NEmiRJ9kpyyMD4HyQ5fsTb3C/J80e5Dc1shog0efYCfhkiVbW0qt494m3uR3cpEGkkPDtLmoAk2wFnA/OAWcA7gJXA+4FHA7cDR1bV6iRfAy4FXgQ8Fji6ja8EtgV+DLyrDS+squOSnA78FHgqsBvdjzkX013z6dKqOrLV4wDgbcA2wL8BR1XVvUluoLuG2e8DWwGvoruO2SV0lwRZQ3dhwm+M4vXRzOWRiDQxBwE3V9Uz2w8vvwx8EHhlu2LsEuCkgflnV9XedL+UPqGqfg68FfhMVe1VVZ8Zso05dNca+3PgXLorA+wJPKM1he0I/C3wkqp6NrCc7orKY25v5acCf1lVN9BdSeDktk0DRJudPzaUJuZq4L1J3gOcB9wJPB1Y1v1+jVnA6oH5P9+ex64uOxHntl9UXw3cWlVXAyRZ0dYxD9gD+Fbb5tbAd9azzVc8hH2TejNEpAmoqn9N8hy6Po13AcuAFVX1vPUsMnaF2QeY+P/Z2DK/4FevUPuLto4HgGVVddhm3Ka0SWzOkiYgyROA+6rqk8B7gX2AuWNXN06yVZI9N7Kae+guUtnXJcALxq6e3O4w+ZQRb1PaIENEmphnAJe1mx29ha5/45XAe9oVY69k42dBXQTskeTKJK95qBWoqjXAkcCn25WSL6HriN+Qc4GXt23+9kPdprQxnp0lSerNIxFJUm+GiCSpN0NEktSbISJJ6s0QkST1ZohIknozRCRJvf1/Ibzpi9SrK10AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot = sb.countplot(x='sentiment',data=df, palette='hls')\n",
    "plt.title(\"Verificando classes de sentiment\")\n",
    "plt.show(plot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3) Tratamento dos dados"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### a) Aplicando One Hot Encoding na classe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['classe'] = df['sentiment'].replace(['neg','pos'],[0,1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text_en</th>\n",
       "      <th>text_pt</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>classe</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>24689</th>\n",
       "      <td>24691</td>\n",
       "      <td>What can I say? The little kid inside has alwa...</td>\n",
       "      <td>O que posso dizer? A criança dentro sempre tev...</td>\n",
       "      <td>pos</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22529</th>\n",
       "      <td>22531</td>\n",
       "      <td>The best Batman movie of the 90s no doubt abou...</td>\n",
       "      <td>O melhor filme de Batman dos anos 90, sem dúvi...</td>\n",
       "      <td>pos</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29351</th>\n",
       "      <td>29353</td>\n",
       "      <td>By my \"Kool-Aid drinkers\" remark, I mean that ...</td>\n",
       "      <td>Pelo meu comentário \"Kool-Aid bebedores\", quer...</td>\n",
       "      <td>neg</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28473</th>\n",
       "      <td>28475</td>\n",
       "      <td>Assassin Hausers John Cusak mission is to whac...</td>\n",
       "      <td>Assassin Hausers A missão de John Cusak é atac...</td>\n",
       "      <td>neg</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39594</th>\n",
       "      <td>39596</td>\n",
       "      <td>Ive seen this movie after watching Paltrows ve...</td>\n",
       "      <td>Eu vi esse filme depois de assistir a versão d...</td>\n",
       "      <td>pos</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          id                                            text_en  \\\n",
       "24689  24691  What can I say? The little kid inside has alwa...   \n",
       "22529  22531  The best Batman movie of the 90s no doubt abou...   \n",
       "29351  29353  By my \"Kool-Aid drinkers\" remark, I mean that ...   \n",
       "28473  28475  Assassin Hausers John Cusak mission is to whac...   \n",
       "39594  39596  Ive seen this movie after watching Paltrows ve...   \n",
       "\n",
       "                                                 text_pt sentiment  classe  \n",
       "24689  O que posso dizer? A criança dentro sempre tev...       pos       1  \n",
       "22529  O melhor filme de Batman dos anos 90, sem dúvi...       pos       1  \n",
       "29351  Pelo meu comentário \"Kool-Aid bebedores\", quer...       neg       0  \n",
       "28473  Assassin Hausers A missão de John Cusak é atac...       neg       0  \n",
       "39594  Eu vi esse filme depois de assistir a versão d...       pos       1  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### b) Passando todos caracteres para letra minúscula"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "id": "rAz1ZRXwtag-",
    "outputId": "709c13c8-757d-4d40-8309-1d738bddf1cb"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     /Users/andrerodrigues/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('stopwords')\n",
    "\n",
    "stopwords = nltk.corpus.stopwords.words('portuguese')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text_en</th>\n",
       "      <th>text_pt</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>classe</th>\n",
       "      <th>texto_proc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>24689</th>\n",
       "      <td>24691</td>\n",
       "      <td>What can I say? The little kid inside has alwa...</td>\n",
       "      <td>O que posso dizer? A criança dentro sempre tev...</td>\n",
       "      <td>pos</td>\n",
       "      <td>1</td>\n",
       "      <td>o que posso dizer? a criança dentro sempre tev...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22529</th>\n",
       "      <td>22531</td>\n",
       "      <td>The best Batman movie of the 90s no doubt abou...</td>\n",
       "      <td>O melhor filme de Batman dos anos 90, sem dúvi...</td>\n",
       "      <td>pos</td>\n",
       "      <td>1</td>\n",
       "      <td>o melhor filme de batman dos anos 90, sem dúvi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29351</th>\n",
       "      <td>29353</td>\n",
       "      <td>By my \"Kool-Aid drinkers\" remark, I mean that ...</td>\n",
       "      <td>Pelo meu comentário \"Kool-Aid bebedores\", quer...</td>\n",
       "      <td>neg</td>\n",
       "      <td>0</td>\n",
       "      <td>pelo meu comentário \"kool-aid bebedores\", quer...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28473</th>\n",
       "      <td>28475</td>\n",
       "      <td>Assassin Hausers John Cusak mission is to whac...</td>\n",
       "      <td>Assassin Hausers A missão de John Cusak é atac...</td>\n",
       "      <td>neg</td>\n",
       "      <td>0</td>\n",
       "      <td>assassin hausers a missão de john cusak é atac...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39594</th>\n",
       "      <td>39596</td>\n",
       "      <td>Ive seen this movie after watching Paltrows ve...</td>\n",
       "      <td>Eu vi esse filme depois de assistir a versão d...</td>\n",
       "      <td>pos</td>\n",
       "      <td>1</td>\n",
       "      <td>eu vi esse filme depois de assistir a versão d...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          id                                            text_en  \\\n",
       "24689  24691  What can I say? The little kid inside has alwa...   \n",
       "22529  22531  The best Batman movie of the 90s no doubt abou...   \n",
       "29351  29353  By my \"Kool-Aid drinkers\" remark, I mean that ...   \n",
       "28473  28475  Assassin Hausers John Cusak mission is to whac...   \n",
       "39594  39596  Ive seen this movie after watching Paltrows ve...   \n",
       "\n",
       "                                                 text_pt sentiment  classe  \\\n",
       "24689  O que posso dizer? A criança dentro sempre tev...       pos       1   \n",
       "22529  O melhor filme de Batman dos anos 90, sem dúvi...       pos       1   \n",
       "29351  Pelo meu comentário \"Kool-Aid bebedores\", quer...       neg       0   \n",
       "28473  Assassin Hausers A missão de John Cusak é atac...       neg       0   \n",
       "39594  Eu vi esse filme depois de assistir a versão d...       pos       1   \n",
       "\n",
       "                                              texto_proc  \n",
       "24689  o que posso dizer? a criança dentro sempre tev...  \n",
       "22529  o melhor filme de batman dos anos 90, sem dúvi...  \n",
       "29351  pelo meu comentário \"kool-aid bebedores\", quer...  \n",
       "28473  assassin hausers a missão de john cusak é atac...  \n",
       "39594  eu vi esse filme depois de assistir a versão d...  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['texto_proc'] = df.text_pt.apply(lambda texto: texto.lower())\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### c) Eliminando colunas que não interessam para o modelo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>classe</th>\n",
       "      <th>texto_proc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>24689</th>\n",
       "      <td>1</td>\n",
       "      <td>o que posso dizer? a criança dentro sempre tev...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22529</th>\n",
       "      <td>1</td>\n",
       "      <td>o melhor filme de batman dos anos 90, sem dúvi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29351</th>\n",
       "      <td>0</td>\n",
       "      <td>pelo meu comentário \"kool-aid bebedores\", quer...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28473</th>\n",
       "      <td>0</td>\n",
       "      <td>assassin hausers a missão de john cusak é atac...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39594</th>\n",
       "      <td>1</td>\n",
       "      <td>eu vi esse filme depois de assistir a versão d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1266</th>\n",
       "      <td>0</td>\n",
       "      <td>as questões centrais em jogo, god &amp; satan / go...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15262</th>\n",
       "      <td>1</td>\n",
       "      <td>embora eu seja uma garota, felizmente eu tenho...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12611</th>\n",
       "      <td>1</td>\n",
       "      <td>um dos melhores filmes ocidentais de todos os ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12637</th>\n",
       "      <td>1</td>\n",
       "      <td>como um leitor ávido de clive barker, eu realm...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2133</th>\n",
       "      <td>0</td>\n",
       "      <td>se você é um fã da série original, não veja es...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>49459 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       classe                                         texto_proc\n",
       "24689       1  o que posso dizer? a criança dentro sempre tev...\n",
       "22529       1  o melhor filme de batman dos anos 90, sem dúvi...\n",
       "29351       0  pelo meu comentário \"kool-aid bebedores\", quer...\n",
       "28473       0  assassin hausers a missão de john cusak é atac...\n",
       "39594       1  eu vi esse filme depois de assistir a versão d...\n",
       "...       ...                                                ...\n",
       "1266        0  as questões centrais em jogo, god & satan / go...\n",
       "15262       1  embora eu seja uma garota, felizmente eu tenho...\n",
       "12611       1  um dos melhores filmes ocidentais de todos os ...\n",
       "12637       1  como um leitor ávido de clive barker, eu realm...\n",
       "2133        0  se você é um fã da série original, não veja es...\n",
       "\n",
       "[49459 rows x 2 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1 = df.drop(columns=['id','text_en','text_pt','sentiment'], axis=1)\n",
    "df1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4) VECTORIZING"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Enquanto o processo de Tokenização separa o texto em pequenos trechos, normalmente uma palavra ou símbolo, (token) a vetorização é o processo de agregar medidas aos tokens ou n-gramas para gerar features\n",
    "\n",
    "* Uma vez que geramos as features aplicando uma das estratégias de vetorização, fica simples treinar um modelo de machine learning para, por exemplo, classificar documentos."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### VETORIZAÇÃO POR CONTAGEM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# vect = CountVectorizer(ngram_range=(1,1), stop_words=stopwords)\n",
    "# # vect = CountVectorizer(ngram_range=(2,2))\n",
    "# vect.fit(df1.texto_proc)\n",
    "# text_vect = vect.transform(df1.texto_proc)   \n",
    "# print(text_vect.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**O problema na vetorização por contagem é que pode haver uma grande disparidade entre os documentos levando o algoritmo a dar maior relevância para aqueles com maior número de palavras ou tokens repetidos. Portanto, não usaremos essa forma de vetorização.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TUTv5MDZE0jr"
   },
   "source": [
    "#### TF - Frequência de Termos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Consiste na frequência relativa de um mesmo n-grama ou token em um documento."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 223
    },
    "colab_type": "code",
    "id": "Wp4frwerLyx8",
    "outputId": "d3e8cab8-a589-4ae1-b8e4-c8f6b303ff84"
   },
   "outputs": [],
   "source": [
    "# from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "# vect = TfidfVectorizer(ngram_range=(1,1), use_idf=True, norm='l2', stop_words=stopwords)\n",
    "# # vect = TfidfVectorizer(ngram_range=(2,2), use_idf=True, norm='l1')\n",
    "# vect.fit(df1.texto_proc)\n",
    "# tf_vect = vect.transform(df1.texto_proc)\n",
    "# print(tf_vect.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**O problema do TF é que, assim como a Contagem de Termos, os n-gramas que aparecem com maior frequência irão desbalancear as features e nem sempre os termos que se repetem muito agregarão informação. Portanto, não usaremos essa forma de vetorização.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dxG2g5mInypI"
   },
   "source": [
    "#### TF-IDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* É a combinação da frequência de termos multiplicado com um elemento de “peso”,  o inverso da frequência em documentos define o “peso” que será usado."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "alnbcBIeAD5Y"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(49459, 659746)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "import string\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "import string\n",
    "import nltk\n",
    "nltk.download('punkt')\n",
    "\n",
    "trans_table = {ord(x): None for x in string.punctuation + string.digits}    \n",
    "\n",
    "def tokenize(text):\n",
    "        # Remover pontuação e dígitos de qualquer palavra com pelo menos dois caracteres. \n",
    "        tokens = [word for word in nltk.word_tokenize(text.translate(trans_table), language=\"portuguese\") if len(word) > 1] #if len(word) > 1 \n",
    "        return tokens\n",
    "\n",
    "vect = TfidfVectorizer(\n",
    "    ngram_range=(1,2)  # (1, 2) means unigrams and bigrams.\n",
    "    , use_idf=True     # Enable inverse-document-frequency reweighting.\n",
    "    , norm='l2'        # ‘l2’: Sum of squares of vector elements is 1. \n",
    "                       # The cosine similarity between two vectors is their dot product when l2 norm has been applied.\n",
    "    , tokenizer = tokenize  \n",
    "    , stop_words=stopwords\n",
    "    , max_df=0.9         # When building the vocabulary ignore terms that have a document frequency strictly higher than the given threshold\n",
    "    , min_df=2           # When building the vocabulary ignore terms that have a document frequency strictly lower than the given threshold. \n",
    "    , analyzer = 'word'  # Whether the feature should be made of word or character n-grams.\n",
    "    , strip_accents = 'unicode' # Remove accents and perform other character normalization during the preprocessing step.\n",
    ")\n",
    "\n",
    "vect.fit(df1.texto_proc)\n",
    "tfidf_vect = vect.transform(df1.texto_proc)\n",
    "print(tfidf_vect.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Foram testadas diversas combinações de parametrização do vetorizador TF-IDF até encontrar aquela que produzisse a melhor acurácia. Segue abaixo o resumo desses testes:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://i.ibb.co/VT3zNqg/Screen-Shot-2019-12-21-at-07-02-17.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5) Testando acurácia obtida pelos Classificadores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### a) Separating Train and Test samples for KNN, DT, NB, RF and AB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = tfidf_vect\n",
    "# X = StandardScaler(with_mean=False).fit_transform(tfidf_vect) #Verificamos que a normalização dos dados produzia baixa acurácia \n",
    "y = df1.classe\n",
    "\n",
    "X_train,X_test,y_train,y_test = train_test_split(\n",
    "    X, \n",
    "    y, \n",
    "    test_size = 0.2, # Vamos usar a relação 80/20 para separar os dados de treino e teste.\n",
    "    random_state = 13\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5XYETMD3vSXE"
   },
   "source": [
    "#### b) KNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "fs2A6QixvSXF"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8053983016579054\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.81      0.80      0.80      4930\n",
      "           1       0.80      0.81      0.81      4962\n",
      "\n",
      "    accuracy                           0.81      9892\n",
      "   macro avg       0.81      0.81      0.81      9892\n",
      "weighted avg       0.81      0.81      0.81      9892\n",
      "\n",
      "0.8053983016579054\n"
     ]
    }
   ],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "knn = KNeighborsClassifier(n_neighbors = 10)\n",
    "knn.fit(X_train, y_train)\n",
    "y_pred_knn = knn.predict(X_test)\n",
    "accuracy = accuracy_score(y_pred_knn, y_test)\n",
    "print(accuracy) \n",
    "\n",
    "cm = confusion_matrix(y_test, y_pred_knn)\n",
    "print(classification_report(y_test, y_pred_knn))\n",
    "print(metrics.accuracy_score(y_test, y_pred_knn))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### c) Decision Tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7097654670440761\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.71      0.71      0.71      4930\n",
      "           1       0.71      0.71      0.71      4962\n",
      "\n",
      "    accuracy                           0.71      9892\n",
      "   macro avg       0.71      0.71      0.71      9892\n",
      "weighted avg       0.71      0.71      0.71      9892\n",
      "\n",
      "0.7097654670440761\n"
     ]
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "# tree = DecisionTreeClassifier(random_state=42)\n",
    "# tree.fit(X_train, y_train)\n",
    "# print('D Tree: ', tree.score(X, df['classe']))\n",
    "\n",
    "\n",
    "dt = DecisionTreeClassifier(random_state=42)\n",
    "dt.fit(X_train, y_train)\n",
    "y_pred_dt = dt.predict(X_test)\n",
    "accuracy = accuracy_score(y_pred_dt, y_test)\n",
    "print(accuracy)\n",
    "\n",
    "\n",
    "cm = confusion_matrix(y_test, y_pred_dt)\n",
    "print(classification_report(y_test, y_pred_dt))\n",
    "print(metrics.accuracy_score(y_test, y_pred_dt))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MfX7rEgEvSXd"
   },
   "source": [
    "#### d) Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "FWGnrZgKvSXe",
    "outputId": "f0d106cc-545d-468b-8132-f5cf653c9223"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7640517589971695"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "modelRfc= RandomForestClassifier(random_state=1)\n",
    "modelRfc.fit(X_train, y_train)\n",
    "modelRfc.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DH5Rb-2CvSXj"
   },
   "source": [
    "#### e) Boosting utilizando  o algoritmo AdaBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9e5P1n2TvSXk",
    "outputId": "e1ba85fa-b858-4516-8fa0-0eb23b7efa61"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.761928831378892"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "\n",
    "modelADB = AdaBoostClassifier(random_state=1, n_estimators = 25)\n",
    "modelADB.fit(X_train, y_train)\n",
    "modelADB.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### f) Support Vector Machine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### i) Faz a validação via crossvalidation (k-fold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Acuracia(clf,X,y):\n",
    "    resultados = cross_val_predict(clf, X, y, cv=5)\n",
    "    return metrics.accuracy_score(y,resultados)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ii) Treinamento do Modelo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Calculo do MinMaxScaler\n",
    "#https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html\n",
    "#X_std = (X - X.min(axis=0)) / (X.max(axis=0) - X.min(axis=0))\n",
    "#X_scaled = X_std * (max - min) + min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Calculo do StandardScaler\n",
    "#https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html\n",
    "#z = (x - u) / s - where u is the mean of the training samples and and s is the standard deviation of the training samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "pip_1 = Pipeline([\n",
    "#     ('scaler',StandardScaler(with_mean=False)), #Verificamos que a normalização dos dados produzia baixa acurácia \n",
    "    ('clf', svm.SVC())\n",
    "])\n",
    "\n",
    "# pip_2 = Pipeline([\n",
    "# #     ('min_max_scaler', MaxAbsScaler()), #Verificamos que a normalização dos dados produzia baixa acurácia \n",
    "#     ('clf', svm.SVC())\n",
    "# ])\n",
    "\n",
    "# pip_3 = Pipeline([\n",
    "# #     ('scaler',StandardScaler(with_mean=False)),\n",
    "#     ('clf', svm.SVC(kernel='rbf'))\n",
    "# ])\n",
    "\n",
    "pip_4 = Pipeline([\n",
    "#     ('scaler',StandardScaler(with_mean=False)),\n",
    "    ('clf', svm.SVC(kernel='poly'))\n",
    "])\n",
    "\n",
    "pip_5 = Pipeline([\n",
    "#     ('scaler',StandardScaler(with_mean=False)),\n",
    "    ('clf', svm.SVC(kernel='linear'))\n",
    "])\n",
    "\n",
    "pip_6 = Pipeline([\n",
    "#     ('scaler',StandardScaler(with_mean=False)),\n",
    "    ('clf', svm.LinearSVC()) # Similar to SVC with parameter kernel=’linear’, but implemented in terms of liblinear rather than libsvm, \n",
    "                             # so it has more flexibility in the choice of penalties and loss functions and should scale better to large \n",
    "                             # numbers of samples.\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### iii) Teste de acurácia usando o pipeline 'pip_1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5007177662306153"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Acuracia(pip_1, tfidf_vect, df1.classe)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### iv) Teste de acurácia usando o pipeline 'pip_2'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Acuracia(pip_2, tfidf_vect, df1.classe)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### v) Testando Kernels "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Kernel rbf**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Acuracia(pip_3, tfidf_vect, df1.classe) --> Identico ao resultado já obtido no pip_1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Kernel Polynomial**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5007177662306153"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Acuracia(pip_4, tfidf_vect, df1.classe)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Kernel Linear**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9072160779635657"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Acuracia(pip_5, tfidf_vect, df1.classe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9079035160435917"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Acuracia(pip_6, tfidf_vect, df1.classe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A melhor acurácia encontrada com Classificadores foi de 90.79% usando SVM com kernel linear e parametrização default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "TrabalhoFinal_AR_v1.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}