information_retrieval.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.8"
    },
    "colab": {
      "name": "information_retrieval.ipynb",
      "provenance": [],
      "collapsed_sections": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bdkwQ36739Wi"
      },
      "source": [
        "# Information Retrieval\n",
        "This project applies Information Retrieval (IR) knowledge in practice with\n",
        "popular Python toolkits such as NLTK and Whoosh. The test collection is about 4,000 documents from US Government web sites and the topics are 15\n",
        "needs for government information."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QSChF98s8ErB",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 431
        },
        "outputId": "bcdfb9b6-df11-447c-8b71-c836c555477d"
      },
      "source": [
        "!pip install whoosh\n",
        "!pip install pytrec_eval\n",
        "!pip install wget"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting whoosh\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ba/19/24d0f1f454a2c1eb689ca28d2f178db81e5024f42d82729a4ff6771155cf/Whoosh-2.7.4-py2.py3-none-any.whl (468kB)\n",
            "\r\u001b[K     |▊                               | 10kB 17.3MB/s eta 0:00:01\r\u001b[K     |█▍                              | 20kB 6.9MB/s eta 0:00:01\r\u001b[K     |██                              | 30kB 7.9MB/s eta 0:00:01\r\u001b[K     |██▉                             | 40kB 9.0MB/s eta 0:00:01\r\u001b[K     |███▌                            | 51kB 7.2MB/s eta 0:00:01\r\u001b[K     |████▏                           | 61kB 7.6MB/s eta 0:00:01\r\u001b[K     |█████                           | 71kB 8.2MB/s eta 0:00:01\r\u001b[K     |█████▋                          | 81kB 9.1MB/s eta 0:00:01\r\u001b[K     |██████▎                         | 92kB 9.3MB/s eta 0:00:01\r\u001b[K     |███████                         | 102kB 9.8MB/s eta 0:00:01\r\u001b[K     |███████▊                        | 112kB 9.8MB/s eta 0:00:01\r\u001b[K     |████████▍                       | 122kB 9.8MB/s eta 0:00:01\r\u001b[K     |█████████                       | 133kB 9.8MB/s eta 0:00:01\r\u001b[K     |█████████▉                      | 143kB 9.8MB/s eta 0:00:01\r\u001b[K     |██████████▌                     | 153kB 9.8MB/s eta 0:00:01\r\u001b[K     |███████████▏                    | 163kB 9.8MB/s eta 0:00:01\r\u001b[K     |███████████▉                    | 174kB 9.8MB/s eta 0:00:01\r\u001b[K     |████████████▋                   | 184kB 9.8MB/s eta 0:00:01\r\u001b[K     |█████████████▎                  | 194kB 9.8MB/s eta 0:00:01\r\u001b[K     |██████████████                  | 204kB 9.8MB/s eta 0:00:01\r\u001b[K     |██████████████▊                 | 215kB 9.8MB/s eta 0:00:01\r\u001b[K     |███████████████▍                | 225kB 9.8MB/s eta 0:00:01\r\u001b[K     |████████████████                | 235kB 9.8MB/s eta 0:00:01\r\u001b[K     |████████████████▊               | 245kB 9.8MB/s eta 0:00:01\r\u001b[K     |█████████████████▌              | 256kB 9.8MB/s eta 0:00:01\r\u001b[K     |██████████████████▏             | 266kB 9.8MB/s eta 0:00:01\r\u001b[K     |██████████████████▉             | 276kB 9.8MB/s eta 0:00:01\r\u001b[K     |███████████████████▋            | 286kB 9.8MB/s eta 0:00:01\r\u001b[K     |████████████████████▎           | 296kB 9.8MB/s eta 0:00:01\r\u001b[K     |█████████████████████           | 307kB 9.8MB/s eta 0:00:01\r\u001b[K     |█████████████████████▊          | 317kB 9.8MB/s eta 0:00:01\r\u001b[K     |██████████████████████▍         | 327kB 9.8MB/s eta 0:00:01\r\u001b[K     |███████████████████████         | 337kB 9.8MB/s eta 0:00:01\r\u001b[K     |███████████████████████▊        | 348kB 9.8MB/s eta 0:00:01\r\u001b[K     |████████████████████████▌       | 358kB 9.8MB/s eta 0:00:01\r\u001b[K     |█████████████████████████▏      | 368kB 9.8MB/s eta 0:00:01\r\u001b[K     |█████████████████████████▉      | 378kB 9.8MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▋     | 389kB 9.8MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▎    | 399kB 9.8MB/s eta 0:00:01\r\u001b[K     |████████████████████████████    | 409kB 9.8MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▋   | 419kB 9.8MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▍  | 430kB 9.8MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████  | 440kB 9.8MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▊ | 450kB 9.8MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████▌| 460kB 9.8MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 471kB 9.8MB/s \n",
            "\u001b[?25hInstalling collected packages: whoosh\n",
            "Successfully installed whoosh-2.7.4\n",
            "Collecting pytrec_eval\n",
            "  Downloading https://files.pythonhosted.org/packages/2e/03/e6e84df6a7c1265579ab26bbe30ff7f8c22745aa77e0799bba471c0a3a19/pytrec_eval-0.5.tar.gz\n",
            "Building wheels for collected packages: pytrec-eval\n",
            "  Building wheel for pytrec-eval (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pytrec-eval: filename=pytrec_eval-0.5-cp36-cp36m-linux_x86_64.whl size=263773 sha256=ba9bcffb130566ec3b1d58b2b65234537729f431671d3547b2f7ab90ccf01fba\n",
            "  Stored in directory: /root/.cache/pip/wheels/55/66/40/1779aa0a8eb66e088669befe286f695cdfe420ba91ce662127\n",
            "Successfully built pytrec-eval\n",
            "Installing collected packages: pytrec-eval\n",
            "Successfully installed pytrec-eval-0.5\n",
            "Collecting wget\n",
            "  Downloading https://files.pythonhosted.org/packages/47/6a/62e288da7bcda82b935ff0c6cfe542970f04e29c756b0e147251b2fb251f/wget-3.2.zip\n",
            "Building wheels for collected packages: wget\n",
            "  Building wheel for wget (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for wget: filename=wget-3.2-cp36-none-any.whl size=9682 sha256=0646992ad2680a28456f7b77e083df77541a092580c70aa9dbd2f78a9cce6d31\n",
            "  Stored in directory: /root/.cache/pip/wheels/40/15/30/7d8f7cea2902b4db79e3fea550d7d7b85ecb27ef992b618f3f\n",
            "Successfully built wget\n",
            "Installing collected packages: wget\n",
            "Successfully installed wget-3.2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0HIF5uBV8cRy",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "41ef08b5-862d-4574-a939-aa25dc859cae"
      },
      "source": [
        "import wget\n",
        "wget.download(\"https://github.com/MIE451-1513-2019/course-datasets/raw/master/government.zip\", \"government.zip\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'government.zip'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xN-TAsbPAsRm",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "e1cf17b4-51c0-4a83-c845-340618157056"
      },
      "source": [
        "!unzip government.zip"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Archive:  government.zip\n",
            "   creating: government/\n",
            "  inflating: government/topics-with-full-descriptions.txt  \n",
            "  inflating: government/gov.topics   \n",
            "  inflating: government/gov.qrels    \n",
            "   creating: government/documents/\n",
            "   creating: government/documents/61/\n",
            "  inflating: government/documents/61/G00-61-2800209  \n",
            "  inflating: government/documents/61/G00-61-1192048  \n",
            "  inflating: government/documents/61/G00-61-1118212  \n",
            "  inflating: government/documents/61/G00-61-0749882  \n",
            "  inflating: government/documents/61/G00-61-2230501  \n",
            "  inflating: government/documents/61/G00-61-0680698  \n",
            "  inflating: government/documents/61/G00-61-0551387  \n",
            "  inflating: government/documents/61/G00-61-2575433  \n",
            "  inflating: government/documents/61/G00-61-0469713  \n",
            "  inflating: government/documents/61/G00-61-0280746  \n",
            "  inflating: government/documents/61/G00-61-2574316  \n",
            "  inflating: government/documents/61/G00-61-3933997  \n",
            "  inflating: government/documents/61/G00-61-3290635  \n",
            "  inflating: government/documents/61/G00-61-1109698  \n",
            "  inflating: government/documents/61/G00-61-2451131  \n",
            "  inflating: government/documents/61/G00-61-0655651  \n",
            "  inflating: government/documents/61/G00-61-3358134  \n",
            "  inflating: government/documents/61/G00-61-2612130  \n",
            "  inflating: government/documents/61/G00-61-0974677  \n",
            "  inflating: government/documents/61/G00-61-0614325  \n",
            "  inflating: government/documents/61/G00-61-3907226  \n",
            "  inflating: government/documents/61/G00-61-3278363  \n",
            "  inflating: government/documents/61/G00-61-3174915  \n",
            "  inflating: government/documents/61/G00-61-3298515  \n",
            "  inflating: government/documents/61/G00-61-2752439  \n",
            "  inflating: government/documents/61/G00-61-0619616  \n",
            "  inflating: government/documents/61/G00-61-4092373  \n",
            "  inflating: government/documents/61/G00-61-1131563  \n",
            "  inflating: government/documents/61/G00-61-2231909  \n",
            "  inflating: government/documents/61/G00-61-2295836  \n",
            "  inflating: government/documents/61/G00-61-0113829  \n",
            "  inflating: government/documents/61/G00-61-4150601  \n",
            "  inflating: government/documents/61/G00-61-3120987  \n",
            "  inflating: government/documents/61/G00-61-3894960  \n",
            "  inflating: government/documents/61/G00-61-0194948  \n",
            "  inflating: government/documents/61/G00-61-1484848  \n",
            "  inflating: government/documents/61/G00-61-3205908  \n",
            "  inflating: government/documents/61/G00-61-1436475  \n",
            "  inflating: government/documents/61/G00-61-2756070  \n",
            "  inflating: government/documents/61/G00-61-3516303  \n",
            "  inflating: government/documents/61/G00-61-3300557  \n",
            "   creating: government/documents/95/\n",
            "  inflating: government/documents/95/G00-95-3755341  \n",
            "  inflating: government/documents/95/G00-95-3892368  \n",
            "  inflating: government/documents/95/G00-95-3548730  \n",
            "  inflating: government/documents/95/G00-95-0741422  \n",
            "  inflating: government/documents/95/G00-95-0346292  \n",
            "  inflating: government/documents/95/G00-95-0612583  \n",
            "  inflating: government/documents/95/G00-95-3002374  \n",
            "  inflating: government/documents/95/G00-95-0454461  \n",
            "  inflating: government/documents/95/G00-95-0110661  \n",
            "  inflating: government/documents/95/G00-95-2914435  \n",
            "  inflating: government/documents/95/G00-95-2542025  \n",
            "  inflating: government/documents/95/G00-95-2028787  \n",
            "  inflating: government/documents/95/G00-95-0404721  \n",
            "  inflating: government/documents/95/G00-95-4067546  \n",
            "  inflating: government/documents/95/G00-95-1558260  \n",
            "  inflating: government/documents/95/G00-95-3043676  \n",
            "  inflating: government/documents/95/G00-95-2005790  \n",
            "  inflating: government/documents/95/G00-95-2305328  \n",
            "  inflating: government/documents/95/G00-95-0747101  \n",
            "  inflating: government/documents/95/G00-95-3337324  \n",
            "  inflating: government/documents/95/G00-95-0098370  \n",
            "  inflating: government/documents/95/G00-95-0181123  \n",
            "  inflating: government/documents/95/G00-95-3775505  \n",
            "  inflating: government/documents/95/G00-95-1976041  \n",
            "  inflating: government/documents/95/G00-95-3919947  \n",
            "  inflating: government/documents/95/G00-95-0371226  \n",
            "  inflating: government/documents/95/G00-95-0866612  \n",
            "  inflating: government/documents/95/G00-95-3378051  \n",
            "  inflating: government/documents/95/G00-95-3474611  \n",
            "  inflating: government/documents/95/G00-95-0287843  \n",
            "  inflating: government/documents/95/G00-95-0708680  \n",
            "  inflating: government/documents/95/G00-95-0963044  \n",
            "  inflating: government/documents/95/G00-95-1144392  \n",
            "  inflating: government/documents/95/G00-95-2095908  \n",
            "  inflating: government/documents/95/G00-95-1860595  \n",
            "  inflating: government/documents/95/G00-95-0702657  \n",
            "  inflating: government/documents/95/G00-95-2016892  \n",
            "  inflating: government/documents/95/G00-95-4007498  \n",
            "  inflating: government/documents/95/G00-95-1867597  \n",
            "  inflating: government/documents/95/G00-95-0245120  \n",
            "  inflating: government/documents/95/G00-95-2757938  \n",
            "  inflating: government/documents/95/G00-95-2566183  \n",
            "  inflating: government/documents/95/G00-95-3157593  \n",
            "  inflating: government/documents/95/G00-95-3870835  \n",
            "  inflating: government/documents/95/G00-95-3428981  \n",
            "   creating: government/documents/59/\n",
            "  inflating: government/documents/59/G00-59-2973625  \n",
            "  inflating: government/documents/59/G00-59-3520616  \n",
            "  inflating: government/documents/59/G00-59-0940465  \n",
            "  inflating: government/documents/59/G00-59-0879264  \n",
            "  inflating: government/documents/59/G00-59-0523165  \n",
            "  inflating: government/documents/59/G00-59-3730691  \n",
            "  inflating: government/documents/59/G00-59-0746520  \n",
            "  inflating: government/documents/59/G00-59-2546088  \n",
            "  inflating: government/documents/59/G00-59-3066205  \n",
            "  inflating: government/documents/59/G00-59-2956812  \n",
            "  inflating: government/documents/59/G00-59-3171342  \n",
            "  inflating: government/documents/59/G00-59-0515161  \n",
            "  inflating: government/documents/59/G00-59-2783898  \n",
            "  inflating: government/documents/59/G00-59-0607040  \n",
            "  inflating: government/documents/59/G00-59-3622783  \n",
            "  inflating: government/documents/59/G00-59-2483560  \n",
            "  inflating: government/documents/59/G00-59-2500550  \n",
            "  inflating: government/documents/59/G00-59-2284677  \n",
            "  inflating: government/documents/59/G00-59-1913612  \n",
            "  inflating: government/documents/59/G00-59-0641167  \n",
            "  inflating: government/documents/59/G00-59-3586444  \n",
            "  inflating: government/documents/59/G00-59-2832535  \n",
            "  inflating: government/documents/59/G00-59-2531522  \n",
            "  inflating: government/documents/59/G00-59-2306802  \n",
            "  inflating: government/documents/59/G00-59-1196753  \n",
            "  inflating: government/documents/59/G00-59-0759361  \n",
            "  inflating: government/documents/59/G00-59-3459439  \n",
            "  inflating: government/documents/59/G00-59-1181108  \n",
            "  inflating: government/documents/59/G00-59-3752265  \n",
            "  inflating: government/documents/59/G00-59-2617080  \n",
            "  inflating: government/documents/59/G00-59-2294330  \n",
            "  inflating: government/documents/59/G00-59-0937091  \n",
            "  inflating: government/documents/59/G00-59-3429520  \n",
            "  inflating: government/documents/59/G00-59-2970562  \n",
            "  inflating: government/documents/59/G00-59-1311489  \n",
            "  inflating: government/documents/59/G00-59-1606146  \n",
            "  inflating: government/documents/59/G00-59-0786269  \n",
            "  inflating: government/documents/59/G00-59-2927976  \n",
            "  inflating: government/documents/59/G00-59-1490078  \n",
            "  inflating: government/documents/59/G00-59-0624747  \n",
            "  inflating: government/documents/59/G00-59-2349259  \n",
            "  inflating: government/documents/59/G00-59-4041593  \n",
            "   creating: government/documents/92/\n",
            "  inflating: government/documents/92/G00-92-1604853  \n",
            "  inflating: government/documents/92/G00-92-3210776  \n",
            "  inflating: government/documents/92/G00-92-0578141  \n",
            "  inflating: government/documents/92/G00-92-1820424  \n",
            "  inflating: government/documents/92/G00-92-1132845  \n",
            "  inflating: government/documents/92/G00-92-2489145  \n",
            "  inflating: government/documents/92/G00-92-2053892  \n",
            "  inflating: government/documents/92/G00-92-4174427  \n",
            "  inflating: government/documents/92/G00-92-3815340  \n",
            "  inflating: government/documents/92/G00-92-1073414  \n",
            "  inflating: government/documents/92/G00-92-2878129  \n",
            "  inflating: government/documents/92/G00-92-0364153  \n",
            "  inflating: government/documents/92/G00-92-0527701  \n",
            "  inflating: government/documents/92/G00-92-0915653  \n",
            "  inflating: government/documents/92/G00-92-1620651  \n",
            "  inflating: government/documents/92/G00-92-0142623  \n",
            "  inflating: government/documents/92/G00-92-2985229  \n",
            "  inflating: government/documents/92/G00-92-3108734  \n",
            "  inflating: government/documents/92/G00-92-1469046  \n",
            "  inflating: government/documents/92/G00-92-2126100  \n",
            "  inflating: government/documents/92/G00-92-2974327  \n",
            "  inflating: government/documents/92/G00-92-2525720  \n",
            "  inflating: government/documents/92/G00-92-1739122  \n",
            "  inflating: government/documents/92/G00-92-2864043  \n",
            "  inflating: government/documents/92/G00-92-2877090  \n",
            "  inflating: government/documents/92/G00-92-0602302  \n",
            "  inflating: government/documents/92/G00-92-2113170  \n",
            "   creating: government/documents/66/\n",
            "  inflating: government/documents/66/G00-66-0841001  \n",
            "  inflating: government/documents/66/G00-66-3554105  \n",
            "  inflating: government/documents/66/G00-66-1333335  \n",
            "  inflating: government/documents/66/G00-66-3193359  \n",
            "  inflating: government/documents/66/G00-66-1627965  \n",
            "  inflating: government/documents/66/G00-66-3432910  \n",
            "  inflating: government/documents/66/G00-66-2028129  \n",
            "  inflating: government/documents/66/G00-66-1817290  \n",
            "  inflating: government/documents/66/G00-66-3105045  \n",
            "  inflating: government/documents/66/G00-66-2198995  \n",
            "  inflating: government/documents/66/G00-66-2992415  \n",
            "  inflating: government/documents/66/G00-66-0145846  \n",
            "  inflating: government/documents/66/G00-66-2867691  \n",
            "  inflating: government/documents/66/G00-66-3504465  \n",
            "  inflating: government/documents/66/G00-66-0313677  \n",
            "  inflating: government/documents/66/G00-66-2988065  \n",
            "  inflating: government/documents/66/G00-66-0135225  \n",
            "  inflating: government/documents/66/G00-66-0456093  \n",
            "  inflating: government/documents/66/G00-66-3863804  \n",
            "  inflating: government/documents/66/G00-66-2926228  \n",
            "  inflating: government/documents/66/G00-66-0999480  \n",
            "  inflating: government/documents/66/G00-66-2747468  \n",
            "  inflating: government/documents/66/G00-66-0779934  \n",
            "  inflating: government/documents/66/G00-66-3921879  \n",
            "  inflating: government/documents/66/G00-66-1584015  \n",
            "  inflating: government/documents/66/G00-66-2084493  \n",
            "  inflating: government/documents/66/G00-66-3503754  \n",
            "  inflating: government/documents/66/G00-66-4094171  \n",
            "  inflating: government/documents/66/G00-66-0781154  \n",
            "  inflating: government/documents/66/G00-66-1204421  \n",
            "  inflating: government/documents/66/G00-66-1114257  \n",
            "  inflating: government/documents/66/G00-66-2062069  \n",
            "   creating: government/documents/50/\n",
            "  inflating: government/documents/50/G00-50-2455054  \n",
            "  inflating: government/documents/50/G00-50-0205013  \n",
            "  inflating: government/documents/50/G00-50-2469238  \n",
            "  inflating: government/documents/50/G00-50-3779341  \n",
            "  inflating: government/documents/50/G00-50-3188252  \n",
            "  inflating: government/documents/50/G00-50-2576270  \n",
            "  inflating: government/documents/50/G00-50-0181286  \n",
            "  inflating: government/documents/50/G00-50-0406042  \n",
            "  inflating: government/documents/50/G00-50-2714935  \n",
            "  inflating: government/documents/50/G00-50-0448299  \n",
            "  inflating: government/documents/50/G00-50-2285222  \n",
            "  inflating: government/documents/50/G00-50-2059900  \n",
            "  inflating: government/documents/50/G00-50-1131981  \n",
            "  inflating: government/documents/50/G00-50-1032874  \n",
            "  inflating: government/documents/50/G00-50-0363109  \n",
            "  inflating: government/documents/50/G00-50-1877646  \n",
            "  inflating: government/documents/50/G00-50-3958975  \n",
            "  inflating: government/documents/50/G00-50-1978727  \n",
            "  inflating: government/documents/50/G00-50-2589385  \n",
            "  inflating: government/documents/50/G00-50-2855245  \n",
            "  inflating: government/documents/50/G00-50-3469727  \n",
            "  inflating: government/documents/50/G00-50-1452188  \n",
            "  inflating: government/documents/50/G00-50-1708114  \n",
            "  inflating: government/documents/50/G00-50-1789161  \n",
            "  inflating: government/documents/50/G00-50-3984946  \n",
            "  inflating: government/documents/50/G00-50-2017678  \n",
            "  inflating: government/documents/50/G00-50-0496728  \n",
            "  inflating: government/documents/50/G00-50-3513186  \n",
            "  inflating: government/documents/50/G00-50-2794468  \n",
            "  inflating: government/documents/50/G00-50-1467118  \n",
            "  inflating: government/documents/50/G00-50-2851146  \n",
            "  inflating: government/documents/50/G00-50-3231467  \n",
            "  inflating: government/documents/50/G00-50-1155977  \n",
            "  inflating: government/documents/50/G00-50-3342463  \n",
            "  inflating: government/documents/50/G00-50-3102480  \n",
            "  inflating: government/documents/50/G00-50-0155644  \n",
            "  inflating: government/documents/50/G00-50-3277586  \n",
            "  inflating: government/documents/50/G00-50-1809007  \n",
            "  inflating: government/documents/50/G00-50-2694870  \n",
            "  inflating: government/documents/50/G00-50-1075346  \n",
            "  inflating: government/documents/50/G00-50-0979736  \n",
            "  inflating: government/documents/50/G00-50-2186799  \n",
            "   creating: government/documents/68/\n",
            "  inflating: government/documents/68/G00-68-1600868  \n",
            "  inflating: government/documents/68/G00-68-1850883  \n",
            "  inflating: government/documents/68/G00-68-2999154  \n",
            "  inflating: government/documents/68/G00-68-0000000  \n",
            "  inflating: government/documents/68/G00-68-0895618  \n",
            "  inflating: government/documents/68/G00-68-1332243  \n",
            "  inflating: government/documents/68/G00-68-2966133  \n",
            "  inflating: government/documents/68/G00-68-0439463  \n",
            "  inflating: government/documents/68/G00-68-4089689  \n",
            "  inflating: government/documents/68/G00-68-4057729  \n",
            "  inflating: government/documents/68/G00-68-2997163  \n",
            "  inflating: government/documents/68/G00-68-1075400  \n",
            "  inflating: government/documents/68/G00-68-2675464  \n",
            "  inflating: government/documents/68/G00-68-0171274  \n",
            "  inflating: government/documents/68/G00-68-0570212  \n",
            "  inflating: government/documents/68/G00-68-1134000  \n",
            "  inflating: government/documents/68/G00-68-0320800  \n",
            "  inflating: government/documents/68/G00-68-0361254  \n",
            "  inflating: government/documents/68/G00-68-2200781  \n",
            "  inflating: government/documents/68/G00-68-1283154  \n",
            "  inflating: government/documents/68/G00-68-2213331  \n",
            "  inflating: government/documents/68/G00-68-2716539  \n",
            "  inflating: government/documents/68/G00-68-3644310  \n",
            "  inflating: government/documents/68/G00-68-2708425  \n",
            "  inflating: government/documents/68/G00-68-2813282  \n",
            "  inflating: government/documents/68/G00-68-3443432  \n",
            "  inflating: government/documents/68/G00-68-2814465  \n",
            "  inflating: government/documents/68/G00-68-0570869  \n",
            "  inflating: government/documents/68/G00-68-3661801  \n",
            "  inflating: government/documents/68/G00-68-1081000  \n",
            "  inflating: government/documents/68/G00-68-1968214  \n",
            "  inflating: government/documents/68/G00-68-2471724  \n",
            "  inflating: government/documents/68/G00-68-1782021  \n",
            "  inflating: government/documents/68/G00-68-0121247  \n",
            "  inflating: government/documents/68/G00-68-3071788  \n",
            "  inflating: government/documents/68/G00-68-1537883  \n",
            "  inflating: government/documents/68/G00-68-2512403  \n",
            "  inflating: government/documents/68/G00-68-3094820  \n",
            "   creating: government/documents/57/\n",
            "  inflating: government/documents/57/G00-57-3533857  \n",
            "  inflating: government/documents/57/G00-57-3463009  \n",
            "  inflating: government/documents/57/G00-57-1902924  \n",
            "  inflating: government/documents/57/G00-57-1288615  \n",
            "  inflating: government/documents/57/G00-57-1457708  \n",
            "  inflating: government/documents/57/G00-57-0508479  \n",
            "  inflating: government/documents/57/G00-57-0204493  \n",
            "  inflating: government/documents/57/G00-57-3572192  \n",
            "  inflating: government/documents/57/G00-57-1534179  \n",
            "  inflating: government/documents/57/G00-57-3117640  \n",
            "  inflating: government/documents/57/G00-57-2179164  \n",
            "  inflating: government/documents/57/G00-57-0197281  \n",
            "  inflating: government/documents/57/G00-57-0803089  \n",
            "  inflating: government/documents/57/G00-57-3600665  \n",
            "  inflating: government/documents/57/G00-57-3919967  \n",
            "  inflating: government/documents/57/G00-57-2250448  \n",
            "  inflating: government/documents/57/G00-57-3290405  \n",
            "  inflating: government/documents/57/G00-57-0253951  \n",
            "  inflating: government/documents/57/G00-57-1531658  \n",
            "  inflating: government/documents/57/G00-57-1321673  \n",
            "  inflating: government/documents/57/G00-57-1064839  \n",
            "  inflating: government/documents/57/G00-57-3649689  \n",
            "   creating: government/documents/03/\n",
            "  inflating: government/documents/03/G00-03-1342151  \n",
            "  inflating: government/documents/03/G00-03-2784515  \n",
            "  inflating: government/documents/03/G00-03-1066311  \n",
            "  inflating: government/documents/03/G00-03-3965443  \n",
            "  inflating: government/documents/03/G00-03-1803323  \n",
            "  inflating: government/documents/03/G00-03-2010455  \n",
            "  inflating: government/documents/03/G00-03-3573783  \n",
            "  inflating: government/documents/03/G00-03-1585062  \n",
            "  inflating: government/documents/03/G00-03-2589138  \n",
            "  inflating: government/documents/03/G00-03-0697220  \n",
            "  inflating: government/documents/03/G00-03-0131692  \n",
            "  inflating: government/documents/03/G00-03-1280908  \n",
            "  inflating: government/documents/03/G00-03-1263503  \n",
            "  inflating: government/documents/03/G00-03-1730521  \n",
            "  inflating: government/documents/03/G00-03-0000000  \n",
            "  inflating: government/documents/03/G00-03-0670479  \n",
            "  inflating: government/documents/03/G00-03-0783642  \n",
            "  inflating: government/documents/03/G00-03-3769691  \n",
            "  inflating: government/documents/03/G00-03-2504093  \n",
            "  inflating: government/documents/03/G00-03-0843413  \n",
            "  inflating: government/documents/03/G00-03-2245885  \n",
            "  inflating: government/documents/03/G00-03-1610340  \n",
            "  inflating: government/documents/03/G00-03-0467446  \n",
            "  inflating: government/documents/03/G00-03-0645441  \n",
            "  inflating: government/documents/03/G00-03-2344848  \n",
            "  inflating: government/documents/03/G00-03-2429650  \n",
            "  inflating: government/documents/03/G00-03-0931579  \n",
            "  inflating: government/documents/03/G00-03-0492327  \n",
            "  inflating: government/documents/03/G00-03-1771651  \n",
            "  inflating: government/documents/03/G00-03-2855342  \n",
            "  inflating: government/documents/03/G00-03-2613061  \n",
            "  inflating: government/documents/03/G00-03-1921890  \n",
            "  inflating: government/documents/03/G00-03-1334876  \n",
            "  inflating: government/documents/03/G00-03-2338574  \n",
            "  inflating: government/documents/03/G00-03-1927535  \n",
            "  inflating: government/documents/03/G00-03-1058112  \n",
            "  inflating: government/documents/03/G00-03-0589290  \n",
            "  inflating: government/documents/03/G00-03-1704627  \n",
            "  inflating: government/documents/03/G00-03-1895476  \n",
            "  inflating: government/documents/03/G00-03-0858819  \n",
            "  inflating: government/documents/03/G00-03-0111352  \n",
            "  inflating: government/documents/03/G00-03-1891070  \n",
            "  inflating: government/documents/03/G00-03-3391669  \n",
            "  inflating: government/documents/03/G00-03-2027773  \n",
            "  inflating: government/documents/03/G00-03-2173112  \n",
            "  inflating: government/documents/03/G00-03-0259893  \n",
            "  inflating: government/documents/03/G00-03-1789465  \n",
            "  inflating: government/documents/03/G00-03-4136743  \n",
            "  inflating: government/documents/03/G00-03-0197620  \n",
            "  inflating: government/documents/03/G00-03-2200929  \n",
            "  inflating: government/documents/03/G00-03-0766684  \n",
            "  inflating: government/documents/03/G00-03-3626113  \n",
            "  inflating: government/documents/03/G00-03-3843811  \n",
            "  inflating: government/documents/03/G00-03-0366425  \n",
            "  inflating: government/documents/03/G00-03-0073170  \n",
            "  inflating: government/documents/03/G00-03-1993772  \n",
            "  inflating: government/documents/03/G00-03-1159864  \n",
            "  inflating: government/documents/03/G00-03-1847819  \n",
            "  inflating: government/documents/03/G00-03-1485396  \n",
            "  inflating: government/documents/03/G00-03-0316753  \n",
            "  inflating: government/documents/03/G00-03-4012606  \n",
            "  inflating: government/documents/03/G00-03-2944444  \n",
            "  inflating: government/documents/03/G00-03-4076972  \n",
            "  inflating: government/documents/03/G00-03-4112125  \n",
            "  inflating: government/documents/03/G00-03-1291019  \n",
            "  inflating: government/documents/03/G00-03-2484932  \n",
            "  inflating: government/documents/03/G00-03-2224505  \n",
            "  inflating: government/documents/03/G00-03-0192125  \n",
            "  inflating: government/documents/03/G00-03-3107666  \n",
            "  inflating: government/documents/03/G00-03-0037616  \n",
            "  inflating: government/documents/03/G00-03-3078273  \n",
            "  inflating: government/documents/03/G00-03-2930314  \n",
            "  inflating: government/documents/03/G00-03-0968831  \n",
            "  inflating: government/documents/03/G00-03-1109594  \n",
            "  inflating: government/documents/03/G00-03-0991043  \n",
            "  inflating: government/documents/03/G00-03-1387912  \n",
            "  inflating: government/documents/03/G00-03-3996998  \n",
            "  inflating: government/documents/03/G00-03-2104296  \n",
            "  inflating: government/documents/03/G00-03-2424527  \n",
            "  inflating: government/documents/03/G00-03-2837152  \n",
            "  inflating: government/documents/03/G00-03-2507815  \n",
            "  inflating: government/documents/03/G00-03-2669273  \n",
            "  inflating: government/documents/03/G00-03-3056063  \n",
            "  inflating: government/documents/03/G00-03-3137314  \n",
            "  inflating: government/documents/03/G00-03-3018423  \n",
            "  inflating: government/documents/03/G00-03-2515741  \n",
            "  inflating: government/documents/03/G00-03-1496820  \n",
            "  inflating: government/documents/03/G00-03-1898526  \n",
            "  inflating: government/documents/03/G00-03-3700882  \n",
            "  inflating: government/documents/03/G00-03-3340798  \n",
            "  inflating: government/documents/03/G00-03-1048747  \n",
            "  inflating: government/documents/03/G00-03-2042174  \n",
            "  inflating: government/documents/03/G00-03-2694354  \n",
            "  inflating: government/documents/03/G00-03-3386274  \n",
            "  inflating: government/documents/03/G00-03-1410659  \n",
            "  inflating: government/documents/03/G00-03-2833729  \n",
            "  inflating: government/documents/03/G00-03-0496104  \n",
            "  inflating: government/documents/03/G00-03-1848726  \n",
            "  inflating: government/documents/03/G00-03-1606496  \n",
            "  inflating: government/documents/03/G00-03-0718385  \n",
            "  inflating: government/documents/03/G00-03-3297483  \n",
            "  inflating: government/documents/03/G00-03-0215521  \n",
            "  inflating: government/documents/03/G00-03-2718778  \n",
            "  inflating: government/documents/03/G00-03-2685070  \n",
            "  inflating: government/documents/03/G00-03-0266073  \n",
            "  inflating: government/documents/03/G00-03-4098316  \n",
            "  inflating: government/documents/03/G00-03-1664683  \n",
            "  inflating: government/documents/03/G00-03-2081400  \n",
            "  inflating: government/documents/03/G00-03-3480212  \n",
            "  inflating: government/documents/03/G00-03-0886424  \n",
            "  inflating: government/documents/03/G00-03-2187270  \n",
            "  inflating: government/documents/03/G00-03-2288219  \n",
            "  inflating: government/documents/03/G00-03-1970588  \n",
            "  inflating: government/documents/03/G00-03-0991989  \n",
            "  inflating: government/documents/03/G00-03-2869813  \n",
            "  inflating: government/documents/03/G00-03-1944370  \n",
            "  inflating: government/documents/03/G00-03-2905818  \n",
            "   creating: government/documents/04/\n",
            "  inflating: government/documents/04/G00-04-2035920  \n",
            "  inflating: government/documents/04/G00-04-1409246  \n",
            "  inflating: government/documents/04/G00-04-2134942  \n",
            "  inflating: government/documents/04/G00-04-2448974  \n",
            "  inflating: government/documents/04/G00-04-3271001  \n",
            "  inflating: government/documents/04/G00-04-2716519  \n",
            "  inflating: government/documents/04/G00-04-0928329  \n",
            "  inflating: government/documents/04/G00-04-3507451  \n",
            "  inflating: government/documents/04/G00-04-2858389  \n",
            "  inflating: government/documents/04/G00-04-1736888  \n",
            "  inflating: government/documents/04/G00-04-0014570  \n",
            "  inflating: government/documents/04/G00-04-0276405  \n",
            "  inflating: government/documents/04/G00-04-3693239  \n",
            "  inflating: government/documents/04/G00-04-1018827  \n",
            "  inflating: government/documents/04/G00-04-0296440  \n",
            "  inflating: government/documents/04/G00-04-3812745  \n",
            "  inflating: government/documents/04/G00-04-1731548  \n",
            "  inflating: government/documents/04/G00-04-1016314  \n",
            "  inflating: government/documents/04/G00-04-0971232  \n",
            "  inflating: government/documents/04/G00-04-3209697  \n",
            "  inflating: government/documents/04/G00-04-3066455  \n",
            "  inflating: government/documents/04/G00-04-1877943  \n",
            "  inflating: government/documents/04/G00-04-1309344  \n",
            "  inflating: government/documents/04/G00-04-0730361  \n",
            "  inflating: government/documents/04/G00-04-0177999  \n",
            "  inflating: government/documents/04/G00-04-0274135  \n",
            "  inflating: government/documents/04/G00-04-1452093  \n",
            "  inflating: government/documents/04/G00-04-0629714  \n",
            "  inflating: government/documents/04/G00-04-3302540  \n",
            "  inflating: government/documents/04/G00-04-3171772  \n",
            "  inflating: government/documents/04/G00-04-1324842  \n",
            "  inflating: government/documents/04/G00-04-2486353  \n",
            "  inflating: government/documents/04/G00-04-0398471  \n",
            "  inflating: government/documents/04/G00-04-2417515  \n",
            "  inflating: government/documents/04/G00-04-1138116  \n",
            "  inflating: government/documents/04/G00-04-0785331  \n",
            "  inflating: government/documents/04/G00-04-0334581  \n",
            "  inflating: government/documents/04/G00-04-3495223  \n",
            "  inflating: government/documents/04/G00-04-2203486  \n",
            "  inflating: government/documents/04/G00-04-1013476  \n",
            "  inflating: government/documents/04/G00-04-2280100  \n",
            "  inflating: government/documents/04/G00-04-1410400  \n",
            "  inflating: government/documents/04/G00-04-1444976  \n",
            "  inflating: government/documents/04/G00-04-1286103  \n",
            "  inflating: government/documents/04/G00-04-0450272  \n",
            "  inflating: government/documents/04/G00-04-3156568  \n",
            "  inflating: government/documents/04/G00-04-2903547  \n",
            "  inflating: government/documents/04/G00-04-2668409  \n",
            "  inflating: government/documents/04/G00-04-3718557  \n",
            "  inflating: government/documents/04/G00-04-4166204  \n",
            "  inflating: government/documents/04/G00-04-0005904  \n",
            "  inflating: government/documents/04/G00-04-2728438  \n",
            "  inflating: government/documents/04/G00-04-0209451  \n",
            "  inflating: government/documents/04/G00-04-1046533  \n",
            "  inflating: government/documents/04/G00-04-3778286  \n",
            "  inflating: government/documents/04/G00-04-3123084  \n",
            "  inflating: government/documents/04/G00-04-0357335  \n",
            "  inflating: government/documents/04/G00-04-1860602  \n",
            "  inflating: government/documents/04/G00-04-0827201  \n",
            "  inflating: government/documents/04/G00-04-2832744  \n",
            "  inflating: government/documents/04/G00-04-4078219  \n",
            "  inflating: government/documents/04/G00-04-3662045  \n",
            "  inflating: government/documents/04/G00-04-1864045  \n",
            "  inflating: government/documents/04/G00-04-0313635  \n",
            "  inflating: government/documents/04/G00-04-3085148  \n",
            "  inflating: government/documents/04/G00-04-0019148  \n",
            "  inflating: government/documents/04/G00-04-4008583  \n",
            "  inflating: government/documents/04/G00-04-4026648  \n",
            "  inflating: government/documents/04/G00-04-2662918  \n",
            "  inflating: government/documents/04/G00-04-3402496  \n",
            "  inflating: government/documents/04/G00-04-1881110  \n",
            "  inflating: government/documents/04/G00-04-1415677  \n",
            "  inflating: government/documents/04/G00-04-0128177  \n",
            "  inflating: government/documents/04/G00-04-0316260  \n",
            "  inflating: government/documents/04/G00-04-1562575  \n",
            "  inflating: government/documents/04/G00-04-3198667  \n",
            "  inflating: government/documents/04/G00-04-0146241  \n",
            "  inflating: government/documents/04/G00-04-1652768  \n",
            "  inflating: government/documents/04/G00-04-1277049  \n",
            "  inflating: government/documents/04/G00-04-1034628  \n",
            "  inflating: government/documents/04/G00-04-0767639  \n",
            "  inflating: government/documents/04/G00-04-3558844  \n",
            "  inflating: government/documents/04/G00-04-0503448  \n",
            "  inflating: government/documents/04/G00-04-3372687  \n",
            "  inflating: government/documents/04/G00-04-3257250  \n",
            "  inflating: government/documents/04/G00-04-3319765  \n",
            "  inflating: government/documents/04/G00-04-2753870  \n",
            "  inflating: government/documents/04/G00-04-3355472  \n",
            "  inflating: government/documents/04/G00-04-2088268  \n",
            "  inflating: government/documents/04/G00-04-0775119  \n",
            "  inflating: government/documents/04/G00-04-0292200  \n",
            "  inflating: government/documents/04/G00-04-1824402  \n",
            "  inflating: government/documents/04/G00-04-1999795  \n",
            "  inflating: government/documents/04/G00-04-3931456  \n",
            "  inflating: government/documents/04/G00-04-0412407  \n",
            "  inflating: government/documents/04/G00-04-1213346  \n",
            "  inflating: government/documents/04/G00-04-1527192  \n",
            "  inflating: government/documents/04/G00-04-2255170  \n",
            "  inflating: government/documents/04/G00-04-1296411  \n",
            "  inflating: government/documents/04/G00-04-1430928  \n",
            "  inflating: government/documents/04/G00-04-0561356  \n",
            "  inflating: government/documents/04/G00-04-0538799  \n",
            "  inflating: government/documents/04/G00-04-0039956  \n",
            "  inflating: government/documents/04/G00-04-3016417  \n",
            "  inflating: government/documents/04/G00-04-2926894  \n",
            "  inflating: government/documents/04/G00-04-2494323  \n",
            "  inflating: government/documents/04/G00-04-1891352  \n",
            "  inflating: government/documents/04/G00-04-3972022  \n",
            "  inflating: government/documents/04/G00-04-1094086  \n",
            "   creating: government/documents/32/\n",
            "  inflating: government/documents/32/G00-32-1907807  \n",
            "  inflating: government/documents/32/G00-32-2907392  \n",
            "  inflating: government/documents/32/G00-32-0348108  \n",
            "  inflating: government/documents/32/G00-32-3515764  \n",
            "  inflating: government/documents/32/G00-32-2305396  \n",
            "  inflating: government/documents/32/G00-32-3853899  \n",
            "  inflating: government/documents/32/G00-32-3090268  \n",
            "  inflating: government/documents/32/G00-32-2951843  \n",
            "  inflating: government/documents/32/G00-32-3296216  \n",
            "  inflating: government/documents/32/G00-32-1032618  \n",
            "  inflating: government/documents/32/G00-32-0551737  \n",
            "  inflating: government/documents/32/G00-32-1470031  \n",
            "  inflating: government/documents/32/G00-32-2506617  \n",
            "  inflating: government/documents/32/G00-32-0503644  \n",
            "  inflating: government/documents/32/G00-32-1745098  \n",
            "  inflating: government/documents/32/G00-32-2762525  \n",
            "  inflating: government/documents/32/G00-32-0376398  \n",
            "  inflating: government/documents/32/G00-32-0921903  \n",
            "  inflating: government/documents/32/G00-32-1570697  \n",
            "  inflating: government/documents/32/G00-32-0695646  \n",
            "  inflating: government/documents/32/G00-32-2887958  \n",
            "  inflating: government/documents/32/G00-32-1626772  \n",
            "  inflating: government/documents/32/G00-32-0245184  \n",
            "  inflating: government/documents/32/G00-32-2166728  \n",
            "   creating: government/documents/35/\n",
            "  inflating: government/documents/35/G00-35-0873391  \n",
            "  inflating: government/documents/35/G00-35-3882703  \n",
            "  inflating: government/documents/35/G00-35-0886432  \n",
            "  inflating: government/documents/35/G00-35-0169522  \n",
            "  inflating: government/documents/35/G00-35-1974416  \n",
            "  inflating: government/documents/35/G00-35-1545751  \n",
            "  inflating: government/documents/35/G00-35-3085404  \n",
            "  inflating: government/documents/35/G00-35-0053029  \n",
            "  inflating: government/documents/35/G00-35-1946377  \n",
            "  inflating: government/documents/35/G00-35-4101243  \n",
            "  inflating: government/documents/35/G00-35-2639523  \n",
            "  inflating: government/documents/35/G00-35-3820003  \n",
            "  inflating: government/documents/35/G00-35-3432192  \n",
            "  inflating: government/documents/35/G00-35-0946984  \n",
            "  inflating: government/documents/35/G00-35-0968818  \n",
            "  inflating: government/documents/35/G00-35-4018350  \n",
            "  inflating: government/documents/35/G00-35-1714937  \n",
            "  inflating: government/documents/35/G00-35-1576806  \n",
            "  inflating: government/documents/35/G00-35-3362733  \n",
            "  inflating: government/documents/35/G00-35-4003014  \n",
            "  inflating: government/documents/35/G00-35-3292355  \n",
            "  inflating: government/documents/35/G00-35-2527252  \n",
            "  inflating: government/documents/35/G00-35-2972457  \n",
            "  inflating: government/documents/35/G00-35-2715236  \n",
            "  inflating: government/documents/35/G00-35-0922387  \n",
            "  inflating: government/documents/35/G00-35-1566196  \n",
            "  inflating: government/documents/35/G00-35-3406418  \n",
            "  inflating: government/documents/35/G00-35-2799823  \n",
            "  inflating: government/documents/35/G00-35-1295262  \n",
            "  inflating: government/documents/35/G00-35-2190090  \n",
            "  inflating: government/documents/35/G00-35-3237468  \n",
            "   creating: government/documents/69/\n",
            "  inflating: government/documents/69/G00-69-1910137  \n",
            "  inflating: government/documents/69/G00-69-2249049  \n",
            "  inflating: government/documents/69/G00-69-2570177  \n",
            "  inflating: government/documents/69/G00-69-0550726  \n",
            "  inflating: government/documents/69/G00-69-1797385  \n",
            "  inflating: government/documents/69/G00-69-2578528  \n",
            "  inflating: government/documents/69/G00-69-3581376  \n",
            "  inflating: government/documents/69/G00-69-1939272  \n",
            "  inflating: government/documents/69/G00-69-0779075  \n",
            "  inflating: government/documents/69/G00-69-1691089  \n",
            "  inflating: government/documents/69/G00-69-3929158  \n",
            "  inflating: government/documents/69/G00-69-2828869  \n",
            "  inflating: government/documents/69/G00-69-3235793  \n",
            "  inflating: government/documents/69/G00-69-0897350  \n",
            "  inflating: government/documents/69/G00-69-1630483  \n",
            "  inflating: government/documents/69/G00-69-0971359  \n",
            "  inflating: government/documents/69/G00-69-1092129  \n",
            "  inflating: government/documents/69/G00-69-2544761  \n",
            "  inflating: government/documents/69/G00-69-0204239  \n",
            "  inflating: government/documents/69/G00-69-0992063  \n",
            "  inflating: government/documents/69/G00-69-4177280  \n",
            "  inflating: government/documents/69/G00-69-3727013  \n",
            "  inflating: government/documents/69/G00-69-2698310  \n",
            "  inflating: government/documents/69/G00-69-0005329  \n",
            "  inflating: government/documents/69/G00-69-1435616  \n",
            "  inflating: government/documents/69/G00-69-1265256  \n",
            "  inflating: government/documents/69/G00-69-0982341  \n",
            "  inflating: government/documents/69/G00-69-0750141  \n",
            "  inflating: government/documents/69/G00-69-0601069  \n",
            "  inflating: government/documents/69/G00-69-1645072  \n",
            "  inflating: government/documents/69/G00-69-0402252  \n",
            "  inflating: government/documents/69/G00-69-0920828  \n",
            "  inflating: government/documents/69/G00-69-1932028  \n",
            "  inflating: government/documents/69/G00-69-2353421  \n",
            "  inflating: government/documents/69/G00-69-3541913  \n",
            "  inflating: government/documents/69/G00-69-0092136  \n",
            "   creating: government/documents/56/\n",
            "  inflating: government/documents/56/G00-56-3267412  \n",
            "  inflating: government/documents/56/G00-56-1880693  \n",
            "  inflating: government/documents/56/G00-56-1706812  \n",
            "  inflating: government/documents/56/G00-56-2928520  \n",
            "  inflating: government/documents/56/G00-56-2550885  \n",
            "  inflating: government/documents/56/G00-56-3884873  \n",
            "  inflating: government/documents/56/G00-56-0323218  \n",
            "  inflating: government/documents/56/G00-56-0327569  \n",
            "  inflating: government/documents/56/G00-56-1236267  \n",
            "  inflating: government/documents/56/G00-56-4034448  \n",
            "  inflating: government/documents/56/G00-56-0479901  \n",
            "  inflating: government/documents/56/G00-56-1300001  \n",
            "  inflating: government/documents/56/G00-56-4151981  \n",
            "  inflating: government/documents/56/G00-56-2329426  \n",
            "  inflating: government/documents/56/G00-56-1963034  \n",
            "  inflating: government/documents/56/G00-56-4183095  \n",
            "  inflating: government/documents/56/G00-56-2140972  \n",
            "  inflating: government/documents/56/G00-56-0639217  \n",
            "  inflating: government/documents/56/G00-56-1774854  \n",
            "  inflating: government/documents/56/G00-56-3262021  \n",
            "  inflating: government/documents/56/G00-56-0597632  \n",
            "  inflating: government/documents/56/G00-56-3249614  \n",
            "  inflating: government/documents/56/G00-56-3011495  \n",
            "  inflating: government/documents/56/G00-56-2914829  \n",
            "  inflating: government/documents/56/G00-56-2425288  \n",
            "  inflating: government/documents/56/G00-56-0844216  \n",
            "  inflating: government/documents/56/G00-56-0703159  \n",
            "   creating: government/documents/51/\n",
            "  inflating: government/documents/51/G00-51-1102099  \n",
            "  inflating: government/documents/51/G00-51-1483504  \n",
            "  inflating: government/documents/51/G00-51-1191692  \n",
            "  inflating: government/documents/51/G00-51-0089678  \n",
            "  inflating: government/documents/51/G00-51-0815543  \n",
            "  inflating: government/documents/51/G00-51-2923921  \n",
            "  inflating: government/documents/51/G00-51-2949538  \n",
            "  inflating: government/documents/51/G00-51-3264753  \n",
            "  inflating: government/documents/51/G00-51-3768244  \n",
            "  inflating: government/documents/51/G00-51-3348554  \n",
            "  inflating: government/documents/51/G00-51-0400594  \n",
            "  inflating: government/documents/51/G00-51-3613612  \n",
            "  inflating: government/documents/51/G00-51-3553948  \n",
            "  inflating: government/documents/51/G00-51-1785570  \n",
            "  inflating: government/documents/51/G00-51-0498788  \n",
            "  inflating: government/documents/51/G00-51-1924264  \n",
            "  inflating: government/documents/51/G00-51-1758488  \n",
            "  inflating: government/documents/51/G00-51-0687659  \n",
            "  inflating: government/documents/51/G00-51-0560959  \n",
            "  inflating: government/documents/51/G00-51-2032061  \n",
            "  inflating: government/documents/51/G00-51-1817629  \n",
            "   creating: government/documents/58/\n",
            "  inflating: government/documents/58/G00-58-3729581  \n",
            "  inflating: government/documents/58/G00-58-0743096  \n",
            "  inflating: government/documents/58/G00-58-1844953  \n",
            "  inflating: government/documents/58/G00-58-2162083  \n",
            "  inflating: government/documents/58/G00-58-0554573  \n",
            "  inflating: government/documents/58/G00-58-0058694  \n",
            "  inflating: government/documents/58/G00-58-3268744  \n",
            "  inflating: government/documents/58/G00-58-3804365  \n",
            "  inflating: government/documents/58/G00-58-0251906  \n",
            "  inflating: government/documents/58/G00-58-0468921  \n",
            "  inflating: government/documents/58/G00-58-1033990  \n",
            "  inflating: government/documents/58/G00-58-0264438  \n",
            "  inflating: government/documents/58/G00-58-2205749  \n",
            "  inflating: government/documents/58/G00-58-1407270  \n",
            "  inflating: government/documents/58/G00-58-3498167  \n",
            "  inflating: government/documents/58/G00-58-4057573  \n",
            "  inflating: government/documents/58/G00-58-0570362  \n",
            "  inflating: government/documents/58/G00-58-1938337  \n",
            "  inflating: government/documents/58/G00-58-1212629  \n",
            "  inflating: government/documents/58/G00-58-2989879  \n",
            "  inflating: government/documents/58/G00-58-2446566  \n",
            "  inflating: government/documents/58/G00-58-1795138  \n",
            "  inflating: government/documents/58/G00-58-1737760  \n",
            "  inflating: government/documents/58/G00-58-3634250  \n",
            "  inflating: government/documents/58/G00-58-2464974  \n",
            "  inflating: government/documents/58/G00-58-0287418  \n",
            "  inflating: government/documents/58/G00-58-3090637  \n",
            "  inflating: government/documents/58/G00-58-3937064  \n",
            "  inflating: government/documents/58/G00-58-1856156  \n",
            "  inflating: government/documents/58/G00-58-1939983  \n",
            "  inflating: government/documents/58/G00-58-3411830  \n",
            "  inflating: government/documents/58/G00-58-2466520  \n",
            "  inflating: government/documents/58/G00-58-3488759  \n",
            "  inflating: government/documents/58/G00-58-0435410  \n",
            "   creating: government/documents/67/\n",
            "  inflating: government/documents/67/G00-67-3682125  \n",
            "  inflating: government/documents/67/G00-67-2771108  \n",
            "  inflating: government/documents/67/G00-67-2003399  \n",
            "  inflating: government/documents/67/G00-67-0920658  \n",
            "  inflating: government/documents/67/G00-67-1626082  \n",
            "  inflating: government/documents/67/G00-67-2481553  \n",
            "  inflating: government/documents/67/G00-67-1986470  \n",
            "  inflating: government/documents/67/G00-67-2253092  \n",
            "  inflating: government/documents/67/G00-67-1653323  \n",
            "  inflating: government/documents/67/G00-67-0351803  \n",
            "  inflating: government/documents/67/G00-67-0637954  \n",
            "  inflating: government/documents/67/G00-67-0051429  \n",
            "  inflating: government/documents/67/G00-67-4134018  \n",
            "  inflating: government/documents/67/G00-67-0077172  \n",
            "  inflating: government/documents/67/G00-67-0152545  \n",
            "  inflating: government/documents/67/G00-67-0795339  \n",
            "  inflating: government/documents/67/G00-67-3819584  \n",
            "  inflating: government/documents/67/G00-67-3875805  \n",
            "  inflating: government/documents/67/G00-67-1920188  \n",
            "  inflating: government/documents/67/G00-67-1477759  \n",
            "  inflating: government/documents/67/G00-67-2808769  \n",
            "  inflating: government/documents/67/G00-67-2278397  \n",
            "  inflating: government/documents/67/G00-67-1176122  \n",
            "  inflating: government/documents/67/G00-67-3888789  \n",
            "  inflating: government/documents/67/G00-67-0564291  \n",
            "  inflating: government/documents/67/G00-67-0038636  \n",
            "  inflating: government/documents/67/G00-67-3223440  \n",
            "  inflating: government/documents/67/G00-67-1727933  \n",
            "  inflating: government/documents/67/G00-67-3813180  \n",
            "  inflating: government/documents/67/G00-67-2999597  \n",
            "  inflating: government/documents/67/G00-67-3852352  \n",
            "  inflating: government/documents/67/G00-67-3538835  \n",
            "  inflating: government/documents/67/G00-67-3598246  \n",
            "  inflating: government/documents/67/G00-67-3576955  \n",
            "  inflating: government/documents/67/G00-67-1876619  \n",
            "  inflating: government/documents/67/G00-67-2780214  \n",
            "   creating: government/documents/93/\n",
            "  inflating: government/documents/93/G00-93-1016588  \n",
            "  inflating: government/documents/93/G00-93-1024042  \n",
            "  inflating: government/documents/93/G00-93-2004577  \n",
            "  inflating: government/documents/93/G00-93-0389371  \n",
            "  inflating: government/documents/93/G00-93-1698165  \n",
            "  inflating: government/documents/93/G00-93-3182623  \n",
            "  inflating: government/documents/93/G00-93-0843597  \n",
            "  inflating: government/documents/93/G00-93-3967570  \n",
            "  inflating: government/documents/93/G00-93-1237623  \n",
            "  inflating: government/documents/93/G00-93-1352793  \n",
            "  inflating: government/documents/93/G00-93-0901470  \n",
            "  inflating: government/documents/93/G00-93-3473671  \n",
            "  inflating: government/documents/93/G00-93-0870338  \n",
            "  inflating: government/documents/93/G00-93-2563459  \n",
            "  inflating: government/documents/93/G00-93-2470909  \n",
            "  inflating: government/documents/93/G00-93-4160214  \n",
            "  inflating: government/documents/93/G00-93-1371217  \n",
            "  inflating: government/documents/93/G00-93-2344882  \n",
            "  inflating: government/documents/93/G00-93-1265148  \n",
            "  inflating: government/documents/93/G00-93-3702508  \n",
            "  inflating: government/documents/93/G00-93-2875149  \n",
            "  inflating: government/documents/93/G00-93-1159393  \n",
            "  inflating: government/documents/93/G00-93-1635202  \n",
            "  inflating: government/documents/93/G00-93-0470547  \n",
            "  inflating: government/documents/93/G00-93-1783036  \n",
            "  inflating: government/documents/93/G00-93-0207532  \n",
            "  inflating: government/documents/93/G00-93-3425780  \n",
            "  inflating: government/documents/93/G00-93-3791059  \n",
            "  inflating: government/documents/93/G00-93-1203370  \n",
            "  inflating: government/documents/93/G00-93-4012578  \n",
            "   creating: government/documents/94/\n",
            "  inflating: government/documents/94/G00-94-0087632  \n",
            "  inflating: government/documents/94/G00-94-0583920  \n",
            "  inflating: government/documents/94/G00-94-3782465  \n",
            "  inflating: government/documents/94/G00-94-2502192  \n",
            "  inflating: government/documents/94/G00-94-0708359  \n",
            "  inflating: government/documents/94/G00-94-2662608  \n",
            "  inflating: government/documents/94/G00-94-1007927  \n",
            "  inflating: government/documents/94/G00-94-2249557  \n",
            "  inflating: government/documents/94/G00-94-2611218  \n",
            "  inflating: government/documents/94/G00-94-1934206  \n",
            "  inflating: government/documents/94/G00-94-1187833  \n",
            "  inflating: government/documents/94/G00-94-1117794  \n",
            "  inflating: government/documents/94/G00-94-3501831  \n",
            "  inflating: government/documents/94/G00-94-1368593  \n",
            "  inflating: government/documents/94/G00-94-2971833  \n",
            "  inflating: government/documents/94/G00-94-1761460  \n",
            "  inflating: government/documents/94/G00-94-4041492  \n",
            "  inflating: government/documents/94/G00-94-0216582  \n",
            "  inflating: government/documents/94/G00-94-4070183  \n",
            "  inflating: government/documents/94/G00-94-3097702  \n",
            "  inflating: government/documents/94/G00-94-1328230  \n",
            "  inflating: government/documents/94/G00-94-3306767  \n",
            "  inflating: government/documents/94/G00-94-1189296  \n",
            "  inflating: government/documents/94/G00-94-4046104  \n",
            "  inflating: government/documents/94/G00-94-1978508  \n",
            "  inflating: government/documents/94/G00-94-3433584  \n",
            "  inflating: government/documents/94/G00-94-3227831  \n",
            "  inflating: government/documents/94/G00-94-3725883  \n",
            "  inflating: government/documents/94/G00-94-0326199  \n",
            "  inflating: government/documents/94/G00-94-1190048  \n",
            "  inflating: government/documents/94/G00-94-2620896  \n",
            "  inflating: government/documents/94/G00-94-3105853  \n",
            "  inflating: government/documents/94/G00-94-2625788  \n",
            "  inflating: government/documents/94/G00-94-1294899  \n",
            "   creating: government/documents/60/\n",
            "  inflating: government/documents/60/G00-60-0922955  \n",
            "  inflating: government/documents/60/G00-60-1185974  \n",
            "  inflating: government/documents/60/G00-60-0338104  \n",
            "  inflating: government/documents/60/G00-60-2564326  \n",
            "  inflating: government/documents/60/G00-60-2428516  \n",
            "  inflating: government/documents/60/G00-60-1981260  \n",
            "  inflating: government/documents/60/G00-60-1440029  \n",
            "  inflating: government/documents/60/G00-60-0543216  \n",
            "  inflating: government/documents/60/G00-60-0879221  \n",
            "  inflating: government/documents/60/G00-60-0321329  \n",
            "  inflating: government/documents/60/G00-60-0305942  \n",
            "  inflating: government/documents/60/G00-60-3617373  \n",
            "  inflating: government/documents/60/G00-60-1532208  \n",
            "  inflating: government/documents/60/G00-60-2433761  \n",
            "  inflating: government/documents/60/G00-60-1570476  \n",
            "  inflating: government/documents/60/G00-60-3914816  \n",
            "  inflating: government/documents/60/G00-60-0631416  \n",
            "  inflating: government/documents/60/G00-60-3249607  \n",
            "  inflating: government/documents/60/G00-60-0986120  \n",
            "  inflating: government/documents/60/G00-60-0593267  \n",
            "  inflating: government/documents/60/G00-60-0199705  \n",
            "  inflating: government/documents/60/G00-60-1494612  \n",
            "  inflating: government/documents/60/G00-60-0739588  \n",
            "  inflating: government/documents/60/G00-60-2112580  \n",
            "  inflating: government/documents/60/G00-60-4138309  \n",
            "  inflating: government/documents/60/G00-60-1236252  \n",
            "  inflating: government/documents/60/G00-60-4042439  \n",
            "  inflating: government/documents/60/G00-60-1487771  \n",
            "   creating: government/documents/34/\n",
            "  inflating: government/documents/34/G00-34-0729018  \n",
            "  inflating: government/documents/34/G00-34-0655826  \n",
            "  inflating: government/documents/34/G00-34-1018795  \n",
            "  inflating: government/documents/34/G00-34-0444524  \n",
            "  inflating: government/documents/34/G00-34-3591274  \n",
            "  inflating: government/documents/34/G00-34-3698894  \n",
            "  inflating: government/documents/34/G00-34-1044519  \n",
            "  inflating: government/documents/34/G00-34-3494159  \n",
            "  inflating: government/documents/34/G00-34-0061753  \n",
            "  inflating: government/documents/34/G00-34-0984356  \n",
            "  inflating: government/documents/34/G00-34-0427966  \n",
            "  inflating: government/documents/34/G00-34-2691400  \n",
            "  inflating: government/documents/34/G00-34-3584290  \n",
            "  inflating: government/documents/34/G00-34-4113161  \n",
            "  inflating: government/documents/34/G00-34-1517989  \n",
            "  inflating: government/documents/34/G00-34-1094860  \n",
            "  inflating: government/documents/34/G00-34-4042132  \n",
            "  inflating: government/documents/34/G00-34-2999931  \n",
            "  inflating: government/documents/34/G00-34-0760154  \n",
            "  inflating: government/documents/34/G00-34-2408824  \n",
            "  inflating: government/documents/34/G00-34-2902209  \n",
            "  inflating: government/documents/34/G00-34-1275218  \n",
            "  inflating: government/documents/34/G00-34-2179336  \n",
            "  inflating: government/documents/34/G00-34-1467420  \n",
            "  inflating: government/documents/34/G00-34-1882006  \n",
            "  inflating: government/documents/34/G00-34-2047694  \n",
            "  inflating: government/documents/34/G00-34-0234491  \n",
            "  inflating: government/documents/34/G00-34-0751149  \n",
            "  inflating: government/documents/34/G00-34-3331337  \n",
            "  inflating: government/documents/34/G00-34-3408606  \n",
            "  inflating: government/documents/34/G00-34-0424105  \n",
            "   creating: government/documents/33/\n",
            "  inflating: government/documents/33/G00-33-1434621  \n",
            "  inflating: government/documents/33/G00-33-0806248  \n",
            "  inflating: government/documents/33/G00-33-4173024  \n",
            "  inflating: government/documents/33/G00-33-2347653  \n",
            "  inflating: government/documents/33/G00-33-2762518  \n",
            "  inflating: government/documents/33/G00-33-4156365  \n",
            "  inflating: government/documents/33/G00-33-2002560  \n",
            "  inflating: government/documents/33/G00-33-1425672  \n",
            "  inflating: government/documents/33/G00-33-1046511  \n",
            "  inflating: government/documents/33/G00-33-0293490  \n",
            "  inflating: government/documents/33/G00-33-2857182  \n",
            "  inflating: government/documents/33/G00-33-2815908  \n",
            "  inflating: government/documents/33/G00-33-3833530  \n",
            "  inflating: government/documents/33/G00-33-1729611  \n",
            "  inflating: government/documents/33/G00-33-3539855  \n",
            "  inflating: government/documents/33/G00-33-3621878  \n",
            "  inflating: government/documents/33/G00-33-1173483  \n",
            "  inflating: government/documents/33/G00-33-0504218  \n",
            "  inflating: government/documents/33/G00-33-1897527  \n",
            "  inflating: government/documents/33/G00-33-2821056  \n",
            "  inflating: government/documents/33/G00-33-0053624  \n",
            "  inflating: government/documents/33/G00-33-1942384  \n",
            "  inflating: government/documents/33/G00-33-2494908  \n",
            "  inflating: government/documents/33/G00-33-0737891  \n",
            "  inflating: government/documents/33/G00-33-0503127  \n",
            "  inflating: government/documents/33/G00-33-0486825  \n",
            "  inflating: government/documents/33/G00-33-3016812  \n",
            "  inflating: government/documents/33/G00-33-0670497  \n",
            "  inflating: government/documents/33/G00-33-2461443  \n",
            "  inflating: government/documents/33/G00-33-2034731  \n",
            "  inflating: government/documents/33/G00-33-2327900  \n",
            "  inflating: government/documents/33/G00-33-2469103  \n",
            "  inflating: government/documents/33/G00-33-2074619  \n",
            "   creating: government/documents/05/\n",
            "  inflating: government/documents/05/G00-05-0719078  \n",
            "  inflating: government/documents/05/G00-05-1727482  \n",
            "  inflating: government/documents/05/G00-05-2303988  \n",
            "  inflating: government/documents/05/G00-05-2928830  \n",
            "  inflating: government/documents/05/G00-05-2114059  \n",
            "  inflating: government/documents/05/G00-05-4010060  \n",
            "  inflating: government/documents/05/G00-05-1064100  \n",
            "  inflating: government/documents/05/G00-05-3814025  \n",
            "  inflating: government/documents/05/G00-05-2357004  \n",
            "  inflating: government/documents/05/G00-05-1786459  \n",
            "  inflating: government/documents/05/G00-05-3224912  \n",
            "  inflating: government/documents/05/G00-05-3680680  \n",
            "  inflating: government/documents/05/G00-05-2762833  \n",
            "  inflating: government/documents/05/G00-05-3604034  \n",
            "  inflating: government/documents/05/G00-05-0035717  \n",
            "  inflating: government/documents/05/G00-05-0768168  \n",
            "  inflating: government/documents/05/G00-05-3848213  \n",
            "  inflating: government/documents/05/G00-05-3170756  \n",
            "  inflating: government/documents/05/G00-05-3637817  \n",
            "  inflating: government/documents/05/G00-05-1434764  \n",
            "  inflating: government/documents/05/G00-05-1550998  \n",
            "  inflating: government/documents/05/G00-05-3357276  \n",
            "  inflating: government/documents/05/G00-05-2805580  \n",
            "  inflating: government/documents/05/G00-05-0230946  \n",
            "  inflating: government/documents/05/G00-05-1408124  \n",
            "  inflating: government/documents/05/G00-05-2480927  \n",
            "  inflating: government/documents/05/G00-05-1023764  \n",
            "  inflating: government/documents/05/G00-05-0771435  \n",
            "  inflating: government/documents/05/G00-05-1904668  \n",
            "  inflating: government/documents/05/G00-05-3846958  \n",
            "  inflating: government/documents/05/G00-05-0081436  \n",
            "  inflating: government/documents/05/G00-05-2787549  \n",
            "  inflating: government/documents/05/G00-05-3024540  \n",
            "  inflating: government/documents/05/G00-05-3121512  \n",
            "  inflating: government/documents/05/G00-05-1004743  \n",
            "  inflating: government/documents/05/G00-05-2393778  \n",
            "  inflating: government/documents/05/G00-05-3579944  \n",
            "  inflating: government/documents/05/G00-05-3086404  \n",
            "  inflating: government/documents/05/G00-05-3051665  \n",
            "  inflating: government/documents/05/G00-05-1639628  \n",
            "  inflating: government/documents/05/G00-05-3783969  \n",
            "  inflating: government/documents/05/G00-05-2071164  \n",
            "  inflating: government/documents/05/G00-05-0212361  \n",
            "  inflating: government/documents/05/G00-05-2323106  \n",
            "  inflating: government/documents/05/G00-05-3539237  \n",
            "  inflating: government/documents/05/G00-05-0911594  \n",
            "  inflating: government/documents/05/G00-05-1655757  \n",
            "  inflating: government/documents/05/G00-05-1131809  \n",
            "  inflating: government/documents/05/G00-05-2034933  \n",
            "  inflating: government/documents/05/G00-05-2079308  \n",
            "  inflating: government/documents/05/G00-05-3353179  \n",
            "  inflating: government/documents/05/G00-05-2231767  \n",
            "  inflating: government/documents/05/G00-05-1030627  \n",
            "  inflating: government/documents/05/G00-05-0272455  \n",
            "  inflating: government/documents/05/G00-05-1146514  \n",
            "  inflating: government/documents/05/G00-05-1517784  \n",
            "  inflating: government/documents/05/G00-05-0067428  \n",
            "  inflating: government/documents/05/G00-05-2082516  \n",
            "  inflating: government/documents/05/G00-05-3390354  \n",
            "  inflating: government/documents/05/G00-05-3832940  \n",
            "  inflating: government/documents/05/G00-05-0137212  \n",
            "  inflating: government/documents/05/G00-05-3854564  \n",
            "  inflating: government/documents/05/G00-05-3345831  \n",
            "  inflating: government/documents/05/G00-05-0750309  \n",
            "  inflating: government/documents/05/G00-05-1324928  \n",
            "  inflating: government/documents/05/G00-05-3569844  \n",
            "  inflating: government/documents/05/G00-05-2552502  \n",
            "  inflating: government/documents/05/G00-05-3703958  \n",
            "  inflating: government/documents/05/G00-05-1961641  \n",
            "  inflating: government/documents/05/G00-05-1894408  \n",
            "  inflating: government/documents/05/G00-05-4035656  \n",
            "  inflating: government/documents/05/G00-05-2203144  \n",
            "  inflating: government/documents/05/G00-05-3251004  \n",
            "  inflating: government/documents/05/G00-05-3932135  \n",
            "  inflating: government/documents/05/G00-05-3764273  \n",
            "  inflating: government/documents/05/G00-05-0758583  \n",
            "  inflating: government/documents/05/G00-05-1470237  \n",
            "  inflating: government/documents/05/G00-05-1392191  \n",
            "  inflating: government/documents/05/G00-05-1619768  \n",
            "  inflating: government/documents/05/G00-05-2863370  \n",
            "  inflating: government/documents/05/G00-05-3526931  \n",
            "  inflating: government/documents/05/G00-05-0927949  \n",
            "  inflating: government/documents/05/G00-05-2948873  \n",
            "  inflating: government/documents/05/G00-05-0448193  \n",
            "  inflating: government/documents/05/G00-05-0013004  \n",
            "  inflating: government/documents/05/G00-05-2782257  \n",
            "  inflating: government/documents/05/G00-05-0764940  \n",
            "  inflating: government/documents/05/G00-05-2988698  \n",
            "  inflating: government/documents/05/G00-05-0110566  \n",
            "  inflating: government/documents/05/G00-05-1620629  \n",
            "  inflating: government/documents/05/G00-05-0623935  \n",
            "  inflating: government/documents/05/G00-05-0743969  \n",
            "  inflating: government/documents/05/G00-05-0709830  \n",
            "  inflating: government/documents/05/G00-05-3437619  \n",
            "  inflating: government/documents/05/G00-05-1478323  \n",
            "  inflating: government/documents/05/G00-05-4049017  \n",
            "  inflating: government/documents/05/G00-05-1642328  \n",
            "  inflating: government/documents/05/G00-05-3463998  \n",
            "  inflating: government/documents/05/G00-05-1315394  \n",
            "  inflating: government/documents/05/G00-05-1267454  \n",
            "  inflating: government/documents/05/G00-05-2191460  \n",
            "  inflating: government/documents/05/G00-05-0619345  \n",
            "  inflating: government/documents/05/G00-05-3046426  \n",
            "  inflating: government/documents/05/G00-05-0390218  \n",
            "  inflating: government/documents/05/G00-05-3391805  \n",
            "  inflating: government/documents/05/G00-05-1218739  \n",
            "  inflating: government/documents/05/G00-05-2966257  \n",
            "  inflating: government/documents/05/G00-05-2684553  \n",
            "  inflating: government/documents/05/G00-05-3621484  \n",
            "  inflating: government/documents/05/G00-05-3904373  \n",
            "   creating: government/documents/02/\n",
            "  inflating: government/documents/02/G00-02-1934959  \n",
            "  inflating: government/documents/02/G00-02-3032184  \n",
            "  inflating: government/documents/02/G00-02-1551768  \n",
            "  inflating: government/documents/02/G00-02-1411565  \n",
            "  inflating: government/documents/02/G00-02-3830677  \n",
            "  inflating: government/documents/02/G00-02-0901987  \n",
            "  inflating: government/documents/02/G00-02-2153966  \n",
            "  inflating: government/documents/02/G00-02-4047734  \n",
            "  inflating: government/documents/02/G00-02-0447211  \n",
            "  inflating: government/documents/02/G00-02-0866559  \n",
            "  inflating: government/documents/02/G00-02-1147404  \n",
            "  inflating: government/documents/02/G00-02-3232307  \n",
            "  inflating: government/documents/02/G00-02-1467454  \n",
            "  inflating: government/documents/02/G00-02-3479535  \n",
            "  inflating: government/documents/02/G00-02-2405910  \n",
            "  inflating: government/documents/02/G00-02-0761842  \n",
            "  inflating: government/documents/02/G00-02-2681918  \n",
            "  inflating: government/documents/02/G00-02-3228661  \n",
            "  inflating: government/documents/02/G00-02-1833746  \n",
            "  inflating: government/documents/02/G00-02-2373378  \n",
            "  inflating: government/documents/02/G00-02-4147101  \n",
            "  inflating: government/documents/02/G00-02-3338510  \n",
            "  inflating: government/documents/02/G00-02-1392373  \n",
            "  inflating: government/documents/02/G00-02-3416895  \n",
            "  inflating: government/documents/02/G00-02-3199651  \n",
            "  inflating: government/documents/02/G00-02-0777152  \n",
            "  inflating: government/documents/02/G00-02-1652207  \n",
            "  inflating: government/documents/02/G00-02-3718994  \n",
            "  inflating: government/documents/02/G00-02-0533322  \n",
            "  inflating: government/documents/02/G00-02-0524966  \n",
            "  inflating: government/documents/02/G00-02-2211051  \n",
            "  inflating: government/documents/02/G00-02-3361139  \n",
            "  inflating: government/documents/02/G00-02-2542528  \n",
            "  inflating: government/documents/02/G00-02-1762113  \n",
            "  inflating: government/documents/02/G00-02-1486711  \n",
            "  inflating: government/documents/02/G00-02-1759190  \n",
            "  inflating: government/documents/02/G00-02-1479121  \n",
            "  inflating: government/documents/02/G00-02-1720397  \n",
            "  inflating: government/documents/02/G00-02-0114963  \n",
            "  inflating: government/documents/02/G00-02-0267227  \n",
            "  inflating: government/documents/02/G00-02-1463953  \n",
            "  inflating: government/documents/02/G00-02-1569978  \n",
            "  inflating: government/documents/02/G00-02-0003148  \n",
            "  inflating: government/documents/02/G00-02-2601378  \n",
            "  inflating: government/documents/02/G00-02-3720828  \n",
            "  inflating: government/documents/02/G00-02-4140077  \n",
            "  inflating: government/documents/02/G00-02-2039026  \n",
            "  inflating: government/documents/02/G00-02-4193228  \n",
            "  inflating: government/documents/02/G00-02-1212269  \n",
            "  inflating: government/documents/02/G00-02-1862854  \n",
            "  inflating: government/documents/02/G00-02-3376624  \n",
            "  inflating: government/documents/02/G00-02-0646693  \n",
            "  inflating: government/documents/02/G00-02-0256830  \n",
            "  inflating: government/documents/02/G00-02-0146077  \n",
            "  inflating: government/documents/02/G00-02-2116569  \n",
            "  inflating: government/documents/02/G00-02-0541868  \n",
            "  inflating: government/documents/02/G00-02-2095762  \n",
            "  inflating: government/documents/02/G00-02-0555602  \n",
            "  inflating: government/documents/02/G00-02-1710004  \n",
            "  inflating: government/documents/02/G00-02-1372443  \n",
            "  inflating: government/documents/02/G00-02-1818683  \n",
            "  inflating: government/documents/02/G00-02-2594969  \n",
            "  inflating: government/documents/02/G00-02-2454692  \n",
            "  inflating: government/documents/02/G00-02-3293233  \n",
            "  inflating: government/documents/02/G00-02-0916641  \n",
            "  inflating: government/documents/02/G00-02-1103235  \n",
            "  inflating: government/documents/02/G00-02-2469809  \n",
            "  inflating: government/documents/02/G00-02-0599362  \n",
            "  inflating: government/documents/02/G00-02-2120122  \n",
            "  inflating: government/documents/02/G00-02-2129251  \n",
            "  inflating: government/documents/02/G00-02-0069667  \n",
            "  inflating: government/documents/02/G00-02-0735704  \n",
            "  inflating: government/documents/02/G00-02-0135035  \n",
            "  inflating: government/documents/02/G00-02-2407017  \n",
            "  inflating: government/documents/02/G00-02-1482390  \n",
            "  inflating: government/documents/02/G00-02-2667090  \n",
            "  inflating: government/documents/02/G00-02-1178594  \n",
            "  inflating: government/documents/02/G00-02-0345288  \n",
            "  inflating: government/documents/02/G00-02-4102909  \n",
            "  inflating: government/documents/02/G00-02-3848847  \n",
            "  inflating: government/documents/02/G00-02-1420159  \n",
            "  inflating: government/documents/02/G00-02-2545922  \n",
            "  inflating: government/documents/02/G00-02-1497234  \n",
            "  inflating: government/documents/02/G00-02-3406615  \n",
            "  inflating: government/documents/02/G00-02-4085175  \n",
            "  inflating: government/documents/02/G00-02-4120271  \n",
            "  inflating: government/documents/02/G00-02-0832578  \n",
            "  inflating: government/documents/02/G00-02-4057099  \n",
            "  inflating: government/documents/02/G00-02-2980653  \n",
            "  inflating: government/documents/02/G00-02-0322304  \n",
            "  inflating: government/documents/02/G00-02-0162779  \n",
            "  inflating: government/documents/02/G00-02-2397510  \n",
            "  inflating: government/documents/02/G00-02-2888949  \n",
            "  inflating: government/documents/02/G00-02-1239993  \n",
            "  inflating: government/documents/02/G00-02-0506607  \n",
            "  inflating: government/documents/02/G00-02-4012189  \n",
            "  inflating: government/documents/02/G00-02-0351712  \n",
            "  inflating: government/documents/02/G00-02-0460286  \n",
            "  inflating: government/documents/02/G00-02-3981961  \n",
            "  inflating: government/documents/02/G00-02-3773912  \n",
            "  inflating: government/documents/02/G00-02-1055630  \n",
            "  inflating: government/documents/02/G00-02-2091979  \n",
            "  inflating: government/documents/02/G00-02-2698369  \n",
            "  inflating: government/documents/02/G00-02-3103535  \n",
            "  inflating: government/documents/02/G00-02-0638292  \n",
            "  inflating: government/documents/02/G00-02-0709635  \n",
            "  inflating: government/documents/02/G00-02-0510219  \n",
            "  inflating: government/documents/02/G00-02-3807812  \n",
            "  inflating: government/documents/02/G00-02-1268088  \n",
            "  inflating: government/documents/02/G00-02-3474376  \n",
            "  inflating: government/documents/02/G00-02-1247639  \n",
            "   creating: government/documents/20/\n",
            "  inflating: government/documents/20/G00-20-0663829  \n",
            "  inflating: government/documents/20/G00-20-1122738  \n",
            "  inflating: government/documents/20/G00-20-3712065  \n",
            "  inflating: government/documents/20/G00-20-2958823  \n",
            "  inflating: government/documents/20/G00-20-2560826  \n",
            "  inflating: government/documents/20/G00-20-3831007  \n",
            "  inflating: government/documents/20/G00-20-1661394  \n",
            "  inflating: government/documents/20/G00-20-3178406  \n",
            "  inflating: government/documents/20/G00-20-0457640  \n",
            "  inflating: government/documents/20/G00-20-4189759  \n",
            "  inflating: government/documents/20/G00-20-1725319  \n",
            "  inflating: government/documents/20/G00-20-1794595  \n",
            "  inflating: government/documents/20/G00-20-4162998  \n",
            "  inflating: government/documents/20/G00-20-4115606  \n",
            "  inflating: government/documents/20/G00-20-2738363  \n",
            "  inflating: government/documents/20/G00-20-2252890  \n",
            "  inflating: government/documents/20/G00-20-0702603  \n",
            "  inflating: government/documents/20/G00-20-2050171  \n",
            "  inflating: government/documents/20/G00-20-0411511  \n",
            "  inflating: government/documents/20/G00-20-1390240  \n",
            "  inflating: government/documents/20/G00-20-1847296  \n",
            "  inflating: government/documents/20/G00-20-3485920  \n",
            "  inflating: government/documents/20/G00-20-3257357  \n",
            "  inflating: government/documents/20/G00-20-3691478  \n",
            "  inflating: government/documents/20/G00-20-1424362  \n",
            "  inflating: government/documents/20/G00-20-0962516  \n",
            "  inflating: government/documents/20/G00-20-1895990  \n",
            "  inflating: government/documents/20/G00-20-1229953  \n",
            "  inflating: government/documents/20/G00-20-2822947  \n",
            "  inflating: government/documents/20/G00-20-1054384  \n",
            "  inflating: government/documents/20/G00-20-1369470  \n",
            "  inflating: government/documents/20/G00-20-0365918  \n",
            "  inflating: government/documents/20/G00-20-3165511  \n",
            "  inflating: government/documents/20/G00-20-0452715  \n",
            "  inflating: government/documents/20/G00-20-3716305  \n",
            "  inflating: government/documents/20/G00-20-4144119  \n",
            "  inflating: government/documents/20/G00-20-3839216  \n",
            "  inflating: government/documents/20/G00-20-3678512  \n",
            "  inflating: government/documents/20/G00-20-0665105  \n",
            "   creating: government/documents/18/\n",
            "  inflating: government/documents/18/G00-18-3067678  \n",
            "  inflating: government/documents/18/G00-18-1959000  \n",
            "  inflating: government/documents/18/G00-18-3038584  \n",
            "  inflating: government/documents/18/G00-18-3456151  \n",
            "  inflating: government/documents/18/G00-18-2123316  \n",
            "  inflating: government/documents/18/G00-18-2681214  \n",
            "  inflating: government/documents/18/G00-18-3177883  \n",
            "  inflating: government/documents/18/G00-18-4062690  \n",
            "  inflating: government/documents/18/G00-18-1650811  \n",
            "  inflating: government/documents/18/G00-18-0299814  \n",
            "  inflating: government/documents/18/G00-18-3303047  \n",
            "  inflating: government/documents/18/G00-18-2965637  \n",
            "  inflating: government/documents/18/G00-18-3414640  \n",
            "  inflating: government/documents/18/G00-18-2332031  \n",
            "  inflating: government/documents/18/G00-18-2665440  \n",
            "  inflating: government/documents/18/G00-18-0398891  \n",
            "  inflating: government/documents/18/G00-18-2295266  \n",
            "  inflating: government/documents/18/G00-18-3201584  \n",
            "  inflating: government/documents/18/G00-18-2893727  \n",
            "  inflating: government/documents/18/G00-18-0124713  \n",
            "  inflating: government/documents/18/G00-18-1512949  \n",
            "  inflating: government/documents/18/G00-18-2276510  \n",
            "  inflating: government/documents/18/G00-18-3998125  \n",
            "  inflating: government/documents/18/G00-18-2902056  \n",
            "  inflating: government/documents/18/G00-18-3387229  \n",
            "  inflating: government/documents/18/G00-18-1572043  \n",
            "  inflating: government/documents/18/G00-18-4042118  \n",
            "  inflating: government/documents/18/G00-18-3191888  \n",
            "  inflating: government/documents/18/G00-18-0390418  \n",
            "  inflating: government/documents/18/G00-18-3647860  \n",
            "  inflating: government/documents/18/G00-18-3835186  \n",
            "  inflating: government/documents/18/G00-18-2154531  \n",
            "  inflating: government/documents/18/G00-18-0270371  \n",
            "  inflating: government/documents/18/G00-18-4015501  \n",
            "  inflating: government/documents/18/G00-18-3212032  \n",
            "  inflating: government/documents/18/G00-18-1492286  \n",
            "  inflating: government/documents/18/G00-18-1660493  \n",
            "  inflating: government/documents/18/G00-18-3874901  \n",
            "  inflating: government/documents/18/G00-18-2415462  \n",
            "  inflating: government/documents/18/G00-18-0401173  \n",
            "  inflating: government/documents/18/G00-18-1062516  \n",
            "  inflating: government/documents/18/G00-18-0374568  \n",
            "   creating: government/documents/27/\n",
            "  inflating: government/documents/27/G00-27-2423052  \n",
            "  inflating: government/documents/27/G00-27-4066921  \n",
            "  inflating: government/documents/27/G00-27-3762182  \n",
            "  inflating: government/documents/27/G00-27-1222379  \n",
            "  inflating: government/documents/27/G00-27-1471056  \n",
            "  inflating: government/documents/27/G00-27-3111091  \n",
            "  inflating: government/documents/27/G00-27-1315760  \n",
            "  inflating: government/documents/27/G00-27-0576256  \n",
            "  inflating: government/documents/27/G00-27-2669897  \n",
            "  inflating: government/documents/27/G00-27-0038969  \n",
            "  inflating: government/documents/27/G00-27-0709614  \n",
            "  inflating: government/documents/27/G00-27-1492903  \n",
            "  inflating: government/documents/27/G00-27-2159399  \n",
            "  inflating: government/documents/27/G00-27-0369252  \n",
            "  inflating: government/documents/27/G00-27-0613752  \n",
            "  inflating: government/documents/27/G00-27-0561556  \n",
            "  inflating: government/documents/27/G00-27-3174236  \n",
            "  inflating: government/documents/27/G00-27-1448789  \n",
            "  inflating: government/documents/27/G00-27-4125444  \n",
            "  inflating: government/documents/27/G00-27-2344245  \n",
            "  inflating: government/documents/27/G00-27-2578980  \n",
            "  inflating: government/documents/27/G00-27-2319443  \n",
            "  inflating: government/documents/27/G00-27-3649313  \n",
            "  inflating: government/documents/27/G00-27-2014375  \n",
            "  inflating: government/documents/27/G00-27-0467489  \n",
            "  inflating: government/documents/27/G00-27-3209717  \n",
            "  inflating: government/documents/27/G00-27-3165010  \n",
            "  inflating: government/documents/27/G00-27-2661531  \n",
            "  inflating: government/documents/27/G00-27-2048511  \n",
            "   creating: government/documents/11/\n",
            "  inflating: government/documents/11/G00-11-3066108  \n",
            "  inflating: government/documents/11/G00-11-1357603  \n",
            "  inflating: government/documents/11/G00-11-0752088  \n",
            "  inflating: government/documents/11/G00-11-0641521  \n",
            "  inflating: government/documents/11/G00-11-0021900  \n",
            "  inflating: government/documents/11/G00-11-3020925  \n",
            "  inflating: government/documents/11/G00-11-0221068  \n",
            "  inflating: government/documents/11/G00-11-0873436  \n",
            "  inflating: government/documents/11/G00-11-0354032  \n",
            "  inflating: government/documents/11/G00-11-0502740  \n",
            "  inflating: government/documents/11/G00-11-0737967  \n",
            "  inflating: government/documents/11/G00-11-1493113  \n",
            "  inflating: government/documents/11/G00-11-0700527  \n",
            "  inflating: government/documents/11/G00-11-0223618  \n",
            "  inflating: government/documents/11/G00-11-1703665  \n",
            "  inflating: government/documents/11/G00-11-0243725  \n",
            "  inflating: government/documents/11/G00-11-1453588  \n",
            "  inflating: government/documents/11/G00-11-0750274  \n",
            "  inflating: government/documents/11/G00-11-1277616  \n",
            "  inflating: government/documents/11/G00-11-3446258  \n",
            "  inflating: government/documents/11/G00-11-1650256  \n",
            "  inflating: government/documents/11/G00-11-2005001  \n",
            "  inflating: government/documents/11/G00-11-2906082  \n",
            "  inflating: government/documents/11/G00-11-0752432  \n",
            "  inflating: government/documents/11/G00-11-0732122  \n",
            "  inflating: government/documents/11/G00-11-3028759  \n",
            "  inflating: government/documents/11/G00-11-0920146  \n",
            "  inflating: government/documents/11/G00-11-3027786  \n",
            "  inflating: government/documents/11/G00-11-0182346  \n",
            "  inflating: government/documents/11/G00-11-0456889  \n",
            "  inflating: government/documents/11/G00-11-0101683  \n",
            "  inflating: government/documents/11/G00-11-3596661  \n",
            "  inflating: government/documents/11/G00-11-2857124  \n",
            "  inflating: government/documents/11/G00-11-0770745  \n",
            "  inflating: government/documents/11/G00-11-4105562  \n",
            "  inflating: government/documents/11/G00-11-4000101  \n",
            "  inflating: government/documents/11/G00-11-3911519  \n",
            "  inflating: government/documents/11/G00-11-3415802  \n",
            "  inflating: government/documents/11/G00-11-1714751  \n",
            "  inflating: government/documents/11/G00-11-1987490  \n",
            "  inflating: government/documents/11/G00-11-3436507  \n",
            "  inflating: government/documents/11/G00-11-3550835  \n",
            "  inflating: government/documents/11/G00-11-3610332  \n",
            "  inflating: government/documents/11/G00-11-2958190  \n",
            "  inflating: government/documents/11/G00-11-3704445  \n",
            "  inflating: government/documents/11/G00-11-1389065  \n",
            "  inflating: government/documents/11/G00-11-1809773  \n",
            "  inflating: government/documents/11/G00-11-1749431  \n",
            "  inflating: government/documents/11/G00-11-3773963  \n",
            "  inflating: government/documents/11/G00-11-0184364  \n",
            "  inflating: government/documents/11/G00-11-0225161  \n",
            "  inflating: government/documents/11/G00-11-3448596  \n",
            "  inflating: government/documents/11/G00-11-0042157  \n",
            "  inflating: government/documents/11/G00-11-2137476  \n",
            "  inflating: government/documents/11/G00-11-2590367  \n",
            "  inflating: government/documents/11/G00-11-2416074  \n",
            "  inflating: government/documents/11/G00-11-2794349  \n",
            "  inflating: government/documents/11/G00-11-3493478  \n",
            "  inflating: government/documents/11/G00-11-0575949  \n",
            "  inflating: government/documents/11/G00-11-2000231  \n",
            "  inflating: government/documents/11/G00-11-0909095  \n",
            "  inflating: government/documents/11/G00-11-0450519  \n",
            "   creating: government/documents/29/\n",
            "  inflating: government/documents/29/G00-29-1445117  \n",
            "  inflating: government/documents/29/G00-29-1316899  \n",
            "  inflating: government/documents/29/G00-29-3298118  \n",
            "  inflating: government/documents/29/G00-29-0283205  \n",
            "  inflating: government/documents/29/G00-29-1467190  \n",
            "  inflating: government/documents/29/G00-29-3781694  \n",
            "  inflating: government/documents/29/G00-29-0173341  \n",
            "  inflating: government/documents/29/G00-29-2968932  \n",
            "  inflating: government/documents/29/G00-29-4124951  \n",
            "  inflating: government/documents/29/G00-29-0924012  \n",
            "  inflating: government/documents/29/G00-29-3005305  \n",
            "  inflating: government/documents/29/G00-29-1544211  \n",
            "  inflating: government/documents/29/G00-29-2589930  \n",
            "  inflating: government/documents/29/G00-29-0601812  \n",
            "  inflating: government/documents/29/G00-29-0596915  \n",
            "  inflating: government/documents/29/G00-29-3538849  \n",
            "  inflating: government/documents/29/G00-29-3689678  \n",
            "  inflating: government/documents/29/G00-29-3111356  \n",
            "  inflating: government/documents/29/G00-29-1949417  \n",
            "  inflating: government/documents/29/G00-29-2689569  \n",
            "  inflating: government/documents/29/G00-29-1907477  \n",
            "  inflating: government/documents/29/G00-29-2643876  \n",
            "  inflating: government/documents/29/G00-29-2908102  \n",
            "  inflating: government/documents/29/G00-29-4107050  \n",
            "  inflating: government/documents/29/G00-29-0864992  \n",
            "  inflating: government/documents/29/G00-29-0994948  \n",
            "  inflating: government/documents/29/G00-29-2343040  \n",
            "  inflating: government/documents/29/G00-29-1977811  \n",
            "  inflating: government/documents/29/G00-29-2159472  \n",
            "  inflating: government/documents/29/G00-29-2678164  \n",
            "  inflating: government/documents/29/G00-29-0371794  \n",
            "  inflating: government/documents/29/G00-29-3865087  \n",
            "  inflating: government/documents/29/G00-29-2466763  \n",
            "  inflating: government/documents/29/G00-29-1284645  \n",
            "  inflating: government/documents/29/G00-29-3287865  \n",
            "  inflating: government/documents/29/G00-29-3326169  \n",
            "  inflating: government/documents/29/G00-29-3434101  \n",
            "   creating: government/documents/16/\n",
            "  inflating: government/documents/16/G00-16-3395917  \n",
            "  inflating: government/documents/16/G00-16-0165956  \n",
            "  inflating: government/documents/16/G00-16-2479016  \n",
            "  inflating: government/documents/16/G00-16-3871314  \n",
            "  inflating: government/documents/16/G00-16-2494170  \n",
            "  inflating: government/documents/16/G00-16-0858333  \n",
            "  inflating: government/documents/16/G00-16-2165994  \n",
            "  inflating: government/documents/16/G00-16-0625124  \n",
            "  inflating: government/documents/16/G00-16-1636821  \n",
            "  inflating: government/documents/16/G00-16-1314498  \n",
            "  inflating: government/documents/16/G00-16-1821796  \n",
            "  inflating: government/documents/16/G00-16-3952320  \n",
            "  inflating: government/documents/16/G00-16-2086899  \n",
            "  inflating: government/documents/16/G00-16-2895420  \n",
            "  inflating: government/documents/16/G00-16-2432196  \n",
            "  inflating: government/documents/16/G00-16-3820731  \n",
            "  inflating: government/documents/16/G00-16-1355187  \n",
            "  inflating: government/documents/16/G00-16-2750126  \n",
            "  inflating: government/documents/16/G00-16-0059045  \n",
            "  inflating: government/documents/16/G00-16-1925033  \n",
            "  inflating: government/documents/16/G00-16-1854115  \n",
            "  inflating: government/documents/16/G00-16-1088156  \n",
            "  inflating: government/documents/16/G00-16-2422077  \n",
            "  inflating: government/documents/16/G00-16-1397775  \n",
            "  inflating: government/documents/16/G00-16-0550405  \n",
            "  inflating: government/documents/16/G00-16-4067677  \n",
            "  inflating: government/documents/16/G00-16-2963616  \n",
            "   creating: government/documents/42/\n",
            "  inflating: government/documents/42/G00-42-4051351  \n",
            "  inflating: government/documents/42/G00-42-3197837  \n",
            "  inflating: government/documents/42/G00-42-1930382  \n",
            "  inflating: government/documents/42/G00-42-3697613  \n",
            "  inflating: government/documents/42/G00-42-1830673  \n",
            "  inflating: government/documents/42/G00-42-1060597  \n",
            "  inflating: government/documents/42/G00-42-2344886  \n",
            "  inflating: government/documents/42/G00-42-1312823  \n",
            "  inflating: government/documents/42/G00-42-3117860  \n",
            "  inflating: government/documents/42/G00-42-2134920  \n",
            "  inflating: government/documents/42/G00-42-1280190  \n",
            "  inflating: government/documents/42/G00-42-1042518  \n",
            "  inflating: government/documents/42/G00-42-0907520  \n",
            "  inflating: government/documents/42/G00-42-3826078  \n",
            "  inflating: government/documents/42/G00-42-2097415  \n",
            "  inflating: government/documents/42/G00-42-0547574  \n",
            "  inflating: government/documents/42/G00-42-3384778  \n",
            "  inflating: government/documents/42/G00-42-3198851  \n",
            "  inflating: government/documents/42/G00-42-0703388  \n",
            "  inflating: government/documents/42/G00-42-1249322  \n",
            "  inflating: government/documents/42/G00-42-1455285  \n",
            "  inflating: government/documents/42/G00-42-1179044  \n",
            "  inflating: government/documents/42/G00-42-2047705  \n",
            "  inflating: government/documents/42/G00-42-1834554  \n",
            "  inflating: government/documents/42/G00-42-0000000  \n",
            "   creating: government/documents/89/\n",
            "  inflating: government/documents/89/G00-89-2699905  \n",
            "  inflating: government/documents/89/G00-89-0436944  \n",
            "  inflating: government/documents/89/G00-89-3995270  \n",
            "  inflating: government/documents/89/G00-89-0026341  \n",
            "  inflating: government/documents/89/G00-89-0044196  \n",
            "  inflating: government/documents/89/G00-89-4174369  \n",
            "  inflating: government/documents/89/G00-89-3143135  \n",
            "  inflating: government/documents/89/G00-89-1405587  \n",
            "  inflating: government/documents/89/G00-89-3698265  \n",
            "  inflating: government/documents/89/G00-89-3734824  \n",
            "  inflating: government/documents/89/G00-89-1037295  \n",
            "  inflating: government/documents/89/G00-89-3353735  \n",
            "  inflating: government/documents/89/G00-89-0341103  \n",
            "  inflating: government/documents/89/G00-89-0439830  \n",
            "  inflating: government/documents/89/G00-89-1075429  \n",
            "  inflating: government/documents/89/G00-89-3576342  \n",
            "  inflating: government/documents/89/G00-89-3970763  \n",
            "  inflating: government/documents/89/G00-89-2116689  \n",
            "  inflating: government/documents/89/G00-89-0780322  \n",
            "  inflating: government/documents/89/G00-89-3204498  \n",
            "  inflating: government/documents/89/G00-89-3578129  \n",
            "  inflating: government/documents/89/G00-89-0896876  \n",
            "  inflating: government/documents/89/G00-89-1575855  \n",
            "  inflating: government/documents/89/G00-89-3859322  \n",
            "  inflating: government/documents/89/G00-89-1794146  \n",
            "  inflating: government/documents/89/G00-89-0106321  \n",
            "  inflating: government/documents/89/G00-89-0590575  \n",
            "  inflating: government/documents/89/G00-89-0575056  \n",
            "  inflating: government/documents/89/G00-89-0076357  \n",
            "  inflating: government/documents/89/G00-89-3274987  \n",
            "  inflating: government/documents/89/G00-89-0765423  \n",
            "  inflating: government/documents/89/G00-89-0735391  \n",
            "  inflating: government/documents/89/G00-89-0031748  \n",
            "  inflating: government/documents/89/G00-89-0000000  \n",
            "  inflating: government/documents/89/G00-89-1911969  \n",
            "   creating: government/documents/45/\n",
            "  inflating: government/documents/45/G00-45-1444296  \n",
            "  inflating: government/documents/45/G00-45-0809730  \n",
            "  inflating: government/documents/45/G00-45-1398664  \n",
            "  inflating: government/documents/45/G00-45-1301562  \n",
            "  inflating: government/documents/45/G00-45-1367785  \n",
            "  inflating: government/documents/45/G00-45-4032177  \n",
            "  inflating: government/documents/45/G00-45-1486246  \n",
            "  inflating: government/documents/45/G00-45-0568572  \n",
            "  inflating: government/documents/45/G00-45-2538832  \n",
            "  inflating: government/documents/45/G00-45-2200333  \n",
            "  inflating: government/documents/45/G00-45-4075347  \n",
            "  inflating: government/documents/45/G00-45-4025848  \n",
            "  inflating: government/documents/45/G00-45-2623195  \n",
            "  inflating: government/documents/45/G00-45-0288610  \n",
            "  inflating: government/documents/45/G00-45-1977767  \n",
            "  inflating: government/documents/45/G00-45-3385318  \n",
            "  inflating: government/documents/45/G00-45-2019689  \n",
            "  inflating: government/documents/45/G00-45-0332043  \n",
            "  inflating: government/documents/45/G00-45-0636707  \n",
            "  inflating: government/documents/45/G00-45-0778096  \n",
            "  inflating: government/documents/45/G00-45-1254880  \n",
            "  inflating: government/documents/45/G00-45-1365497  \n",
            "  inflating: government/documents/45/G00-45-0349368  \n",
            "  inflating: government/documents/45/G00-45-1836148  \n",
            "  inflating: government/documents/45/G00-45-0006211  \n",
            "  inflating: government/documents/45/G00-45-1896827  \n",
            "  inflating: government/documents/45/G00-45-3351605  \n",
            "  inflating: government/documents/45/G00-45-0770756  \n",
            "  inflating: government/documents/45/G00-45-0028267  \n",
            "  inflating: government/documents/45/G00-45-2059894  \n",
            "  inflating: government/documents/45/G00-45-3659650  \n",
            "   creating: government/documents/73/\n",
            "  inflating: government/documents/73/G00-73-0660295  \n",
            "  inflating: government/documents/73/G00-73-0828819  \n",
            "  inflating: government/documents/73/G00-73-0679375  \n",
            "  inflating: government/documents/73/G00-73-3287750  \n",
            "  inflating: government/documents/73/G00-73-3056531  \n",
            "  inflating: government/documents/73/G00-73-2514072  \n",
            "  inflating: government/documents/73/G00-73-2016800  \n",
            "  inflating: government/documents/73/G00-73-1397304  \n",
            "  inflating: government/documents/73/G00-73-2863491  \n",
            "  inflating: government/documents/73/G00-73-3632837  \n",
            "  inflating: government/documents/73/G00-73-4188628  \n",
            "  inflating: government/documents/73/G00-73-0473672  \n",
            "  inflating: government/documents/73/G00-73-0532559  \n",
            "  inflating: government/documents/73/G00-73-0299639  \n",
            "  inflating: government/documents/73/G00-73-2028789  \n",
            "  inflating: government/documents/73/G00-73-0543133  \n",
            "  inflating: government/documents/73/G00-73-2821514  \n",
            "  inflating: government/documents/73/G00-73-1175424  \n",
            "  inflating: government/documents/73/G00-73-0028862  \n",
            "  inflating: government/documents/73/G00-73-1631747  \n",
            "  inflating: government/documents/73/G00-73-2648815  \n",
            "  inflating: government/documents/73/G00-73-3912994  \n",
            "  inflating: government/documents/73/G00-73-0692182  \n",
            "  inflating: government/documents/73/G00-73-2660773  \n",
            "  inflating: government/documents/73/G00-73-1212891  \n",
            "  inflating: government/documents/73/G00-73-1766422  \n",
            "  inflating: government/documents/73/G00-73-3314008  \n",
            "  inflating: government/documents/73/G00-73-3937547  \n",
            "  inflating: government/documents/73/G00-73-4011981  \n",
            "  inflating: government/documents/73/G00-73-3170442  \n",
            "  inflating: government/documents/73/G00-73-1499832  \n",
            "  inflating: government/documents/73/G00-73-2173103  \n",
            "  inflating: government/documents/73/G00-73-2455063  \n",
            "   creating: government/documents/87/\n",
            "  inflating: government/documents/87/G00-87-4011875  \n",
            "  inflating: government/documents/87/G00-87-3716404  \n",
            "  inflating: government/documents/87/G00-87-1718107  \n",
            "  inflating: government/documents/87/G00-87-1278195  \n",
            "  inflating: government/documents/87/G00-87-3746944  \n",
            "  inflating: government/documents/87/G00-87-1706069  \n",
            "  inflating: government/documents/87/G00-87-0803651  \n",
            "  inflating: government/documents/87/G00-87-2683912  \n",
            "  inflating: government/documents/87/G00-87-3904576  \n",
            "  inflating: government/documents/87/G00-87-2595258  \n",
            "  inflating: government/documents/87/G00-87-1292162  \n",
            "  inflating: government/documents/87/G00-87-0152864  \n",
            "  inflating: government/documents/87/G00-87-2907453  \n",
            "  inflating: government/documents/87/G00-87-1492488  \n",
            "  inflating: government/documents/87/G00-87-1860591  \n",
            "  inflating: government/documents/87/G00-87-2357947  \n",
            "  inflating: government/documents/87/G00-87-1872370  \n",
            "  inflating: government/documents/87/G00-87-1353606  \n",
            "  inflating: government/documents/87/G00-87-1583924  \n",
            "  inflating: government/documents/87/G00-87-2869473  \n",
            "  inflating: government/documents/87/G00-87-0765639  \n",
            "  inflating: government/documents/87/G00-87-3743915  \n",
            "  inflating: government/documents/87/G00-87-2615310  \n",
            "  inflating: government/documents/87/G00-87-0777423  \n",
            "  inflating: government/documents/87/G00-87-2843876  \n",
            "  inflating: government/documents/87/G00-87-2625357  \n",
            "  inflating: government/documents/87/G00-87-0601767  \n",
            "  inflating: government/documents/87/G00-87-1422945  \n",
            "  inflating: government/documents/87/G00-87-2091907  \n",
            "  inflating: government/documents/87/G00-87-2839296  \n",
            "  inflating: government/documents/87/G00-87-2023843  \n",
            "  inflating: government/documents/87/G00-87-0559585  \n",
            "  inflating: government/documents/87/G00-87-2416961  \n",
            "  inflating: government/documents/87/G00-87-1262283  \n",
            "  inflating: government/documents/87/G00-87-3226757  \n",
            "   creating: government/documents/80/\n",
            "  inflating: government/documents/80/G00-80-1408326  \n",
            "  inflating: government/documents/80/G00-80-3757269  \n",
            "  inflating: government/documents/80/G00-80-2865414  \n",
            "  inflating: government/documents/80/G00-80-1849469  \n",
            "  inflating: government/documents/80/G00-80-2913262  \n",
            "  inflating: government/documents/80/G00-80-2670974  \n",
            "  inflating: government/documents/80/G00-80-0624606  \n",
            "  inflating: government/documents/80/G00-80-1369664  \n",
            "  inflating: government/documents/80/G00-80-0022286  \n",
            "  inflating: government/documents/80/G00-80-0235938  \n",
            "  inflating: government/documents/80/G00-80-1413929  \n",
            "  inflating: government/documents/80/G00-80-2769217  \n",
            "  inflating: government/documents/80/G00-80-0615081  \n",
            "  inflating: government/documents/80/G00-80-2348597  \n",
            "  inflating: government/documents/80/G00-80-0939068  \n",
            "  inflating: government/documents/80/G00-80-0880722  \n",
            "  inflating: government/documents/80/G00-80-2174888  \n",
            "  inflating: government/documents/80/G00-80-1927471  \n",
            "  inflating: government/documents/80/G00-80-4041999  \n",
            "  inflating: government/documents/80/G00-80-2792256  \n",
            "  inflating: government/documents/80/G00-80-1342091  \n",
            "  inflating: government/documents/80/G00-80-0569048  \n",
            "  inflating: government/documents/80/G00-80-1682195  \n",
            "  inflating: government/documents/80/G00-80-3027088  \n",
            "  inflating: government/documents/80/G00-80-2737758  \n",
            "  inflating: government/documents/80/G00-80-2143781  \n",
            "  inflating: government/documents/80/G00-80-2773459  \n",
            "  inflating: government/documents/80/G00-80-3361450  \n",
            "  inflating: government/documents/80/G00-80-1361789  \n",
            "  inflating: government/documents/80/G00-80-3548931  \n",
            "  inflating: government/documents/80/G00-80-0018653  \n",
            "  inflating: government/documents/80/G00-80-1112147  \n",
            "  inflating: government/documents/80/G00-80-3838765  \n",
            "  inflating: government/documents/80/G00-80-2126087  \n",
            "   creating: government/documents/74/\n",
            "  inflating: government/documents/74/G00-74-1029096  \n",
            "  inflating: government/documents/74/G00-74-2775206  \n",
            "  inflating: government/documents/74/G00-74-3136755  \n",
            "  inflating: government/documents/74/G00-74-0751435  \n",
            "  inflating: government/documents/74/G00-74-3926994  \n",
            "  inflating: government/documents/74/G00-74-1546702  \n",
            "  inflating: government/documents/74/G00-74-1664646  \n",
            "  inflating: government/documents/74/G00-74-2304784  \n",
            "  inflating: government/documents/74/G00-74-3136022  \n",
            "  inflating: government/documents/74/G00-74-2056534  \n",
            "  inflating: government/documents/74/G00-74-1330452  \n",
            "  inflating: government/documents/74/G00-74-0115318  \n",
            "  inflating: government/documents/74/G00-74-2680835  \n",
            "  inflating: government/documents/74/G00-74-0697089  \n",
            "  inflating: government/documents/74/G00-74-4109187  \n",
            "  inflating: government/documents/74/G00-74-0468546  \n",
            "  inflating: government/documents/74/G00-74-1291250  \n",
            "  inflating: government/documents/74/G00-74-2685794  \n",
            "  inflating: government/documents/74/G00-74-4030396  \n",
            "  inflating: government/documents/74/G00-74-3829250  \n",
            "  inflating: government/documents/74/G00-74-2695472  \n",
            "  inflating: government/documents/74/G00-74-1280036  \n",
            "  inflating: government/documents/74/G00-74-2621580  \n",
            "  inflating: government/documents/74/G00-74-1394517  \n",
            "  inflating: government/documents/74/G00-74-3490580  \n",
            "  inflating: government/documents/74/G00-74-1802348  \n",
            "  inflating: government/documents/74/G00-74-1944411  \n",
            "  inflating: government/documents/74/G00-74-2098221  \n",
            "  inflating: government/documents/74/G00-74-3530847  \n",
            "  inflating: government/documents/74/G00-74-2972556  \n",
            "   creating: government/documents/28/\n",
            "  inflating: government/documents/28/G00-28-3936026  \n",
            "  inflating: government/documents/28/G00-28-0596013  \n",
            "  inflating: government/documents/28/G00-28-0007323  \n",
            "  inflating: government/documents/28/G00-28-2286602  \n",
            "  inflating: government/documents/28/G00-28-0626415  \n",
            "  inflating: government/documents/28/G00-28-1462904  \n",
            "  inflating: government/documents/28/G00-28-3705847  \n",
            "  inflating: government/documents/28/G00-28-2516057  \n",
            "  inflating: government/documents/28/G00-28-0647082  \n",
            "  inflating: government/documents/28/G00-28-0666504  \n",
            "  inflating: government/documents/28/G00-28-4051184  \n",
            "  inflating: government/documents/28/G00-28-3568199  \n",
            "  inflating: government/documents/28/G00-28-3558822  \n",
            "  inflating: government/documents/28/G00-28-0052103  \n",
            "  inflating: government/documents/28/G00-28-0691401  \n",
            "  inflating: government/documents/28/G00-28-1394199  \n",
            "  inflating: government/documents/28/G00-28-2735649  \n",
            "  inflating: government/documents/28/G00-28-3095840  \n",
            "  inflating: government/documents/28/G00-28-3598417  \n",
            "  inflating: government/documents/28/G00-28-1994826  \n",
            "  inflating: government/documents/28/G00-28-2254800  \n",
            "  inflating: government/documents/28/G00-28-3127444  \n",
            "  inflating: government/documents/28/G00-28-0628343  \n",
            "  inflating: government/documents/28/G00-28-3518400  \n",
            "  inflating: government/documents/28/G00-28-0286112  \n",
            "  inflating: government/documents/28/G00-28-2499271  \n",
            "  inflating: government/documents/28/G00-28-0956700  \n",
            "  inflating: government/documents/28/G00-28-3008506  \n",
            "  inflating: government/documents/28/G00-28-0463390  \n",
            "  inflating: government/documents/28/G00-28-2250792  \n",
            "  inflating: government/documents/28/G00-28-0867888  \n",
            "  inflating: government/documents/28/G00-28-2535373  \n",
            "  inflating: government/documents/28/G00-28-1450394  \n",
            "  inflating: government/documents/28/G00-28-0847842  \n",
            "   creating: government/documents/17/\n",
            "  inflating: government/documents/17/G00-17-0194527  \n",
            "  inflating: government/documents/17/G00-17-2521440  \n",
            "  inflating: government/documents/17/G00-17-4060420  \n",
            "  inflating: government/documents/17/G00-17-2881652  \n",
            "  inflating: government/documents/17/G00-17-0059415  \n",
            "  inflating: government/documents/17/G00-17-1893099  \n",
            "  inflating: government/documents/17/G00-17-3586054  \n",
            "  inflating: government/documents/17/G00-17-4048455  \n",
            "  inflating: government/documents/17/G00-17-0942793  \n",
            "  inflating: government/documents/17/G00-17-1319093  \n",
            "  inflating: government/documents/17/G00-17-3429531  \n",
            "  inflating: government/documents/17/G00-17-0868458  \n",
            "  inflating: government/documents/17/G00-17-2871119  \n",
            "  inflating: government/documents/17/G00-17-1961907  \n",
            "  inflating: government/documents/17/G00-17-2133216  \n",
            "  inflating: government/documents/17/G00-17-1387724  \n",
            "  inflating: government/documents/17/G00-17-3612287  \n",
            "  inflating: government/documents/17/G00-17-1802707  \n",
            "  inflating: government/documents/17/G00-17-1289316  \n",
            "  inflating: government/documents/17/G00-17-1852046  \n",
            "  inflating: government/documents/17/G00-17-3630567  \n",
            "  inflating: government/documents/17/G00-17-1742060  \n",
            "  inflating: government/documents/17/G00-17-3941780  \n",
            "  inflating: government/documents/17/G00-17-1193464  \n",
            "  inflating: government/documents/17/G00-17-2845878  \n",
            "  inflating: government/documents/17/G00-17-1200812  \n",
            "  inflating: government/documents/17/G00-17-1006883  \n",
            "  inflating: government/documents/17/G00-17-3563194  \n",
            "  inflating: government/documents/17/G00-17-0641985  \n",
            "  inflating: government/documents/17/G00-17-2974871  \n",
            "  inflating: government/documents/17/G00-17-2173043  \n",
            "  inflating: government/documents/17/G00-17-0424066  \n",
            "  inflating: government/documents/17/G00-17-0115496  \n",
            "  inflating: government/documents/17/G00-17-3433782  \n",
            "  inflating: government/documents/17/G00-17-1818132  \n",
            "  inflating: government/documents/17/G00-17-3027297  \n",
            "  inflating: government/documents/17/G00-17-0275027  \n",
            "  inflating: government/documents/17/G00-17-0502557  \n",
            "   creating: government/documents/10/\n",
            "  inflating: government/documents/10/G00-10-2836439  \n",
            "  inflating: government/documents/10/G00-10-2085310  \n",
            "  inflating: government/documents/10/G00-10-4163494  \n",
            "  inflating: government/documents/10/G00-10-0704121  \n",
            "  inflating: government/documents/10/G00-10-0136170  \n",
            "  inflating: government/documents/10/G00-10-3849661  \n",
            "  inflating: government/documents/10/G00-10-2344253  \n",
            "  inflating: government/documents/10/G00-10-1086431  \n",
            "  inflating: government/documents/10/G00-10-3149786  \n",
            "  inflating: government/documents/10/G00-10-3730888  \n",
            "  inflating: government/documents/10/G00-10-2871392  \n",
            "  inflating: government/documents/10/G00-10-0952892  \n",
            "  inflating: government/documents/10/G00-10-4131219  \n",
            "  inflating: government/documents/10/G00-10-3713897  \n",
            "  inflating: government/documents/10/G00-10-2753840  \n",
            "  inflating: government/documents/10/G00-10-2990313  \n",
            "  inflating: government/documents/10/G00-10-3222851  \n",
            "  inflating: government/documents/10/G00-10-1437219  \n",
            "  inflating: government/documents/10/G00-10-2011620  \n",
            "  inflating: government/documents/10/G00-10-2754349  \n",
            "  inflating: government/documents/10/G00-10-1308595  \n",
            "  inflating: government/documents/10/G00-10-0460101  \n",
            "  inflating: government/documents/10/G00-10-1001660  \n",
            "  inflating: government/documents/10/G00-10-0844304  \n",
            "  inflating: government/documents/10/G00-10-1055581  \n",
            "  inflating: government/documents/10/G00-10-0673072  \n",
            "  inflating: government/documents/10/G00-10-4133035  \n",
            "  inflating: government/documents/10/G00-10-3331481  \n",
            "  inflating: government/documents/10/G00-10-0361005  \n",
            "  inflating: government/documents/10/G00-10-0525087  \n",
            "  inflating: government/documents/10/G00-10-0818874  \n",
            "  inflating: government/documents/10/G00-10-1316251  \n",
            "  inflating: government/documents/10/G00-10-0690201  \n",
            "  inflating: government/documents/10/G00-10-2024294  \n",
            "  inflating: government/documents/10/G00-10-3117896  \n",
            "  inflating: government/documents/10/G00-10-2989359  \n",
            "  inflating: government/documents/10/G00-10-2673853  \n",
            "  inflating: government/documents/10/G00-10-3137809  \n",
            "  inflating: government/documents/10/G00-10-2477398  \n",
            "  inflating: government/documents/10/G00-10-0106475  \n",
            "  inflating: government/documents/10/G00-10-0664217  \n",
            "  inflating: government/documents/10/G00-10-2719392  \n",
            "  inflating: government/documents/10/G00-10-0798979  \n",
            "  inflating: government/documents/10/G00-10-2151573  \n",
            "  inflating: government/documents/10/G00-10-4116640  \n",
            "  inflating: government/documents/10/G00-10-0814583  \n",
            "  inflating: government/documents/10/G00-10-3892683  \n",
            "  inflating: government/documents/10/G00-10-1034424  \n",
            "  inflating: government/documents/10/G00-10-4006951  \n",
            "  inflating: government/documents/10/G00-10-1470109  \n",
            "  inflating: government/documents/10/G00-10-1931044  \n",
            "  inflating: government/documents/10/G00-10-0741937  \n",
            "  inflating: government/documents/10/G00-10-0603637  \n",
            "  inflating: government/documents/10/G00-10-1731941  \n",
            "  inflating: government/documents/10/G00-10-0140128  \n",
            "  inflating: government/documents/10/G00-10-1494526  \n",
            "  inflating: government/documents/10/G00-10-0994585  \n",
            "  inflating: government/documents/10/G00-10-0281044  \n",
            "  inflating: government/documents/10/G00-10-1389053  \n",
            "  inflating: government/documents/10/G00-10-1135665  \n",
            "  inflating: government/documents/10/G00-10-1966794  \n",
            "  inflating: government/documents/10/G00-10-4080725  \n",
            "  inflating: government/documents/10/G00-10-2392581  \n",
            "  inflating: government/documents/10/G00-10-1960260  \n",
            "  inflating: government/documents/10/G00-10-0305437  \n",
            "  inflating: government/documents/10/G00-10-1843562  \n",
            "  inflating: government/documents/10/G00-10-0668261  \n",
            "  inflating: government/documents/10/G00-10-0085031  \n",
            "  inflating: government/documents/10/G00-10-1133627  \n",
            "  inflating: government/documents/10/G00-10-3982634  \n",
            "  inflating: government/documents/10/G00-10-4170332  \n",
            "  inflating: government/documents/10/G00-10-3302265  \n",
            "  inflating: government/documents/10/G00-10-4076121  \n",
            "  inflating: government/documents/10/G00-10-0865843  \n",
            "  inflating: government/documents/10/G00-10-0002983  \n",
            "  inflating: government/documents/10/G00-10-0300401  \n",
            "  inflating: government/documents/10/G00-10-2917540  \n",
            "  inflating: government/documents/10/G00-10-3941892  \n",
            "  inflating: government/documents/10/G00-10-3758881  \n",
            "  inflating: government/documents/10/G00-10-2794731  \n",
            "  inflating: government/documents/10/G00-10-2109040  \n",
            "  inflating: government/documents/10/G00-10-1806241  \n",
            "  inflating: government/documents/10/G00-10-4090850  \n",
            "  inflating: government/documents/10/G00-10-2771685  \n",
            "  inflating: government/documents/10/G00-10-1443691  \n",
            "  inflating: government/documents/10/G00-10-2946905  \n",
            "  inflating: government/documents/10/G00-10-1754459  \n",
            "  inflating: government/documents/10/G00-10-0666948  \n",
            "  inflating: government/documents/10/G00-10-3170109  \n",
            "  inflating: government/documents/10/G00-10-0506282  \n",
            "  inflating: government/documents/10/G00-10-0606850  \n",
            "  inflating: government/documents/10/G00-10-3382518  \n",
            "  inflating: government/documents/10/G00-10-0479458  \n",
            "  inflating: government/documents/10/G00-10-3858249  \n",
            "  inflating: government/documents/10/G00-10-0902276  \n",
            "  inflating: government/documents/10/G00-10-2861833  \n",
            "  inflating: government/documents/10/G00-10-2418697  \n",
            "  inflating: government/documents/10/G00-10-0571379  \n",
            "  inflating: government/documents/10/G00-10-1509107  \n",
            "  inflating: government/documents/10/G00-10-3392667  \n",
            "  inflating: government/documents/10/G00-10-3017657  \n",
            "  inflating: government/documents/10/G00-10-1957853  \n",
            "  inflating: government/documents/10/G00-10-1266735  \n",
            "  inflating: government/documents/10/G00-10-1232247  \n",
            "  inflating: government/documents/10/G00-10-3116656  \n",
            "  inflating: government/documents/10/G00-10-1856404  \n",
            "  inflating: government/documents/10/G00-10-0074663  \n",
            "  inflating: government/documents/10/G00-10-0852358  \n",
            "  inflating: government/documents/10/G00-10-3090724  \n",
            "   creating: government/documents/19/\n",
            "  inflating: government/documents/19/G00-19-2536913  \n",
            "  inflating: government/documents/19/G00-19-2401614  \n",
            "  inflating: government/documents/19/G00-19-3991415  \n",
            "  inflating: government/documents/19/G00-19-2706356  \n",
            "  inflating: government/documents/19/G00-19-1884413  \n",
            "  inflating: government/documents/19/G00-19-2280669  \n",
            "  inflating: government/documents/19/G00-19-0127152  \n",
            "  inflating: government/documents/19/G00-19-1613877  \n",
            "  inflating: government/documents/19/G00-19-3750659  \n",
            "  inflating: government/documents/19/G00-19-2625954  \n",
            "  inflating: government/documents/19/G00-19-0629904  \n",
            "  inflating: government/documents/19/G00-19-2896720  \n",
            "  inflating: government/documents/19/G00-19-2800132  \n",
            "  inflating: government/documents/19/G00-19-1335309  \n",
            "  inflating: government/documents/19/G00-19-2066142  \n",
            "  inflating: government/documents/19/G00-19-2289281  \n",
            "  inflating: government/documents/19/G00-19-1043569  \n",
            "  inflating: government/documents/19/G00-19-2831556  \n",
            "  inflating: government/documents/19/G00-19-2572981  \n",
            "  inflating: government/documents/19/G00-19-3375396  \n",
            "  inflating: government/documents/19/G00-19-1878600  \n",
            "  inflating: government/documents/19/G00-19-1969434  \n",
            "  inflating: government/documents/19/G00-19-3592413  \n",
            "  inflating: government/documents/19/G00-19-1194370  \n",
            "  inflating: government/documents/19/G00-19-0753142  \n",
            "  inflating: government/documents/19/G00-19-2902623  \n",
            "  inflating: government/documents/19/G00-19-3360256  \n",
            "  inflating: government/documents/19/G00-19-4082665  \n",
            "  inflating: government/documents/19/G00-19-3641122  \n",
            "  inflating: government/documents/19/G00-19-2784490  \n",
            "  inflating: government/documents/19/G00-19-1464709  \n",
            "   creating: government/documents/26/\n",
            "  inflating: government/documents/26/G00-26-2454883  \n",
            "  inflating: government/documents/26/G00-26-3218156  \n",
            "  inflating: government/documents/26/G00-26-3904897  \n",
            "  inflating: government/documents/26/G00-26-1939983  \n",
            "  inflating: government/documents/26/G00-26-3181060  \n",
            "  inflating: government/documents/26/G00-26-0622450  \n",
            "  inflating: government/documents/26/G00-26-2881811  \n",
            "  inflating: government/documents/26/G00-26-1895117  \n",
            "  inflating: government/documents/26/G00-26-4176011  \n",
            "  inflating: government/documents/26/G00-26-0088644  \n",
            "  inflating: government/documents/26/G00-26-0885396  \n",
            "  inflating: government/documents/26/G00-26-2210420  \n",
            "  inflating: government/documents/26/G00-26-3007890  \n",
            "  inflating: government/documents/26/G00-26-0214876  \n",
            "  inflating: government/documents/26/G00-26-4012518  \n",
            "  inflating: government/documents/26/G00-26-3719300  \n",
            "  inflating: government/documents/26/G00-26-1925008  \n",
            "  inflating: government/documents/26/G00-26-1653948  \n",
            "  inflating: government/documents/26/G00-26-3134051  \n",
            "  inflating: government/documents/26/G00-26-1962814  \n",
            "  inflating: government/documents/26/G00-26-2247612  \n",
            "  inflating: government/documents/26/G00-26-0823698  \n",
            "  inflating: government/documents/26/G00-26-2502814  \n",
            "  inflating: government/documents/26/G00-26-1048210  \n",
            "  inflating: government/documents/26/G00-26-2224339  \n",
            "  inflating: government/documents/26/G00-26-0735531  \n",
            "  inflating: government/documents/26/G00-26-1904362  \n",
            "  inflating: government/documents/26/G00-26-2049181  \n",
            "  inflating: government/documents/26/G00-26-2650369  \n",
            "   creating: government/documents/21/\n",
            "  inflating: government/documents/21/G00-21-1192711  \n",
            "  inflating: government/documents/21/G00-21-0210232  \n",
            "  inflating: government/documents/21/G00-21-2001848  \n",
            "  inflating: government/documents/21/G00-21-0118799  \n",
            "  inflating: government/documents/21/G00-21-2737022  \n",
            "  inflating: government/documents/21/G00-21-0039963  \n",
            "  inflating: government/documents/21/G00-21-3009650  \n",
            "  inflating: government/documents/21/G00-21-1046543  \n",
            "  inflating: government/documents/21/G00-21-3464887  \n",
            "  inflating: government/documents/21/G00-21-2004003  \n",
            "  inflating: government/documents/21/G00-21-2018063  \n",
            "  inflating: government/documents/21/G00-21-1840014  \n",
            "  inflating: government/documents/21/G00-21-0886695  \n",
            "  inflating: government/documents/21/G00-21-1858074  \n",
            "  inflating: government/documents/21/G00-21-0308165  \n",
            "  inflating: government/documents/21/G00-21-2773039  \n",
            "  inflating: government/documents/21/G00-21-4119651  \n",
            "  inflating: government/documents/21/G00-21-0158978  \n",
            "  inflating: government/documents/21/G00-21-0073681  \n",
            "  inflating: government/documents/21/G00-21-0841529  \n",
            "  inflating: government/documents/21/G00-21-0494028  \n",
            "  inflating: government/documents/21/G00-21-1529615  \n",
            "  inflating: government/documents/21/G00-21-0582633  \n",
            "  inflating: government/documents/21/G00-21-1973048  \n",
            "  inflating: government/documents/21/G00-21-0649032  \n",
            "  inflating: government/documents/21/G00-21-3669563  \n",
            "  inflating: government/documents/21/G00-21-1589793  \n",
            "  inflating: government/documents/21/G00-21-3428605  \n",
            "  inflating: government/documents/21/G00-21-1714281  \n",
            "  inflating: government/documents/21/G00-21-2114990  \n",
            "  inflating: government/documents/21/G00-21-3411003  \n",
            "  inflating: government/documents/21/G00-21-1188169  \n",
            "  inflating: government/documents/21/G00-21-2000014  \n",
            "  inflating: government/documents/21/G00-21-2316505  \n",
            "  inflating: government/documents/21/G00-21-0501419  \n",
            "  inflating: government/documents/21/G00-21-2229498  \n",
            "  inflating: government/documents/21/G00-21-2495856  \n",
            "  inflating: government/documents/21/G00-21-4148394  \n",
            "  inflating: government/documents/21/G00-21-2194623  \n",
            "  inflating: government/documents/21/G00-21-2096695  \n",
            "  inflating: government/documents/21/G00-21-1073405  \n",
            "  inflating: government/documents/21/G00-21-4103271  \n",
            "  inflating: government/documents/21/G00-21-2949722  \n",
            "  inflating: government/documents/21/G00-21-3628344  \n",
            "  inflating: government/documents/21/G00-21-2223465  \n",
            "  inflating: government/documents/21/G00-21-0691653  \n",
            "  inflating: government/documents/21/G00-21-1913322  \n",
            "  inflating: government/documents/21/G00-21-3025953  \n",
            "  inflating: government/documents/21/G00-21-4161915  \n",
            "  inflating: government/documents/21/G00-21-4157028  \n",
            "   creating: government/documents/75/\n",
            "  inflating: government/documents/75/G00-75-1172730  \n",
            "  inflating: government/documents/75/G00-75-2761171  \n",
            "  inflating: government/documents/75/G00-75-3862870  \n",
            "  inflating: government/documents/75/G00-75-1015577  \n",
            "  inflating: government/documents/75/G00-75-0353792  \n",
            "  inflating: government/documents/75/G00-75-0360346  \n",
            "  inflating: government/documents/75/G00-75-2696512  \n",
            "  inflating: government/documents/75/G00-75-0507147  \n",
            "  inflating: government/documents/75/G00-75-2320438  \n",
            "  inflating: government/documents/75/G00-75-0755884  \n",
            "  inflating: government/documents/75/G00-75-2759811  \n",
            "  inflating: government/documents/75/G00-75-3017232  \n",
            "  inflating: government/documents/75/G00-75-3405135  \n",
            "  inflating: government/documents/75/G00-75-4083533  \n",
            "  inflating: government/documents/75/G00-75-3337203  \n",
            "  inflating: government/documents/75/G00-75-3188024  \n",
            "  inflating: government/documents/75/G00-75-1809770  \n",
            "  inflating: government/documents/75/G00-75-3333385  \n",
            "  inflating: government/documents/75/G00-75-2371200  \n",
            "  inflating: government/documents/75/G00-75-2649744  \n",
            "  inflating: government/documents/75/G00-75-2904152  \n",
            "  inflating: government/documents/75/G00-75-2391076  \n",
            "  inflating: government/documents/75/G00-75-3286457  \n",
            "  inflating: government/documents/75/G00-75-3743073  \n",
            "  inflating: government/documents/75/G00-75-3524215  \n",
            "  inflating: government/documents/75/G00-75-0577272  \n",
            "  inflating: government/documents/75/G00-75-2536079  \n",
            "  inflating: government/documents/75/G00-75-2944787  \n",
            "  inflating: government/documents/75/G00-75-1062837  \n",
            "  inflating: government/documents/75/G00-75-3633903  \n",
            "   creating: government/documents/81/\n",
            "  inflating: government/documents/81/G00-81-0534217  \n",
            "  inflating: government/documents/81/G00-81-1053799  \n",
            "  inflating: government/documents/81/G00-81-3163620  \n",
            "  inflating: government/documents/81/G00-81-4061846  \n",
            "  inflating: government/documents/81/G00-81-2370175  \n",
            "  inflating: government/documents/81/G00-81-3117219  \n",
            "  inflating: government/documents/81/G00-81-2576319  \n",
            "  inflating: government/documents/81/G00-81-3518475  \n",
            "  inflating: government/documents/81/G00-81-3327332  \n",
            "  inflating: government/documents/81/G00-81-0049650  \n",
            "  inflating: government/documents/81/G00-81-2418942  \n",
            "  inflating: government/documents/81/G00-81-0575496  \n",
            "  inflating: government/documents/81/G00-81-3308468  \n",
            "  inflating: government/documents/81/G00-81-1156977  \n",
            "  inflating: government/documents/81/G00-81-3230314  \n",
            "  inflating: government/documents/81/G00-81-3624185  \n",
            "  inflating: government/documents/81/G00-81-2715832  \n",
            "  inflating: government/documents/81/G00-81-3142599  \n",
            "  inflating: government/documents/81/G00-81-1989367  \n",
            "  inflating: government/documents/81/G00-81-2858567  \n",
            "  inflating: government/documents/81/G00-81-3820772  \n",
            "  inflating: government/documents/81/G00-81-2848764  \n",
            "  inflating: government/documents/81/G00-81-3225951  \n",
            "  inflating: government/documents/81/G00-81-3282800  \n",
            "  inflating: government/documents/81/G00-81-2669959  \n",
            "  inflating: government/documents/81/G00-81-3698412  \n",
            "  inflating: government/documents/81/G00-81-1858365  \n",
            "  inflating: government/documents/81/G00-81-2981070  \n",
            "   creating: government/documents/86/\n",
            "  inflating: government/documents/86/G00-86-0725465  \n",
            "  inflating: government/documents/86/G00-86-3794943  \n",
            "  inflating: government/documents/86/G00-86-0241003  \n",
            "  inflating: government/documents/86/G00-86-3900916  \n",
            "  inflating: government/documents/86/G00-86-1782415  \n",
            "  inflating: government/documents/86/G00-86-0073571  \n",
            "  inflating: government/documents/86/G00-86-3927645  \n",
            "  inflating: government/documents/86/G00-86-2719389  \n",
            "  inflating: government/documents/86/G00-86-0344148  \n",
            "  inflating: government/documents/86/G00-86-1693753  \n",
            "  inflating: government/documents/86/G00-86-3555591  \n",
            "  inflating: government/documents/86/G00-86-4022163  \n",
            "  inflating: government/documents/86/G00-86-0847220  \n",
            "  inflating: government/documents/86/G00-86-3182901  \n",
            "  inflating: government/documents/86/G00-86-3607394  \n",
            "  inflating: government/documents/86/G00-86-3730053  \n",
            "  inflating: government/documents/86/G00-86-2811015  \n",
            "  inflating: government/documents/86/G00-86-2289049  \n",
            "  inflating: government/documents/86/G00-86-0821288  \n",
            "  inflating: government/documents/86/G00-86-3137534  \n",
            "  inflating: government/documents/86/G00-86-3719816  \n",
            "  inflating: government/documents/86/G00-86-3503020  \n",
            "  inflating: government/documents/86/G00-86-3214229  \n",
            "  inflating: government/documents/86/G00-86-1671461  \n",
            "  inflating: government/documents/86/G00-86-0578862  \n",
            "  inflating: government/documents/86/G00-86-2059462  \n",
            "  inflating: government/documents/86/G00-86-0205886  \n",
            "  inflating: government/documents/86/G00-86-0751959  \n",
            "  inflating: government/documents/86/G00-86-2926152  \n",
            "  inflating: government/documents/86/G00-86-0913823  \n",
            "   creating: government/documents/72/\n",
            "  inflating: government/documents/72/G00-72-0516110  \n",
            "  inflating: government/documents/72/G00-72-3133704  \n",
            "  inflating: government/documents/72/G00-72-3509260  \n",
            "  inflating: government/documents/72/G00-72-1269594  \n",
            "  inflating: government/documents/72/G00-72-4180373  \n",
            "  inflating: government/documents/72/G00-72-2677026  \n",
            "  inflating: government/documents/72/G00-72-3566643  \n",
            "  inflating: government/documents/72/G00-72-1340883  \n",
            "  inflating: government/documents/72/G00-72-3535112  \n",
            "  inflating: government/documents/72/G00-72-2974059  \n",
            "  inflating: government/documents/72/G00-72-2592548  \n",
            "  inflating: government/documents/72/G00-72-3750946  \n",
            "  inflating: government/documents/72/G00-72-0527645  \n",
            "  inflating: government/documents/72/G00-72-3043721  \n",
            "  inflating: government/documents/72/G00-72-2900002  \n",
            "  inflating: government/documents/72/G00-72-3730240  \n",
            "  inflating: government/documents/72/G00-72-1851030  \n",
            "  inflating: government/documents/72/G00-72-1850510  \n",
            "  inflating: government/documents/72/G00-72-3332576  \n",
            "  inflating: government/documents/72/G00-72-0385489  \n",
            "  inflating: government/documents/72/G00-72-1267721  \n",
            "  inflating: government/documents/72/G00-72-1618219  \n",
            "  inflating: government/documents/72/G00-72-3522367  \n",
            "  inflating: government/documents/72/G00-72-2362421  \n",
            "  inflating: government/documents/72/G00-72-3760458  \n",
            "  inflating: government/documents/72/G00-72-2178498  \n",
            "  inflating: government/documents/72/G00-72-4030171  \n",
            "  inflating: government/documents/72/G00-72-3310984  \n",
            "  inflating: government/documents/72/G00-72-4059007  \n",
            "  inflating: government/documents/72/G00-72-1085257  \n",
            "   creating: government/documents/44/\n",
            "  inflating: government/documents/44/G00-44-3774931  \n",
            "  inflating: government/documents/44/G00-44-3191702  \n",
            "  inflating: government/documents/44/G00-44-3433686  \n",
            "  inflating: government/documents/44/G00-44-0092425  \n",
            "  inflating: government/documents/44/G00-44-1176683  \n",
            "  inflating: government/documents/44/G00-44-1932731  \n",
            "  inflating: government/documents/44/G00-44-0975793  \n",
            "  inflating: government/documents/44/G00-44-3240389  \n",
            "  inflating: government/documents/44/G00-44-2857194  \n",
            "  inflating: government/documents/44/G00-44-0995015  \n",
            "  inflating: government/documents/44/G00-44-1237736  \n",
            "  inflating: government/documents/44/G00-44-1772001  \n",
            "  inflating: government/documents/44/G00-44-0550737  \n",
            "  inflating: government/documents/44/G00-44-2379326  \n",
            "  inflating: government/documents/44/G00-44-3934324  \n",
            "  inflating: government/documents/44/G00-44-2135933  \n",
            "  inflating: government/documents/44/G00-44-1666740  \n",
            "  inflating: government/documents/44/G00-44-3924724  \n",
            "  inflating: government/documents/44/G00-44-0776281  \n",
            "  inflating: government/documents/44/G00-44-0827206  \n",
            "  inflating: government/documents/44/G00-44-1482914  \n",
            "  inflating: government/documents/44/G00-44-0040735  \n",
            "  inflating: government/documents/44/G00-44-2026102  \n",
            "  inflating: government/documents/44/G00-44-1544965  \n",
            "  inflating: government/documents/44/G00-44-1684353  \n",
            "  inflating: government/documents/44/G00-44-0764394  \n",
            "  inflating: government/documents/44/G00-44-2717261  \n",
            "  inflating: government/documents/44/G00-44-0841542  \n",
            "  inflating: government/documents/44/G00-44-0819855  \n",
            "  inflating: government/documents/44/G00-44-1554688  \n",
            "  inflating: government/documents/44/G00-44-2315826  \n",
            "  inflating: government/documents/44/G00-44-0168977  \n",
            "  inflating: government/documents/44/G00-44-1409165  \n",
            "   creating: government/documents/43/\n",
            "  inflating: government/documents/43/G00-43-3227387  \n",
            "  inflating: government/documents/43/G00-43-0106349  \n",
            "  inflating: government/documents/43/G00-43-2750291  \n",
            "  inflating: government/documents/43/G00-43-1143881  \n",
            "  inflating: government/documents/43/G00-43-2937459  \n",
            "  inflating: government/documents/43/G00-43-2860595  \n",
            "  inflating: government/documents/43/G00-43-2127379  \n",
            "  inflating: government/documents/43/G00-43-1822848  \n",
            "  inflating: government/documents/43/G00-43-3013510  \n",
            "  inflating: government/documents/43/G00-43-2871497  \n",
            "  inflating: government/documents/43/G00-43-3114822  \n",
            "  inflating: government/documents/43/G00-43-1817122  \n",
            "  inflating: government/documents/43/G00-43-2037829  \n",
            "  inflating: government/documents/43/G00-43-0782190  \n",
            "  inflating: government/documents/43/G00-43-0913072  \n",
            "  inflating: government/documents/43/G00-43-1931415  \n",
            "  inflating: government/documents/43/G00-43-0086742  \n",
            "  inflating: government/documents/43/G00-43-3414854  \n",
            "  inflating: government/documents/43/G00-43-2174906  \n",
            "  inflating: government/documents/43/G00-43-0410361  \n",
            "  inflating: government/documents/43/G00-43-3607125  \n",
            "  inflating: government/documents/43/G00-43-1317779  \n",
            "  inflating: government/documents/43/G00-43-1722468  \n",
            "  inflating: government/documents/43/G00-43-1985610  \n",
            "  inflating: government/documents/43/G00-43-0579156  \n",
            "  inflating: government/documents/43/G00-43-0903725  \n",
            "  inflating: government/documents/43/G00-43-4175875  \n",
            "  inflating: government/documents/43/G00-43-2308261  \n",
            "  inflating: government/documents/43/G00-43-1343266  \n",
            "  inflating: government/documents/43/G00-43-0195911  \n",
            "  inflating: government/documents/43/G00-43-0072878  \n",
            "  inflating: government/documents/43/G00-43-2549110  \n",
            "  inflating: government/documents/43/G00-43-2715210  \n",
            "  inflating: government/documents/43/G00-43-1458990  \n",
            "  inflating: government/documents/43/G00-43-3812747  \n",
            "  inflating: government/documents/43/G00-43-3062032  \n",
            "   creating: government/documents/88/\n",
            "  inflating: government/documents/88/G00-88-1447109  \n",
            "  inflating: government/documents/88/G00-88-2083848  \n",
            "  inflating: government/documents/88/G00-88-1080103  \n",
            "  inflating: government/documents/88/G00-88-1543350  \n",
            "  inflating: government/documents/88/G00-88-0128325  \n",
            "  inflating: government/documents/88/G00-88-2066673  \n",
            "  inflating: government/documents/88/G00-88-3039621  \n",
            "  inflating: government/documents/88/G00-88-2629440  \n",
            "  inflating: government/documents/88/G00-88-0648048  \n",
            "  inflating: government/documents/88/G00-88-2223375  \n",
            "  inflating: government/documents/88/G00-88-2072755  \n",
            "  inflating: government/documents/88/G00-88-2403767  \n",
            "  inflating: government/documents/88/G00-88-0775356  \n",
            "  inflating: government/documents/88/G00-88-2509522  \n",
            "  inflating: government/documents/88/G00-88-1214313  \n",
            "  inflating: government/documents/88/G00-88-1482208  \n",
            "  inflating: government/documents/88/G00-88-1028808  \n",
            "  inflating: government/documents/88/G00-88-0049153  \n",
            "  inflating: government/documents/88/G00-88-4059871  \n",
            "  inflating: government/documents/88/G00-88-2853984  \n",
            "  inflating: government/documents/88/G00-88-0162149  \n",
            "  inflating: government/documents/88/G00-88-3510277  \n",
            "  inflating: government/documents/88/G00-88-1594543  \n",
            "  inflating: government/documents/88/G00-88-3436258  \n",
            "  inflating: government/documents/88/G00-88-1403371  \n",
            "  inflating: government/documents/88/G00-88-2177598  \n",
            "  inflating: government/documents/88/G00-88-3380937  \n",
            "  inflating: government/documents/88/G00-88-2120454  \n",
            "  inflating: government/documents/88/G00-88-2830711  \n",
            "  inflating: government/documents/88/G00-88-1894712  \n",
            "   creating: government/documents/07/\n",
            "  inflating: government/documents/07/G00-07-3137365  \n",
            "  inflating: government/documents/07/G00-07-0600291  \n",
            "  inflating: government/documents/07/G00-07-1711598  \n",
            "  inflating: government/documents/07/G00-07-3712517  \n",
            "  inflating: government/documents/07/G00-07-3187807  \n",
            "  inflating: government/documents/07/G00-07-2819623  \n",
            "  inflating: government/documents/07/G00-07-0151026  \n",
            "  inflating: government/documents/07/G00-07-1018832  \n",
            "  inflating: government/documents/07/G00-07-3659195  \n",
            "  inflating: government/documents/07/G00-07-1172041  \n",
            "  inflating: government/documents/07/G00-07-1262808  \n",
            "  inflating: government/documents/07/G00-07-0355407  \n",
            "  inflating: government/documents/07/G00-07-2826375  \n",
            "  inflating: government/documents/07/G00-07-3227299  \n",
            "  inflating: government/documents/07/G00-07-0430412  \n",
            "  inflating: government/documents/07/G00-07-4009621  \n",
            "  inflating: government/documents/07/G00-07-3064254  \n",
            "  inflating: government/documents/07/G00-07-2244658  \n",
            "  inflating: government/documents/07/G00-07-2536891  \n",
            "  inflating: government/documents/07/G00-07-1561076  \n",
            "  inflating: government/documents/07/G00-07-1903067  \n",
            "  inflating: government/documents/07/G00-07-0443905  \n",
            "  inflating: government/documents/07/G00-07-3853050  \n",
            "  inflating: government/documents/07/G00-07-1577386  \n",
            "  inflating: government/documents/07/G00-07-0040146  \n",
            "  inflating: government/documents/07/G00-07-4080218  \n",
            "  inflating: government/documents/07/G00-07-0085827  \n",
            "  inflating: government/documents/07/G00-07-3745310  \n",
            "  inflating: government/documents/07/G00-07-3425686  \n",
            "  inflating: government/documents/07/G00-07-2687847  \n",
            "  inflating: government/documents/07/G00-07-3695881  \n",
            "  inflating: government/documents/07/G00-07-2371962  \n",
            "  inflating: government/documents/07/G00-07-0894074  \n",
            "  inflating: government/documents/07/G00-07-2710935  \n",
            "  inflating: government/documents/07/G00-07-1479427  \n",
            "  inflating: government/documents/07/G00-07-0886188  \n",
            "  inflating: government/documents/07/G00-07-3578277  \n",
            "  inflating: government/documents/07/G00-07-0302662  \n",
            "  inflating: government/documents/07/G00-07-3154026  \n",
            "  inflating: government/documents/07/G00-07-3283651  \n",
            "  inflating: government/documents/07/G00-07-1233793  \n",
            "  inflating: government/documents/07/G00-07-1116868  \n",
            "  inflating: government/documents/07/G00-07-3593654  \n",
            "  inflating: government/documents/07/G00-07-1993864  \n",
            "  inflating: government/documents/07/G00-07-2654723  \n",
            "  inflating: government/documents/07/G00-07-3252715  \n",
            "  inflating: government/documents/07/G00-07-1433889  \n",
            "  inflating: government/documents/07/G00-07-3859091  \n",
            "  inflating: government/documents/07/G00-07-1415326  \n",
            "  inflating: government/documents/07/G00-07-1556157  \n",
            "  inflating: government/documents/07/G00-07-0709451  \n",
            "  inflating: government/documents/07/G00-07-1097127  \n",
            "  inflating: government/documents/07/G00-07-3318478  \n",
            "  inflating: government/documents/07/G00-07-3237370  \n",
            "  inflating: government/documents/07/G00-07-4070041  \n",
            "  inflating: government/documents/07/G00-07-3972872  \n",
            "  inflating: government/documents/07/G00-07-2941106  \n",
            "  inflating: government/documents/07/G00-07-0297207  \n",
            "  inflating: government/documents/07/G00-07-0121280  \n",
            "  inflating: government/documents/07/G00-07-0475488  \n",
            "  inflating: government/documents/07/G00-07-0978415  \n",
            "  inflating: government/documents/07/G00-07-0261374  \n",
            "  inflating: government/documents/07/G00-07-1531381  \n",
            "  inflating: government/documents/07/G00-07-0484771  \n",
            "  inflating: government/documents/07/G00-07-0393523  \n",
            "  inflating: government/documents/07/G00-07-1735413  \n",
            "  inflating: government/documents/07/G00-07-1762179  \n",
            "  inflating: government/documents/07/G00-07-2561555  \n",
            "  inflating: government/documents/07/G00-07-4190786  \n",
            "  inflating: government/documents/07/G00-07-1680987  \n",
            "  inflating: government/documents/07/G00-07-3092558  \n",
            "  inflating: government/documents/07/G00-07-2785200  \n",
            "  inflating: government/documents/07/G00-07-0452218  \n",
            "  inflating: government/documents/07/G00-07-2059058  \n",
            "  inflating: government/documents/07/G00-07-1984572  \n",
            "  inflating: government/documents/07/G00-07-0975373  \n",
            "  inflating: government/documents/07/G00-07-1161610  \n",
            "  inflating: government/documents/07/G00-07-2767374  \n",
            "  inflating: government/documents/07/G00-07-3170301  \n",
            "  inflating: government/documents/07/G00-07-0004006  \n",
            "  inflating: government/documents/07/G00-07-0325733  \n",
            "  inflating: government/documents/07/G00-07-0000000  \n",
            "  inflating: government/documents/07/G00-07-0926644  \n",
            "  inflating: government/documents/07/G00-07-2159515  \n",
            "  inflating: government/documents/07/G00-07-1593969  \n",
            "  inflating: government/documents/07/G00-07-1025422  \n",
            "  inflating: government/documents/07/G00-07-2659757  \n",
            "  inflating: government/documents/07/G00-07-2291379  \n",
            "  inflating: government/documents/07/G00-07-2984484  \n",
            "  inflating: government/documents/07/G00-07-0733299  \n",
            "  inflating: government/documents/07/G00-07-2746381  \n",
            "  inflating: government/documents/07/G00-07-1052664  \n",
            "  inflating: government/documents/07/G00-07-0502700  \n",
            "  inflating: government/documents/07/G00-07-1453949  \n",
            "   creating: government/documents/38/\n",
            "  inflating: government/documents/38/G00-38-2341175  \n",
            "  inflating: government/documents/38/G00-38-1913766  \n",
            "  inflating: government/documents/38/G00-38-2268943  \n",
            "  inflating: government/documents/38/G00-38-1744932  \n",
            "  inflating: government/documents/38/G00-38-2908882  \n",
            "  inflating: government/documents/38/G00-38-0022868  \n",
            "  inflating: government/documents/38/G00-38-4007589  \n",
            "  inflating: government/documents/38/G00-38-2869018  \n",
            "  inflating: government/documents/38/G00-38-1602755  \n",
            "  inflating: government/documents/38/G00-38-2501293  \n",
            "  inflating: government/documents/38/G00-38-4074069  \n",
            "  inflating: government/documents/38/G00-38-1569212  \n",
            "  inflating: government/documents/38/G00-38-1883567  \n",
            "  inflating: government/documents/38/G00-38-0413107  \n",
            "  inflating: government/documents/38/G00-38-2378615  \n",
            "  inflating: government/documents/38/G00-38-2907822  \n",
            "  inflating: government/documents/38/G00-38-0854859  \n",
            "  inflating: government/documents/38/G00-38-0691497  \n",
            "  inflating: government/documents/38/G00-38-4001700  \n",
            "  inflating: government/documents/38/G00-38-4104339  \n",
            "  inflating: government/documents/38/G00-38-2776509  \n",
            "  inflating: government/documents/38/G00-38-2407266  \n",
            "  inflating: government/documents/38/G00-38-1736026  \n",
            "  inflating: government/documents/38/G00-38-0178600  \n",
            "  inflating: government/documents/38/G00-38-2515094  \n",
            "  inflating: government/documents/38/G00-38-0391658  \n",
            "  inflating: government/documents/38/G00-38-2705227  \n",
            "  inflating: government/documents/38/G00-38-0046564  \n",
            "  inflating: government/documents/38/G00-38-0601955  \n",
            "  inflating: government/documents/38/G00-38-2184733  \n",
            "  inflating: government/documents/38/G00-38-2335871  \n",
            "   creating: government/documents/00/\n",
            "  inflating: government/documents/00/G00-00-2056373  \n",
            "  inflating: government/documents/00/G00-00-2717332  \n",
            "  inflating: government/documents/00/G00-00-2550100  \n",
            "  inflating: government/documents/00/G00-00-3234392  \n",
            "  inflating: government/documents/00/G00-00-0114013  \n",
            "  inflating: government/documents/00/G00-00-3694638  \n",
            "  inflating: government/documents/00/G00-00-1636809  \n",
            "  inflating: government/documents/00/G00-00-0681214  \n",
            "  inflating: government/documents/00/G00-00-2121363  \n",
            "  inflating: government/documents/00/G00-00-0537233  \n",
            "  inflating: government/documents/00/G00-00-4000194  \n",
            "  inflating: government/documents/00/G00-00-2723824  \n",
            "  inflating: government/documents/00/G00-00-2416713  \n",
            "  inflating: government/documents/00/G00-00-2764912  \n",
            "  inflating: government/documents/00/G00-00-1958915  \n",
            "  inflating: government/documents/00/G00-00-2308409  \n",
            "  inflating: government/documents/00/G00-00-3874865  \n",
            "  inflating: government/documents/00/G00-00-3057225  \n",
            "  inflating: government/documents/00/G00-00-1711483  \n",
            "  inflating: government/documents/00/G00-00-3717374  \n",
            "  inflating: government/documents/00/G00-00-1961652  \n",
            "  inflating: government/documents/00/G00-00-2257471  \n",
            "  inflating: government/documents/00/G00-00-1496317  \n",
            "  inflating: government/documents/00/G00-00-1006224  \n",
            "  inflating: government/documents/00/G00-00-1432013  \n",
            "  inflating: government/documents/00/G00-00-3536797  \n",
            "  inflating: government/documents/00/G00-00-3784664  \n",
            "  inflating: government/documents/00/G00-00-1540114  \n",
            "  inflating: government/documents/00/G00-00-4009264  \n",
            "  inflating: government/documents/00/G00-00-3215146  \n",
            "  inflating: government/documents/00/G00-00-2884900  \n",
            "  inflating: government/documents/00/G00-00-2430093  \n",
            "  inflating: government/documents/00/G00-00-0386885  \n",
            "  inflating: government/documents/00/G00-00-0666975  \n",
            "  inflating: government/documents/00/G00-00-2461093  \n",
            "  inflating: government/documents/00/G00-00-1902126  \n",
            "  inflating: government/documents/00/G00-00-0158061  \n",
            "  inflating: government/documents/00/G00-00-3800163  \n",
            "  inflating: government/documents/00/G00-00-1525163  \n",
            "  inflating: government/documents/00/G00-00-0088569  \n",
            "  inflating: government/documents/00/G00-00-0307496  \n",
            "  inflating: government/documents/00/G00-00-2833797  \n",
            "  inflating: government/documents/00/G00-00-1842603  \n",
            "  inflating: government/documents/00/G00-00-3031135  \n",
            "  inflating: government/documents/00/G00-00-1497190  \n",
            "  inflating: government/documents/00/G00-00-2274733  \n",
            "  inflating: government/documents/00/G00-00-0409603  \n",
            "  inflating: government/documents/00/G00-00-1289792  \n",
            "  inflating: government/documents/00/G00-00-3056005  \n",
            "  inflating: government/documents/00/G00-00-0165832  \n",
            "  inflating: government/documents/00/G00-00-2154947  \n",
            "  inflating: government/documents/00/G00-00-0990763  \n",
            "  inflating: government/documents/00/G00-00-0350612  \n",
            "  inflating: government/documents/00/G00-00-2052165  \n",
            "  inflating: government/documents/00/G00-00-2694952  \n",
            "  inflating: government/documents/00/G00-00-3451251  \n",
            "  inflating: government/documents/00/G00-00-1569310  \n",
            "  inflating: government/documents/00/G00-00-1696968  \n",
            "  inflating: government/documents/00/G00-00-1277838  \n",
            "  inflating: government/documents/00/G00-00-2016453  \n",
            "  inflating: government/documents/00/G00-00-3667859  \n",
            "  inflating: government/documents/00/G00-00-1568254  \n",
            "  inflating: government/documents/00/G00-00-1996128  \n",
            "  inflating: government/documents/00/G00-00-2888052  \n",
            "  inflating: government/documents/00/G00-00-1829458  \n",
            "  inflating: government/documents/00/G00-00-1246456  \n",
            "  inflating: government/documents/00/G00-00-2491056  \n",
            "  inflating: government/documents/00/G00-00-3253540  \n",
            "  inflating: government/documents/00/G00-00-0809130  \n",
            "  inflating: government/documents/00/G00-00-0458107  \n",
            "  inflating: government/documents/00/G00-00-0297437  \n",
            "  inflating: government/documents/00/G00-00-0945765  \n",
            "  inflating: government/documents/00/G00-00-1101781  \n",
            "  inflating: government/documents/00/G00-00-1794090  \n",
            "  inflating: government/documents/00/G00-00-1733953  \n",
            "  inflating: government/documents/00/G00-00-3952056  \n",
            "  inflating: government/documents/00/G00-00-3473051  \n",
            "  inflating: government/documents/00/G00-00-1717907  \n",
            "  inflating: government/documents/00/G00-00-2791497  \n",
            "  inflating: government/documents/00/G00-00-0503555  \n",
            "  inflating: government/documents/00/G00-00-2534723  \n",
            "  inflating: government/documents/00/G00-00-2737496  \n",
            "  inflating: government/documents/00/G00-00-0502670  \n",
            "  inflating: government/documents/00/G00-00-1541287  \n",
            "  inflating: government/documents/00/G00-00-1591495  \n",
            "  inflating: government/documents/00/G00-00-1037714  \n",
            "  inflating: government/documents/00/G00-00-1210895  \n",
            "  inflating: government/documents/00/G00-00-0465038  \n",
            "  inflating: government/documents/00/G00-00-0124389  \n",
            "  inflating: government/documents/00/G00-00-3295822  \n",
            "  inflating: government/documents/00/G00-00-0663832  \n",
            "  inflating: government/documents/00/G00-00-3098344  \n",
            "  inflating: government/documents/00/G00-00-1474937  \n",
            "  inflating: government/documents/00/G00-00-2649810  \n",
            "  inflating: government/documents/00/G00-00-0799717  \n",
            "  inflating: government/documents/00/G00-00-2917282  \n",
            "  inflating: government/documents/00/G00-00-2682418  \n",
            "  inflating: government/documents/00/G00-00-3905014  \n",
            "  inflating: government/documents/00/G00-00-0803995  \n",
            "  inflating: government/documents/00/G00-00-2040865  \n",
            "  inflating: government/documents/00/G00-00-2515121  \n",
            "  inflating: government/documents/00/G00-00-1168484  \n",
            "  inflating: government/documents/00/G00-00-2548786  \n",
            "  inflating: government/documents/00/G00-00-2235134  \n",
            "  inflating: government/documents/00/G00-00-0575834  \n",
            "  inflating: government/documents/00/G00-00-2164772  \n",
            "  inflating: government/documents/00/G00-00-3566921  \n",
            "   creating: government/documents/36/\n",
            "  inflating: government/documents/36/G00-36-2127662  \n",
            "  inflating: government/documents/36/G00-36-0199656  \n",
            "  inflating: government/documents/36/G00-36-0132989  \n",
            "  inflating: government/documents/36/G00-36-1275993  \n",
            "  inflating: government/documents/36/G00-36-1960831  \n",
            "  inflating: government/documents/36/G00-36-0856406  \n",
            "  inflating: government/documents/36/G00-36-3425457  \n",
            "  inflating: government/documents/36/G00-36-3945230  \n",
            "  inflating: government/documents/36/G00-36-2916036  \n",
            "  inflating: government/documents/36/G00-36-3309025  \n",
            "  inflating: government/documents/36/G00-36-3470419  \n",
            "  inflating: government/documents/36/G00-36-0496639  \n",
            "  inflating: government/documents/36/G00-36-2458572  \n",
            "  inflating: government/documents/36/G00-36-1433110  \n",
            "  inflating: government/documents/36/G00-36-0850178  \n",
            "  inflating: government/documents/36/G00-36-2625159  \n",
            "  inflating: government/documents/36/G00-36-0221750  \n",
            "  inflating: government/documents/36/G00-36-2123062  \n",
            "  inflating: government/documents/36/G00-36-3047736  \n",
            "  inflating: government/documents/36/G00-36-3734429  \n",
            "  inflating: government/documents/36/G00-36-3746009  \n",
            "  inflating: government/documents/36/G00-36-2093615  \n",
            "  inflating: government/documents/36/G00-36-1135316  \n",
            "  inflating: government/documents/36/G00-36-2943291  \n",
            "  inflating: government/documents/36/G00-36-1620676  \n",
            "  inflating: government/documents/36/G00-36-3192459  \n",
            "  inflating: government/documents/36/G00-36-3147953  \n",
            "  inflating: government/documents/36/G00-36-1401603  \n",
            "  inflating: government/documents/36/G00-36-0017202  \n",
            "  inflating: government/documents/36/G00-36-3174389  \n",
            "  inflating: government/documents/36/G00-36-1040840  \n",
            "  inflating: government/documents/36/G00-36-4040609  \n",
            "  inflating: government/documents/36/G00-36-2788975  \n",
            "  inflating: government/documents/36/G00-36-2296936  \n",
            "  inflating: government/documents/36/G00-36-3599633  \n",
            "  inflating: government/documents/36/G00-36-1666289  \n",
            "  inflating: government/documents/36/G00-36-3549612  \n",
            "  inflating: government/documents/36/G00-36-3884264  \n",
            "  inflating: government/documents/36/G00-36-0475923  \n",
            "  inflating: government/documents/36/G00-36-0387611  \n",
            "  inflating: government/documents/36/G00-36-0882517  \n",
            "  inflating: government/documents/36/G00-36-2319087  \n",
            "   creating: government/documents/09/\n",
            "  inflating: government/documents/09/G00-09-0737106  \n",
            "  inflating: government/documents/09/G00-09-0783436  \n",
            "  inflating: government/documents/09/G00-09-4133514  \n",
            "  inflating: government/documents/09/G00-09-0532119  \n",
            "  inflating: government/documents/09/G00-09-3548030  \n",
            "  inflating: government/documents/09/G00-09-2627884  \n",
            "  inflating: government/documents/09/G00-09-1843567  \n",
            "  inflating: government/documents/09/G00-09-1427842  \n",
            "  inflating: government/documents/09/G00-09-0997903  \n",
            "  inflating: government/documents/09/G00-09-3984077  \n",
            "  inflating: government/documents/09/G00-09-0003342  \n",
            "  inflating: government/documents/09/G00-09-1246924  \n",
            "  inflating: government/documents/09/G00-09-3673255  \n",
            "  inflating: government/documents/09/G00-09-3662094  \n",
            "  inflating: government/documents/09/G00-09-1393028  \n",
            "  inflating: government/documents/09/G00-09-3225165  \n",
            "  inflating: government/documents/09/G00-09-0541164  \n",
            "  inflating: government/documents/09/G00-09-0071686  \n",
            "  inflating: government/documents/09/G00-09-0006344  \n",
            "  inflating: government/documents/09/G00-09-2272592  \n",
            "  inflating: government/documents/09/G00-09-2693851  \n",
            "  inflating: government/documents/09/G00-09-1011869  \n",
            "  inflating: government/documents/09/G00-09-4116125  \n",
            "  inflating: government/documents/09/G00-09-1434921  \n",
            "  inflating: government/documents/09/G00-09-0651144  \n",
            "  inflating: government/documents/09/G00-09-2790190  \n",
            "  inflating: government/documents/09/G00-09-3229156  \n",
            "  inflating: government/documents/09/G00-09-1560577  \n",
            "  inflating: government/documents/09/G00-09-3997123  \n",
            "  inflating: government/documents/09/G00-09-3375933  \n",
            "  inflating: government/documents/09/G00-09-3958168  \n",
            "  inflating: government/documents/09/G00-09-0487171  \n",
            "  inflating: government/documents/09/G00-09-1783499  \n",
            "  inflating: government/documents/09/G00-09-0660674  \n",
            "  inflating: government/documents/09/G00-09-1704111  \n",
            "  inflating: government/documents/09/G00-09-0808205  \n",
            "  inflating: government/documents/09/G00-09-2874929  \n",
            "  inflating: government/documents/09/G00-09-0749366  \n",
            "  inflating: government/documents/09/G00-09-3830284  \n",
            "  inflating: government/documents/09/G00-09-3262338  \n",
            "  inflating: government/documents/09/G00-09-2736495  \n",
            "  inflating: government/documents/09/G00-09-1022942  \n",
            "  inflating: government/documents/09/G00-09-2926931  \n",
            "  inflating: government/documents/09/G00-09-2977161  \n",
            "  inflating: government/documents/09/G00-09-1805506  \n",
            "  inflating: government/documents/09/G00-09-0536161  \n",
            "  inflating: government/documents/09/G00-09-2671845  \n",
            "  inflating: government/documents/09/G00-09-1846234  \n",
            "  inflating: government/documents/09/G00-09-0081961  \n",
            "  inflating: government/documents/09/G00-09-3637766  \n",
            "  inflating: government/documents/09/G00-09-3442668  \n",
            "  inflating: government/documents/09/G00-09-2147303  \n",
            "  inflating: government/documents/09/G00-09-2400329  \n",
            "  inflating: government/documents/09/G00-09-2332892  \n",
            "  inflating: government/documents/09/G00-09-3580452  \n",
            "  inflating: government/documents/09/G00-09-2336678  \n",
            "  inflating: government/documents/09/G00-09-1771088  \n",
            "  inflating: government/documents/09/G00-09-1165988  \n",
            "  inflating: government/documents/09/G00-09-3393396  \n",
            "  inflating: government/documents/09/G00-09-1833418  \n",
            "  inflating: government/documents/09/G00-09-1585165  \n",
            "  inflating: government/documents/09/G00-09-1177845  \n",
            "  inflating: government/documents/09/G00-09-0351581  \n",
            "  inflating: government/documents/09/G00-09-3243231  \n",
            "  inflating: government/documents/09/G00-09-0774298  \n",
            "  inflating: government/documents/09/G00-09-0245064  \n",
            "  inflating: government/documents/09/G00-09-0441906  \n",
            "  inflating: government/documents/09/G00-09-1193469  \n",
            "  inflating: government/documents/09/G00-09-3879836  \n",
            "  inflating: government/documents/09/G00-09-2426973  \n",
            "  inflating: government/documents/09/G00-09-2418472  \n",
            "  inflating: government/documents/09/G00-09-3881335  \n",
            "  inflating: government/documents/09/G00-09-2033802  \n",
            "  inflating: government/documents/09/G00-09-3931526  \n",
            "  inflating: government/documents/09/G00-09-4172401  \n",
            "  inflating: government/documents/09/G00-09-3538654  \n",
            "  inflating: government/documents/09/G00-09-3865948  \n",
            "  inflating: government/documents/09/G00-09-0395878  \n",
            "  inflating: government/documents/09/G00-09-3749355  \n",
            "  inflating: government/documents/09/G00-09-2011903  \n",
            "  inflating: government/documents/09/G00-09-1239961  \n",
            "  inflating: government/documents/09/G00-09-0186781  \n",
            "  inflating: government/documents/09/G00-09-0600961  \n",
            "  inflating: government/documents/09/G00-09-0408094  \n",
            "  inflating: government/documents/09/G00-09-1345111  \n",
            "  inflating: government/documents/09/G00-09-0888336  \n",
            "  inflating: government/documents/09/G00-09-2549321  \n",
            "  inflating: government/documents/09/G00-09-3069955  \n",
            "  inflating: government/documents/09/G00-09-0141522  \n",
            "  inflating: government/documents/09/G00-09-1404763  \n",
            "  inflating: government/documents/09/G00-09-3199095  \n",
            "  inflating: government/documents/09/G00-09-2781232  \n",
            "  inflating: government/documents/09/G00-09-1294904  \n",
            "  inflating: government/documents/09/G00-09-0826360  \n",
            "  inflating: government/documents/09/G00-09-1310323  \n",
            "  inflating: government/documents/09/G00-09-1753041  \n",
            "  inflating: government/documents/09/G00-09-0524984  \n",
            "  inflating: government/documents/09/G00-09-3470441  \n",
            "  inflating: government/documents/09/G00-09-2641304  \n",
            "  inflating: government/documents/09/G00-09-3657559  \n",
            "  inflating: government/documents/09/G00-09-2719242  \n",
            "  inflating: government/documents/09/G00-09-1222042  \n",
            "  inflating: government/documents/09/G00-09-0550713  \n",
            "  inflating: government/documents/09/G00-09-0472163  \n",
            "  inflating: government/documents/09/G00-09-1443772  \n",
            "  inflating: government/documents/09/G00-09-1670600  \n",
            "  inflating: government/documents/09/G00-09-0025460  \n",
            "  inflating: government/documents/09/G00-09-2725871  \n",
            "  inflating: government/documents/09/G00-09-3116633  \n",
            "  inflating: government/documents/09/G00-09-3195749  \n",
            "  inflating: government/documents/09/G00-09-2058065  \n",
            "  inflating: government/documents/09/G00-09-2834741  \n",
            "  inflating: government/documents/09/G00-09-2310962  \n",
            "  inflating: government/documents/09/G00-09-2605995  \n",
            "  inflating: government/documents/09/G00-09-2003930  \n",
            "   creating: government/documents/31/\n",
            "  inflating: government/documents/31/G00-31-3612634  \n",
            "  inflating: government/documents/31/G00-31-3415573  \n",
            "  inflating: government/documents/31/G00-31-3733602  \n",
            "  inflating: government/documents/31/G00-31-0429249  \n",
            "  inflating: government/documents/31/G00-31-0124417  \n",
            "  inflating: government/documents/31/G00-31-3509855  \n",
            "  inflating: government/documents/31/G00-31-0699333  \n",
            "  inflating: government/documents/31/G00-31-0221893  \n",
            "  inflating: government/documents/31/G00-31-1130222  \n",
            "  inflating: government/documents/31/G00-31-0298409  \n",
            "  inflating: government/documents/31/G00-31-3759587  \n",
            "  inflating: government/documents/31/G00-31-0402761  \n",
            "  inflating: government/documents/31/G00-31-1069354  \n",
            "  inflating: government/documents/31/G00-31-1308855  \n",
            "  inflating: government/documents/31/G00-31-4029518  \n",
            "  inflating: government/documents/31/G00-31-1216640  \n",
            "  inflating: government/documents/31/G00-31-4099841  \n",
            "  inflating: government/documents/31/G00-31-0926120  \n",
            "  inflating: government/documents/31/G00-31-0283929  \n",
            "  inflating: government/documents/31/G00-31-2378409  \n",
            "  inflating: government/documents/31/G00-31-2428996  \n",
            "  inflating: government/documents/31/G00-31-3599554  \n",
            "  inflating: government/documents/31/G00-31-2037123  \n",
            "  inflating: government/documents/31/G00-31-2454019  \n",
            "  inflating: government/documents/31/G00-31-1568234  \n",
            "  inflating: government/documents/31/G00-31-2127726  \n",
            "  inflating: government/documents/31/G00-31-2469242  \n",
            "  inflating: government/documents/31/G00-31-0445278  \n",
            "  inflating: government/documents/31/G00-31-1383043  \n",
            "  inflating: government/documents/31/G00-31-0295976  \n",
            "  inflating: government/documents/31/G00-31-1166839  \n",
            "  inflating: government/documents/31/G00-31-3447261  \n",
            "  inflating: government/documents/31/G00-31-1918557  \n",
            "  inflating: government/documents/31/G00-31-2583171  \n",
            "  inflating: government/documents/31/G00-31-0319287  \n",
            "  inflating: government/documents/31/G00-31-2417708  \n",
            "   creating: government/documents/91/\n",
            "  inflating: government/documents/91/G00-91-3031381  \n",
            "  inflating: government/documents/91/G00-91-3069407  \n",
            "  inflating: government/documents/91/G00-91-2870219  \n",
            "  inflating: government/documents/91/G00-91-3052709  \n",
            "  inflating: government/documents/91/G00-91-3432870  \n",
            "  inflating: government/documents/91/G00-91-3902992  \n",
            "  inflating: government/documents/91/G00-91-0485812  \n",
            "  inflating: government/documents/91/G00-91-0518238  \n",
            "  inflating: government/documents/91/G00-91-0134664  \n",
            "  inflating: government/documents/91/G00-91-0853670  \n",
            "  inflating: government/documents/91/G00-91-1239655  \n",
            "  inflating: government/documents/91/G00-91-4075015  \n",
            "  inflating: government/documents/91/G00-91-3065008  \n",
            "  inflating: government/documents/91/G00-91-3807220  \n",
            "  inflating: government/documents/91/G00-91-2843491  \n",
            "  inflating: government/documents/91/G00-91-3728081  \n",
            "  inflating: government/documents/91/G00-91-3056625  \n",
            "  inflating: government/documents/91/G00-91-3181951  \n",
            "  inflating: government/documents/91/G00-91-1127108  \n",
            "  inflating: government/documents/91/G00-91-1567424  \n",
            "  inflating: government/documents/91/G00-91-4153808  \n",
            "  inflating: government/documents/91/G00-91-2643625  \n",
            "  inflating: government/documents/91/G00-91-2032916  \n",
            "  inflating: government/documents/91/G00-91-3561460  \n",
            "  inflating: government/documents/91/G00-91-1900231  \n",
            "  inflating: government/documents/91/G00-91-1932391  \n",
            "  inflating: government/documents/91/G00-91-0820679  \n",
            "  inflating: government/documents/91/G00-91-3576409  \n",
            "  inflating: government/documents/91/G00-91-3997333  \n",
            "  inflating: government/documents/91/G00-91-1609512  \n",
            "  inflating: government/documents/91/G00-91-1069766  \n",
            "  inflating: government/documents/91/G00-91-2432483  \n",
            "  inflating: government/documents/91/G00-91-2992636  \n",
            "  inflating: government/documents/91/G00-91-1645119  \n",
            "   creating: government/documents/65/\n",
            "  inflating: government/documents/65/G00-65-0727765  \n",
            "  inflating: government/documents/65/G00-65-3200331  \n",
            "  inflating: government/documents/65/G00-65-1530657  \n",
            "  inflating: government/documents/65/G00-65-2360977  \n",
            "  inflating: government/documents/65/G00-65-0442094  \n",
            "  inflating: government/documents/65/G00-65-4020897  \n",
            "  inflating: government/documents/65/G00-65-4078383  \n",
            "  inflating: government/documents/65/G00-65-2516704  \n",
            "  inflating: government/documents/65/G00-65-3431980  \n",
            "  inflating: government/documents/65/G00-65-0410705  \n",
            "  inflating: government/documents/65/G00-65-0178367  \n",
            "  inflating: government/documents/65/G00-65-2859168  \n",
            "  inflating: government/documents/65/G00-65-0346398  \n",
            "  inflating: government/documents/65/G00-65-3635077  \n",
            "  inflating: government/documents/65/G00-65-3333979  \n",
            "  inflating: government/documents/65/G00-65-2598086  \n",
            "  inflating: government/documents/65/G00-65-3253157  \n",
            "  inflating: government/documents/65/G00-65-0000000  \n",
            "  inflating: government/documents/65/G00-65-4189024  \n",
            "  inflating: government/documents/65/G00-65-0162935  \n",
            "  inflating: government/documents/65/G00-65-3011991  \n",
            "  inflating: government/documents/65/G00-65-3240095  \n",
            "  inflating: government/documents/65/G00-65-1451613  \n",
            "  inflating: government/documents/65/G00-65-1737276  \n",
            "  inflating: government/documents/65/G00-65-0761364  \n",
            "   creating: government/documents/62/\n",
            "  inflating: government/documents/62/G00-62-4016584  \n",
            "  inflating: government/documents/62/G00-62-3810067  \n",
            "  inflating: government/documents/62/G00-62-2773089  \n",
            "  inflating: government/documents/62/G00-62-2477266  \n",
            "  inflating: government/documents/62/G00-62-0629836  \n",
            "  inflating: government/documents/62/G00-62-1249629  \n",
            "  inflating: government/documents/62/G00-62-2252378  \n",
            "  inflating: government/documents/62/G00-62-2030604  \n",
            "  inflating: government/documents/62/G00-62-0503229  \n",
            "  inflating: government/documents/62/G00-62-2530435  \n",
            "  inflating: government/documents/62/G00-62-0234683  \n",
            "  inflating: government/documents/62/G00-62-0337312  \n",
            "  inflating: government/documents/62/G00-62-2042932  \n",
            "  inflating: government/documents/62/G00-62-2515352  \n",
            "  inflating: government/documents/62/G00-62-1152151  \n",
            "  inflating: government/documents/62/G00-62-2271006  \n",
            "  inflating: government/documents/62/G00-62-1561542  \n",
            "  inflating: government/documents/62/G00-62-0415887  \n",
            "  inflating: government/documents/62/G00-62-0837921  \n",
            "  inflating: government/documents/62/G00-62-3597770  \n",
            "  inflating: government/documents/62/G00-62-3289850  \n",
            "  inflating: government/documents/62/G00-62-3385176  \n",
            "  inflating: government/documents/62/G00-62-2738054  \n",
            "  inflating: government/documents/62/G00-62-2501423  \n",
            "  inflating: government/documents/62/G00-62-0870952  \n",
            "  inflating: government/documents/62/G00-62-1736084  \n",
            "  inflating: government/documents/62/G00-62-3832697  \n",
            "  inflating: government/documents/62/G00-62-2366852  \n",
            "  inflating: government/documents/62/G00-62-1084142  \n",
            "  inflating: government/documents/62/G00-62-2860802  \n",
            "  inflating: government/documents/62/G00-62-3414482  \n",
            "  inflating: government/documents/62/G00-62-2770020  \n",
            "  inflating: government/documents/62/G00-62-3405159  \n",
            "   creating: government/documents/96/\n",
            "  inflating: government/documents/96/G00-96-1781066  \n",
            "  inflating: government/documents/96/G00-96-3080106  \n",
            "  inflating: government/documents/96/G00-96-1395116  \n",
            "  inflating: government/documents/96/G00-96-2780114  \n",
            "  inflating: government/documents/96/G00-96-2382474  \n",
            "  inflating: government/documents/96/G00-96-3597536  \n",
            "  inflating: government/documents/96/G00-96-3323352  \n",
            "  inflating: government/documents/96/G00-96-1415966  \n",
            "  inflating: government/documents/96/G00-96-3191929  \n",
            "  inflating: government/documents/96/G00-96-3984409  \n",
            "  inflating: government/documents/96/G00-96-3182721  \n",
            "  inflating: government/documents/96/G00-96-0203840  \n",
            "  inflating: government/documents/96/G00-96-0772936  \n",
            "  inflating: government/documents/96/G00-96-3317696  \n",
            "  inflating: government/documents/96/G00-96-1446108  \n",
            "  inflating: government/documents/96/G00-96-0307080  \n",
            "  inflating: government/documents/96/G00-96-2089068  \n",
            "  inflating: government/documents/96/G00-96-3359934  \n",
            "  inflating: government/documents/96/G00-96-2085859  \n",
            "  inflating: government/documents/96/G00-96-0218578  \n",
            "  inflating: government/documents/96/G00-96-2877140  \n",
            "  inflating: government/documents/96/G00-96-3079391  \n",
            "  inflating: government/documents/96/G00-96-0371133  \n",
            "   creating: government/documents/54/\n",
            "  inflating: government/documents/54/G00-54-1525808  \n",
            "  inflating: government/documents/54/G00-54-1217779  \n",
            "  inflating: government/documents/54/G00-54-3434200  \n",
            "  inflating: government/documents/54/G00-54-1409324  \n",
            "  inflating: government/documents/54/G00-54-2622017  \n",
            "  inflating: government/documents/54/G00-54-0007236  \n",
            "  inflating: government/documents/54/G00-54-0210787  \n",
            "  inflating: government/documents/54/G00-54-2576117  \n",
            "  inflating: government/documents/54/G00-54-3304332  \n",
            "  inflating: government/documents/54/G00-54-1196886  \n",
            "  inflating: government/documents/54/G00-54-2958371  \n",
            "  inflating: government/documents/54/G00-54-0690719  \n",
            "  inflating: government/documents/54/G00-54-0590753  \n",
            "  inflating: government/documents/54/G00-54-3319519  \n",
            "  inflating: government/documents/54/G00-54-3242132  \n",
            "  inflating: government/documents/54/G00-54-4103368  \n",
            "  inflating: government/documents/54/G00-54-3996340  \n",
            "  inflating: government/documents/54/G00-54-3445232  \n",
            "  inflating: government/documents/54/G00-54-3141493  \n",
            "  inflating: government/documents/54/G00-54-2352427  \n",
            "  inflating: government/documents/54/G00-54-1834977  \n",
            "  inflating: government/documents/54/G00-54-1823367  \n",
            "  inflating: government/documents/54/G00-54-0037064  \n",
            "  inflating: government/documents/54/G00-54-2231037  \n",
            "  inflating: government/documents/54/G00-54-0840997  \n",
            "  inflating: government/documents/54/G00-54-0905235  \n",
            "  inflating: government/documents/54/G00-54-3411561  \n",
            "  inflating: government/documents/54/G00-54-3644456  \n",
            "  inflating: government/documents/54/G00-54-0764224  \n",
            "   creating: government/documents/98/\n",
            "  inflating: government/documents/98/G00-98-2000334  \n",
            "  inflating: government/documents/98/G00-98-2611944  \n",
            "  inflating: government/documents/98/G00-98-1515436  \n",
            "  inflating: government/documents/98/G00-98-3741879  \n",
            "  inflating: government/documents/98/G00-98-3651358  \n",
            "  inflating: government/documents/98/G00-98-1047338  \n",
            "  inflating: government/documents/98/G00-98-4068688  \n",
            "  inflating: government/documents/98/G00-98-2339773  \n",
            "  inflating: government/documents/98/G00-98-2108936  \n",
            "  inflating: government/documents/98/G00-98-0827654  \n",
            "  inflating: government/documents/98/G00-98-2574886  \n",
            "  inflating: government/documents/98/G00-98-2120171  \n",
            "  inflating: government/documents/98/G00-98-2772007  \n",
            "  inflating: government/documents/98/G00-98-3517069  \n",
            "  inflating: government/documents/98/G00-98-0789839  \n",
            "  inflating: government/documents/98/G00-98-3268177  \n",
            "  inflating: government/documents/98/G00-98-2242918  \n",
            "  inflating: government/documents/98/G00-98-3717752  \n",
            "  inflating: government/documents/98/G00-98-0496126  \n",
            "  inflating: government/documents/98/G00-98-0272801  \n",
            "  inflating: government/documents/98/G00-98-2827164  \n",
            "  inflating: government/documents/98/G00-98-3312136  \n",
            "  inflating: government/documents/98/G00-98-1822743  \n",
            "  inflating: government/documents/98/G00-98-1302793  \n",
            "  inflating: government/documents/98/G00-98-1706780  \n",
            "  inflating: government/documents/98/G00-98-3795801  \n",
            "  inflating: government/documents/98/G00-98-2231918  \n",
            "  inflating: government/documents/98/G00-98-1785584  \n",
            "  inflating: government/documents/98/G00-98-3853828  \n",
            "  inflating: government/documents/98/G00-98-0052809  \n",
            "  inflating: government/documents/98/G00-98-1780609  \n",
            "  inflating: government/documents/98/G00-98-4172459  \n",
            "  inflating: government/documents/98/G00-98-1192401  \n",
            "  inflating: government/documents/98/G00-98-0261576  \n",
            "  inflating: government/documents/98/G00-98-3138271  \n",
            "  inflating: government/documents/98/G00-98-0000000  \n",
            "  inflating: government/documents/98/G00-98-3928796  \n",
            "  inflating: government/documents/98/G00-98-3866861  \n",
            "  inflating: government/documents/98/G00-98-1908158  \n",
            "  inflating: government/documents/98/G00-98-1861074  \n",
            "  inflating: government/documents/98/G00-98-0309243  \n",
            "   creating: government/documents/53/\n",
            "  inflating: government/documents/53/G00-53-2952068  \n",
            "  inflating: government/documents/53/G00-53-0850674  \n",
            "  inflating: government/documents/53/G00-53-2230464  \n",
            "  inflating: government/documents/53/G00-53-3329958  \n",
            "  inflating: government/documents/53/G00-53-1477767  \n",
            "  inflating: government/documents/53/G00-53-0998741  \n",
            "  inflating: government/documents/53/G00-53-2220063  \n",
            "  inflating: government/documents/53/G00-53-0436974  \n",
            "  inflating: government/documents/53/G00-53-0192769  \n",
            "  inflating: government/documents/53/G00-53-1122362  \n",
            "  inflating: government/documents/53/G00-53-3870218  \n",
            "  inflating: government/documents/53/G00-53-3569283  \n",
            "  inflating: government/documents/53/G00-53-0302989  \n",
            "  inflating: government/documents/53/G00-53-1129399  \n",
            "  inflating: government/documents/53/G00-53-0187653  \n",
            "  inflating: government/documents/53/G00-53-0277699  \n",
            "  inflating: government/documents/53/G00-53-4094998  \n",
            "  inflating: government/documents/53/G00-53-1684082  \n",
            "  inflating: government/documents/53/G00-53-2539578  \n",
            "  inflating: government/documents/53/G00-53-1996585  \n",
            "  inflating: government/documents/53/G00-53-3817888  \n",
            "  inflating: government/documents/53/G00-53-2171886  \n",
            "  inflating: government/documents/53/G00-53-0764096  \n",
            "  inflating: government/documents/53/G00-53-0263242  \n",
            "  inflating: government/documents/53/G00-53-1733327  \n",
            "  inflating: government/documents/53/G00-53-2925225  \n",
            "  inflating: government/documents/53/G00-53-3502518  \n",
            "  inflating: government/documents/53/G00-53-1466983  \n",
            "  inflating: government/documents/53/G00-53-1799996  \n",
            "  inflating: government/documents/53/G00-53-1764251  \n",
            "  inflating: government/documents/53/G00-53-0798521  \n",
            "  inflating: government/documents/53/G00-53-3937257  \n",
            "  inflating: government/documents/53/G00-53-0783449  \n",
            "  inflating: government/documents/53/G00-53-3228747  \n",
            "  inflating: government/documents/53/G00-53-0318533  \n",
            "  inflating: government/documents/53/G00-53-1998631  \n",
            "  inflating: government/documents/53/G00-53-4080590  \n",
            "   creating: government/documents/30/\n",
            "  inflating: government/documents/30/G00-30-1592409  \n",
            "  inflating: government/documents/30/G00-30-2122407  \n",
            "  inflating: government/documents/30/G00-30-0221651  \n",
            "  inflating: government/documents/30/G00-30-0401307  \n",
            "  inflating: government/documents/30/G00-30-0192480  \n",
            "  inflating: government/documents/30/G00-30-2788847  \n",
            "  inflating: government/documents/30/G00-30-1042227  \n",
            "  inflating: government/documents/30/G00-30-0665654  \n",
            "  inflating: government/documents/30/G00-30-0826855  \n",
            "  inflating: government/documents/30/G00-30-3435120  \n",
            "  inflating: government/documents/30/G00-30-3725345  \n",
            "  inflating: government/documents/30/G00-30-3863788  \n",
            "  inflating: government/documents/30/G00-30-1718009  \n",
            "  inflating: government/documents/30/G00-30-3936140  \n",
            "  inflating: government/documents/30/G00-30-0208335  \n",
            "  inflating: government/documents/30/G00-30-2977443  \n",
            "  inflating: government/documents/30/G00-30-2725539  \n",
            "  inflating: government/documents/30/G00-30-3726817  \n",
            "  inflating: government/documents/30/G00-30-1689251  \n",
            "  inflating: government/documents/30/G00-30-2065672  \n",
            "  inflating: government/documents/30/G00-30-0632487  \n",
            "  inflating: government/documents/30/G00-30-3794060  \n",
            "  inflating: government/documents/30/G00-30-0263169  \n",
            "  inflating: government/documents/30/G00-30-2216168  \n",
            "  inflating: government/documents/30/G00-30-1518511  \n",
            "  inflating: government/documents/30/G00-30-0064687  \n",
            "  inflating: government/documents/30/G00-30-3320486  \n",
            "  inflating: government/documents/30/G00-30-0025674  \n",
            "  inflating: government/documents/30/G00-30-0129773  \n",
            "  inflating: government/documents/30/G00-30-1702552  \n",
            "  inflating: government/documents/30/G00-30-0754079  \n",
            "  inflating: government/documents/30/G00-30-3783984  \n",
            "  inflating: government/documents/30/G00-30-1463695  \n",
            "  inflating: government/documents/30/G00-30-1962408  \n",
            "  inflating: government/documents/30/G00-30-1986213  \n",
            "  inflating: government/documents/30/G00-30-2283848  \n",
            "  inflating: government/documents/30/G00-30-1273791  \n",
            "  inflating: government/documents/30/G00-30-0937559  \n",
            "  inflating: government/documents/30/G00-30-0813675  \n",
            "   creating: government/documents/37/\n",
            "  inflating: government/documents/37/G00-37-0047781  \n",
            "  inflating: government/documents/37/G00-37-3695093  \n",
            "  inflating: government/documents/37/G00-37-2777794  \n",
            "  inflating: government/documents/37/G00-37-3787077  \n",
            "  inflating: government/documents/37/G00-37-4086752  \n",
            "  inflating: government/documents/37/G00-37-0563843  \n",
            "  inflating: government/documents/37/G00-37-1427392  \n",
            "  inflating: government/documents/37/G00-37-3530053  \n",
            "  inflating: government/documents/37/G00-37-3209394  \n",
            "  inflating: government/documents/37/G00-37-2483194  \n",
            "  inflating: government/documents/37/G00-37-0278690  \n",
            "  inflating: government/documents/37/G00-37-3798231  \n",
            "  inflating: government/documents/37/G00-37-3964850  \n",
            "  inflating: government/documents/37/G00-37-2508472  \n",
            "  inflating: government/documents/37/G00-37-0586634  \n",
            "  inflating: government/documents/37/G00-37-0776747  \n",
            "  inflating: government/documents/37/G00-37-3618349  \n",
            "  inflating: government/documents/37/G00-37-3803525  \n",
            "  inflating: government/documents/37/G00-37-0534979  \n",
            "  inflating: government/documents/37/G00-37-2426106  \n",
            "  inflating: government/documents/37/G00-37-1588543  \n",
            "  inflating: government/documents/37/G00-37-2381637  \n",
            "  inflating: government/documents/37/G00-37-2754165  \n",
            "   creating: government/documents/08/\n",
            "  inflating: government/documents/08/G00-08-3197187  \n",
            "  inflating: government/documents/08/G00-08-3586003  \n",
            "  inflating: government/documents/08/G00-08-0883392  \n",
            "  inflating: government/documents/08/G00-08-1594396  \n",
            "  inflating: government/documents/08/G00-08-0239407  \n",
            "  inflating: government/documents/08/G00-08-0317811  \n",
            "  inflating: government/documents/08/G00-08-1725210  \n",
            "  inflating: government/documents/08/G00-08-2650693  \n",
            "  inflating: government/documents/08/G00-08-3225599  \n",
            "  inflating: government/documents/08/G00-08-3850896  \n",
            "  inflating: government/documents/08/G00-08-1822422  \n",
            "  inflating: government/documents/08/G00-08-3950773  \n",
            "  inflating: government/documents/08/G00-08-3649734  \n",
            "  inflating: government/documents/08/G00-08-2098651  \n",
            "  inflating: government/documents/08/G00-08-3234501  \n",
            "  inflating: government/documents/08/G00-08-1394976  \n",
            "  inflating: government/documents/08/G00-08-0601924  \n",
            "  inflating: government/documents/08/G00-08-1915469  \n",
            "  inflating: government/documents/08/G00-08-3197667  \n",
            "  inflating: government/documents/08/G00-08-1084598  \n",
            "  inflating: government/documents/08/G00-08-0028902  \n",
            "  inflating: government/documents/08/G00-08-2096166  \n",
            "  inflating: government/documents/08/G00-08-3090509  \n",
            "  inflating: government/documents/08/G00-08-2497475  \n",
            "  inflating: government/documents/08/G00-08-3768107  \n",
            "  inflating: government/documents/08/G00-08-0371584  \n",
            "  inflating: government/documents/08/G00-08-3044569  \n",
            "  inflating: government/documents/08/G00-08-2645826  \n",
            "  inflating: government/documents/08/G00-08-3839955  \n",
            "  inflating: government/documents/08/G00-08-0893800  \n",
            "  inflating: government/documents/08/G00-08-3550429  \n",
            "  inflating: government/documents/08/G00-08-0052369  \n",
            "  inflating: government/documents/08/G00-08-1314254  \n",
            "  inflating: government/documents/08/G00-08-2571457  \n",
            "  inflating: government/documents/08/G00-08-1667883  \n",
            "  inflating: government/documents/08/G00-08-2772972  \n",
            "  inflating: government/documents/08/G00-08-0165895  \n",
            "  inflating: government/documents/08/G00-08-2570942  \n",
            "  inflating: government/documents/08/G00-08-2515170  \n",
            "  inflating: government/documents/08/G00-08-1048959  \n",
            "  inflating: government/documents/08/G00-08-3809731  \n",
            "  inflating: government/documents/08/G00-08-1000901  \n",
            "  inflating: government/documents/08/G00-08-1651053  \n",
            "  inflating: government/documents/08/G00-08-3257336  \n",
            "  inflating: government/documents/08/G00-08-1442145  \n",
            "  inflating: government/documents/08/G00-08-2429119  \n",
            "  inflating: government/documents/08/G00-08-3693971  \n",
            "  inflating: government/documents/08/G00-08-3572006  \n",
            "  inflating: government/documents/08/G00-08-2154490  \n",
            "  inflating: government/documents/08/G00-08-1398884  \n",
            "  inflating: government/documents/08/G00-08-3062942  \n",
            "  inflating: government/documents/08/G00-08-1232764  \n",
            "  inflating: government/documents/08/G00-08-0804085  \n",
            "  inflating: government/documents/08/G00-08-2869885  \n",
            "  inflating: government/documents/08/G00-08-1255046  \n",
            "  inflating: government/documents/08/G00-08-4015249  \n",
            "  inflating: government/documents/08/G00-08-1482914  \n",
            "  inflating: government/documents/08/G00-08-3388000  \n",
            "  inflating: government/documents/08/G00-08-3097262  \n",
            "  inflating: government/documents/08/G00-08-1077424  \n",
            "  inflating: government/documents/08/G00-08-1886521  \n",
            "  inflating: government/documents/08/G00-08-3685669  \n",
            "  inflating: government/documents/08/G00-08-0210913  \n",
            "  inflating: government/documents/08/G00-08-0995170  \n",
            "  inflating: government/documents/08/G00-08-2472523  \n",
            "  inflating: government/documents/08/G00-08-3824835  \n",
            "  inflating: government/documents/08/G00-08-3276373  \n",
            "  inflating: government/documents/08/G00-08-1409974  \n",
            "  inflating: government/documents/08/G00-08-2330518  \n",
            "  inflating: government/documents/08/G00-08-2577728  \n",
            "  inflating: government/documents/08/G00-08-2045138  \n",
            "  inflating: government/documents/08/G00-08-0264118  \n",
            "  inflating: government/documents/08/G00-08-1362469  \n",
            "  inflating: government/documents/08/G00-08-2963185  \n",
            "  inflating: government/documents/08/G00-08-2666680  \n",
            "  inflating: government/documents/08/G00-08-0716288  \n",
            "  inflating: government/documents/08/G00-08-2258958  \n",
            "  inflating: government/documents/08/G00-08-3780534  \n",
            "  inflating: government/documents/08/G00-08-2701029  \n",
            "  inflating: government/documents/08/G00-08-1793114  \n",
            "  inflating: government/documents/08/G00-08-0900666  \n",
            "  inflating: government/documents/08/G00-08-0417552  \n",
            "  inflating: government/documents/08/G00-08-0748245  \n",
            "  inflating: government/documents/08/G00-08-1007258  \n",
            "  inflating: government/documents/08/G00-08-0122936  \n",
            "  inflating: government/documents/08/G00-08-1609604  \n",
            "  inflating: government/documents/08/G00-08-1343599  \n",
            "  inflating: government/documents/08/G00-08-0808916  \n",
            "  inflating: government/documents/08/G00-08-0121513  \n",
            "  inflating: government/documents/08/G00-08-2342711  \n",
            "  inflating: government/documents/08/G00-08-1145623  \n",
            "  inflating: government/documents/08/G00-08-3960707  \n",
            "  inflating: government/documents/08/G00-08-1730514  \n",
            "  inflating: government/documents/08/G00-08-4001402  \n",
            "  inflating: government/documents/08/G00-08-4094617  \n",
            "  inflating: government/documents/08/G00-08-0085414  \n",
            "  inflating: government/documents/08/G00-08-3354269  \n",
            "  inflating: government/documents/08/G00-08-1337903  \n",
            "  inflating: government/documents/08/G00-08-3570730  \n",
            "  inflating: government/documents/08/G00-08-0825121  \n",
            "  inflating: government/documents/08/G00-08-2309305  \n",
            "  inflating: government/documents/08/G00-08-1370039  \n",
            "  inflating: government/documents/08/G00-08-0868177  \n",
            "  inflating: government/documents/08/G00-08-1473899  \n",
            "  inflating: government/documents/08/G00-08-2817882  \n",
            "  inflating: government/documents/08/G00-08-1618266  \n",
            "  inflating: government/documents/08/G00-08-0187263  \n",
            "  inflating: government/documents/08/G00-08-0164002  \n",
            "  inflating: government/documents/08/G00-08-3261667  \n",
            "   creating: government/documents/01/\n",
            "  inflating: government/documents/01/G00-01-4115952  \n",
            "  inflating: government/documents/01/G00-01-4030707  \n",
            "  inflating: government/documents/01/G00-01-1806077  \n",
            "  inflating: government/documents/01/G00-01-1596652  \n",
            "  inflating: government/documents/01/G00-01-0190637  \n",
            "  inflating: government/documents/01/G00-01-3237294  \n",
            "  inflating: government/documents/01/G00-01-1727987  \n",
            "  inflating: government/documents/01/G00-01-2389428  \n",
            "  inflating: government/documents/01/G00-01-1078697  \n",
            "  inflating: government/documents/01/G00-01-3035856  \n",
            "  inflating: government/documents/01/G00-01-1070603  \n",
            "  inflating: government/documents/01/G00-01-3160119  \n",
            "  inflating: government/documents/01/G00-01-3160857  \n",
            "  inflating: government/documents/01/G00-01-1548658  \n",
            "  inflating: government/documents/01/G00-01-0552893  \n",
            "  inflating: government/documents/01/G00-01-1555515  \n",
            "  inflating: government/documents/01/G00-01-2822805  \n",
            "  inflating: government/documents/01/G00-01-2973983  \n",
            "  inflating: government/documents/01/G00-01-0423383  \n",
            "  inflating: government/documents/01/G00-01-2561428  \n",
            "  inflating: government/documents/01/G00-01-1735338  \n",
            "  inflating: government/documents/01/G00-01-0488061  \n",
            "  inflating: government/documents/01/G00-01-3657801  \n",
            "  inflating: government/documents/01/G00-01-0996489  \n",
            "  inflating: government/documents/01/G00-01-3793940  \n",
            "  inflating: government/documents/01/G00-01-1939422  \n",
            "  inflating: government/documents/01/G00-01-1903746  \n",
            "  inflating: government/documents/01/G00-01-0726854  \n",
            "  inflating: government/documents/01/G00-01-2816216  \n",
            "  inflating: government/documents/01/G00-01-2206923  \n",
            "  inflating: government/documents/01/G00-01-0725221  \n",
            "  inflating: government/documents/01/G00-01-0658356  \n",
            "  inflating: government/documents/01/G00-01-0116531  \n",
            "  inflating: government/documents/01/G00-01-3251318  \n",
            "  inflating: government/documents/01/G00-01-2134408  \n",
            "  inflating: government/documents/01/G00-01-2800720  \n",
            "  inflating: government/documents/01/G00-01-1131482  \n",
            "  inflating: government/documents/01/G00-01-1814152  \n",
            "  inflating: government/documents/01/G00-01-1726882  \n",
            "  inflating: government/documents/01/G00-01-2796220  \n",
            "  inflating: government/documents/01/G00-01-1031819  \n",
            "  inflating: government/documents/01/G00-01-0631689  \n",
            "  inflating: government/documents/01/G00-01-0885829  \n",
            "  inflating: government/documents/01/G00-01-3749742  \n",
            "  inflating: government/documents/01/G00-01-0080587  \n",
            "  inflating: government/documents/01/G00-01-3217456  \n",
            "  inflating: government/documents/01/G00-01-0745853  \n",
            "  inflating: government/documents/01/G00-01-0270065  \n",
            "  inflating: government/documents/01/G00-01-3851180  \n",
            "  inflating: government/documents/01/G00-01-0325111  \n",
            "  inflating: government/documents/01/G00-01-1996694  \n",
            "  inflating: government/documents/01/G00-01-1855370  \n",
            "  inflating: government/documents/01/G00-01-0604374  \n",
            "  inflating: government/documents/01/G00-01-0682299  \n",
            "  inflating: government/documents/01/G00-01-0879152  \n",
            "  inflating: government/documents/01/G00-01-1619611  \n",
            "  inflating: government/documents/01/G00-01-2162850  \n",
            "  inflating: government/documents/01/G00-01-2689026  \n",
            "  inflating: government/documents/01/G00-01-2432252  \n",
            "  inflating: government/documents/01/G00-01-0320371  \n",
            "  inflating: government/documents/01/G00-01-2851595  \n",
            "  inflating: government/documents/01/G00-01-0409474  \n",
            "  inflating: government/documents/01/G00-01-0645068  \n",
            "  inflating: government/documents/01/G00-01-0797609  \n",
            "  inflating: government/documents/01/G00-01-0847467  \n",
            "  inflating: government/documents/01/G00-01-1404039  \n",
            "  inflating: government/documents/01/G00-01-3919230  \n",
            "  inflating: government/documents/01/G00-01-2172635  \n",
            "  inflating: government/documents/01/G00-01-2898660  \n",
            "  inflating: government/documents/01/G00-01-3742866  \n",
            "  inflating: government/documents/01/G00-01-1184872  \n",
            "  inflating: government/documents/01/G00-01-1928582  \n",
            "  inflating: government/documents/01/G00-01-1676158  \n",
            "  inflating: government/documents/01/G00-01-3645577  \n",
            "  inflating: government/documents/01/G00-01-3568472  \n",
            "  inflating: government/documents/01/G00-01-0784065  \n",
            "  inflating: government/documents/01/G00-01-1906654  \n",
            "  inflating: government/documents/01/G00-01-2776526  \n",
            "  inflating: government/documents/01/G00-01-2292288  \n",
            "  inflating: government/documents/01/G00-01-3672955  \n",
            "  inflating: government/documents/01/G00-01-3511414  \n",
            "  inflating: government/documents/01/G00-01-2038545  \n",
            "  inflating: government/documents/01/G00-01-1434889  \n",
            "  inflating: government/documents/01/G00-01-0261040  \n",
            "  inflating: government/documents/01/G00-01-3395222  \n",
            "  inflating: government/documents/01/G00-01-1171351  \n",
            "  inflating: government/documents/01/G00-01-0400712  \n",
            "  inflating: government/documents/01/G00-01-2154945  \n",
            "  inflating: government/documents/01/G00-01-3825845  \n",
            "  inflating: government/documents/01/G00-01-3224659  \n",
            "  inflating: government/documents/01/G00-01-2271700  \n",
            "  inflating: government/documents/01/G00-01-1504689  \n",
            "  inflating: government/documents/01/G00-01-3584374  \n",
            "  inflating: government/documents/01/G00-01-0420692  \n",
            "  inflating: government/documents/01/G00-01-3570364  \n",
            "  inflating: government/documents/01/G00-01-3999065  \n",
            "  inflating: government/documents/01/G00-01-2931694  \n",
            "  inflating: government/documents/01/G00-01-4041583  \n",
            "  inflating: government/documents/01/G00-01-0861661  \n",
            "  inflating: government/documents/01/G00-01-1579878  \n",
            "  inflating: government/documents/01/G00-01-1146668  \n",
            "  inflating: government/documents/01/G00-01-3377224  \n",
            "  inflating: government/documents/01/G00-01-1005021  \n",
            "  inflating: government/documents/01/G00-01-1207426  \n",
            "  inflating: government/documents/01/G00-01-0241150  \n",
            "  inflating: government/documents/01/G00-01-2064929  \n",
            "   creating: government/documents/06/\n",
            "  inflating: government/documents/06/G00-06-3906164  \n",
            "  inflating: government/documents/06/G00-06-1678194  \n",
            "  inflating: government/documents/06/G00-06-3460256  \n",
            "  inflating: government/documents/06/G00-06-1136931  \n",
            "  inflating: government/documents/06/G00-06-3762749  \n",
            "  inflating: government/documents/06/G00-06-3647346  \n",
            "  inflating: government/documents/06/G00-06-0690672  \n",
            "  inflating: government/documents/06/G00-06-1472959  \n",
            "  inflating: government/documents/06/G00-06-0975502  \n",
            "  inflating: government/documents/06/G00-06-1859727  \n",
            "  inflating: government/documents/06/G00-06-0872838  \n",
            "  inflating: government/documents/06/G00-06-1866615  \n",
            "  inflating: government/documents/06/G00-06-1975174  \n",
            "  inflating: government/documents/06/G00-06-2386747  \n",
            "  inflating: government/documents/06/G00-06-2032400  \n",
            "  inflating: government/documents/06/G00-06-3386226  \n",
            "  inflating: government/documents/06/G00-06-3652121  \n",
            "  inflating: government/documents/06/G00-06-3205886  \n",
            "  inflating: government/documents/06/G00-06-0818508  \n",
            "  inflating: government/documents/06/G00-06-1888561  \n",
            "  inflating: government/documents/06/G00-06-0397604  \n",
            "  inflating: government/documents/06/G00-06-2323986  \n",
            "  inflating: government/documents/06/G00-06-2496948  \n",
            "  inflating: government/documents/06/G00-06-2454895  \n",
            "  inflating: government/documents/06/G00-06-1939824  \n",
            "  inflating: government/documents/06/G00-06-2996037  \n",
            "  inflating: government/documents/06/G00-06-3519612  \n",
            "  inflating: government/documents/06/G00-06-2300997  \n",
            "  inflating: government/documents/06/G00-06-0014141  \n",
            "  inflating: government/documents/06/G00-06-2741751  \n",
            "  inflating: government/documents/06/G00-06-3861158  \n",
            "  inflating: government/documents/06/G00-06-0959549  \n",
            "  inflating: government/documents/06/G00-06-2519665  \n",
            "  inflating: government/documents/06/G00-06-0125838  \n",
            "  inflating: government/documents/06/G00-06-0310476  \n",
            "  inflating: government/documents/06/G00-06-4103448  \n",
            "  inflating: government/documents/06/G00-06-3270322  \n",
            "  inflating: government/documents/06/G00-06-2351076  \n",
            "  inflating: government/documents/06/G00-06-1755720  \n",
            "  inflating: government/documents/06/G00-06-1757034  \n",
            "  inflating: government/documents/06/G00-06-0790599  \n",
            "  inflating: government/documents/06/G00-06-0670692  \n",
            "  inflating: government/documents/06/G00-06-0808843  \n",
            "  inflating: government/documents/06/G00-06-0568210  \n",
            "  inflating: government/documents/06/G00-06-2661322  \n",
            "  inflating: government/documents/06/G00-06-0334457  \n",
            "  inflating: government/documents/06/G00-06-3817709  \n",
            "  inflating: government/documents/06/G00-06-1662495  \n",
            "  inflating: government/documents/06/G00-06-2290619  \n",
            "  inflating: government/documents/06/G00-06-3939376  \n",
            "  inflating: government/documents/06/G00-06-1168781  \n",
            "  inflating: government/documents/06/G00-06-0233752  \n",
            "  inflating: government/documents/06/G00-06-3965004  \n",
            "  inflating: government/documents/06/G00-06-1160574  \n",
            "  inflating: government/documents/06/G00-06-0364321  \n",
            "  inflating: government/documents/06/G00-06-1950576  \n",
            "  inflating: government/documents/06/G00-06-1039005  \n",
            "  inflating: government/documents/06/G00-06-3009863  \n",
            "  inflating: government/documents/06/G00-06-0689219  \n",
            "  inflating: government/documents/06/G00-06-3972161  \n",
            "  inflating: government/documents/06/G00-06-0369715  \n",
            "  inflating: government/documents/06/G00-06-0239914  \n",
            "  inflating: government/documents/06/G00-06-4191921  \n",
            "  inflating: government/documents/06/G00-06-2053433  \n",
            "  inflating: government/documents/06/G00-06-0769719  \n",
            "  inflating: government/documents/06/G00-06-2446510  \n",
            "  inflating: government/documents/06/G00-06-1727652  \n",
            "  inflating: government/documents/06/G00-06-2221728  \n",
            "  inflating: government/documents/06/G00-06-3152191  \n",
            "  inflating: government/documents/06/G00-06-1798790  \n",
            "  inflating: government/documents/06/G00-06-3115645  \n",
            "  inflating: government/documents/06/G00-06-0615550  \n",
            "  inflating: government/documents/06/G00-06-2965032  \n",
            "  inflating: government/documents/06/G00-06-0807908  \n",
            "  inflating: government/documents/06/G00-06-0418517  \n",
            "  inflating: government/documents/06/G00-06-2853218  \n",
            "  inflating: government/documents/06/G00-06-1598494  \n",
            "  inflating: government/documents/06/G00-06-3431951  \n",
            "  inflating: government/documents/06/G00-06-1913581  \n",
            "  inflating: government/documents/06/G00-06-2984337  \n",
            "  inflating: government/documents/06/G00-06-2330537  \n",
            "  inflating: government/documents/06/G00-06-2028177  \n",
            "  inflating: government/documents/06/G00-06-2884817  \n",
            "  inflating: government/documents/06/G00-06-0644931  \n",
            "  inflating: government/documents/06/G00-06-3089090  \n",
            "  inflating: government/documents/06/G00-06-2445973  \n",
            "  inflating: government/documents/06/G00-06-1617935  \n",
            "  inflating: government/documents/06/G00-06-1479477  \n",
            "  inflating: government/documents/06/G00-06-0990611  \n",
            "  inflating: government/documents/06/G00-06-2471691  \n",
            "  inflating: government/documents/06/G00-06-3054024  \n",
            "  inflating: government/documents/06/G00-06-0000000  \n",
            "  inflating: government/documents/06/G00-06-3512712  \n",
            "  inflating: government/documents/06/G00-06-2971917  \n",
            "  inflating: government/documents/06/G00-06-4038523  \n",
            "  inflating: government/documents/06/G00-06-2562505  \n",
            "  inflating: government/documents/06/G00-06-1310773  \n",
            "  inflating: government/documents/06/G00-06-1506948  \n",
            "  inflating: government/documents/06/G00-06-2929062  \n",
            "  inflating: government/documents/06/G00-06-1679297  \n",
            "  inflating: government/documents/06/G00-06-1661850  \n",
            "  inflating: government/documents/06/G00-06-4174747  \n",
            "   creating: government/documents/39/\n",
            "  inflating: government/documents/39/G00-39-2879677  \n",
            "  inflating: government/documents/39/G00-39-1567138  \n",
            "  inflating: government/documents/39/G00-39-4042977  \n",
            "  inflating: government/documents/39/G00-39-1768013  \n",
            "  inflating: government/documents/39/G00-39-0764093  \n",
            "  inflating: government/documents/39/G00-39-3855269  \n",
            "  inflating: government/documents/39/G00-39-2294981  \n",
            "  inflating: government/documents/39/G00-39-0841574  \n",
            "  inflating: government/documents/39/G00-39-3477058  \n",
            "  inflating: government/documents/39/G00-39-0860982  \n",
            "  inflating: government/documents/39/G00-39-3850837  \n",
            "  inflating: government/documents/39/G00-39-3954319  \n",
            "  inflating: government/documents/39/G00-39-1521404  \n",
            "  inflating: government/documents/39/G00-39-3687485  \n",
            "  inflating: government/documents/39/G00-39-4066050  \n",
            "  inflating: government/documents/39/G00-39-3348542  \n",
            "  inflating: government/documents/39/G00-39-0996057  \n",
            "  inflating: government/documents/39/G00-39-1676475  \n",
            "  inflating: government/documents/39/G00-39-0000000  \n",
            "  inflating: government/documents/39/G00-39-0104519  \n",
            "  inflating: government/documents/39/G00-39-3962892  \n",
            "  inflating: government/documents/39/G00-39-2581160  \n",
            "   creating: government/documents/99/\n",
            "  inflating: government/documents/99/G00-99-2951955  \n",
            "  inflating: government/documents/99/G00-99-2401551  \n",
            "  inflating: government/documents/99/G00-99-0804220  \n",
            "  inflating: government/documents/99/G00-99-3220476  \n",
            "  inflating: government/documents/99/G00-99-2447525  \n",
            "  inflating: government/documents/99/G00-99-1815453  \n",
            "  inflating: government/documents/99/G00-99-2721898  \n",
            "  inflating: government/documents/99/G00-99-2748693  \n",
            "  inflating: government/documents/99/G00-99-0399954  \n",
            "  inflating: government/documents/99/G00-99-1148872  \n",
            "  inflating: government/documents/99/G00-99-1946317  \n",
            "  inflating: government/documents/99/G00-99-2653385  \n",
            "  inflating: government/documents/99/G00-99-3597135  \n",
            "  inflating: government/documents/99/G00-99-3472326  \n",
            "  inflating: government/documents/99/G00-99-1825829  \n",
            "  inflating: government/documents/99/G00-99-2416431  \n",
            "  inflating: government/documents/99/G00-99-2186810  \n",
            "  inflating: government/documents/99/G00-99-3094164  \n",
            "  inflating: government/documents/99/G00-99-2247765  \n",
            "  inflating: government/documents/99/G00-99-3505460  \n",
            "  inflating: government/documents/99/G00-99-0456158  \n",
            "  inflating: government/documents/99/G00-99-0140748  \n",
            "  inflating: government/documents/99/G00-99-1417036  \n",
            "  inflating: government/documents/99/G00-99-2279811  \n",
            "  inflating: government/documents/99/G00-99-3432861  \n",
            "  inflating: government/documents/99/G00-99-0181704  \n",
            "  inflating: government/documents/99/G00-99-2536821  \n",
            "  inflating: government/documents/99/G00-99-3441506  \n",
            "  inflating: government/documents/99/G00-99-3624858  \n",
            "  inflating: government/documents/99/G00-99-2441663  \n",
            "   creating: government/documents/52/\n",
            "  inflating: government/documents/52/G00-52-1251046  \n",
            "  inflating: government/documents/52/G00-52-3419290  \n",
            "  inflating: government/documents/52/G00-52-1601190  \n",
            "  inflating: government/documents/52/G00-52-4025676  \n",
            "  inflating: government/documents/52/G00-52-0351180  \n",
            "  inflating: government/documents/52/G00-52-3178258  \n",
            "  inflating: government/documents/52/G00-52-3666949  \n",
            "  inflating: government/documents/52/G00-52-2638272  \n",
            "  inflating: government/documents/52/G00-52-1354748  \n",
            "  inflating: government/documents/52/G00-52-3462296  \n",
            "  inflating: government/documents/52/G00-52-2390317  \n",
            "  inflating: government/documents/52/G00-52-0410632  \n",
            "  inflating: government/documents/52/G00-52-1940908  \n",
            "  inflating: government/documents/52/G00-52-2256719  \n",
            "  inflating: government/documents/52/G00-52-2562368  \n",
            "  inflating: government/documents/52/G00-52-2522868  \n",
            "  inflating: government/documents/52/G00-52-4071167  \n",
            "  inflating: government/documents/52/G00-52-2991521  \n",
            "  inflating: government/documents/52/G00-52-2387745  \n",
            "  inflating: government/documents/52/G00-52-3560721  \n",
            "  inflating: government/documents/52/G00-52-0025559  \n",
            "  inflating: government/documents/52/G00-52-1091815  \n",
            "  inflating: government/documents/52/G00-52-3677624  \n",
            "  inflating: government/documents/52/G00-52-3395425  \n",
            "  inflating: government/documents/52/G00-52-1332764  \n",
            "  inflating: government/documents/52/G00-52-1105273  \n",
            "  inflating: government/documents/52/G00-52-1455795  \n",
            "  inflating: government/documents/52/G00-52-1224492  \n",
            "  inflating: government/documents/52/G00-52-3079304  \n",
            "   creating: government/documents/55/\n",
            "  inflating: government/documents/55/G00-55-3900830  \n",
            "  inflating: government/documents/55/G00-55-3690786  \n",
            "  inflating: government/documents/55/G00-55-3300136  \n",
            "  inflating: government/documents/55/G00-55-3346667  \n",
            "  inflating: government/documents/55/G00-55-0975670  \n",
            "  inflating: government/documents/55/G00-55-0304547  \n",
            "  inflating: government/documents/55/G00-55-1952930  \n",
            "  inflating: government/documents/55/G00-55-0771592  \n",
            "  inflating: government/documents/55/G00-55-2195428  \n",
            "  inflating: government/documents/55/G00-55-1994431  \n",
            "  inflating: government/documents/55/G00-55-0643570  \n",
            "  inflating: government/documents/55/G00-55-0000000  \n",
            "  inflating: government/documents/55/G00-55-1255864  \n",
            "  inflating: government/documents/55/G00-55-3670302  \n",
            "  inflating: government/documents/55/G00-55-0513571  \n",
            "  inflating: government/documents/55/G00-55-1380697  \n",
            "  inflating: government/documents/55/G00-55-0300099  \n",
            "  inflating: government/documents/55/G00-55-2853576  \n",
            "  inflating: government/documents/55/G00-55-0330602  \n",
            "  inflating: government/documents/55/G00-55-3773526  \n",
            "  inflating: government/documents/55/G00-55-1156700  \n",
            "  inflating: government/documents/55/G00-55-0221126  \n",
            "  inflating: government/documents/55/G00-55-1296936  \n",
            "  inflating: government/documents/55/G00-55-3685771  \n",
            "  inflating: government/documents/55/G00-55-0075732  \n",
            "  inflating: government/documents/55/G00-55-3547612  \n",
            "  inflating: government/documents/55/G00-55-3581191  \n",
            "  inflating: government/documents/55/G00-55-0185605  \n",
            "  inflating: government/documents/55/G00-55-3532648  \n",
            "  inflating: government/documents/55/G00-55-0789960  \n",
            "  inflating: government/documents/55/G00-55-2655624  \n",
            "  inflating: government/documents/55/G00-55-0989881  \n",
            "  inflating: government/documents/55/G00-55-0804983  \n",
            "  inflating: government/documents/55/G00-55-3817584  \n",
            "  inflating: government/documents/55/G00-55-0473793  \n",
            "   creating: government/documents/97/\n",
            "  inflating: government/documents/97/G00-97-2327881  \n",
            "  inflating: government/documents/97/G00-97-3940310  \n",
            "  inflating: government/documents/97/G00-97-2296786  \n",
            "  inflating: government/documents/97/G00-97-3639968  \n",
            "  inflating: government/documents/97/G00-97-1443049  \n",
            "  inflating: government/documents/97/G00-97-1817307  \n",
            "  inflating: government/documents/97/G00-97-3097941  \n",
            "  inflating: government/documents/97/G00-97-0304787  \n",
            "  inflating: government/documents/97/G00-97-2750090  \n",
            "  inflating: government/documents/97/G00-97-0293538  \n",
            "  inflating: government/documents/97/G00-97-4114543  \n",
            "  inflating: government/documents/97/G00-97-2471178  \n",
            "  inflating: government/documents/97/G00-97-1855812  \n",
            "  inflating: government/documents/97/G00-97-3162638  \n",
            "  inflating: government/documents/97/G00-97-3418169  \n",
            "  inflating: government/documents/97/G00-97-2616059  \n",
            "  inflating: government/documents/97/G00-97-2415717  \n",
            "  inflating: government/documents/97/G00-97-3608484  \n",
            "  inflating: government/documents/97/G00-97-2698410  \n",
            "  inflating: government/documents/97/G00-97-1687597  \n",
            "  inflating: government/documents/97/G00-97-2702150  \n",
            "  inflating: government/documents/97/G00-97-3422992  \n",
            "  inflating: government/documents/97/G00-97-2256677  \n",
            "  inflating: government/documents/97/G00-97-1730213  \n",
            "  inflating: government/documents/97/G00-97-3966956  \n",
            "  inflating: government/documents/97/G00-97-1475424  \n",
            "  inflating: government/documents/97/G00-97-3182735  \n",
            "  inflating: government/documents/97/G00-97-0745395  \n",
            "  inflating: government/documents/97/G00-97-3210379  \n",
            "  inflating: government/documents/97/G00-97-3618510  \n",
            "  inflating: government/documents/97/G00-97-2215955  \n",
            "  inflating: government/documents/97/G00-97-3200260  \n",
            "  inflating: government/documents/97/G00-97-3156431  \n",
            "  inflating: government/documents/97/G00-97-1988575  \n",
            "  inflating: government/documents/97/G00-97-2546590  \n",
            "  inflating: government/documents/97/G00-97-2162561  \n",
            "   creating: government/documents/63/\n",
            "  inflating: government/documents/63/G00-63-0953850  \n",
            "  inflating: government/documents/63/G00-63-3313516  \n",
            "  inflating: government/documents/63/G00-63-3880216  \n",
            "  inflating: government/documents/63/G00-63-3377143  \n",
            "  inflating: government/documents/63/G00-63-0465539  \n",
            "  inflating: government/documents/63/G00-63-2430531  \n",
            "  inflating: government/documents/63/G00-63-3813621  \n",
            "  inflating: government/documents/63/G00-63-0927287  \n",
            "  inflating: government/documents/63/G00-63-4005575  \n",
            "  inflating: government/documents/63/G00-63-0571652  \n",
            "  inflating: government/documents/63/G00-63-0482665  \n",
            "  inflating: government/documents/63/G00-63-0071667  \n",
            "  inflating: government/documents/63/G00-63-3855590  \n",
            "  inflating: government/documents/63/G00-63-2233776  \n",
            "  inflating: government/documents/63/G00-63-1218088  \n",
            "  inflating: government/documents/63/G00-63-0151995  \n",
            "  inflating: government/documents/63/G00-63-3154141  \n",
            "  inflating: government/documents/63/G00-63-1389256  \n",
            "  inflating: government/documents/63/G00-63-2487643  \n",
            "  inflating: government/documents/63/G00-63-0999632  \n",
            "  inflating: government/documents/63/G00-63-3049414  \n",
            "  inflating: government/documents/63/G00-63-1991583  \n",
            "  inflating: government/documents/63/G00-63-2350953  \n",
            "  inflating: government/documents/63/G00-63-2410805  \n",
            "  inflating: government/documents/63/G00-63-3708749  \n",
            "  inflating: government/documents/63/G00-63-3078014  \n",
            "  inflating: government/documents/63/G00-63-1804150  \n",
            "  inflating: government/documents/63/G00-63-1863069  \n",
            "   creating: government/documents/64/\n",
            "  inflating: government/documents/64/G00-64-2338458  \n",
            "  inflating: government/documents/64/G00-64-0258456  \n",
            "  inflating: government/documents/64/G00-64-0652907  \n",
            "  inflating: government/documents/64/G00-64-3191505  \n",
            "  inflating: government/documents/64/G00-64-3503951  \n",
            "  inflating: government/documents/64/G00-64-1194280  \n",
            "  inflating: government/documents/64/G00-64-2601514  \n",
            "  inflating: government/documents/64/G00-64-3476206  \n",
            "  inflating: government/documents/64/G00-64-1391646  \n",
            "  inflating: government/documents/64/G00-64-1861217  \n",
            "  inflating: government/documents/64/G00-64-2302639  \n",
            "  inflating: government/documents/64/G00-64-3545040  \n",
            "  inflating: government/documents/64/G00-64-1872775  \n",
            "  inflating: government/documents/64/G00-64-0195193  \n",
            "  inflating: government/documents/64/G00-64-3316475  \n",
            "  inflating: government/documents/64/G00-64-1938075  \n",
            "  inflating: government/documents/64/G00-64-4180535  \n",
            "  inflating: government/documents/64/G00-64-0367997  \n",
            "  inflating: government/documents/64/G00-64-3940663  \n",
            "  inflating: government/documents/64/G00-64-4055414  \n",
            "  inflating: government/documents/64/G00-64-2095540  \n",
            "  inflating: government/documents/64/G00-64-1871899  \n",
            "  inflating: government/documents/64/G00-64-3410298  \n",
            "  inflating: government/documents/64/G00-64-2137020  \n",
            "  inflating: government/documents/64/G00-64-2167825  \n",
            "  inflating: government/documents/64/G00-64-2759283  \n",
            "  inflating: government/documents/64/G00-64-3063239  \n",
            "   creating: government/documents/90/\n",
            "  inflating: government/documents/90/G00-90-0222848  \n",
            "  inflating: government/documents/90/G00-90-0648858  \n",
            "  inflating: government/documents/90/G00-90-0874739  \n",
            "  inflating: government/documents/90/G00-90-0144649  \n",
            "  inflating: government/documents/90/G00-90-2048674  \n",
            "  inflating: government/documents/90/G00-90-4194053  \n",
            "  inflating: government/documents/90/G00-90-2229321  \n",
            "  inflating: government/documents/90/G00-90-1438033  \n",
            "  inflating: government/documents/90/G00-90-4024223  \n",
            "  inflating: government/documents/90/G00-90-2241076  \n",
            "  inflating: government/documents/90/G00-90-3871013  \n",
            "  inflating: government/documents/90/G00-90-0342721  \n",
            "  inflating: government/documents/90/G00-90-0231194  \n",
            "  inflating: government/documents/90/G00-90-0833131  \n",
            "  inflating: government/documents/90/G00-90-3199989  \n",
            "  inflating: government/documents/90/G00-90-1799846  \n",
            "  inflating: government/documents/90/G00-90-1020923  \n",
            "  inflating: government/documents/90/G00-90-3847071  \n",
            "  inflating: government/documents/90/G00-90-2203817  \n",
            "  inflating: government/documents/90/G00-90-4179525  \n",
            "  inflating: government/documents/90/G00-90-3161999  \n",
            "  inflating: government/documents/90/G00-90-2621216  \n",
            "  inflating: government/documents/90/G00-90-3665860  \n",
            "  inflating: government/documents/90/G00-90-3801679  \n",
            "  inflating: government/documents/90/G00-90-3221944  \n",
            "  inflating: government/documents/90/G00-90-4073444  \n",
            "  inflating: government/documents/90/G00-90-3593960  \n",
            "  inflating: government/documents/90/G00-90-1818167  \n",
            "   creating: government/documents/46/\n",
            "  inflating: government/documents/46/G00-46-1439567  \n",
            "  inflating: government/documents/46/G00-46-2942390  \n",
            "  inflating: government/documents/46/G00-46-0268880  \n",
            "  inflating: government/documents/46/G00-46-0029122  \n",
            "  inflating: government/documents/46/G00-46-2682762  \n",
            "  inflating: government/documents/46/G00-46-3696083  \n",
            "  inflating: government/documents/46/G00-46-2945933  \n",
            "  inflating: government/documents/46/G00-46-0977266  \n",
            "  inflating: government/documents/46/G00-46-4090374  \n",
            "  inflating: government/documents/46/G00-46-1289711  \n",
            "  inflating: government/documents/46/G00-46-3161714  \n",
            "  inflating: government/documents/46/G00-46-1122260  \n",
            "  inflating: government/documents/46/G00-46-0825851  \n",
            "  inflating: government/documents/46/G00-46-3010333  \n",
            "  inflating: government/documents/46/G00-46-1985230  \n",
            "  inflating: government/documents/46/G00-46-3490898  \n",
            "  inflating: government/documents/46/G00-46-3411629  \n",
            "  inflating: government/documents/46/G00-46-3374400  \n",
            "  inflating: government/documents/46/G00-46-3492739  \n",
            "  inflating: government/documents/46/G00-46-1324941  \n",
            "  inflating: government/documents/46/G00-46-2760766  \n",
            "  inflating: government/documents/46/G00-46-2606261  \n",
            "  inflating: government/documents/46/G00-46-3319208  \n",
            "  inflating: government/documents/46/G00-46-1396158  \n",
            "  inflating: government/documents/46/G00-46-3258958  \n",
            "  inflating: government/documents/46/G00-46-1260939  \n",
            "  inflating: government/documents/46/G00-46-1883668  \n",
            "  inflating: government/documents/46/G00-46-0840102  \n",
            "  inflating: government/documents/46/G00-46-2926034  \n",
            "  inflating: government/documents/46/G00-46-2672308  \n",
            "  inflating: government/documents/46/G00-46-0122941  \n",
            "  inflating: government/documents/46/G00-46-2131394  \n",
            "   creating: government/documents/79/\n",
            "  inflating: government/documents/79/G00-79-3835198  \n",
            "  inflating: government/documents/79/G00-79-3824822  \n",
            "  inflating: government/documents/79/G00-79-0173131  \n",
            "  inflating: government/documents/79/G00-79-1442062  \n",
            "  inflating: government/documents/79/G00-79-2864917  \n",
            "  inflating: government/documents/79/G00-79-0590819  \n",
            "  inflating: government/documents/79/G00-79-2569923  \n",
            "  inflating: government/documents/79/G00-79-1032207  \n",
            "  inflating: government/documents/79/G00-79-2399698  \n",
            "  inflating: government/documents/79/G00-79-3109769  \n",
            "  inflating: government/documents/79/G00-79-2923133  \n",
            "  inflating: government/documents/79/G00-79-3007729  \n",
            "  inflating: government/documents/79/G00-79-2532479  \n",
            "  inflating: government/documents/79/G00-79-1457025  \n",
            "  inflating: government/documents/79/G00-79-2892445  \n",
            "  inflating: government/documents/79/G00-79-1489615  \n",
            "  inflating: government/documents/79/G00-79-0620805  \n",
            "  inflating: government/documents/79/G00-79-3679705  \n",
            "  inflating: government/documents/79/G00-79-2642089  \n",
            "  inflating: government/documents/79/G00-79-4144643  \n",
            "  inflating: government/documents/79/G00-79-3110767  \n",
            "  inflating: government/documents/79/G00-79-0102365  \n",
            "  inflating: government/documents/79/G00-79-3918530  \n",
            "  inflating: government/documents/79/G00-79-3852367  \n",
            "  inflating: government/documents/79/G00-79-2524737  \n",
            "  inflating: government/documents/79/G00-79-3570948  \n",
            "  inflating: government/documents/79/G00-79-0177455  \n",
            "  inflating: government/documents/79/G00-79-0790244  \n",
            "  inflating: government/documents/79/G00-79-1526684  \n",
            "  inflating: government/documents/79/G00-79-0406865  \n",
            "  inflating: government/documents/79/G00-79-1460340  \n",
            "  inflating: government/documents/79/G00-79-0218457  \n",
            "   creating: government/documents/41/\n",
            "  inflating: government/documents/41/G00-41-1695955  \n",
            "  inflating: government/documents/41/G00-41-2303639  \n",
            "  inflating: government/documents/41/G00-41-1145282  \n",
            "  inflating: government/documents/41/G00-41-1795167  \n",
            "  inflating: government/documents/41/G00-41-2311908  \n",
            "  inflating: government/documents/41/G00-41-1248910  \n",
            "  inflating: government/documents/41/G00-41-0964856  \n",
            "  inflating: government/documents/41/G00-41-2101298  \n",
            "  inflating: government/documents/41/G00-41-2344977  \n",
            "  inflating: government/documents/41/G00-41-1515340  \n",
            "  inflating: government/documents/41/G00-41-3719681  \n",
            "  inflating: government/documents/41/G00-41-2516655  \n",
            "  inflating: government/documents/41/G00-41-3022936  \n",
            "  inflating: government/documents/41/G00-41-4061473  \n",
            "  inflating: government/documents/41/G00-41-2480002  \n",
            "  inflating: government/documents/41/G00-41-1833498  \n",
            "  inflating: government/documents/41/G00-41-1343353  \n",
            "  inflating: government/documents/41/G00-41-3874597  \n",
            "  inflating: government/documents/41/G00-41-3659577  \n",
            "  inflating: government/documents/41/G00-41-0410179  \n",
            "  inflating: government/documents/41/G00-41-0541989  \n",
            "  inflating: government/documents/41/G00-41-3128445  \n",
            "  inflating: government/documents/41/G00-41-2962330  \n",
            "  inflating: government/documents/41/G00-41-3118613  \n",
            "  inflating: government/documents/41/G00-41-2190427  \n",
            "  inflating: government/documents/41/G00-41-1157769  \n",
            "  inflating: government/documents/41/G00-41-3817559  \n",
            "  inflating: government/documents/41/G00-41-3928454  \n",
            "  inflating: government/documents/41/G00-41-0604118  \n",
            "  inflating: government/documents/41/G00-41-0390790  \n",
            "  inflating: government/documents/41/G00-41-0098543  \n",
            "  inflating: government/documents/41/G00-41-1781943  \n",
            "  inflating: government/documents/41/G00-41-3469869  \n",
            "  inflating: government/documents/41/G00-41-3243233  \n",
            "  inflating: government/documents/41/G00-41-0928621  \n",
            "  inflating: government/documents/41/G00-41-3558692  \n",
            "   creating: government/documents/83/\n",
            "  inflating: government/documents/83/G00-83-3031351  \n",
            "  inflating: government/documents/83/G00-83-1610541  \n",
            "  inflating: government/documents/83/G00-83-2446108  \n",
            "  inflating: government/documents/83/G00-83-2979577  \n",
            "  inflating: government/documents/83/G00-83-1007925  \n",
            "  inflating: government/documents/83/G00-83-1599680  \n",
            "  inflating: government/documents/83/G00-83-3663520  \n",
            "  inflating: government/documents/83/G00-83-4148764  \n",
            "  inflating: government/documents/83/G00-83-3277638  \n",
            "  inflating: government/documents/83/G00-83-2711254  \n",
            "  inflating: government/documents/83/G00-83-1019314  \n",
            "  inflating: government/documents/83/G00-83-3087336  \n",
            "  inflating: government/documents/83/G00-83-0109387  \n",
            "  inflating: government/documents/83/G00-83-1056919  \n",
            "  inflating: government/documents/83/G00-83-3551240  \n",
            "  inflating: government/documents/83/G00-83-0087723  \n",
            "  inflating: government/documents/83/G00-83-2279979  \n",
            "  inflating: government/documents/83/G00-83-2023652  \n",
            "  inflating: government/documents/83/G00-83-3561112  \n",
            "  inflating: government/documents/83/G00-83-0694106  \n",
            "  inflating: government/documents/83/G00-83-1566739  \n",
            "  inflating: government/documents/83/G00-83-2143372  \n",
            "  inflating: government/documents/83/G00-83-1047969  \n",
            "   creating: government/documents/77/\n",
            "  inflating: government/documents/77/G00-77-3295130  \n",
            "  inflating: government/documents/77/G00-77-0359759  \n",
            "  inflating: government/documents/77/G00-77-2783812  \n",
            "  inflating: government/documents/77/G00-77-1684446  \n",
            "  inflating: government/documents/77/G00-77-1281904  \n",
            "  inflating: government/documents/77/G00-77-4073754  \n",
            "  inflating: government/documents/77/G00-77-0489289  \n",
            "  inflating: government/documents/77/G00-77-0645360  \n",
            "  inflating: government/documents/77/G00-77-3130212  \n",
            "  inflating: government/documents/77/G00-77-2301754  \n",
            "  inflating: government/documents/77/G00-77-3788231  \n",
            "  inflating: government/documents/77/G00-77-2698734  \n",
            "  inflating: government/documents/77/G00-77-3888391  \n",
            "  inflating: government/documents/77/G00-77-4152588  \n",
            "  inflating: government/documents/77/G00-77-1847967  \n",
            "  inflating: government/documents/77/G00-77-3589551  \n",
            "  inflating: government/documents/77/G00-77-1082226  \n",
            "  inflating: government/documents/77/G00-77-1867784  \n",
            "  inflating: government/documents/77/G00-77-1094989  \n",
            "  inflating: government/documents/77/G00-77-1693859  \n",
            "  inflating: government/documents/77/G00-77-1078597  \n",
            "  inflating: government/documents/77/G00-77-1505767  \n",
            "  inflating: government/documents/77/G00-77-0932280  \n",
            "  inflating: government/documents/77/G00-77-2977425  \n",
            "  inflating: government/documents/77/G00-77-1178393  \n",
            "  inflating: government/documents/77/G00-77-0448511  \n",
            "  inflating: government/documents/77/G00-77-2949998  \n",
            "  inflating: government/documents/77/G00-77-1808783  \n",
            "  inflating: government/documents/77/G00-77-1913440  \n",
            "  inflating: government/documents/77/G00-77-3991216  \n",
            "  inflating: government/documents/77/G00-77-4032514  \n",
            "  inflating: government/documents/77/G00-77-0452516  \n",
            "   creating: government/documents/48/\n",
            "  inflating: government/documents/48/G00-48-2560174  \n",
            "  inflating: government/documents/48/G00-48-0271104  \n",
            "  inflating: government/documents/48/G00-48-0930923  \n",
            "  inflating: government/documents/48/G00-48-1279631  \n",
            "  inflating: government/documents/48/G00-48-2454581  \n",
            "  inflating: government/documents/48/G00-48-0783760  \n",
            "  inflating: government/documents/48/G00-48-2907514  \n",
            "  inflating: government/documents/48/G00-48-2210520  \n",
            "  inflating: government/documents/48/G00-48-0966375  \n",
            "  inflating: government/documents/48/G00-48-2228619  \n",
            "  inflating: government/documents/48/G00-48-1527977  \n",
            "  inflating: government/documents/48/G00-48-2351509  \n",
            "  inflating: government/documents/48/G00-48-2303181  \n",
            "  inflating: government/documents/48/G00-48-1761099  \n",
            "  inflating: government/documents/48/G00-48-0413701  \n",
            "  inflating: government/documents/48/G00-48-2464830  \n",
            "  inflating: government/documents/48/G00-48-1306900  \n",
            "  inflating: government/documents/48/G00-48-1495892  \n",
            "  inflating: government/documents/48/G00-48-0374136  \n",
            "  inflating: government/documents/48/G00-48-3798484  \n",
            "  inflating: government/documents/48/G00-48-2924130  \n",
            "  inflating: government/documents/48/G00-48-2946724  \n",
            "  inflating: government/documents/48/G00-48-0640832  \n",
            "  inflating: government/documents/48/G00-48-1322117  \n",
            "  inflating: government/documents/48/G00-48-2460416  \n",
            "  inflating: government/documents/48/G00-48-0636130  \n",
            "  inflating: government/documents/48/G00-48-1176934  \n",
            "   creating: government/documents/70/\n",
            "  inflating: government/documents/70/G00-70-0567151  \n",
            "  inflating: government/documents/70/G00-70-1984499  \n",
            "  inflating: government/documents/70/G00-70-1022522  \n",
            "  inflating: government/documents/70/G00-70-3783746  \n",
            "  inflating: government/documents/70/G00-70-3678320  \n",
            "  inflating: government/documents/70/G00-70-3118212  \n",
            "  inflating: government/documents/70/G00-70-2361074  \n",
            "  inflating: government/documents/70/G00-70-3522936  \n",
            "  inflating: government/documents/70/G00-70-1221597  \n",
            "  inflating: government/documents/70/G00-70-2216707  \n",
            "  inflating: government/documents/70/G00-70-0991062  \n",
            "  inflating: government/documents/70/G00-70-3835972  \n",
            "  inflating: government/documents/70/G00-70-0142458  \n",
            "  inflating: government/documents/70/G00-70-2681284  \n",
            "  inflating: government/documents/70/G00-70-2867841  \n",
            "  inflating: government/documents/70/G00-70-2643119  \n",
            "  inflating: government/documents/70/G00-70-1025934  \n",
            "  inflating: government/documents/70/G00-70-1933374  \n",
            "  inflating: government/documents/70/G00-70-3751929  \n",
            "  inflating: government/documents/70/G00-70-1086552  \n",
            "  inflating: government/documents/70/G00-70-3859350  \n",
            "  inflating: government/documents/70/G00-70-0954490  \n",
            "  inflating: government/documents/70/G00-70-2163268  \n",
            "  inflating: government/documents/70/G00-70-3424520  \n",
            "  inflating: government/documents/70/G00-70-0775862  \n",
            "  inflating: government/documents/70/G00-70-2747752  \n",
            "  inflating: government/documents/70/G00-70-3168895  \n",
            "  inflating: government/documents/70/G00-70-2787853  \n",
            "  inflating: government/documents/70/G00-70-2194949  \n",
            "  inflating: government/documents/70/G00-70-0992046  \n",
            "  inflating: government/documents/70/G00-70-0878215  \n",
            "   creating: government/documents/84/\n",
            "  inflating: government/documents/84/G00-84-3054336  \n",
            "  inflating: government/documents/84/G00-84-1633108  \n",
            "  inflating: government/documents/84/G00-84-0557790  \n",
            "  inflating: government/documents/84/G00-84-3542587  \n",
            "  inflating: government/documents/84/G00-84-2647789  \n",
            "  inflating: government/documents/84/G00-84-3794357  \n",
            "  inflating: government/documents/84/G00-84-0410852  \n",
            "  inflating: government/documents/84/G00-84-0031129  \n",
            "  inflating: government/documents/84/G00-84-0274223  \n",
            "  inflating: government/documents/84/G00-84-0060779  \n",
            "  inflating: government/documents/84/G00-84-2792674  \n",
            "  inflating: government/documents/84/G00-84-3705449  \n",
            "  inflating: government/documents/84/G00-84-0610346  \n",
            "  inflating: government/documents/84/G00-84-3128630  \n",
            "  inflating: government/documents/84/G00-84-2576377  \n",
            "  inflating: government/documents/84/G00-84-2503293  \n",
            "  inflating: government/documents/84/G00-84-0144159  \n",
            "  inflating: government/documents/84/G00-84-2538245  \n",
            "  inflating: government/documents/84/G00-84-0120057  \n",
            "  inflating: government/documents/84/G00-84-3790991  \n",
            "  inflating: government/documents/84/G00-84-2308958  \n",
            "  inflating: government/documents/84/G00-84-1325129  \n",
            "  inflating: government/documents/84/G00-84-2217112  \n",
            "  inflating: government/documents/84/G00-84-0253801  \n",
            "  inflating: government/documents/84/G00-84-0259130  \n",
            "  inflating: government/documents/84/G00-84-2509432  \n",
            "  inflating: government/documents/84/G00-84-2738083  \n",
            "  inflating: government/documents/84/G00-84-3976222  \n",
            "  inflating: government/documents/84/G00-84-0937148  \n",
            "  inflating: government/documents/84/G00-84-2850617  \n",
            "  inflating: government/documents/84/G00-84-3349019  \n",
            "  inflating: government/documents/84/G00-84-2265835  \n",
            "  inflating: government/documents/84/G00-84-1089027  \n",
            "  inflating: government/documents/84/G00-84-0860875  \n",
            "   creating: government/documents/24/\n",
            "  inflating: government/documents/24/G00-24-0028328  \n",
            "  inflating: government/documents/24/G00-24-2808188  \n",
            "  inflating: government/documents/24/G00-24-1789618  \n",
            "  inflating: government/documents/24/G00-24-0610470  \n",
            "  inflating: government/documents/24/G00-24-0633652  \n",
            "  inflating: government/documents/24/G00-24-1638239  \n",
            "  inflating: government/documents/24/G00-24-1092308  \n",
            "  inflating: government/documents/24/G00-24-3340215  \n",
            "  inflating: government/documents/24/G00-24-1604470  \n",
            "  inflating: government/documents/24/G00-24-0537894  \n",
            "  inflating: government/documents/24/G00-24-2248175  \n",
            "  inflating: government/documents/24/G00-24-0519991  \n",
            "  inflating: government/documents/24/G00-24-3965592  \n",
            "  inflating: government/documents/24/G00-24-2277582  \n",
            "  inflating: government/documents/24/G00-24-4000056  \n",
            "  inflating: government/documents/24/G00-24-3625720  \n",
            "  inflating: government/documents/24/G00-24-3733379  \n",
            "  inflating: government/documents/24/G00-24-2914363  \n",
            "  inflating: government/documents/24/G00-24-0016657  \n",
            "  inflating: government/documents/24/G00-24-0940390  \n",
            "  inflating: government/documents/24/G00-24-0730942  \n",
            "  inflating: government/documents/24/G00-24-4085400  \n",
            "   creating: government/documents/23/\n",
            "  inflating: government/documents/23/G00-23-1251181  \n",
            "  inflating: government/documents/23/G00-23-0632783  \n",
            "  inflating: government/documents/23/G00-23-3059476  \n",
            "  inflating: government/documents/23/G00-23-3143323  \n",
            "  inflating: government/documents/23/G00-23-3399952  \n",
            "  inflating: government/documents/23/G00-23-2879237  \n",
            "  inflating: government/documents/23/G00-23-1595494  \n",
            "  inflating: government/documents/23/G00-23-3963973  \n",
            "  inflating: government/documents/23/G00-23-1800567  \n",
            "  inflating: government/documents/23/G00-23-1071960  \n",
            "  inflating: government/documents/23/G00-23-3149835  \n",
            "  inflating: government/documents/23/G00-23-2717675  \n",
            "  inflating: government/documents/23/G00-23-4089123  \n",
            "  inflating: government/documents/23/G00-23-3366177  \n",
            "  inflating: government/documents/23/G00-23-3822276  \n",
            "  inflating: government/documents/23/G00-23-3986715  \n",
            "  inflating: government/documents/23/G00-23-0825886  \n",
            "  inflating: government/documents/23/G00-23-0917457  \n",
            "  inflating: government/documents/23/G00-23-3770912  \n",
            "  inflating: government/documents/23/G00-23-3513349  \n",
            "  inflating: government/documents/23/G00-23-0817674  \n",
            "  inflating: government/documents/23/G00-23-0060528  \n",
            "  inflating: government/documents/23/G00-23-4092941  \n",
            "  inflating: government/documents/23/G00-23-2420837  \n",
            "  inflating: government/documents/23/G00-23-0678612  \n",
            "  inflating: government/documents/23/G00-23-2826118  \n",
            "  inflating: government/documents/23/G00-23-3098217  \n",
            "  inflating: government/documents/23/G00-23-1010771  \n",
            "  inflating: government/documents/23/G00-23-2860153  \n",
            "  inflating: government/documents/23/G00-23-0937782  \n",
            "  inflating: government/documents/23/G00-23-2403039  \n",
            "  inflating: government/documents/23/G00-23-3352202  \n",
            "  inflating: government/documents/23/G00-23-3119665  \n",
            "  inflating: government/documents/23/G00-23-1078147  \n",
            "  inflating: government/documents/23/G00-23-0479567  \n",
            "   creating: government/documents/15/\n",
            "  inflating: government/documents/15/G00-15-1586300  \n",
            "  inflating: government/documents/15/G00-15-1796298  \n",
            "  inflating: government/documents/15/G00-15-2160921  \n",
            "  inflating: government/documents/15/G00-15-1980890  \n",
            "  inflating: government/documents/15/G00-15-1040508  \n",
            "  inflating: government/documents/15/G00-15-3874791  \n",
            "  inflating: government/documents/15/G00-15-1769233  \n",
            "  inflating: government/documents/15/G00-15-2697443  \n",
            "  inflating: government/documents/15/G00-15-3425024  \n",
            "  inflating: government/documents/15/G00-15-2705772  \n",
            "  inflating: government/documents/15/G00-15-0389618  \n",
            "  inflating: government/documents/15/G00-15-4189524  \n",
            "  inflating: government/documents/15/G00-15-1584021  \n",
            "  inflating: government/documents/15/G00-15-4078353  \n",
            "  inflating: government/documents/15/G00-15-1144697  \n",
            "  inflating: government/documents/15/G00-15-2918418  \n",
            "  inflating: government/documents/15/G00-15-1718631  \n",
            "  inflating: government/documents/15/G00-15-3646536  \n",
            "  inflating: government/documents/15/G00-15-3106058  \n",
            "  inflating: government/documents/15/G00-15-1032639  \n",
            "  inflating: government/documents/15/G00-15-1460278  \n",
            "  inflating: government/documents/15/G00-15-3429810  \n",
            "  inflating: government/documents/15/G00-15-0501460  \n",
            "  inflating: government/documents/15/G00-15-2555140  \n",
            "  inflating: government/documents/15/G00-15-1296457  \n",
            "  inflating: government/documents/15/G00-15-3964548  \n",
            "  inflating: government/documents/15/G00-15-3549062  \n",
            "  inflating: government/documents/15/G00-15-2159501  \n",
            "  inflating: government/documents/15/G00-15-2334035  \n",
            "  inflating: government/documents/15/G00-15-3863880  \n",
            "  inflating: government/documents/15/G00-15-3557365  \n",
            "  inflating: government/documents/15/G00-15-0800500  \n",
            "  inflating: government/documents/15/G00-15-2101996  \n",
            "  inflating: government/documents/15/G00-15-3335359  \n",
            "  inflating: government/documents/15/G00-15-3695921  \n",
            "  inflating: government/documents/15/G00-15-4094582  \n",
            "   creating: government/documents/12/\n",
            "  inflating: government/documents/12/G00-12-3819195  \n",
            "  inflating: government/documents/12/G00-12-3452859  \n",
            "  inflating: government/documents/12/G00-12-4053815  \n",
            "  inflating: government/documents/12/G00-12-1397051  \n",
            "  inflating: government/documents/12/G00-12-0331334  \n",
            "  inflating: government/documents/12/G00-12-2272119  \n",
            "  inflating: government/documents/12/G00-12-3511705  \n",
            "  inflating: government/documents/12/G00-12-0039547  \n",
            "  inflating: government/documents/12/G00-12-1537007  \n",
            "  inflating: government/documents/12/G00-12-0801786  \n",
            "  inflating: government/documents/12/G00-12-3438391  \n",
            "  inflating: government/documents/12/G00-12-0015737  \n",
            "  inflating: government/documents/12/G00-12-1926166  \n",
            "  inflating: government/documents/12/G00-12-3249880  \n",
            "  inflating: government/documents/12/G00-12-1154511  \n",
            "  inflating: government/documents/12/G00-12-1043081  \n",
            "  inflating: government/documents/12/G00-12-3806551  \n",
            "  inflating: government/documents/12/G00-12-2788084  \n",
            "  inflating: government/documents/12/G00-12-2689046  \n",
            "  inflating: government/documents/12/G00-12-3727083  \n",
            "  inflating: government/documents/12/G00-12-3365465  \n",
            "  inflating: government/documents/12/G00-12-1245383  \n",
            "  inflating: government/documents/12/G00-12-2310354  \n",
            "  inflating: government/documents/12/G00-12-2680588  \n",
            "  inflating: government/documents/12/G00-12-1259911  \n",
            "  inflating: government/documents/12/G00-12-1837057  \n",
            "  inflating: government/documents/12/G00-12-2075864  \n",
            "  inflating: government/documents/12/G00-12-1418162  \n",
            "  inflating: government/documents/12/G00-12-3729065  \n",
            "  inflating: government/documents/12/G00-12-1714147  \n",
            "  inflating: government/documents/12/G00-12-2092230  \n",
            "  inflating: government/documents/12/G00-12-1139144  \n",
            "  inflating: government/documents/12/G00-12-2704265  \n",
            "  inflating: government/documents/12/G00-12-0024883  \n",
            "  inflating: government/documents/12/G00-12-0572170  \n",
            "  inflating: government/documents/12/G00-12-0711547  \n",
            "  inflating: government/documents/12/G00-12-1131975  \n",
            "  inflating: government/documents/12/G00-12-1779763  \n",
            "  inflating: government/documents/12/G00-12-0399012  \n",
            "  inflating: government/documents/12/G00-12-2357538  \n",
            "  inflating: government/documents/12/G00-12-1831266  \n",
            "  inflating: government/documents/12/G00-12-3326183  \n",
            "  inflating: government/documents/12/G00-12-3766260  \n",
            "  inflating: government/documents/12/G00-12-1790785  \n",
            "  inflating: government/documents/12/G00-12-0326213  \n",
            "  inflating: government/documents/12/G00-12-1746233  \n",
            "   creating: government/documents/85/\n",
            "  inflating: government/documents/85/G00-85-3944555  \n",
            "  inflating: government/documents/85/G00-85-3966586  \n",
            "  inflating: government/documents/85/G00-85-2970310  \n",
            "  inflating: government/documents/85/G00-85-0894040  \n",
            "  inflating: government/documents/85/G00-85-2822054  \n",
            "  inflating: government/documents/85/G00-85-3065292  \n",
            "  inflating: government/documents/85/G00-85-3240813  \n",
            "  inflating: government/documents/85/G00-85-0776822  \n",
            "  inflating: government/documents/85/G00-85-0364728  \n",
            "  inflating: government/documents/85/G00-85-2784544  \n",
            "  inflating: government/documents/85/G00-85-3077778  \n",
            "  inflating: government/documents/85/G00-85-0552445  \n",
            "  inflating: government/documents/85/G00-85-2732861  \n",
            "  inflating: government/documents/85/G00-85-1525415  \n",
            "  inflating: government/documents/85/G00-85-0189213  \n",
            "  inflating: government/documents/85/G00-85-0542492  \n",
            "  inflating: government/documents/85/G00-85-3812751  \n",
            "  inflating: government/documents/85/G00-85-4030813  \n",
            "  inflating: government/documents/85/G00-85-1063363  \n",
            "  inflating: government/documents/85/G00-85-3608680  \n",
            "  inflating: government/documents/85/G00-85-0221097  \n",
            "  inflating: government/documents/85/G00-85-4173589  \n",
            "  inflating: government/documents/85/G00-85-3490684  \n",
            "  inflating: government/documents/85/G00-85-2805930  \n",
            "  inflating: government/documents/85/G00-85-2193284  \n",
            "   creating: government/documents/71/\n",
            "  inflating: government/documents/71/G00-71-4081967  \n",
            "  inflating: government/documents/71/G00-71-0886607  \n",
            "  inflating: government/documents/71/G00-71-0183650  \n",
            "  inflating: government/documents/71/G00-71-1370524  \n",
            "  inflating: government/documents/71/G00-71-2116591  \n",
            "  inflating: government/documents/71/G00-71-3227595  \n",
            "  inflating: government/documents/71/G00-71-2869595  \n",
            "  inflating: government/documents/71/G00-71-2435350  \n",
            "  inflating: government/documents/71/G00-71-2077906  \n",
            "  inflating: government/documents/71/G00-71-0161321  \n",
            "  inflating: government/documents/71/G00-71-1781567  \n",
            "  inflating: government/documents/71/G00-71-3454228  \n",
            "  inflating: government/documents/71/G00-71-2803474  \n",
            "  inflating: government/documents/71/G00-71-3304650  \n",
            "  inflating: government/documents/71/G00-71-2027559  \n",
            "  inflating: government/documents/71/G00-71-1768133  \n",
            "  inflating: government/documents/71/G00-71-0907060  \n",
            "  inflating: government/documents/71/G00-71-2522032  \n",
            "  inflating: government/documents/71/G00-71-3257178  \n",
            "  inflating: government/documents/71/G00-71-2511610  \n",
            "  inflating: government/documents/71/G00-71-0999126  \n",
            "  inflating: government/documents/71/G00-71-2512854  \n",
            "   creating: government/documents/76/\n",
            "  inflating: government/documents/76/G00-76-1093417  \n",
            "  inflating: government/documents/76/G00-76-1350144  \n",
            "  inflating: government/documents/76/G00-76-0232183  \n",
            "  inflating: government/documents/76/G00-76-4136817  \n",
            "  inflating: government/documents/76/G00-76-1965909  \n",
            "  inflating: government/documents/76/G00-76-1170213  \n",
            "  inflating: government/documents/76/G00-76-2297937  \n",
            "  inflating: government/documents/76/G00-76-2205746  \n",
            "  inflating: government/documents/76/G00-76-1390720  \n",
            "  inflating: government/documents/76/G00-76-4172346  \n",
            "  inflating: government/documents/76/G00-76-0659802  \n",
            "  inflating: government/documents/76/G00-76-2498376  \n",
            "  inflating: government/documents/76/G00-76-3461519  \n",
            "  inflating: government/documents/76/G00-76-2513112  \n",
            "  inflating: government/documents/76/G00-76-1290170  \n",
            "  inflating: government/documents/76/G00-76-3273936  \n",
            "  inflating: government/documents/76/G00-76-2841899  \n",
            "  inflating: government/documents/76/G00-76-1292662  \n",
            "  inflating: government/documents/76/G00-76-2323292  \n",
            "  inflating: government/documents/76/G00-76-0311333  \n",
            "  inflating: government/documents/76/G00-76-2535403  \n",
            "  inflating: government/documents/76/G00-76-3635292  \n",
            "  inflating: government/documents/76/G00-76-2126358  \n",
            "  inflating: government/documents/76/G00-76-4175746  \n",
            "  inflating: government/documents/76/G00-76-1074169  \n",
            "  inflating: government/documents/76/G00-76-2978158  \n",
            "  inflating: government/documents/76/G00-76-0415824  \n",
            "  inflating: government/documents/76/G00-76-1211427  \n",
            "  inflating: government/documents/76/G00-76-2156458  \n",
            "  inflating: government/documents/76/G00-76-1157350  \n",
            "  inflating: government/documents/76/G00-76-3380097  \n",
            "  inflating: government/documents/76/G00-76-3480643  \n",
            "  inflating: government/documents/76/G00-76-3289607  \n",
            "   creating: government/documents/82/\n",
            "  inflating: government/documents/82/G00-82-2957676  \n",
            "  inflating: government/documents/82/G00-82-3293970  \n",
            "  inflating: government/documents/82/G00-82-1000302  \n",
            "  inflating: government/documents/82/G00-82-4103082  \n",
            "  inflating: government/documents/82/G00-82-3018441  \n",
            "  inflating: government/documents/82/G00-82-0211909  \n",
            "  inflating: government/documents/82/G00-82-0223004  \n",
            "  inflating: government/documents/82/G00-82-2961859  \n",
            "  inflating: government/documents/82/G00-82-4126643  \n",
            "  inflating: government/documents/82/G00-82-3251309  \n",
            "  inflating: government/documents/82/G00-82-2477299  \n",
            "  inflating: government/documents/82/G00-82-0453865  \n",
            "  inflating: government/documents/82/G00-82-3144058  \n",
            "  inflating: government/documents/82/G00-82-1268271  \n",
            "  inflating: government/documents/82/G00-82-0465076  \n",
            "  inflating: government/documents/82/G00-82-3336838  \n",
            "  inflating: government/documents/82/G00-82-2554245  \n",
            "  inflating: government/documents/82/G00-82-0922843  \n",
            "  inflating: government/documents/82/G00-82-1153141  \n",
            "  inflating: government/documents/82/G00-82-0793880  \n",
            "  inflating: government/documents/82/G00-82-1147697  \n",
            "  inflating: government/documents/82/G00-82-3702933  \n",
            "  inflating: government/documents/82/G00-82-3540073  \n",
            "  inflating: government/documents/82/G00-82-0740873  \n",
            "  inflating: government/documents/82/G00-82-0443423  \n",
            "  inflating: government/documents/82/G00-82-3349829  \n",
            "  inflating: government/documents/82/G00-82-2383456  \n",
            "  inflating: government/documents/82/G00-82-1906104  \n",
            "  inflating: government/documents/82/G00-82-0471625  \n",
            "  inflating: government/documents/82/G00-82-1744219  \n",
            "  inflating: government/documents/82/G00-82-0872634  \n",
            "  inflating: government/documents/82/G00-82-2065688  \n",
            "   creating: government/documents/49/\n",
            "  inflating: government/documents/49/G00-49-0846427  \n",
            "  inflating: government/documents/49/G00-49-0019145  \n",
            "  inflating: government/documents/49/G00-49-3068476  \n",
            "  inflating: government/documents/49/G00-49-2415497  \n",
            "  inflating: government/documents/49/G00-49-3982133  \n",
            "  inflating: government/documents/49/G00-49-0669001  \n",
            "  inflating: government/documents/49/G00-49-3672923  \n",
            "  inflating: government/documents/49/G00-49-0195605  \n",
            "  inflating: government/documents/49/G00-49-0053784  \n",
            "  inflating: government/documents/49/G00-49-1312433  \n",
            "  inflating: government/documents/49/G00-49-2630728  \n",
            "  inflating: government/documents/49/G00-49-0969756  \n",
            "  inflating: government/documents/49/G00-49-1872970  \n",
            "  inflating: government/documents/49/G00-49-3594785  \n",
            "  inflating: government/documents/49/G00-49-4093982  \n",
            "  inflating: government/documents/49/G00-49-2094879  \n",
            "  inflating: government/documents/49/G00-49-2898676  \n",
            "  inflating: government/documents/49/G00-49-2029636  \n",
            "  inflating: government/documents/49/G00-49-3465808  \n",
            "  inflating: government/documents/49/G00-49-2396146  \n",
            "  inflating: government/documents/49/G00-49-2685671  \n",
            "  inflating: government/documents/49/G00-49-1264822  \n",
            "  inflating: government/documents/49/G00-49-2701894  \n",
            "  inflating: government/documents/49/G00-49-1558204  \n",
            "  inflating: government/documents/49/G00-49-2121083  \n",
            "  inflating: government/documents/49/G00-49-0535670  \n",
            "  inflating: government/documents/49/G00-49-2101886  \n",
            "  inflating: government/documents/49/G00-49-2602614  \n",
            "   creating: government/documents/40/\n",
            "  inflating: government/documents/40/G00-40-2952351  \n",
            "  inflating: government/documents/40/G00-40-2126286  \n",
            "  inflating: government/documents/40/G00-40-1089727  \n",
            "  inflating: government/documents/40/G00-40-0934119  \n",
            "  inflating: government/documents/40/G00-40-3007949  \n",
            "  inflating: government/documents/40/G00-40-1960014  \n",
            "  inflating: government/documents/40/G00-40-3823068  \n",
            "  inflating: government/documents/40/G00-40-0657116  \n",
            "  inflating: government/documents/40/G00-40-1599376  \n",
            "  inflating: government/documents/40/G00-40-4123169  \n",
            "  inflating: government/documents/40/G00-40-3182751  \n",
            "  inflating: government/documents/40/G00-40-0635599  \n",
            "  inflating: government/documents/40/G00-40-3894524  \n",
            "  inflating: government/documents/40/G00-40-0540396  \n",
            "  inflating: government/documents/40/G00-40-4182751  \n",
            "  inflating: government/documents/40/G00-40-1496001  \n",
            "  inflating: government/documents/40/G00-40-2173588  \n",
            "  inflating: government/documents/40/G00-40-2134089  \n",
            "  inflating: government/documents/40/G00-40-2268123  \n",
            "  inflating: government/documents/40/G00-40-3277263  \n",
            "  inflating: government/documents/40/G00-40-1115652  \n",
            "  inflating: government/documents/40/G00-40-0492331  \n",
            "  inflating: government/documents/40/G00-40-3917304  \n",
            "  inflating: government/documents/40/G00-40-0194454  \n",
            "  inflating: government/documents/40/G00-40-4080000  \n",
            "  inflating: government/documents/40/G00-40-3261210  \n",
            "  inflating: government/documents/40/G00-40-2976726  \n",
            "  inflating: government/documents/40/G00-40-1505368  \n",
            "  inflating: government/documents/40/G00-40-2206408  \n",
            "  inflating: government/documents/40/G00-40-3544501  \n",
            "  inflating: government/documents/40/G00-40-2284866  \n",
            "  inflating: government/documents/40/G00-40-1469629  \n",
            "  inflating: government/documents/40/G00-40-1199314  \n",
            "  inflating: government/documents/40/G00-40-2252497  \n",
            "  inflating: government/documents/40/G00-40-0252660  \n",
            "  inflating: government/documents/40/G00-40-3402544  \n",
            "  inflating: government/documents/40/G00-40-0682400  \n",
            "  inflating: government/documents/40/G00-40-2295287  \n",
            "  inflating: government/documents/40/G00-40-2934506  \n",
            "  inflating: government/documents/40/G00-40-1070064  \n",
            "   creating: government/documents/47/\n",
            "  inflating: government/documents/47/G00-47-1058225  \n",
            "  inflating: government/documents/47/G00-47-1494409  \n",
            "  inflating: government/documents/47/G00-47-1562233  \n",
            "  inflating: government/documents/47/G00-47-2455664  \n",
            "  inflating: government/documents/47/G00-47-1761257  \n",
            "  inflating: government/documents/47/G00-47-1267820  \n",
            "  inflating: government/documents/47/G00-47-1044907  \n",
            "  inflating: government/documents/47/G00-47-3118394  \n",
            "  inflating: government/documents/47/G00-47-3367105  \n",
            "  inflating: government/documents/47/G00-47-1880383  \n",
            "  inflating: government/documents/47/G00-47-0128679  \n",
            "  inflating: government/documents/47/G00-47-1591952  \n",
            "  inflating: government/documents/47/G00-47-1933881  \n",
            "  inflating: government/documents/47/G00-47-2918149  \n",
            "  inflating: government/documents/47/G00-47-1040824  \n",
            "  inflating: government/documents/47/G00-47-3472789  \n",
            "  inflating: government/documents/47/G00-47-0594475  \n",
            "  inflating: government/documents/47/G00-47-2191998  \n",
            "  inflating: government/documents/47/G00-47-0197492  \n",
            "  inflating: government/documents/47/G00-47-0072041  \n",
            "  inflating: government/documents/47/G00-47-4117952  \n",
            "  inflating: government/documents/47/G00-47-1094341  \n",
            "  inflating: government/documents/47/G00-47-2967546  \n",
            "  inflating: government/documents/47/G00-47-3501009  \n",
            "  inflating: government/documents/47/G00-47-1504694  \n",
            "  inflating: government/documents/47/G00-47-1352318  \n",
            "  inflating: government/documents/47/G00-47-2117970  \n",
            "  inflating: government/documents/47/G00-47-3819051  \n",
            "  inflating: government/documents/47/G00-47-3296441  \n",
            "   creating: government/documents/78/\n",
            "  inflating: government/documents/78/G00-78-3557324  \n",
            "  inflating: government/documents/78/G00-78-0412688  \n",
            "  inflating: government/documents/78/G00-78-0419893  \n",
            "  inflating: government/documents/78/G00-78-3080721  \n",
            "  inflating: government/documents/78/G00-78-1086448  \n",
            "  inflating: government/documents/78/G00-78-2271043  \n",
            "  inflating: government/documents/78/G00-78-1392606  \n",
            "  inflating: government/documents/78/G00-78-1575317  \n",
            "  inflating: government/documents/78/G00-78-3963422  \n",
            "  inflating: government/documents/78/G00-78-1593253  \n",
            "  inflating: government/documents/78/G00-78-3059004  \n",
            "  inflating: government/documents/78/G00-78-2116321  \n",
            "  inflating: government/documents/78/G00-78-0877232  \n",
            "  inflating: government/documents/78/G00-78-3978118  \n",
            "  inflating: government/documents/78/G00-78-1364426  \n",
            "  inflating: government/documents/78/G00-78-3451918  \n",
            "  inflating: government/documents/78/G00-78-0045864  \n",
            "  inflating: government/documents/78/G00-78-2047390  \n",
            "  inflating: government/documents/78/G00-78-2804464  \n",
            "  inflating: government/documents/78/G00-78-1144993  \n",
            "  inflating: government/documents/78/G00-78-1609511  \n",
            "  inflating: government/documents/78/G00-78-3388040  \n",
            "  inflating: government/documents/78/G00-78-2696016  \n",
            "  inflating: government/documents/78/G00-78-0686285  \n",
            "  inflating: government/documents/78/G00-78-2551063  \n",
            "  inflating: government/documents/78/G00-78-1531079  \n",
            "  inflating: government/documents/78/G00-78-0515193  \n",
            "  inflating: government/documents/78/G00-78-3252910  \n",
            "  inflating: government/documents/78/G00-78-0709063  \n",
            "  inflating: government/documents/78/G00-78-4117174  \n",
            "  inflating: government/documents/78/G00-78-0544440  \n",
            "  inflating: government/documents/78/G00-78-2978026  \n",
            "  inflating: government/documents/78/G00-78-2441218  \n",
            "  inflating: government/documents/78/G00-78-0143635  \n",
            "  inflating: government/documents/78/G00-78-3691063  \n",
            "   creating: government/documents/13/\n",
            "  inflating: government/documents/13/G00-13-2475665  \n",
            "  inflating: government/documents/13/G00-13-3897486  \n",
            "  inflating: government/documents/13/G00-13-2160659  \n",
            "  inflating: government/documents/13/G00-13-0091418  \n",
            "  inflating: government/documents/13/G00-13-3039445  \n",
            "  inflating: government/documents/13/G00-13-0969542  \n",
            "  inflating: government/documents/13/G00-13-3400792  \n",
            "  inflating: government/documents/13/G00-13-2963939  \n",
            "  inflating: government/documents/13/G00-13-0220876  \n",
            "  inflating: government/documents/13/G00-13-4020618  \n",
            "  inflating: government/documents/13/G00-13-1993809  \n",
            "  inflating: government/documents/13/G00-13-1543158  \n",
            "  inflating: government/documents/13/G00-13-2874993  \n",
            "  inflating: government/documents/13/G00-13-3621195  \n",
            "  inflating: government/documents/13/G00-13-3656253  \n",
            "  inflating: government/documents/13/G00-13-0135077  \n",
            "  inflating: government/documents/13/G00-13-1078618  \n",
            "  inflating: government/documents/13/G00-13-1939935  \n",
            "  inflating: government/documents/13/G00-13-3911701  \n",
            "  inflating: government/documents/13/G00-13-1005487  \n",
            "  inflating: government/documents/13/G00-13-1085651  \n",
            "  inflating: government/documents/13/G00-13-1931082  \n",
            "  inflating: government/documents/13/G00-13-2106516  \n",
            "  inflating: government/documents/13/G00-13-1105403  \n",
            "  inflating: government/documents/13/G00-13-1139439  \n",
            "  inflating: government/documents/13/G00-13-1476848  \n",
            "  inflating: government/documents/13/G00-13-2965738  \n",
            "  inflating: government/documents/13/G00-13-3991878  \n",
            "  inflating: government/documents/13/G00-13-3394460  \n",
            "   creating: government/documents/14/\n",
            "  inflating: government/documents/14/G00-14-1873492  \n",
            "  inflating: government/documents/14/G00-14-2857303  \n",
            "  inflating: government/documents/14/G00-14-1375985  \n",
            "  inflating: government/documents/14/G00-14-3400567  \n",
            "  inflating: government/documents/14/G00-14-1002771  \n",
            "  inflating: government/documents/14/G00-14-1263885  \n",
            "  inflating: government/documents/14/G00-14-1054960  \n",
            "  inflating: government/documents/14/G00-14-3661579  \n",
            "  inflating: government/documents/14/G00-14-2961673  \n",
            "  inflating: government/documents/14/G00-14-4159159  \n",
            "  inflating: government/documents/14/G00-14-2161877  \n",
            "  inflating: government/documents/14/G00-14-2266369  \n",
            "  inflating: government/documents/14/G00-14-3787870  \n",
            "  inflating: government/documents/14/G00-14-0844999  \n",
            "  inflating: government/documents/14/G00-14-2429748  \n",
            "  inflating: government/documents/14/G00-14-1585650  \n",
            "  inflating: government/documents/14/G00-14-2358559  \n",
            "  inflating: government/documents/14/G00-14-0931254  \n",
            "  inflating: government/documents/14/G00-14-2670446  \n",
            "  inflating: government/documents/14/G00-14-1850196  \n",
            "  inflating: government/documents/14/G00-14-2310358  \n",
            "  inflating: government/documents/14/G00-14-1553441  \n",
            "  inflating: government/documents/14/G00-14-1818851  \n",
            "  inflating: government/documents/14/G00-14-2329542  \n",
            "  inflating: government/documents/14/G00-14-1443227  \n",
            "   creating: government/documents/22/\n",
            "  inflating: government/documents/22/G00-22-1281276  \n",
            "  inflating: government/documents/22/G00-22-3205119  \n",
            "  inflating: government/documents/22/G00-22-2100718  \n",
            "  inflating: government/documents/22/G00-22-3347016  \n",
            "  inflating: government/documents/22/G00-22-3220001  \n",
            "  inflating: government/documents/22/G00-22-2230354  \n",
            "  inflating: government/documents/22/G00-22-1017526  \n",
            "  inflating: government/documents/22/G00-22-2086631  \n",
            "  inflating: government/documents/22/G00-22-3347794  \n",
            "  inflating: government/documents/22/G00-22-1746423  \n",
            "  inflating: government/documents/22/G00-22-0100444  \n",
            "  inflating: government/documents/22/G00-22-3826966  \n",
            "  inflating: government/documents/22/G00-22-1831204  \n",
            "  inflating: government/documents/22/G00-22-2985512  \n",
            "  inflating: government/documents/22/G00-22-0206671  \n",
            "  inflating: government/documents/22/G00-22-0786298  \n",
            "  inflating: government/documents/22/G00-22-0643864  \n",
            "  inflating: government/documents/22/G00-22-1635915  \n",
            "  inflating: government/documents/22/G00-22-1796180  \n",
            "  inflating: government/documents/22/G00-22-0400264  \n",
            "  inflating: government/documents/22/G00-22-2253031  \n",
            "  inflating: government/documents/22/G00-22-3983334  \n",
            "  inflating: government/documents/22/G00-22-2107598  \n",
            "  inflating: government/documents/22/G00-22-3171818  \n",
            "  inflating: government/documents/22/G00-22-2631365  \n",
            "  inflating: government/documents/22/G00-22-1192355  \n",
            "  inflating: government/documents/22/G00-22-0354429  \n",
            "  inflating: government/documents/22/G00-22-0586020  \n",
            "  inflating: government/documents/22/G00-22-0301057  \n",
            "  inflating: government/documents/22/G00-22-2864155  \n",
            "  inflating: government/documents/22/G00-22-2230919  \n",
            "  inflating: government/documents/22/G00-22-2651181  \n",
            "  inflating: government/documents/22/G00-22-3396139  \n",
            "  inflating: government/documents/22/G00-22-0689568  \n",
            "  inflating: government/documents/22/G00-22-2758386  \n",
            "   creating: government/documents/25/\n",
            "  inflating: government/documents/25/G00-25-0029321  \n",
            "  inflating: government/documents/25/G00-25-0946134  \n",
            "  inflating: government/documents/25/G00-25-1076458  \n",
            "  inflating: government/documents/25/G00-25-0953762  \n",
            "  inflating: government/documents/25/G00-25-0565492  \n",
            "  inflating: government/documents/25/G00-25-3064175  \n",
            "  inflating: government/documents/25/G00-25-2855153  \n",
            "  inflating: government/documents/25/G00-25-2361981  \n",
            "  inflating: government/documents/25/G00-25-1502844  \n",
            "  inflating: government/documents/25/G00-25-1856150  \n",
            "  inflating: government/documents/25/G00-25-0870759  \n",
            "  inflating: government/documents/25/G00-25-1633405  \n",
            "  inflating: government/documents/25/G00-25-0917121  \n",
            "  inflating: government/documents/25/G00-25-3640781  \n",
            "  inflating: government/documents/25/G00-25-0617747  \n",
            "  inflating: government/documents/25/G00-25-4035691  \n",
            "  inflating: government/documents/25/G00-25-0436750  \n",
            "  inflating: government/documents/25/G00-25-3369034  \n",
            "  inflating: government/documents/25/G00-25-3378717  \n",
            "  inflating: government/documents/25/G00-25-0025068  \n",
            "  inflating: government/documents/25/G00-25-1743771  \n",
            "  inflating: government/documents/25/G00-25-1916045  \n",
            "  inflating: government/documents/25/G00-25-1029982  \n",
            "  inflating: government/documents/25/G00-25-0435040  \n",
            "  inflating: government/documents/25/G00-25-0827853  \n",
            "  inflating: government/documents/25/G00-25-3027752  \n",
            "  inflating: government/documents/25/G00-25-1453487  \n",
            "  inflating: government/documents/25/G00-25-1928405  \n",
            "  inflating: government/documents/25/G00-25-1583528  \n",
            "  inflating: government/documents/25/G00-25-3018984  \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7VNv24P839Wk",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "outputId": "5df3ed6a-ea17-4859-a46a-255dcf05c555"
      },
      "source": [
        "# imports\n",
        "# Put all your imports here\n",
        "from whoosh import index, writing\n",
        "from whoosh.fields import Schema, TEXT, KEYWORD, ID, STORED\n",
        "from whoosh.analysis import *\n",
        "from whoosh.qparser import QueryParser\n",
        "import os.path\n",
        "from pathlib import Path\n",
        "import tempfile\n",
        "import subprocess\n",
        "import pytrec_eval\n",
        "import nltk\n",
        "from nltk.stem import *\n",
        "nltk.download(\"wordnet\")\n",
        "from whoosh.analysis import Filter\n",
        "from whoosh.scoring import BM25F"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/wordnet.zip.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6CTt1NZr39Wo"
      },
      "source": [
        "DATA_DIR = \"government\"\n",
        "#\n",
        "# Put other path constants here\n",
        "document_dir=os.path.join(DATA_DIR,'documents')\n",
        "topic_file=os.path.join(DATA_DIR,'gov.topics')\n",
        "qrels_file=os.path.join(DATA_DIR,'gov.qrels')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MaebEQWj39Wt"
      },
      "source": [
        "## Part 1: Indexing and querying\n",
        "Index the government documents, run the queries (\\topics\") through (vanilla) Whoosh as a baseline\n",
        "system, and run pytrec eval to compare Whoosh’s results with human judgments."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v5ZkoByd39Wv"
      },
      "source": [
        "# Put your code for creating the index here (you can add more cells).\n",
        "# Make sure you save the final index in the variable INDEX_Q2, your query parser in QP_Q2, and your searcher in SEARCHER_Q2\n",
        "def createIndex(schema):\n",
        "    indexDir=tempfile.mkdtemp()\n",
        "    return index.create_in(indexDir,schema)\n",
        "mySchema=Schema(file_path=ID(stored=True),file_content=TEXT(analyzer=RegexTokenizer()))\n",
        "myIndex=createIndex(mySchema)\n",
        "filesToIndex=[str(filePath) for filePath in Path(document_dir).glob(\"**/*\") if filePath.is_file()]\n",
        "def addFilesToIndex(indexObj, fileList):\n",
        "    writer=writing.BufferedWriter(indexObj, period=None, limit=1000)\n",
        "    try:\n",
        "        for docNum, filePath in enumerate(fileList):\n",
        "            with open(filePath,'r',encoding='utf-8') as f:\n",
        "                fileContent=f.read()\n",
        "                writer.add_document(file_path=filePath, file_content=fileContent)\n",
        "    finally:\n",
        "        writer.close()\n",
        "addFilesToIndex(myIndex,filesToIndex)\n",
        "\n",
        "\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ibPPoKXF39Wx"
      },
      "source": [
        "INDEX_Q2=myIndex # Replace None with your index for Q2\n",
        "QP_Q2=QueryParser('file_content',schema=myIndex.schema) # Replace None with your query parser for Q2\n",
        "SEARCHER_Q2=myIndex.searcher() # Replace None with your searcher for Q2"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JlhFSMMOHPkW"
      },
      "source": [
        "def pyTrecEval(topicFile, qrelsFile, queryParser, searcher):\n",
        "    # Load topic file - a list of topics(search phrases) used for evalutation\n",
        "    with open(topicFile, \"r\") as tf:\n",
        "        topics = tf.read().splitlines()\n",
        "\n",
        "    # create an output file to which we'll write our results\n",
        "    tempOutputFile = tempfile.mkstemp()[1]\n",
        "    with open(tempOutputFile, \"w\") as outputTRECFile:\n",
        "        # for each evaluated topic:\n",
        "        # build a query and record the results in the file in TREC_EVAL format\n",
        "        for topic in topics:\n",
        "            topic_id, topic_phrase = tuple(topic.split(\" \", 1))\n",
        "            #print(topic_id, topic_phrase)\n",
        "            topicQuery = queryParser.parse(topic_phrase)\n",
        "            topicResults = searcher.search(topicQuery, limit=None)\n",
        "            for (docnum, result) in enumerate(topicResults):\n",
        "                score = topicResults.score(docnum)\n",
        "                #print(\"%s Q0 %s %d %lf test\\n\" % (topic_id, os.path.basename(result[\"file_path\"]), docnum, score))\n",
        "                outputTRECFile.write(\"%s Q0 %s %d %lf test\\n\" % (topic_id, os.path.basename(result[\"file_path\"]), docnum, score))\n",
        "    with open(qrelsFile, 'r') as f_qrel:\n",
        "        qrel = pytrec_eval.parse_qrel(f_qrel)\n",
        "\n",
        "    with open(tempOutputFile, 'r') as f_run:\n",
        "        run = pytrec_eval.parse_run(f_run)\n",
        "\n",
        "    evaluator = pytrec_eval.RelevanceEvaluator(\n",
        "        qrel, pytrec_eval.supported_measures)\n",
        "\n",
        "    results = evaluator.evaluate(run)\n",
        "    def print_line(measure, scope, value):\n",
        "        print('{:25s}{:8s}{:.4f}'.format(measure, scope, value))\n",
        "\n",
        "    for query_id, query_measures in results.items():\n",
        "        for measure, value in query_measures.items():\n",
        "            if measure == \"runid\":\n",
        "              continue\n",
        "            print_line(measure, query_id, value)\n",
        "    for measure in query_measures.keys():\n",
        "        if measure == \"runid\":\n",
        "              continue\n",
        "        print_line(\n",
        "            measure,\n",
        "            'all',\n",
        "            pytrec_eval.compute_aggregated_measure(\n",
        "                measure,\n",
        "                [query_measures[measure]\n",
        "                 for query_measures in results.values()]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Oa_ZABJtNg_D",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "af87aac8-f916-4a68-ed72-a6d0de703787"
      },
      "source": [
        "pyTrecEval(topic_file, qrels_file, QP_Q2, SEARCHER_Q2) "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "num_q                    1       1.0000\n",
            "num_ret                  1       1.0000\n",
            "num_rel                  1       5.0000\n",
            "num_rel_ret              1       0.0000\n",
            "map                      1       0.0000\n",
            "gm_map                   1       -11.5129\n",
            "Rprec                    1       0.0000\n",
            "bpref                    1       0.0000\n",
            "recip_rank               1       0.0000\n",
            "iprec_at_recall_0.00     1       0.0000\n",
            "iprec_at_recall_0.10     1       0.0000\n",
            "iprec_at_recall_0.20     1       0.0000\n",
            "iprec_at_recall_0.30     1       0.0000\n",
            "iprec_at_recall_0.40     1       0.0000\n",
            "iprec_at_recall_0.50     1       0.0000\n",
            "iprec_at_recall_0.60     1       0.0000\n",
            "iprec_at_recall_0.70     1       0.0000\n",
            "iprec_at_recall_0.80     1       0.0000\n",
            "iprec_at_recall_0.90     1       0.0000\n",
            "iprec_at_recall_1.00     1       0.0000\n",
            "P_5                      1       0.0000\n",
            "P_10                     1       0.0000\n",
            "P_15                     1       0.0000\n",
            "P_20                     1       0.0000\n",
            "P_30                     1       0.0000\n",
            "P_100                    1       0.0000\n",
            "P_200                    1       0.0000\n",
            "P_500                    1       0.0000\n",
            "P_1000                   1       0.0000\n",
            "relstring                1       0.0000\n",
            "recall_5                 1       0.0000\n",
            "recall_10                1       0.0000\n",
            "recall_15                1       0.0000\n",
            "recall_20                1       0.0000\n",
            "recall_30                1       0.0000\n",
            "recall_100               1       0.0000\n",
            "recall_200               1       0.0000\n",
            "recall_500               1       0.0000\n",
            "recall_1000              1       0.0000\n",
            "infAP                    1       0.0000\n",
            "gm_bpref                 1       -11.5129\n",
            "Rprec_mult_0.20          1       0.0000\n",
            "Rprec_mult_0.40          1       0.0000\n",
            "Rprec_mult_0.60          1       0.0000\n",
            "Rprec_mult_0.80          1       0.0000\n",
            "Rprec_mult_1.00          1       0.0000\n",
            "Rprec_mult_1.20          1       0.0000\n",
            "Rprec_mult_1.40          1       0.0000\n",
            "Rprec_mult_1.60          1       0.0000\n",
            "Rprec_mult_1.80          1       0.0000\n",
            "Rprec_mult_2.00          1       0.0000\n",
            "utility                  1       -1.0000\n",
            "11pt_avg                 1       0.0000\n",
            "binG                     1       0.0000\n",
            "G                        1       0.0000\n",
            "ndcg                     1       0.0000\n",
            "ndcg_rel                 1       0.0000\n",
            "Rndcg                    1       0.0000\n",
            "ndcg_cut_5               1       0.0000\n",
            "ndcg_cut_10              1       0.0000\n",
            "ndcg_cut_15              1       0.0000\n",
            "ndcg_cut_20              1       0.0000\n",
            "ndcg_cut_30              1       0.0000\n",
            "ndcg_cut_100             1       0.0000\n",
            "ndcg_cut_200             1       0.0000\n",
            "ndcg_cut_500             1       0.0000\n",
            "ndcg_cut_1000            1       0.0000\n",
            "map_cut_5                1       0.0000\n",
            "map_cut_10               1       0.0000\n",
            "map_cut_15               1       0.0000\n",
            "map_cut_20               1       0.0000\n",
            "map_cut_30               1       0.0000\n",
            "map_cut_100              1       0.0000\n",
            "map_cut_200              1       0.0000\n",
            "map_cut_500              1       0.0000\n",
            "map_cut_1000             1       0.0000\n",
            "relative_P_5             1       0.0000\n",
            "relative_P_10            1       0.0000\n",
            "relative_P_15            1       0.0000\n",
            "relative_P_20            1       0.0000\n",
            "relative_P_30            1       0.0000\n",
            "relative_P_100           1       0.0000\n",
            "relative_P_200           1       0.0000\n",
            "relative_P_500           1       0.0000\n",
            "relative_P_1000          1       0.0000\n",
            "success_1                1       0.0000\n",
            "success_5                1       0.0000\n",
            "success_10               1       0.0000\n",
            "set_P                    1       0.0000\n",
            "set_relative_P           1       0.0000\n",
            "set_recall               1       0.0000\n",
            "set_map                  1       0.0000\n",
            "set_F                    1       0.0000\n",
            "num_nonrel_judged_ret    1       1.0000\n",
            "num_q                    2       1.0000\n",
            "num_ret                  2       6.0000\n",
            "num_rel                  2       2.0000\n",
            "num_rel_ret              2       0.0000\n",
            "map                      2       0.0000\n",
            "gm_map                   2       -11.5129\n",
            "Rprec                    2       0.0000\n",
            "bpref                    2       0.0000\n",
            "recip_rank               2       0.0000\n",
            "iprec_at_recall_0.00     2       0.0000\n",
            "iprec_at_recall_0.10     2       0.0000\n",
            "iprec_at_recall_0.20     2       0.0000\n",
            "iprec_at_recall_0.30     2       0.0000\n",
            "iprec_at_recall_0.40     2       0.0000\n",
            "iprec_at_recall_0.50     2       0.0000\n",
            "iprec_at_recall_0.60     2       0.0000\n",
            "iprec_at_recall_0.70     2       0.0000\n",
            "iprec_at_recall_0.80     2       0.0000\n",
            "iprec_at_recall_0.90     2       0.0000\n",
            "iprec_at_recall_1.00     2       0.0000\n",
            "P_5                      2       0.0000\n",
            "P_10                     2       0.0000\n",
            "P_15                     2       0.0000\n",
            "P_20                     2       0.0000\n",
            "P_30                     2       0.0000\n",
            "P_100                    2       0.0000\n",
            "P_200                    2       0.0000\n",
            "P_500                    2       0.0000\n",
            "P_1000                   2       0.0000\n",
            "relstring                2       0.0000\n",
            "recall_5                 2       0.0000\n",
            "recall_10                2       0.0000\n",
            "recall_15                2       0.0000\n",
            "recall_20                2       0.0000\n",
            "recall_30                2       0.0000\n",
            "recall_100               2       0.0000\n",
            "recall_200               2       0.0000\n",
            "recall_500               2       0.0000\n",
            "recall_1000              2       0.0000\n",
            "infAP                    2       0.0000\n",
            "gm_bpref                 2       -11.5129\n",
            "Rprec_mult_0.20          2       0.0000\n",
            "Rprec_mult_0.40          2       0.0000\n",
            "Rprec_mult_0.60          2       0.0000\n",
            "Rprec_mult_0.80          2       0.0000\n",
            "Rprec_mult_1.00          2       0.0000\n",
            "Rprec_mult_1.20          2       0.0000\n",
            "Rprec_mult_1.40          2       0.0000\n",
            "Rprec_mult_1.60          2       0.0000\n",
            "Rprec_mult_1.80          2       0.0000\n",
            "Rprec_mult_2.00          2       0.0000\n",
            "utility                  2       -6.0000\n",
            "11pt_avg                 2       0.0000\n",
            "binG                     2       0.0000\n",
            "G                        2       0.0000\n",
            "ndcg                     2       0.0000\n",
            "ndcg_rel                 2       0.0000\n",
            "Rndcg                    2       0.0000\n",
            "ndcg_cut_5               2       0.0000\n",
            "ndcg_cut_10              2       0.0000\n",
            "ndcg_cut_15              2       0.0000\n",
            "ndcg_cut_20              2       0.0000\n",
            "ndcg_cut_30              2       0.0000\n",
            "ndcg_cut_100             2       0.0000\n",
            "ndcg_cut_200             2       0.0000\n",
            "ndcg_cut_500             2       0.0000\n",
            "ndcg_cut_1000            2       0.0000\n",
            "map_cut_5                2       0.0000\n",
            "map_cut_10               2       0.0000\n",
            "map_cut_15               2       0.0000\n",
            "map_cut_20               2       0.0000\n",
            "map_cut_30               2       0.0000\n",
            "map_cut_100              2       0.0000\n",
            "map_cut_200              2       0.0000\n",
            "map_cut_500              2       0.0000\n",
            "map_cut_1000             2       0.0000\n",
            "relative_P_5             2       0.0000\n",
            "relative_P_10            2       0.0000\n",
            "relative_P_15            2       0.0000\n",
            "relative_P_20            2       0.0000\n",
            "relative_P_30            2       0.0000\n",
            "relative_P_100           2       0.0000\n",
            "relative_P_200           2       0.0000\n",
            "relative_P_500           2       0.0000\n",
            "relative_P_1000          2       0.0000\n",
            "success_1                2       0.0000\n",
            "success_5                2       0.0000\n",
            "success_10               2       0.0000\n",
            "set_P                    2       0.0000\n",
            "set_relative_P           2       0.0000\n",
            "set_recall               2       0.0000\n",
            "set_map                  2       0.0000\n",
            "set_F                    2       0.0000\n",
            "num_nonrel_judged_ret    2       4.0000\n",
            "num_q                    4       1.0000\n",
            "num_ret                  4       12.0000\n",
            "num_rel                  4       4.0000\n",
            "num_rel_ret              4       1.0000\n",
            "map                      4       0.0312\n",
            "gm_map                   4       -3.4657\n",
            "Rprec                    4       0.0000\n",
            "bpref                    4       0.0000\n",
            "recip_rank               4       0.1250\n",
            "iprec_at_recall_0.00     4       0.1250\n",
            "iprec_at_recall_0.10     4       0.1250\n",
            "iprec_at_recall_0.20     4       0.1250\n",
            "iprec_at_recall_0.30     4       0.0000\n",
            "iprec_at_recall_0.40     4       0.0000\n",
            "iprec_at_recall_0.50     4       0.0000\n",
            "iprec_at_recall_0.60     4       0.0000\n",
            "iprec_at_recall_0.70     4       0.0000\n",
            "iprec_at_recall_0.80     4       0.0000\n",
            "iprec_at_recall_0.90     4       0.0000\n",
            "iprec_at_recall_1.00     4       0.0000\n",
            "P_5                      4       0.0000\n",
            "P_10                     4       0.1000\n",
            "P_15                     4       0.0667\n",
            "P_20                     4       0.0500\n",
            "P_30                     4       0.0333\n",
            "P_100                    4       0.0100\n",
            "P_200                    4       0.0050\n",
            "P_500                    4       0.0020\n",
            "P_1000                   4       0.0010\n",
            "relstring                4       0.0000\n",
            "recall_5                 4       0.0000\n",
            "recall_10                4       0.2500\n",
            "recall_15                4       0.2500\n",
            "recall_20                4       0.2500\n",
            "recall_30                4       0.2500\n",
            "recall_100               4       0.2500\n",
            "recall_200               4       0.2500\n",
            "recall_500               4       0.2500\n",
            "recall_1000              4       0.2500\n",
            "infAP                    4       0.0313\n",
            "gm_bpref                 4       -11.5129\n",
            "Rprec_mult_0.20          4       0.0000\n",
            "Rprec_mult_0.40          4       0.0000\n",
            "Rprec_mult_0.60          4       0.0000\n",
            "Rprec_mult_0.80          4       0.0000\n",
            "Rprec_mult_1.00          4       0.0000\n",
            "Rprec_mult_1.20          4       0.0000\n",
            "Rprec_mult_1.40          4       0.0000\n",
            "Rprec_mult_1.60          4       0.0000\n",
            "Rprec_mult_1.80          4       0.1250\n",
            "Rprec_mult_2.00          4       0.1250\n",
            "utility                  4       -10.0000\n",
            "11pt_avg                 4       0.0341\n",
            "binG                     4       0.0789\n",
            "G                        4       0.0789\n",
            "ndcg                     4       0.1232\n",
            "ndcg_rel                 4       0.1232\n",
            "Rndcg                    4       0.0616\n",
            "ndcg_cut_5               4       0.0000\n",
            "ndcg_cut_10              4       0.1232\n",
            "ndcg_cut_15              4       0.1232\n",
            "ndcg_cut_20              4       0.1232\n",
            "ndcg_cut_30              4       0.1232\n",
            "ndcg_cut_100             4       0.1232\n",
            "ndcg_cut_200             4       0.1232\n",
            "ndcg_cut_500             4       0.1232\n",
            "ndcg_cut_1000            4       0.1232\n",
            "map_cut_5                4       0.0000\n",
            "map_cut_10               4       0.0312\n",
            "map_cut_15               4       0.0312\n",
            "map_cut_20               4       0.0312\n",
            "map_cut_30               4       0.0312\n",
            "map_cut_100              4       0.0312\n",
            "map_cut_200              4       0.0312\n",
            "map_cut_500              4       0.0312\n",
            "map_cut_1000             4       0.0312\n",
            "relative_P_5             4       0.0000\n",
            "relative_P_10            4       0.2500\n",
            "relative_P_15            4       0.2500\n",
            "relative_P_20            4       0.2500\n",
            "relative_P_30            4       0.2500\n",
            "relative_P_100           4       0.2500\n",
            "relative_P_200           4       0.2500\n",
            "relative_P_500           4       0.2500\n",
            "relative_P_1000          4       0.2500\n",
            "success_1                4       0.0000\n",
            "success_5                4       0.0000\n",
            "success_10               4       1.0000\n",
            "set_P                    4       0.0833\n",
            "set_relative_P           4       0.2500\n",
            "set_recall               4       0.2500\n",
            "set_map                  4       0.0208\n",
            "set_F                    4       0.1250\n",
            "num_nonrel_judged_ret    4       9.0000\n",
            "num_q                    6       1.0000\n",
            "num_ret                  6       5.0000\n",
            "num_rel                  6       1.0000\n",
            "num_rel_ret              6       0.0000\n",
            "map                      6       0.0000\n",
            "gm_map                   6       -11.5129\n",
            "Rprec                    6       0.0000\n",
            "bpref                    6       0.0000\n",
            "recip_rank               6       0.0000\n",
            "iprec_at_recall_0.00     6       0.0000\n",
            "iprec_at_recall_0.10     6       0.0000\n",
            "iprec_at_recall_0.20     6       0.0000\n",
            "iprec_at_recall_0.30     6       0.0000\n",
            "iprec_at_recall_0.40     6       0.0000\n",
            "iprec_at_recall_0.50     6       0.0000\n",
            "iprec_at_recall_0.60     6       0.0000\n",
            "iprec_at_recall_0.70     6       0.0000\n",
            "iprec_at_recall_0.80     6       0.0000\n",
            "iprec_at_recall_0.90     6       0.0000\n",
            "iprec_at_recall_1.00     6       0.0000\n",
            "P_5                      6       0.0000\n",
            "P_10                     6       0.0000\n",
            "P_15                     6       0.0000\n",
            "P_20                     6       0.0000\n",
            "P_30                     6       0.0000\n",
            "P_100                    6       0.0000\n",
            "P_200                    6       0.0000\n",
            "P_500                    6       0.0000\n",
            "P_1000                   6       0.0000\n",
            "relstring                6       0.0000\n",
            "recall_5                 6       0.0000\n",
            "recall_10                6       0.0000\n",
            "recall_15                6       0.0000\n",
            "recall_20                6       0.0000\n",
            "recall_30                6       0.0000\n",
            "recall_100               6       0.0000\n",
            "recall_200               6       0.0000\n",
            "recall_500               6       0.0000\n",
            "recall_1000              6       0.0000\n",
            "infAP                    6       0.0000\n",
            "gm_bpref                 6       -11.5129\n",
            "Rprec_mult_0.20          6       0.0000\n",
            "Rprec_mult_0.40          6       0.0000\n",
            "Rprec_mult_0.60          6       0.0000\n",
            "Rprec_mult_0.80          6       0.0000\n",
            "Rprec_mult_1.00          6       0.0000\n",
            "Rprec_mult_1.20          6       0.0000\n",
            "Rprec_mult_1.40          6       0.0000\n",
            "Rprec_mult_1.60          6       0.0000\n",
            "Rprec_mult_1.80          6       0.0000\n",
            "Rprec_mult_2.00          6       0.0000\n",
            "utility                  6       -5.0000\n",
            "11pt_avg                 6       0.0000\n",
            "binG                     6       0.0000\n",
            "G                        6       0.0000\n",
            "ndcg                     6       0.0000\n",
            "ndcg_rel                 6       0.0000\n",
            "Rndcg                    6       0.0000\n",
            "ndcg_cut_5               6       0.0000\n",
            "ndcg_cut_10              6       0.0000\n",
            "ndcg_cut_15              6       0.0000\n",
            "ndcg_cut_20              6       0.0000\n",
            "ndcg_cut_30              6       0.0000\n",
            "ndcg_cut_100             6       0.0000\n",
            "ndcg_cut_200             6       0.0000\n",
            "ndcg_cut_500             6       0.0000\n",
            "ndcg_cut_1000            6       0.0000\n",
            "map_cut_5                6       0.0000\n",
            "map_cut_10               6       0.0000\n",
            "map_cut_15               6       0.0000\n",
            "map_cut_20               6       0.0000\n",
            "map_cut_30               6       0.0000\n",
            "map_cut_100              6       0.0000\n",
            "map_cut_200              6       0.0000\n",
            "map_cut_500              6       0.0000\n",
            "map_cut_1000             6       0.0000\n",
            "relative_P_5             6       0.0000\n",
            "relative_P_10            6       0.0000\n",
            "relative_P_15            6       0.0000\n",
            "relative_P_20            6       0.0000\n",
            "relative_P_30            6       0.0000\n",
            "relative_P_100           6       0.0000\n",
            "relative_P_200           6       0.0000\n",
            "relative_P_500           6       0.0000\n",
            "relative_P_1000          6       0.0000\n",
            "success_1                6       0.0000\n",
            "success_5                6       0.0000\n",
            "success_10               6       0.0000\n",
            "set_P                    6       0.0000\n",
            "set_relative_P           6       0.0000\n",
            "set_recall               6       0.0000\n",
            "set_map                  6       0.0000\n",
            "set_F                    6       0.0000\n",
            "num_nonrel_judged_ret    6       4.0000\n",
            "num_q                    7       1.0000\n",
            "num_ret                  7       7.0000\n",
            "num_rel                  7       3.0000\n",
            "num_rel_ret              7       0.0000\n",
            "map                      7       0.0000\n",
            "gm_map                   7       -11.5129\n",
            "Rprec                    7       0.0000\n",
            "bpref                    7       0.0000\n",
            "recip_rank               7       0.0000\n",
            "iprec_at_recall_0.00     7       0.0000\n",
            "iprec_at_recall_0.10     7       0.0000\n",
            "iprec_at_recall_0.20     7       0.0000\n",
            "iprec_at_recall_0.30     7       0.0000\n",
            "iprec_at_recall_0.40     7       0.0000\n",
            "iprec_at_recall_0.50     7       0.0000\n",
            "iprec_at_recall_0.60     7       0.0000\n",
            "iprec_at_recall_0.70     7       0.0000\n",
            "iprec_at_recall_0.80     7       0.0000\n",
            "iprec_at_recall_0.90     7       0.0000\n",
            "iprec_at_recall_1.00     7       0.0000\n",
            "P_5                      7       0.0000\n",
            "P_10                     7       0.0000\n",
            "P_15                     7       0.0000\n",
            "P_20                     7       0.0000\n",
            "P_30                     7       0.0000\n",
            "P_100                    7       0.0000\n",
            "P_200                    7       0.0000\n",
            "P_500                    7       0.0000\n",
            "P_1000                   7       0.0000\n",
            "relstring                7       0.0000\n",
            "recall_5                 7       0.0000\n",
            "recall_10                7       0.0000\n",
            "recall_15                7       0.0000\n",
            "recall_20                7       0.0000\n",
            "recall_30                7       0.0000\n",
            "recall_100               7       0.0000\n",
            "recall_200               7       0.0000\n",
            "recall_500               7       0.0000\n",
            "recall_1000              7       0.0000\n",
            "infAP                    7       0.0000\n",
            "gm_bpref                 7       -11.5129\n",
            "Rprec_mult_0.20          7       0.0000\n",
            "Rprec_mult_0.40          7       0.0000\n",
            "Rprec_mult_0.60          7       0.0000\n",
            "Rprec_mult_0.80          7       0.0000\n",
            "Rprec_mult_1.00          7       0.0000\n",
            "Rprec_mult_1.20          7       0.0000\n",
            "Rprec_mult_1.40          7       0.0000\n",
            "Rprec_mult_1.60          7       0.0000\n",
            "Rprec_mult_1.80          7       0.0000\n",
            "Rprec_mult_2.00          7       0.0000\n",
            "utility                  7       -7.0000\n",
            "11pt_avg                 7       0.0000\n",
            "binG                     7       0.0000\n",
            "G                        7       0.0000\n",
            "ndcg                     7       0.0000\n",
            "ndcg_rel                 7       0.0000\n",
            "Rndcg                    7       0.0000\n",
            "ndcg_cut_5               7       0.0000\n",
            "ndcg_cut_10              7       0.0000\n",
            "ndcg_cut_15              7       0.0000\n",
            "ndcg_cut_20              7       0.0000\n",
            "ndcg_cut_30              7       0.0000\n",
            "ndcg_cut_100             7       0.0000\n",
            "ndcg_cut_200             7       0.0000\n",
            "ndcg_cut_500             7       0.0000\n",
            "ndcg_cut_1000            7       0.0000\n",
            "map_cut_5                7       0.0000\n",
            "map_cut_10               7       0.0000\n",
            "map_cut_15               7       0.0000\n",
            "map_cut_20               7       0.0000\n",
            "map_cut_30               7       0.0000\n",
            "map_cut_100              7       0.0000\n",
            "map_cut_200              7       0.0000\n",
            "map_cut_500              7       0.0000\n",
            "map_cut_1000             7       0.0000\n",
            "relative_P_5             7       0.0000\n",
            "relative_P_10            7       0.0000\n",
            "relative_P_15            7       0.0000\n",
            "relative_P_20            7       0.0000\n",
            "relative_P_30            7       0.0000\n",
            "relative_P_100           7       0.0000\n",
            "relative_P_200           7       0.0000\n",
            "relative_P_500           7       0.0000\n",
            "relative_P_1000          7       0.0000\n",
            "success_1                7       0.0000\n",
            "success_5                7       0.0000\n",
            "success_10               7       0.0000\n",
            "set_P                    7       0.0000\n",
            "set_relative_P           7       0.0000\n",
            "set_recall               7       0.0000\n",
            "set_map                  7       0.0000\n",
            "set_F                    7       0.0000\n",
            "num_nonrel_judged_ret    7       5.0000\n",
            "num_q                    9       1.0000\n",
            "num_ret                  9       2.0000\n",
            "num_rel                  9       1.0000\n",
            "num_rel_ret              9       0.0000\n",
            "map                      9       0.0000\n",
            "gm_map                   9       -11.5129\n",
            "Rprec                    9       0.0000\n",
            "bpref                    9       0.0000\n",
            "recip_rank               9       0.0000\n",
            "iprec_at_recall_0.00     9       0.0000\n",
            "iprec_at_recall_0.10     9       0.0000\n",
            "iprec_at_recall_0.20     9       0.0000\n",
            "iprec_at_recall_0.30     9       0.0000\n",
            "iprec_at_recall_0.40     9       0.0000\n",
            "iprec_at_recall_0.50     9       0.0000\n",
            "iprec_at_recall_0.60     9       0.0000\n",
            "iprec_at_recall_0.70     9       0.0000\n",
            "iprec_at_recall_0.80     9       0.0000\n",
            "iprec_at_recall_0.90     9       0.0000\n",
            "iprec_at_recall_1.00     9       0.0000\n",
            "P_5                      9       0.0000\n",
            "P_10                     9       0.0000\n",
            "P_15                     9       0.0000\n",
            "P_20                     9       0.0000\n",
            "P_30                     9       0.0000\n",
            "P_100                    9       0.0000\n",
            "P_200                    9       0.0000\n",
            "P_500                    9       0.0000\n",
            "P_1000                   9       0.0000\n",
            "relstring                9       0.0000\n",
            "recall_5                 9       0.0000\n",
            "recall_10                9       0.0000\n",
            "recall_15                9       0.0000\n",
            "recall_20                9       0.0000\n",
            "recall_30                9       0.0000\n",
            "recall_100               9       0.0000\n",
            "recall_200               9       0.0000\n",
            "recall_500               9       0.0000\n",
            "recall_1000              9       0.0000\n",
            "infAP                    9       0.0000\n",
            "gm_bpref                 9       -11.5129\n",
            "Rprec_mult_0.20          9       0.0000\n",
            "Rprec_mult_0.40          9       0.0000\n",
            "Rprec_mult_0.60          9       0.0000\n",
            "Rprec_mult_0.80          9       0.0000\n",
            "Rprec_mult_1.00          9       0.0000\n",
            "Rprec_mult_1.20          9       0.0000\n",
            "Rprec_mult_1.40          9       0.0000\n",
            "Rprec_mult_1.60          9       0.0000\n",
            "Rprec_mult_1.80          9       0.0000\n",
            "Rprec_mult_2.00          9       0.0000\n",
            "utility                  9       -2.0000\n",
            "11pt_avg                 9       0.0000\n",
            "binG                     9       0.0000\n",
            "G                        9       0.0000\n",
            "ndcg                     9       0.0000\n",
            "ndcg_rel                 9       0.0000\n",
            "Rndcg                    9       0.0000\n",
            "ndcg_cut_5               9       0.0000\n",
            "ndcg_cut_10              9       0.0000\n",
            "ndcg_cut_15              9       0.0000\n",
            "ndcg_cut_20              9       0.0000\n",
            "ndcg_cut_30              9       0.0000\n",
            "ndcg_cut_100             9       0.0000\n",
            "ndcg_cut_200             9       0.0000\n",
            "ndcg_cut_500             9       0.0000\n",
            "ndcg_cut_1000            9       0.0000\n",
            "map_cut_5                9       0.0000\n",
            "map_cut_10               9       0.0000\n",
            "map_cut_15               9       0.0000\n",
            "map_cut_20               9       0.0000\n",
            "map_cut_30               9       0.0000\n",
            "map_cut_100              9       0.0000\n",
            "map_cut_200              9       0.0000\n",
            "map_cut_500              9       0.0000\n",
            "map_cut_1000             9       0.0000\n",
            "relative_P_5             9       0.0000\n",
            "relative_P_10            9       0.0000\n",
            "relative_P_15            9       0.0000\n",
            "relative_P_20            9       0.0000\n",
            "relative_P_30            9       0.0000\n",
            "relative_P_100           9       0.0000\n",
            "relative_P_200           9       0.0000\n",
            "relative_P_500           9       0.0000\n",
            "relative_P_1000          9       0.0000\n",
            "success_1                9       0.0000\n",
            "success_5                9       0.0000\n",
            "success_10               9       0.0000\n",
            "set_P                    9       0.0000\n",
            "set_relative_P           9       0.0000\n",
            "set_recall               9       0.0000\n",
            "set_map                  9       0.0000\n",
            "set_F                    9       0.0000\n",
            "num_nonrel_judged_ret    9       2.0000\n",
            "num_q                    10      1.0000\n",
            "num_ret                  10      16.0000\n",
            "num_rel                  10      1.0000\n",
            "num_rel_ret              10      1.0000\n",
            "map                      10      0.1667\n",
            "gm_map                   10      -1.7918\n",
            "Rprec                    10      0.0000\n",
            "bpref                    10      0.0000\n",
            "recip_rank               10      0.1667\n",
            "iprec_at_recall_0.00     10      0.1667\n",
            "iprec_at_recall_0.10     10      0.1667\n",
            "iprec_at_recall_0.20     10      0.1667\n",
            "iprec_at_recall_0.30     10      0.1667\n",
            "iprec_at_recall_0.40     10      0.1667\n",
            "iprec_at_recall_0.50     10      0.1667\n",
            "iprec_at_recall_0.60     10      0.1667\n",
            "iprec_at_recall_0.70     10      0.1667\n",
            "iprec_at_recall_0.80     10      0.1667\n",
            "iprec_at_recall_0.90     10      0.1667\n",
            "iprec_at_recall_1.00     10      0.1667\n",
            "P_5                      10      0.0000\n",
            "P_10                     10      0.1000\n",
            "P_15                     10      0.0667\n",
            "P_20                     10      0.0500\n",
            "P_30                     10      0.0333\n",
            "P_100                    10      0.0100\n",
            "P_200                    10      0.0050\n",
            "P_500                    10      0.0020\n",
            "P_1000                   10      0.0010\n",
            "relstring                10      0.0000\n",
            "recall_5                 10      0.0000\n",
            "recall_10                10      1.0000\n",
            "recall_15                10      1.0000\n",
            "recall_20                10      1.0000\n",
            "recall_30                10      1.0000\n",
            "recall_100               10      1.0000\n",
            "recall_200               10      1.0000\n",
            "recall_500               10      1.0000\n",
            "recall_1000              10      1.0000\n",
            "infAP                    10      0.1667\n",
            "gm_bpref                 10      -11.5129\n",
            "Rprec_mult_0.20          10      0.0000\n",
            "Rprec_mult_0.40          10      0.0000\n",
            "Rprec_mult_0.60          10      0.0000\n",
            "Rprec_mult_0.80          10      0.0000\n",
            "Rprec_mult_1.00          10      0.0000\n",
            "Rprec_mult_1.20          10      0.0000\n",
            "Rprec_mult_1.40          10      0.0000\n",
            "Rprec_mult_1.60          10      0.0000\n",
            "Rprec_mult_1.80          10      0.0000\n",
            "Rprec_mult_2.00          10      0.0000\n",
            "utility                  10      -14.0000\n",
            "11pt_avg                 10      0.1667\n",
            "binG                     10      0.3562\n",
            "G                        10      0.3562\n",
            "ndcg                     10      0.3562\n",
            "ndcg_rel                 10      0.3562\n",
            "Rndcg                    10      0.1781\n",
            "ndcg_cut_5               10      0.0000\n",
            "ndcg_cut_10              10      0.3562\n",
            "ndcg_cut_15              10      0.3562\n",
            "ndcg_cut_20              10      0.3562\n",
            "ndcg_cut_30              10      0.3562\n",
            "ndcg_cut_100             10      0.3562\n",
            "ndcg_cut_200             10      0.3562\n",
            "ndcg_cut_500             10      0.3562\n",
            "ndcg_cut_1000            10      0.3562\n",
            "map_cut_5                10      0.0000\n",
            "map_cut_10               10      0.1667\n",
            "map_cut_15               10      0.1667\n",
            "map_cut_20               10      0.1667\n",
            "map_cut_30               10      0.1667\n",
            "map_cut_100              10      0.1667\n",
            "map_cut_200              10      0.1667\n",
            "map_cut_500              10      0.1667\n",
            "map_cut_1000             10      0.1667\n",
            "relative_P_5             10      0.0000\n",
            "relative_P_10            10      1.0000\n",
            "relative_P_15            10      1.0000\n",
            "relative_P_20            10      1.0000\n",
            "relative_P_30            10      1.0000\n",
            "relative_P_100           10      1.0000\n",
            "relative_P_200           10      1.0000\n",
            "relative_P_500           10      1.0000\n",
            "relative_P_1000          10      1.0000\n",
            "success_1                10      0.0000\n",
            "success_5                10      0.0000\n",
            "success_10               10      1.0000\n",
            "set_P                    10      0.0625\n",
            "set_relative_P           10      1.0000\n",
            "set_recall               10      1.0000\n",
            "set_map                  10      0.0625\n",
            "set_F                    10      0.1176\n",
            "num_nonrel_judged_ret    10      10.0000\n",
            "num_q                    14      1.0000\n",
            "num_ret                  14      7.0000\n",
            "num_rel                  14      1.0000\n",
            "num_rel_ret              14      1.0000\n",
            "map                      14      0.2500\n",
            "gm_map                   14      -1.3863\n",
            "Rprec                    14      0.0000\n",
            "bpref                    14      0.0000\n",
            "recip_rank               14      0.2500\n",
            "iprec_at_recall_0.00     14      0.2500\n",
            "iprec_at_recall_0.10     14      0.2500\n",
            "iprec_at_recall_0.20     14      0.2500\n",
            "iprec_at_recall_0.30     14      0.2500\n",
            "iprec_at_recall_0.40     14      0.2500\n",
            "iprec_at_recall_0.50     14      0.2500\n",
            "iprec_at_recall_0.60     14      0.2500\n",
            "iprec_at_recall_0.70     14      0.2500\n",
            "iprec_at_recall_0.80     14      0.2500\n",
            "iprec_at_recall_0.90     14      0.2500\n",
            "iprec_at_recall_1.00     14      0.2500\n",
            "P_5                      14      0.2000\n",
            "P_10                     14      0.1000\n",
            "P_15                     14      0.0667\n",
            "P_20                     14      0.0500\n",
            "P_30                     14      0.0333\n",
            "P_100                    14      0.0100\n",
            "P_200                    14      0.0050\n",
            "P_500                    14      0.0020\n",
            "P_1000                   14      0.0010\n",
            "relstring                14      0.0000\n",
            "recall_5                 14      1.0000\n",
            "recall_10                14      1.0000\n",
            "recall_15                14      1.0000\n",
            "recall_20                14      1.0000\n",
            "recall_30                14      1.0000\n",
            "recall_100               14      1.0000\n",
            "recall_200               14      1.0000\n",
            "recall_500               14      1.0000\n",
            "recall_1000              14      1.0000\n",
            "infAP                    14      0.2500\n",
            "gm_bpref                 14      -11.5129\n",
            "Rprec_mult_0.20          14      0.0000\n",
            "Rprec_mult_0.40          14      0.0000\n",
            "Rprec_mult_0.60          14      0.0000\n",
            "Rprec_mult_0.80          14      0.0000\n",
            "Rprec_mult_1.00          14      0.0000\n",
            "Rprec_mult_1.20          14      0.0000\n",
            "Rprec_mult_1.40          14      0.0000\n",
            "Rprec_mult_1.60          14      0.0000\n",
            "Rprec_mult_1.80          14      0.0000\n",
            "Rprec_mult_2.00          14      0.0000\n",
            "utility                  14      -5.0000\n",
            "11pt_avg                 14      0.2500\n",
            "binG                     14      0.4307\n",
            "G                        14      0.4307\n",
            "ndcg                     14      0.4307\n",
            "ndcg_rel                 14      0.4307\n",
            "Rndcg                    14      0.2153\n",
            "ndcg_cut_5               14      0.4307\n",
            "ndcg_cut_10              14      0.4307\n",
            "ndcg_cut_15              14      0.4307\n",
            "ndcg_cut_20              14      0.4307\n",
            "ndcg_cut_30              14      0.4307\n",
            "ndcg_cut_100             14      0.4307\n",
            "ndcg_cut_200             14      0.4307\n",
            "ndcg_cut_500             14      0.4307\n",
            "ndcg_cut_1000            14      0.4307\n",
            "map_cut_5                14      0.2500\n",
            "map_cut_10               14      0.2500\n",
            "map_cut_15               14      0.2500\n",
            "map_cut_20               14      0.2500\n",
            "map_cut_30               14      0.2500\n",
            "map_cut_100              14      0.2500\n",
            "map_cut_200              14      0.2500\n",
            "map_cut_500              14      0.2500\n",
            "map_cut_1000             14      0.2500\n",
            "relative_P_5             14      1.0000\n",
            "relative_P_10            14      1.0000\n",
            "relative_P_15            14      1.0000\n",
            "relative_P_20            14      1.0000\n",
            "relative_P_30            14      1.0000\n",
            "relative_P_100           14      1.0000\n",
            "relative_P_200           14      1.0000\n",
            "relative_P_500           14      1.0000\n",
            "relative_P_1000          14      1.0000\n",
            "success_1                14      0.0000\n",
            "success_5                14      1.0000\n",
            "success_10               14      1.0000\n",
            "set_P                    14      0.1429\n",
            "set_relative_P           14      1.0000\n",
            "set_recall               14      1.0000\n",
            "set_map                  14      0.1429\n",
            "set_F                    14      0.2500\n",
            "num_nonrel_judged_ret    14      5.0000\n",
            "num_q                    16      1.0000\n",
            "num_ret                  16      7.0000\n",
            "num_rel                  16      7.0000\n",
            "num_rel_ret              16      0.0000\n",
            "map                      16      0.0000\n",
            "gm_map                   16      -11.5129\n",
            "Rprec                    16      0.0000\n",
            "bpref                    16      0.0000\n",
            "recip_rank               16      0.0000\n",
            "iprec_at_recall_0.00     16      0.0000\n",
            "iprec_at_recall_0.10     16      0.0000\n",
            "iprec_at_recall_0.20     16      0.0000\n",
            "iprec_at_recall_0.30     16      0.0000\n",
            "iprec_at_recall_0.40     16      0.0000\n",
            "iprec_at_recall_0.50     16      0.0000\n",
            "iprec_at_recall_0.60     16      0.0000\n",
            "iprec_at_recall_0.70     16      0.0000\n",
            "iprec_at_recall_0.80     16      0.0000\n",
            "iprec_at_recall_0.90     16      0.0000\n",
            "iprec_at_recall_1.00     16      0.0000\n",
            "P_5                      16      0.0000\n",
            "P_10                     16      0.0000\n",
            "P_15                     16      0.0000\n",
            "P_20                     16      0.0000\n",
            "P_30                     16      0.0000\n",
            "P_100                    16      0.0000\n",
            "P_200                    16      0.0000\n",
            "P_500                    16      0.0000\n",
            "P_1000                   16      0.0000\n",
            "relstring                16      0.0000\n",
            "recall_5                 16      0.0000\n",
            "recall_10                16      0.0000\n",
            "recall_15                16      0.0000\n",
            "recall_20                16      0.0000\n",
            "recall_30                16      0.0000\n",
            "recall_100               16      0.0000\n",
            "recall_200               16      0.0000\n",
            "recall_500               16      0.0000\n",
            "recall_1000              16      0.0000\n",
            "infAP                    16      0.0000\n",
            "gm_bpref                 16      -11.5129\n",
            "Rprec_mult_0.20          16      0.0000\n",
            "Rprec_mult_0.40          16      0.0000\n",
            "Rprec_mult_0.60          16      0.0000\n",
            "Rprec_mult_0.80          16      0.0000\n",
            "Rprec_mult_1.00          16      0.0000\n",
            "Rprec_mult_1.20          16      0.0000\n",
            "Rprec_mult_1.40          16      0.0000\n",
            "Rprec_mult_1.60          16      0.0000\n",
            "Rprec_mult_1.80          16      0.0000\n",
            "Rprec_mult_2.00          16      0.0000\n",
            "utility                  16      -7.0000\n",
            "11pt_avg                 16      0.0000\n",
            "binG                     16      0.0000\n",
            "G                        16      0.0000\n",
            "ndcg                     16      0.0000\n",
            "ndcg_rel                 16      0.0000\n",
            "Rndcg                    16      0.0000\n",
            "ndcg_cut_5               16      0.0000\n",
            "ndcg_cut_10              16      0.0000\n",
            "ndcg_cut_15              16      0.0000\n",
            "ndcg_cut_20              16      0.0000\n",
            "ndcg_cut_30              16      0.0000\n",
            "ndcg_cut_100             16      0.0000\n",
            "ndcg_cut_200             16      0.0000\n",
            "ndcg_cut_500             16      0.0000\n",
            "ndcg_cut_1000            16      0.0000\n",
            "map_cut_5                16      0.0000\n",
            "map_cut_10               16      0.0000\n",
            "map_cut_15               16      0.0000\n",
            "map_cut_20               16      0.0000\n",
            "map_cut_30               16      0.0000\n",
            "map_cut_100              16      0.0000\n",
            "map_cut_200              16      0.0000\n",
            "map_cut_500              16      0.0000\n",
            "map_cut_1000             16      0.0000\n",
            "relative_P_5             16      0.0000\n",
            "relative_P_10            16      0.0000\n",
            "relative_P_15            16      0.0000\n",
            "relative_P_20            16      0.0000\n",
            "relative_P_30            16      0.0000\n",
            "relative_P_100           16      0.0000\n",
            "relative_P_200           16      0.0000\n",
            "relative_P_500           16      0.0000\n",
            "relative_P_1000          16      0.0000\n",
            "success_1                16      0.0000\n",
            "success_5                16      0.0000\n",
            "success_10               16      0.0000\n",
            "set_P                    16      0.0000\n",
            "set_relative_P           16      0.0000\n",
            "set_recall               16      0.0000\n",
            "set_map                  16      0.0000\n",
            "set_F                    16      0.0000\n",
            "num_nonrel_judged_ret    16      6.0000\n",
            "num_q                    18      1.0000\n",
            "num_ret                  18      17.0000\n",
            "num_rel                  18      1.0000\n",
            "num_rel_ret              18      1.0000\n",
            "map                      18      1.0000\n",
            "gm_map                   18      0.0000\n",
            "Rprec                    18      1.0000\n",
            "bpref                    18      1.0000\n",
            "recip_rank               18      1.0000\n",
            "iprec_at_recall_0.00     18      1.0000\n",
            "iprec_at_recall_0.10     18      1.0000\n",
            "iprec_at_recall_0.20     18      1.0000\n",
            "iprec_at_recall_0.30     18      1.0000\n",
            "iprec_at_recall_0.40     18      1.0000\n",
            "iprec_at_recall_0.50     18      1.0000\n",
            "iprec_at_recall_0.60     18      1.0000\n",
            "iprec_at_recall_0.70     18      1.0000\n",
            "iprec_at_recall_0.80     18      1.0000\n",
            "iprec_at_recall_0.90     18      1.0000\n",
            "iprec_at_recall_1.00     18      1.0000\n",
            "P_5                      18      0.2000\n",
            "P_10                     18      0.1000\n",
            "P_15                     18      0.0667\n",
            "P_20                     18      0.0500\n",
            "P_30                     18      0.0333\n",
            "P_100                    18      0.0100\n",
            "P_200                    18      0.0050\n",
            "P_500                    18      0.0020\n",
            "P_1000                   18      0.0010\n",
            "relstring                18      0.0000\n",
            "recall_5                 18      1.0000\n",
            "recall_10                18      1.0000\n",
            "recall_15                18      1.0000\n",
            "recall_20                18      1.0000\n",
            "recall_30                18      1.0000\n",
            "recall_100               18      1.0000\n",
            "recall_200               18      1.0000\n",
            "recall_500               18      1.0000\n",
            "recall_1000              18      1.0000\n",
            "infAP                    18      1.0000\n",
            "gm_bpref                 18      0.0000\n",
            "Rprec_mult_0.20          18      1.0000\n",
            "Rprec_mult_0.40          18      1.0000\n",
            "Rprec_mult_0.60          18      1.0000\n",
            "Rprec_mult_0.80          18      1.0000\n",
            "Rprec_mult_1.00          18      1.0000\n",
            "Rprec_mult_1.20          18      0.5000\n",
            "Rprec_mult_1.40          18      0.5000\n",
            "Rprec_mult_1.60          18      0.5000\n",
            "Rprec_mult_1.80          18      0.5000\n",
            "Rprec_mult_2.00          18      0.5000\n",
            "utility                  18      -15.0000\n",
            "11pt_avg                 18      1.0000\n",
            "binG                     18      1.0000\n",
            "G                        18      1.0000\n",
            "ndcg                     18      1.0000\n",
            "ndcg_rel                 18      1.0000\n",
            "Rndcg                    18      1.0000\n",
            "ndcg_cut_5               18      1.0000\n",
            "ndcg_cut_10              18      1.0000\n",
            "ndcg_cut_15              18      1.0000\n",
            "ndcg_cut_20              18      1.0000\n",
            "ndcg_cut_30              18      1.0000\n",
            "ndcg_cut_100             18      1.0000\n",
            "ndcg_cut_200             18      1.0000\n",
            "ndcg_cut_500             18      1.0000\n",
            "ndcg_cut_1000            18      1.0000\n",
            "map_cut_5                18      1.0000\n",
            "map_cut_10               18      1.0000\n",
            "map_cut_15               18      1.0000\n",
            "map_cut_20               18      1.0000\n",
            "map_cut_30               18      1.0000\n",
            "map_cut_100              18      1.0000\n",
            "map_cut_200              18      1.0000\n",
            "map_cut_500              18      1.0000\n",
            "map_cut_1000             18      1.0000\n",
            "relative_P_5             18      1.0000\n",
            "relative_P_10            18      1.0000\n",
            "relative_P_15            18      1.0000\n",
            "relative_P_20            18      1.0000\n",
            "relative_P_30            18      1.0000\n",
            "relative_P_100           18      1.0000\n",
            "relative_P_200           18      1.0000\n",
            "relative_P_500           18      1.0000\n",
            "relative_P_1000          18      1.0000\n",
            "success_1                18      1.0000\n",
            "success_5                18      1.0000\n",
            "success_10               18      1.0000\n",
            "set_P                    18      0.0588\n",
            "set_relative_P           18      1.0000\n",
            "set_recall               18      1.0000\n",
            "set_map                  18      0.0588\n",
            "set_F                    18      0.1111\n",
            "num_nonrel_judged_ret    18      16.0000\n",
            "num_q                    22      1.0000\n",
            "num_ret                  22      17.0000\n",
            "num_rel                  22      1.0000\n",
            "num_rel_ret              22      1.0000\n",
            "map                      22      0.2000\n",
            "gm_map                   22      -1.6094\n",
            "Rprec                    22      0.0000\n",
            "bpref                    22      0.0000\n",
            "recip_rank               22      0.2000\n",
            "iprec_at_recall_0.00     22      0.2000\n",
            "iprec_at_recall_0.10     22      0.2000\n",
            "iprec_at_recall_0.20     22      0.2000\n",
            "iprec_at_recall_0.30     22      0.2000\n",
            "iprec_at_recall_0.40     22      0.2000\n",
            "iprec_at_recall_0.50     22      0.2000\n",
            "iprec_at_recall_0.60     22      0.2000\n",
            "iprec_at_recall_0.70     22      0.2000\n",
            "iprec_at_recall_0.80     22      0.2000\n",
            "iprec_at_recall_0.90     22      0.2000\n",
            "iprec_at_recall_1.00     22      0.2000\n",
            "P_5                      22      0.2000\n",
            "P_10                     22      0.1000\n",
            "P_15                     22      0.0667\n",
            "P_20                     22      0.0500\n",
            "P_30                     22      0.0333\n",
            "P_100                    22      0.0100\n",
            "P_200                    22      0.0050\n",
            "P_500                    22      0.0020\n",
            "P_1000                   22      0.0010\n",
            "relstring                22      0.0000\n",
            "recall_5                 22      1.0000\n",
            "recall_10                22      1.0000\n",
            "recall_15                22      1.0000\n",
            "recall_20                22      1.0000\n",
            "recall_30                22      1.0000\n",
            "recall_100               22      1.0000\n",
            "recall_200               22      1.0000\n",
            "recall_500               22      1.0000\n",
            "recall_1000              22      1.0000\n",
            "infAP                    22      0.2000\n",
            "gm_bpref                 22      -11.5129\n",
            "Rprec_mult_0.20          22      0.0000\n",
            "Rprec_mult_0.40          22      0.0000\n",
            "Rprec_mult_0.60          22      0.0000\n",
            "Rprec_mult_0.80          22      0.0000\n",
            "Rprec_mult_1.00          22      0.0000\n",
            "Rprec_mult_1.20          22      0.0000\n",
            "Rprec_mult_1.40          22      0.0000\n",
            "Rprec_mult_1.60          22      0.0000\n",
            "Rprec_mult_1.80          22      0.0000\n",
            "Rprec_mult_2.00          22      0.0000\n",
            "utility                  22      -15.0000\n",
            "11pt_avg                 22      0.2000\n",
            "binG                     22      0.3869\n",
            "G                        22      0.3869\n",
            "ndcg                     22      0.3869\n",
            "ndcg_rel                 22      0.3869\n",
            "Rndcg                    22      0.1934\n",
            "ndcg_cut_5               22      0.3869\n",
            "ndcg_cut_10              22      0.3869\n",
            "ndcg_cut_15              22      0.3869\n",
            "ndcg_cut_20              22      0.3869\n",
            "ndcg_cut_30              22      0.3869\n",
            "ndcg_cut_100             22      0.3869\n",
            "ndcg_cut_200             22      0.3869\n",
            "ndcg_cut_500             22      0.3869\n",
            "ndcg_cut_1000            22      0.3869\n",
            "map_cut_5                22      0.2000\n",
            "map_cut_10               22      0.2000\n",
            "map_cut_15               22      0.2000\n",
            "map_cut_20               22      0.2000\n",
            "map_cut_30               22      0.2000\n",
            "map_cut_100              22      0.2000\n",
            "map_cut_200              22      0.2000\n",
            "map_cut_500              22      0.2000\n",
            "map_cut_1000             22      0.2000\n",
            "relative_P_5             22      1.0000\n",
            "relative_P_10            22      1.0000\n",
            "relative_P_15            22      1.0000\n",
            "relative_P_20            22      1.0000\n",
            "relative_P_30            22      1.0000\n",
            "relative_P_100           22      1.0000\n",
            "relative_P_200           22      1.0000\n",
            "relative_P_500           22      1.0000\n",
            "relative_P_1000          22      1.0000\n",
            "success_1                22      0.0000\n",
            "success_5                22      1.0000\n",
            "success_10               22      1.0000\n",
            "set_P                    22      0.0588\n",
            "set_relative_P           22      1.0000\n",
            "set_recall               22      1.0000\n",
            "set_map                  22      0.0588\n",
            "set_F                    22      0.1111\n",
            "num_nonrel_judged_ret    22      15.0000\n",
            "num_q                    24      1.0000\n",
            "num_ret                  24      7.0000\n",
            "num_rel                  24      1.0000\n",
            "num_rel_ret              24      1.0000\n",
            "map                      24      1.0000\n",
            "gm_map                   24      0.0000\n",
            "Rprec                    24      1.0000\n",
            "bpref                    24      1.0000\n",
            "recip_rank               24      1.0000\n",
            "iprec_at_recall_0.00     24      1.0000\n",
            "iprec_at_recall_0.10     24      1.0000\n",
            "iprec_at_recall_0.20     24      1.0000\n",
            "iprec_at_recall_0.30     24      1.0000\n",
            "iprec_at_recall_0.40     24      1.0000\n",
            "iprec_at_recall_0.50     24      1.0000\n",
            "iprec_at_recall_0.60     24      1.0000\n",
            "iprec_at_recall_0.70     24      1.0000\n",
            "iprec_at_recall_0.80     24      1.0000\n",
            "iprec_at_recall_0.90     24      1.0000\n",
            "iprec_at_recall_1.00     24      1.0000\n",
            "P_5                      24      0.2000\n",
            "P_10                     24      0.1000\n",
            "P_15                     24      0.0667\n",
            "P_20                     24      0.0500\n",
            "P_30                     24      0.0333\n",
            "P_100                    24      0.0100\n",
            "P_200                    24      0.0050\n",
            "P_500                    24      0.0020\n",
            "P_1000                   24      0.0010\n",
            "relstring                24      0.0000\n",
            "recall_5                 24      1.0000\n",
            "recall_10                24      1.0000\n",
            "recall_15                24      1.0000\n",
            "recall_20                24      1.0000\n",
            "recall_30                24      1.0000\n",
            "recall_100               24      1.0000\n",
            "recall_200               24      1.0000\n",
            "recall_500               24      1.0000\n",
            "recall_1000              24      1.0000\n",
            "infAP                    24      1.0000\n",
            "gm_bpref                 24      0.0000\n",
            "Rprec_mult_0.20          24      1.0000\n",
            "Rprec_mult_0.40          24      1.0000\n",
            "Rprec_mult_0.60          24      1.0000\n",
            "Rprec_mult_0.80          24      1.0000\n",
            "Rprec_mult_1.00          24      1.0000\n",
            "Rprec_mult_1.20          24      0.5000\n",
            "Rprec_mult_1.40          24      0.5000\n",
            "Rprec_mult_1.60          24      0.5000\n",
            "Rprec_mult_1.80          24      0.5000\n",
            "Rprec_mult_2.00          24      0.5000\n",
            "utility                  24      -5.0000\n",
            "11pt_avg                 24      1.0000\n",
            "binG                     24      1.0000\n",
            "G                        24      1.0000\n",
            "ndcg                     24      1.0000\n",
            "ndcg_rel                 24      1.0000\n",
            "Rndcg                    24      1.0000\n",
            "ndcg_cut_5               24      1.0000\n",
            "ndcg_cut_10              24      1.0000\n",
            "ndcg_cut_15              24      1.0000\n",
            "ndcg_cut_20              24      1.0000\n",
            "ndcg_cut_30              24      1.0000\n",
            "ndcg_cut_100             24      1.0000\n",
            "ndcg_cut_200             24      1.0000\n",
            "ndcg_cut_500             24      1.0000\n",
            "ndcg_cut_1000            24      1.0000\n",
            "map_cut_5                24      1.0000\n",
            "map_cut_10               24      1.0000\n",
            "map_cut_15               24      1.0000\n",
            "map_cut_20               24      1.0000\n",
            "map_cut_30               24      1.0000\n",
            "map_cut_100              24      1.0000\n",
            "map_cut_200              24      1.0000\n",
            "map_cut_500              24      1.0000\n",
            "map_cut_1000             24      1.0000\n",
            "relative_P_5             24      1.0000\n",
            "relative_P_10            24      1.0000\n",
            "relative_P_15            24      1.0000\n",
            "relative_P_20            24      1.0000\n",
            "relative_P_30            24      1.0000\n",
            "relative_P_100           24      1.0000\n",
            "relative_P_200           24      1.0000\n",
            "relative_P_500           24      1.0000\n",
            "relative_P_1000          24      1.0000\n",
            "success_1                24      1.0000\n",
            "success_5                24      1.0000\n",
            "success_10               24      1.0000\n",
            "set_P                    24      0.1429\n",
            "set_relative_P           24      1.0000\n",
            "set_recall               24      1.0000\n",
            "set_map                  24      0.1429\n",
            "set_F                    24      0.2500\n",
            "num_nonrel_judged_ret    24      6.0000\n",
            "num_q                    26      1.0000\n",
            "num_ret                  26      25.0000\n",
            "num_rel                  26      3.0000\n",
            "num_rel_ret              26      1.0000\n",
            "map                      26      0.1111\n",
            "gm_map                   26      -2.1972\n",
            "Rprec                    26      0.3333\n",
            "bpref                    26      0.1111\n",
            "recip_rank               26      0.3333\n",
            "iprec_at_recall_0.00     26      0.3333\n",
            "iprec_at_recall_0.10     26      0.3333\n",
            "iprec_at_recall_0.20     26      0.3333\n",
            "iprec_at_recall_0.30     26      0.3333\n",
            "iprec_at_recall_0.40     26      0.0000\n",
            "iprec_at_recall_0.50     26      0.0000\n",
            "iprec_at_recall_0.60     26      0.0000\n",
            "iprec_at_recall_0.70     26      0.0000\n",
            "iprec_at_recall_0.80     26      0.0000\n",
            "iprec_at_recall_0.90     26      0.0000\n",
            "iprec_at_recall_1.00     26      0.0000\n",
            "P_5                      26      0.2000\n",
            "P_10                     26      0.1000\n",
            "P_15                     26      0.0667\n",
            "P_20                     26      0.0500\n",
            "P_30                     26      0.0333\n",
            "P_100                    26      0.0100\n",
            "P_200                    26      0.0050\n",
            "P_500                    26      0.0020\n",
            "P_1000                   26      0.0010\n",
            "relstring                26      0.0000\n",
            "recall_5                 26      0.3333\n",
            "recall_10                26      0.3333\n",
            "recall_15                26      0.3333\n",
            "recall_20                26      0.3333\n",
            "recall_30                26      0.3333\n",
            "recall_100               26      0.3333\n",
            "recall_200               26      0.3333\n",
            "recall_500               26      0.3333\n",
            "recall_1000              26      0.3333\n",
            "infAP                    26      0.1111\n",
            "gm_bpref                 26      -2.1972\n",
            "Rprec_mult_0.20          26      0.0000\n",
            "Rprec_mult_0.40          26      0.0000\n",
            "Rprec_mult_0.60          26      0.0000\n",
            "Rprec_mult_0.80          26      0.3333\n",
            "Rprec_mult_1.00          26      0.3333\n",
            "Rprec_mult_1.20          26      0.2500\n",
            "Rprec_mult_1.40          26      0.2000\n",
            "Rprec_mult_1.60          26      0.2000\n",
            "Rprec_mult_1.80          26      0.1667\n",
            "Rprec_mult_2.00          26      0.1667\n",
            "utility                  26      -23.0000\n",
            "11pt_avg                 26      0.1212\n",
            "binG                     26      0.1667\n",
            "G                        26      0.1667\n",
            "ndcg                     26      0.2346\n",
            "ndcg_rel                 26      0.2346\n",
            "Rndcg                    26      0.2346\n",
            "ndcg_cut_5               26      0.2346\n",
            "ndcg_cut_10              26      0.2346\n",
            "ndcg_cut_15              26      0.2346\n",
            "ndcg_cut_20              26      0.2346\n",
            "ndcg_cut_30              26      0.2346\n",
            "ndcg_cut_100             26      0.2346\n",
            "ndcg_cut_200             26      0.2346\n",
            "ndcg_cut_500             26      0.2346\n",
            "ndcg_cut_1000            26      0.2346\n",
            "map_cut_5                26      0.1111\n",
            "map_cut_10               26      0.1111\n",
            "map_cut_15               26      0.1111\n",
            "map_cut_20               26      0.1111\n",
            "map_cut_30               26      0.1111\n",
            "map_cut_100              26      0.1111\n",
            "map_cut_200              26      0.1111\n",
            "map_cut_500              26      0.1111\n",
            "map_cut_1000             26      0.1111\n",
            "relative_P_5             26      0.3333\n",
            "relative_P_10            26      0.3333\n",
            "relative_P_15            26      0.3333\n",
            "relative_P_20            26      0.3333\n",
            "relative_P_30            26      0.3333\n",
            "relative_P_100           26      0.3333\n",
            "relative_P_200           26      0.3333\n",
            "relative_P_500           26      0.3333\n",
            "relative_P_1000          26      0.3333\n",
            "success_1                26      0.0000\n",
            "success_5                26      1.0000\n",
            "success_10               26      1.0000\n",
            "set_P                    26      0.0400\n",
            "set_relative_P           26      0.3333\n",
            "set_recall               26      0.3333\n",
            "set_map                  26      0.0133\n",
            "set_F                    26      0.0714\n",
            "num_nonrel_judged_ret    26      22.0000\n",
            "num_q                    28      1.0000\n",
            "num_ret                  28      22.0000\n",
            "num_rel                  28      2.0000\n",
            "num_rel_ret              28      0.0000\n",
            "map                      28      0.0000\n",
            "gm_map                   28      -11.5129\n",
            "Rprec                    28      0.0000\n",
            "bpref                    28      0.0000\n",
            "recip_rank               28      0.0000\n",
            "iprec_at_recall_0.00     28      0.0000\n",
            "iprec_at_recall_0.10     28      0.0000\n",
            "iprec_at_recall_0.20     28      0.0000\n",
            "iprec_at_recall_0.30     28      0.0000\n",
            "iprec_at_recall_0.40     28      0.0000\n",
            "iprec_at_recall_0.50     28      0.0000\n",
            "iprec_at_recall_0.60     28      0.0000\n",
            "iprec_at_recall_0.70     28      0.0000\n",
            "iprec_at_recall_0.80     28      0.0000\n",
            "iprec_at_recall_0.90     28      0.0000\n",
            "iprec_at_recall_1.00     28      0.0000\n",
            "P_5                      28      0.0000\n",
            "P_10                     28      0.0000\n",
            "P_15                     28      0.0000\n",
            "P_20                     28      0.0000\n",
            "P_30                     28      0.0000\n",
            "P_100                    28      0.0000\n",
            "P_200                    28      0.0000\n",
            "P_500                    28      0.0000\n",
            "P_1000                   28      0.0000\n",
            "relstring                28      0.0000\n",
            "recall_5                 28      0.0000\n",
            "recall_10                28      0.0000\n",
            "recall_15                28      0.0000\n",
            "recall_20                28      0.0000\n",
            "recall_30                28      0.0000\n",
            "recall_100               28      0.0000\n",
            "recall_200               28      0.0000\n",
            "recall_500               28      0.0000\n",
            "recall_1000              28      0.0000\n",
            "infAP                    28      0.0000\n",
            "gm_bpref                 28      -11.5129\n",
            "Rprec_mult_0.20          28      0.0000\n",
            "Rprec_mult_0.40          28      0.0000\n",
            "Rprec_mult_0.60          28      0.0000\n",
            "Rprec_mult_0.80          28      0.0000\n",
            "Rprec_mult_1.00          28      0.0000\n",
            "Rprec_mult_1.20          28      0.0000\n",
            "Rprec_mult_1.40          28      0.0000\n",
            "Rprec_mult_1.60          28      0.0000\n",
            "Rprec_mult_1.80          28      0.0000\n",
            "Rprec_mult_2.00          28      0.0000\n",
            "utility                  28      -22.0000\n",
            "11pt_avg                 28      0.0000\n",
            "binG                     28      0.0000\n",
            "G                        28      0.0000\n",
            "ndcg                     28      0.0000\n",
            "ndcg_rel                 28      0.0000\n",
            "Rndcg                    28      0.0000\n",
            "ndcg_cut_5               28      0.0000\n",
            "ndcg_cut_10              28      0.0000\n",
            "ndcg_cut_15              28      0.0000\n",
            "ndcg_cut_20              28      0.0000\n",
            "ndcg_cut_30              28      0.0000\n",
            "ndcg_cut_100             28      0.0000\n",
            "ndcg_cut_200             28      0.0000\n",
            "ndcg_cut_500             28      0.0000\n",
            "ndcg_cut_1000            28      0.0000\n",
            "map_cut_5                28      0.0000\n",
            "map_cut_10               28      0.0000\n",
            "map_cut_15               28      0.0000\n",
            "map_cut_20               28      0.0000\n",
            "map_cut_30               28      0.0000\n",
            "map_cut_100              28      0.0000\n",
            "map_cut_200              28      0.0000\n",
            "map_cut_500              28      0.0000\n",
            "map_cut_1000             28      0.0000\n",
            "relative_P_5             28      0.0000\n",
            "relative_P_10            28      0.0000\n",
            "relative_P_15            28      0.0000\n",
            "relative_P_20            28      0.0000\n",
            "relative_P_30            28      0.0000\n",
            "relative_P_100           28      0.0000\n",
            "relative_P_200           28      0.0000\n",
            "relative_P_500           28      0.0000\n",
            "relative_P_1000          28      0.0000\n",
            "success_1                28      0.0000\n",
            "success_5                28      0.0000\n",
            "success_10               28      0.0000\n",
            "set_P                    28      0.0000\n",
            "set_relative_P           28      0.0000\n",
            "set_recall               28      0.0000\n",
            "set_map                  28      0.0000\n",
            "set_F                    28      0.0000\n",
            "num_nonrel_judged_ret    28      21.0000\n",
            "num_q                    all     14.0000\n",
            "num_ret                  all     151.0000\n",
            "num_rel                  all     33.0000\n",
            "num_rel_ret              all     7.0000\n",
            "map                      all     0.1971\n",
            "gm_map                   all     0.0015\n",
            "Rprec                    all     0.1667\n",
            "bpref                    all     0.1508\n",
            "recip_rank               all     0.2196\n",
            "iprec_at_recall_0.00     all     0.2196\n",
            "iprec_at_recall_0.10     all     0.2196\n",
            "iprec_at_recall_0.20     all     0.2196\n",
            "iprec_at_recall_0.30     all     0.2107\n",
            "iprec_at_recall_0.40     all     0.1869\n",
            "iprec_at_recall_0.50     all     0.1869\n",
            "iprec_at_recall_0.60     all     0.1869\n",
            "iprec_at_recall_0.70     all     0.1869\n",
            "iprec_at_recall_0.80     all     0.1869\n",
            "iprec_at_recall_0.90     all     0.1869\n",
            "iprec_at_recall_1.00     all     0.1869\n",
            "P_5                      all     0.0714\n",
            "P_10                     all     0.0500\n",
            "P_15                     all     0.0333\n",
            "P_20                     all     0.0250\n",
            "P_30                     all     0.0167\n",
            "P_100                    all     0.0050\n",
            "P_200                    all     0.0025\n",
            "P_500                    all     0.0010\n",
            "P_1000                   all     0.0005\n",
            "relstring                all     0.0000\n",
            "recall_5                 all     0.3095\n",
            "recall_10                all     0.3988\n",
            "recall_15                all     0.3988\n",
            "recall_20                all     0.3988\n",
            "recall_30                all     0.3988\n",
            "recall_100               all     0.3988\n",
            "recall_200               all     0.3988\n",
            "recall_500               all     0.3988\n",
            "recall_1000              all     0.3988\n",
            "infAP                    all     0.1971\n",
            "gm_bpref                 all     0.0001\n",
            "Rprec_mult_0.20          all     0.1429\n",
            "Rprec_mult_0.40          all     0.1429\n",
            "Rprec_mult_0.60          all     0.1429\n",
            "Rprec_mult_0.80          all     0.1667\n",
            "Rprec_mult_1.00          all     0.1667\n",
            "Rprec_mult_1.20          all     0.0893\n",
            "Rprec_mult_1.40          all     0.0857\n",
            "Rprec_mult_1.60          all     0.0857\n",
            "Rprec_mult_1.80          all     0.0923\n",
            "Rprec_mult_2.00          all     0.0923\n",
            "utility                  all     -9.7857\n",
            "11pt_avg                 all     0.1980\n",
            "binG                     all     0.2442\n",
            "G                        all     0.2442\n",
            "ndcg                     all     0.2523\n",
            "ndcg_rel                 all     0.2523\n",
            "Rndcg                    all     0.2059\n",
            "ndcg_cut_5               all     0.2180\n",
            "ndcg_cut_10              all     0.2523\n",
            "ndcg_cut_15              all     0.2523\n",
            "ndcg_cut_20              all     0.2523\n",
            "ndcg_cut_30              all     0.2523\n",
            "ndcg_cut_100             all     0.2523\n",
            "ndcg_cut_200             all     0.2523\n",
            "ndcg_cut_500             all     0.2523\n",
            "ndcg_cut_1000            all     0.2523\n",
            "map_cut_5                all     0.1829\n",
            "map_cut_10               all     0.1971\n",
            "map_cut_15               all     0.1971\n",
            "map_cut_20               all     0.1971\n",
            "map_cut_30               all     0.1971\n",
            "map_cut_100              all     0.1971\n",
            "map_cut_200              all     0.1971\n",
            "map_cut_500              all     0.1971\n",
            "map_cut_1000             all     0.1971\n",
            "relative_P_5             all     0.3095\n",
            "relative_P_10            all     0.3988\n",
            "relative_P_15            all     0.3988\n",
            "relative_P_20            all     0.3988\n",
            "relative_P_30            all     0.3988\n",
            "relative_P_100           all     0.3988\n",
            "relative_P_200           all     0.3988\n",
            "relative_P_500           all     0.3988\n",
            "relative_P_1000          all     0.3988\n",
            "success_1                all     0.1429\n",
            "success_5                all     0.3571\n",
            "success_10               all     0.5000\n",
            "set_P                    all     0.0421\n",
            "set_relative_P           all     0.3988\n",
            "set_recall               all     0.3988\n",
            "set_map                  all     0.0357\n",
            "set_F                    all     0.0740\n",
            "num_nonrel_judged_ret    all     126.0000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N_e8DZ3K39W0"
      },
      "source": [
        "## Part 2: Improving performance"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yD0gzOt239W1"
      },
      "source": [
        "Look at where the baseline Whoosh system did well, or badly:  \n",
        "For topic 28: Early Childhood Education, the relevant document G00-02-0541868(false negative) should be ranked highly but it is not in the return document list, because each word in the query appears in that document in lower case, but the analyzer did not change upper case to lower case and searcher treats upper case and lower case differently. And the document G00-75-2371200(false positive) got a high score because most of the words are in upper case in that document and all three words in the query appear multiple times in that document."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ul4KyS4aQzRF"
      },
      "source": [
        "def printRelName(topicFile, qrelsFile, queryParser, searcher, id):\n",
        "  with open(topicFile, \"r\") as tf:\n",
        "        topics = tf.read().splitlines()\n",
        "  for topic in topics:\n",
        "        topic_id, topic_phrase = tuple(topic.split(\" \", 1))\n",
        "        if topic_id == id:\n",
        "          print(\"---------------------------Topic_id and Topic_phrase----------------------------------\")\n",
        "          print(topic_id, topic_phrase)\n",
        "          topicQuery = queryParser.parse(topic_phrase)\n",
        "          topicResults = searcher.search(topicQuery, limit=None)\n",
        "          print(\"---------------------------Return documents----------------------------------\")\n",
        "          for (docnum, result) in enumerate(topicResults):\n",
        "              score = topicResults.score(docnum)\n",
        "              print(\"%s Q0 %s %d %lf test\" % (topic_id, os.path.basename(result[\"file_path\"]), docnum, score))\n",
        "          print(\"---------------------------Relevant documents----------------------------------\")\n",
        "          with open(qrelsFile, 'r') as f_qrel:\n",
        "            qrels = f_qrel.readlines()\n",
        "            for i in qrels:\n",
        "              qid, _, doc, rel = i.rstrip().split(\" \")\n",
        "              if qid == id and rel == \"1\":\n",
        "                print(i.rstrip())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2blPXfPaQ7JY",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 521
        },
        "outputId": "d739a2ec-5f96-458b-a0bd-28228b7171f8"
      },
      "source": [
        "printRelName(topic_file, qrels_file, QP_Q2, SEARCHER_Q2, \"28\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "---------------------------Topic_id and Topic_phrase----------------------------------\n",
            "28 Early Childhood Education\n",
            "---------------------------Return documents----------------------------------\n",
            "28 Q0 G00-75-2371200 0 24.505669 test\n",
            "28 Q0 G00-93-3702508 1 21.152986 test\n",
            "28 Q0 G00-99-2279811 2 20.492992 test\n",
            "28 Q0 G00-30-2788847 3 20.139545 test\n",
            "28 Q0 G00-31-0429249 4 19.988408 test\n",
            "28 Q0 G00-61-3894960 5 19.960652 test\n",
            "28 Q0 G00-48-1527977 6 19.960652 test\n",
            "28 Q0 G00-93-4160214 7 19.960652 test\n",
            "28 Q0 G00-78-2978026 8 19.848213 test\n",
            "28 Q0 G00-28-3705847 9 19.793504 test\n",
            "28 Q0 G00-50-3231467 10 19.318859 test\n",
            "28 Q0 G00-74-2972556 11 19.244841 test\n",
            "28 Q0 G00-77-3295130 12 19.031552 test\n",
            "28 Q0 G00-91-3997333 13 18.808227 test\n",
            "28 Q0 G00-93-1203370 14 18.558858 test\n",
            "28 Q0 G00-82-0211909 15 18.217313 test\n",
            "28 Q0 G00-16-2494170 16 17.101565 test\n",
            "28 Q0 G00-09-4172401 17 16.713747 test\n",
            "28 Q0 G00-27-2159399 18 13.352770 test\n",
            "28 Q0 G00-82-3144058 19 12.296384 test\n",
            "28 Q0 G00-65-4078383 20 11.960488 test\n",
            "28 Q0 G00-39-3477058 21 11.365612 test\n",
            "---------------------------Relevant documents----------------------------------\n",
            "28 0 G00-02-0541868 1\n",
            "28 0 G00-54-2576117 1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tzwPVMgc39W1"
      },
      "source": [
        "Based on the analysis, improve the baseline."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mAef6qiu39W2"
      },
      "source": [
        "# Put your code for creating the index here (you can add more cells).\n",
        "# Make sure you save the final index in the variable INDEX_Q3, your query parser in QP_Q3, and your searcher in SEARCHER_Q3\n",
        "LwrAnalyzer = RegexTokenizer() | LowercaseFilter() \n",
        "mySchema2 = Schema(file_path = ID(stored=True), file_content = TEXT(analyzer = LwrAnalyzer))\n",
        "myIndex2 = createIndex(mySchema2)\n",
        "addFilesToIndex(myIndex2, filesToIndex)\n",
        "myQueryParser2 = QueryParser(\"file_content\", schema=myIndex2.schema)\n",
        "mySearcher2 = myIndex2.searcher(weighting=BM25F(B=0.7,K1=1))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_H6yUwtd39W4"
      },
      "source": [
        "INDEX_Q3 = myIndex2 # Replace None with your index for Q3\n",
        "QP_Q3 = myQueryParser2 # Replace None with your query parser for Q3\n",
        "SEARCHER_Q3 = mySearcher2 # Replace None with your searcher for Q3"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mgjBlLosb85P",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "69215962-1855-463d-c53f-6aea3792c3e9"
      },
      "source": [
        "pyTrecEval(topic_file, qrels_file, QP_Q3, SEARCHER_Q3) "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "num_q                    1       1.0000\n",
            "num_ret                  1       3.0000\n",
            "num_rel                  1       5.0000\n",
            "num_rel_ret              1       0.0000\n",
            "map                      1       0.0000\n",
            "gm_map                   1       -11.5129\n",
            "Rprec                    1       0.0000\n",
            "bpref                    1       0.0000\n",
            "recip_rank               1       0.0000\n",
            "iprec_at_recall_0.00     1       0.0000\n",
            "iprec_at_recall_0.10     1       0.0000\n",
            "iprec_at_recall_0.20     1       0.0000\n",
            "iprec_at_recall_0.30     1       0.0000\n",
            "iprec_at_recall_0.40     1       0.0000\n",
            "iprec_at_recall_0.50     1       0.0000\n",
            "iprec_at_recall_0.60     1       0.0000\n",
            "iprec_at_recall_0.70     1       0.0000\n",
            "iprec_at_recall_0.80     1       0.0000\n",
            "iprec_at_recall_0.90     1       0.0000\n",
            "iprec_at_recall_1.00     1       0.0000\n",
            "P_5                      1       0.0000\n",
            "P_10                     1       0.0000\n",
            "P_15                     1       0.0000\n",
            "P_20                     1       0.0000\n",
            "P_30                     1       0.0000\n",
            "P_100                    1       0.0000\n",
            "P_200                    1       0.0000\n",
            "P_500                    1       0.0000\n",
            "P_1000                   1       0.0000\n",
            "relstring                1       0.0000\n",
            "recall_5                 1       0.0000\n",
            "recall_10                1       0.0000\n",
            "recall_15                1       0.0000\n",
            "recall_20                1       0.0000\n",
            "recall_30                1       0.0000\n",
            "recall_100               1       0.0000\n",
            "recall_200               1       0.0000\n",
            "recall_500               1       0.0000\n",
            "recall_1000              1       0.0000\n",
            "infAP                    1       0.0000\n",
            "gm_bpref                 1       -11.5129\n",
            "Rprec_mult_0.20          1       0.0000\n",
            "Rprec_mult_0.40          1       0.0000\n",
            "Rprec_mult_0.60          1       0.0000\n",
            "Rprec_mult_0.80          1       0.0000\n",
            "Rprec_mult_1.00          1       0.0000\n",
            "Rprec_mult_1.20          1       0.0000\n",
            "Rprec_mult_1.40          1       0.0000\n",
            "Rprec_mult_1.60          1       0.0000\n",
            "Rprec_mult_1.80          1       0.0000\n",
            "Rprec_mult_2.00          1       0.0000\n",
            "utility                  1       -3.0000\n",
            "11pt_avg                 1       0.0000\n",
            "binG                     1       0.0000\n",
            "G                        1       0.0000\n",
            "ndcg                     1       0.0000\n",
            "ndcg_rel                 1       0.0000\n",
            "Rndcg                    1       0.0000\n",
            "ndcg_cut_5               1       0.0000\n",
            "ndcg_cut_10              1       0.0000\n",
            "ndcg_cut_15              1       0.0000\n",
            "ndcg_cut_20              1       0.0000\n",
            "ndcg_cut_30              1       0.0000\n",
            "ndcg_cut_100             1       0.0000\n",
            "ndcg_cut_200             1       0.0000\n",
            "ndcg_cut_500             1       0.0000\n",
            "ndcg_cut_1000            1       0.0000\n",
            "map_cut_5                1       0.0000\n",
            "map_cut_10               1       0.0000\n",
            "map_cut_15               1       0.0000\n",
            "map_cut_20               1       0.0000\n",
            "map_cut_30               1       0.0000\n",
            "map_cut_100              1       0.0000\n",
            "map_cut_200              1       0.0000\n",
            "map_cut_500              1       0.0000\n",
            "map_cut_1000             1       0.0000\n",
            "relative_P_5             1       0.0000\n",
            "relative_P_10            1       0.0000\n",
            "relative_P_15            1       0.0000\n",
            "relative_P_20            1       0.0000\n",
            "relative_P_30            1       0.0000\n",
            "relative_P_100           1       0.0000\n",
            "relative_P_200           1       0.0000\n",
            "relative_P_500           1       0.0000\n",
            "relative_P_1000          1       0.0000\n",
            "success_1                1       0.0000\n",
            "success_5                1       0.0000\n",
            "success_10               1       0.0000\n",
            "set_P                    1       0.0000\n",
            "set_relative_P           1       0.0000\n",
            "set_recall               1       0.0000\n",
            "set_map                  1       0.0000\n",
            "set_F                    1       0.0000\n",
            "num_nonrel_judged_ret    1       3.0000\n",
            "num_q                    2       1.0000\n",
            "num_ret                  2       13.0000\n",
            "num_rel                  2       2.0000\n",
            "num_rel_ret              2       1.0000\n",
            "map                      2       0.2500\n",
            "gm_map                   2       -1.3863\n",
            "Rprec                    2       0.5000\n",
            "bpref                    2       0.2500\n",
            "recip_rank               2       0.5000\n",
            "iprec_at_recall_0.00     2       0.5000\n",
            "iprec_at_recall_0.10     2       0.5000\n",
            "iprec_at_recall_0.20     2       0.5000\n",
            "iprec_at_recall_0.30     2       0.5000\n",
            "iprec_at_recall_0.40     2       0.5000\n",
            "iprec_at_recall_0.50     2       0.5000\n",
            "iprec_at_recall_0.60     2       0.0000\n",
            "iprec_at_recall_0.70     2       0.0000\n",
            "iprec_at_recall_0.80     2       0.0000\n",
            "iprec_at_recall_0.90     2       0.0000\n",
            "iprec_at_recall_1.00     2       0.0000\n",
            "P_5                      2       0.2000\n",
            "P_10                     2       0.1000\n",
            "P_15                     2       0.0667\n",
            "P_20                     2       0.0500\n",
            "P_30                     2       0.0333\n",
            "P_100                    2       0.0100\n",
            "P_200                    2       0.0050\n",
            "P_500                    2       0.0020\n",
            "P_1000                   2       0.0010\n",
            "relstring                2       0.0000\n",
            "recall_5                 2       0.5000\n",
            "recall_10                2       0.5000\n",
            "recall_15                2       0.5000\n",
            "recall_20                2       0.5000\n",
            "recall_30                2       0.5000\n",
            "recall_100               2       0.5000\n",
            "recall_200               2       0.5000\n",
            "recall_500               2       0.5000\n",
            "recall_1000              2       0.5000\n",
            "infAP                    2       0.2500\n",
            "gm_bpref                 2       -1.3863\n",
            "Rprec_mult_0.20          2       0.0000\n",
            "Rprec_mult_0.40          2       0.0000\n",
            "Rprec_mult_0.60          2       0.5000\n",
            "Rprec_mult_0.80          2       0.5000\n",
            "Rprec_mult_1.00          2       0.5000\n",
            "Rprec_mult_1.20          2       0.3333\n",
            "Rprec_mult_1.40          2       0.3333\n",
            "Rprec_mult_1.60          2       0.2500\n",
            "Rprec_mult_1.80          2       0.2500\n",
            "Rprec_mult_2.00          2       0.2500\n",
            "utility                  2       -11.0000\n",
            "11pt_avg                 2       0.2727\n",
            "binG                     2       0.3155\n",
            "G                        2       0.3155\n",
            "ndcg                     2       0.3869\n",
            "ndcg_rel                 2       0.3869\n",
            "Rndcg                    2       0.3869\n",
            "ndcg_cut_5               2       0.3869\n",
            "ndcg_cut_10              2       0.3869\n",
            "ndcg_cut_15              2       0.3869\n",
            "ndcg_cut_20              2       0.3869\n",
            "ndcg_cut_30              2       0.3869\n",
            "ndcg_cut_100             2       0.3869\n",
            "ndcg_cut_200             2       0.3869\n",
            "ndcg_cut_500             2       0.3869\n",
            "ndcg_cut_1000            2       0.3869\n",
            "map_cut_5                2       0.2500\n",
            "map_cut_10               2       0.2500\n",
            "map_cut_15               2       0.2500\n",
            "map_cut_20               2       0.2500\n",
            "map_cut_30               2       0.2500\n",
            "map_cut_100              2       0.2500\n",
            "map_cut_200              2       0.2500\n",
            "map_cut_500              2       0.2500\n",
            "map_cut_1000             2       0.2500\n",
            "relative_P_5             2       0.5000\n",
            "relative_P_10            2       0.5000\n",
            "relative_P_15            2       0.5000\n",
            "relative_P_20            2       0.5000\n",
            "relative_P_30            2       0.5000\n",
            "relative_P_100           2       0.5000\n",
            "relative_P_200           2       0.5000\n",
            "relative_P_500           2       0.5000\n",
            "relative_P_1000          2       0.5000\n",
            "success_1                2       0.0000\n",
            "success_5                2       1.0000\n",
            "success_10               2       1.0000\n",
            "set_P                    2       0.0769\n",
            "set_relative_P           2       0.5000\n",
            "set_recall               2       0.5000\n",
            "set_map                  2       0.0385\n",
            "set_F                    2       0.1333\n",
            "num_nonrel_judged_ret    2       9.0000\n",
            "num_q                    4       1.0000\n",
            "num_ret                  4       32.0000\n",
            "num_rel                  4       4.0000\n",
            "num_rel_ret              4       2.0000\n",
            "map                      4       0.5000\n",
            "gm_map                   4       -0.6931\n",
            "Rprec                    4       0.5000\n",
            "bpref                    4       0.5000\n",
            "recip_rank               4       1.0000\n",
            "iprec_at_recall_0.00     4       1.0000\n",
            "iprec_at_recall_0.10     4       1.0000\n",
            "iprec_at_recall_0.20     4       1.0000\n",
            "iprec_at_recall_0.30     4       1.0000\n",
            "iprec_at_recall_0.40     4       1.0000\n",
            "iprec_at_recall_0.50     4       1.0000\n",
            "iprec_at_recall_0.60     4       0.0000\n",
            "iprec_at_recall_0.70     4       0.0000\n",
            "iprec_at_recall_0.80     4       0.0000\n",
            "iprec_at_recall_0.90     4       0.0000\n",
            "iprec_at_recall_1.00     4       0.0000\n",
            "P_5                      4       0.4000\n",
            "P_10                     4       0.2000\n",
            "P_15                     4       0.1333\n",
            "P_20                     4       0.1000\n",
            "P_30                     4       0.0667\n",
            "P_100                    4       0.0200\n",
            "P_200                    4       0.0100\n",
            "P_500                    4       0.0040\n",
            "P_1000                   4       0.0020\n",
            "relstring                4       0.0000\n",
            "recall_5                 4       0.5000\n",
            "recall_10                4       0.5000\n",
            "recall_15                4       0.5000\n",
            "recall_20                4       0.5000\n",
            "recall_30                4       0.5000\n",
            "recall_100               4       0.5000\n",
            "recall_200               4       0.5000\n",
            "recall_500               4       0.5000\n",
            "recall_1000              4       0.5000\n",
            "infAP                    4       0.5000\n",
            "gm_bpref                 4       -0.6931\n",
            "Rprec_mult_0.20          4       1.0000\n",
            "Rprec_mult_0.40          4       1.0000\n",
            "Rprec_mult_0.60          4       0.6667\n",
            "Rprec_mult_0.80          4       0.5000\n",
            "Rprec_mult_1.00          4       0.5000\n",
            "Rprec_mult_1.20          4       0.4000\n",
            "Rprec_mult_1.40          4       0.3333\n",
            "Rprec_mult_1.60          4       0.2857\n",
            "Rprec_mult_1.80          4       0.2500\n",
            "Rprec_mult_2.00          4       0.2500\n",
            "utility                  4       -28.0000\n",
            "11pt_avg                 4       0.5455\n",
            "binG                     4       0.5000\n",
            "G                        4       0.5000\n",
            "ndcg                     4       0.6367\n",
            "ndcg_rel                 4       0.8183\n",
            "Rndcg                    4       0.6367\n",
            "ndcg_cut_5               4       0.6367\n",
            "ndcg_cut_10              4       0.6367\n",
            "ndcg_cut_15              4       0.6367\n",
            "ndcg_cut_20              4       0.6367\n",
            "ndcg_cut_30              4       0.6367\n",
            "ndcg_cut_100             4       0.6367\n",
            "ndcg_cut_200             4       0.6367\n",
            "ndcg_cut_500             4       0.6367\n",
            "ndcg_cut_1000            4       0.6367\n",
            "map_cut_5                4       0.5000\n",
            "map_cut_10               4       0.5000\n",
            "map_cut_15               4       0.5000\n",
            "map_cut_20               4       0.5000\n",
            "map_cut_30               4       0.5000\n",
            "map_cut_100              4       0.5000\n",
            "map_cut_200              4       0.5000\n",
            "map_cut_500              4       0.5000\n",
            "map_cut_1000             4       0.5000\n",
            "relative_P_5             4       0.5000\n",
            "relative_P_10            4       0.5000\n",
            "relative_P_15            4       0.5000\n",
            "relative_P_20            4       0.5000\n",
            "relative_P_30            4       0.5000\n",
            "relative_P_100           4       0.5000\n",
            "relative_P_200           4       0.5000\n",
            "relative_P_500           4       0.5000\n",
            "relative_P_1000          4       0.5000\n",
            "success_1                4       1.0000\n",
            "success_5                4       1.0000\n",
            "success_10               4       1.0000\n",
            "set_P                    4       0.0625\n",
            "set_relative_P           4       0.5000\n",
            "set_recall               4       0.5000\n",
            "set_map                  4       0.0312\n",
            "set_F                    4       0.1111\n",
            "num_nonrel_judged_ret    4       23.0000\n",
            "num_q                    6       1.0000\n",
            "num_ret                  6       9.0000\n",
            "num_rel                  6       1.0000\n",
            "num_rel_ret              6       0.0000\n",
            "map                      6       0.0000\n",
            "gm_map                   6       -11.5129\n",
            "Rprec                    6       0.0000\n",
            "bpref                    6       0.0000\n",
            "recip_rank               6       0.0000\n",
            "iprec_at_recall_0.00     6       0.0000\n",
            "iprec_at_recall_0.10     6       0.0000\n",
            "iprec_at_recall_0.20     6       0.0000\n",
            "iprec_at_recall_0.30     6       0.0000\n",
            "iprec_at_recall_0.40     6       0.0000\n",
            "iprec_at_recall_0.50     6       0.0000\n",
            "iprec_at_recall_0.60     6       0.0000\n",
            "iprec_at_recall_0.70     6       0.0000\n",
            "iprec_at_recall_0.80     6       0.0000\n",
            "iprec_at_recall_0.90     6       0.0000\n",
            "iprec_at_recall_1.00     6       0.0000\n",
            "P_5                      6       0.0000\n",
            "P_10                     6       0.0000\n",
            "P_15                     6       0.0000\n",
            "P_20                     6       0.0000\n",
            "P_30                     6       0.0000\n",
            "P_100                    6       0.0000\n",
            "P_200                    6       0.0000\n",
            "P_500                    6       0.0000\n",
            "P_1000                   6       0.0000\n",
            "relstring                6       0.0000\n",
            "recall_5                 6       0.0000\n",
            "recall_10                6       0.0000\n",
            "recall_15                6       0.0000\n",
            "recall_20                6       0.0000\n",
            "recall_30                6       0.0000\n",
            "recall_100               6       0.0000\n",
            "recall_200               6       0.0000\n",
            "recall_500               6       0.0000\n",
            "recall_1000              6       0.0000\n",
            "infAP                    6       0.0000\n",
            "gm_bpref                 6       -11.5129\n",
            "Rprec_mult_0.20          6       0.0000\n",
            "Rprec_mult_0.40          6       0.0000\n",
            "Rprec_mult_0.60          6       0.0000\n",
            "Rprec_mult_0.80          6       0.0000\n",
            "Rprec_mult_1.00          6       0.0000\n",
            "Rprec_mult_1.20          6       0.0000\n",
            "Rprec_mult_1.40          6       0.0000\n",
            "Rprec_mult_1.60          6       0.0000\n",
            "Rprec_mult_1.80          6       0.0000\n",
            "Rprec_mult_2.00          6       0.0000\n",
            "utility                  6       -9.0000\n",
            "11pt_avg                 6       0.0000\n",
            "binG                     6       0.0000\n",
            "G                        6       0.0000\n",
            "ndcg                     6       0.0000\n",
            "ndcg_rel                 6       0.0000\n",
            "Rndcg                    6       0.0000\n",
            "ndcg_cut_5               6       0.0000\n",
            "ndcg_cut_10              6       0.0000\n",
            "ndcg_cut_15              6       0.0000\n",
            "ndcg_cut_20              6       0.0000\n",
            "ndcg_cut_30              6       0.0000\n",
            "ndcg_cut_100             6       0.0000\n",
            "ndcg_cut_200             6       0.0000\n",
            "ndcg_cut_500             6       0.0000\n",
            "ndcg_cut_1000            6       0.0000\n",
            "map_cut_5                6       0.0000\n",
            "map_cut_10               6       0.0000\n",
            "map_cut_15               6       0.0000\n",
            "map_cut_20               6       0.0000\n",
            "map_cut_30               6       0.0000\n",
            "map_cut_100              6       0.0000\n",
            "map_cut_200              6       0.0000\n",
            "map_cut_500              6       0.0000\n",
            "map_cut_1000             6       0.0000\n",
            "relative_P_5             6       0.0000\n",
            "relative_P_10            6       0.0000\n",
            "relative_P_15            6       0.0000\n",
            "relative_P_20            6       0.0000\n",
            "relative_P_30            6       0.0000\n",
            "relative_P_100           6       0.0000\n",
            "relative_P_200           6       0.0000\n",
            "relative_P_500           6       0.0000\n",
            "relative_P_1000          6       0.0000\n",
            "success_1                6       0.0000\n",
            "success_5                6       0.0000\n",
            "success_10               6       0.0000\n",
            "set_P                    6       0.0000\n",
            "set_relative_P           6       0.0000\n",
            "set_recall               6       0.0000\n",
            "set_map                  6       0.0000\n",
            "set_F                    6       0.0000\n",
            "num_nonrel_judged_ret    6       8.0000\n",
            "num_q                    7       1.0000\n",
            "num_ret                  7       14.0000\n",
            "num_rel                  7       3.0000\n",
            "num_rel_ret              7       0.0000\n",
            "map                      7       0.0000\n",
            "gm_map                   7       -11.5129\n",
            "Rprec                    7       0.0000\n",
            "bpref                    7       0.0000\n",
            "recip_rank               7       0.0000\n",
            "iprec_at_recall_0.00     7       0.0000\n",
            "iprec_at_recall_0.10     7       0.0000\n",
            "iprec_at_recall_0.20     7       0.0000\n",
            "iprec_at_recall_0.30     7       0.0000\n",
            "iprec_at_recall_0.40     7       0.0000\n",
            "iprec_at_recall_0.50     7       0.0000\n",
            "iprec_at_recall_0.60     7       0.0000\n",
            "iprec_at_recall_0.70     7       0.0000\n",
            "iprec_at_recall_0.80     7       0.0000\n",
            "iprec_at_recall_0.90     7       0.0000\n",
            "iprec_at_recall_1.00     7       0.0000\n",
            "P_5                      7       0.0000\n",
            "P_10                     7       0.0000\n",
            "P_15                     7       0.0000\n",
            "P_20                     7       0.0000\n",
            "P_30                     7       0.0000\n",
            "P_100                    7       0.0000\n",
            "P_200                    7       0.0000\n",
            "P_500                    7       0.0000\n",
            "P_1000                   7       0.0000\n",
            "relstring                7       0.0000\n",
            "recall_5                 7       0.0000\n",
            "recall_10                7       0.0000\n",
            "recall_15                7       0.0000\n",
            "recall_20                7       0.0000\n",
            "recall_30                7       0.0000\n",
            "recall_100               7       0.0000\n",
            "recall_200               7       0.0000\n",
            "recall_500               7       0.0000\n",
            "recall_1000              7       0.0000\n",
            "infAP                    7       0.0000\n",
            "gm_bpref                 7       -11.5129\n",
            "Rprec_mult_0.20          7       0.0000\n",
            "Rprec_mult_0.40          7       0.0000\n",
            "Rprec_mult_0.60          7       0.0000\n",
            "Rprec_mult_0.80          7       0.0000\n",
            "Rprec_mult_1.00          7       0.0000\n",
            "Rprec_mult_1.20          7       0.0000\n",
            "Rprec_mult_1.40          7       0.0000\n",
            "Rprec_mult_1.60          7       0.0000\n",
            "Rprec_mult_1.80          7       0.0000\n",
            "Rprec_mult_2.00          7       0.0000\n",
            "utility                  7       -14.0000\n",
            "11pt_avg                 7       0.0000\n",
            "binG                     7       0.0000\n",
            "G                        7       0.0000\n",
            "ndcg                     7       0.0000\n",
            "ndcg_rel                 7       0.0000\n",
            "Rndcg                    7       0.0000\n",
            "ndcg_cut_5               7       0.0000\n",
            "ndcg_cut_10              7       0.0000\n",
            "ndcg_cut_15              7       0.0000\n",
            "ndcg_cut_20              7       0.0000\n",
            "ndcg_cut_30              7       0.0000\n",
            "ndcg_cut_100             7       0.0000\n",
            "ndcg_cut_200             7       0.0000\n",
            "ndcg_cut_500             7       0.0000\n",
            "ndcg_cut_1000            7       0.0000\n",
            "map_cut_5                7       0.0000\n",
            "map_cut_10               7       0.0000\n",
            "map_cut_15               7       0.0000\n",
            "map_cut_20               7       0.0000\n",
            "map_cut_30               7       0.0000\n",
            "map_cut_100              7       0.0000\n",
            "map_cut_200              7       0.0000\n",
            "map_cut_500              7       0.0000\n",
            "map_cut_1000             7       0.0000\n",
            "relative_P_5             7       0.0000\n",
            "relative_P_10            7       0.0000\n",
            "relative_P_15            7       0.0000\n",
            "relative_P_20            7       0.0000\n",
            "relative_P_30            7       0.0000\n",
            "relative_P_100           7       0.0000\n",
            "relative_P_200           7       0.0000\n",
            "relative_P_500           7       0.0000\n",
            "relative_P_1000          7       0.0000\n",
            "success_1                7       0.0000\n",
            "success_5                7       0.0000\n",
            "success_10               7       0.0000\n",
            "set_P                    7       0.0000\n",
            "set_relative_P           7       0.0000\n",
            "set_recall               7       0.0000\n",
            "set_map                  7       0.0000\n",
            "set_F                    7       0.0000\n",
            "num_nonrel_judged_ret    7       11.0000\n",
            "num_q                    9       1.0000\n",
            "num_ret                  9       2.0000\n",
            "num_rel                  9       1.0000\n",
            "num_rel_ret              9       0.0000\n",
            "map                      9       0.0000\n",
            "gm_map                   9       -11.5129\n",
            "Rprec                    9       0.0000\n",
            "bpref                    9       0.0000\n",
            "recip_rank               9       0.0000\n",
            "iprec_at_recall_0.00     9       0.0000\n",
            "iprec_at_recall_0.10     9       0.0000\n",
            "iprec_at_recall_0.20     9       0.0000\n",
            "iprec_at_recall_0.30     9       0.0000\n",
            "iprec_at_recall_0.40     9       0.0000\n",
            "iprec_at_recall_0.50     9       0.0000\n",
            "iprec_at_recall_0.60     9       0.0000\n",
            "iprec_at_recall_0.70     9       0.0000\n",
            "iprec_at_recall_0.80     9       0.0000\n",
            "iprec_at_recall_0.90     9       0.0000\n",
            "iprec_at_recall_1.00     9       0.0000\n",
            "P_5                      9       0.0000\n",
            "P_10                     9       0.0000\n",
            "P_15                     9       0.0000\n",
            "P_20                     9       0.0000\n",
            "P_30                     9       0.0000\n",
            "P_100                    9       0.0000\n",
            "P_200                    9       0.0000\n",
            "P_500                    9       0.0000\n",
            "P_1000                   9       0.0000\n",
            "relstring                9       0.0000\n",
            "recall_5                 9       0.0000\n",
            "recall_10                9       0.0000\n",
            "recall_15                9       0.0000\n",
            "recall_20                9       0.0000\n",
            "recall_30                9       0.0000\n",
            "recall_100               9       0.0000\n",
            "recall_200               9       0.0000\n",
            "recall_500               9       0.0000\n",
            "recall_1000              9       0.0000\n",
            "infAP                    9       0.0000\n",
            "gm_bpref                 9       -11.5129\n",
            "Rprec_mult_0.20          9       0.0000\n",
            "Rprec_mult_0.40          9       0.0000\n",
            "Rprec_mult_0.60          9       0.0000\n",
            "Rprec_mult_0.80          9       0.0000\n",
            "Rprec_mult_1.00          9       0.0000\n",
            "Rprec_mult_1.20          9       0.0000\n",
            "Rprec_mult_1.40          9       0.0000\n",
            "Rprec_mult_1.60          9       0.0000\n",
            "Rprec_mult_1.80          9       0.0000\n",
            "Rprec_mult_2.00          9       0.0000\n",
            "utility                  9       -2.0000\n",
            "11pt_avg                 9       0.0000\n",
            "binG                     9       0.0000\n",
            "G                        9       0.0000\n",
            "ndcg                     9       0.0000\n",
            "ndcg_rel                 9       0.0000\n",
            "Rndcg                    9       0.0000\n",
            "ndcg_cut_5               9       0.0000\n",
            "ndcg_cut_10              9       0.0000\n",
            "ndcg_cut_15              9       0.0000\n",
            "ndcg_cut_20              9       0.0000\n",
            "ndcg_cut_30              9       0.0000\n",
            "ndcg_cut_100             9       0.0000\n",
            "ndcg_cut_200             9       0.0000\n",
            "ndcg_cut_500             9       0.0000\n",
            "ndcg_cut_1000            9       0.0000\n",
            "map_cut_5                9       0.0000\n",
            "map_cut_10               9       0.0000\n",
            "map_cut_15               9       0.0000\n",
            "map_cut_20               9       0.0000\n",
            "map_cut_30               9       0.0000\n",
            "map_cut_100              9       0.0000\n",
            "map_cut_200              9       0.0000\n",
            "map_cut_500              9       0.0000\n",
            "map_cut_1000             9       0.0000\n",
            "relative_P_5             9       0.0000\n",
            "relative_P_10            9       0.0000\n",
            "relative_P_15            9       0.0000\n",
            "relative_P_20            9       0.0000\n",
            "relative_P_30            9       0.0000\n",
            "relative_P_100           9       0.0000\n",
            "relative_P_200           9       0.0000\n",
            "relative_P_500           9       0.0000\n",
            "relative_P_1000          9       0.0000\n",
            "success_1                9       0.0000\n",
            "success_5                9       0.0000\n",
            "success_10               9       0.0000\n",
            "set_P                    9       0.0000\n",
            "set_relative_P           9       0.0000\n",
            "set_recall               9       0.0000\n",
            "set_map                  9       0.0000\n",
            "set_F                    9       0.0000\n",
            "num_nonrel_judged_ret    9       2.0000\n",
            "num_q                    10      1.0000\n",
            "num_ret                  10      24.0000\n",
            "num_rel                  10      1.0000\n",
            "num_rel_ret              10      1.0000\n",
            "map                      10      0.3333\n",
            "gm_map                   10      -1.0986\n",
            "Rprec                    10      0.0000\n",
            "bpref                    10      0.0000\n",
            "recip_rank               10      0.3333\n",
            "iprec_at_recall_0.00     10      0.3333\n",
            "iprec_at_recall_0.10     10      0.3333\n",
            "iprec_at_recall_0.20     10      0.3333\n",
            "iprec_at_recall_0.30     10      0.3333\n",
            "iprec_at_recall_0.40     10      0.3333\n",
            "iprec_at_recall_0.50     10      0.3333\n",
            "iprec_at_recall_0.60     10      0.3333\n",
            "iprec_at_recall_0.70     10      0.3333\n",
            "iprec_at_recall_0.80     10      0.3333\n",
            "iprec_at_recall_0.90     10      0.3333\n",
            "iprec_at_recall_1.00     10      0.3333\n",
            "P_5                      10      0.2000\n",
            "P_10                     10      0.1000\n",
            "P_15                     10      0.0667\n",
            "P_20                     10      0.0500\n",
            "P_30                     10      0.0333\n",
            "P_100                    10      0.0100\n",
            "P_200                    10      0.0050\n",
            "P_500                    10      0.0020\n",
            "P_1000                   10      0.0010\n",
            "relstring                10      0.0000\n",
            "recall_5                 10      1.0000\n",
            "recall_10                10      1.0000\n",
            "recall_15                10      1.0000\n",
            "recall_20                10      1.0000\n",
            "recall_30                10      1.0000\n",
            "recall_100               10      1.0000\n",
            "recall_200               10      1.0000\n",
            "recall_500               10      1.0000\n",
            "recall_1000              10      1.0000\n",
            "infAP                    10      0.3333\n",
            "gm_bpref                 10      -11.5129\n",
            "Rprec_mult_0.20          10      0.0000\n",
            "Rprec_mult_0.40          10      0.0000\n",
            "Rprec_mult_0.60          10      0.0000\n",
            "Rprec_mult_0.80          10      0.0000\n",
            "Rprec_mult_1.00          10      0.0000\n",
            "Rprec_mult_1.20          10      0.0000\n",
            "Rprec_mult_1.40          10      0.0000\n",
            "Rprec_mult_1.60          10      0.0000\n",
            "Rprec_mult_1.80          10      0.0000\n",
            "Rprec_mult_2.00          10      0.0000\n",
            "utility                  10      -22.0000\n",
            "11pt_avg                 10      0.3333\n",
            "binG                     10      0.5000\n",
            "G                        10      0.5000\n",
            "ndcg                     10      0.5000\n",
            "ndcg_rel                 10      0.5000\n",
            "Rndcg                    10      0.2500\n",
            "ndcg_cut_5               10      0.5000\n",
            "ndcg_cut_10              10      0.5000\n",
            "ndcg_cut_15              10      0.5000\n",
            "ndcg_cut_20              10      0.5000\n",
            "ndcg_cut_30              10      0.5000\n",
            "ndcg_cut_100             10      0.5000\n",
            "ndcg_cut_200             10      0.5000\n",
            "ndcg_cut_500             10      0.5000\n",
            "ndcg_cut_1000            10      0.5000\n",
            "map_cut_5                10      0.3333\n",
            "map_cut_10               10      0.3333\n",
            "map_cut_15               10      0.3333\n",
            "map_cut_20               10      0.3333\n",
            "map_cut_30               10      0.3333\n",
            "map_cut_100              10      0.3333\n",
            "map_cut_200              10      0.3333\n",
            "map_cut_500              10      0.3333\n",
            "map_cut_1000             10      0.3333\n",
            "relative_P_5             10      1.0000\n",
            "relative_P_10            10      1.0000\n",
            "relative_P_15            10      1.0000\n",
            "relative_P_20            10      1.0000\n",
            "relative_P_30            10      1.0000\n",
            "relative_P_100           10      1.0000\n",
            "relative_P_200           10      1.0000\n",
            "relative_P_500           10      1.0000\n",
            "relative_P_1000          10      1.0000\n",
            "success_1                10      0.0000\n",
            "success_5                10      1.0000\n",
            "success_10               10      1.0000\n",
            "set_P                    10      0.0417\n",
            "set_relative_P           10      1.0000\n",
            "set_recall               10      1.0000\n",
            "set_map                  10      0.0417\n",
            "set_F                    10      0.0800\n",
            "num_nonrel_judged_ret    10      11.0000\n",
            "num_q                    14      1.0000\n",
            "num_ret                  14      24.0000\n",
            "num_rel                  14      1.0000\n",
            "num_rel_ret              14      1.0000\n",
            "map                      14      1.0000\n",
            "gm_map                   14      0.0000\n",
            "Rprec                    14      1.0000\n",
            "bpref                    14      1.0000\n",
            "recip_rank               14      1.0000\n",
            "iprec_at_recall_0.00     14      1.0000\n",
            "iprec_at_recall_0.10     14      1.0000\n",
            "iprec_at_recall_0.20     14      1.0000\n",
            "iprec_at_recall_0.30     14      1.0000\n",
            "iprec_at_recall_0.40     14      1.0000\n",
            "iprec_at_recall_0.50     14      1.0000\n",
            "iprec_at_recall_0.60     14      1.0000\n",
            "iprec_at_recall_0.70     14      1.0000\n",
            "iprec_at_recall_0.80     14      1.0000\n",
            "iprec_at_recall_0.90     14      1.0000\n",
            "iprec_at_recall_1.00     14      1.0000\n",
            "P_5                      14      0.2000\n",
            "P_10                     14      0.1000\n",
            "P_15                     14      0.0667\n",
            "P_20                     14      0.0500\n",
            "P_30                     14      0.0333\n",
            "P_100                    14      0.0100\n",
            "P_200                    14      0.0050\n",
            "P_500                    14      0.0020\n",
            "P_1000                   14      0.0010\n",
            "relstring                14      0.0000\n",
            "recall_5                 14      1.0000\n",
            "recall_10                14      1.0000\n",
            "recall_15                14      1.0000\n",
            "recall_20                14      1.0000\n",
            "recall_30                14      1.0000\n",
            "recall_100               14      1.0000\n",
            "recall_200               14      1.0000\n",
            "recall_500               14      1.0000\n",
            "recall_1000              14      1.0000\n",
            "infAP                    14      1.0000\n",
            "gm_bpref                 14      0.0000\n",
            "Rprec_mult_0.20          14      1.0000\n",
            "Rprec_mult_0.40          14      1.0000\n",
            "Rprec_mult_0.60          14      1.0000\n",
            "Rprec_mult_0.80          14      1.0000\n",
            "Rprec_mult_1.00          14      1.0000\n",
            "Rprec_mult_1.20          14      0.5000\n",
            "Rprec_mult_1.40          14      0.5000\n",
            "Rprec_mult_1.60          14      0.5000\n",
            "Rprec_mult_1.80          14      0.5000\n",
            "Rprec_mult_2.00          14      0.5000\n",
            "utility                  14      -22.0000\n",
            "11pt_avg                 14      1.0000\n",
            "binG                     14      1.0000\n",
            "G                        14      1.0000\n",
            "ndcg                     14      1.0000\n",
            "ndcg_rel                 14      1.0000\n",
            "Rndcg                    14      1.0000\n",
            "ndcg_cut_5               14      1.0000\n",
            "ndcg_cut_10              14      1.0000\n",
            "ndcg_cut_15              14      1.0000\n",
            "ndcg_cut_20              14      1.0000\n",
            "ndcg_cut_30              14      1.0000\n",
            "ndcg_cut_100             14      1.0000\n",
            "ndcg_cut_200             14      1.0000\n",
            "ndcg_cut_500             14      1.0000\n",
            "ndcg_cut_1000            14      1.0000\n",
            "map_cut_5                14      1.0000\n",
            "map_cut_10               14      1.0000\n",
            "map_cut_15               14      1.0000\n",
            "map_cut_20               14      1.0000\n",
            "map_cut_30               14      1.0000\n",
            "map_cut_100              14      1.0000\n",
            "map_cut_200              14      1.0000\n",
            "map_cut_500              14      1.0000\n",
            "map_cut_1000             14      1.0000\n",
            "relative_P_5             14      1.0000\n",
            "relative_P_10            14      1.0000\n",
            "relative_P_15            14      1.0000\n",
            "relative_P_20            14      1.0000\n",
            "relative_P_30            14      1.0000\n",
            "relative_P_100           14      1.0000\n",
            "relative_P_200           14      1.0000\n",
            "relative_P_500           14      1.0000\n",
            "relative_P_1000          14      1.0000\n",
            "success_1                14      1.0000\n",
            "success_5                14      1.0000\n",
            "success_10               14      1.0000\n",
            "set_P                    14      0.0417\n",
            "set_relative_P           14      1.0000\n",
            "set_recall               14      1.0000\n",
            "set_map                  14      0.0417\n",
            "set_F                    14      0.0800\n",
            "num_nonrel_judged_ret    14      18.0000\n",
            "num_q                    16      1.0000\n",
            "num_ret                  16      19.0000\n",
            "num_rel                  16      7.0000\n",
            "num_rel_ret              16      0.0000\n",
            "map                      16      0.0000\n",
            "gm_map                   16      -11.5129\n",
            "Rprec                    16      0.0000\n",
            "bpref                    16      0.0000\n",
            "recip_rank               16      0.0000\n",
            "iprec_at_recall_0.00     16      0.0000\n",
            "iprec_at_recall_0.10     16      0.0000\n",
            "iprec_at_recall_0.20     16      0.0000\n",
            "iprec_at_recall_0.30     16      0.0000\n",
            "iprec_at_recall_0.40     16      0.0000\n",
            "iprec_at_recall_0.50     16      0.0000\n",
            "iprec_at_recall_0.60     16      0.0000\n",
            "iprec_at_recall_0.70     16      0.0000\n",
            "iprec_at_recall_0.80     16      0.0000\n",
            "iprec_at_recall_0.90     16      0.0000\n",
            "iprec_at_recall_1.00     16      0.0000\n",
            "P_5                      16      0.0000\n",
            "P_10                     16      0.0000\n",
            "P_15                     16      0.0000\n",
            "P_20                     16      0.0000\n",
            "P_30                     16      0.0000\n",
            "P_100                    16      0.0000\n",
            "P_200                    16      0.0000\n",
            "P_500                    16      0.0000\n",
            "P_1000                   16      0.0000\n",
            "relstring                16      0.0000\n",
            "recall_5                 16      0.0000\n",
            "recall_10                16      0.0000\n",
            "recall_15                16      0.0000\n",
            "recall_20                16      0.0000\n",
            "recall_30                16      0.0000\n",
            "recall_100               16      0.0000\n",
            "recall_200               16      0.0000\n",
            "recall_500               16      0.0000\n",
            "recall_1000              16      0.0000\n",
            "infAP                    16      0.0000\n",
            "gm_bpref                 16      -11.5129\n",
            "Rprec_mult_0.20          16      0.0000\n",
            "Rprec_mult_0.40          16      0.0000\n",
            "Rprec_mult_0.60          16      0.0000\n",
            "Rprec_mult_0.80          16      0.0000\n",
            "Rprec_mult_1.00          16      0.0000\n",
            "Rprec_mult_1.20          16      0.0000\n",
            "Rprec_mult_1.40          16      0.0000\n",
            "Rprec_mult_1.60          16      0.0000\n",
            "Rprec_mult_1.80          16      0.0000\n",
            "Rprec_mult_2.00          16      0.0000\n",
            "utility                  16      -19.0000\n",
            "11pt_avg                 16      0.0000\n",
            "binG                     16      0.0000\n",
            "G                        16      0.0000\n",
            "ndcg                     16      0.0000\n",
            "ndcg_rel                 16      0.0000\n",
            "Rndcg                    16      0.0000\n",
            "ndcg_cut_5               16      0.0000\n",
            "ndcg_cut_10              16      0.0000\n",
            "ndcg_cut_15              16      0.0000\n",
            "ndcg_cut_20              16      0.0000\n",
            "ndcg_cut_30              16      0.0000\n",
            "ndcg_cut_100             16      0.0000\n",
            "ndcg_cut_200             16      0.0000\n",
            "ndcg_cut_500             16      0.0000\n",
            "ndcg_cut_1000            16      0.0000\n",
            "map_cut_5                16      0.0000\n",
            "map_cut_10               16      0.0000\n",
            "map_cut_15               16      0.0000\n",
            "map_cut_20               16      0.0000\n",
            "map_cut_30               16      0.0000\n",
            "map_cut_100              16      0.0000\n",
            "map_cut_200              16      0.0000\n",
            "map_cut_500              16      0.0000\n",
            "map_cut_1000             16      0.0000\n",
            "relative_P_5             16      0.0000\n",
            "relative_P_10            16      0.0000\n",
            "relative_P_15            16      0.0000\n",
            "relative_P_20            16      0.0000\n",
            "relative_P_30            16      0.0000\n",
            "relative_P_100           16      0.0000\n",
            "relative_P_200           16      0.0000\n",
            "relative_P_500           16      0.0000\n",
            "relative_P_1000          16      0.0000\n",
            "success_1                16      0.0000\n",
            "success_5                16      0.0000\n",
            "success_10               16      0.0000\n",
            "set_P                    16      0.0000\n",
            "set_relative_P           16      0.0000\n",
            "set_recall               16      0.0000\n",
            "set_map                  16      0.0000\n",
            "set_F                    16      0.0000\n",
            "num_nonrel_judged_ret    16      15.0000\n",
            "num_q                    18      1.0000\n",
            "num_ret                  18      22.0000\n",
            "num_rel                  18      1.0000\n",
            "num_rel_ret              18      1.0000\n",
            "map                      18      1.0000\n",
            "gm_map                   18      0.0000\n",
            "Rprec                    18      1.0000\n",
            "bpref                    18      1.0000\n",
            "recip_rank               18      1.0000\n",
            "iprec_at_recall_0.00     18      1.0000\n",
            "iprec_at_recall_0.10     18      1.0000\n",
            "iprec_at_recall_0.20     18      1.0000\n",
            "iprec_at_recall_0.30     18      1.0000\n",
            "iprec_at_recall_0.40     18      1.0000\n",
            "iprec_at_recall_0.50     18      1.0000\n",
            "iprec_at_recall_0.60     18      1.0000\n",
            "iprec_at_recall_0.70     18      1.0000\n",
            "iprec_at_recall_0.80     18      1.0000\n",
            "iprec_at_recall_0.90     18      1.0000\n",
            "iprec_at_recall_1.00     18      1.0000\n",
            "P_5                      18      0.2000\n",
            "P_10                     18      0.1000\n",
            "P_15                     18      0.0667\n",
            "P_20                     18      0.0500\n",
            "P_30                     18      0.0333\n",
            "P_100                    18      0.0100\n",
            "P_200                    18      0.0050\n",
            "P_500                    18      0.0020\n",
            "P_1000                   18      0.0010\n",
            "relstring                18      0.0000\n",
            "recall_5                 18      1.0000\n",
            "recall_10                18      1.0000\n",
            "recall_15                18      1.0000\n",
            "recall_20                18      1.0000\n",
            "recall_30                18      1.0000\n",
            "recall_100               18      1.0000\n",
            "recall_200               18      1.0000\n",
            "recall_500               18      1.0000\n",
            "recall_1000              18      1.0000\n",
            "infAP                    18      1.0000\n",
            "gm_bpref                 18      0.0000\n",
            "Rprec_mult_0.20          18      1.0000\n",
            "Rprec_mult_0.40          18      1.0000\n",
            "Rprec_mult_0.60          18      1.0000\n",
            "Rprec_mult_0.80          18      1.0000\n",
            "Rprec_mult_1.00          18      1.0000\n",
            "Rprec_mult_1.20          18      0.5000\n",
            "Rprec_mult_1.40          18      0.5000\n",
            "Rprec_mult_1.60          18      0.5000\n",
            "Rprec_mult_1.80          18      0.5000\n",
            "Rprec_mult_2.00          18      0.5000\n",
            "utility                  18      -20.0000\n",
            "11pt_avg                 18      1.0000\n",
            "binG                     18      1.0000\n",
            "G                        18      1.0000\n",
            "ndcg                     18      1.0000\n",
            "ndcg_rel                 18      1.0000\n",
            "Rndcg                    18      1.0000\n",
            "ndcg_cut_5               18      1.0000\n",
            "ndcg_cut_10              18      1.0000\n",
            "ndcg_cut_15              18      1.0000\n",
            "ndcg_cut_20              18      1.0000\n",
            "ndcg_cut_30              18      1.0000\n",
            "ndcg_cut_100             18      1.0000\n",
            "ndcg_cut_200             18      1.0000\n",
            "ndcg_cut_500             18      1.0000\n",
            "ndcg_cut_1000            18      1.0000\n",
            "map_cut_5                18      1.0000\n",
            "map_cut_10               18      1.0000\n",
            "map_cut_15               18      1.0000\n",
            "map_cut_20               18      1.0000\n",
            "map_cut_30               18      1.0000\n",
            "map_cut_100              18      1.0000\n",
            "map_cut_200              18      1.0000\n",
            "map_cut_500              18      1.0000\n",
            "map_cut_1000             18      1.0000\n",
            "relative_P_5             18      1.0000\n",
            "relative_P_10            18      1.0000\n",
            "relative_P_15            18      1.0000\n",
            "relative_P_20            18      1.0000\n",
            "relative_P_30            18      1.0000\n",
            "relative_P_100           18      1.0000\n",
            "relative_P_200           18      1.0000\n",
            "relative_P_500           18      1.0000\n",
            "relative_P_1000          18      1.0000\n",
            "success_1                18      1.0000\n",
            "success_5                18      1.0000\n",
            "success_10               18      1.0000\n",
            "set_P                    18      0.0455\n",
            "set_relative_P           18      1.0000\n",
            "set_recall               18      1.0000\n",
            "set_map                  18      0.0455\n",
            "set_F                    18      0.0870\n",
            "num_nonrel_judged_ret    18      21.0000\n",
            "num_q                    19      1.0000\n",
            "num_ret                  19      5.0000\n",
            "num_rel                  19      2.0000\n",
            "num_rel_ret              19      1.0000\n",
            "map                      19      0.5000\n",
            "gm_map                   19      -0.6931\n",
            "Rprec                    19      0.5000\n",
            "bpref                    19      0.5000\n",
            "recip_rank               19      1.0000\n",
            "iprec_at_recall_0.00     19      1.0000\n",
            "iprec_at_recall_0.10     19      1.0000\n",
            "iprec_at_recall_0.20     19      1.0000\n",
            "iprec_at_recall_0.30     19      1.0000\n",
            "iprec_at_recall_0.40     19      1.0000\n",
            "iprec_at_recall_0.50     19      1.0000\n",
            "iprec_at_recall_0.60     19      0.0000\n",
            "iprec_at_recall_0.70     19      0.0000\n",
            "iprec_at_recall_0.80     19      0.0000\n",
            "iprec_at_recall_0.90     19      0.0000\n",
            "iprec_at_recall_1.00     19      0.0000\n",
            "P_5                      19      0.2000\n",
            "P_10                     19      0.1000\n",
            "P_15                     19      0.0667\n",
            "P_20                     19      0.0500\n",
            "P_30                     19      0.0333\n",
            "P_100                    19      0.0100\n",
            "P_200                    19      0.0050\n",
            "P_500                    19      0.0020\n",
            "P_1000                   19      0.0010\n",
            "relstring                19      0.0000\n",
            "recall_5                 19      0.5000\n",
            "recall_10                19      0.5000\n",
            "recall_15                19      0.5000\n",
            "recall_20                19      0.5000\n",
            "recall_30                19      0.5000\n",
            "recall_100               19      0.5000\n",
            "recall_200               19      0.5000\n",
            "recall_500               19      0.5000\n",
            "recall_1000              19      0.5000\n",
            "infAP                    19      0.5000\n",
            "gm_bpref                 19      -0.6931\n",
            "Rprec_mult_0.20          19      1.0000\n",
            "Rprec_mult_0.40          19      1.0000\n",
            "Rprec_mult_0.60          19      0.5000\n",
            "Rprec_mult_0.80          19      0.5000\n",
            "Rprec_mult_1.00          19      0.5000\n",
            "Rprec_mult_1.20          19      0.3333\n",
            "Rprec_mult_1.40          19      0.3333\n",
            "Rprec_mult_1.60          19      0.2500\n",
            "Rprec_mult_1.80          19      0.2500\n",
            "Rprec_mult_2.00          19      0.2500\n",
            "utility                  19      -3.0000\n",
            "11pt_avg                 19      0.5455\n",
            "binG                     19      0.5000\n",
            "G                        19      0.5000\n",
            "ndcg                     19      0.6131\n",
            "ndcg_rel                 19      0.8066\n",
            "Rndcg                    19      0.6131\n",
            "ndcg_cut_5               19      0.6131\n",
            "ndcg_cut_10              19      0.6131\n",
            "ndcg_cut_15              19      0.6131\n",
            "ndcg_cut_20              19      0.6131\n",
            "ndcg_cut_30              19      0.6131\n",
            "ndcg_cut_100             19      0.6131\n",
            "ndcg_cut_200             19      0.6131\n",
            "ndcg_cut_500             19      0.6131\n",
            "ndcg_cut_1000            19      0.6131\n",
            "map_cut_5                19      0.5000\n",
            "map_cut_10               19      0.5000\n",
            "map_cut_15               19      0.5000\n",
            "map_cut_20               19      0.5000\n",
            "map_cut_30               19      0.5000\n",
            "map_cut_100              19      0.5000\n",
            "map_cut_200              19      0.5000\n",
            "map_cut_500              19      0.5000\n",
            "map_cut_1000             19      0.5000\n",
            "relative_P_5             19      0.5000\n",
            "relative_P_10            19      0.5000\n",
            "relative_P_15            19      0.5000\n",
            "relative_P_20            19      0.5000\n",
            "relative_P_30            19      0.5000\n",
            "relative_P_100           19      0.5000\n",
            "relative_P_200           19      0.5000\n",
            "relative_P_500           19      0.5000\n",
            "relative_P_1000          19      0.5000\n",
            "success_1                19      1.0000\n",
            "success_5                19      1.0000\n",
            "success_10               19      1.0000\n",
            "set_P                    19      0.2000\n",
            "set_relative_P           19      0.5000\n",
            "set_recall               19      0.5000\n",
            "set_map                  19      0.1000\n",
            "set_F                    19      0.2857\n",
            "num_nonrel_judged_ret    19      4.0000\n",
            "num_q                    22      1.0000\n",
            "num_ret                  22      30.0000\n",
            "num_rel                  22      1.0000\n",
            "num_rel_ret              22      1.0000\n",
            "map                      22      0.0714\n",
            "gm_map                   22      -2.6391\n",
            "Rprec                    22      0.0000\n",
            "bpref                    22      0.0000\n",
            "recip_rank               22      0.0714\n",
            "iprec_at_recall_0.00     22      0.0714\n",
            "iprec_at_recall_0.10     22      0.0714\n",
            "iprec_at_recall_0.20     22      0.0714\n",
            "iprec_at_recall_0.30     22      0.0714\n",
            "iprec_at_recall_0.40     22      0.0714\n",
            "iprec_at_recall_0.50     22      0.0714\n",
            "iprec_at_recall_0.60     22      0.0714\n",
            "iprec_at_recall_0.70     22      0.0714\n",
            "iprec_at_recall_0.80     22      0.0714\n",
            "iprec_at_recall_0.90     22      0.0714\n",
            "iprec_at_recall_1.00     22      0.0714\n",
            "P_5                      22      0.0000\n",
            "P_10                     22      0.0000\n",
            "P_15                     22      0.0667\n",
            "P_20                     22      0.0500\n",
            "P_30                     22      0.0333\n",
            "P_100                    22      0.0100\n",
            "P_200                    22      0.0050\n",
            "P_500                    22      0.0020\n",
            "P_1000                   22      0.0010\n",
            "relstring                22      0.0000\n",
            "recall_5                 22      0.0000\n",
            "recall_10                22      0.0000\n",
            "recall_15                22      1.0000\n",
            "recall_20                22      1.0000\n",
            "recall_30                22      1.0000\n",
            "recall_100               22      1.0000\n",
            "recall_200               22      1.0000\n",
            "recall_500               22      1.0000\n",
            "recall_1000              22      1.0000\n",
            "infAP                    22      0.0714\n",
            "gm_bpref                 22      -11.5129\n",
            "Rprec_mult_0.20          22      0.0000\n",
            "Rprec_mult_0.40          22      0.0000\n",
            "Rprec_mult_0.60          22      0.0000\n",
            "Rprec_mult_0.80          22      0.0000\n",
            "Rprec_mult_1.00          22      0.0000\n",
            "Rprec_mult_1.20          22      0.0000\n",
            "Rprec_mult_1.40          22      0.0000\n",
            "Rprec_mult_1.60          22      0.0000\n",
            "Rprec_mult_1.80          22      0.0000\n",
            "Rprec_mult_2.00          22      0.0000\n",
            "utility                  22      -28.0000\n",
            "11pt_avg                 22      0.0714\n",
            "binG                     22      0.2560\n",
            "G                        22      0.2560\n",
            "ndcg                     22      0.2560\n",
            "ndcg_rel                 22      0.2560\n",
            "Rndcg                    22      0.1280\n",
            "ndcg_cut_5               22      0.0000\n",
            "ndcg_cut_10              22      0.0000\n",
            "ndcg_cut_15              22      0.2560\n",
            "ndcg_cut_20              22      0.2560\n",
            "ndcg_cut_30              22      0.2560\n",
            "ndcg_cut_100             22      0.2560\n",
            "ndcg_cut_200             22      0.2560\n",
            "ndcg_cut_500             22      0.2560\n",
            "ndcg_cut_1000            22      0.2560\n",
            "map_cut_5                22      0.0000\n",
            "map_cut_10               22      0.0000\n",
            "map_cut_15               22      0.0714\n",
            "map_cut_20               22      0.0714\n",
            "map_cut_30               22      0.0714\n",
            "map_cut_100              22      0.0714\n",
            "map_cut_200              22      0.0714\n",
            "map_cut_500              22      0.0714\n",
            "map_cut_1000             22      0.0714\n",
            "relative_P_5             22      0.0000\n",
            "relative_P_10            22      0.0000\n",
            "relative_P_15            22      1.0000\n",
            "relative_P_20            22      1.0000\n",
            "relative_P_30            22      1.0000\n",
            "relative_P_100           22      1.0000\n",
            "relative_P_200           22      1.0000\n",
            "relative_P_500           22      1.0000\n",
            "relative_P_1000          22      1.0000\n",
            "success_1                22      0.0000\n",
            "success_5                22      0.0000\n",
            "success_10               22      0.0000\n",
            "set_P                    22      0.0333\n",
            "set_relative_P           22      1.0000\n",
            "set_recall               22      1.0000\n",
            "set_map                  22      0.0333\n",
            "set_F                    22      0.0645\n",
            "num_nonrel_judged_ret    22      27.0000\n",
            "num_q                    24      1.0000\n",
            "num_ret                  24      17.0000\n",
            "num_rel                  24      1.0000\n",
            "num_rel_ret              24      1.0000\n",
            "map                      24      1.0000\n",
            "gm_map                   24      0.0000\n",
            "Rprec                    24      1.0000\n",
            "bpref                    24      1.0000\n",
            "recip_rank               24      1.0000\n",
            "iprec_at_recall_0.00     24      1.0000\n",
            "iprec_at_recall_0.10     24      1.0000\n",
            "iprec_at_recall_0.20     24      1.0000\n",
            "iprec_at_recall_0.30     24      1.0000\n",
            "iprec_at_recall_0.40     24      1.0000\n",
            "iprec_at_recall_0.50     24      1.0000\n",
            "iprec_at_recall_0.60     24      1.0000\n",
            "iprec_at_recall_0.70     24      1.0000\n",
            "iprec_at_recall_0.80     24      1.0000\n",
            "iprec_at_recall_0.90     24      1.0000\n",
            "iprec_at_recall_1.00     24      1.0000\n",
            "P_5                      24      0.2000\n",
            "P_10                     24      0.1000\n",
            "P_15                     24      0.0667\n",
            "P_20                     24      0.0500\n",
            "P_30                     24      0.0333\n",
            "P_100                    24      0.0100\n",
            "P_200                    24      0.0050\n",
            "P_500                    24      0.0020\n",
            "P_1000                   24      0.0010\n",
            "relstring                24      0.0000\n",
            "recall_5                 24      1.0000\n",
            "recall_10                24      1.0000\n",
            "recall_15                24      1.0000\n",
            "recall_20                24      1.0000\n",
            "recall_30                24      1.0000\n",
            "recall_100               24      1.0000\n",
            "recall_200               24      1.0000\n",
            "recall_500               24      1.0000\n",
            "recall_1000              24      1.0000\n",
            "infAP                    24      1.0000\n",
            "gm_bpref                 24      0.0000\n",
            "Rprec_mult_0.20          24      1.0000\n",
            "Rprec_mult_0.40          24      1.0000\n",
            "Rprec_mult_0.60          24      1.0000\n",
            "Rprec_mult_0.80          24      1.0000\n",
            "Rprec_mult_1.00          24      1.0000\n",
            "Rprec_mult_1.20          24      0.5000\n",
            "Rprec_mult_1.40          24      0.5000\n",
            "Rprec_mult_1.60          24      0.5000\n",
            "Rprec_mult_1.80          24      0.5000\n",
            "Rprec_mult_2.00          24      0.5000\n",
            "utility                  24      -15.0000\n",
            "11pt_avg                 24      1.0000\n",
            "binG                     24      1.0000\n",
            "G                        24      1.0000\n",
            "ndcg                     24      1.0000\n",
            "ndcg_rel                 24      1.0000\n",
            "Rndcg                    24      1.0000\n",
            "ndcg_cut_5               24      1.0000\n",
            "ndcg_cut_10              24      1.0000\n",
            "ndcg_cut_15              24      1.0000\n",
            "ndcg_cut_20              24      1.0000\n",
            "ndcg_cut_30              24      1.0000\n",
            "ndcg_cut_100             24      1.0000\n",
            "ndcg_cut_200             24      1.0000\n",
            "ndcg_cut_500             24      1.0000\n",
            "ndcg_cut_1000            24      1.0000\n",
            "map_cut_5                24      1.0000\n",
            "map_cut_10               24      1.0000\n",
            "map_cut_15               24      1.0000\n",
            "map_cut_20               24      1.0000\n",
            "map_cut_30               24      1.0000\n",
            "map_cut_100              24      1.0000\n",
            "map_cut_200              24      1.0000\n",
            "map_cut_500              24      1.0000\n",
            "map_cut_1000             24      1.0000\n",
            "relative_P_5             24      1.0000\n",
            "relative_P_10            24      1.0000\n",
            "relative_P_15            24      1.0000\n",
            "relative_P_20            24      1.0000\n",
            "relative_P_30            24      1.0000\n",
            "relative_P_100           24      1.0000\n",
            "relative_P_200           24      1.0000\n",
            "relative_P_500           24      1.0000\n",
            "relative_P_1000          24      1.0000\n",
            "success_1                24      1.0000\n",
            "success_5                24      1.0000\n",
            "success_10               24      1.0000\n",
            "set_P                    24      0.0588\n",
            "set_relative_P           24      1.0000\n",
            "set_recall               24      1.0000\n",
            "set_map                  24      0.0588\n",
            "set_F                    24      0.1111\n",
            "num_nonrel_judged_ret    24      14.0000\n",
            "num_q                    26      1.0000\n",
            "num_ret                  26      40.0000\n",
            "num_rel                  26      3.0000\n",
            "num_rel_ret              26      1.0000\n",
            "map                      26      0.0556\n",
            "gm_map                   26      -2.8904\n",
            "Rprec                    26      0.0000\n",
            "bpref                    26      0.0000\n",
            "recip_rank               26      0.1667\n",
            "iprec_at_recall_0.00     26      0.1667\n",
            "iprec_at_recall_0.10     26      0.1667\n",
            "iprec_at_recall_0.20     26      0.1667\n",
            "iprec_at_recall_0.30     26      0.1667\n",
            "iprec_at_recall_0.40     26      0.0000\n",
            "iprec_at_recall_0.50     26      0.0000\n",
            "iprec_at_recall_0.60     26      0.0000\n",
            "iprec_at_recall_0.70     26      0.0000\n",
            "iprec_at_recall_0.80     26      0.0000\n",
            "iprec_at_recall_0.90     26      0.0000\n",
            "iprec_at_recall_1.00     26      0.0000\n",
            "P_5                      26      0.0000\n",
            "P_10                     26      0.1000\n",
            "P_15                     26      0.0667\n",
            "P_20                     26      0.0500\n",
            "P_30                     26      0.0333\n",
            "P_100                    26      0.0100\n",
            "P_200                    26      0.0050\n",
            "P_500                    26      0.0020\n",
            "P_1000                   26      0.0010\n",
            "relstring                26      0.0000\n",
            "recall_5                 26      0.0000\n",
            "recall_10                26      0.3333\n",
            "recall_15                26      0.3333\n",
            "recall_20                26      0.3333\n",
            "recall_30                26      0.3333\n",
            "recall_100               26      0.3333\n",
            "recall_200               26      0.3333\n",
            "recall_500               26      0.3333\n",
            "recall_1000              26      0.3333\n",
            "infAP                    26      0.0556\n",
            "gm_bpref                 26      -11.5129\n",
            "Rprec_mult_0.20          26      0.0000\n",
            "Rprec_mult_0.40          26      0.0000\n",
            "Rprec_mult_0.60          26      0.0000\n",
            "Rprec_mult_0.80          26      0.0000\n",
            "Rprec_mult_1.00          26      0.0000\n",
            "Rprec_mult_1.20          26      0.0000\n",
            "Rprec_mult_1.40          26      0.0000\n",
            "Rprec_mult_1.60          26      0.0000\n",
            "Rprec_mult_1.80          26      0.1667\n",
            "Rprec_mult_2.00          26      0.1667\n",
            "utility                  26      -38.0000\n",
            "11pt_avg                 26      0.0606\n",
            "binG                     26      0.1187\n",
            "G                        26      0.1187\n",
            "ndcg                     26      0.1672\n",
            "ndcg_rel                 26      0.1672\n",
            "Rndcg                    26      0.0836\n",
            "ndcg_cut_5               26      0.0000\n",
            "ndcg_cut_10              26      0.1672\n",
            "ndcg_cut_15              26      0.1672\n",
            "ndcg_cut_20              26      0.1672\n",
            "ndcg_cut_30              26      0.1672\n",
            "ndcg_cut_100             26      0.1672\n",
            "ndcg_cut_200             26      0.1672\n",
            "ndcg_cut_500             26      0.1672\n",
            "ndcg_cut_1000            26      0.1672\n",
            "map_cut_5                26      0.0000\n",
            "map_cut_10               26      0.0556\n",
            "map_cut_15               26      0.0556\n",
            "map_cut_20               26      0.0556\n",
            "map_cut_30               26      0.0556\n",
            "map_cut_100              26      0.0556\n",
            "map_cut_200              26      0.0556\n",
            "map_cut_500              26      0.0556\n",
            "map_cut_1000             26      0.0556\n",
            "relative_P_5             26      0.0000\n",
            "relative_P_10            26      0.3333\n",
            "relative_P_15            26      0.3333\n",
            "relative_P_20            26      0.3333\n",
            "relative_P_30            26      0.3333\n",
            "relative_P_100           26      0.3333\n",
            "relative_P_200           26      0.3333\n",
            "relative_P_500           26      0.3333\n",
            "relative_P_1000          26      0.3333\n",
            "success_1                26      0.0000\n",
            "success_5                26      0.0000\n",
            "success_10               26      1.0000\n",
            "set_P                    26      0.0250\n",
            "set_relative_P           26      0.3333\n",
            "set_recall               26      0.3333\n",
            "set_map                  26      0.0083\n",
            "set_F                    26      0.0465\n",
            "num_nonrel_judged_ret    26      31.0000\n",
            "num_q                    28      1.0000\n",
            "num_ret                  28      36.0000\n",
            "num_rel                  28      2.0000\n",
            "num_rel_ret              28      2.0000\n",
            "map                      28      0.0698\n",
            "gm_map                   28      -2.6622\n",
            "Rprec                    28      0.0000\n",
            "bpref                    28      0.0000\n",
            "recip_rank               28      0.0526\n",
            "iprec_at_recall_0.00     28      0.0870\n",
            "iprec_at_recall_0.10     28      0.0870\n",
            "iprec_at_recall_0.20     28      0.0870\n",
            "iprec_at_recall_0.30     28      0.0870\n",
            "iprec_at_recall_0.40     28      0.0870\n",
            "iprec_at_recall_0.50     28      0.0870\n",
            "iprec_at_recall_0.60     28      0.0870\n",
            "iprec_at_recall_0.70     28      0.0870\n",
            "iprec_at_recall_0.80     28      0.0870\n",
            "iprec_at_recall_0.90     28      0.0870\n",
            "iprec_at_recall_1.00     28      0.0870\n",
            "P_5                      28      0.0000\n",
            "P_10                     28      0.0000\n",
            "P_15                     28      0.0000\n",
            "P_20                     28      0.0500\n",
            "P_30                     28      0.0667\n",
            "P_100                    28      0.0200\n",
            "P_200                    28      0.0100\n",
            "P_500                    28      0.0040\n",
            "P_1000                   28      0.0020\n",
            "relstring                28      0.0000\n",
            "recall_5                 28      0.0000\n",
            "recall_10                28      0.0000\n",
            "recall_15                28      0.0000\n",
            "recall_20                28      0.5000\n",
            "recall_30                28      1.0000\n",
            "recall_100               28      1.0000\n",
            "recall_200               28      1.0000\n",
            "recall_500               28      1.0000\n",
            "recall_1000              28      1.0000\n",
            "infAP                    28      0.0698\n",
            "gm_bpref                 28      -11.5129\n",
            "Rprec_mult_0.20          28      0.0000\n",
            "Rprec_mult_0.40          28      0.0000\n",
            "Rprec_mult_0.60          28      0.0000\n",
            "Rprec_mult_0.80          28      0.0000\n",
            "Rprec_mult_1.00          28      0.0000\n",
            "Rprec_mult_1.20          28      0.0000\n",
            "Rprec_mult_1.40          28      0.0000\n",
            "Rprec_mult_1.60          28      0.0000\n",
            "Rprec_mult_1.80          28      0.0000\n",
            "Rprec_mult_2.00          28      0.0000\n",
            "utility                  28      -32.0000\n",
            "11pt_avg                 28      0.0870\n",
            "binG                     28      0.2262\n",
            "G                        28      0.2262\n",
            "ndcg                     28      0.2756\n",
            "ndcg_rel                 28      0.2087\n",
            "Rndcg                    28      0.1378\n",
            "ndcg_cut_5               28      0.0000\n",
            "ndcg_cut_10              28      0.0000\n",
            "ndcg_cut_15              28      0.0000\n",
            "ndcg_cut_20              28      0.1419\n",
            "ndcg_cut_30              28      0.2756\n",
            "ndcg_cut_100             28      0.2756\n",
            "ndcg_cut_200             28      0.2756\n",
            "ndcg_cut_500             28      0.2756\n",
            "ndcg_cut_1000            28      0.2756\n",
            "map_cut_5                28      0.0000\n",
            "map_cut_10               28      0.0000\n",
            "map_cut_15               28      0.0000\n",
            "map_cut_20               28      0.0263\n",
            "map_cut_30               28      0.0698\n",
            "map_cut_100              28      0.0698\n",
            "map_cut_200              28      0.0698\n",
            "map_cut_500              28      0.0698\n",
            "map_cut_1000             28      0.0698\n",
            "relative_P_5             28      0.0000\n",
            "relative_P_10            28      0.0000\n",
            "relative_P_15            28      0.0000\n",
            "relative_P_20            28      0.5000\n",
            "relative_P_30            28      1.0000\n",
            "relative_P_100           28      1.0000\n",
            "relative_P_200           28      1.0000\n",
            "relative_P_500           28      1.0000\n",
            "relative_P_1000          28      1.0000\n",
            "success_1                28      0.0000\n",
            "success_5                28      0.0000\n",
            "success_10               28      0.0000\n",
            "set_P                    28      0.0556\n",
            "set_relative_P           28      1.0000\n",
            "set_recall               28      1.0000\n",
            "set_map                  28      0.0556\n",
            "set_F                    28      0.1053\n",
            "num_nonrel_judged_ret    28      25.0000\n",
            "num_q                    all     15.0000\n",
            "num_ret                  all     290.0000\n",
            "num_rel                  all     35.0000\n",
            "num_rel_ret              all     12.0000\n",
            "map                      all     0.3187\n",
            "gm_map                   all     0.0096\n",
            "Rprec                    all     0.3000\n",
            "bpref                    all     0.2833\n",
            "recip_rank               all     0.4083\n",
            "iprec_at_recall_0.00     all     0.4106\n",
            "iprec_at_recall_0.10     all     0.4106\n",
            "iprec_at_recall_0.20     all     0.4106\n",
            "iprec_at_recall_0.30     all     0.4106\n",
            "iprec_at_recall_0.40     all     0.3994\n",
            "iprec_at_recall_0.50     all     0.3994\n",
            "iprec_at_recall_0.60     all     0.2328\n",
            "iprec_at_recall_0.70     all     0.2328\n",
            "iprec_at_recall_0.80     all     0.2328\n",
            "iprec_at_recall_0.90     all     0.2328\n",
            "iprec_at_recall_1.00     all     0.2328\n",
            "P_5                      all     0.1067\n",
            "P_10                     all     0.0600\n",
            "P_15                     all     0.0444\n",
            "P_20                     all     0.0367\n",
            "P_30                     all     0.0267\n",
            "P_100                    all     0.0080\n",
            "P_200                    all     0.0040\n",
            "P_500                    all     0.0016\n",
            "P_1000                   all     0.0008\n",
            "relstring                all     0.0000\n",
            "recall_5                 all     0.3667\n",
            "recall_10                all     0.3889\n",
            "recall_15                all     0.4556\n",
            "recall_20                all     0.4889\n",
            "recall_30                all     0.5222\n",
            "recall_100               all     0.5222\n",
            "recall_200               all     0.5222\n",
            "recall_500               all     0.5222\n",
            "recall_1000              all     0.5222\n",
            "infAP                    all     0.3187\n",
            "gm_bpref                 all     0.0008\n",
            "Rprec_mult_0.20          all     0.3333\n",
            "Rprec_mult_0.40          all     0.3333\n",
            "Rprec_mult_0.60          all     0.3111\n",
            "Rprec_mult_0.80          all     0.3000\n",
            "Rprec_mult_1.00          all     0.3000\n",
            "Rprec_mult_1.20          all     0.1711\n",
            "Rprec_mult_1.40          all     0.1667\n",
            "Rprec_mult_1.60          all     0.1524\n",
            "Rprec_mult_1.80          all     0.1611\n",
            "Rprec_mult_2.00          all     0.1611\n",
            "utility                  all     -17.7333\n",
            "11pt_avg                 all     0.3277\n",
            "binG                     all     0.3611\n",
            "G                        all     0.3611\n",
            "ndcg                     all     0.3890\n",
            "ndcg_rel                 all     0.4096\n",
            "Rndcg                    all     0.3491\n",
            "ndcg_cut_5               all     0.3424\n",
            "ndcg_cut_10              all     0.3536\n",
            "ndcg_cut_15              all     0.3707\n",
            "ndcg_cut_20              all     0.3801\n",
            "ndcg_cut_30              all     0.3890\n",
            "ndcg_cut_100             all     0.3890\n",
            "ndcg_cut_200             all     0.3890\n",
            "ndcg_cut_500             all     0.3890\n",
            "ndcg_cut_1000            all     0.3890\n",
            "map_cut_5                all     0.3056\n",
            "map_cut_10               all     0.3093\n",
            "map_cut_15               all     0.3140\n",
            "map_cut_20               all     0.3158\n",
            "map_cut_30               all     0.3187\n",
            "map_cut_100              all     0.3187\n",
            "map_cut_200              all     0.3187\n",
            "map_cut_500              all     0.3187\n",
            "map_cut_1000             all     0.3187\n",
            "relative_P_5             all     0.3667\n",
            "relative_P_10            all     0.3889\n",
            "relative_P_15            all     0.4556\n",
            "relative_P_20            all     0.4889\n",
            "relative_P_30            all     0.5222\n",
            "relative_P_100           all     0.5222\n",
            "relative_P_200           all     0.5222\n",
            "relative_P_500           all     0.5222\n",
            "relative_P_1000          all     0.5222\n",
            "success_1                all     0.3333\n",
            "success_5                all     0.4667\n",
            "success_10               all     0.5333\n",
            "set_P                    all     0.0427\n",
            "set_relative_P           all     0.5222\n",
            "set_recall               all     0.5222\n",
            "set_map                  all     0.0303\n",
            "set_F                    all     0.0736\n",
            "num_nonrel_judged_ret    all     222.0000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mD4nxtDK39W7"
      },
      "source": [
        "\n",
        "I modified the analyzer used in schema so it can change upper case word to lower case, the overall map score increased from 0.1971 to 0.3187 and the false negative cases were imporved, because num_rel_ret is 2, which means the system can now retreive all relevant documents for topic 28. I tried to tune the parameter of BM25F() to reduce the effect of word frequency on scoring, but the false positive cases were not imporved."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hl4AUf0n39W9"
      },
      "source": [
        "\n",
        "I would say overall this is a good approach, even though it damaged the performance of some cases, but the overall map score increased, which means more relevant documents are retreived and their ranking is higher than before. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vs6teq2k39W-"
      },
      "source": [
        "## Part 3: Search engine optimization\n",
        "Try alternative techniques (tokenizer, filters, stemmers, and scoring functions, or other techniques) to improve performance."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2-ZpjRdN39XD"
      },
      "source": [
        "# Put your code for creating the index here (you can add more cells).\n",
        "# Make sure you save the final index in the variable INDEX_Q4, your query parser in QP_Q4, and your searcher in SEARCHER_Q4\n",
        "\n",
        "class CustomFilter(Filter):\n",
        "    is_morph = True\n",
        "    def __init__(self, filterFunc, *args, **kwargs):\n",
        "        self.customFilter = filterFunc\n",
        "        self.args = args\n",
        "        self.kwargs = kwargs\n",
        "    def __eq__(self):\n",
        "        return (other\n",
        "                and self.__class__ is other.__class__)\n",
        "    def __call__(self, tokens):\n",
        "        for t in tokens:\n",
        "            if t.mode == 'query': # if called by query parser\n",
        "                t.text = self.customFilter(t.text, *self.args, **self.kwargs)\n",
        "                yield t\n",
        "            else: # == 'index' if called by indexer\n",
        "                t.text = self.customFilter(t.text, *self.args, **self.kwargs)\n",
        "                yield t\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IC1X5ddN-qO2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "6d72a4ed-4159-43da-b4ea-e9bcf9dc77a4"
      },
      "source": [
        "#Remove stop words and change every word to lower case\n",
        "New_Analyzer=RegexTokenizer() | LowercaseFilter()| StopFilter()\n",
        "New_Schema=Schema(file_path = ID(stored=True), file_content = TEXT(analyzer = New_Analyzer))\n",
        "NEW_INDEX = createIndex(New_Schema)\n",
        "addFilesToIndex(NEW_INDEX, filesToIndex)\n",
        "NEW_QP = QueryParser(\"file_content\", schema=NEW_INDEX.schema)\n",
        "NEW_SEARCHER = NEW_INDEX.searcher()\n",
        "pyTrecEval(topic_file, qrels_file, NEW_QP, NEW_SEARCHER) \n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "num_q                    1       1.0000\n",
            "num_ret                  1       3.0000\n",
            "num_rel                  1       5.0000\n",
            "num_rel_ret              1       0.0000\n",
            "map                      1       0.0000\n",
            "gm_map                   1       -11.5129\n",
            "Rprec                    1       0.0000\n",
            "bpref                    1       0.0000\n",
            "recip_rank               1       0.0000\n",
            "iprec_at_recall_0.00     1       0.0000\n",
            "iprec_at_recall_0.10     1       0.0000\n",
            "iprec_at_recall_0.20     1       0.0000\n",
            "iprec_at_recall_0.30     1       0.0000\n",
            "iprec_at_recall_0.40     1       0.0000\n",
            "iprec_at_recall_0.50     1       0.0000\n",
            "iprec_at_recall_0.60     1       0.0000\n",
            "iprec_at_recall_0.70     1       0.0000\n",
            "iprec_at_recall_0.80     1       0.0000\n",
            "iprec_at_recall_0.90     1       0.0000\n",
            "iprec_at_recall_1.00     1       0.0000\n",
            "P_5                      1       0.0000\n",
            "P_10                     1       0.0000\n",
            "P_15                     1       0.0000\n",
            "P_20                     1       0.0000\n",
            "P_30                     1       0.0000\n",
            "P_100                    1       0.0000\n",
            "P_200                    1       0.0000\n",
            "P_500                    1       0.0000\n",
            "P_1000                   1       0.0000\n",
            "relstring                1       0.0000\n",
            "recall_5                 1       0.0000\n",
            "recall_10                1       0.0000\n",
            "recall_15                1       0.0000\n",
            "recall_20                1       0.0000\n",
            "recall_30                1       0.0000\n",
            "recall_100               1       0.0000\n",
            "recall_200               1       0.0000\n",
            "recall_500               1       0.0000\n",
            "recall_1000              1       0.0000\n",
            "infAP                    1       0.0000\n",
            "gm_bpref                 1       -11.5129\n",
            "Rprec_mult_0.20          1       0.0000\n",
            "Rprec_mult_0.40          1       0.0000\n",
            "Rprec_mult_0.60          1       0.0000\n",
            "Rprec_mult_0.80          1       0.0000\n",
            "Rprec_mult_1.00          1       0.0000\n",
            "Rprec_mult_1.20          1       0.0000\n",
            "Rprec_mult_1.40          1       0.0000\n",
            "Rprec_mult_1.60          1       0.0000\n",
            "Rprec_mult_1.80          1       0.0000\n",
            "Rprec_mult_2.00          1       0.0000\n",
            "utility                  1       -3.0000\n",
            "11pt_avg                 1       0.0000\n",
            "binG                     1       0.0000\n",
            "G                        1       0.0000\n",
            "ndcg                     1       0.0000\n",
            "ndcg_rel                 1       0.0000\n",
            "Rndcg                    1       0.0000\n",
            "ndcg_cut_5               1       0.0000\n",
            "ndcg_cut_10              1       0.0000\n",
            "ndcg_cut_15              1       0.0000\n",
            "ndcg_cut_20              1       0.0000\n",
            "ndcg_cut_30              1       0.0000\n",
            "ndcg_cut_100             1       0.0000\n",
            "ndcg_cut_200             1       0.0000\n",
            "ndcg_cut_500             1       0.0000\n",
            "ndcg_cut_1000            1       0.0000\n",
            "map_cut_5                1       0.0000\n",
            "map_cut_10               1       0.0000\n",
            "map_cut_15               1       0.0000\n",
            "map_cut_20               1       0.0000\n",
            "map_cut_30               1       0.0000\n",
            "map_cut_100              1       0.0000\n",
            "map_cut_200              1       0.0000\n",
            "map_cut_500              1       0.0000\n",
            "map_cut_1000             1       0.0000\n",
            "relative_P_5             1       0.0000\n",
            "relative_P_10            1       0.0000\n",
            "relative_P_15            1       0.0000\n",
            "relative_P_20            1       0.0000\n",
            "relative_P_30            1       0.0000\n",
            "relative_P_100           1       0.0000\n",
            "relative_P_200           1       0.0000\n",
            "relative_P_500           1       0.0000\n",
            "relative_P_1000          1       0.0000\n",
            "success_1                1       0.0000\n",
            "success_5                1       0.0000\n",
            "success_10               1       0.0000\n",
            "set_P                    1       0.0000\n",
            "set_relative_P           1       0.0000\n",
            "set_recall               1       0.0000\n",
            "set_map                  1       0.0000\n",
            "set_F                    1       0.0000\n",
            "num_nonrel_judged_ret    1       3.0000\n",
            "num_q                    2       1.0000\n",
            "num_ret                  2       13.0000\n",
            "num_rel                  2       2.0000\n",
            "num_rel_ret              2       1.0000\n",
            "map                      2       0.5000\n",
            "gm_map                   2       -0.6931\n",
            "Rprec                    2       0.5000\n",
            "bpref                    2       0.5000\n",
            "recip_rank               2       1.0000\n",
            "iprec_at_recall_0.00     2       1.0000\n",
            "iprec_at_recall_0.10     2       1.0000\n",
            "iprec_at_recall_0.20     2       1.0000\n",
            "iprec_at_recall_0.30     2       1.0000\n",
            "iprec_at_recall_0.40     2       1.0000\n",
            "iprec_at_recall_0.50     2       1.0000\n",
            "iprec_at_recall_0.60     2       0.0000\n",
            "iprec_at_recall_0.70     2       0.0000\n",
            "iprec_at_recall_0.80     2       0.0000\n",
            "iprec_at_recall_0.90     2       0.0000\n",
            "iprec_at_recall_1.00     2       0.0000\n",
            "P_5                      2       0.2000\n",
            "P_10                     2       0.1000\n",
            "P_15                     2       0.0667\n",
            "P_20                     2       0.0500\n",
            "P_30                     2       0.0333\n",
            "P_100                    2       0.0100\n",
            "P_200                    2       0.0050\n",
            "P_500                    2       0.0020\n",
            "P_1000                   2       0.0010\n",
            "relstring                2       0.0000\n",
            "recall_5                 2       0.5000\n",
            "recall_10                2       0.5000\n",
            "recall_15                2       0.5000\n",
            "recall_20                2       0.5000\n",
            "recall_30                2       0.5000\n",
            "recall_100               2       0.5000\n",
            "recall_200               2       0.5000\n",
            "recall_500               2       0.5000\n",
            "recall_1000              2       0.5000\n",
            "infAP                    2       0.5000\n",
            "gm_bpref                 2       -0.6931\n",
            "Rprec_mult_0.20          2       1.0000\n",
            "Rprec_mult_0.40          2       1.0000\n",
            "Rprec_mult_0.60          2       0.5000\n",
            "Rprec_mult_0.80          2       0.5000\n",
            "Rprec_mult_1.00          2       0.5000\n",
            "Rprec_mult_1.20          2       0.3333\n",
            "Rprec_mult_1.40          2       0.3333\n",
            "Rprec_mult_1.60          2       0.2500\n",
            "Rprec_mult_1.80          2       0.2500\n",
            "Rprec_mult_2.00          2       0.2500\n",
            "utility                  2       -11.0000\n",
            "11pt_avg                 2       0.5455\n",
            "binG                     2       0.5000\n",
            "G                        2       0.5000\n",
            "ndcg                     2       0.6131\n",
            "ndcg_rel                 2       0.8066\n",
            "Rndcg                    2       0.6131\n",
            "ndcg_cut_5               2       0.6131\n",
            "ndcg_cut_10              2       0.6131\n",
            "ndcg_cut_15              2       0.6131\n",
            "ndcg_cut_20              2       0.6131\n",
            "ndcg_cut_30              2       0.6131\n",
            "ndcg_cut_100             2       0.6131\n",
            "ndcg_cut_200             2       0.6131\n",
            "ndcg_cut_500             2       0.6131\n",
            "ndcg_cut_1000            2       0.6131\n",
            "map_cut_5                2       0.5000\n",
            "map_cut_10               2       0.5000\n",
            "map_cut_15               2       0.5000\n",
            "map_cut_20               2       0.5000\n",
            "map_cut_30               2       0.5000\n",
            "map_cut_100              2       0.5000\n",
            "map_cut_200              2       0.5000\n",
            "map_cut_500              2       0.5000\n",
            "map_cut_1000             2       0.5000\n",
            "relative_P_5             2       0.5000\n",
            "relative_P_10            2       0.5000\n",
            "relative_P_15            2       0.5000\n",
            "relative_P_20            2       0.5000\n",
            "relative_P_30            2       0.5000\n",
            "relative_P_100           2       0.5000\n",
            "relative_P_200           2       0.5000\n",
            "relative_P_500           2       0.5000\n",
            "relative_P_1000          2       0.5000\n",
            "success_1                2       1.0000\n",
            "success_5                2       1.0000\n",
            "success_10               2       1.0000\n",
            "set_P                    2       0.0769\n",
            "set_relative_P           2       0.5000\n",
            "set_recall               2       0.5000\n",
            "set_map                  2       0.0385\n",
            "set_F                    2       0.1333\n",
            "num_nonrel_judged_ret    2       9.0000\n",
            "num_q                    4       1.0000\n",
            "num_ret                  4       32.0000\n",
            "num_rel                  4       4.0000\n",
            "num_rel_ret              4       2.0000\n",
            "map                      4       0.5000\n",
            "gm_map                   4       -0.6931\n",
            "Rprec                    4       0.5000\n",
            "bpref                    4       0.5000\n",
            "recip_rank               4       1.0000\n",
            "iprec_at_recall_0.00     4       1.0000\n",
            "iprec_at_recall_0.10     4       1.0000\n",
            "iprec_at_recall_0.20     4       1.0000\n",
            "iprec_at_recall_0.30     4       1.0000\n",
            "iprec_at_recall_0.40     4       1.0000\n",
            "iprec_at_recall_0.50     4       1.0000\n",
            "iprec_at_recall_0.60     4       0.0000\n",
            "iprec_at_recall_0.70     4       0.0000\n",
            "iprec_at_recall_0.80     4       0.0000\n",
            "iprec_at_recall_0.90     4       0.0000\n",
            "iprec_at_recall_1.00     4       0.0000\n",
            "P_5                      4       0.4000\n",
            "P_10                     4       0.2000\n",
            "P_15                     4       0.1333\n",
            "P_20                     4       0.1000\n",
            "P_30                     4       0.0667\n",
            "P_100                    4       0.0200\n",
            "P_200                    4       0.0100\n",
            "P_500                    4       0.0040\n",
            "P_1000                   4       0.0020\n",
            "relstring                4       0.0000\n",
            "recall_5                 4       0.5000\n",
            "recall_10                4       0.5000\n",
            "recall_15                4       0.5000\n",
            "recall_20                4       0.5000\n",
            "recall_30                4       0.5000\n",
            "recall_100               4       0.5000\n",
            "recall_200               4       0.5000\n",
            "recall_500               4       0.5000\n",
            "recall_1000              4       0.5000\n",
            "infAP                    4       0.5000\n",
            "gm_bpref                 4       -0.6931\n",
            "Rprec_mult_0.20          4       1.0000\n",
            "Rprec_mult_0.40          4       1.0000\n",
            "Rprec_mult_0.60          4       0.6667\n",
            "Rprec_mult_0.80          4       0.5000\n",
            "Rprec_mult_1.00          4       0.5000\n",
            "Rprec_mult_1.20          4       0.4000\n",
            "Rprec_mult_1.40          4       0.3333\n",
            "Rprec_mult_1.60          4       0.2857\n",
            "Rprec_mult_1.80          4       0.2500\n",
            "Rprec_mult_2.00          4       0.2500\n",
            "utility                  4       -28.0000\n",
            "11pt_avg                 4       0.5455\n",
            "binG                     4       0.5000\n",
            "G                        4       0.5000\n",
            "ndcg                     4       0.6367\n",
            "ndcg_rel                 4       0.8183\n",
            "Rndcg                    4       0.6367\n",
            "ndcg_cut_5               4       0.6367\n",
            "ndcg_cut_10              4       0.6367\n",
            "ndcg_cut_15              4       0.6367\n",
            "ndcg_cut_20              4       0.6367\n",
            "ndcg_cut_30              4       0.6367\n",
            "ndcg_cut_100             4       0.6367\n",
            "ndcg_cut_200             4       0.6367\n",
            "ndcg_cut_500             4       0.6367\n",
            "ndcg_cut_1000            4       0.6367\n",
            "map_cut_5                4       0.5000\n",
            "map_cut_10               4       0.5000\n",
            "map_cut_15               4       0.5000\n",
            "map_cut_20               4       0.5000\n",
            "map_cut_30               4       0.5000\n",
            "map_cut_100              4       0.5000\n",
            "map_cut_200              4       0.5000\n",
            "map_cut_500              4       0.5000\n",
            "map_cut_1000             4       0.5000\n",
            "relative_P_5             4       0.5000\n",
            "relative_P_10            4       0.5000\n",
            "relative_P_15            4       0.5000\n",
            "relative_P_20            4       0.5000\n",
            "relative_P_30            4       0.5000\n",
            "relative_P_100           4       0.5000\n",
            "relative_P_200           4       0.5000\n",
            "relative_P_500           4       0.5000\n",
            "relative_P_1000          4       0.5000\n",
            "success_1                4       1.0000\n",
            "success_5                4       1.0000\n",
            "success_10               4       1.0000\n",
            "set_P                    4       0.0625\n",
            "set_relative_P           4       0.5000\n",
            "set_recall               4       0.5000\n",
            "set_map                  4       0.0312\n",
            "set_F                    4       0.1111\n",
            "num_nonrel_judged_ret    4       23.0000\n",
            "num_q                    6       1.0000\n",
            "num_ret                  6       9.0000\n",
            "num_rel                  6       1.0000\n",
            "num_rel_ret              6       0.0000\n",
            "map                      6       0.0000\n",
            "gm_map                   6       -11.5129\n",
            "Rprec                    6       0.0000\n",
            "bpref                    6       0.0000\n",
            "recip_rank               6       0.0000\n",
            "iprec_at_recall_0.00     6       0.0000\n",
            "iprec_at_recall_0.10     6       0.0000\n",
            "iprec_at_recall_0.20     6       0.0000\n",
            "iprec_at_recall_0.30     6       0.0000\n",
            "iprec_at_recall_0.40     6       0.0000\n",
            "iprec_at_recall_0.50     6       0.0000\n",
            "iprec_at_recall_0.60     6       0.0000\n",
            "iprec_at_recall_0.70     6       0.0000\n",
            "iprec_at_recall_0.80     6       0.0000\n",
            "iprec_at_recall_0.90     6       0.0000\n",
            "iprec_at_recall_1.00     6       0.0000\n",
            "P_5                      6       0.0000\n",
            "P_10                     6       0.0000\n",
            "P_15                     6       0.0000\n",
            "P_20                     6       0.0000\n",
            "P_30                     6       0.0000\n",
            "P_100                    6       0.0000\n",
            "P_200                    6       0.0000\n",
            "P_500                    6       0.0000\n",
            "P_1000                   6       0.0000\n",
            "relstring                6       0.0000\n",
            "recall_5                 6       0.0000\n",
            "recall_10                6       0.0000\n",
            "recall_15                6       0.0000\n",
            "recall_20                6       0.0000\n",
            "recall_30                6       0.0000\n",
            "recall_100               6       0.0000\n",
            "recall_200               6       0.0000\n",
            "recall_500               6       0.0000\n",
            "recall_1000              6       0.0000\n",
            "infAP                    6       0.0000\n",
            "gm_bpref                 6       -11.5129\n",
            "Rprec_mult_0.20          6       0.0000\n",
            "Rprec_mult_0.40          6       0.0000\n",
            "Rprec_mult_0.60          6       0.0000\n",
            "Rprec_mult_0.80          6       0.0000\n",
            "Rprec_mult_1.00          6       0.0000\n",
            "Rprec_mult_1.20          6       0.0000\n",
            "Rprec_mult_1.40          6       0.0000\n",
            "Rprec_mult_1.60          6       0.0000\n",
            "Rprec_mult_1.80          6       0.0000\n",
            "Rprec_mult_2.00          6       0.0000\n",
            "utility                  6       -9.0000\n",
            "11pt_avg                 6       0.0000\n",
            "binG                     6       0.0000\n",
            "G                        6       0.0000\n",
            "ndcg                     6       0.0000\n",
            "ndcg_rel                 6       0.0000\n",
            "Rndcg                    6       0.0000\n",
            "ndcg_cut_5               6       0.0000\n",
            "ndcg_cut_10              6       0.0000\n",
            "ndcg_cut_15              6       0.0000\n",
            "ndcg_cut_20              6       0.0000\n",
            "ndcg_cut_30              6       0.0000\n",
            "ndcg_cut_100             6       0.0000\n",
            "ndcg_cut_200             6       0.0000\n",
            "ndcg_cut_500             6       0.0000\n",
            "ndcg_cut_1000            6       0.0000\n",
            "map_cut_5                6       0.0000\n",
            "map_cut_10               6       0.0000\n",
            "map_cut_15               6       0.0000\n",
            "map_cut_20               6       0.0000\n",
            "map_cut_30               6       0.0000\n",
            "map_cut_100              6       0.0000\n",
            "map_cut_200              6       0.0000\n",
            "map_cut_500              6       0.0000\n",
            "map_cut_1000             6       0.0000\n",
            "relative_P_5             6       0.0000\n",
            "relative_P_10            6       0.0000\n",
            "relative_P_15            6       0.0000\n",
            "relative_P_20            6       0.0000\n",
            "relative_P_30            6       0.0000\n",
            "relative_P_100           6       0.0000\n",
            "relative_P_200           6       0.0000\n",
            "relative_P_500           6       0.0000\n",
            "relative_P_1000          6       0.0000\n",
            "success_1                6       0.0000\n",
            "success_5                6       0.0000\n",
            "success_10               6       0.0000\n",
            "set_P                    6       0.0000\n",
            "set_relative_P           6       0.0000\n",
            "set_recall               6       0.0000\n",
            "set_map                  6       0.0000\n",
            "set_F                    6       0.0000\n",
            "num_nonrel_judged_ret    6       8.0000\n",
            "num_q                    7       1.0000\n",
            "num_ret                  7       14.0000\n",
            "num_rel                  7       3.0000\n",
            "num_rel_ret              7       0.0000\n",
            "map                      7       0.0000\n",
            "gm_map                   7       -11.5129\n",
            "Rprec                    7       0.0000\n",
            "bpref                    7       0.0000\n",
            "recip_rank               7       0.0000\n",
            "iprec_at_recall_0.00     7       0.0000\n",
            "iprec_at_recall_0.10     7       0.0000\n",
            "iprec_at_recall_0.20     7       0.0000\n",
            "iprec_at_recall_0.30     7       0.0000\n",
            "iprec_at_recall_0.40     7       0.0000\n",
            "iprec_at_recall_0.50     7       0.0000\n",
            "iprec_at_recall_0.60     7       0.0000\n",
            "iprec_at_recall_0.70     7       0.0000\n",
            "iprec_at_recall_0.80     7       0.0000\n",
            "iprec_at_recall_0.90     7       0.0000\n",
            "iprec_at_recall_1.00     7       0.0000\n",
            "P_5                      7       0.0000\n",
            "P_10                     7       0.0000\n",
            "P_15                     7       0.0000\n",
            "P_20                     7       0.0000\n",
            "P_30                     7       0.0000\n",
            "P_100                    7       0.0000\n",
            "P_200                    7       0.0000\n",
            "P_500                    7       0.0000\n",
            "P_1000                   7       0.0000\n",
            "relstring                7       0.0000\n",
            "recall_5                 7       0.0000\n",
            "recall_10                7       0.0000\n",
            "recall_15                7       0.0000\n",
            "recall_20                7       0.0000\n",
            "recall_30                7       0.0000\n",
            "recall_100               7       0.0000\n",
            "recall_200               7       0.0000\n",
            "recall_500               7       0.0000\n",
            "recall_1000              7       0.0000\n",
            "infAP                    7       0.0000\n",
            "gm_bpref                 7       -11.5129\n",
            "Rprec_mult_0.20          7       0.0000\n",
            "Rprec_mult_0.40          7       0.0000\n",
            "Rprec_mult_0.60          7       0.0000\n",
            "Rprec_mult_0.80          7       0.0000\n",
            "Rprec_mult_1.00          7       0.0000\n",
            "Rprec_mult_1.20          7       0.0000\n",
            "Rprec_mult_1.40          7       0.0000\n",
            "Rprec_mult_1.60          7       0.0000\n",
            "Rprec_mult_1.80          7       0.0000\n",
            "Rprec_mult_2.00          7       0.0000\n",
            "utility                  7       -14.0000\n",
            "11pt_avg                 7       0.0000\n",
            "binG                     7       0.0000\n",
            "G                        7       0.0000\n",
            "ndcg                     7       0.0000\n",
            "ndcg_rel                 7       0.0000\n",
            "Rndcg                    7       0.0000\n",
            "ndcg_cut_5               7       0.0000\n",
            "ndcg_cut_10              7       0.0000\n",
            "ndcg_cut_15              7       0.0000\n",
            "ndcg_cut_20              7       0.0000\n",
            "ndcg_cut_30              7       0.0000\n",
            "ndcg_cut_100             7       0.0000\n",
            "ndcg_cut_200             7       0.0000\n",
            "ndcg_cut_500             7       0.0000\n",
            "ndcg_cut_1000            7       0.0000\n",
            "map_cut_5                7       0.0000\n",
            "map_cut_10               7       0.0000\n",
            "map_cut_15               7       0.0000\n",
            "map_cut_20               7       0.0000\n",
            "map_cut_30               7       0.0000\n",
            "map_cut_100              7       0.0000\n",
            "map_cut_200              7       0.0000\n",
            "map_cut_500              7       0.0000\n",
            "map_cut_1000             7       0.0000\n",
            "relative_P_5             7       0.0000\n",
            "relative_P_10            7       0.0000\n",
            "relative_P_15            7       0.0000\n",
            "relative_P_20            7       0.0000\n",
            "relative_P_30            7       0.0000\n",
            "relative_P_100           7       0.0000\n",
            "relative_P_200           7       0.0000\n",
            "relative_P_500           7       0.0000\n",
            "relative_P_1000          7       0.0000\n",
            "success_1                7       0.0000\n",
            "success_5                7       0.0000\n",
            "success_10               7       0.0000\n",
            "set_P                    7       0.0000\n",
            "set_relative_P           7       0.0000\n",
            "set_recall               7       0.0000\n",
            "set_map                  7       0.0000\n",
            "set_F                    7       0.0000\n",
            "num_nonrel_judged_ret    7       11.0000\n",
            "num_q                    9       1.0000\n",
            "num_ret                  9       2.0000\n",
            "num_rel                  9       1.0000\n",
            "num_rel_ret              9       0.0000\n",
            "map                      9       0.0000\n",
            "gm_map                   9       -11.5129\n",
            "Rprec                    9       0.0000\n",
            "bpref                    9       0.0000\n",
            "recip_rank               9       0.0000\n",
            "iprec_at_recall_0.00     9       0.0000\n",
            "iprec_at_recall_0.10     9       0.0000\n",
            "iprec_at_recall_0.20     9       0.0000\n",
            "iprec_at_recall_0.30     9       0.0000\n",
            "iprec_at_recall_0.40     9       0.0000\n",
            "iprec_at_recall_0.50     9       0.0000\n",
            "iprec_at_recall_0.60     9       0.0000\n",
            "iprec_at_recall_0.70     9       0.0000\n",
            "iprec_at_recall_0.80     9       0.0000\n",
            "iprec_at_recall_0.90     9       0.0000\n",
            "iprec_at_recall_1.00     9       0.0000\n",
            "P_5                      9       0.0000\n",
            "P_10                     9       0.0000\n",
            "P_15                     9       0.0000\n",
            "P_20                     9       0.0000\n",
            "P_30                     9       0.0000\n",
            "P_100                    9       0.0000\n",
            "P_200                    9       0.0000\n",
            "P_500                    9       0.0000\n",
            "P_1000                   9       0.0000\n",
            "relstring                9       0.0000\n",
            "recall_5                 9       0.0000\n",
            "recall_10                9       0.0000\n",
            "recall_15                9       0.0000\n",
            "recall_20                9       0.0000\n",
            "recall_30                9       0.0000\n",
            "recall_100               9       0.0000\n",
            "recall_200               9       0.0000\n",
            "recall_500               9       0.0000\n",
            "recall_1000              9       0.0000\n",
            "infAP                    9       0.0000\n",
            "gm_bpref                 9       -11.5129\n",
            "Rprec_mult_0.20          9       0.0000\n",
            "Rprec_mult_0.40          9       0.0000\n",
            "Rprec_mult_0.60          9       0.0000\n",
            "Rprec_mult_0.80          9       0.0000\n",
            "Rprec_mult_1.00          9       0.0000\n",
            "Rprec_mult_1.20          9       0.0000\n",
            "Rprec_mult_1.40          9       0.0000\n",
            "Rprec_mult_1.60          9       0.0000\n",
            "Rprec_mult_1.80          9       0.0000\n",
            "Rprec_mult_2.00          9       0.0000\n",
            "utility                  9       -2.0000\n",
            "11pt_avg                 9       0.0000\n",
            "binG                     9       0.0000\n",
            "G                        9       0.0000\n",
            "ndcg                     9       0.0000\n",
            "ndcg_rel                 9       0.0000\n",
            "Rndcg                    9       0.0000\n",
            "ndcg_cut_5               9       0.0000\n",
            "ndcg_cut_10              9       0.0000\n",
            "ndcg_cut_15              9       0.0000\n",
            "ndcg_cut_20              9       0.0000\n",
            "ndcg_cut_30              9       0.0000\n",
            "ndcg_cut_100             9       0.0000\n",
            "ndcg_cut_200             9       0.0000\n",
            "ndcg_cut_500             9       0.0000\n",
            "ndcg_cut_1000            9       0.0000\n",
            "map_cut_5                9       0.0000\n",
            "map_cut_10               9       0.0000\n",
            "map_cut_15               9       0.0000\n",
            "map_cut_20               9       0.0000\n",
            "map_cut_30               9       0.0000\n",
            "map_cut_100              9       0.0000\n",
            "map_cut_200              9       0.0000\n",
            "map_cut_500              9       0.0000\n",
            "map_cut_1000             9       0.0000\n",
            "relative_P_5             9       0.0000\n",
            "relative_P_10            9       0.0000\n",
            "relative_P_15            9       0.0000\n",
            "relative_P_20            9       0.0000\n",
            "relative_P_30            9       0.0000\n",
            "relative_P_100           9       0.0000\n",
            "relative_P_200           9       0.0000\n",
            "relative_P_500           9       0.0000\n",
            "relative_P_1000          9       0.0000\n",
            "success_1                9       0.0000\n",
            "success_5                9       0.0000\n",
            "success_10               9       0.0000\n",
            "set_P                    9       0.0000\n",
            "set_relative_P           9       0.0000\n",
            "set_recall               9       0.0000\n",
            "set_map                  9       0.0000\n",
            "set_F                    9       0.0000\n",
            "num_nonrel_judged_ret    9       2.0000\n",
            "num_q                    10      1.0000\n",
            "num_ret                  10      24.0000\n",
            "num_rel                  10      1.0000\n",
            "num_rel_ret              10      1.0000\n",
            "map                      10      0.3333\n",
            "gm_map                   10      -1.0986\n",
            "Rprec                    10      0.0000\n",
            "bpref                    10      0.0000\n",
            "recip_rank               10      0.3333\n",
            "iprec_at_recall_0.00     10      0.3333\n",
            "iprec_at_recall_0.10     10      0.3333\n",
            "iprec_at_recall_0.20     10      0.3333\n",
            "iprec_at_recall_0.30     10      0.3333\n",
            "iprec_at_recall_0.40     10      0.3333\n",
            "iprec_at_recall_0.50     10      0.3333\n",
            "iprec_at_recall_0.60     10      0.3333\n",
            "iprec_at_recall_0.70     10      0.3333\n",
            "iprec_at_recall_0.80     10      0.3333\n",
            "iprec_at_recall_0.90     10      0.3333\n",
            "iprec_at_recall_1.00     10      0.3333\n",
            "P_5                      10      0.2000\n",
            "P_10                     10      0.1000\n",
            "P_15                     10      0.0667\n",
            "P_20                     10      0.0500\n",
            "P_30                     10      0.0333\n",
            "P_100                    10      0.0100\n",
            "P_200                    10      0.0050\n",
            "P_500                    10      0.0020\n",
            "P_1000                   10      0.0010\n",
            "relstring                10      0.0000\n",
            "recall_5                 10      1.0000\n",
            "recall_10                10      1.0000\n",
            "recall_15                10      1.0000\n",
            "recall_20                10      1.0000\n",
            "recall_30                10      1.0000\n",
            "recall_100               10      1.0000\n",
            "recall_200               10      1.0000\n",
            "recall_500               10      1.0000\n",
            "recall_1000              10      1.0000\n",
            "infAP                    10      0.3333\n",
            "gm_bpref                 10      -11.5129\n",
            "Rprec_mult_0.20          10      0.0000\n",
            "Rprec_mult_0.40          10      0.0000\n",
            "Rprec_mult_0.60          10      0.0000\n",
            "Rprec_mult_0.80          10      0.0000\n",
            "Rprec_mult_1.00          10      0.0000\n",
            "Rprec_mult_1.20          10      0.0000\n",
            "Rprec_mult_1.40          10      0.0000\n",
            "Rprec_mult_1.60          10      0.0000\n",
            "Rprec_mult_1.80          10      0.0000\n",
            "Rprec_mult_2.00          10      0.0000\n",
            "utility                  10      -22.0000\n",
            "11pt_avg                 10      0.3333\n",
            "binG                     10      0.5000\n",
            "G                        10      0.5000\n",
            "ndcg                     10      0.5000\n",
            "ndcg_rel                 10      0.5000\n",
            "Rndcg                    10      0.2500\n",
            "ndcg_cut_5               10      0.5000\n",
            "ndcg_cut_10              10      0.5000\n",
            "ndcg_cut_15              10      0.5000\n",
            "ndcg_cut_20              10      0.5000\n",
            "ndcg_cut_30              10      0.5000\n",
            "ndcg_cut_100             10      0.5000\n",
            "ndcg_cut_200             10      0.5000\n",
            "ndcg_cut_500             10      0.5000\n",
            "ndcg_cut_1000            10      0.5000\n",
            "map_cut_5                10      0.3333\n",
            "map_cut_10               10      0.3333\n",
            "map_cut_15               10      0.3333\n",
            "map_cut_20               10      0.3333\n",
            "map_cut_30               10      0.3333\n",
            "map_cut_100              10      0.3333\n",
            "map_cut_200              10      0.3333\n",
            "map_cut_500              10      0.3333\n",
            "map_cut_1000             10      0.3333\n",
            "relative_P_5             10      1.0000\n",
            "relative_P_10            10      1.0000\n",
            "relative_P_15            10      1.0000\n",
            "relative_P_20            10      1.0000\n",
            "relative_P_30            10      1.0000\n",
            "relative_P_100           10      1.0000\n",
            "relative_P_200           10      1.0000\n",
            "relative_P_500           10      1.0000\n",
            "relative_P_1000          10      1.0000\n",
            "success_1                10      0.0000\n",
            "success_5                10      1.0000\n",
            "success_10               10      1.0000\n",
            "set_P                    10      0.0417\n",
            "set_relative_P           10      1.0000\n",
            "set_recall               10      1.0000\n",
            "set_map                  10      0.0417\n",
            "set_F                    10      0.0800\n",
            "num_nonrel_judged_ret    10      11.0000\n",
            "num_q                    14      1.0000\n",
            "num_ret                  14      24.0000\n",
            "num_rel                  14      1.0000\n",
            "num_rel_ret              14      1.0000\n",
            "map                      14      1.0000\n",
            "gm_map                   14      0.0000\n",
            "Rprec                    14      1.0000\n",
            "bpref                    14      1.0000\n",
            "recip_rank               14      1.0000\n",
            "iprec_at_recall_0.00     14      1.0000\n",
            "iprec_at_recall_0.10     14      1.0000\n",
            "iprec_at_recall_0.20     14      1.0000\n",
            "iprec_at_recall_0.30     14      1.0000\n",
            "iprec_at_recall_0.40     14      1.0000\n",
            "iprec_at_recall_0.50     14      1.0000\n",
            "iprec_at_recall_0.60     14      1.0000\n",
            "iprec_at_recall_0.70     14      1.0000\n",
            "iprec_at_recall_0.80     14      1.0000\n",
            "iprec_at_recall_0.90     14      1.0000\n",
            "iprec_at_recall_1.00     14      1.0000\n",
            "P_5                      14      0.2000\n",
            "P_10                     14      0.1000\n",
            "P_15                     14      0.0667\n",
            "P_20                     14      0.0500\n",
            "P_30                     14      0.0333\n",
            "P_100                    14      0.0100\n",
            "P_200                    14      0.0050\n",
            "P_500                    14      0.0020\n",
            "P_1000                   14      0.0010\n",
            "relstring                14      0.0000\n",
            "recall_5                 14      1.0000\n",
            "recall_10                14      1.0000\n",
            "recall_15                14      1.0000\n",
            "recall_20                14      1.0000\n",
            "recall_30                14      1.0000\n",
            "recall_100               14      1.0000\n",
            "recall_200               14      1.0000\n",
            "recall_500               14      1.0000\n",
            "recall_1000              14      1.0000\n",
            "infAP                    14      1.0000\n",
            "gm_bpref                 14      0.0000\n",
            "Rprec_mult_0.20          14      1.0000\n",
            "Rprec_mult_0.40          14      1.0000\n",
            "Rprec_mult_0.60          14      1.0000\n",
            "Rprec_mult_0.80          14      1.0000\n",
            "Rprec_mult_1.00          14      1.0000\n",
            "Rprec_mult_1.20          14      0.5000\n",
            "Rprec_mult_1.40          14      0.5000\n",
            "Rprec_mult_1.60          14      0.5000\n",
            "Rprec_mult_1.80          14      0.5000\n",
            "Rprec_mult_2.00          14      0.5000\n",
            "utility                  14      -22.0000\n",
            "11pt_avg                 14      1.0000\n",
            "binG                     14      1.0000\n",
            "G                        14      1.0000\n",
            "ndcg                     14      1.0000\n",
            "ndcg_rel                 14      1.0000\n",
            "Rndcg                    14      1.0000\n",
            "ndcg_cut_5               14      1.0000\n",
            "ndcg_cut_10              14      1.0000\n",
            "ndcg_cut_15              14      1.0000\n",
            "ndcg_cut_20              14      1.0000\n",
            "ndcg_cut_30              14      1.0000\n",
            "ndcg_cut_100             14      1.0000\n",
            "ndcg_cut_200             14      1.0000\n",
            "ndcg_cut_500             14      1.0000\n",
            "ndcg_cut_1000            14      1.0000\n",
            "map_cut_5                14      1.0000\n",
            "map_cut_10               14      1.0000\n",
            "map_cut_15               14      1.0000\n",
            "map_cut_20               14      1.0000\n",
            "map_cut_30               14      1.0000\n",
            "map_cut_100              14      1.0000\n",
            "map_cut_200              14      1.0000\n",
            "map_cut_500              14      1.0000\n",
            "map_cut_1000             14      1.0000\n",
            "relative_P_5             14      1.0000\n",
            "relative_P_10            14      1.0000\n",
            "relative_P_15            14      1.0000\n",
            "relative_P_20            14      1.0000\n",
            "relative_P_30            14      1.0000\n",
            "relative_P_100           14      1.0000\n",
            "relative_P_200           14      1.0000\n",
            "relative_P_500           14      1.0000\n",
            "relative_P_1000          14      1.0000\n",
            "success_1                14      1.0000\n",
            "success_5                14      1.0000\n",
            "success_10               14      1.0000\n",
            "set_P                    14      0.0417\n",
            "set_relative_P           14      1.0000\n",
            "set_recall               14      1.0000\n",
            "set_map                  14      0.0417\n",
            "set_F                    14      0.0800\n",
            "num_nonrel_judged_ret    14      18.0000\n",
            "num_q                    16      1.0000\n",
            "num_ret                  16      19.0000\n",
            "num_rel                  16      7.0000\n",
            "num_rel_ret              16      0.0000\n",
            "map                      16      0.0000\n",
            "gm_map                   16      -11.5129\n",
            "Rprec                    16      0.0000\n",
            "bpref                    16      0.0000\n",
            "recip_rank               16      0.0000\n",
            "iprec_at_recall_0.00     16      0.0000\n",
            "iprec_at_recall_0.10     16      0.0000\n",
            "iprec_at_recall_0.20     16      0.0000\n",
            "iprec_at_recall_0.30     16      0.0000\n",
            "iprec_at_recall_0.40     16      0.0000\n",
            "iprec_at_recall_0.50     16      0.0000\n",
            "iprec_at_recall_0.60     16      0.0000\n",
            "iprec_at_recall_0.70     16      0.0000\n",
            "iprec_at_recall_0.80     16      0.0000\n",
            "iprec_at_recall_0.90     16      0.0000\n",
            "iprec_at_recall_1.00     16      0.0000\n",
            "P_5                      16      0.0000\n",
            "P_10                     16      0.0000\n",
            "P_15                     16      0.0000\n",
            "P_20                     16      0.0000\n",
            "P_30                     16      0.0000\n",
            "P_100                    16      0.0000\n",
            "P_200                    16      0.0000\n",
            "P_500                    16      0.0000\n",
            "P_1000                   16      0.0000\n",
            "relstring                16      0.0000\n",
            "recall_5                 16      0.0000\n",
            "recall_10                16      0.0000\n",
            "recall_15                16      0.0000\n",
            "recall_20                16      0.0000\n",
            "recall_30                16      0.0000\n",
            "recall_100               16      0.0000\n",
            "recall_200               16      0.0000\n",
            "recall_500               16      0.0000\n",
            "recall_1000              16      0.0000\n",
            "infAP                    16      0.0000\n",
            "gm_bpref                 16      -11.5129\n",
            "Rprec_mult_0.20          16      0.0000\n",
            "Rprec_mult_0.40          16      0.0000\n",
            "Rprec_mult_0.60          16      0.0000\n",
            "Rprec_mult_0.80          16      0.0000\n",
            "Rprec_mult_1.00          16      0.0000\n",
            "Rprec_mult_1.20          16      0.0000\n",
            "Rprec_mult_1.40          16      0.0000\n",
            "Rprec_mult_1.60          16      0.0000\n",
            "Rprec_mult_1.80          16      0.0000\n",
            "Rprec_mult_2.00          16      0.0000\n",
            "utility                  16      -19.0000\n",
            "11pt_avg                 16      0.0000\n",
            "binG                     16      0.0000\n",
            "G                        16      0.0000\n",
            "ndcg                     16      0.0000\n",
            "ndcg_rel                 16      0.0000\n",
            "Rndcg                    16      0.0000\n",
            "ndcg_cut_5               16      0.0000\n",
            "ndcg_cut_10              16      0.0000\n",
            "ndcg_cut_15              16      0.0000\n",
            "ndcg_cut_20              16      0.0000\n",
            "ndcg_cut_30              16      0.0000\n",
            "ndcg_cut_100             16      0.0000\n",
            "ndcg_cut_200             16      0.0000\n",
            "ndcg_cut_500             16      0.0000\n",
            "ndcg_cut_1000            16      0.0000\n",
            "map_cut_5                16      0.0000\n",
            "map_cut_10               16      0.0000\n",
            "map_cut_15               16      0.0000\n",
            "map_cut_20               16      0.0000\n",
            "map_cut_30               16      0.0000\n",
            "map_cut_100              16      0.0000\n",
            "map_cut_200              16      0.0000\n",
            "map_cut_500              16      0.0000\n",
            "map_cut_1000             16      0.0000\n",
            "relative_P_5             16      0.0000\n",
            "relative_P_10            16      0.0000\n",
            "relative_P_15            16      0.0000\n",
            "relative_P_20            16      0.0000\n",
            "relative_P_30            16      0.0000\n",
            "relative_P_100           16      0.0000\n",
            "relative_P_200           16      0.0000\n",
            "relative_P_500           16      0.0000\n",
            "relative_P_1000          16      0.0000\n",
            "success_1                16      0.0000\n",
            "success_5                16      0.0000\n",
            "success_10               16      0.0000\n",
            "set_P                    16      0.0000\n",
            "set_relative_P           16      0.0000\n",
            "set_recall               16      0.0000\n",
            "set_map                  16      0.0000\n",
            "set_F                    16      0.0000\n",
            "num_nonrel_judged_ret    16      15.0000\n",
            "num_q                    18      1.0000\n",
            "num_ret                  18      22.0000\n",
            "num_rel                  18      1.0000\n",
            "num_rel_ret              18      1.0000\n",
            "map                      18      1.0000\n",
            "gm_map                   18      0.0000\n",
            "Rprec                    18      1.0000\n",
            "bpref                    18      1.0000\n",
            "recip_rank               18      1.0000\n",
            "iprec_at_recall_0.00     18      1.0000\n",
            "iprec_at_recall_0.10     18      1.0000\n",
            "iprec_at_recall_0.20     18      1.0000\n",
            "iprec_at_recall_0.30     18      1.0000\n",
            "iprec_at_recall_0.40     18      1.0000\n",
            "iprec_at_recall_0.50     18      1.0000\n",
            "iprec_at_recall_0.60     18      1.0000\n",
            "iprec_at_recall_0.70     18      1.0000\n",
            "iprec_at_recall_0.80     18      1.0000\n",
            "iprec_at_recall_0.90     18      1.0000\n",
            "iprec_at_recall_1.00     18      1.0000\n",
            "P_5                      18      0.2000\n",
            "P_10                     18      0.1000\n",
            "P_15                     18      0.0667\n",
            "P_20                     18      0.0500\n",
            "P_30                     18      0.0333\n",
            "P_100                    18      0.0100\n",
            "P_200                    18      0.0050\n",
            "P_500                    18      0.0020\n",
            "P_1000                   18      0.0010\n",
            "relstring                18      0.0000\n",
            "recall_5                 18      1.0000\n",
            "recall_10                18      1.0000\n",
            "recall_15                18      1.0000\n",
            "recall_20                18      1.0000\n",
            "recall_30                18      1.0000\n",
            "recall_100               18      1.0000\n",
            "recall_200               18      1.0000\n",
            "recall_500               18      1.0000\n",
            "recall_1000              18      1.0000\n",
            "infAP                    18      1.0000\n",
            "gm_bpref                 18      0.0000\n",
            "Rprec_mult_0.20          18      1.0000\n",
            "Rprec_mult_0.40          18      1.0000\n",
            "Rprec_mult_0.60          18      1.0000\n",
            "Rprec_mult_0.80          18      1.0000\n",
            "Rprec_mult_1.00          18      1.0000\n",
            "Rprec_mult_1.20          18      0.5000\n",
            "Rprec_mult_1.40          18      0.5000\n",
            "Rprec_mult_1.60          18      0.5000\n",
            "Rprec_mult_1.80          18      0.5000\n",
            "Rprec_mult_2.00          18      0.5000\n",
            "utility                  18      -20.0000\n",
            "11pt_avg                 18      1.0000\n",
            "binG                     18      1.0000\n",
            "G                        18      1.0000\n",
            "ndcg                     18      1.0000\n",
            "ndcg_rel                 18      1.0000\n",
            "Rndcg                    18      1.0000\n",
            "ndcg_cut_5               18      1.0000\n",
            "ndcg_cut_10              18      1.0000\n",
            "ndcg_cut_15              18      1.0000\n",
            "ndcg_cut_20              18      1.0000\n",
            "ndcg_cut_30              18      1.0000\n",
            "ndcg_cut_100             18      1.0000\n",
            "ndcg_cut_200             18      1.0000\n",
            "ndcg_cut_500             18      1.0000\n",
            "ndcg_cut_1000            18      1.0000\n",
            "map_cut_5                18      1.0000\n",
            "map_cut_10               18      1.0000\n",
            "map_cut_15               18      1.0000\n",
            "map_cut_20               18      1.0000\n",
            "map_cut_30               18      1.0000\n",
            "map_cut_100              18      1.0000\n",
            "map_cut_200              18      1.0000\n",
            "map_cut_500              18      1.0000\n",
            "map_cut_1000             18      1.0000\n",
            "relative_P_5             18      1.0000\n",
            "relative_P_10            18      1.0000\n",
            "relative_P_15            18      1.0000\n",
            "relative_P_20            18      1.0000\n",
            "relative_P_30            18      1.0000\n",
            "relative_P_100           18      1.0000\n",
            "relative_P_200           18      1.0000\n",
            "relative_P_500           18      1.0000\n",
            "relative_P_1000          18      1.0000\n",
            "success_1                18      1.0000\n",
            "success_5                18      1.0000\n",
            "success_10               18      1.0000\n",
            "set_P                    18      0.0455\n",
            "set_relative_P           18      1.0000\n",
            "set_recall               18      1.0000\n",
            "set_map                  18      0.0455\n",
            "set_F                    18      0.0870\n",
            "num_nonrel_judged_ret    18      21.0000\n",
            "num_q                    19      1.0000\n",
            "num_ret                  19      5.0000\n",
            "num_rel                  19      2.0000\n",
            "num_rel_ret              19      1.0000\n",
            "map                      19      0.5000\n",
            "gm_map                   19      -0.6931\n",
            "Rprec                    19      0.5000\n",
            "bpref                    19      0.5000\n",
            "recip_rank               19      1.0000\n",
            "iprec_at_recall_0.00     19      1.0000\n",
            "iprec_at_recall_0.10     19      1.0000\n",
            "iprec_at_recall_0.20     19      1.0000\n",
            "iprec_at_recall_0.30     19      1.0000\n",
            "iprec_at_recall_0.40     19      1.0000\n",
            "iprec_at_recall_0.50     19      1.0000\n",
            "iprec_at_recall_0.60     19      0.0000\n",
            "iprec_at_recall_0.70     19      0.0000\n",
            "iprec_at_recall_0.80     19      0.0000\n",
            "iprec_at_recall_0.90     19      0.0000\n",
            "iprec_at_recall_1.00     19      0.0000\n",
            "P_5                      19      0.2000\n",
            "P_10                     19      0.1000\n",
            "P_15                     19      0.0667\n",
            "P_20                     19      0.0500\n",
            "P_30                     19      0.0333\n",
            "P_100                    19      0.0100\n",
            "P_200                    19      0.0050\n",
            "P_500                    19      0.0020\n",
            "P_1000                   19      0.0010\n",
            "relstring                19      0.0000\n",
            "recall_5                 19      0.5000\n",
            "recall_10                19      0.5000\n",
            "recall_15                19      0.5000\n",
            "recall_20                19      0.5000\n",
            "recall_30                19      0.5000\n",
            "recall_100               19      0.5000\n",
            "recall_200               19      0.5000\n",
            "recall_500               19      0.5000\n",
            "recall_1000              19      0.5000\n",
            "infAP                    19      0.5000\n",
            "gm_bpref                 19      -0.6931\n",
            "Rprec_mult_0.20          19      1.0000\n",
            "Rprec_mult_0.40          19      1.0000\n",
            "Rprec_mult_0.60          19      0.5000\n",
            "Rprec_mult_0.80          19      0.5000\n",
            "Rprec_mult_1.00          19      0.5000\n",
            "Rprec_mult_1.20          19      0.3333\n",
            "Rprec_mult_1.40          19      0.3333\n",
            "Rprec_mult_1.60          19      0.2500\n",
            "Rprec_mult_1.80          19      0.2500\n",
            "Rprec_mult_2.00          19      0.2500\n",
            "utility                  19      -3.0000\n",
            "11pt_avg                 19      0.5455\n",
            "binG                     19      0.5000\n",
            "G                        19      0.5000\n",
            "ndcg                     19      0.6131\n",
            "ndcg_rel                 19      0.8066\n",
            "Rndcg                    19      0.6131\n",
            "ndcg_cut_5               19      0.6131\n",
            "ndcg_cut_10              19      0.6131\n",
            "ndcg_cut_15              19      0.6131\n",
            "ndcg_cut_20              19      0.6131\n",
            "ndcg_cut_30              19      0.6131\n",
            "ndcg_cut_100             19      0.6131\n",
            "ndcg_cut_200             19      0.6131\n",
            "ndcg_cut_500             19      0.6131\n",
            "ndcg_cut_1000            19      0.6131\n",
            "map_cut_5                19      0.5000\n",
            "map_cut_10               19      0.5000\n",
            "map_cut_15               19      0.5000\n",
            "map_cut_20               19      0.5000\n",
            "map_cut_30               19      0.5000\n",
            "map_cut_100              19      0.5000\n",
            "map_cut_200              19      0.5000\n",
            "map_cut_500              19      0.5000\n",
            "map_cut_1000             19      0.5000\n",
            "relative_P_5             19      0.5000\n",
            "relative_P_10            19      0.5000\n",
            "relative_P_15            19      0.5000\n",
            "relative_P_20            19      0.5000\n",
            "relative_P_30            19      0.5000\n",
            "relative_P_100           19      0.5000\n",
            "relative_P_200           19      0.5000\n",
            "relative_P_500           19      0.5000\n",
            "relative_P_1000          19      0.5000\n",
            "success_1                19      1.0000\n",
            "success_5                19      1.0000\n",
            "success_10               19      1.0000\n",
            "set_P                    19      0.2000\n",
            "set_relative_P           19      0.5000\n",
            "set_recall               19      0.5000\n",
            "set_map                  19      0.1000\n",
            "set_F                    19      0.2857\n",
            "num_nonrel_judged_ret    19      4.0000\n",
            "num_q                    22      1.0000\n",
            "num_ret                  22      34.0000\n",
            "num_rel                  22      1.0000\n",
            "num_rel_ret              22      1.0000\n",
            "map                      22      0.0769\n",
            "gm_map                   22      -2.5649\n",
            "Rprec                    22      0.0000\n",
            "bpref                    22      0.0000\n",
            "recip_rank               22      0.0769\n",
            "iprec_at_recall_0.00     22      0.0769\n",
            "iprec_at_recall_0.10     22      0.0769\n",
            "iprec_at_recall_0.20     22      0.0769\n",
            "iprec_at_recall_0.30     22      0.0769\n",
            "iprec_at_recall_0.40     22      0.0769\n",
            "iprec_at_recall_0.50     22      0.0769\n",
            "iprec_at_recall_0.60     22      0.0769\n",
            "iprec_at_recall_0.70     22      0.0769\n",
            "iprec_at_recall_0.80     22      0.0769\n",
            "iprec_at_recall_0.90     22      0.0769\n",
            "iprec_at_recall_1.00     22      0.0769\n",
            "P_5                      22      0.0000\n",
            "P_10                     22      0.0000\n",
            "P_15                     22      0.0667\n",
            "P_20                     22      0.0500\n",
            "P_30                     22      0.0333\n",
            "P_100                    22      0.0100\n",
            "P_200                    22      0.0050\n",
            "P_500                    22      0.0020\n",
            "P_1000                   22      0.0010\n",
            "relstring                22      0.0000\n",
            "recall_5                 22      0.0000\n",
            "recall_10                22      0.0000\n",
            "recall_15                22      1.0000\n",
            "recall_20                22      1.0000\n",
            "recall_30                22      1.0000\n",
            "recall_100               22      1.0000\n",
            "recall_200               22      1.0000\n",
            "recall_500               22      1.0000\n",
            "recall_1000              22      1.0000\n",
            "infAP                    22      0.0769\n",
            "gm_bpref                 22      -11.5129\n",
            "Rprec_mult_0.20          22      0.0000\n",
            "Rprec_mult_0.40          22      0.0000\n",
            "Rprec_mult_0.60          22      0.0000\n",
            "Rprec_mult_0.80          22      0.0000\n",
            "Rprec_mult_1.00          22      0.0000\n",
            "Rprec_mult_1.20          22      0.0000\n",
            "Rprec_mult_1.40          22      0.0000\n",
            "Rprec_mult_1.60          22      0.0000\n",
            "Rprec_mult_1.80          22      0.0000\n",
            "Rprec_mult_2.00          22      0.0000\n",
            "utility                  22      -32.0000\n",
            "11pt_avg                 22      0.0769\n",
            "binG                     22      0.2626\n",
            "G                        22      0.2626\n",
            "ndcg                     22      0.2626\n",
            "ndcg_rel                 22      0.2626\n",
            "Rndcg                    22      0.1313\n",
            "ndcg_cut_5               22      0.0000\n",
            "ndcg_cut_10              22      0.0000\n",
            "ndcg_cut_15              22      0.2626\n",
            "ndcg_cut_20              22      0.2626\n",
            "ndcg_cut_30              22      0.2626\n",
            "ndcg_cut_100             22      0.2626\n",
            "ndcg_cut_200             22      0.2626\n",
            "ndcg_cut_500             22      0.2626\n",
            "ndcg_cut_1000            22      0.2626\n",
            "map_cut_5                22      0.0000\n",
            "map_cut_10               22      0.0000\n",
            "map_cut_15               22      0.0769\n",
            "map_cut_20               22      0.0769\n",
            "map_cut_30               22      0.0769\n",
            "map_cut_100              22      0.0769\n",
            "map_cut_200              22      0.0769\n",
            "map_cut_500              22      0.0769\n",
            "map_cut_1000             22      0.0769\n",
            "relative_P_5             22      0.0000\n",
            "relative_P_10            22      0.0000\n",
            "relative_P_15            22      1.0000\n",
            "relative_P_20            22      1.0000\n",
            "relative_P_30            22      1.0000\n",
            "relative_P_100           22      1.0000\n",
            "relative_P_200           22      1.0000\n",
            "relative_P_500           22      1.0000\n",
            "relative_P_1000          22      1.0000\n",
            "success_1                22      0.0000\n",
            "success_5                22      0.0000\n",
            "success_10               22      0.0000\n",
            "set_P                    22      0.0294\n",
            "set_relative_P           22      1.0000\n",
            "set_recall               22      1.0000\n",
            "set_map                  22      0.0294\n",
            "set_F                    22      0.0571\n",
            "num_nonrel_judged_ret    22      30.0000\n",
            "num_q                    24      1.0000\n",
            "num_ret                  24      17.0000\n",
            "num_rel                  24      1.0000\n",
            "num_rel_ret              24      1.0000\n",
            "map                      24      1.0000\n",
            "gm_map                   24      0.0000\n",
            "Rprec                    24      1.0000\n",
            "bpref                    24      1.0000\n",
            "recip_rank               24      1.0000\n",
            "iprec_at_recall_0.00     24      1.0000\n",
            "iprec_at_recall_0.10     24      1.0000\n",
            "iprec_at_recall_0.20     24      1.0000\n",
            "iprec_at_recall_0.30     24      1.0000\n",
            "iprec_at_recall_0.40     24      1.0000\n",
            "iprec_at_recall_0.50     24      1.0000\n",
            "iprec_at_recall_0.60     24      1.0000\n",
            "iprec_at_recall_0.70     24      1.0000\n",
            "iprec_at_recall_0.80     24      1.0000\n",
            "iprec_at_recall_0.90     24      1.0000\n",
            "iprec_at_recall_1.00     24      1.0000\n",
            "P_5                      24      0.2000\n",
            "P_10                     24      0.1000\n",
            "P_15                     24      0.0667\n",
            "P_20                     24      0.0500\n",
            "P_30                     24      0.0333\n",
            "P_100                    24      0.0100\n",
            "P_200                    24      0.0050\n",
            "P_500                    24      0.0020\n",
            "P_1000                   24      0.0010\n",
            "relstring                24      0.0000\n",
            "recall_5                 24      1.0000\n",
            "recall_10                24      1.0000\n",
            "recall_15                24      1.0000\n",
            "recall_20                24      1.0000\n",
            "recall_30                24      1.0000\n",
            "recall_100               24      1.0000\n",
            "recall_200               24      1.0000\n",
            "recall_500               24      1.0000\n",
            "recall_1000              24      1.0000\n",
            "infAP                    24      1.0000\n",
            "gm_bpref                 24      0.0000\n",
            "Rprec_mult_0.20          24      1.0000\n",
            "Rprec_mult_0.40          24      1.0000\n",
            "Rprec_mult_0.60          24      1.0000\n",
            "Rprec_mult_0.80          24      1.0000\n",
            "Rprec_mult_1.00          24      1.0000\n",
            "Rprec_mult_1.20          24      0.5000\n",
            "Rprec_mult_1.40          24      0.5000\n",
            "Rprec_mult_1.60          24      0.5000\n",
            "Rprec_mult_1.80          24      0.5000\n",
            "Rprec_mult_2.00          24      0.5000\n",
            "utility                  24      -15.0000\n",
            "11pt_avg                 24      1.0000\n",
            "binG                     24      1.0000\n",
            "G                        24      1.0000\n",
            "ndcg                     24      1.0000\n",
            "ndcg_rel                 24      1.0000\n",
            "Rndcg                    24      1.0000\n",
            "ndcg_cut_5               24      1.0000\n",
            "ndcg_cut_10              24      1.0000\n",
            "ndcg_cut_15              24      1.0000\n",
            "ndcg_cut_20              24      1.0000\n",
            "ndcg_cut_30              24      1.0000\n",
            "ndcg_cut_100             24      1.0000\n",
            "ndcg_cut_200             24      1.0000\n",
            "ndcg_cut_500             24      1.0000\n",
            "ndcg_cut_1000            24      1.0000\n",
            "map_cut_5                24      1.0000\n",
            "map_cut_10               24      1.0000\n",
            "map_cut_15               24      1.0000\n",
            "map_cut_20               24      1.0000\n",
            "map_cut_30               24      1.0000\n",
            "map_cut_100              24      1.0000\n",
            "map_cut_200              24      1.0000\n",
            "map_cut_500              24      1.0000\n",
            "map_cut_1000             24      1.0000\n",
            "relative_P_5             24      1.0000\n",
            "relative_P_10            24      1.0000\n",
            "relative_P_15            24      1.0000\n",
            "relative_P_20            24      1.0000\n",
            "relative_P_30            24      1.0000\n",
            "relative_P_100           24      1.0000\n",
            "relative_P_200           24      1.0000\n",
            "relative_P_500           24      1.0000\n",
            "relative_P_1000          24      1.0000\n",
            "success_1                24      1.0000\n",
            "success_5                24      1.0000\n",
            "success_10               24      1.0000\n",
            "set_P                    24      0.0588\n",
            "set_relative_P           24      1.0000\n",
            "set_recall               24      1.0000\n",
            "set_map                  24      0.0588\n",
            "set_F                    24      0.1111\n",
            "num_nonrel_judged_ret    24      14.0000\n",
            "num_q                    26      1.0000\n",
            "num_ret                  26      40.0000\n",
            "num_rel                  26      3.0000\n",
            "num_rel_ret              26      1.0000\n",
            "map                      26      0.0667\n",
            "gm_map                   26      -2.7081\n",
            "Rprec                    26      0.0000\n",
            "bpref                    26      0.0000\n",
            "recip_rank               26      0.2000\n",
            "iprec_at_recall_0.00     26      0.2000\n",
            "iprec_at_recall_0.10     26      0.2000\n",
            "iprec_at_recall_0.20     26      0.2000\n",
            "iprec_at_recall_0.30     26      0.2000\n",
            "iprec_at_recall_0.40     26      0.0000\n",
            "iprec_at_recall_0.50     26      0.0000\n",
            "iprec_at_recall_0.60     26      0.0000\n",
            "iprec_at_recall_0.70     26      0.0000\n",
            "iprec_at_recall_0.80     26      0.0000\n",
            "iprec_at_recall_0.90     26      0.0000\n",
            "iprec_at_recall_1.00     26      0.0000\n",
            "P_5                      26      0.2000\n",
            "P_10                     26      0.1000\n",
            "P_15                     26      0.0667\n",
            "P_20                     26      0.0500\n",
            "P_30                     26      0.0333\n",
            "P_100                    26      0.0100\n",
            "P_200                    26      0.0050\n",
            "P_500                    26      0.0020\n",
            "P_1000                   26      0.0010\n",
            "relstring                26      0.0000\n",
            "recall_5                 26      0.3333\n",
            "recall_10                26      0.3333\n",
            "recall_15                26      0.3333\n",
            "recall_20                26      0.3333\n",
            "recall_30                26      0.3333\n",
            "recall_100               26      0.3333\n",
            "recall_200               26      0.3333\n",
            "recall_500               26      0.3333\n",
            "recall_1000              26      0.3333\n",
            "infAP                    26      0.0667\n",
            "gm_bpref                 26      -11.5129\n",
            "Rprec_mult_0.20          26      0.0000\n",
            "Rprec_mult_0.40          26      0.0000\n",
            "Rprec_mult_0.60          26      0.0000\n",
            "Rprec_mult_0.80          26      0.0000\n",
            "Rprec_mult_1.00          26      0.0000\n",
            "Rprec_mult_1.20          26      0.0000\n",
            "Rprec_mult_1.40          26      0.2000\n",
            "Rprec_mult_1.60          26      0.2000\n",
            "Rprec_mult_1.80          26      0.1667\n",
            "Rprec_mult_2.00          26      0.1667\n",
            "utility                  26      -38.0000\n",
            "11pt_avg                 26      0.0727\n",
            "binG                     26      0.1290\n",
            "G                        26      0.1290\n",
            "ndcg                     26      0.1815\n",
            "ndcg_rel                 26      0.1815\n",
            "Rndcg                    26      0.0908\n",
            "ndcg_cut_5               26      0.1815\n",
            "ndcg_cut_10              26      0.1815\n",
            "ndcg_cut_15              26      0.1815\n",
            "ndcg_cut_20              26      0.1815\n",
            "ndcg_cut_30              26      0.1815\n",
            "ndcg_cut_100             26      0.1815\n",
            "ndcg_cut_200             26      0.1815\n",
            "ndcg_cut_500             26      0.1815\n",
            "ndcg_cut_1000            26      0.1815\n",
            "map_cut_5                26      0.0667\n",
            "map_cut_10               26      0.0667\n",
            "map_cut_15               26      0.0667\n",
            "map_cut_20               26      0.0667\n",
            "map_cut_30               26      0.0667\n",
            "map_cut_100              26      0.0667\n",
            "map_cut_200              26      0.0667\n",
            "map_cut_500              26      0.0667\n",
            "map_cut_1000             26      0.0667\n",
            "relative_P_5             26      0.3333\n",
            "relative_P_10            26      0.3333\n",
            "relative_P_15            26      0.3333\n",
            "relative_P_20            26      0.3333\n",
            "relative_P_30            26      0.3333\n",
            "relative_P_100           26      0.3333\n",
            "relative_P_200           26      0.3333\n",
            "relative_P_500           26      0.3333\n",
            "relative_P_1000          26      0.3333\n",
            "success_1                26      0.0000\n",
            "success_5                26      1.0000\n",
            "success_10               26      1.0000\n",
            "set_P                    26      0.0250\n",
            "set_relative_P           26      0.3333\n",
            "set_recall               26      0.3333\n",
            "set_map                  26      0.0083\n",
            "set_F                    26      0.0465\n",
            "num_nonrel_judged_ret    26      31.0000\n",
            "num_q                    28      1.0000\n",
            "num_ret                  28      36.0000\n",
            "num_rel                  28      2.0000\n",
            "num_rel_ret              28      2.0000\n",
            "map                      28      0.0729\n",
            "gm_map                   28      -2.6188\n",
            "Rprec                    28      0.0000\n",
            "bpref                    28      0.0000\n",
            "recip_rank               28      0.0588\n",
            "iprec_at_recall_0.00     28      0.0870\n",
            "iprec_at_recall_0.10     28      0.0870\n",
            "iprec_at_recall_0.20     28      0.0870\n",
            "iprec_at_recall_0.30     28      0.0870\n",
            "iprec_at_recall_0.40     28      0.0870\n",
            "iprec_at_recall_0.50     28      0.0870\n",
            "iprec_at_recall_0.60     28      0.0870\n",
            "iprec_at_recall_0.70     28      0.0870\n",
            "iprec_at_recall_0.80     28      0.0870\n",
            "iprec_at_recall_0.90     28      0.0870\n",
            "iprec_at_recall_1.00     28      0.0870\n",
            "P_5                      28      0.0000\n",
            "P_10                     28      0.0000\n",
            "P_15                     28      0.0000\n",
            "P_20                     28      0.0500\n",
            "P_30                     28      0.0667\n",
            "P_100                    28      0.0200\n",
            "P_200                    28      0.0100\n",
            "P_500                    28      0.0040\n",
            "P_1000                   28      0.0020\n",
            "relstring                28      0.0000\n",
            "recall_5                 28      0.0000\n",
            "recall_10                28      0.0000\n",
            "recall_15                28      0.0000\n",
            "recall_20                28      0.5000\n",
            "recall_30                28      1.0000\n",
            "recall_100               28      1.0000\n",
            "recall_200               28      1.0000\n",
            "recall_500               28      1.0000\n",
            "recall_1000              28      1.0000\n",
            "infAP                    28      0.0729\n",
            "gm_bpref                 28      -11.5129\n",
            "Rprec_mult_0.20          28      0.0000\n",
            "Rprec_mult_0.40          28      0.0000\n",
            "Rprec_mult_0.60          28      0.0000\n",
            "Rprec_mult_0.80          28      0.0000\n",
            "Rprec_mult_1.00          28      0.0000\n",
            "Rprec_mult_1.20          28      0.0000\n",
            "Rprec_mult_1.40          28      0.0000\n",
            "Rprec_mult_1.60          28      0.0000\n",
            "Rprec_mult_1.80          28      0.0000\n",
            "Rprec_mult_2.00          28      0.0000\n",
            "utility                  28      -32.0000\n",
            "11pt_avg                 28      0.0870\n",
            "binG                     28      0.2304\n",
            "G                        28      0.2304\n",
            "ndcg                     28      0.2808\n",
            "ndcg_rel                 28      0.2139\n",
            "Rndcg                    28      0.1404\n",
            "ndcg_cut_5               28      0.0000\n",
            "ndcg_cut_10              28      0.0000\n",
            "ndcg_cut_15              28      0.0000\n",
            "ndcg_cut_20              28      0.1470\n",
            "ndcg_cut_30              28      0.2808\n",
            "ndcg_cut_100             28      0.2808\n",
            "ndcg_cut_200             28      0.2808\n",
            "ndcg_cut_500             28      0.2808\n",
            "ndcg_cut_1000            28      0.2808\n",
            "map_cut_5                28      0.0000\n",
            "map_cut_10               28      0.0000\n",
            "map_cut_15               28      0.0000\n",
            "map_cut_20               28      0.0294\n",
            "map_cut_30               28      0.0729\n",
            "map_cut_100              28      0.0729\n",
            "map_cut_200              28      0.0729\n",
            "map_cut_500              28      0.0729\n",
            "map_cut_1000             28      0.0729\n",
            "relative_P_5             28      0.0000\n",
            "relative_P_10            28      0.0000\n",
            "relative_P_15            28      0.0000\n",
            "relative_P_20            28      0.5000\n",
            "relative_P_30            28      1.0000\n",
            "relative_P_100           28      1.0000\n",
            "relative_P_200           28      1.0000\n",
            "relative_P_500           28      1.0000\n",
            "relative_P_1000          28      1.0000\n",
            "success_1                28      0.0000\n",
            "success_5                28      0.0000\n",
            "success_10               28      0.0000\n",
            "set_P                    28      0.0556\n",
            "set_relative_P           28      1.0000\n",
            "set_recall               28      1.0000\n",
            "set_map                  28      0.0556\n",
            "set_F                    28      0.1053\n",
            "num_nonrel_judged_ret    28      25.0000\n",
            "num_q                    all     15.0000\n",
            "num_ret                  all     294.0000\n",
            "num_rel                  all     35.0000\n",
            "num_rel_ret              all     12.0000\n",
            "map                      all     0.3367\n",
            "gm_map                   all     0.0103\n",
            "Rprec                    all     0.3000\n",
            "bpref                    all     0.3000\n",
            "recip_rank               all     0.4446\n",
            "iprec_at_recall_0.00     all     0.4465\n",
            "iprec_at_recall_0.10     all     0.4465\n",
            "iprec_at_recall_0.20     all     0.4465\n",
            "iprec_at_recall_0.30     all     0.4465\n",
            "iprec_at_recall_0.40     all     0.4331\n",
            "iprec_at_recall_0.50     all     0.4331\n",
            "iprec_at_recall_0.60     all     0.2331\n",
            "iprec_at_recall_0.70     all     0.2331\n",
            "iprec_at_recall_0.80     all     0.2331\n",
            "iprec_at_recall_0.90     all     0.2331\n",
            "iprec_at_recall_1.00     all     0.2331\n",
            "P_5                      all     0.1200\n",
            "P_10                     all     0.0600\n",
            "P_15                     all     0.0444\n",
            "P_20                     all     0.0367\n",
            "P_30                     all     0.0267\n",
            "P_100                    all     0.0080\n",
            "P_200                    all     0.0040\n",
            "P_500                    all     0.0016\n",
            "P_1000                   all     0.0008\n",
            "relstring                all     0.0000\n",
            "recall_5                 all     0.3889\n",
            "recall_10                all     0.3889\n",
            "recall_15                all     0.4556\n",
            "recall_20                all     0.4889\n",
            "recall_30                all     0.5222\n",
            "recall_100               all     0.5222\n",
            "recall_200               all     0.5222\n",
            "recall_500               all     0.5222\n",
            "recall_1000              all     0.5222\n",
            "infAP                    all     0.3367\n",
            "gm_bpref                 all     0.0009\n",
            "Rprec_mult_0.20          all     0.4000\n",
            "Rprec_mult_0.40          all     0.4000\n",
            "Rprec_mult_0.60          all     0.3111\n",
            "Rprec_mult_0.80          all     0.3000\n",
            "Rprec_mult_1.00          all     0.3000\n",
            "Rprec_mult_1.20          all     0.1711\n",
            "Rprec_mult_1.40          all     0.1800\n",
            "Rprec_mult_1.60          all     0.1657\n",
            "Rprec_mult_1.80          all     0.1611\n",
            "Rprec_mult_2.00          all     0.1611\n",
            "utility                  all     -18.0000\n",
            "11pt_avg                 all     0.3471\n",
            "binG                     all     0.3748\n",
            "G                        all     0.3748\n",
            "ndcg                     all     0.4059\n",
            "ndcg_rel                 all     0.4393\n",
            "Rndcg                    all     0.3650\n",
            "ndcg_cut_5               all     0.3696\n",
            "ndcg_cut_10              all     0.3696\n",
            "ndcg_cut_15              all     0.3871\n",
            "ndcg_cut_20              all     0.3969\n",
            "ndcg_cut_30              all     0.4059\n",
            "ndcg_cut_100             all     0.4059\n",
            "ndcg_cut_200             all     0.4059\n",
            "ndcg_cut_500             all     0.4059\n",
            "ndcg_cut_1000            all     0.4059\n",
            "map_cut_5                all     0.3267\n",
            "map_cut_10               all     0.3267\n",
            "map_cut_15               all     0.3318\n",
            "map_cut_20               all     0.3338\n",
            "map_cut_30               all     0.3367\n",
            "map_cut_100              all     0.3367\n",
            "map_cut_200              all     0.3367\n",
            "map_cut_500              all     0.3367\n",
            "map_cut_1000             all     0.3367\n",
            "relative_P_5             all     0.3889\n",
            "relative_P_10            all     0.3889\n",
            "relative_P_15            all     0.4556\n",
            "relative_P_20            all     0.4889\n",
            "relative_P_30            all     0.5222\n",
            "relative_P_100           all     0.5222\n",
            "relative_P_200           all     0.5222\n",
            "relative_P_500           all     0.5222\n",
            "relative_P_1000          all     0.5222\n",
            "success_1                all     0.4000\n",
            "success_5                all     0.5333\n",
            "success_10               all     0.5333\n",
            "set_P                    all     0.0425\n",
            "set_relative_P           all     0.5222\n",
            "set_recall               all     0.5222\n",
            "set_map                  all     0.0300\n",
            "set_F                    all     0.0731\n",
            "num_nonrel_judged_ret    all     225.0000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ofkLyRTNDlbO",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "72e690bd-4f38-4263-aea0-272c9aa7073b"
      },
      "source": [
        "# Add stemmer to previous analyzer\n",
        "New_Analyzer=RegexTokenizer() | LowercaseFilter()| StopFilter()|StemFilter()\n",
        "New_Schema=Schema(file_path = ID(stored=True), file_content = TEXT(analyzer = New_Analyzer))\n",
        "NEW_INDEX = createIndex(New_Schema)\n",
        "addFilesToIndex(NEW_INDEX, filesToIndex)\n",
        "NEW_QP = QueryParser(\"file_content\", schema=NEW_INDEX.schema)\n",
        "NEW_SEARCHER = NEW_INDEX.searcher()\n",
        "pyTrecEval(topic_file, qrels_file, NEW_QP, NEW_SEARCHER) "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "num_q                    1       1.0000\n",
            "num_ret                  1       3.0000\n",
            "num_rel                  1       5.0000\n",
            "num_rel_ret              1       0.0000\n",
            "map                      1       0.0000\n",
            "gm_map                   1       -11.5129\n",
            "Rprec                    1       0.0000\n",
            "bpref                    1       0.0000\n",
            "recip_rank               1       0.0000\n",
            "iprec_at_recall_0.00     1       0.0000\n",
            "iprec_at_recall_0.10     1       0.0000\n",
            "iprec_at_recall_0.20     1       0.0000\n",
            "iprec_at_recall_0.30     1       0.0000\n",
            "iprec_at_recall_0.40     1       0.0000\n",
            "iprec_at_recall_0.50     1       0.0000\n",
            "iprec_at_recall_0.60     1       0.0000\n",
            "iprec_at_recall_0.70     1       0.0000\n",
            "iprec_at_recall_0.80     1       0.0000\n",
            "iprec_at_recall_0.90     1       0.0000\n",
            "iprec_at_recall_1.00     1       0.0000\n",
            "P_5                      1       0.0000\n",
            "P_10                     1       0.0000\n",
            "P_15                     1       0.0000\n",
            "P_20                     1       0.0000\n",
            "P_30                     1       0.0000\n",
            "P_100                    1       0.0000\n",
            "P_200                    1       0.0000\n",
            "P_500                    1       0.0000\n",
            "P_1000                   1       0.0000\n",
            "relstring                1       0.0000\n",
            "recall_5                 1       0.0000\n",
            "recall_10                1       0.0000\n",
            "recall_15                1       0.0000\n",
            "recall_20                1       0.0000\n",
            "recall_30                1       0.0000\n",
            "recall_100               1       0.0000\n",
            "recall_200               1       0.0000\n",
            "recall_500               1       0.0000\n",
            "recall_1000              1       0.0000\n",
            "infAP                    1       0.0000\n",
            "gm_bpref                 1       -11.5129\n",
            "Rprec_mult_0.20          1       0.0000\n",
            "Rprec_mult_0.40          1       0.0000\n",
            "Rprec_mult_0.60          1       0.0000\n",
            "Rprec_mult_0.80          1       0.0000\n",
            "Rprec_mult_1.00          1       0.0000\n",
            "Rprec_mult_1.20          1       0.0000\n",
            "Rprec_mult_1.40          1       0.0000\n",
            "Rprec_mult_1.60          1       0.0000\n",
            "Rprec_mult_1.80          1       0.0000\n",
            "Rprec_mult_2.00          1       0.0000\n",
            "utility                  1       -3.0000\n",
            "11pt_avg                 1       0.0000\n",
            "binG                     1       0.0000\n",
            "G                        1       0.0000\n",
            "ndcg                     1       0.0000\n",
            "ndcg_rel                 1       0.0000\n",
            "Rndcg                    1       0.0000\n",
            "ndcg_cut_5               1       0.0000\n",
            "ndcg_cut_10              1       0.0000\n",
            "ndcg_cut_15              1       0.0000\n",
            "ndcg_cut_20              1       0.0000\n",
            "ndcg_cut_30              1       0.0000\n",
            "ndcg_cut_100             1       0.0000\n",
            "ndcg_cut_200             1       0.0000\n",
            "ndcg_cut_500             1       0.0000\n",
            "ndcg_cut_1000            1       0.0000\n",
            "map_cut_5                1       0.0000\n",
            "map_cut_10               1       0.0000\n",
            "map_cut_15               1       0.0000\n",
            "map_cut_20               1       0.0000\n",
            "map_cut_30               1       0.0000\n",
            "map_cut_100              1       0.0000\n",
            "map_cut_200              1       0.0000\n",
            "map_cut_500              1       0.0000\n",
            "map_cut_1000             1       0.0000\n",
            "relative_P_5             1       0.0000\n",
            "relative_P_10            1       0.0000\n",
            "relative_P_15            1       0.0000\n",
            "relative_P_20            1       0.0000\n",
            "relative_P_30            1       0.0000\n",
            "relative_P_100           1       0.0000\n",
            "relative_P_200           1       0.0000\n",
            "relative_P_500           1       0.0000\n",
            "relative_P_1000          1       0.0000\n",
            "success_1                1       0.0000\n",
            "success_5                1       0.0000\n",
            "success_10               1       0.0000\n",
            "set_P                    1       0.0000\n",
            "set_relative_P           1       0.0000\n",
            "set_recall               1       0.0000\n",
            "set_map                  1       0.0000\n",
            "set_F                    1       0.0000\n",
            "num_nonrel_judged_ret    1       3.0000\n",
            "num_q                    2       1.0000\n",
            "num_ret                  2       13.0000\n",
            "num_rel                  2       2.0000\n",
            "num_rel_ret              2       1.0000\n",
            "map                      2       0.5000\n",
            "gm_map                   2       -0.6931\n",
            "Rprec                    2       0.5000\n",
            "bpref                    2       0.5000\n",
            "recip_rank               2       1.0000\n",
            "iprec_at_recall_0.00     2       1.0000\n",
            "iprec_at_recall_0.10     2       1.0000\n",
            "iprec_at_recall_0.20     2       1.0000\n",
            "iprec_at_recall_0.30     2       1.0000\n",
            "iprec_at_recall_0.40     2       1.0000\n",
            "iprec_at_recall_0.50     2       1.0000\n",
            "iprec_at_recall_0.60     2       0.0000\n",
            "iprec_at_recall_0.70     2       0.0000\n",
            "iprec_at_recall_0.80     2       0.0000\n",
            "iprec_at_recall_0.90     2       0.0000\n",
            "iprec_at_recall_1.00     2       0.0000\n",
            "P_5                      2       0.2000\n",
            "P_10                     2       0.1000\n",
            "P_15                     2       0.0667\n",
            "P_20                     2       0.0500\n",
            "P_30                     2       0.0333\n",
            "P_100                    2       0.0100\n",
            "P_200                    2       0.0050\n",
            "P_500                    2       0.0020\n",
            "P_1000                   2       0.0010\n",
            "relstring                2       0.0000\n",
            "recall_5                 2       0.5000\n",
            "recall_10                2       0.5000\n",
            "recall_15                2       0.5000\n",
            "recall_20                2       0.5000\n",
            "recall_30                2       0.5000\n",
            "recall_100               2       0.5000\n",
            "recall_200               2       0.5000\n",
            "recall_500               2       0.5000\n",
            "recall_1000              2       0.5000\n",
            "infAP                    2       0.5000\n",
            "gm_bpref                 2       -0.6931\n",
            "Rprec_mult_0.20          2       1.0000\n",
            "Rprec_mult_0.40          2       1.0000\n",
            "Rprec_mult_0.60          2       0.5000\n",
            "Rprec_mult_0.80          2       0.5000\n",
            "Rprec_mult_1.00          2       0.5000\n",
            "Rprec_mult_1.20          2       0.3333\n",
            "Rprec_mult_1.40          2       0.3333\n",
            "Rprec_mult_1.60          2       0.2500\n",
            "Rprec_mult_1.80          2       0.2500\n",
            "Rprec_mult_2.00          2       0.2500\n",
            "utility                  2       -11.0000\n",
            "11pt_avg                 2       0.5455\n",
            "binG                     2       0.5000\n",
            "G                        2       0.5000\n",
            "ndcg                     2       0.6131\n",
            "ndcg_rel                 2       0.8066\n",
            "Rndcg                    2       0.6131\n",
            "ndcg_cut_5               2       0.6131\n",
            "ndcg_cut_10              2       0.6131\n",
            "ndcg_cut_15              2       0.6131\n",
            "ndcg_cut_20              2       0.6131\n",
            "ndcg_cut_30              2       0.6131\n",
            "ndcg_cut_100             2       0.6131\n",
            "ndcg_cut_200             2       0.6131\n",
            "ndcg_cut_500             2       0.6131\n",
            "ndcg_cut_1000            2       0.6131\n",
            "map_cut_5                2       0.5000\n",
            "map_cut_10               2       0.5000\n",
            "map_cut_15               2       0.5000\n",
            "map_cut_20               2       0.5000\n",
            "map_cut_30               2       0.5000\n",
            "map_cut_100              2       0.5000\n",
            "map_cut_200              2       0.5000\n",
            "map_cut_500              2       0.5000\n",
            "map_cut_1000             2       0.5000\n",
            "relative_P_5             2       0.5000\n",
            "relative_P_10            2       0.5000\n",
            "relative_P_15            2       0.5000\n",
            "relative_P_20            2       0.5000\n",
            "relative_P_30            2       0.5000\n",
            "relative_P_100           2       0.5000\n",
            "relative_P_200           2       0.5000\n",
            "relative_P_500           2       0.5000\n",
            "relative_P_1000          2       0.5000\n",
            "success_1                2       1.0000\n",
            "success_5                2       1.0000\n",
            "success_10               2       1.0000\n",
            "set_P                    2       0.0769\n",
            "set_relative_P           2       0.5000\n",
            "set_recall               2       0.5000\n",
            "set_map                  2       0.0385\n",
            "set_F                    2       0.1333\n",
            "num_nonrel_judged_ret    2       9.0000\n",
            "num_q                    4       1.0000\n",
            "num_ret                  4       39.0000\n",
            "num_rel                  4       4.0000\n",
            "num_rel_ret              4       3.0000\n",
            "map                      4       0.5375\n",
            "gm_map                   4       -0.6208\n",
            "Rprec                    4       0.5000\n",
            "bpref                    4       0.5000\n",
            "recip_rank               4       1.0000\n",
            "iprec_at_recall_0.00     4       1.0000\n",
            "iprec_at_recall_0.10     4       1.0000\n",
            "iprec_at_recall_0.20     4       1.0000\n",
            "iprec_at_recall_0.30     4       1.0000\n",
            "iprec_at_recall_0.40     4       1.0000\n",
            "iprec_at_recall_0.50     4       1.0000\n",
            "iprec_at_recall_0.60     4       0.1500\n",
            "iprec_at_recall_0.70     4       0.1500\n",
            "iprec_at_recall_0.80     4       0.0000\n",
            "iprec_at_recall_0.90     4       0.0000\n",
            "iprec_at_recall_1.00     4       0.0000\n",
            "P_5                      4       0.4000\n",
            "P_10                     4       0.2000\n",
            "P_15                     4       0.1333\n",
            "P_20                     4       0.1500\n",
            "P_30                     4       0.1000\n",
            "P_100                    4       0.0300\n",
            "P_200                    4       0.0150\n",
            "P_500                    4       0.0060\n",
            "P_1000                   4       0.0030\n",
            "relstring                4       0.0000\n",
            "recall_5                 4       0.5000\n",
            "recall_10                4       0.5000\n",
            "recall_15                4       0.5000\n",
            "recall_20                4       0.7500\n",
            "recall_30                4       0.7500\n",
            "recall_100               4       0.7500\n",
            "recall_200               4       0.7500\n",
            "recall_500               4       0.7500\n",
            "recall_1000              4       0.7500\n",
            "infAP                    4       0.5375\n",
            "gm_bpref                 4       -0.6931\n",
            "Rprec_mult_0.20          4       1.0000\n",
            "Rprec_mult_0.40          4       1.0000\n",
            "Rprec_mult_0.60          4       0.6667\n",
            "Rprec_mult_0.80          4       0.5000\n",
            "Rprec_mult_1.00          4       0.5000\n",
            "Rprec_mult_1.20          4       0.4000\n",
            "Rprec_mult_1.40          4       0.3333\n",
            "Rprec_mult_1.60          4       0.2857\n",
            "Rprec_mult_1.80          4       0.2500\n",
            "Rprec_mult_2.00          4       0.2500\n",
            "utility                  4       -33.0000\n",
            "11pt_avg                 4       0.5727\n",
            "binG                     4       0.5589\n",
            "G                        4       0.5589\n",
            "ndcg                     4       0.7256\n",
            "ndcg_rel                 4       0.8628\n",
            "Rndcg                    4       0.6811\n",
            "ndcg_cut_5               4       0.6367\n",
            "ndcg_cut_10              4       0.6367\n",
            "ndcg_cut_15              4       0.6367\n",
            "ndcg_cut_20              4       0.7256\n",
            "ndcg_cut_30              4       0.7256\n",
            "ndcg_cut_100             4       0.7256\n",
            "ndcg_cut_200             4       0.7256\n",
            "ndcg_cut_500             4       0.7256\n",
            "ndcg_cut_1000            4       0.7256\n",
            "map_cut_5                4       0.5000\n",
            "map_cut_10               4       0.5000\n",
            "map_cut_15               4       0.5000\n",
            "map_cut_20               4       0.5375\n",
            "map_cut_30               4       0.5375\n",
            "map_cut_100              4       0.5375\n",
            "map_cut_200              4       0.5375\n",
            "map_cut_500              4       0.5375\n",
            "map_cut_1000             4       0.5375\n",
            "relative_P_5             4       0.5000\n",
            "relative_P_10            4       0.5000\n",
            "relative_P_15            4       0.5000\n",
            "relative_P_20            4       0.7500\n",
            "relative_P_30            4       0.7500\n",
            "relative_P_100           4       0.7500\n",
            "relative_P_200           4       0.7500\n",
            "relative_P_500           4       0.7500\n",
            "relative_P_1000          4       0.7500\n",
            "success_1                4       1.0000\n",
            "success_5                4       1.0000\n",
            "success_10               4       1.0000\n",
            "set_P                    4       0.0769\n",
            "set_relative_P           4       0.7500\n",
            "set_recall               4       0.7500\n",
            "set_map                  4       0.0577\n",
            "set_F                    4       0.1395\n",
            "num_nonrel_judged_ret    4       24.0000\n",
            "num_q                    6       1.0000\n",
            "num_ret                  6       13.0000\n",
            "num_rel                  6       1.0000\n",
            "num_rel_ret              6       0.0000\n",
            "map                      6       0.0000\n",
            "gm_map                   6       -11.5129\n",
            "Rprec                    6       0.0000\n",
            "bpref                    6       0.0000\n",
            "recip_rank               6       0.0000\n",
            "iprec_at_recall_0.00     6       0.0000\n",
            "iprec_at_recall_0.10     6       0.0000\n",
            "iprec_at_recall_0.20     6       0.0000\n",
            "iprec_at_recall_0.30     6       0.0000\n",
            "iprec_at_recall_0.40     6       0.0000\n",
            "iprec_at_recall_0.50     6       0.0000\n",
            "iprec_at_recall_0.60     6       0.0000\n",
            "iprec_at_recall_0.70     6       0.0000\n",
            "iprec_at_recall_0.80     6       0.0000\n",
            "iprec_at_recall_0.90     6       0.0000\n",
            "iprec_at_recall_1.00     6       0.0000\n",
            "P_5                      6       0.0000\n",
            "P_10                     6       0.0000\n",
            "P_15                     6       0.0000\n",
            "P_20                     6       0.0000\n",
            "P_30                     6       0.0000\n",
            "P_100                    6       0.0000\n",
            "P_200                    6       0.0000\n",
            "P_500                    6       0.0000\n",
            "P_1000                   6       0.0000\n",
            "relstring                6       0.0000\n",
            "recall_5                 6       0.0000\n",
            "recall_10                6       0.0000\n",
            "recall_15                6       0.0000\n",
            "recall_20                6       0.0000\n",
            "recall_30                6       0.0000\n",
            "recall_100               6       0.0000\n",
            "recall_200               6       0.0000\n",
            "recall_500               6       0.0000\n",
            "recall_1000              6       0.0000\n",
            "infAP                    6       0.0000\n",
            "gm_bpref                 6       -11.5129\n",
            "Rprec_mult_0.20          6       0.0000\n",
            "Rprec_mult_0.40          6       0.0000\n",
            "Rprec_mult_0.60          6       0.0000\n",
            "Rprec_mult_0.80          6       0.0000\n",
            "Rprec_mult_1.00          6       0.0000\n",
            "Rprec_mult_1.20          6       0.0000\n",
            "Rprec_mult_1.40          6       0.0000\n",
            "Rprec_mult_1.60          6       0.0000\n",
            "Rprec_mult_1.80          6       0.0000\n",
            "Rprec_mult_2.00          6       0.0000\n",
            "utility                  6       -13.0000\n",
            "11pt_avg                 6       0.0000\n",
            "binG                     6       0.0000\n",
            "G                        6       0.0000\n",
            "ndcg                     6       0.0000\n",
            "ndcg_rel                 6       0.0000\n",
            "Rndcg                    6       0.0000\n",
            "ndcg_cut_5               6       0.0000\n",
            "ndcg_cut_10              6       0.0000\n",
            "ndcg_cut_15              6       0.0000\n",
            "ndcg_cut_20              6       0.0000\n",
            "ndcg_cut_30              6       0.0000\n",
            "ndcg_cut_100             6       0.0000\n",
            "ndcg_cut_200             6       0.0000\n",
            "ndcg_cut_500             6       0.0000\n",
            "ndcg_cut_1000            6       0.0000\n",
            "map_cut_5                6       0.0000\n",
            "map_cut_10               6       0.0000\n",
            "map_cut_15               6       0.0000\n",
            "map_cut_20               6       0.0000\n",
            "map_cut_30               6       0.0000\n",
            "map_cut_100              6       0.0000\n",
            "map_cut_200              6       0.0000\n",
            "map_cut_500              6       0.0000\n",
            "map_cut_1000             6       0.0000\n",
            "relative_P_5             6       0.0000\n",
            "relative_P_10            6       0.0000\n",
            "relative_P_15            6       0.0000\n",
            "relative_P_20            6       0.0000\n",
            "relative_P_30            6       0.0000\n",
            "relative_P_100           6       0.0000\n",
            "relative_P_200           6       0.0000\n",
            "relative_P_500           6       0.0000\n",
            "relative_P_1000          6       0.0000\n",
            "success_1                6       0.0000\n",
            "success_5                6       0.0000\n",
            "success_10               6       0.0000\n",
            "set_P                    6       0.0000\n",
            "set_relative_P           6       0.0000\n",
            "set_recall               6       0.0000\n",
            "set_map                  6       0.0000\n",
            "set_F                    6       0.0000\n",
            "num_nonrel_judged_ret    6       12.0000\n",
            "num_q                    7       1.0000\n",
            "num_ret                  7       14.0000\n",
            "num_rel                  7       3.0000\n",
            "num_rel_ret              7       0.0000\n",
            "map                      7       0.0000\n",
            "gm_map                   7       -11.5129\n",
            "Rprec                    7       0.0000\n",
            "bpref                    7       0.0000\n",
            "recip_rank               7       0.0000\n",
            "iprec_at_recall_0.00     7       0.0000\n",
            "iprec_at_recall_0.10     7       0.0000\n",
            "iprec_at_recall_0.20     7       0.0000\n",
            "iprec_at_recall_0.30     7       0.0000\n",
            "iprec_at_recall_0.40     7       0.0000\n",
            "iprec_at_recall_0.50     7       0.0000\n",
            "iprec_at_recall_0.60     7       0.0000\n",
            "iprec_at_recall_0.70     7       0.0000\n",
            "iprec_at_recall_0.80     7       0.0000\n",
            "iprec_at_recall_0.90     7       0.0000\n",
            "iprec_at_recall_1.00     7       0.0000\n",
            "P_5                      7       0.0000\n",
            "P_10                     7       0.0000\n",
            "P_15                     7       0.0000\n",
            "P_20                     7       0.0000\n",
            "P_30                     7       0.0000\n",
            "P_100                    7       0.0000\n",
            "P_200                    7       0.0000\n",
            "P_500                    7       0.0000\n",
            "P_1000                   7       0.0000\n",
            "relstring                7       0.0000\n",
            "recall_5                 7       0.0000\n",
            "recall_10                7       0.0000\n",
            "recall_15                7       0.0000\n",
            "recall_20                7       0.0000\n",
            "recall_30                7       0.0000\n",
            "recall_100               7       0.0000\n",
            "recall_200               7       0.0000\n",
            "recall_500               7       0.0000\n",
            "recall_1000              7       0.0000\n",
            "infAP                    7       0.0000\n",
            "gm_bpref                 7       -11.5129\n",
            "Rprec_mult_0.20          7       0.0000\n",
            "Rprec_mult_0.40          7       0.0000\n",
            "Rprec_mult_0.60          7       0.0000\n",
            "Rprec_mult_0.80          7       0.0000\n",
            "Rprec_mult_1.00          7       0.0000\n",
            "Rprec_mult_1.20          7       0.0000\n",
            "Rprec_mult_1.40          7       0.0000\n",
            "Rprec_mult_1.60          7       0.0000\n",
            "Rprec_mult_1.80          7       0.0000\n",
            "Rprec_mult_2.00          7       0.0000\n",
            "utility                  7       -14.0000\n",
            "11pt_avg                 7       0.0000\n",
            "binG                     7       0.0000\n",
            "G                        7       0.0000\n",
            "ndcg                     7       0.0000\n",
            "ndcg_rel                 7       0.0000\n",
            "Rndcg                    7       0.0000\n",
            "ndcg_cut_5               7       0.0000\n",
            "ndcg_cut_10              7       0.0000\n",
            "ndcg_cut_15              7       0.0000\n",
            "ndcg_cut_20              7       0.0000\n",
            "ndcg_cut_30              7       0.0000\n",
            "ndcg_cut_100             7       0.0000\n",
            "ndcg_cut_200             7       0.0000\n",
            "ndcg_cut_500             7       0.0000\n",
            "ndcg_cut_1000            7       0.0000\n",
            "map_cut_5                7       0.0000\n",
            "map_cut_10               7       0.0000\n",
            "map_cut_15               7       0.0000\n",
            "map_cut_20               7       0.0000\n",
            "map_cut_30               7       0.0000\n",
            "map_cut_100              7       0.0000\n",
            "map_cut_200              7       0.0000\n",
            "map_cut_500              7       0.0000\n",
            "map_cut_1000             7       0.0000\n",
            "relative_P_5             7       0.0000\n",
            "relative_P_10            7       0.0000\n",
            "relative_P_15            7       0.0000\n",
            "relative_P_20            7       0.0000\n",
            "relative_P_30            7       0.0000\n",
            "relative_P_100           7       0.0000\n",
            "relative_P_200           7       0.0000\n",
            "relative_P_500           7       0.0000\n",
            "relative_P_1000          7       0.0000\n",
            "success_1                7       0.0000\n",
            "success_5                7       0.0000\n",
            "success_10               7       0.0000\n",
            "set_P                    7       0.0000\n",
            "set_relative_P           7       0.0000\n",
            "set_recall               7       0.0000\n",
            "set_map                  7       0.0000\n",
            "set_F                    7       0.0000\n",
            "num_nonrel_judged_ret    7       11.0000\n",
            "num_q                    9       1.0000\n",
            "num_ret                  9       27.0000\n",
            "num_rel                  9       1.0000\n",
            "num_rel_ret              9       1.0000\n",
            "map                      9       0.0714\n",
            "gm_map                   9       -2.6391\n",
            "Rprec                    9       0.0000\n",
            "bpref                    9       0.0000\n",
            "recip_rank               9       0.0714\n",
            "iprec_at_recall_0.00     9       0.0714\n",
            "iprec_at_recall_0.10     9       0.0714\n",
            "iprec_at_recall_0.20     9       0.0714\n",
            "iprec_at_recall_0.30     9       0.0714\n",
            "iprec_at_recall_0.40     9       0.0714\n",
            "iprec_at_recall_0.50     9       0.0714\n",
            "iprec_at_recall_0.60     9       0.0714\n",
            "iprec_at_recall_0.70     9       0.0714\n",
            "iprec_at_recall_0.80     9       0.0714\n",
            "iprec_at_recall_0.90     9       0.0714\n",
            "iprec_at_recall_1.00     9       0.0714\n",
            "P_5                      9       0.0000\n",
            "P_10                     9       0.0000\n",
            "P_15                     9       0.0667\n",
            "P_20                     9       0.0500\n",
            "P_30                     9       0.0333\n",
            "P_100                    9       0.0100\n",
            "P_200                    9       0.0050\n",
            "P_500                    9       0.0020\n",
            "P_1000                   9       0.0010\n",
            "relstring                9       0.0000\n",
            "recall_5                 9       0.0000\n",
            "recall_10                9       0.0000\n",
            "recall_15                9       1.0000\n",
            "recall_20                9       1.0000\n",
            "recall_30                9       1.0000\n",
            "recall_100               9       1.0000\n",
            "recall_200               9       1.0000\n",
            "recall_500               9       1.0000\n",
            "recall_1000              9       1.0000\n",
            "infAP                    9       0.0714\n",
            "gm_bpref                 9       -11.5129\n",
            "Rprec_mult_0.20          9       0.0000\n",
            "Rprec_mult_0.40          9       0.0000\n",
            "Rprec_mult_0.60          9       0.0000\n",
            "Rprec_mult_0.80          9       0.0000\n",
            "Rprec_mult_1.00          9       0.0000\n",
            "Rprec_mult_1.20          9       0.0000\n",
            "Rprec_mult_1.40          9       0.0000\n",
            "Rprec_mult_1.60          9       0.0000\n",
            "Rprec_mult_1.80          9       0.0000\n",
            "Rprec_mult_2.00          9       0.0000\n",
            "utility                  9       -25.0000\n",
            "11pt_avg                 9       0.0714\n",
            "binG                     9       0.2560\n",
            "G                        9       0.2560\n",
            "ndcg                     9       0.2560\n",
            "ndcg_rel                 9       0.2560\n",
            "Rndcg                    9       0.1280\n",
            "ndcg_cut_5               9       0.0000\n",
            "ndcg_cut_10              9       0.0000\n",
            "ndcg_cut_15              9       0.2560\n",
            "ndcg_cut_20              9       0.2560\n",
            "ndcg_cut_30              9       0.2560\n",
            "ndcg_cut_100             9       0.2560\n",
            "ndcg_cut_200             9       0.2560\n",
            "ndcg_cut_500             9       0.2560\n",
            "ndcg_cut_1000            9       0.2560\n",
            "map_cut_5                9       0.0000\n",
            "map_cut_10               9       0.0000\n",
            "map_cut_15               9       0.0714\n",
            "map_cut_20               9       0.0714\n",
            "map_cut_30               9       0.0714\n",
            "map_cut_100              9       0.0714\n",
            "map_cut_200              9       0.0714\n",
            "map_cut_500              9       0.0714\n",
            "map_cut_1000             9       0.0714\n",
            "relative_P_5             9       0.0000\n",
            "relative_P_10            9       0.0000\n",
            "relative_P_15            9       1.0000\n",
            "relative_P_20            9       1.0000\n",
            "relative_P_30            9       1.0000\n",
            "relative_P_100           9       1.0000\n",
            "relative_P_200           9       1.0000\n",
            "relative_P_500           9       1.0000\n",
            "relative_P_1000          9       1.0000\n",
            "success_1                9       0.0000\n",
            "success_5                9       0.0000\n",
            "success_10               9       0.0000\n",
            "set_P                    9       0.0370\n",
            "set_relative_P           9       1.0000\n",
            "set_recall               9       1.0000\n",
            "set_map                  9       0.0370\n",
            "set_F                    9       0.0714\n",
            "num_nonrel_judged_ret    9       23.0000\n",
            "num_q                    10      1.0000\n",
            "num_ret                  10      42.0000\n",
            "num_rel                  10      1.0000\n",
            "num_rel_ret              10      1.0000\n",
            "map                      10      0.2500\n",
            "gm_map                   10      -1.3863\n",
            "Rprec                    10      0.0000\n",
            "bpref                    10      0.0000\n",
            "recip_rank               10      0.2500\n",
            "iprec_at_recall_0.00     10      0.2500\n",
            "iprec_at_recall_0.10     10      0.2500\n",
            "iprec_at_recall_0.20     10      0.2500\n",
            "iprec_at_recall_0.30     10      0.2500\n",
            "iprec_at_recall_0.40     10      0.2500\n",
            "iprec_at_recall_0.50     10      0.2500\n",
            "iprec_at_recall_0.60     10      0.2500\n",
            "iprec_at_recall_0.70     10      0.2500\n",
            "iprec_at_recall_0.80     10      0.2500\n",
            "iprec_at_recall_0.90     10      0.2500\n",
            "iprec_at_recall_1.00     10      0.2500\n",
            "P_5                      10      0.2000\n",
            "P_10                     10      0.1000\n",
            "P_15                     10      0.0667\n",
            "P_20                     10      0.0500\n",
            "P_30                     10      0.0333\n",
            "P_100                    10      0.0100\n",
            "P_200                    10      0.0050\n",
            "P_500                    10      0.0020\n",
            "P_1000                   10      0.0010\n",
            "relstring                10      0.0000\n",
            "recall_5                 10      1.0000\n",
            "recall_10                10      1.0000\n",
            "recall_15                10      1.0000\n",
            "recall_20                10      1.0000\n",
            "recall_30                10      1.0000\n",
            "recall_100               10      1.0000\n",
            "recall_200               10      1.0000\n",
            "recall_500               10      1.0000\n",
            "recall_1000              10      1.0000\n",
            "infAP                    10      0.2500\n",
            "gm_bpref                 10      -11.5129\n",
            "Rprec_mult_0.20          10      0.0000\n",
            "Rprec_mult_0.40          10      0.0000\n",
            "Rprec_mult_0.60          10      0.0000\n",
            "Rprec_mult_0.80          10      0.0000\n",
            "Rprec_mult_1.00          10      0.0000\n",
            "Rprec_mult_1.20          10      0.0000\n",
            "Rprec_mult_1.40          10      0.0000\n",
            "Rprec_mult_1.60          10      0.0000\n",
            "Rprec_mult_1.80          10      0.0000\n",
            "Rprec_mult_2.00          10      0.0000\n",
            "utility                  10      -40.0000\n",
            "11pt_avg                 10      0.2500\n",
            "binG                     10      0.4307\n",
            "G                        10      0.4307\n",
            "ndcg                     10      0.4307\n",
            "ndcg_rel                 10      0.4307\n",
            "Rndcg                    10      0.2153\n",
            "ndcg_cut_5               10      0.4307\n",
            "ndcg_cut_10              10      0.4307\n",
            "ndcg_cut_15              10      0.4307\n",
            "ndcg_cut_20              10      0.4307\n",
            "ndcg_cut_30              10      0.4307\n",
            "ndcg_cut_100             10      0.4307\n",
            "ndcg_cut_200             10      0.4307\n",
            "ndcg_cut_500             10      0.4307\n",
            "ndcg_cut_1000            10      0.4307\n",
            "map_cut_5                10      0.2500\n",
            "map_cut_10               10      0.2500\n",
            "map_cut_15               10      0.2500\n",
            "map_cut_20               10      0.2500\n",
            "map_cut_30               10      0.2500\n",
            "map_cut_100              10      0.2500\n",
            "map_cut_200              10      0.2500\n",
            "map_cut_500              10      0.2500\n",
            "map_cut_1000             10      0.2500\n",
            "relative_P_5             10      1.0000\n",
            "relative_P_10            10      1.0000\n",
            "relative_P_15            10      1.0000\n",
            "relative_P_20            10      1.0000\n",
            "relative_P_30            10      1.0000\n",
            "relative_P_100           10      1.0000\n",
            "relative_P_200           10      1.0000\n",
            "relative_P_500           10      1.0000\n",
            "relative_P_1000          10      1.0000\n",
            "success_1                10      0.0000\n",
            "success_5                10      1.0000\n",
            "success_10               10      1.0000\n",
            "set_P                    10      0.0238\n",
            "set_relative_P           10      1.0000\n",
            "set_recall               10      1.0000\n",
            "set_map                  10      0.0238\n",
            "set_F                    10      0.0465\n",
            "num_nonrel_judged_ret    10      13.0000\n",
            "num_q                    14      1.0000\n",
            "num_ret                  14      26.0000\n",
            "num_rel                  14      1.0000\n",
            "num_rel_ret              14      1.0000\n",
            "map                      14      1.0000\n",
            "gm_map                   14      0.0000\n",
            "Rprec                    14      1.0000\n",
            "bpref                    14      1.0000\n",
            "recip_rank               14      1.0000\n",
            "iprec_at_recall_0.00     14      1.0000\n",
            "iprec_at_recall_0.10     14      1.0000\n",
            "iprec_at_recall_0.20     14      1.0000\n",
            "iprec_at_recall_0.30     14      1.0000\n",
            "iprec_at_recall_0.40     14      1.0000\n",
            "iprec_at_recall_0.50     14      1.0000\n",
            "iprec_at_recall_0.60     14      1.0000\n",
            "iprec_at_recall_0.70     14      1.0000\n",
            "iprec_at_recall_0.80     14      1.0000\n",
            "iprec_at_recall_0.90     14      1.0000\n",
            "iprec_at_recall_1.00     14      1.0000\n",
            "P_5                      14      0.2000\n",
            "P_10                     14      0.1000\n",
            "P_15                     14      0.0667\n",
            "P_20                     14      0.0500\n",
            "P_30                     14      0.0333\n",
            "P_100                    14      0.0100\n",
            "P_200                    14      0.0050\n",
            "P_500                    14      0.0020\n",
            "P_1000                   14      0.0010\n",
            "relstring                14      0.0000\n",
            "recall_5                 14      1.0000\n",
            "recall_10                14      1.0000\n",
            "recall_15                14      1.0000\n",
            "recall_20                14      1.0000\n",
            "recall_30                14      1.0000\n",
            "recall_100               14      1.0000\n",
            "recall_200               14      1.0000\n",
            "recall_500               14      1.0000\n",
            "recall_1000              14      1.0000\n",
            "infAP                    14      1.0000\n",
            "gm_bpref                 14      0.0000\n",
            "Rprec_mult_0.20          14      1.0000\n",
            "Rprec_mult_0.40          14      1.0000\n",
            "Rprec_mult_0.60          14      1.0000\n",
            "Rprec_mult_0.80          14      1.0000\n",
            "Rprec_mult_1.00          14      1.0000\n",
            "Rprec_mult_1.20          14      0.5000\n",
            "Rprec_mult_1.40          14      0.5000\n",
            "Rprec_mult_1.60          14      0.5000\n",
            "Rprec_mult_1.80          14      0.5000\n",
            "Rprec_mult_2.00          14      0.5000\n",
            "utility                  14      -24.0000\n",
            "11pt_avg                 14      1.0000\n",
            "binG                     14      1.0000\n",
            "G                        14      1.0000\n",
            "ndcg                     14      1.0000\n",
            "ndcg_rel                 14      1.0000\n",
            "Rndcg                    14      1.0000\n",
            "ndcg_cut_5               14      1.0000\n",
            "ndcg_cut_10              14      1.0000\n",
            "ndcg_cut_15              14      1.0000\n",
            "ndcg_cut_20              14      1.0000\n",
            "ndcg_cut_30              14      1.0000\n",
            "ndcg_cut_100             14      1.0000\n",
            "ndcg_cut_200             14      1.0000\n",
            "ndcg_cut_500             14      1.0000\n",
            "ndcg_cut_1000            14      1.0000\n",
            "map_cut_5                14      1.0000\n",
            "map_cut_10               14      1.0000\n",
            "map_cut_15               14      1.0000\n",
            "map_cut_20               14      1.0000\n",
            "map_cut_30               14      1.0000\n",
            "map_cut_100              14      1.0000\n",
            "map_cut_200              14      1.0000\n",
            "map_cut_500              14      1.0000\n",
            "map_cut_1000             14      1.0000\n",
            "relative_P_5             14      1.0000\n",
            "relative_P_10            14      1.0000\n",
            "relative_P_15            14      1.0000\n",
            "relative_P_20            14      1.0000\n",
            "relative_P_30            14      1.0000\n",
            "relative_P_100           14      1.0000\n",
            "relative_P_200           14      1.0000\n",
            "relative_P_500           14      1.0000\n",
            "relative_P_1000          14      1.0000\n",
            "success_1                14      1.0000\n",
            "success_5                14      1.0000\n",
            "success_10               14      1.0000\n",
            "set_P                    14      0.0385\n",
            "set_relative_P           14      1.0000\n",
            "set_recall               14      1.0000\n",
            "set_map                  14      0.0385\n",
            "set_F                    14      0.0741\n",
            "num_nonrel_judged_ret    14      20.0000\n",
            "num_q                    16      1.0000\n",
            "num_ret                  16      24.0000\n",
            "num_rel                  16      7.0000\n",
            "num_rel_ret              16      0.0000\n",
            "map                      16      0.0000\n",
            "gm_map                   16      -11.5129\n",
            "Rprec                    16      0.0000\n",
            "bpref                    16      0.0000\n",
            "recip_rank               16      0.0000\n",
            "iprec_at_recall_0.00     16      0.0000\n",
            "iprec_at_recall_0.10     16      0.0000\n",
            "iprec_at_recall_0.20     16      0.0000\n",
            "iprec_at_recall_0.30     16      0.0000\n",
            "iprec_at_recall_0.40     16      0.0000\n",
            "iprec_at_recall_0.50     16      0.0000\n",
            "iprec_at_recall_0.60     16      0.0000\n",
            "iprec_at_recall_0.70     16      0.0000\n",
            "iprec_at_recall_0.80     16      0.0000\n",
            "iprec_at_recall_0.90     16      0.0000\n",
            "iprec_at_recall_1.00     16      0.0000\n",
            "P_5                      16      0.0000\n",
            "P_10                     16      0.0000\n",
            "P_15                     16      0.0000\n",
            "P_20                     16      0.0000\n",
            "P_30                     16      0.0000\n",
            "P_100                    16      0.0000\n",
            "P_200                    16      0.0000\n",
            "P_500                    16      0.0000\n",
            "P_1000                   16      0.0000\n",
            "relstring                16      0.0000\n",
            "recall_5                 16      0.0000\n",
            "recall_10                16      0.0000\n",
            "recall_15                16      0.0000\n",
            "recall_20                16      0.0000\n",
            "recall_30                16      0.0000\n",
            "recall_100               16      0.0000\n",
            "recall_200               16      0.0000\n",
            "recall_500               16      0.0000\n",
            "recall_1000              16      0.0000\n",
            "infAP                    16      0.0000\n",
            "gm_bpref                 16      -11.5129\n",
            "Rprec_mult_0.20          16      0.0000\n",
            "Rprec_mult_0.40          16      0.0000\n",
            "Rprec_mult_0.60          16      0.0000\n",
            "Rprec_mult_0.80          16      0.0000\n",
            "Rprec_mult_1.00          16      0.0000\n",
            "Rprec_mult_1.20          16      0.0000\n",
            "Rprec_mult_1.40          16      0.0000\n",
            "Rprec_mult_1.60          16      0.0000\n",
            "Rprec_mult_1.80          16      0.0000\n",
            "Rprec_mult_2.00          16      0.0000\n",
            "utility                  16      -24.0000\n",
            "11pt_avg                 16      0.0000\n",
            "binG                     16      0.0000\n",
            "G                        16      0.0000\n",
            "ndcg                     16      0.0000\n",
            "ndcg_rel                 16      0.0000\n",
            "Rndcg                    16      0.0000\n",
            "ndcg_cut_5               16      0.0000\n",
            "ndcg_cut_10              16      0.0000\n",
            "ndcg_cut_15              16      0.0000\n",
            "ndcg_cut_20              16      0.0000\n",
            "ndcg_cut_30              16      0.0000\n",
            "ndcg_cut_100             16      0.0000\n",
            "ndcg_cut_200             16      0.0000\n",
            "ndcg_cut_500             16      0.0000\n",
            "ndcg_cut_1000            16      0.0000\n",
            "map_cut_5                16      0.0000\n",
            "map_cut_10               16      0.0000\n",
            "map_cut_15               16      0.0000\n",
            "map_cut_20               16      0.0000\n",
            "map_cut_30               16      0.0000\n",
            "map_cut_100              16      0.0000\n",
            "map_cut_200              16      0.0000\n",
            "map_cut_500              16      0.0000\n",
            "map_cut_1000             16      0.0000\n",
            "relative_P_5             16      0.0000\n",
            "relative_P_10            16      0.0000\n",
            "relative_P_15            16      0.0000\n",
            "relative_P_20            16      0.0000\n",
            "relative_P_30            16      0.0000\n",
            "relative_P_100           16      0.0000\n",
            "relative_P_200           16      0.0000\n",
            "relative_P_500           16      0.0000\n",
            "relative_P_1000          16      0.0000\n",
            "success_1                16      0.0000\n",
            "success_5                16      0.0000\n",
            "success_10               16      0.0000\n",
            "set_P                    16      0.0000\n",
            "set_relative_P           16      0.0000\n",
            "set_recall               16      0.0000\n",
            "set_map                  16      0.0000\n",
            "set_F                    16      0.0000\n",
            "num_nonrel_judged_ret    16      18.0000\n",
            "num_q                    18      1.0000\n",
            "num_ret                  18      32.0000\n",
            "num_rel                  18      1.0000\n",
            "num_rel_ret              18      1.0000\n",
            "map                      18      1.0000\n",
            "gm_map                   18      0.0000\n",
            "Rprec                    18      1.0000\n",
            "bpref                    18      1.0000\n",
            "recip_rank               18      1.0000\n",
            "iprec_at_recall_0.00     18      1.0000\n",
            "iprec_at_recall_0.10     18      1.0000\n",
            "iprec_at_recall_0.20     18      1.0000\n",
            "iprec_at_recall_0.30     18      1.0000\n",
            "iprec_at_recall_0.40     18      1.0000\n",
            "iprec_at_recall_0.50     18      1.0000\n",
            "iprec_at_recall_0.60     18      1.0000\n",
            "iprec_at_recall_0.70     18      1.0000\n",
            "iprec_at_recall_0.80     18      1.0000\n",
            "iprec_at_recall_0.90     18      1.0000\n",
            "iprec_at_recall_1.00     18      1.0000\n",
            "P_5                      18      0.2000\n",
            "P_10                     18      0.1000\n",
            "P_15                     18      0.0667\n",
            "P_20                     18      0.0500\n",
            "P_30                     18      0.0333\n",
            "P_100                    18      0.0100\n",
            "P_200                    18      0.0050\n",
            "P_500                    18      0.0020\n",
            "P_1000                   18      0.0010\n",
            "relstring                18      0.0000\n",
            "recall_5                 18      1.0000\n",
            "recall_10                18      1.0000\n",
            "recall_15                18      1.0000\n",
            "recall_20                18      1.0000\n",
            "recall_30                18      1.0000\n",
            "recall_100               18      1.0000\n",
            "recall_200               18      1.0000\n",
            "recall_500               18      1.0000\n",
            "recall_1000              18      1.0000\n",
            "infAP                    18      1.0000\n",
            "gm_bpref                 18      0.0000\n",
            "Rprec_mult_0.20          18      1.0000\n",
            "Rprec_mult_0.40          18      1.0000\n",
            "Rprec_mult_0.60          18      1.0000\n",
            "Rprec_mult_0.80          18      1.0000\n",
            "Rprec_mult_1.00          18      1.0000\n",
            "Rprec_mult_1.20          18      0.5000\n",
            "Rprec_mult_1.40          18      0.5000\n",
            "Rprec_mult_1.60          18      0.5000\n",
            "Rprec_mult_1.80          18      0.5000\n",
            "Rprec_mult_2.00          18      0.5000\n",
            "utility                  18      -30.0000\n",
            "11pt_avg                 18      1.0000\n",
            "binG                     18      1.0000\n",
            "G                        18      1.0000\n",
            "ndcg                     18      1.0000\n",
            "ndcg_rel                 18      1.0000\n",
            "Rndcg                    18      1.0000\n",
            "ndcg_cut_5               18      1.0000\n",
            "ndcg_cut_10              18      1.0000\n",
            "ndcg_cut_15              18      1.0000\n",
            "ndcg_cut_20              18      1.0000\n",
            "ndcg_cut_30              18      1.0000\n",
            "ndcg_cut_100             18      1.0000\n",
            "ndcg_cut_200             18      1.0000\n",
            "ndcg_cut_500             18      1.0000\n",
            "ndcg_cut_1000            18      1.0000\n",
            "map_cut_5                18      1.0000\n",
            "map_cut_10               18      1.0000\n",
            "map_cut_15               18      1.0000\n",
            "map_cut_20               18      1.0000\n",
            "map_cut_30               18      1.0000\n",
            "map_cut_100              18      1.0000\n",
            "map_cut_200              18      1.0000\n",
            "map_cut_500              18      1.0000\n",
            "map_cut_1000             18      1.0000\n",
            "relative_P_5             18      1.0000\n",
            "relative_P_10            18      1.0000\n",
            "relative_P_15            18      1.0000\n",
            "relative_P_20            18      1.0000\n",
            "relative_P_30            18      1.0000\n",
            "relative_P_100           18      1.0000\n",
            "relative_P_200           18      1.0000\n",
            "relative_P_500           18      1.0000\n",
            "relative_P_1000          18      1.0000\n",
            "success_1                18      1.0000\n",
            "success_5                18      1.0000\n",
            "success_10               18      1.0000\n",
            "set_P                    18      0.0312\n",
            "set_relative_P           18      1.0000\n",
            "set_recall               18      1.0000\n",
            "set_map                  18      0.0312\n",
            "set_F                    18      0.0606\n",
            "num_nonrel_judged_ret    18      30.0000\n",
            "num_q                    19      1.0000\n",
            "num_ret                  19      5.0000\n",
            "num_rel                  19      2.0000\n",
            "num_rel_ret              19      1.0000\n",
            "map                      19      0.5000\n",
            "gm_map                   19      -0.6931\n",
            "Rprec                    19      0.5000\n",
            "bpref                    19      0.5000\n",
            "recip_rank               19      1.0000\n",
            "iprec_at_recall_0.00     19      1.0000\n",
            "iprec_at_recall_0.10     19      1.0000\n",
            "iprec_at_recall_0.20     19      1.0000\n",
            "iprec_at_recall_0.30     19      1.0000\n",
            "iprec_at_recall_0.40     19      1.0000\n",
            "iprec_at_recall_0.50     19      1.0000\n",
            "iprec_at_recall_0.60     19      0.0000\n",
            "iprec_at_recall_0.70     19      0.0000\n",
            "iprec_at_recall_0.80     19      0.0000\n",
            "iprec_at_recall_0.90     19      0.0000\n",
            "iprec_at_recall_1.00     19      0.0000\n",
            "P_5                      19      0.2000\n",
            "P_10                     19      0.1000\n",
            "P_15                     19      0.0667\n",
            "P_20                     19      0.0500\n",
            "P_30                     19      0.0333\n",
            "P_100                    19      0.0100\n",
            "P_200                    19      0.0050\n",
            "P_500                    19      0.0020\n",
            "P_1000                   19      0.0010\n",
            "relstring                19      0.0000\n",
            "recall_5                 19      0.5000\n",
            "recall_10                19      0.5000\n",
            "recall_15                19      0.5000\n",
            "recall_20                19      0.5000\n",
            "recall_30                19      0.5000\n",
            "recall_100               19      0.5000\n",
            "recall_200               19      0.5000\n",
            "recall_500               19      0.5000\n",
            "recall_1000              19      0.5000\n",
            "infAP                    19      0.5000\n",
            "gm_bpref                 19      -0.6931\n",
            "Rprec_mult_0.20          19      1.0000\n",
            "Rprec_mult_0.40          19      1.0000\n",
            "Rprec_mult_0.60          19      0.5000\n",
            "Rprec_mult_0.80          19      0.5000\n",
            "Rprec_mult_1.00          19      0.5000\n",
            "Rprec_mult_1.20          19      0.3333\n",
            "Rprec_mult_1.40          19      0.3333\n",
            "Rprec_mult_1.60          19      0.2500\n",
            "Rprec_mult_1.80          19      0.2500\n",
            "Rprec_mult_2.00          19      0.2500\n",
            "utility                  19      -3.0000\n",
            "11pt_avg                 19      0.5455\n",
            "binG                     19      0.5000\n",
            "G                        19      0.5000\n",
            "ndcg                     19      0.6131\n",
            "ndcg_rel                 19      0.8066\n",
            "Rndcg                    19      0.6131\n",
            "ndcg_cut_5               19      0.6131\n",
            "ndcg_cut_10              19      0.6131\n",
            "ndcg_cut_15              19      0.6131\n",
            "ndcg_cut_20              19      0.6131\n",
            "ndcg_cut_30              19      0.6131\n",
            "ndcg_cut_100             19      0.6131\n",
            "ndcg_cut_200             19      0.6131\n",
            "ndcg_cut_500             19      0.6131\n",
            "ndcg_cut_1000            19      0.6131\n",
            "map_cut_5                19      0.5000\n",
            "map_cut_10               19      0.5000\n",
            "map_cut_15               19      0.5000\n",
            "map_cut_20               19      0.5000\n",
            "map_cut_30               19      0.5000\n",
            "map_cut_100              19      0.5000\n",
            "map_cut_200              19      0.5000\n",
            "map_cut_500              19      0.5000\n",
            "map_cut_1000             19      0.5000\n",
            "relative_P_5             19      0.5000\n",
            "relative_P_10            19      0.5000\n",
            "relative_P_15            19      0.5000\n",
            "relative_P_20            19      0.5000\n",
            "relative_P_30            19      0.5000\n",
            "relative_P_100           19      0.5000\n",
            "relative_P_200           19      0.5000\n",
            "relative_P_500           19      0.5000\n",
            "relative_P_1000          19      0.5000\n",
            "success_1                19      1.0000\n",
            "success_5                19      1.0000\n",
            "success_10               19      1.0000\n",
            "set_P                    19      0.2000\n",
            "set_relative_P           19      0.5000\n",
            "set_recall               19      0.5000\n",
            "set_map                  19      0.1000\n",
            "set_F                    19      0.2857\n",
            "num_nonrel_judged_ret    19      4.0000\n",
            "num_q                    22      1.0000\n",
            "num_ret                  22      107.0000\n",
            "num_rel                  22      1.0000\n",
            "num_rel_ret              22      1.0000\n",
            "map                      22      0.0370\n",
            "gm_map                   22      -3.2958\n",
            "Rprec                    22      0.0000\n",
            "bpref                    22      0.0000\n",
            "recip_rank               22      0.0370\n",
            "iprec_at_recall_0.00     22      0.0370\n",
            "iprec_at_recall_0.10     22      0.0370\n",
            "iprec_at_recall_0.20     22      0.0370\n",
            "iprec_at_recall_0.30     22      0.0370\n",
            "iprec_at_recall_0.40     22      0.0370\n",
            "iprec_at_recall_0.50     22      0.0370\n",
            "iprec_at_recall_0.60     22      0.0370\n",
            "iprec_at_recall_0.70     22      0.0370\n",
            "iprec_at_recall_0.80     22      0.0370\n",
            "iprec_at_recall_0.90     22      0.0370\n",
            "iprec_at_recall_1.00     22      0.0370\n",
            "P_5                      22      0.0000\n",
            "P_10                     22      0.0000\n",
            "P_15                     22      0.0000\n",
            "P_20                     22      0.0000\n",
            "P_30                     22      0.0333\n",
            "P_100                    22      0.0100\n",
            "P_200                    22      0.0050\n",
            "P_500                    22      0.0020\n",
            "P_1000                   22      0.0010\n",
            "relstring                22      0.0000\n",
            "recall_5                 22      0.0000\n",
            "recall_10                22      0.0000\n",
            "recall_15                22      0.0000\n",
            "recall_20                22      0.0000\n",
            "recall_30                22      1.0000\n",
            "recall_100               22      1.0000\n",
            "recall_200               22      1.0000\n",
            "recall_500               22      1.0000\n",
            "recall_1000              22      1.0000\n",
            "infAP                    22      0.0370\n",
            "gm_bpref                 22      -11.5129\n",
            "Rprec_mult_0.20          22      0.0000\n",
            "Rprec_mult_0.40          22      0.0000\n",
            "Rprec_mult_0.60          22      0.0000\n",
            "Rprec_mult_0.80          22      0.0000\n",
            "Rprec_mult_1.00          22      0.0000\n",
            "Rprec_mult_1.20          22      0.0000\n",
            "Rprec_mult_1.40          22      0.0000\n",
            "Rprec_mult_1.60          22      0.0000\n",
            "Rprec_mult_1.80          22      0.0000\n",
            "Rprec_mult_2.00          22      0.0000\n",
            "utility                  22      -105.0000\n",
            "11pt_avg                 22      0.0370\n",
            "binG                     22      0.2080\n",
            "G                        22      0.2080\n",
            "ndcg                     22      0.2080\n",
            "ndcg_rel                 22      0.2080\n",
            "Rndcg                    22      0.1040\n",
            "ndcg_cut_5               22      0.0000\n",
            "ndcg_cut_10              22      0.0000\n",
            "ndcg_cut_15              22      0.0000\n",
            "ndcg_cut_20              22      0.0000\n",
            "ndcg_cut_30              22      0.2080\n",
            "ndcg_cut_100             22      0.2080\n",
            "ndcg_cut_200             22      0.2080\n",
            "ndcg_cut_500             22      0.2080\n",
            "ndcg_cut_1000            22      0.2080\n",
            "map_cut_5                22      0.0000\n",
            "map_cut_10               22      0.0000\n",
            "map_cut_15               22      0.0000\n",
            "map_cut_20               22      0.0000\n",
            "map_cut_30               22      0.0370\n",
            "map_cut_100              22      0.0370\n",
            "map_cut_200              22      0.0370\n",
            "map_cut_500              22      0.0370\n",
            "map_cut_1000             22      0.0370\n",
            "relative_P_5             22      0.0000\n",
            "relative_P_10            22      0.0000\n",
            "relative_P_15            22      0.0000\n",
            "relative_P_20            22      0.0000\n",
            "relative_P_30            22      1.0000\n",
            "relative_P_100           22      1.0000\n",
            "relative_P_200           22      1.0000\n",
            "relative_P_500           22      1.0000\n",
            "relative_P_1000          22      1.0000\n",
            "success_1                22      0.0000\n",
            "success_5                22      0.0000\n",
            "success_10               22      0.0000\n",
            "set_P                    22      0.0093\n",
            "set_relative_P           22      1.0000\n",
            "set_recall               22      1.0000\n",
            "set_map                  22      0.0093\n",
            "set_F                    22      0.0185\n",
            "num_nonrel_judged_ret    22      72.0000\n",
            "num_q                    24      1.0000\n",
            "num_ret                  24      20.0000\n",
            "num_rel                  24      1.0000\n",
            "num_rel_ret              24      1.0000\n",
            "map                      24      1.0000\n",
            "gm_map                   24      0.0000\n",
            "Rprec                    24      1.0000\n",
            "bpref                    24      1.0000\n",
            "recip_rank               24      1.0000\n",
            "iprec_at_recall_0.00     24      1.0000\n",
            "iprec_at_recall_0.10     24      1.0000\n",
            "iprec_at_recall_0.20     24      1.0000\n",
            "iprec_at_recall_0.30     24      1.0000\n",
            "iprec_at_recall_0.40     24      1.0000\n",
            "iprec_at_recall_0.50     24      1.0000\n",
            "iprec_at_recall_0.60     24      1.0000\n",
            "iprec_at_recall_0.70     24      1.0000\n",
            "iprec_at_recall_0.80     24      1.0000\n",
            "iprec_at_recall_0.90     24      1.0000\n",
            "iprec_at_recall_1.00     24      1.0000\n",
            "P_5                      24      0.2000\n",
            "P_10                     24      0.1000\n",
            "P_15                     24      0.0667\n",
            "P_20                     24      0.0500\n",
            "P_30                     24      0.0333\n",
            "P_100                    24      0.0100\n",
            "P_200                    24      0.0050\n",
            "P_500                    24      0.0020\n",
            "P_1000                   24      0.0010\n",
            "relstring                24      0.0000\n",
            "recall_5                 24      1.0000\n",
            "recall_10                24      1.0000\n",
            "recall_15                24      1.0000\n",
            "recall_20                24      1.0000\n",
            "recall_30                24      1.0000\n",
            "recall_100               24      1.0000\n",
            "recall_200               24      1.0000\n",
            "recall_500               24      1.0000\n",
            "recall_1000              24      1.0000\n",
            "infAP                    24      1.0000\n",
            "gm_bpref                 24      0.0000\n",
            "Rprec_mult_0.20          24      1.0000\n",
            "Rprec_mult_0.40          24      1.0000\n",
            "Rprec_mult_0.60          24      1.0000\n",
            "Rprec_mult_0.80          24      1.0000\n",
            "Rprec_mult_1.00          24      1.0000\n",
            "Rprec_mult_1.20          24      0.5000\n",
            "Rprec_mult_1.40          24      0.5000\n",
            "Rprec_mult_1.60          24      0.5000\n",
            "Rprec_mult_1.80          24      0.5000\n",
            "Rprec_mult_2.00          24      0.5000\n",
            "utility                  24      -18.0000\n",
            "11pt_avg                 24      1.0000\n",
            "binG                     24      1.0000\n",
            "G                        24      1.0000\n",
            "ndcg                     24      1.0000\n",
            "ndcg_rel                 24      1.0000\n",
            "Rndcg                    24      1.0000\n",
            "ndcg_cut_5               24      1.0000\n",
            "ndcg_cut_10              24      1.0000\n",
            "ndcg_cut_15              24      1.0000\n",
            "ndcg_cut_20              24      1.0000\n",
            "ndcg_cut_30              24      1.0000\n",
            "ndcg_cut_100             24      1.0000\n",
            "ndcg_cut_200             24      1.0000\n",
            "ndcg_cut_500             24      1.0000\n",
            "ndcg_cut_1000            24      1.0000\n",
            "map_cut_5                24      1.0000\n",
            "map_cut_10               24      1.0000\n",
            "map_cut_15               24      1.0000\n",
            "map_cut_20               24      1.0000\n",
            "map_cut_30               24      1.0000\n",
            "map_cut_100              24      1.0000\n",
            "map_cut_200              24      1.0000\n",
            "map_cut_500              24      1.0000\n",
            "map_cut_1000             24      1.0000\n",
            "relative_P_5             24      1.0000\n",
            "relative_P_10            24      1.0000\n",
            "relative_P_15            24      1.0000\n",
            "relative_P_20            24      1.0000\n",
            "relative_P_30            24      1.0000\n",
            "relative_P_100           24      1.0000\n",
            "relative_P_200           24      1.0000\n",
            "relative_P_500           24      1.0000\n",
            "relative_P_1000          24      1.0000\n",
            "success_1                24      1.0000\n",
            "success_5                24      1.0000\n",
            "success_10               24      1.0000\n",
            "set_P                    24      0.0500\n",
            "set_relative_P           24      1.0000\n",
            "set_recall               24      1.0000\n",
            "set_map                  24      0.0500\n",
            "set_F                    24      0.0952\n",
            "num_nonrel_judged_ret    24      17.0000\n",
            "num_q                    26      1.0000\n",
            "num_ret                  26      51.0000\n",
            "num_rel                  26      3.0000\n",
            "num_rel_ret              26      2.0000\n",
            "map                      26      0.0771\n",
            "gm_map                   26      -2.5632\n",
            "Rprec                    26      0.0000\n",
            "bpref                    26      0.0000\n",
            "recip_rank               26      0.1667\n",
            "iprec_at_recall_0.00     26      0.1667\n",
            "iprec_at_recall_0.10     26      0.1667\n",
            "iprec_at_recall_0.20     26      0.1667\n",
            "iprec_at_recall_0.30     26      0.1667\n",
            "iprec_at_recall_0.40     26      0.0645\n",
            "iprec_at_recall_0.50     26      0.0645\n",
            "iprec_at_recall_0.60     26      0.0645\n",
            "iprec_at_recall_0.70     26      0.0645\n",
            "iprec_at_recall_0.80     26      0.0000\n",
            "iprec_at_recall_0.90     26      0.0000\n",
            "iprec_at_recall_1.00     26      0.0000\n",
            "P_5                      26      0.0000\n",
            "P_10                     26      0.1000\n",
            "P_15                     26      0.0667\n",
            "P_20                     26      0.0500\n",
            "P_30                     26      0.0333\n",
            "P_100                    26      0.0200\n",
            "P_200                    26      0.0100\n",
            "P_500                    26      0.0040\n",
            "P_1000                   26      0.0020\n",
            "relstring                26      0.0000\n",
            "recall_5                 26      0.0000\n",
            "recall_10                26      0.3333\n",
            "recall_15                26      0.3333\n",
            "recall_20                26      0.3333\n",
            "recall_30                26      0.3333\n",
            "recall_100               26      0.6667\n",
            "recall_200               26      0.6667\n",
            "recall_500               26      0.6667\n",
            "recall_1000              26      0.6667\n",
            "infAP                    26      0.0771\n",
            "gm_bpref                 26      -11.5129\n",
            "Rprec_mult_0.20          26      0.0000\n",
            "Rprec_mult_0.40          26      0.0000\n",
            "Rprec_mult_0.60          26      0.0000\n",
            "Rprec_mult_0.80          26      0.0000\n",
            "Rprec_mult_1.00          26      0.0000\n",
            "Rprec_mult_1.20          26      0.0000\n",
            "Rprec_mult_1.40          26      0.0000\n",
            "Rprec_mult_1.60          26      0.0000\n",
            "Rprec_mult_1.80          26      0.1667\n",
            "Rprec_mult_2.00          26      0.1667\n",
            "utility                  26      -47.0000\n",
            "11pt_avg                 26      0.0841\n",
            "binG                     26      0.1860\n",
            "G                        26      0.1860\n",
            "ndcg                     26      0.2610\n",
            "ndcg_rel                 26      0.2297\n",
            "Rndcg                    26      0.1305\n",
            "ndcg_cut_5               26      0.0000\n",
            "ndcg_cut_10              26      0.1672\n",
            "ndcg_cut_15              26      0.1672\n",
            "ndcg_cut_20              26      0.1672\n",
            "ndcg_cut_30              26      0.1672\n",
            "ndcg_cut_100             26      0.2610\n",
            "ndcg_cut_200             26      0.2610\n",
            "ndcg_cut_500             26      0.2610\n",
            "ndcg_cut_1000            26      0.2610\n",
            "map_cut_5                26      0.0000\n",
            "map_cut_10               26      0.0556\n",
            "map_cut_15               26      0.0556\n",
            "map_cut_20               26      0.0556\n",
            "map_cut_30               26      0.0556\n",
            "map_cut_100              26      0.0771\n",
            "map_cut_200              26      0.0771\n",
            "map_cut_500              26      0.0771\n",
            "map_cut_1000             26      0.0771\n",
            "relative_P_5             26      0.0000\n",
            "relative_P_10            26      0.3333\n",
            "relative_P_15            26      0.3333\n",
            "relative_P_20            26      0.3333\n",
            "relative_P_30            26      0.3333\n",
            "relative_P_100           26      0.6667\n",
            "relative_P_200           26      0.6667\n",
            "relative_P_500           26      0.6667\n",
            "relative_P_1000          26      0.6667\n",
            "success_1                26      0.0000\n",
            "success_5                26      0.0000\n",
            "success_10               26      1.0000\n",
            "set_P                    26      0.0392\n",
            "set_relative_P           26      0.6667\n",
            "set_recall               26      0.6667\n",
            "set_map                  26      0.0261\n",
            "set_F                    26      0.0741\n",
            "num_nonrel_judged_ret    26      36.0000\n",
            "num_q                    28      1.0000\n",
            "num_ret                  28      37.0000\n",
            "num_rel                  28      2.0000\n",
            "num_rel_ret              28      2.0000\n",
            "map                      28      0.0711\n",
            "gm_map                   28      -2.6440\n",
            "Rprec                    28      0.0000\n",
            "bpref                    28      0.0000\n",
            "recip_rank               28      0.0588\n",
            "iprec_at_recall_0.00     28      0.0833\n",
            "iprec_at_recall_0.10     28      0.0833\n",
            "iprec_at_recall_0.20     28      0.0833\n",
            "iprec_at_recall_0.30     28      0.0833\n",
            "iprec_at_recall_0.40     28      0.0833\n",
            "iprec_at_recall_0.50     28      0.0833\n",
            "iprec_at_recall_0.60     28      0.0833\n",
            "iprec_at_recall_0.70     28      0.0833\n",
            "iprec_at_recall_0.80     28      0.0833\n",
            "iprec_at_recall_0.90     28      0.0833\n",
            "iprec_at_recall_1.00     28      0.0833\n",
            "P_5                      28      0.0000\n",
            "P_10                     28      0.0000\n",
            "P_15                     28      0.0000\n",
            "P_20                     28      0.0500\n",
            "P_30                     28      0.0667\n",
            "P_100                    28      0.0200\n",
            "P_200                    28      0.0100\n",
            "P_500                    28      0.0040\n",
            "P_1000                   28      0.0020\n",
            "relstring                28      0.0000\n",
            "recall_5                 28      0.0000\n",
            "recall_10                28      0.0000\n",
            "recall_15                28      0.0000\n",
            "recall_20                28      0.5000\n",
            "recall_30                28      1.0000\n",
            "recall_100               28      1.0000\n",
            "recall_200               28      1.0000\n",
            "recall_500               28      1.0000\n",
            "recall_1000              28      1.0000\n",
            "infAP                    28      0.0711\n",
            "gm_bpref                 28      -11.5129\n",
            "Rprec_mult_0.20          28      0.0000\n",
            "Rprec_mult_0.40          28      0.0000\n",
            "Rprec_mult_0.60          28      0.0000\n",
            "Rprec_mult_0.80          28      0.0000\n",
            "Rprec_mult_1.00          28      0.0000\n",
            "Rprec_mult_1.20          28      0.0000\n",
            "Rprec_mult_1.40          28      0.0000\n",
            "Rprec_mult_1.60          28      0.0000\n",
            "Rprec_mult_1.80          28      0.0000\n",
            "Rprec_mult_2.00          28      0.0000\n",
            "utility                  28      -33.0000\n",
            "11pt_avg                 28      0.0833\n",
            "binG                     28      0.2290\n",
            "G                        28      0.2290\n",
            "ndcg                     28      0.2791\n",
            "ndcg_rel                 28      0.2131\n",
            "Rndcg                    28      0.1395\n",
            "ndcg_cut_5               28      0.0000\n",
            "ndcg_cut_10              28      0.0000\n",
            "ndcg_cut_15              28      0.0000\n",
            "ndcg_cut_20              28      0.1470\n",
            "ndcg_cut_30              28      0.2791\n",
            "ndcg_cut_100             28      0.2791\n",
            "ndcg_cut_200             28      0.2791\n",
            "ndcg_cut_500             28      0.2791\n",
            "ndcg_cut_1000            28      0.2791\n",
            "map_cut_5                28      0.0000\n",
            "map_cut_10               28      0.0000\n",
            "map_cut_15               28      0.0000\n",
            "map_cut_20               28      0.0294\n",
            "map_cut_30               28      0.0711\n",
            "map_cut_100              28      0.0711\n",
            "map_cut_200              28      0.0711\n",
            "map_cut_500              28      0.0711\n",
            "map_cut_1000             28      0.0711\n",
            "relative_P_5             28      0.0000\n",
            "relative_P_10            28      0.0000\n",
            "relative_P_15            28      0.0000\n",
            "relative_P_20            28      0.5000\n",
            "relative_P_30            28      1.0000\n",
            "relative_P_100           28      1.0000\n",
            "relative_P_200           28      1.0000\n",
            "relative_P_500           28      1.0000\n",
            "relative_P_1000          28      1.0000\n",
            "success_1                28      0.0000\n",
            "success_5                28      0.0000\n",
            "success_10               28      0.0000\n",
            "set_P                    28      0.0541\n",
            "set_relative_P           28      1.0000\n",
            "set_recall               28      1.0000\n",
            "set_map                  28      0.0541\n",
            "set_F                    28      0.1026\n",
            "num_nonrel_judged_ret    28      26.0000\n",
            "num_q                    all     15.0000\n",
            "num_ret                  all     453.0000\n",
            "num_rel                  all     35.0000\n",
            "num_rel_ret              all     15.0000\n",
            "map                      all     0.3363\n",
            "gm_map                   all     0.0176\n",
            "Rprec                    all     0.3000\n",
            "bpref                    all     0.3000\n",
            "recip_rank               all     0.4389\n",
            "iprec_at_recall_0.00     all     0.4406\n",
            "iprec_at_recall_0.10     all     0.4406\n",
            "iprec_at_recall_0.20     all     0.4406\n",
            "iprec_at_recall_0.30     all     0.4406\n",
            "iprec_at_recall_0.40     all     0.4338\n",
            "iprec_at_recall_0.50     all     0.4338\n",
            "iprec_at_recall_0.60     all     0.2438\n",
            "iprec_at_recall_0.70     all     0.2438\n",
            "iprec_at_recall_0.80     all     0.2295\n",
            "iprec_at_recall_0.90     all     0.2295\n",
            "iprec_at_recall_1.00     all     0.2295\n",
            "P_5                      all     0.1067\n",
            "P_10                     all     0.0600\n",
            "P_15                     all     0.0444\n",
            "P_20                     all     0.0400\n",
            "P_30                     all     0.0311\n",
            "P_100                    all     0.0100\n",
            "P_200                    all     0.0050\n",
            "P_500                    all     0.0020\n",
            "P_1000                   all     0.0010\n",
            "relstring                all     0.0000\n",
            "recall_5                 all     0.3667\n",
            "recall_10                all     0.3889\n",
            "recall_15                all     0.4556\n",
            "recall_20                all     0.5056\n",
            "recall_30                all     0.6056\n",
            "recall_100               all     0.6278\n",
            "recall_200               all     0.6278\n",
            "recall_500               all     0.6278\n",
            "recall_1000              all     0.6278\n",
            "infAP                    all     0.3363\n",
            "gm_bpref                 all     0.0009\n",
            "Rprec_mult_0.20          all     0.4000\n",
            "Rprec_mult_0.40          all     0.4000\n",
            "Rprec_mult_0.60          all     0.3111\n",
            "Rprec_mult_0.80          all     0.3000\n",
            "Rprec_mult_1.00          all     0.3000\n",
            "Rprec_mult_1.20          all     0.1711\n",
            "Rprec_mult_1.40          all     0.1667\n",
            "Rprec_mult_1.60          all     0.1524\n",
            "Rprec_mult_1.80          all     0.1611\n",
            "Rprec_mult_2.00          all     0.1611\n",
            "utility                  all     -28.2000\n",
            "11pt_avg                 all     0.3460\n",
            "binG                     all     0.3912\n",
            "G                        all     0.3912\n",
            "ndcg                     all     0.4258\n",
            "ndcg_rel                 all     0.4542\n",
            "Rndcg                    all     0.3750\n",
            "ndcg_cut_5               all     0.3529\n",
            "ndcg_cut_10              all     0.3641\n",
            "ndcg_cut_15              all     0.3811\n",
            "ndcg_cut_20              all     0.3968\n",
            "ndcg_cut_30              all     0.4195\n",
            "ndcg_cut_100             all     0.4258\n",
            "ndcg_cut_200             all     0.4258\n",
            "ndcg_cut_500             all     0.4258\n",
            "ndcg_cut_1000            all     0.4258\n",
            "map_cut_5                all     0.3167\n",
            "map_cut_10               all     0.3204\n",
            "map_cut_15               all     0.3251\n",
            "map_cut_20               all     0.3296\n",
            "map_cut_30               all     0.3348\n",
            "map_cut_100              all     0.3363\n",
            "map_cut_200              all     0.3363\n",
            "map_cut_500              all     0.3363\n",
            "map_cut_1000             all     0.3363\n",
            "relative_P_5             all     0.3667\n",
            "relative_P_10            all     0.3889\n",
            "relative_P_15            all     0.4556\n",
            "relative_P_20            all     0.5056\n",
            "relative_P_30            all     0.6056\n",
            "relative_P_100           all     0.6278\n",
            "relative_P_200           all     0.6278\n",
            "relative_P_500           all     0.6278\n",
            "relative_P_1000          all     0.6278\n",
            "success_1                all     0.4000\n",
            "success_5                all     0.4667\n",
            "success_10               all     0.5333\n",
            "set_P                    all     0.0425\n",
            "set_relative_P           all     0.6278\n",
            "set_recall               all     0.6278\n",
            "set_map                  all     0.0311\n",
            "set_F                    all     0.0734\n",
            "num_nonrel_judged_ret    all     318.0000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JMqzySpCH5qo",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "cd04777d-acee-4cba-fafa-00f80260b6f2"
      },
      "source": [
        "#Add lancaster stemmer to previous analyzer\n",
        "New_Analyzer=RegexTokenizer() | LowercaseFilter() | StopFilter()| StemFilter()|CustomFilter(LancasterStemmer().stem)\n",
        "New_Schema=Schema(file_path = ID(stored=True), file_content = TEXT(analyzer = New_Analyzer))\n",
        "NEW_INDEX = createIndex(New_Schema)\n",
        "addFilesToIndex(NEW_INDEX, filesToIndex)\n",
        "NEW_QP = QueryParser(\"file_content\", schema=NEW_INDEX.schema)\n",
        "NEW_SEARCHER = NEW_INDEX.searcher()\n",
        "pyTrecEval(topic_file, qrels_file, NEW_QP, NEW_SEARCHER) \n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "num_q                    1       1.0000\n",
            "num_ret                  1       3.0000\n",
            "num_rel                  1       5.0000\n",
            "num_rel_ret              1       0.0000\n",
            "map                      1       0.0000\n",
            "gm_map                   1       -11.5129\n",
            "Rprec                    1       0.0000\n",
            "bpref                    1       0.0000\n",
            "recip_rank               1       0.0000\n",
            "iprec_at_recall_0.00     1       0.0000\n",
            "iprec_at_recall_0.10     1       0.0000\n",
            "iprec_at_recall_0.20     1       0.0000\n",
            "iprec_at_recall_0.30     1       0.0000\n",
            "iprec_at_recall_0.40     1       0.0000\n",
            "iprec_at_recall_0.50     1       0.0000\n",
            "iprec_at_recall_0.60     1       0.0000\n",
            "iprec_at_recall_0.70     1       0.0000\n",
            "iprec_at_recall_0.80     1       0.0000\n",
            "iprec_at_recall_0.90     1       0.0000\n",
            "iprec_at_recall_1.00     1       0.0000\n",
            "P_5                      1       0.0000\n",
            "P_10                     1       0.0000\n",
            "P_15                     1       0.0000\n",
            "P_20                     1       0.0000\n",
            "P_30                     1       0.0000\n",
            "P_100                    1       0.0000\n",
            "P_200                    1       0.0000\n",
            "P_500                    1       0.0000\n",
            "P_1000                   1       0.0000\n",
            "relstring                1       0.0000\n",
            "recall_5                 1       0.0000\n",
            "recall_10                1       0.0000\n",
            "recall_15                1       0.0000\n",
            "recall_20                1       0.0000\n",
            "recall_30                1       0.0000\n",
            "recall_100               1       0.0000\n",
            "recall_200               1       0.0000\n",
            "recall_500               1       0.0000\n",
            "recall_1000              1       0.0000\n",
            "infAP                    1       0.0000\n",
            "gm_bpref                 1       -11.5129\n",
            "Rprec_mult_0.20          1       0.0000\n",
            "Rprec_mult_0.40          1       0.0000\n",
            "Rprec_mult_0.60          1       0.0000\n",
            "Rprec_mult_0.80          1       0.0000\n",
            "Rprec_mult_1.00          1       0.0000\n",
            "Rprec_mult_1.20          1       0.0000\n",
            "Rprec_mult_1.40          1       0.0000\n",
            "Rprec_mult_1.60          1       0.0000\n",
            "Rprec_mult_1.80          1       0.0000\n",
            "Rprec_mult_2.00          1       0.0000\n",
            "utility                  1       -3.0000\n",
            "11pt_avg                 1       0.0000\n",
            "binG                     1       0.0000\n",
            "G                        1       0.0000\n",
            "ndcg                     1       0.0000\n",
            "ndcg_rel                 1       0.0000\n",
            "Rndcg                    1       0.0000\n",
            "ndcg_cut_5               1       0.0000\n",
            "ndcg_cut_10              1       0.0000\n",
            "ndcg_cut_15              1       0.0000\n",
            "ndcg_cut_20              1       0.0000\n",
            "ndcg_cut_30              1       0.0000\n",
            "ndcg_cut_100             1       0.0000\n",
            "ndcg_cut_200             1       0.0000\n",
            "ndcg_cut_500             1       0.0000\n",
            "ndcg_cut_1000            1       0.0000\n",
            "map_cut_5                1       0.0000\n",
            "map_cut_10               1       0.0000\n",
            "map_cut_15               1       0.0000\n",
            "map_cut_20               1       0.0000\n",
            "map_cut_30               1       0.0000\n",
            "map_cut_100              1       0.0000\n",
            "map_cut_200              1       0.0000\n",
            "map_cut_500              1       0.0000\n",
            "map_cut_1000             1       0.0000\n",
            "relative_P_5             1       0.0000\n",
            "relative_P_10            1       0.0000\n",
            "relative_P_15            1       0.0000\n",
            "relative_P_20            1       0.0000\n",
            "relative_P_30            1       0.0000\n",
            "relative_P_100           1       0.0000\n",
            "relative_P_200           1       0.0000\n",
            "relative_P_500           1       0.0000\n",
            "relative_P_1000          1       0.0000\n",
            "success_1                1       0.0000\n",
            "success_5                1       0.0000\n",
            "success_10               1       0.0000\n",
            "set_P                    1       0.0000\n",
            "set_relative_P           1       0.0000\n",
            "set_recall               1       0.0000\n",
            "set_map                  1       0.0000\n",
            "set_F                    1       0.0000\n",
            "num_nonrel_judged_ret    1       3.0000\n",
            "num_q                    2       1.0000\n",
            "num_ret                  2       13.0000\n",
            "num_rel                  2       2.0000\n",
            "num_rel_ret              2       1.0000\n",
            "map                      2       0.5000\n",
            "gm_map                   2       -0.6931\n",
            "Rprec                    2       0.5000\n",
            "bpref                    2       0.5000\n",
            "recip_rank               2       1.0000\n",
            "iprec_at_recall_0.00     2       1.0000\n",
            "iprec_at_recall_0.10     2       1.0000\n",
            "iprec_at_recall_0.20     2       1.0000\n",
            "iprec_at_recall_0.30     2       1.0000\n",
            "iprec_at_recall_0.40     2       1.0000\n",
            "iprec_at_recall_0.50     2       1.0000\n",
            "iprec_at_recall_0.60     2       0.0000\n",
            "iprec_at_recall_0.70     2       0.0000\n",
            "iprec_at_recall_0.80     2       0.0000\n",
            "iprec_at_recall_0.90     2       0.0000\n",
            "iprec_at_recall_1.00     2       0.0000\n",
            "P_5                      2       0.2000\n",
            "P_10                     2       0.1000\n",
            "P_15                     2       0.0667\n",
            "P_20                     2       0.0500\n",
            "P_30                     2       0.0333\n",
            "P_100                    2       0.0100\n",
            "P_200                    2       0.0050\n",
            "P_500                    2       0.0020\n",
            "P_1000                   2       0.0010\n",
            "relstring                2       0.0000\n",
            "recall_5                 2       0.5000\n",
            "recall_10                2       0.5000\n",
            "recall_15                2       0.5000\n",
            "recall_20                2       0.5000\n",
            "recall_30                2       0.5000\n",
            "recall_100               2       0.5000\n",
            "recall_200               2       0.5000\n",
            "recall_500               2       0.5000\n",
            "recall_1000              2       0.5000\n",
            "infAP                    2       0.5000\n",
            "gm_bpref                 2       -0.6931\n",
            "Rprec_mult_0.20          2       1.0000\n",
            "Rprec_mult_0.40          2       1.0000\n",
            "Rprec_mult_0.60          2       0.5000\n",
            "Rprec_mult_0.80          2       0.5000\n",
            "Rprec_mult_1.00          2       0.5000\n",
            "Rprec_mult_1.20          2       0.3333\n",
            "Rprec_mult_1.40          2       0.3333\n",
            "Rprec_mult_1.60          2       0.2500\n",
            "Rprec_mult_1.80          2       0.2500\n",
            "Rprec_mult_2.00          2       0.2500\n",
            "utility                  2       -11.0000\n",
            "11pt_avg                 2       0.5455\n",
            "binG                     2       0.5000\n",
            "G                        2       0.5000\n",
            "ndcg                     2       0.6131\n",
            "ndcg_rel                 2       0.8066\n",
            "Rndcg                    2       0.6131\n",
            "ndcg_cut_5               2       0.6131\n",
            "ndcg_cut_10              2       0.6131\n",
            "ndcg_cut_15              2       0.6131\n",
            "ndcg_cut_20              2       0.6131\n",
            "ndcg_cut_30              2       0.6131\n",
            "ndcg_cut_100             2       0.6131\n",
            "ndcg_cut_200             2       0.6131\n",
            "ndcg_cut_500             2       0.6131\n",
            "ndcg_cut_1000            2       0.6131\n",
            "map_cut_5                2       0.5000\n",
            "map_cut_10               2       0.5000\n",
            "map_cut_15               2       0.5000\n",
            "map_cut_20               2       0.5000\n",
            "map_cut_30               2       0.5000\n",
            "map_cut_100              2       0.5000\n",
            "map_cut_200              2       0.5000\n",
            "map_cut_500              2       0.5000\n",
            "map_cut_1000             2       0.5000\n",
            "relative_P_5             2       0.5000\n",
            "relative_P_10            2       0.5000\n",
            "relative_P_15            2       0.5000\n",
            "relative_P_20            2       0.5000\n",
            "relative_P_30            2       0.5000\n",
            "relative_P_100           2       0.5000\n",
            "relative_P_200           2       0.5000\n",
            "relative_P_500           2       0.5000\n",
            "relative_P_1000          2       0.5000\n",
            "success_1                2       1.0000\n",
            "success_5                2       1.0000\n",
            "success_10               2       1.0000\n",
            "set_P                    2       0.0769\n",
            "set_relative_P           2       0.5000\n",
            "set_recall               2       0.5000\n",
            "set_map                  2       0.0385\n",
            "set_F                    2       0.1333\n",
            "num_nonrel_judged_ret    2       9.0000\n",
            "num_q                    4       1.0000\n",
            "num_ret                  4       40.0000\n",
            "num_rel                  4       4.0000\n",
            "num_rel_ret              4       3.0000\n",
            "map                      4       0.5375\n",
            "gm_map                   4       -0.6208\n",
            "Rprec                    4       0.5000\n",
            "bpref                    4       0.5000\n",
            "recip_rank               4       1.0000\n",
            "iprec_at_recall_0.00     4       1.0000\n",
            "iprec_at_recall_0.10     4       1.0000\n",
            "iprec_at_recall_0.20     4       1.0000\n",
            "iprec_at_recall_0.30     4       1.0000\n",
            "iprec_at_recall_0.40     4       1.0000\n",
            "iprec_at_recall_0.50     4       1.0000\n",
            "iprec_at_recall_0.60     4       0.1500\n",
            "iprec_at_recall_0.70     4       0.1500\n",
            "iprec_at_recall_0.80     4       0.0000\n",
            "iprec_at_recall_0.90     4       0.0000\n",
            "iprec_at_recall_1.00     4       0.0000\n",
            "P_5                      4       0.4000\n",
            "P_10                     4       0.2000\n",
            "P_15                     4       0.1333\n",
            "P_20                     4       0.1500\n",
            "P_30                     4       0.1000\n",
            "P_100                    4       0.0300\n",
            "P_200                    4       0.0150\n",
            "P_500                    4       0.0060\n",
            "P_1000                   4       0.0030\n",
            "relstring                4       0.0000\n",
            "recall_5                 4       0.5000\n",
            "recall_10                4       0.5000\n",
            "recall_15                4       0.5000\n",
            "recall_20                4       0.7500\n",
            "recall_30                4       0.7500\n",
            "recall_100               4       0.7500\n",
            "recall_200               4       0.7500\n",
            "recall_500               4       0.7500\n",
            "recall_1000              4       0.7500\n",
            "infAP                    4       0.5375\n",
            "gm_bpref                 4       -0.6931\n",
            "Rprec_mult_0.20          4       1.0000\n",
            "Rprec_mult_0.40          4       1.0000\n",
            "Rprec_mult_0.60          4       0.6667\n",
            "Rprec_mult_0.80          4       0.5000\n",
            "Rprec_mult_1.00          4       0.5000\n",
            "Rprec_mult_1.20          4       0.4000\n",
            "Rprec_mult_1.40          4       0.3333\n",
            "Rprec_mult_1.60          4       0.2857\n",
            "Rprec_mult_1.80          4       0.2500\n",
            "Rprec_mult_2.00          4       0.2500\n",
            "utility                  4       -34.0000\n",
            "11pt_avg                 4       0.5727\n",
            "binG                     4       0.5589\n",
            "G                        4       0.5589\n",
            "ndcg                     4       0.7256\n",
            "ndcg_rel                 4       0.8628\n",
            "Rndcg                    4       0.6811\n",
            "ndcg_cut_5               4       0.6367\n",
            "ndcg_cut_10              4       0.6367\n",
            "ndcg_cut_15              4       0.6367\n",
            "ndcg_cut_20              4       0.7256\n",
            "ndcg_cut_30              4       0.7256\n",
            "ndcg_cut_100             4       0.7256\n",
            "ndcg_cut_200             4       0.7256\n",
            "ndcg_cut_500             4       0.7256\n",
            "ndcg_cut_1000            4       0.7256\n",
            "map_cut_5                4       0.5000\n",
            "map_cut_10               4       0.5000\n",
            "map_cut_15               4       0.5000\n",
            "map_cut_20               4       0.5375\n",
            "map_cut_30               4       0.5375\n",
            "map_cut_100              4       0.5375\n",
            "map_cut_200              4       0.5375\n",
            "map_cut_500              4       0.5375\n",
            "map_cut_1000             4       0.5375\n",
            "relative_P_5             4       0.5000\n",
            "relative_P_10            4       0.5000\n",
            "relative_P_15            4       0.5000\n",
            "relative_P_20            4       0.7500\n",
            "relative_P_30            4       0.7500\n",
            "relative_P_100           4       0.7500\n",
            "relative_P_200           4       0.7500\n",
            "relative_P_500           4       0.7500\n",
            "relative_P_1000          4       0.7500\n",
            "success_1                4       1.0000\n",
            "success_5                4       1.0000\n",
            "success_10               4       1.0000\n",
            "set_P                    4       0.0750\n",
            "set_relative_P           4       0.7500\n",
            "set_recall               4       0.7500\n",
            "set_map                  4       0.0563\n",
            "set_F                    4       0.1364\n",
            "num_nonrel_judged_ret    4       25.0000\n",
            "num_q                    6       1.0000\n",
            "num_ret                  6       40.0000\n",
            "num_rel                  6       1.0000\n",
            "num_rel_ret              6       1.0000\n",
            "map                      6       0.1667\n",
            "gm_map                   6       -1.7918\n",
            "Rprec                    6       0.0000\n",
            "bpref                    6       0.0000\n",
            "recip_rank               6       0.1667\n",
            "iprec_at_recall_0.00     6       0.1667\n",
            "iprec_at_recall_0.10     6       0.1667\n",
            "iprec_at_recall_0.20     6       0.1667\n",
            "iprec_at_recall_0.30     6       0.1667\n",
            "iprec_at_recall_0.40     6       0.1667\n",
            "iprec_at_recall_0.50     6       0.1667\n",
            "iprec_at_recall_0.60     6       0.1667\n",
            "iprec_at_recall_0.70     6       0.1667\n",
            "iprec_at_recall_0.80     6       0.1667\n",
            "iprec_at_recall_0.90     6       0.1667\n",
            "iprec_at_recall_1.00     6       0.1667\n",
            "P_5                      6       0.0000\n",
            "P_10                     6       0.1000\n",
            "P_15                     6       0.0667\n",
            "P_20                     6       0.0500\n",
            "P_30                     6       0.0333\n",
            "P_100                    6       0.0100\n",
            "P_200                    6       0.0050\n",
            "P_500                    6       0.0020\n",
            "P_1000                   6       0.0010\n",
            "relstring                6       0.0000\n",
            "recall_5                 6       0.0000\n",
            "recall_10                6       1.0000\n",
            "recall_15                6       1.0000\n",
            "recall_20                6       1.0000\n",
            "recall_30                6       1.0000\n",
            "recall_100               6       1.0000\n",
            "recall_200               6       1.0000\n",
            "recall_500               6       1.0000\n",
            "recall_1000              6       1.0000\n",
            "infAP                    6       0.1667\n",
            "gm_bpref                 6       -11.5129\n",
            "Rprec_mult_0.20          6       0.0000\n",
            "Rprec_mult_0.40          6       0.0000\n",
            "Rprec_mult_0.60          6       0.0000\n",
            "Rprec_mult_0.80          6       0.0000\n",
            "Rprec_mult_1.00          6       0.0000\n",
            "Rprec_mult_1.20          6       0.0000\n",
            "Rprec_mult_1.40          6       0.0000\n",
            "Rprec_mult_1.60          6       0.0000\n",
            "Rprec_mult_1.80          6       0.0000\n",
            "Rprec_mult_2.00          6       0.0000\n",
            "utility                  6       -38.0000\n",
            "11pt_avg                 6       0.1667\n",
            "binG                     6       0.3562\n",
            "G                        6       0.3562\n",
            "ndcg                     6       0.3562\n",
            "ndcg_rel                 6       0.3562\n",
            "Rndcg                    6       0.1781\n",
            "ndcg_cut_5               6       0.0000\n",
            "ndcg_cut_10              6       0.3562\n",
            "ndcg_cut_15              6       0.3562\n",
            "ndcg_cut_20              6       0.3562\n",
            "ndcg_cut_30              6       0.3562\n",
            "ndcg_cut_100             6       0.3562\n",
            "ndcg_cut_200             6       0.3562\n",
            "ndcg_cut_500             6       0.3562\n",
            "ndcg_cut_1000            6       0.3562\n",
            "map_cut_5                6       0.0000\n",
            "map_cut_10               6       0.1667\n",
            "map_cut_15               6       0.1667\n",
            "map_cut_20               6       0.1667\n",
            "map_cut_30               6       0.1667\n",
            "map_cut_100              6       0.1667\n",
            "map_cut_200              6       0.1667\n",
            "map_cut_500              6       0.1667\n",
            "map_cut_1000             6       0.1667\n",
            "relative_P_5             6       0.0000\n",
            "relative_P_10            6       1.0000\n",
            "relative_P_15            6       1.0000\n",
            "relative_P_20            6       1.0000\n",
            "relative_P_30            6       1.0000\n",
            "relative_P_100           6       1.0000\n",
            "relative_P_200           6       1.0000\n",
            "relative_P_500           6       1.0000\n",
            "relative_P_1000          6       1.0000\n",
            "success_1                6       0.0000\n",
            "success_5                6       0.0000\n",
            "success_10               6       1.0000\n",
            "set_P                    6       0.0250\n",
            "set_relative_P           6       1.0000\n",
            "set_recall               6       1.0000\n",
            "set_map                  6       0.0250\n",
            "set_F                    6       0.0488\n",
            "num_nonrel_judged_ret    6       17.0000\n",
            "num_q                    7       1.0000\n",
            "num_ret                  7       14.0000\n",
            "num_rel                  7       3.0000\n",
            "num_rel_ret              7       0.0000\n",
            "map                      7       0.0000\n",
            "gm_map                   7       -11.5129\n",
            "Rprec                    7       0.0000\n",
            "bpref                    7       0.0000\n",
            "recip_rank               7       0.0000\n",
            "iprec_at_recall_0.00     7       0.0000\n",
            "iprec_at_recall_0.10     7       0.0000\n",
            "iprec_at_recall_0.20     7       0.0000\n",
            "iprec_at_recall_0.30     7       0.0000\n",
            "iprec_at_recall_0.40     7       0.0000\n",
            "iprec_at_recall_0.50     7       0.0000\n",
            "iprec_at_recall_0.60     7       0.0000\n",
            "iprec_at_recall_0.70     7       0.0000\n",
            "iprec_at_recall_0.80     7       0.0000\n",
            "iprec_at_recall_0.90     7       0.0000\n",
            "iprec_at_recall_1.00     7       0.0000\n",
            "P_5                      7       0.0000\n",
            "P_10                     7       0.0000\n",
            "P_15                     7       0.0000\n",
            "P_20                     7       0.0000\n",
            "P_30                     7       0.0000\n",
            "P_100                    7       0.0000\n",
            "P_200                    7       0.0000\n",
            "P_500                    7       0.0000\n",
            "P_1000                   7       0.0000\n",
            "relstring                7       0.0000\n",
            "recall_5                 7       0.0000\n",
            "recall_10                7       0.0000\n",
            "recall_15                7       0.0000\n",
            "recall_20                7       0.0000\n",
            "recall_30                7       0.0000\n",
            "recall_100               7       0.0000\n",
            "recall_200               7       0.0000\n",
            "recall_500               7       0.0000\n",
            "recall_1000              7       0.0000\n",
            "infAP                    7       0.0000\n",
            "gm_bpref                 7       -11.5129\n",
            "Rprec_mult_0.20          7       0.0000\n",
            "Rprec_mult_0.40          7       0.0000\n",
            "Rprec_mult_0.60          7       0.0000\n",
            "Rprec_mult_0.80          7       0.0000\n",
            "Rprec_mult_1.00          7       0.0000\n",
            "Rprec_mult_1.20          7       0.0000\n",
            "Rprec_mult_1.40          7       0.0000\n",
            "Rprec_mult_1.60          7       0.0000\n",
            "Rprec_mult_1.80          7       0.0000\n",
            "Rprec_mult_2.00          7       0.0000\n",
            "utility                  7       -14.0000\n",
            "11pt_avg                 7       0.0000\n",
            "binG                     7       0.0000\n",
            "G                        7       0.0000\n",
            "ndcg                     7       0.0000\n",
            "ndcg_rel                 7       0.0000\n",
            "Rndcg                    7       0.0000\n",
            "ndcg_cut_5               7       0.0000\n",
            "ndcg_cut_10              7       0.0000\n",
            "ndcg_cut_15              7       0.0000\n",
            "ndcg_cut_20              7       0.0000\n",
            "ndcg_cut_30              7       0.0000\n",
            "ndcg_cut_100             7       0.0000\n",
            "ndcg_cut_200             7       0.0000\n",
            "ndcg_cut_500             7       0.0000\n",
            "ndcg_cut_1000            7       0.0000\n",
            "map_cut_5                7       0.0000\n",
            "map_cut_10               7       0.0000\n",
            "map_cut_15               7       0.0000\n",
            "map_cut_20               7       0.0000\n",
            "map_cut_30               7       0.0000\n",
            "map_cut_100              7       0.0000\n",
            "map_cut_200              7       0.0000\n",
            "map_cut_500              7       0.0000\n",
            "map_cut_1000             7       0.0000\n",
            "relative_P_5             7       0.0000\n",
            "relative_P_10            7       0.0000\n",
            "relative_P_15            7       0.0000\n",
            "relative_P_20            7       0.0000\n",
            "relative_P_30            7       0.0000\n",
            "relative_P_100           7       0.0000\n",
            "relative_P_200           7       0.0000\n",
            "relative_P_500           7       0.0000\n",
            "relative_P_1000          7       0.0000\n",
            "success_1                7       0.0000\n",
            "success_5                7       0.0000\n",
            "success_10               7       0.0000\n",
            "set_P                    7       0.0000\n",
            "set_relative_P           7       0.0000\n",
            "set_recall               7       0.0000\n",
            "set_map                  7       0.0000\n",
            "set_F                    7       0.0000\n",
            "num_nonrel_judged_ret    7       11.0000\n",
            "num_q                    9       1.0000\n",
            "num_ret                  9       27.0000\n",
            "num_rel                  9       1.0000\n",
            "num_rel_ret              9       1.0000\n",
            "map                      9       0.0556\n",
            "gm_map                   9       -2.8904\n",
            "Rprec                    9       0.0000\n",
            "bpref                    9       0.0000\n",
            "recip_rank               9       0.0556\n",
            "iprec_at_recall_0.00     9       0.0556\n",
            "iprec_at_recall_0.10     9       0.0556\n",
            "iprec_at_recall_0.20     9       0.0556\n",
            "iprec_at_recall_0.30     9       0.0556\n",
            "iprec_at_recall_0.40     9       0.0556\n",
            "iprec_at_recall_0.50     9       0.0556\n",
            "iprec_at_recall_0.60     9       0.0556\n",
            "iprec_at_recall_0.70     9       0.0556\n",
            "iprec_at_recall_0.80     9       0.0556\n",
            "iprec_at_recall_0.90     9       0.0556\n",
            "iprec_at_recall_1.00     9       0.0556\n",
            "P_5                      9       0.0000\n",
            "P_10                     9       0.0000\n",
            "P_15                     9       0.0000\n",
            "P_20                     9       0.0500\n",
            "P_30                     9       0.0333\n",
            "P_100                    9       0.0100\n",
            "P_200                    9       0.0050\n",
            "P_500                    9       0.0020\n",
            "P_1000                   9       0.0010\n",
            "relstring                9       0.0000\n",
            "recall_5                 9       0.0000\n",
            "recall_10                9       0.0000\n",
            "recall_15                9       0.0000\n",
            "recall_20                9       1.0000\n",
            "recall_30                9       1.0000\n",
            "recall_100               9       1.0000\n",
            "recall_200               9       1.0000\n",
            "recall_500               9       1.0000\n",
            "recall_1000              9       1.0000\n",
            "infAP                    9       0.0556\n",
            "gm_bpref                 9       -11.5129\n",
            "Rprec_mult_0.20          9       0.0000\n",
            "Rprec_mult_0.40          9       0.0000\n",
            "Rprec_mult_0.60          9       0.0000\n",
            "Rprec_mult_0.80          9       0.0000\n",
            "Rprec_mult_1.00          9       0.0000\n",
            "Rprec_mult_1.20          9       0.0000\n",
            "Rprec_mult_1.40          9       0.0000\n",
            "Rprec_mult_1.60          9       0.0000\n",
            "Rprec_mult_1.80          9       0.0000\n",
            "Rprec_mult_2.00          9       0.0000\n",
            "utility                  9       -25.0000\n",
            "11pt_avg                 9       0.0556\n",
            "binG                     9       0.2354\n",
            "G                        9       0.2354\n",
            "ndcg                     9       0.2354\n",
            "ndcg_rel                 9       0.2354\n",
            "Rndcg                    9       0.1177\n",
            "ndcg_cut_5               9       0.0000\n",
            "ndcg_cut_10              9       0.0000\n",
            "ndcg_cut_15              9       0.0000\n",
            "ndcg_cut_20              9       0.2354\n",
            "ndcg_cut_30              9       0.2354\n",
            "ndcg_cut_100             9       0.2354\n",
            "ndcg_cut_200             9       0.2354\n",
            "ndcg_cut_500             9       0.2354\n",
            "ndcg_cut_1000            9       0.2354\n",
            "map_cut_5                9       0.0000\n",
            "map_cut_10               9       0.0000\n",
            "map_cut_15               9       0.0000\n",
            "map_cut_20               9       0.0556\n",
            "map_cut_30               9       0.0556\n",
            "map_cut_100              9       0.0556\n",
            "map_cut_200              9       0.0556\n",
            "map_cut_500              9       0.0556\n",
            "map_cut_1000             9       0.0556\n",
            "relative_P_5             9       0.0000\n",
            "relative_P_10            9       0.0000\n",
            "relative_P_15            9       0.0000\n",
            "relative_P_20            9       1.0000\n",
            "relative_P_30            9       1.0000\n",
            "relative_P_100           9       1.0000\n",
            "relative_P_200           9       1.0000\n",
            "relative_P_500           9       1.0000\n",
            "relative_P_1000          9       1.0000\n",
            "success_1                9       0.0000\n",
            "success_5                9       0.0000\n",
            "success_10               9       0.0000\n",
            "set_P                    9       0.0370\n",
            "set_relative_P           9       1.0000\n",
            "set_recall               9       1.0000\n",
            "set_map                  9       0.0370\n",
            "set_F                    9       0.0714\n",
            "num_nonrel_judged_ret    9       23.0000\n",
            "num_q                    10      1.0000\n",
            "num_ret                  10      52.0000\n",
            "num_rel                  10      1.0000\n",
            "num_rel_ret              10      1.0000\n",
            "map                      10      0.2500\n",
            "gm_map                   10      -1.3863\n",
            "Rprec                    10      0.0000\n",
            "bpref                    10      0.0000\n",
            "recip_rank               10      0.2500\n",
            "iprec_at_recall_0.00     10      0.2500\n",
            "iprec_at_recall_0.10     10      0.2500\n",
            "iprec_at_recall_0.20     10      0.2500\n",
            "iprec_at_recall_0.30     10      0.2500\n",
            "iprec_at_recall_0.40     10      0.2500\n",
            "iprec_at_recall_0.50     10      0.2500\n",
            "iprec_at_recall_0.60     10      0.2500\n",
            "iprec_at_recall_0.70     10      0.2500\n",
            "iprec_at_recall_0.80     10      0.2500\n",
            "iprec_at_recall_0.90     10      0.2500\n",
            "iprec_at_recall_1.00     10      0.2500\n",
            "P_5                      10      0.2000\n",
            "P_10                     10      0.1000\n",
            "P_15                     10      0.0667\n",
            "P_20                     10      0.0500\n",
            "P_30                     10      0.0333\n",
            "P_100                    10      0.0100\n",
            "P_200                    10      0.0050\n",
            "P_500                    10      0.0020\n",
            "P_1000                   10      0.0010\n",
            "relstring                10      0.0000\n",
            "recall_5                 10      1.0000\n",
            "recall_10                10      1.0000\n",
            "recall_15                10      1.0000\n",
            "recall_20                10      1.0000\n",
            "recall_30                10      1.0000\n",
            "recall_100               10      1.0000\n",
            "recall_200               10      1.0000\n",
            "recall_500               10      1.0000\n",
            "recall_1000              10      1.0000\n",
            "infAP                    10      0.2500\n",
            "gm_bpref                 10      -11.5129\n",
            "Rprec_mult_0.20          10      0.0000\n",
            "Rprec_mult_0.40          10      0.0000\n",
            "Rprec_mult_0.60          10      0.0000\n",
            "Rprec_mult_0.80          10      0.0000\n",
            "Rprec_mult_1.00          10      0.0000\n",
            "Rprec_mult_1.20          10      0.0000\n",
            "Rprec_mult_1.40          10      0.0000\n",
            "Rprec_mult_1.60          10      0.0000\n",
            "Rprec_mult_1.80          10      0.0000\n",
            "Rprec_mult_2.00          10      0.0000\n",
            "utility                  10      -50.0000\n",
            "11pt_avg                 10      0.2500\n",
            "binG                     10      0.4307\n",
            "G                        10      0.4307\n",
            "ndcg                     10      0.4307\n",
            "ndcg_rel                 10      0.4307\n",
            "Rndcg                    10      0.2153\n",
            "ndcg_cut_5               10      0.4307\n",
            "ndcg_cut_10              10      0.4307\n",
            "ndcg_cut_15              10      0.4307\n",
            "ndcg_cut_20              10      0.4307\n",
            "ndcg_cut_30              10      0.4307\n",
            "ndcg_cut_100             10      0.4307\n",
            "ndcg_cut_200             10      0.4307\n",
            "ndcg_cut_500             10      0.4307\n",
            "ndcg_cut_1000            10      0.4307\n",
            "map_cut_5                10      0.2500\n",
            "map_cut_10               10      0.2500\n",
            "map_cut_15               10      0.2500\n",
            "map_cut_20               10      0.2500\n",
            "map_cut_30               10      0.2500\n",
            "map_cut_100              10      0.2500\n",
            "map_cut_200              10      0.2500\n",
            "map_cut_500              10      0.2500\n",
            "map_cut_1000             10      0.2500\n",
            "relative_P_5             10      1.0000\n",
            "relative_P_10            10      1.0000\n",
            "relative_P_15            10      1.0000\n",
            "relative_P_20            10      1.0000\n",
            "relative_P_30            10      1.0000\n",
            "relative_P_100           10      1.0000\n",
            "relative_P_200           10      1.0000\n",
            "relative_P_500           10      1.0000\n",
            "relative_P_1000          10      1.0000\n",
            "success_1                10      0.0000\n",
            "success_5                10      1.0000\n",
            "success_10               10      1.0000\n",
            "set_P                    10      0.0192\n",
            "set_relative_P           10      1.0000\n",
            "set_recall               10      1.0000\n",
            "set_map                  10      0.0192\n",
            "set_F                    10      0.0377\n",
            "num_nonrel_judged_ret    10      16.0000\n",
            "num_q                    14      1.0000\n",
            "num_ret                  14      26.0000\n",
            "num_rel                  14      1.0000\n",
            "num_rel_ret              14      1.0000\n",
            "map                      14      1.0000\n",
            "gm_map                   14      0.0000\n",
            "Rprec                    14      1.0000\n",
            "bpref                    14      1.0000\n",
            "recip_rank               14      1.0000\n",
            "iprec_at_recall_0.00     14      1.0000\n",
            "iprec_at_recall_0.10     14      1.0000\n",
            "iprec_at_recall_0.20     14      1.0000\n",
            "iprec_at_recall_0.30     14      1.0000\n",
            "iprec_at_recall_0.40     14      1.0000\n",
            "iprec_at_recall_0.50     14      1.0000\n",
            "iprec_at_recall_0.60     14      1.0000\n",
            "iprec_at_recall_0.70     14      1.0000\n",
            "iprec_at_recall_0.80     14      1.0000\n",
            "iprec_at_recall_0.90     14      1.0000\n",
            "iprec_at_recall_1.00     14      1.0000\n",
            "P_5                      14      0.2000\n",
            "P_10                     14      0.1000\n",
            "P_15                     14      0.0667\n",
            "P_20                     14      0.0500\n",
            "P_30                     14      0.0333\n",
            "P_100                    14      0.0100\n",
            "P_200                    14      0.0050\n",
            "P_500                    14      0.0020\n",
            "P_1000                   14      0.0010\n",
            "relstring                14      0.0000\n",
            "recall_5                 14      1.0000\n",
            "recall_10                14      1.0000\n",
            "recall_15                14      1.0000\n",
            "recall_20                14      1.0000\n",
            "recall_30                14      1.0000\n",
            "recall_100               14      1.0000\n",
            "recall_200               14      1.0000\n",
            "recall_500               14      1.0000\n",
            "recall_1000              14      1.0000\n",
            "infAP                    14      1.0000\n",
            "gm_bpref                 14      0.0000\n",
            "Rprec_mult_0.20          14      1.0000\n",
            "Rprec_mult_0.40          14      1.0000\n",
            "Rprec_mult_0.60          14      1.0000\n",
            "Rprec_mult_0.80          14      1.0000\n",
            "Rprec_mult_1.00          14      1.0000\n",
            "Rprec_mult_1.20          14      0.5000\n",
            "Rprec_mult_1.40          14      0.5000\n",
            "Rprec_mult_1.60          14      0.5000\n",
            "Rprec_mult_1.80          14      0.5000\n",
            "Rprec_mult_2.00          14      0.5000\n",
            "utility                  14      -24.0000\n",
            "11pt_avg                 14      1.0000\n",
            "binG                     14      1.0000\n",
            "G                        14      1.0000\n",
            "ndcg                     14      1.0000\n",
            "ndcg_rel                 14      1.0000\n",
            "Rndcg                    14      1.0000\n",
            "ndcg_cut_5               14      1.0000\n",
            "ndcg_cut_10              14      1.0000\n",
            "ndcg_cut_15              14      1.0000\n",
            "ndcg_cut_20              14      1.0000\n",
            "ndcg_cut_30              14      1.0000\n",
            "ndcg_cut_100             14      1.0000\n",
            "ndcg_cut_200             14      1.0000\n",
            "ndcg_cut_500             14      1.0000\n",
            "ndcg_cut_1000            14      1.0000\n",
            "map_cut_5                14      1.0000\n",
            "map_cut_10               14      1.0000\n",
            "map_cut_15               14      1.0000\n",
            "map_cut_20               14      1.0000\n",
            "map_cut_30               14      1.0000\n",
            "map_cut_100              14      1.0000\n",
            "map_cut_200              14      1.0000\n",
            "map_cut_500              14      1.0000\n",
            "map_cut_1000             14      1.0000\n",
            "relative_P_5             14      1.0000\n",
            "relative_P_10            14      1.0000\n",
            "relative_P_15            14      1.0000\n",
            "relative_P_20            14      1.0000\n",
            "relative_P_30            14      1.0000\n",
            "relative_P_100           14      1.0000\n",
            "relative_P_200           14      1.0000\n",
            "relative_P_500           14      1.0000\n",
            "relative_P_1000          14      1.0000\n",
            "success_1                14      1.0000\n",
            "success_5                14      1.0000\n",
            "success_10               14      1.0000\n",
            "set_P                    14      0.0385\n",
            "set_relative_P           14      1.0000\n",
            "set_recall               14      1.0000\n",
            "set_map                  14      0.0385\n",
            "set_F                    14      0.0741\n",
            "num_nonrel_judged_ret    14      20.0000\n",
            "num_q                    16      1.0000\n",
            "num_ret                  16      33.0000\n",
            "num_rel                  16      7.0000\n",
            "num_rel_ret              16      0.0000\n",
            "map                      16      0.0000\n",
            "gm_map                   16      -11.5129\n",
            "Rprec                    16      0.0000\n",
            "bpref                    16      0.0000\n",
            "recip_rank               16      0.0000\n",
            "iprec_at_recall_0.00     16      0.0000\n",
            "iprec_at_recall_0.10     16      0.0000\n",
            "iprec_at_recall_0.20     16      0.0000\n",
            "iprec_at_recall_0.30     16      0.0000\n",
            "iprec_at_recall_0.40     16      0.0000\n",
            "iprec_at_recall_0.50     16      0.0000\n",
            "iprec_at_recall_0.60     16      0.0000\n",
            "iprec_at_recall_0.70     16      0.0000\n",
            "iprec_at_recall_0.80     16      0.0000\n",
            "iprec_at_recall_0.90     16      0.0000\n",
            "iprec_at_recall_1.00     16      0.0000\n",
            "P_5                      16      0.0000\n",
            "P_10                     16      0.0000\n",
            "P_15                     16      0.0000\n",
            "P_20                     16      0.0000\n",
            "P_30                     16      0.0000\n",
            "P_100                    16      0.0000\n",
            "P_200                    16      0.0000\n",
            "P_500                    16      0.0000\n",
            "P_1000                   16      0.0000\n",
            "relstring                16      0.0000\n",
            "recall_5                 16      0.0000\n",
            "recall_10                16      0.0000\n",
            "recall_15                16      0.0000\n",
            "recall_20                16      0.0000\n",
            "recall_30                16      0.0000\n",
            "recall_100               16      0.0000\n",
            "recall_200               16      0.0000\n",
            "recall_500               16      0.0000\n",
            "recall_1000              16      0.0000\n",
            "infAP                    16      0.0000\n",
            "gm_bpref                 16      -11.5129\n",
            "Rprec_mult_0.20          16      0.0000\n",
            "Rprec_mult_0.40          16      0.0000\n",
            "Rprec_mult_0.60          16      0.0000\n",
            "Rprec_mult_0.80          16      0.0000\n",
            "Rprec_mult_1.00          16      0.0000\n",
            "Rprec_mult_1.20          16      0.0000\n",
            "Rprec_mult_1.40          16      0.0000\n",
            "Rprec_mult_1.60          16      0.0000\n",
            "Rprec_mult_1.80          16      0.0000\n",
            "Rprec_mult_2.00          16      0.0000\n",
            "utility                  16      -33.0000\n",
            "11pt_avg                 16      0.0000\n",
            "binG                     16      0.0000\n",
            "G                        16      0.0000\n",
            "ndcg                     16      0.0000\n",
            "ndcg_rel                 16      0.0000\n",
            "Rndcg                    16      0.0000\n",
            "ndcg_cut_5               16      0.0000\n",
            "ndcg_cut_10              16      0.0000\n",
            "ndcg_cut_15              16      0.0000\n",
            "ndcg_cut_20              16      0.0000\n",
            "ndcg_cut_30              16      0.0000\n",
            "ndcg_cut_100             16      0.0000\n",
            "ndcg_cut_200             16      0.0000\n",
            "ndcg_cut_500             16      0.0000\n",
            "ndcg_cut_1000            16      0.0000\n",
            "map_cut_5                16      0.0000\n",
            "map_cut_10               16      0.0000\n",
            "map_cut_15               16      0.0000\n",
            "map_cut_20               16      0.0000\n",
            "map_cut_30               16      0.0000\n",
            "map_cut_100              16      0.0000\n",
            "map_cut_200              16      0.0000\n",
            "map_cut_500              16      0.0000\n",
            "map_cut_1000             16      0.0000\n",
            "relative_P_5             16      0.0000\n",
            "relative_P_10            16      0.0000\n",
            "relative_P_15            16      0.0000\n",
            "relative_P_20            16      0.0000\n",
            "relative_P_30            16      0.0000\n",
            "relative_P_100           16      0.0000\n",
            "relative_P_200           16      0.0000\n",
            "relative_P_500           16      0.0000\n",
            "relative_P_1000          16      0.0000\n",
            "success_1                16      0.0000\n",
            "success_5                16      0.0000\n",
            "success_10               16      0.0000\n",
            "set_P                    16      0.0000\n",
            "set_relative_P           16      0.0000\n",
            "set_recall               16      0.0000\n",
            "set_map                  16      0.0000\n",
            "set_F                    16      0.0000\n",
            "num_nonrel_judged_ret    16      20.0000\n",
            "num_q                    18      1.0000\n",
            "num_ret                  18      32.0000\n",
            "num_rel                  18      1.0000\n",
            "num_rel_ret              18      1.0000\n",
            "map                      18      1.0000\n",
            "gm_map                   18      0.0000\n",
            "Rprec                    18      1.0000\n",
            "bpref                    18      1.0000\n",
            "recip_rank               18      1.0000\n",
            "iprec_at_recall_0.00     18      1.0000\n",
            "iprec_at_recall_0.10     18      1.0000\n",
            "iprec_at_recall_0.20     18      1.0000\n",
            "iprec_at_recall_0.30     18      1.0000\n",
            "iprec_at_recall_0.40     18      1.0000\n",
            "iprec_at_recall_0.50     18      1.0000\n",
            "iprec_at_recall_0.60     18      1.0000\n",
            "iprec_at_recall_0.70     18      1.0000\n",
            "iprec_at_recall_0.80     18      1.0000\n",
            "iprec_at_recall_0.90     18      1.0000\n",
            "iprec_at_recall_1.00     18      1.0000\n",
            "P_5                      18      0.2000\n",
            "P_10                     18      0.1000\n",
            "P_15                     18      0.0667\n",
            "P_20                     18      0.0500\n",
            "P_30                     18      0.0333\n",
            "P_100                    18      0.0100\n",
            "P_200                    18      0.0050\n",
            "P_500                    18      0.0020\n",
            "P_1000                   18      0.0010\n",
            "relstring                18      0.0000\n",
            "recall_5                 18      1.0000\n",
            "recall_10                18      1.0000\n",
            "recall_15                18      1.0000\n",
            "recall_20                18      1.0000\n",
            "recall_30                18      1.0000\n",
            "recall_100               18      1.0000\n",
            "recall_200               18      1.0000\n",
            "recall_500               18      1.0000\n",
            "recall_1000              18      1.0000\n",
            "infAP                    18      1.0000\n",
            "gm_bpref                 18      0.0000\n",
            "Rprec_mult_0.20          18      1.0000\n",
            "Rprec_mult_0.40          18      1.0000\n",
            "Rprec_mult_0.60          18      1.0000\n",
            "Rprec_mult_0.80          18      1.0000\n",
            "Rprec_mult_1.00          18      1.0000\n",
            "Rprec_mult_1.20          18      0.5000\n",
            "Rprec_mult_1.40          18      0.5000\n",
            "Rprec_mult_1.60          18      0.5000\n",
            "Rprec_mult_1.80          18      0.5000\n",
            "Rprec_mult_2.00          18      0.5000\n",
            "utility                  18      -30.0000\n",
            "11pt_avg                 18      1.0000\n",
            "binG                     18      1.0000\n",
            "G                        18      1.0000\n",
            "ndcg                     18      1.0000\n",
            "ndcg_rel                 18      1.0000\n",
            "Rndcg                    18      1.0000\n",
            "ndcg_cut_5               18      1.0000\n",
            "ndcg_cut_10              18      1.0000\n",
            "ndcg_cut_15              18      1.0000\n",
            "ndcg_cut_20              18      1.0000\n",
            "ndcg_cut_30              18      1.0000\n",
            "ndcg_cut_100             18      1.0000\n",
            "ndcg_cut_200             18      1.0000\n",
            "ndcg_cut_500             18      1.0000\n",
            "ndcg_cut_1000            18      1.0000\n",
            "map_cut_5                18      1.0000\n",
            "map_cut_10               18      1.0000\n",
            "map_cut_15               18      1.0000\n",
            "map_cut_20               18      1.0000\n",
            "map_cut_30               18      1.0000\n",
            "map_cut_100              18      1.0000\n",
            "map_cut_200              18      1.0000\n",
            "map_cut_500              18      1.0000\n",
            "map_cut_1000             18      1.0000\n",
            "relative_P_5             18      1.0000\n",
            "relative_P_10            18      1.0000\n",
            "relative_P_15            18      1.0000\n",
            "relative_P_20            18      1.0000\n",
            "relative_P_30            18      1.0000\n",
            "relative_P_100           18      1.0000\n",
            "relative_P_200           18      1.0000\n",
            "relative_P_500           18      1.0000\n",
            "relative_P_1000          18      1.0000\n",
            "success_1                18      1.0000\n",
            "success_5                18      1.0000\n",
            "success_10               18      1.0000\n",
            "set_P                    18      0.0312\n",
            "set_relative_P           18      1.0000\n",
            "set_recall               18      1.0000\n",
            "set_map                  18      0.0312\n",
            "set_F                    18      0.0606\n",
            "num_nonrel_judged_ret    18      30.0000\n",
            "num_q                    19      1.0000\n",
            "num_ret                  19      5.0000\n",
            "num_rel                  19      2.0000\n",
            "num_rel_ret              19      1.0000\n",
            "map                      19      0.5000\n",
            "gm_map                   19      -0.6931\n",
            "Rprec                    19      0.5000\n",
            "bpref                    19      0.5000\n",
            "recip_rank               19      1.0000\n",
            "iprec_at_recall_0.00     19      1.0000\n",
            "iprec_at_recall_0.10     19      1.0000\n",
            "iprec_at_recall_0.20     19      1.0000\n",
            "iprec_at_recall_0.30     19      1.0000\n",
            "iprec_at_recall_0.40     19      1.0000\n",
            "iprec_at_recall_0.50     19      1.0000\n",
            "iprec_at_recall_0.60     19      0.0000\n",
            "iprec_at_recall_0.70     19      0.0000\n",
            "iprec_at_recall_0.80     19      0.0000\n",
            "iprec_at_recall_0.90     19      0.0000\n",
            "iprec_at_recall_1.00     19      0.0000\n",
            "P_5                      19      0.2000\n",
            "P_10                     19      0.1000\n",
            "P_15                     19      0.0667\n",
            "P_20                     19      0.0500\n",
            "P_30                     19      0.0333\n",
            "P_100                    19      0.0100\n",
            "P_200                    19      0.0050\n",
            "P_500                    19      0.0020\n",
            "P_1000                   19      0.0010\n",
            "relstring                19      0.0000\n",
            "recall_5                 19      0.5000\n",
            "recall_10                19      0.5000\n",
            "recall_15                19      0.5000\n",
            "recall_20                19      0.5000\n",
            "recall_30                19      0.5000\n",
            "recall_100               19      0.5000\n",
            "recall_200               19      0.5000\n",
            "recall_500               19      0.5000\n",
            "recall_1000              19      0.5000\n",
            "infAP                    19      0.5000\n",
            "gm_bpref                 19      -0.6931\n",
            "Rprec_mult_0.20          19      1.0000\n",
            "Rprec_mult_0.40          19      1.0000\n",
            "Rprec_mult_0.60          19      0.5000\n",
            "Rprec_mult_0.80          19      0.5000\n",
            "Rprec_mult_1.00          19      0.5000\n",
            "Rprec_mult_1.20          19      0.3333\n",
            "Rprec_mult_1.40          19      0.3333\n",
            "Rprec_mult_1.60          19      0.2500\n",
            "Rprec_mult_1.80          19      0.2500\n",
            "Rprec_mult_2.00          19      0.2500\n",
            "utility                  19      -3.0000\n",
            "11pt_avg                 19      0.5455\n",
            "binG                     19      0.5000\n",
            "G                        19      0.5000\n",
            "ndcg                     19      0.6131\n",
            "ndcg_rel                 19      0.8066\n",
            "Rndcg                    19      0.6131\n",
            "ndcg_cut_5               19      0.6131\n",
            "ndcg_cut_10              19      0.6131\n",
            "ndcg_cut_15              19      0.6131\n",
            "ndcg_cut_20              19      0.6131\n",
            "ndcg_cut_30              19      0.6131\n",
            "ndcg_cut_100             19      0.6131\n",
            "ndcg_cut_200             19      0.6131\n",
            "ndcg_cut_500             19      0.6131\n",
            "ndcg_cut_1000            19      0.6131\n",
            "map_cut_5                19      0.5000\n",
            "map_cut_10               19      0.5000\n",
            "map_cut_15               19      0.5000\n",
            "map_cut_20               19      0.5000\n",
            "map_cut_30               19      0.5000\n",
            "map_cut_100              19      0.5000\n",
            "map_cut_200              19      0.5000\n",
            "map_cut_500              19      0.5000\n",
            "map_cut_1000             19      0.5000\n",
            "relative_P_5             19      0.5000\n",
            "relative_P_10            19      0.5000\n",
            "relative_P_15            19      0.5000\n",
            "relative_P_20            19      0.5000\n",
            "relative_P_30            19      0.5000\n",
            "relative_P_100           19      0.5000\n",
            "relative_P_200           19      0.5000\n",
            "relative_P_500           19      0.5000\n",
            "relative_P_1000          19      0.5000\n",
            "success_1                19      1.0000\n",
            "success_5                19      1.0000\n",
            "success_10               19      1.0000\n",
            "set_P                    19      0.2000\n",
            "set_relative_P           19      0.5000\n",
            "set_recall               19      0.5000\n",
            "set_map                  19      0.1000\n",
            "set_F                    19      0.2857\n",
            "num_nonrel_judged_ret    19      4.0000\n",
            "num_q                    22      1.0000\n",
            "num_ret                  22      107.0000\n",
            "num_rel                  22      1.0000\n",
            "num_rel_ret              22      1.0000\n",
            "map                      22      0.0357\n",
            "gm_map                   22      -3.3322\n",
            "Rprec                    22      0.0000\n",
            "bpref                    22      0.0000\n",
            "recip_rank               22      0.0357\n",
            "iprec_at_recall_0.00     22      0.0357\n",
            "iprec_at_recall_0.10     22      0.0357\n",
            "iprec_at_recall_0.20     22      0.0357\n",
            "iprec_at_recall_0.30     22      0.0357\n",
            "iprec_at_recall_0.40     22      0.0357\n",
            "iprec_at_recall_0.50     22      0.0357\n",
            "iprec_at_recall_0.60     22      0.0357\n",
            "iprec_at_recall_0.70     22      0.0357\n",
            "iprec_at_recall_0.80     22      0.0357\n",
            "iprec_at_recall_0.90     22      0.0357\n",
            "iprec_at_recall_1.00     22      0.0357\n",
            "P_5                      22      0.0000\n",
            "P_10                     22      0.0000\n",
            "P_15                     22      0.0000\n",
            "P_20                     22      0.0000\n",
            "P_30                     22      0.0333\n",
            "P_100                    22      0.0100\n",
            "P_200                    22      0.0050\n",
            "P_500                    22      0.0020\n",
            "P_1000                   22      0.0010\n",
            "relstring                22      0.0000\n",
            "recall_5                 22      0.0000\n",
            "recall_10                22      0.0000\n",
            "recall_15                22      0.0000\n",
            "recall_20                22      0.0000\n",
            "recall_30                22      1.0000\n",
            "recall_100               22      1.0000\n",
            "recall_200               22      1.0000\n",
            "recall_500               22      1.0000\n",
            "recall_1000              22      1.0000\n",
            "infAP                    22      0.0357\n",
            "gm_bpref                 22      -11.5129\n",
            "Rprec_mult_0.20          22      0.0000\n",
            "Rprec_mult_0.40          22      0.0000\n",
            "Rprec_mult_0.60          22      0.0000\n",
            "Rprec_mult_0.80          22      0.0000\n",
            "Rprec_mult_1.00          22      0.0000\n",
            "Rprec_mult_1.20          22      0.0000\n",
            "Rprec_mult_1.40          22      0.0000\n",
            "Rprec_mult_1.60          22      0.0000\n",
            "Rprec_mult_1.80          22      0.0000\n",
            "Rprec_mult_2.00          22      0.0000\n",
            "utility                  22      -105.0000\n",
            "11pt_avg                 22      0.0357\n",
            "binG                     22      0.2058\n",
            "G                        22      0.2058\n",
            "ndcg                     22      0.2058\n",
            "ndcg_rel                 22      0.2058\n",
            "Rndcg                    22      0.1029\n",
            "ndcg_cut_5               22      0.0000\n",
            "ndcg_cut_10              22      0.0000\n",
            "ndcg_cut_15              22      0.0000\n",
            "ndcg_cut_20              22      0.0000\n",
            "ndcg_cut_30              22      0.2058\n",
            "ndcg_cut_100             22      0.2058\n",
            "ndcg_cut_200             22      0.2058\n",
            "ndcg_cut_500             22      0.2058\n",
            "ndcg_cut_1000            22      0.2058\n",
            "map_cut_5                22      0.0000\n",
            "map_cut_10               22      0.0000\n",
            "map_cut_15               22      0.0000\n",
            "map_cut_20               22      0.0000\n",
            "map_cut_30               22      0.0357\n",
            "map_cut_100              22      0.0357\n",
            "map_cut_200              22      0.0357\n",
            "map_cut_500              22      0.0357\n",
            "map_cut_1000             22      0.0357\n",
            "relative_P_5             22      0.0000\n",
            "relative_P_10            22      0.0000\n",
            "relative_P_15            22      0.0000\n",
            "relative_P_20            22      0.0000\n",
            "relative_P_30            22      1.0000\n",
            "relative_P_100           22      1.0000\n",
            "relative_P_200           22      1.0000\n",
            "relative_P_500           22      1.0000\n",
            "relative_P_1000          22      1.0000\n",
            "success_1                22      0.0000\n",
            "success_5                22      0.0000\n",
            "success_10               22      0.0000\n",
            "set_P                    22      0.0093\n",
            "set_relative_P           22      1.0000\n",
            "set_recall               22      1.0000\n",
            "set_map                  22      0.0093\n",
            "set_F                    22      0.0185\n",
            "num_nonrel_judged_ret    22      72.0000\n",
            "num_q                    24      1.0000\n",
            "num_ret                  24      23.0000\n",
            "num_rel                  24      1.0000\n",
            "num_rel_ret              24      1.0000\n",
            "map                      24      1.0000\n",
            "gm_map                   24      0.0000\n",
            "Rprec                    24      1.0000\n",
            "bpref                    24      1.0000\n",
            "recip_rank               24      1.0000\n",
            "iprec_at_recall_0.00     24      1.0000\n",
            "iprec_at_recall_0.10     24      1.0000\n",
            "iprec_at_recall_0.20     24      1.0000\n",
            "iprec_at_recall_0.30     24      1.0000\n",
            "iprec_at_recall_0.40     24      1.0000\n",
            "iprec_at_recall_0.50     24      1.0000\n",
            "iprec_at_recall_0.60     24      1.0000\n",
            "iprec_at_recall_0.70     24      1.0000\n",
            "iprec_at_recall_0.80     24      1.0000\n",
            "iprec_at_recall_0.90     24      1.0000\n",
            "iprec_at_recall_1.00     24      1.0000\n",
            "P_5                      24      0.2000\n",
            "P_10                     24      0.1000\n",
            "P_15                     24      0.0667\n",
            "P_20                     24      0.0500\n",
            "P_30                     24      0.0333\n",
            "P_100                    24      0.0100\n",
            "P_200                    24      0.0050\n",
            "P_500                    24      0.0020\n",
            "P_1000                   24      0.0010\n",
            "relstring                24      0.0000\n",
            "recall_5                 24      1.0000\n",
            "recall_10                24      1.0000\n",
            "recall_15                24      1.0000\n",
            "recall_20                24      1.0000\n",
            "recall_30                24      1.0000\n",
            "recall_100               24      1.0000\n",
            "recall_200               24      1.0000\n",
            "recall_500               24      1.0000\n",
            "recall_1000              24      1.0000\n",
            "infAP                    24      1.0000\n",
            "gm_bpref                 24      0.0000\n",
            "Rprec_mult_0.20          24      1.0000\n",
            "Rprec_mult_0.40          24      1.0000\n",
            "Rprec_mult_0.60          24      1.0000\n",
            "Rprec_mult_0.80          24      1.0000\n",
            "Rprec_mult_1.00          24      1.0000\n",
            "Rprec_mult_1.20          24      0.5000\n",
            "Rprec_mult_1.40          24      0.5000\n",
            "Rprec_mult_1.60          24      0.5000\n",
            "Rprec_mult_1.80          24      0.5000\n",
            "Rprec_mult_2.00          24      0.5000\n",
            "utility                  24      -21.0000\n",
            "11pt_avg                 24      1.0000\n",
            "binG                     24      1.0000\n",
            "G                        24      1.0000\n",
            "ndcg                     24      1.0000\n",
            "ndcg_rel                 24      1.0000\n",
            "Rndcg                    24      1.0000\n",
            "ndcg_cut_5               24      1.0000\n",
            "ndcg_cut_10              24      1.0000\n",
            "ndcg_cut_15              24      1.0000\n",
            "ndcg_cut_20              24      1.0000\n",
            "ndcg_cut_30              24      1.0000\n",
            "ndcg_cut_100             24      1.0000\n",
            "ndcg_cut_200             24      1.0000\n",
            "ndcg_cut_500             24      1.0000\n",
            "ndcg_cut_1000            24      1.0000\n",
            "map_cut_5                24      1.0000\n",
            "map_cut_10               24      1.0000\n",
            "map_cut_15               24      1.0000\n",
            "map_cut_20               24      1.0000\n",
            "map_cut_30               24      1.0000\n",
            "map_cut_100              24      1.0000\n",
            "map_cut_200              24      1.0000\n",
            "map_cut_500              24      1.0000\n",
            "map_cut_1000             24      1.0000\n",
            "relative_P_5             24      1.0000\n",
            "relative_P_10            24      1.0000\n",
            "relative_P_15            24      1.0000\n",
            "relative_P_20            24      1.0000\n",
            "relative_P_30            24      1.0000\n",
            "relative_P_100           24      1.0000\n",
            "relative_P_200           24      1.0000\n",
            "relative_P_500           24      1.0000\n",
            "relative_P_1000          24      1.0000\n",
            "success_1                24      1.0000\n",
            "success_5                24      1.0000\n",
            "success_10               24      1.0000\n",
            "set_P                    24      0.0435\n",
            "set_relative_P           24      1.0000\n",
            "set_recall               24      1.0000\n",
            "set_map                  24      0.0435\n",
            "set_F                    24      0.0833\n",
            "num_nonrel_judged_ret    24      18.0000\n",
            "num_q                    26      1.0000\n",
            "num_ret                  26      52.0000\n",
            "num_rel                  26      3.0000\n",
            "num_rel_ret              26      2.0000\n",
            "map                      26      0.0771\n",
            "gm_map                   26      -2.5632\n",
            "Rprec                    26      0.0000\n",
            "bpref                    26      0.0000\n",
            "recip_rank               26      0.1667\n",
            "iprec_at_recall_0.00     26      0.1667\n",
            "iprec_at_recall_0.10     26      0.1667\n",
            "iprec_at_recall_0.20     26      0.1667\n",
            "iprec_at_recall_0.30     26      0.1667\n",
            "iprec_at_recall_0.40     26      0.0645\n",
            "iprec_at_recall_0.50     26      0.0645\n",
            "iprec_at_recall_0.60     26      0.0645\n",
            "iprec_at_recall_0.70     26      0.0645\n",
            "iprec_at_recall_0.80     26      0.0000\n",
            "iprec_at_recall_0.90     26      0.0000\n",
            "iprec_at_recall_1.00     26      0.0000\n",
            "P_5                      26      0.0000\n",
            "P_10                     26      0.1000\n",
            "P_15                     26      0.0667\n",
            "P_20                     26      0.0500\n",
            "P_30                     26      0.0333\n",
            "P_100                    26      0.0200\n",
            "P_200                    26      0.0100\n",
            "P_500                    26      0.0040\n",
            "P_1000                   26      0.0020\n",
            "relstring                26      0.0000\n",
            "recall_5                 26      0.0000\n",
            "recall_10                26      0.3333\n",
            "recall_15                26      0.3333\n",
            "recall_20                26      0.3333\n",
            "recall_30                26      0.3333\n",
            "recall_100               26      0.6667\n",
            "recall_200               26      0.6667\n",
            "recall_500               26      0.6667\n",
            "recall_1000              26      0.6667\n",
            "infAP                    26      0.0771\n",
            "gm_bpref                 26      -11.5129\n",
            "Rprec_mult_0.20          26      0.0000\n",
            "Rprec_mult_0.40          26      0.0000\n",
            "Rprec_mult_0.60          26      0.0000\n",
            "Rprec_mult_0.80          26      0.0000\n",
            "Rprec_mult_1.00          26      0.0000\n",
            "Rprec_mult_1.20          26      0.0000\n",
            "Rprec_mult_1.40          26      0.0000\n",
            "Rprec_mult_1.60          26      0.0000\n",
            "Rprec_mult_1.80          26      0.1667\n",
            "Rprec_mult_2.00          26      0.1667\n",
            "utility                  26      -48.0000\n",
            "11pt_avg                 26      0.0841\n",
            "binG                     26      0.1860\n",
            "G                        26      0.1860\n",
            "ndcg                     26      0.2610\n",
            "ndcg_rel                 26      0.2297\n",
            "Rndcg                    26      0.1305\n",
            "ndcg_cut_5               26      0.0000\n",
            "ndcg_cut_10              26      0.1672\n",
            "ndcg_cut_15              26      0.1672\n",
            "ndcg_cut_20              26      0.1672\n",
            "ndcg_cut_30              26      0.1672\n",
            "ndcg_cut_100             26      0.2610\n",
            "ndcg_cut_200             26      0.2610\n",
            "ndcg_cut_500             26      0.2610\n",
            "ndcg_cut_1000            26      0.2610\n",
            "map_cut_5                26      0.0000\n",
            "map_cut_10               26      0.0556\n",
            "map_cut_15               26      0.0556\n",
            "map_cut_20               26      0.0556\n",
            "map_cut_30               26      0.0556\n",
            "map_cut_100              26      0.0771\n",
            "map_cut_200              26      0.0771\n",
            "map_cut_500              26      0.0771\n",
            "map_cut_1000             26      0.0771\n",
            "relative_P_5             26      0.0000\n",
            "relative_P_10            26      0.3333\n",
            "relative_P_15            26      0.3333\n",
            "relative_P_20            26      0.3333\n",
            "relative_P_30            26      0.3333\n",
            "relative_P_100           26      0.6667\n",
            "relative_P_200           26      0.6667\n",
            "relative_P_500           26      0.6667\n",
            "relative_P_1000          26      0.6667\n",
            "success_1                26      0.0000\n",
            "success_5                26      0.0000\n",
            "success_10               26      1.0000\n",
            "set_P                    26      0.0385\n",
            "set_relative_P           26      0.6667\n",
            "set_recall               26      0.6667\n",
            "set_map                  26      0.0256\n",
            "set_F                    26      0.0727\n",
            "num_nonrel_judged_ret    26      36.0000\n",
            "num_q                    28      1.0000\n",
            "num_ret                  28      52.0000\n",
            "num_rel                  28      2.0000\n",
            "num_rel_ret              28      2.0000\n",
            "map                      28      0.2262\n",
            "gm_map                   28      -1.4864\n",
            "Rprec                    28      0.0000\n",
            "bpref                    28      0.0000\n",
            "recip_rank               28      0.1667\n",
            "iprec_at_recall_0.00     28      0.2857\n",
            "iprec_at_recall_0.10     28      0.2857\n",
            "iprec_at_recall_0.20     28      0.2857\n",
            "iprec_at_recall_0.30     28      0.2857\n",
            "iprec_at_recall_0.40     28      0.2857\n",
            "iprec_at_recall_0.50     28      0.2857\n",
            "iprec_at_recall_0.60     28      0.2857\n",
            "iprec_at_recall_0.70     28      0.2857\n",
            "iprec_at_recall_0.80     28      0.2857\n",
            "iprec_at_recall_0.90     28      0.2857\n",
            "iprec_at_recall_1.00     28      0.2857\n",
            "P_5                      28      0.0000\n",
            "P_10                     28      0.2000\n",
            "P_15                     28      0.1333\n",
            "P_20                     28      0.1000\n",
            "P_30                     28      0.0667\n",
            "P_100                    28      0.0200\n",
            "P_200                    28      0.0100\n",
            "P_500                    28      0.0040\n",
            "P_1000                   28      0.0020\n",
            "relstring                28      0.0000\n",
            "recall_5                 28      0.0000\n",
            "recall_10                28      1.0000\n",
            "recall_15                28      1.0000\n",
            "recall_20                28      1.0000\n",
            "recall_30                28      1.0000\n",
            "recall_100               28      1.0000\n",
            "recall_200               28      1.0000\n",
            "recall_500               28      1.0000\n",
            "recall_1000              28      1.0000\n",
            "infAP                    28      0.2262\n",
            "gm_bpref                 28      -11.5129\n",
            "Rprec_mult_0.20          28      0.0000\n",
            "Rprec_mult_0.40          28      0.0000\n",
            "Rprec_mult_0.60          28      0.0000\n",
            "Rprec_mult_0.80          28      0.0000\n",
            "Rprec_mult_1.00          28      0.0000\n",
            "Rprec_mult_1.20          28      0.0000\n",
            "Rprec_mult_1.40          28      0.0000\n",
            "Rprec_mult_1.60          28      0.0000\n",
            "Rprec_mult_1.80          28      0.0000\n",
            "Rprec_mult_2.00          28      0.0000\n",
            "utility                  28      -48.0000\n",
            "11pt_avg                 28      0.2857\n",
            "binG                     28      0.3562\n",
            "G                        28      0.3562\n",
            "ndcg                     28      0.4228\n",
            "ndcg_rel                 28      0.3206\n",
            "Rndcg                    28      0.2114\n",
            "ndcg_cut_5               28      0.0000\n",
            "ndcg_cut_10              28      0.4228\n",
            "ndcg_cut_15              28      0.4228\n",
            "ndcg_cut_20              28      0.4228\n",
            "ndcg_cut_30              28      0.4228\n",
            "ndcg_cut_100             28      0.4228\n",
            "ndcg_cut_200             28      0.4228\n",
            "ndcg_cut_500             28      0.4228\n",
            "ndcg_cut_1000            28      0.4228\n",
            "map_cut_5                28      0.0000\n",
            "map_cut_10               28      0.2262\n",
            "map_cut_15               28      0.2262\n",
            "map_cut_20               28      0.2262\n",
            "map_cut_30               28      0.2262\n",
            "map_cut_100              28      0.2262\n",
            "map_cut_200              28      0.2262\n",
            "map_cut_500              28      0.2262\n",
            "map_cut_1000             28      0.2262\n",
            "relative_P_5             28      0.0000\n",
            "relative_P_10            28      1.0000\n",
            "relative_P_15            28      1.0000\n",
            "relative_P_20            28      1.0000\n",
            "relative_P_30            28      1.0000\n",
            "relative_P_100           28      1.0000\n",
            "relative_P_200           28      1.0000\n",
            "relative_P_500           28      1.0000\n",
            "relative_P_1000          28      1.0000\n",
            "success_1                28      0.0000\n",
            "success_5                28      0.0000\n",
            "success_10               28      1.0000\n",
            "set_P                    28      0.0385\n",
            "set_relative_P           28      1.0000\n",
            "set_recall               28      1.0000\n",
            "set_map                  28      0.0385\n",
            "set_F                    28      0.0741\n",
            "num_nonrel_judged_ret    28      28.0000\n",
            "num_q                    all     15.0000\n",
            "num_ret                  all     519.0000\n",
            "num_rel                  all     35.0000\n",
            "num_rel_ret              all     16.0000\n",
            "map                      all     0.3566\n",
            "gm_map                   all     0.0357\n",
            "Rprec                    all     0.3000\n",
            "bpref                    all     0.3000\n",
            "recip_rank               all     0.4561\n",
            "iprec_at_recall_0.00     all     0.4640\n",
            "iprec_at_recall_0.10     all     0.4640\n",
            "iprec_at_recall_0.20     all     0.4640\n",
            "iprec_at_recall_0.30     all     0.4640\n",
            "iprec_at_recall_0.40     all     0.4572\n",
            "iprec_at_recall_0.50     all     0.4572\n",
            "iprec_at_recall_0.60     all     0.2672\n",
            "iprec_at_recall_0.70     all     0.2672\n",
            "iprec_at_recall_0.80     all     0.2529\n",
            "iprec_at_recall_0.90     all     0.2529\n",
            "iprec_at_recall_1.00     all     0.2529\n",
            "P_5                      all     0.1067\n",
            "P_10                     all     0.0800\n",
            "P_15                     all     0.0533\n",
            "P_20                     all     0.0467\n",
            "P_30                     all     0.0333\n",
            "P_100                    all     0.0107\n",
            "P_200                    all     0.0053\n",
            "P_500                    all     0.0021\n",
            "P_1000                   all     0.0011\n",
            "relstring                all     0.0000\n",
            "recall_5                 all     0.3667\n",
            "recall_10                all     0.5222\n",
            "recall_15                all     0.5222\n",
            "recall_20                all     0.6056\n",
            "recall_30                all     0.6722\n",
            "recall_100               all     0.6944\n",
            "recall_200               all     0.6944\n",
            "recall_500               all     0.6944\n",
            "recall_1000              all     0.6944\n",
            "infAP                    all     0.3566\n",
            "gm_bpref                 all     0.0009\n",
            "Rprec_mult_0.20          all     0.4000\n",
            "Rprec_mult_0.40          all     0.4000\n",
            "Rprec_mult_0.60          all     0.3111\n",
            "Rprec_mult_0.80          all     0.3000\n",
            "Rprec_mult_1.00          all     0.3000\n",
            "Rprec_mult_1.20          all     0.1711\n",
            "Rprec_mult_1.40          all     0.1667\n",
            "Rprec_mult_1.60          all     0.1524\n",
            "Rprec_mult_1.80          all     0.1611\n",
            "Rprec_mult_2.00          all     0.1611\n",
            "utility                  all     -32.4667\n",
            "11pt_avg                 all     0.3694\n",
            "binG                     all     0.4219\n",
            "G                        all     0.4219\n",
            "ndcg                     all     0.4576\n",
            "ndcg_rel                 all     0.4836\n",
            "Rndcg                    all     0.3909\n",
            "ndcg_cut_5               all     0.3529\n",
            "ndcg_cut_10              all     0.4160\n",
            "ndcg_cut_15              all     0.4160\n",
            "ndcg_cut_20              all     0.4376\n",
            "ndcg_cut_30              all     0.4513\n",
            "ndcg_cut_100             all     0.4576\n",
            "ndcg_cut_200             all     0.4576\n",
            "ndcg_cut_500             all     0.4576\n",
            "ndcg_cut_1000            all     0.4576\n",
            "map_cut_5                all     0.3167\n",
            "map_cut_10               all     0.3466\n",
            "map_cut_15               all     0.3466\n",
            "map_cut_20               all     0.3528\n",
            "map_cut_30               all     0.3551\n",
            "map_cut_100              all     0.3566\n",
            "map_cut_200              all     0.3566\n",
            "map_cut_500              all     0.3566\n",
            "map_cut_1000             all     0.3566\n",
            "relative_P_5             all     0.3667\n",
            "relative_P_10            all     0.5222\n",
            "relative_P_15            all     0.5222\n",
            "relative_P_20            all     0.6056\n",
            "relative_P_30            all     0.6722\n",
            "relative_P_100           all     0.6944\n",
            "relative_P_200           all     0.6944\n",
            "relative_P_500           all     0.6944\n",
            "relative_P_1000          all     0.6944\n",
            "success_1                all     0.4000\n",
            "success_5                all     0.4667\n",
            "success_10               all     0.6667\n",
            "set_P                    all     0.0422\n",
            "set_relative_P           all     0.6944\n",
            "set_recall               all     0.6944\n",
            "set_map                  all     0.0308\n",
            "set_F                    all     0.0731\n",
            "num_nonrel_judged_ret    all     332.0000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U5t4u36W1bUS",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "68786329-d5ad-41d2-9956-f96e246e3488"
      },
      "source": [
        "#Change scoring parameter\n",
        "from whoosh.scoring import BM25F\n",
        "Best_Analyzer=RegexTokenizer() | LowercaseFilter() | StopFilter()| StemFilter()|CustomFilter(LancasterStemmer().stem)\n",
        "Best_Schema=Schema(file_path = ID(stored=True), file_content = TEXT(analyzer = Best_Analyzer))\n",
        "BEST_INDEX = createIndex(Best_Schema)\n",
        "addFilesToIndex(BEST_INDEX, filesToIndex)\n",
        "BEST_QP = QueryParser(\"file_content\", schema=BEST_INDEX.schema)\n",
        "BEST_SEARCHER = BEST_INDEX.searcher(weighting=BM25F(B=0.5,K1=1.2))\n",
        "pyTrecEval(topic_file, qrels_file, BEST_QP, BEST_SEARCHER) \n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "num_q                    1       1.0000\n",
            "num_ret                  1       3.0000\n",
            "num_rel                  1       5.0000\n",
            "num_rel_ret              1       0.0000\n",
            "map                      1       0.0000\n",
            "gm_map                   1       -11.5129\n",
            "Rprec                    1       0.0000\n",
            "bpref                    1       0.0000\n",
            "recip_rank               1       0.0000\n",
            "iprec_at_recall_0.00     1       0.0000\n",
            "iprec_at_recall_0.10     1       0.0000\n",
            "iprec_at_recall_0.20     1       0.0000\n",
            "iprec_at_recall_0.30     1       0.0000\n",
            "iprec_at_recall_0.40     1       0.0000\n",
            "iprec_at_recall_0.50     1       0.0000\n",
            "iprec_at_recall_0.60     1       0.0000\n",
            "iprec_at_recall_0.70     1       0.0000\n",
            "iprec_at_recall_0.80     1       0.0000\n",
            "iprec_at_recall_0.90     1       0.0000\n",
            "iprec_at_recall_1.00     1       0.0000\n",
            "P_5                      1       0.0000\n",
            "P_10                     1       0.0000\n",
            "P_15                     1       0.0000\n",
            "P_20                     1       0.0000\n",
            "P_30                     1       0.0000\n",
            "P_100                    1       0.0000\n",
            "P_200                    1       0.0000\n",
            "P_500                    1       0.0000\n",
            "P_1000                   1       0.0000\n",
            "relstring                1       0.0000\n",
            "recall_5                 1       0.0000\n",
            "recall_10                1       0.0000\n",
            "recall_15                1       0.0000\n",
            "recall_20                1       0.0000\n",
            "recall_30                1       0.0000\n",
            "recall_100               1       0.0000\n",
            "recall_200               1       0.0000\n",
            "recall_500               1       0.0000\n",
            "recall_1000              1       0.0000\n",
            "infAP                    1       0.0000\n",
            "gm_bpref                 1       -11.5129\n",
            "Rprec_mult_0.20          1       0.0000\n",
            "Rprec_mult_0.40          1       0.0000\n",
            "Rprec_mult_0.60          1       0.0000\n",
            "Rprec_mult_0.80          1       0.0000\n",
            "Rprec_mult_1.00          1       0.0000\n",
            "Rprec_mult_1.20          1       0.0000\n",
            "Rprec_mult_1.40          1       0.0000\n",
            "Rprec_mult_1.60          1       0.0000\n",
            "Rprec_mult_1.80          1       0.0000\n",
            "Rprec_mult_2.00          1       0.0000\n",
            "utility                  1       -3.0000\n",
            "11pt_avg                 1       0.0000\n",
            "binG                     1       0.0000\n",
            "G                        1       0.0000\n",
            "ndcg                     1       0.0000\n",
            "ndcg_rel                 1       0.0000\n",
            "Rndcg                    1       0.0000\n",
            "ndcg_cut_5               1       0.0000\n",
            "ndcg_cut_10              1       0.0000\n",
            "ndcg_cut_15              1       0.0000\n",
            "ndcg_cut_20              1       0.0000\n",
            "ndcg_cut_30              1       0.0000\n",
            "ndcg_cut_100             1       0.0000\n",
            "ndcg_cut_200             1       0.0000\n",
            "ndcg_cut_500             1       0.0000\n",
            "ndcg_cut_1000            1       0.0000\n",
            "map_cut_5                1       0.0000\n",
            "map_cut_10               1       0.0000\n",
            "map_cut_15               1       0.0000\n",
            "map_cut_20               1       0.0000\n",
            "map_cut_30               1       0.0000\n",
            "map_cut_100              1       0.0000\n",
            "map_cut_200              1       0.0000\n",
            "map_cut_500              1       0.0000\n",
            "map_cut_1000             1       0.0000\n",
            "relative_P_5             1       0.0000\n",
            "relative_P_10            1       0.0000\n",
            "relative_P_15            1       0.0000\n",
            "relative_P_20            1       0.0000\n",
            "relative_P_30            1       0.0000\n",
            "relative_P_100           1       0.0000\n",
            "relative_P_200           1       0.0000\n",
            "relative_P_500           1       0.0000\n",
            "relative_P_1000          1       0.0000\n",
            "success_1                1       0.0000\n",
            "success_5                1       0.0000\n",
            "success_10               1       0.0000\n",
            "set_P                    1       0.0000\n",
            "set_relative_P           1       0.0000\n",
            "set_recall               1       0.0000\n",
            "set_map                  1       0.0000\n",
            "set_F                    1       0.0000\n",
            "num_nonrel_judged_ret    1       3.0000\n",
            "num_q                    2       1.0000\n",
            "num_ret                  2       13.0000\n",
            "num_rel                  2       2.0000\n",
            "num_rel_ret              2       1.0000\n",
            "map                      2       0.5000\n",
            "gm_map                   2       -0.6931\n",
            "Rprec                    2       0.5000\n",
            "bpref                    2       0.5000\n",
            "recip_rank               2       1.0000\n",
            "iprec_at_recall_0.00     2       1.0000\n",
            "iprec_at_recall_0.10     2       1.0000\n",
            "iprec_at_recall_0.20     2       1.0000\n",
            "iprec_at_recall_0.30     2       1.0000\n",
            "iprec_at_recall_0.40     2       1.0000\n",
            "iprec_at_recall_0.50     2       1.0000\n",
            "iprec_at_recall_0.60     2       0.0000\n",
            "iprec_at_recall_0.70     2       0.0000\n",
            "iprec_at_recall_0.80     2       0.0000\n",
            "iprec_at_recall_0.90     2       0.0000\n",
            "iprec_at_recall_1.00     2       0.0000\n",
            "P_5                      2       0.2000\n",
            "P_10                     2       0.1000\n",
            "P_15                     2       0.0667\n",
            "P_20                     2       0.0500\n",
            "P_30                     2       0.0333\n",
            "P_100                    2       0.0100\n",
            "P_200                    2       0.0050\n",
            "P_500                    2       0.0020\n",
            "P_1000                   2       0.0010\n",
            "relstring                2       0.0000\n",
            "recall_5                 2       0.5000\n",
            "recall_10                2       0.5000\n",
            "recall_15                2       0.5000\n",
            "recall_20                2       0.5000\n",
            "recall_30                2       0.5000\n",
            "recall_100               2       0.5000\n",
            "recall_200               2       0.5000\n",
            "recall_500               2       0.5000\n",
            "recall_1000              2       0.5000\n",
            "infAP                    2       0.5000\n",
            "gm_bpref                 2       -0.6931\n",
            "Rprec_mult_0.20          2       1.0000\n",
            "Rprec_mult_0.40          2       1.0000\n",
            "Rprec_mult_0.60          2       0.5000\n",
            "Rprec_mult_0.80          2       0.5000\n",
            "Rprec_mult_1.00          2       0.5000\n",
            "Rprec_mult_1.20          2       0.3333\n",
            "Rprec_mult_1.40          2       0.3333\n",
            "Rprec_mult_1.60          2       0.2500\n",
            "Rprec_mult_1.80          2       0.2500\n",
            "Rprec_mult_2.00          2       0.2500\n",
            "utility                  2       -11.0000\n",
            "11pt_avg                 2       0.5455\n",
            "binG                     2       0.5000\n",
            "G                        2       0.5000\n",
            "ndcg                     2       0.6131\n",
            "ndcg_rel                 2       0.8066\n",
            "Rndcg                    2       0.6131\n",
            "ndcg_cut_5               2       0.6131\n",
            "ndcg_cut_10              2       0.6131\n",
            "ndcg_cut_15              2       0.6131\n",
            "ndcg_cut_20              2       0.6131\n",
            "ndcg_cut_30              2       0.6131\n",
            "ndcg_cut_100             2       0.6131\n",
            "ndcg_cut_200             2       0.6131\n",
            "ndcg_cut_500             2       0.6131\n",
            "ndcg_cut_1000            2       0.6131\n",
            "map_cut_5                2       0.5000\n",
            "map_cut_10               2       0.5000\n",
            "map_cut_15               2       0.5000\n",
            "map_cut_20               2       0.5000\n",
            "map_cut_30               2       0.5000\n",
            "map_cut_100              2       0.5000\n",
            "map_cut_200              2       0.5000\n",
            "map_cut_500              2       0.5000\n",
            "map_cut_1000             2       0.5000\n",
            "relative_P_5             2       0.5000\n",
            "relative_P_10            2       0.5000\n",
            "relative_P_15            2       0.5000\n",
            "relative_P_20            2       0.5000\n",
            "relative_P_30            2       0.5000\n",
            "relative_P_100           2       0.5000\n",
            "relative_P_200           2       0.5000\n",
            "relative_P_500           2       0.5000\n",
            "relative_P_1000          2       0.5000\n",
            "success_1                2       1.0000\n",
            "success_5                2       1.0000\n",
            "success_10               2       1.0000\n",
            "set_P                    2       0.0769\n",
            "set_relative_P           2       0.5000\n",
            "set_recall               2       0.5000\n",
            "set_map                  2       0.0385\n",
            "set_F                    2       0.1333\n",
            "num_nonrel_judged_ret    2       9.0000\n",
            "num_q                    4       1.0000\n",
            "num_ret                  4       40.0000\n",
            "num_rel                  4       4.0000\n",
            "num_rel_ret              4       3.0000\n",
            "map                      4       0.5375\n",
            "gm_map                   4       -0.6208\n",
            "Rprec                    4       0.5000\n",
            "bpref                    4       0.5000\n",
            "recip_rank               4       1.0000\n",
            "iprec_at_recall_0.00     4       1.0000\n",
            "iprec_at_recall_0.10     4       1.0000\n",
            "iprec_at_recall_0.20     4       1.0000\n",
            "iprec_at_recall_0.30     4       1.0000\n",
            "iprec_at_recall_0.40     4       1.0000\n",
            "iprec_at_recall_0.50     4       1.0000\n",
            "iprec_at_recall_0.60     4       0.1500\n",
            "iprec_at_recall_0.70     4       0.1500\n",
            "iprec_at_recall_0.80     4       0.0000\n",
            "iprec_at_recall_0.90     4       0.0000\n",
            "iprec_at_recall_1.00     4       0.0000\n",
            "P_5                      4       0.4000\n",
            "P_10                     4       0.2000\n",
            "P_15                     4       0.1333\n",
            "P_20                     4       0.1500\n",
            "P_30                     4       0.1000\n",
            "P_100                    4       0.0300\n",
            "P_200                    4       0.0150\n",
            "P_500                    4       0.0060\n",
            "P_1000                   4       0.0030\n",
            "relstring                4       0.0000\n",
            "recall_5                 4       0.5000\n",
            "recall_10                4       0.5000\n",
            "recall_15                4       0.5000\n",
            "recall_20                4       0.7500\n",
            "recall_30                4       0.7500\n",
            "recall_100               4       0.7500\n",
            "recall_200               4       0.7500\n",
            "recall_500               4       0.7500\n",
            "recall_1000              4       0.7500\n",
            "infAP                    4       0.5375\n",
            "gm_bpref                 4       -0.6931\n",
            "Rprec_mult_0.20          4       1.0000\n",
            "Rprec_mult_0.40          4       1.0000\n",
            "Rprec_mult_0.60          4       0.6667\n",
            "Rprec_mult_0.80          4       0.5000\n",
            "Rprec_mult_1.00          4       0.5000\n",
            "Rprec_mult_1.20          4       0.4000\n",
            "Rprec_mult_1.40          4       0.3333\n",
            "Rprec_mult_1.60          4       0.2857\n",
            "Rprec_mult_1.80          4       0.2500\n",
            "Rprec_mult_2.00          4       0.2500\n",
            "utility                  4       -34.0000\n",
            "11pt_avg                 4       0.5727\n",
            "binG                     4       0.5589\n",
            "G                        4       0.5589\n",
            "ndcg                     4       0.7256\n",
            "ndcg_rel                 4       0.8628\n",
            "Rndcg                    4       0.6811\n",
            "ndcg_cut_5               4       0.6367\n",
            "ndcg_cut_10              4       0.6367\n",
            "ndcg_cut_15              4       0.6367\n",
            "ndcg_cut_20              4       0.7256\n",
            "ndcg_cut_30              4       0.7256\n",
            "ndcg_cut_100             4       0.7256\n",
            "ndcg_cut_200             4       0.7256\n",
            "ndcg_cut_500             4       0.7256\n",
            "ndcg_cut_1000            4       0.7256\n",
            "map_cut_5                4       0.5000\n",
            "map_cut_10               4       0.5000\n",
            "map_cut_15               4       0.5000\n",
            "map_cut_20               4       0.5375\n",
            "map_cut_30               4       0.5375\n",
            "map_cut_100              4       0.5375\n",
            "map_cut_200              4       0.5375\n",
            "map_cut_500              4       0.5375\n",
            "map_cut_1000             4       0.5375\n",
            "relative_P_5             4       0.5000\n",
            "relative_P_10            4       0.5000\n",
            "relative_P_15            4       0.5000\n",
            "relative_P_20            4       0.7500\n",
            "relative_P_30            4       0.7500\n",
            "relative_P_100           4       0.7500\n",
            "relative_P_200           4       0.7500\n",
            "relative_P_500           4       0.7500\n",
            "relative_P_1000          4       0.7500\n",
            "success_1                4       1.0000\n",
            "success_5                4       1.0000\n",
            "success_10               4       1.0000\n",
            "set_P                    4       0.0750\n",
            "set_relative_P           4       0.7500\n",
            "set_recall               4       0.7500\n",
            "set_map                  4       0.0563\n",
            "set_F                    4       0.1364\n",
            "num_nonrel_judged_ret    4       25.0000\n",
            "num_q                    6       1.0000\n",
            "num_ret                  6       40.0000\n",
            "num_rel                  6       1.0000\n",
            "num_rel_ret              6       1.0000\n",
            "map                      6       0.1429\n",
            "gm_map                   6       -1.9459\n",
            "Rprec                    6       0.0000\n",
            "bpref                    6       0.0000\n",
            "recip_rank               6       0.1429\n",
            "iprec_at_recall_0.00     6       0.1429\n",
            "iprec_at_recall_0.10     6       0.1429\n",
            "iprec_at_recall_0.20     6       0.1429\n",
            "iprec_at_recall_0.30     6       0.1429\n",
            "iprec_at_recall_0.40     6       0.1429\n",
            "iprec_at_recall_0.50     6       0.1429\n",
            "iprec_at_recall_0.60     6       0.1429\n",
            "iprec_at_recall_0.70     6       0.1429\n",
            "iprec_at_recall_0.80     6       0.1429\n",
            "iprec_at_recall_0.90     6       0.1429\n",
            "iprec_at_recall_1.00     6       0.1429\n",
            "P_5                      6       0.0000\n",
            "P_10                     6       0.1000\n",
            "P_15                     6       0.0667\n",
            "P_20                     6       0.0500\n",
            "P_30                     6       0.0333\n",
            "P_100                    6       0.0100\n",
            "P_200                    6       0.0050\n",
            "P_500                    6       0.0020\n",
            "P_1000                   6       0.0010\n",
            "relstring                6       0.0000\n",
            "recall_5                 6       0.0000\n",
            "recall_10                6       1.0000\n",
            "recall_15                6       1.0000\n",
            "recall_20                6       1.0000\n",
            "recall_30                6       1.0000\n",
            "recall_100               6       1.0000\n",
            "recall_200               6       1.0000\n",
            "recall_500               6       1.0000\n",
            "recall_1000              6       1.0000\n",
            "infAP                    6       0.1429\n",
            "gm_bpref                 6       -11.5129\n",
            "Rprec_mult_0.20          6       0.0000\n",
            "Rprec_mult_0.40          6       0.0000\n",
            "Rprec_mult_0.60          6       0.0000\n",
            "Rprec_mult_0.80          6       0.0000\n",
            "Rprec_mult_1.00          6       0.0000\n",
            "Rprec_mult_1.20          6       0.0000\n",
            "Rprec_mult_1.40          6       0.0000\n",
            "Rprec_mult_1.60          6       0.0000\n",
            "Rprec_mult_1.80          6       0.0000\n",
            "Rprec_mult_2.00          6       0.0000\n",
            "utility                  6       -38.0000\n",
            "11pt_avg                 6       0.1429\n",
            "binG                     6       0.3333\n",
            "G                        6       0.3333\n",
            "ndcg                     6       0.3333\n",
            "ndcg_rel                 6       0.3333\n",
            "Rndcg                    6       0.1667\n",
            "ndcg_cut_5               6       0.0000\n",
            "ndcg_cut_10              6       0.3333\n",
            "ndcg_cut_15              6       0.3333\n",
            "ndcg_cut_20              6       0.3333\n",
            "ndcg_cut_30              6       0.3333\n",
            "ndcg_cut_100             6       0.3333\n",
            "ndcg_cut_200             6       0.3333\n",
            "ndcg_cut_500             6       0.3333\n",
            "ndcg_cut_1000            6       0.3333\n",
            "map_cut_5                6       0.0000\n",
            "map_cut_10               6       0.1429\n",
            "map_cut_15               6       0.1429\n",
            "map_cut_20               6       0.1429\n",
            "map_cut_30               6       0.1429\n",
            "map_cut_100              6       0.1429\n",
            "map_cut_200              6       0.1429\n",
            "map_cut_500              6       0.1429\n",
            "map_cut_1000             6       0.1429\n",
            "relative_P_5             6       0.0000\n",
            "relative_P_10            6       1.0000\n",
            "relative_P_15            6       1.0000\n",
            "relative_P_20            6       1.0000\n",
            "relative_P_30            6       1.0000\n",
            "relative_P_100           6       1.0000\n",
            "relative_P_200           6       1.0000\n",
            "relative_P_500           6       1.0000\n",
            "relative_P_1000          6       1.0000\n",
            "success_1                6       0.0000\n",
            "success_5                6       0.0000\n",
            "success_10               6       1.0000\n",
            "set_P                    6       0.0250\n",
            "set_relative_P           6       1.0000\n",
            "set_recall               6       1.0000\n",
            "set_map                  6       0.0250\n",
            "set_F                    6       0.0488\n",
            "num_nonrel_judged_ret    6       17.0000\n",
            "num_q                    7       1.0000\n",
            "num_ret                  7       14.0000\n",
            "num_rel                  7       3.0000\n",
            "num_rel_ret              7       0.0000\n",
            "map                      7       0.0000\n",
            "gm_map                   7       -11.5129\n",
            "Rprec                    7       0.0000\n",
            "bpref                    7       0.0000\n",
            "recip_rank               7       0.0000\n",
            "iprec_at_recall_0.00     7       0.0000\n",
            "iprec_at_recall_0.10     7       0.0000\n",
            "iprec_at_recall_0.20     7       0.0000\n",
            "iprec_at_recall_0.30     7       0.0000\n",
            "iprec_at_recall_0.40     7       0.0000\n",
            "iprec_at_recall_0.50     7       0.0000\n",
            "iprec_at_recall_0.60     7       0.0000\n",
            "iprec_at_recall_0.70     7       0.0000\n",
            "iprec_at_recall_0.80     7       0.0000\n",
            "iprec_at_recall_0.90     7       0.0000\n",
            "iprec_at_recall_1.00     7       0.0000\n",
            "P_5                      7       0.0000\n",
            "P_10                     7       0.0000\n",
            "P_15                     7       0.0000\n",
            "P_20                     7       0.0000\n",
            "P_30                     7       0.0000\n",
            "P_100                    7       0.0000\n",
            "P_200                    7       0.0000\n",
            "P_500                    7       0.0000\n",
            "P_1000                   7       0.0000\n",
            "relstring                7       0.0000\n",
            "recall_5                 7       0.0000\n",
            "recall_10                7       0.0000\n",
            "recall_15                7       0.0000\n",
            "recall_20                7       0.0000\n",
            "recall_30                7       0.0000\n",
            "recall_100               7       0.0000\n",
            "recall_200               7       0.0000\n",
            "recall_500               7       0.0000\n",
            "recall_1000              7       0.0000\n",
            "infAP                    7       0.0000\n",
            "gm_bpref                 7       -11.5129\n",
            "Rprec_mult_0.20          7       0.0000\n",
            "Rprec_mult_0.40          7       0.0000\n",
            "Rprec_mult_0.60          7       0.0000\n",
            "Rprec_mult_0.80          7       0.0000\n",
            "Rprec_mult_1.00          7       0.0000\n",
            "Rprec_mult_1.20          7       0.0000\n",
            "Rprec_mult_1.40          7       0.0000\n",
            "Rprec_mult_1.60          7       0.0000\n",
            "Rprec_mult_1.80          7       0.0000\n",
            "Rprec_mult_2.00          7       0.0000\n",
            "utility                  7       -14.0000\n",
            "11pt_avg                 7       0.0000\n",
            "binG                     7       0.0000\n",
            "G                        7       0.0000\n",
            "ndcg                     7       0.0000\n",
            "ndcg_rel                 7       0.0000\n",
            "Rndcg                    7       0.0000\n",
            "ndcg_cut_5               7       0.0000\n",
            "ndcg_cut_10              7       0.0000\n",
            "ndcg_cut_15              7       0.0000\n",
            "ndcg_cut_20              7       0.0000\n",
            "ndcg_cut_30              7       0.0000\n",
            "ndcg_cut_100             7       0.0000\n",
            "ndcg_cut_200             7       0.0000\n",
            "ndcg_cut_500             7       0.0000\n",
            "ndcg_cut_1000            7       0.0000\n",
            "map_cut_5                7       0.0000\n",
            "map_cut_10               7       0.0000\n",
            "map_cut_15               7       0.0000\n",
            "map_cut_20               7       0.0000\n",
            "map_cut_30               7       0.0000\n",
            "map_cut_100              7       0.0000\n",
            "map_cut_200              7       0.0000\n",
            "map_cut_500              7       0.0000\n",
            "map_cut_1000             7       0.0000\n",
            "relative_P_5             7       0.0000\n",
            "relative_P_10            7       0.0000\n",
            "relative_P_15            7       0.0000\n",
            "relative_P_20            7       0.0000\n",
            "relative_P_30            7       0.0000\n",
            "relative_P_100           7       0.0000\n",
            "relative_P_200           7       0.0000\n",
            "relative_P_500           7       0.0000\n",
            "relative_P_1000          7       0.0000\n",
            "success_1                7       0.0000\n",
            "success_5                7       0.0000\n",
            "success_10               7       0.0000\n",
            "set_P                    7       0.0000\n",
            "set_relative_P           7       0.0000\n",
            "set_recall               7       0.0000\n",
            "set_map                  7       0.0000\n",
            "set_F                    7       0.0000\n",
            "num_nonrel_judged_ret    7       11.0000\n",
            "num_q                    9       1.0000\n",
            "num_ret                  9       27.0000\n",
            "num_rel                  9       1.0000\n",
            "num_rel_ret              9       1.0000\n",
            "map                      9       0.2000\n",
            "gm_map                   9       -1.6094\n",
            "Rprec                    9       0.0000\n",
            "bpref                    9       0.0000\n",
            "recip_rank               9       0.2000\n",
            "iprec_at_recall_0.00     9       0.2000\n",
            "iprec_at_recall_0.10     9       0.2000\n",
            "iprec_at_recall_0.20     9       0.2000\n",
            "iprec_at_recall_0.30     9       0.2000\n",
            "iprec_at_recall_0.40     9       0.2000\n",
            "iprec_at_recall_0.50     9       0.2000\n",
            "iprec_at_recall_0.60     9       0.2000\n",
            "iprec_at_recall_0.70     9       0.2000\n",
            "iprec_at_recall_0.80     9       0.2000\n",
            "iprec_at_recall_0.90     9       0.2000\n",
            "iprec_at_recall_1.00     9       0.2000\n",
            "P_5                      9       0.2000\n",
            "P_10                     9       0.1000\n",
            "P_15                     9       0.0667\n",
            "P_20                     9       0.0500\n",
            "P_30                     9       0.0333\n",
            "P_100                    9       0.0100\n",
            "P_200                    9       0.0050\n",
            "P_500                    9       0.0020\n",
            "P_1000                   9       0.0010\n",
            "relstring                9       0.0000\n",
            "recall_5                 9       1.0000\n",
            "recall_10                9       1.0000\n",
            "recall_15                9       1.0000\n",
            "recall_20                9       1.0000\n",
            "recall_30                9       1.0000\n",
            "recall_100               9       1.0000\n",
            "recall_200               9       1.0000\n",
            "recall_500               9       1.0000\n",
            "recall_1000              9       1.0000\n",
            "infAP                    9       0.2000\n",
            "gm_bpref                 9       -11.5129\n",
            "Rprec_mult_0.20          9       0.0000\n",
            "Rprec_mult_0.40          9       0.0000\n",
            "Rprec_mult_0.60          9       0.0000\n",
            "Rprec_mult_0.80          9       0.0000\n",
            "Rprec_mult_1.00          9       0.0000\n",
            "Rprec_mult_1.20          9       0.0000\n",
            "Rprec_mult_1.40          9       0.0000\n",
            "Rprec_mult_1.60          9       0.0000\n",
            "Rprec_mult_1.80          9       0.0000\n",
            "Rprec_mult_2.00          9       0.0000\n",
            "utility                  9       -25.0000\n",
            "11pt_avg                 9       0.2000\n",
            "binG                     9       0.3869\n",
            "G                        9       0.3869\n",
            "ndcg                     9       0.3869\n",
            "ndcg_rel                 9       0.3869\n",
            "Rndcg                    9       0.1934\n",
            "ndcg_cut_5               9       0.3869\n",
            "ndcg_cut_10              9       0.3869\n",
            "ndcg_cut_15              9       0.3869\n",
            "ndcg_cut_20              9       0.3869\n",
            "ndcg_cut_30              9       0.3869\n",
            "ndcg_cut_100             9       0.3869\n",
            "ndcg_cut_200             9       0.3869\n",
            "ndcg_cut_500             9       0.3869\n",
            "ndcg_cut_1000            9       0.3869\n",
            "map_cut_5                9       0.2000\n",
            "map_cut_10               9       0.2000\n",
            "map_cut_15               9       0.2000\n",
            "map_cut_20               9       0.2000\n",
            "map_cut_30               9       0.2000\n",
            "map_cut_100              9       0.2000\n",
            "map_cut_200              9       0.2000\n",
            "map_cut_500              9       0.2000\n",
            "map_cut_1000             9       0.2000\n",
            "relative_P_5             9       1.0000\n",
            "relative_P_10            9       1.0000\n",
            "relative_P_15            9       1.0000\n",
            "relative_P_20            9       1.0000\n",
            "relative_P_30            9       1.0000\n",
            "relative_P_100           9       1.0000\n",
            "relative_P_200           9       1.0000\n",
            "relative_P_500           9       1.0000\n",
            "relative_P_1000          9       1.0000\n",
            "success_1                9       0.0000\n",
            "success_5                9       1.0000\n",
            "success_10               9       1.0000\n",
            "set_P                    9       0.0370\n",
            "set_relative_P           9       1.0000\n",
            "set_recall               9       1.0000\n",
            "set_map                  9       0.0370\n",
            "set_F                    9       0.0714\n",
            "num_nonrel_judged_ret    9       23.0000\n",
            "num_q                    10      1.0000\n",
            "num_ret                  10      52.0000\n",
            "num_rel                  10      1.0000\n",
            "num_rel_ret              10      1.0000\n",
            "map                      10      0.3333\n",
            "gm_map                   10      -1.0986\n",
            "Rprec                    10      0.0000\n",
            "bpref                    10      0.0000\n",
            "recip_rank               10      0.3333\n",
            "iprec_at_recall_0.00     10      0.3333\n",
            "iprec_at_recall_0.10     10      0.3333\n",
            "iprec_at_recall_0.20     10      0.3333\n",
            "iprec_at_recall_0.30     10      0.3333\n",
            "iprec_at_recall_0.40     10      0.3333\n",
            "iprec_at_recall_0.50     10      0.3333\n",
            "iprec_at_recall_0.60     10      0.3333\n",
            "iprec_at_recall_0.70     10      0.3333\n",
            "iprec_at_recall_0.80     10      0.3333\n",
            "iprec_at_recall_0.90     10      0.3333\n",
            "iprec_at_recall_1.00     10      0.3333\n",
            "P_5                      10      0.2000\n",
            "P_10                     10      0.1000\n",
            "P_15                     10      0.0667\n",
            "P_20                     10      0.0500\n",
            "P_30                     10      0.0333\n",
            "P_100                    10      0.0100\n",
            "P_200                    10      0.0050\n",
            "P_500                    10      0.0020\n",
            "P_1000                   10      0.0010\n",
            "relstring                10      0.0000\n",
            "recall_5                 10      1.0000\n",
            "recall_10                10      1.0000\n",
            "recall_15                10      1.0000\n",
            "recall_20                10      1.0000\n",
            "recall_30                10      1.0000\n",
            "recall_100               10      1.0000\n",
            "recall_200               10      1.0000\n",
            "recall_500               10      1.0000\n",
            "recall_1000              10      1.0000\n",
            "infAP                    10      0.3333\n",
            "gm_bpref                 10      -11.5129\n",
            "Rprec_mult_0.20          10      0.0000\n",
            "Rprec_mult_0.40          10      0.0000\n",
            "Rprec_mult_0.60          10      0.0000\n",
            "Rprec_mult_0.80          10      0.0000\n",
            "Rprec_mult_1.00          10      0.0000\n",
            "Rprec_mult_1.20          10      0.0000\n",
            "Rprec_mult_1.40          10      0.0000\n",
            "Rprec_mult_1.60          10      0.0000\n",
            "Rprec_mult_1.80          10      0.0000\n",
            "Rprec_mult_2.00          10      0.0000\n",
            "utility                  10      -50.0000\n",
            "11pt_avg                 10      0.3333\n",
            "binG                     10      0.5000\n",
            "G                        10      0.5000\n",
            "ndcg                     10      0.5000\n",
            "ndcg_rel                 10      0.5000\n",
            "Rndcg                    10      0.2500\n",
            "ndcg_cut_5               10      0.5000\n",
            "ndcg_cut_10              10      0.5000\n",
            "ndcg_cut_15              10      0.5000\n",
            "ndcg_cut_20              10      0.5000\n",
            "ndcg_cut_30              10      0.5000\n",
            "ndcg_cut_100             10      0.5000\n",
            "ndcg_cut_200             10      0.5000\n",
            "ndcg_cut_500             10      0.5000\n",
            "ndcg_cut_1000            10      0.5000\n",
            "map_cut_5                10      0.3333\n",
            "map_cut_10               10      0.3333\n",
            "map_cut_15               10      0.3333\n",
            "map_cut_20               10      0.3333\n",
            "map_cut_30               10      0.3333\n",
            "map_cut_100              10      0.3333\n",
            "map_cut_200              10      0.3333\n",
            "map_cut_500              10      0.3333\n",
            "map_cut_1000             10      0.3333\n",
            "relative_P_5             10      1.0000\n",
            "relative_P_10            10      1.0000\n",
            "relative_P_15            10      1.0000\n",
            "relative_P_20            10      1.0000\n",
            "relative_P_30            10      1.0000\n",
            "relative_P_100           10      1.0000\n",
            "relative_P_200           10      1.0000\n",
            "relative_P_500           10      1.0000\n",
            "relative_P_1000          10      1.0000\n",
            "success_1                10      0.0000\n",
            "success_5                10      1.0000\n",
            "success_10               10      1.0000\n",
            "set_P                    10      0.0192\n",
            "set_relative_P           10      1.0000\n",
            "set_recall               10      1.0000\n",
            "set_map                  10      0.0192\n",
            "set_F                    10      0.0377\n",
            "num_nonrel_judged_ret    10      16.0000\n",
            "num_q                    14      1.0000\n",
            "num_ret                  14      26.0000\n",
            "num_rel                  14      1.0000\n",
            "num_rel_ret              14      1.0000\n",
            "map                      14      1.0000\n",
            "gm_map                   14      0.0000\n",
            "Rprec                    14      1.0000\n",
            "bpref                    14      1.0000\n",
            "recip_rank               14      1.0000\n",
            "iprec_at_recall_0.00     14      1.0000\n",
            "iprec_at_recall_0.10     14      1.0000\n",
            "iprec_at_recall_0.20     14      1.0000\n",
            "iprec_at_recall_0.30     14      1.0000\n",
            "iprec_at_recall_0.40     14      1.0000\n",
            "iprec_at_recall_0.50     14      1.0000\n",
            "iprec_at_recall_0.60     14      1.0000\n",
            "iprec_at_recall_0.70     14      1.0000\n",
            "iprec_at_recall_0.80     14      1.0000\n",
            "iprec_at_recall_0.90     14      1.0000\n",
            "iprec_at_recall_1.00     14      1.0000\n",
            "P_5                      14      0.2000\n",
            "P_10                     14      0.1000\n",
            "P_15                     14      0.0667\n",
            "P_20                     14      0.0500\n",
            "P_30                     14      0.0333\n",
            "P_100                    14      0.0100\n",
            "P_200                    14      0.0050\n",
            "P_500                    14      0.0020\n",
            "P_1000                   14      0.0010\n",
            "relstring                14      0.0000\n",
            "recall_5                 14      1.0000\n",
            "recall_10                14      1.0000\n",
            "recall_15                14      1.0000\n",
            "recall_20                14      1.0000\n",
            "recall_30                14      1.0000\n",
            "recall_100               14      1.0000\n",
            "recall_200               14      1.0000\n",
            "recall_500               14      1.0000\n",
            "recall_1000              14      1.0000\n",
            "infAP                    14      1.0000\n",
            "gm_bpref                 14      0.0000\n",
            "Rprec_mult_0.20          14      1.0000\n",
            "Rprec_mult_0.40          14      1.0000\n",
            "Rprec_mult_0.60          14      1.0000\n",
            "Rprec_mult_0.80          14      1.0000\n",
            "Rprec_mult_1.00          14      1.0000\n",
            "Rprec_mult_1.20          14      0.5000\n",
            "Rprec_mult_1.40          14      0.5000\n",
            "Rprec_mult_1.60          14      0.5000\n",
            "Rprec_mult_1.80          14      0.5000\n",
            "Rprec_mult_2.00          14      0.5000\n",
            "utility                  14      -24.0000\n",
            "11pt_avg                 14      1.0000\n",
            "binG                     14      1.0000\n",
            "G                        14      1.0000\n",
            "ndcg                     14      1.0000\n",
            "ndcg_rel                 14      1.0000\n",
            "Rndcg                    14      1.0000\n",
            "ndcg_cut_5               14      1.0000\n",
            "ndcg_cut_10              14      1.0000\n",
            "ndcg_cut_15              14      1.0000\n",
            "ndcg_cut_20              14      1.0000\n",
            "ndcg_cut_30              14      1.0000\n",
            "ndcg_cut_100             14      1.0000\n",
            "ndcg_cut_200             14      1.0000\n",
            "ndcg_cut_500             14      1.0000\n",
            "ndcg_cut_1000            14      1.0000\n",
            "map_cut_5                14      1.0000\n",
            "map_cut_10               14      1.0000\n",
            "map_cut_15               14      1.0000\n",
            "map_cut_20               14      1.0000\n",
            "map_cut_30               14      1.0000\n",
            "map_cut_100              14      1.0000\n",
            "map_cut_200              14      1.0000\n",
            "map_cut_500              14      1.0000\n",
            "map_cut_1000             14      1.0000\n",
            "relative_P_5             14      1.0000\n",
            "relative_P_10            14      1.0000\n",
            "relative_P_15            14      1.0000\n",
            "relative_P_20            14      1.0000\n",
            "relative_P_30            14      1.0000\n",
            "relative_P_100           14      1.0000\n",
            "relative_P_200           14      1.0000\n",
            "relative_P_500           14      1.0000\n",
            "relative_P_1000          14      1.0000\n",
            "success_1                14      1.0000\n",
            "success_5                14      1.0000\n",
            "success_10               14      1.0000\n",
            "set_P                    14      0.0385\n",
            "set_relative_P           14      1.0000\n",
            "set_recall               14      1.0000\n",
            "set_map                  14      0.0385\n",
            "set_F                    14      0.0741\n",
            "num_nonrel_judged_ret    14      20.0000\n",
            "num_q                    16      1.0000\n",
            "num_ret                  16      33.0000\n",
            "num_rel                  16      7.0000\n",
            "num_rel_ret              16      0.0000\n",
            "map                      16      0.0000\n",
            "gm_map                   16      -11.5129\n",
            "Rprec                    16      0.0000\n",
            "bpref                    16      0.0000\n",
            "recip_rank               16      0.0000\n",
            "iprec_at_recall_0.00     16      0.0000\n",
            "iprec_at_recall_0.10     16      0.0000\n",
            "iprec_at_recall_0.20     16      0.0000\n",
            "iprec_at_recall_0.30     16      0.0000\n",
            "iprec_at_recall_0.40     16      0.0000\n",
            "iprec_at_recall_0.50     16      0.0000\n",
            "iprec_at_recall_0.60     16      0.0000\n",
            "iprec_at_recall_0.70     16      0.0000\n",
            "iprec_at_recall_0.80     16      0.0000\n",
            "iprec_at_recall_0.90     16      0.0000\n",
            "iprec_at_recall_1.00     16      0.0000\n",
            "P_5                      16      0.0000\n",
            "P_10                     16      0.0000\n",
            "P_15                     16      0.0000\n",
            "P_20                     16      0.0000\n",
            "P_30                     16      0.0000\n",
            "P_100                    16      0.0000\n",
            "P_200                    16      0.0000\n",
            "P_500                    16      0.0000\n",
            "P_1000                   16      0.0000\n",
            "relstring                16      0.0000\n",
            "recall_5                 16      0.0000\n",
            "recall_10                16      0.0000\n",
            "recall_15                16      0.0000\n",
            "recall_20                16      0.0000\n",
            "recall_30                16      0.0000\n",
            "recall_100               16      0.0000\n",
            "recall_200               16      0.0000\n",
            "recall_500               16      0.0000\n",
            "recall_1000              16      0.0000\n",
            "infAP                    16      0.0000\n",
            "gm_bpref                 16      -11.5129\n",
            "Rprec_mult_0.20          16      0.0000\n",
            "Rprec_mult_0.40          16      0.0000\n",
            "Rprec_mult_0.60          16      0.0000\n",
            "Rprec_mult_0.80          16      0.0000\n",
            "Rprec_mult_1.00          16      0.0000\n",
            "Rprec_mult_1.20          16      0.0000\n",
            "Rprec_mult_1.40          16      0.0000\n",
            "Rprec_mult_1.60          16      0.0000\n",
            "Rprec_mult_1.80          16      0.0000\n",
            "Rprec_mult_2.00          16      0.0000\n",
            "utility                  16      -33.0000\n",
            "11pt_avg                 16      0.0000\n",
            "binG                     16      0.0000\n",
            "G                        16      0.0000\n",
            "ndcg                     16      0.0000\n",
            "ndcg_rel                 16      0.0000\n",
            "Rndcg                    16      0.0000\n",
            "ndcg_cut_5               16      0.0000\n",
            "ndcg_cut_10              16      0.0000\n",
            "ndcg_cut_15              16      0.0000\n",
            "ndcg_cut_20              16      0.0000\n",
            "ndcg_cut_30              16      0.0000\n",
            "ndcg_cut_100             16      0.0000\n",
            "ndcg_cut_200             16      0.0000\n",
            "ndcg_cut_500             16      0.0000\n",
            "ndcg_cut_1000            16      0.0000\n",
            "map_cut_5                16      0.0000\n",
            "map_cut_10               16      0.0000\n",
            "map_cut_15               16      0.0000\n",
            "map_cut_20               16      0.0000\n",
            "map_cut_30               16      0.0000\n",
            "map_cut_100              16      0.0000\n",
            "map_cut_200              16      0.0000\n",
            "map_cut_500              16      0.0000\n",
            "map_cut_1000             16      0.0000\n",
            "relative_P_5             16      0.0000\n",
            "relative_P_10            16      0.0000\n",
            "relative_P_15            16      0.0000\n",
            "relative_P_20            16      0.0000\n",
            "relative_P_30            16      0.0000\n",
            "relative_P_100           16      0.0000\n",
            "relative_P_200           16      0.0000\n",
            "relative_P_500           16      0.0000\n",
            "relative_P_1000          16      0.0000\n",
            "success_1                16      0.0000\n",
            "success_5                16      0.0000\n",
            "success_10               16      0.0000\n",
            "set_P                    16      0.0000\n",
            "set_relative_P           16      0.0000\n",
            "set_recall               16      0.0000\n",
            "set_map                  16      0.0000\n",
            "set_F                    16      0.0000\n",
            "num_nonrel_judged_ret    16      20.0000\n",
            "num_q                    18      1.0000\n",
            "num_ret                  18      32.0000\n",
            "num_rel                  18      1.0000\n",
            "num_rel_ret              18      1.0000\n",
            "map                      18      1.0000\n",
            "gm_map                   18      0.0000\n",
            "Rprec                    18      1.0000\n",
            "bpref                    18      1.0000\n",
            "recip_rank               18      1.0000\n",
            "iprec_at_recall_0.00     18      1.0000\n",
            "iprec_at_recall_0.10     18      1.0000\n",
            "iprec_at_recall_0.20     18      1.0000\n",
            "iprec_at_recall_0.30     18      1.0000\n",
            "iprec_at_recall_0.40     18      1.0000\n",
            "iprec_at_recall_0.50     18      1.0000\n",
            "iprec_at_recall_0.60     18      1.0000\n",
            "iprec_at_recall_0.70     18      1.0000\n",
            "iprec_at_recall_0.80     18      1.0000\n",
            "iprec_at_recall_0.90     18      1.0000\n",
            "iprec_at_recall_1.00     18      1.0000\n",
            "P_5                      18      0.2000\n",
            "P_10                     18      0.1000\n",
            "P_15                     18      0.0667\n",
            "P_20                     18      0.0500\n",
            "P_30                     18      0.0333\n",
            "P_100                    18      0.0100\n",
            "P_200                    18      0.0050\n",
            "P_500                    18      0.0020\n",
            "P_1000                   18      0.0010\n",
            "relstring                18      0.0000\n",
            "recall_5                 18      1.0000\n",
            "recall_10                18      1.0000\n",
            "recall_15                18      1.0000\n",
            "recall_20                18      1.0000\n",
            "recall_30                18      1.0000\n",
            "recall_100               18      1.0000\n",
            "recall_200               18      1.0000\n",
            "recall_500               18      1.0000\n",
            "recall_1000              18      1.0000\n",
            "infAP                    18      1.0000\n",
            "gm_bpref                 18      0.0000\n",
            "Rprec_mult_0.20          18      1.0000\n",
            "Rprec_mult_0.40          18      1.0000\n",
            "Rprec_mult_0.60          18      1.0000\n",
            "Rprec_mult_0.80          18      1.0000\n",
            "Rprec_mult_1.00          18      1.0000\n",
            "Rprec_mult_1.20          18      0.5000\n",
            "Rprec_mult_1.40          18      0.5000\n",
            "Rprec_mult_1.60          18      0.5000\n",
            "Rprec_mult_1.80          18      0.5000\n",
            "Rprec_mult_2.00          18      0.5000\n",
            "utility                  18      -30.0000\n",
            "11pt_avg                 18      1.0000\n",
            "binG                     18      1.0000\n",
            "G                        18      1.0000\n",
            "ndcg                     18      1.0000\n",
            "ndcg_rel                 18      1.0000\n",
            "Rndcg                    18      1.0000\n",
            "ndcg_cut_5               18      1.0000\n",
            "ndcg_cut_10              18      1.0000\n",
            "ndcg_cut_15              18      1.0000\n",
            "ndcg_cut_20              18      1.0000\n",
            "ndcg_cut_30              18      1.0000\n",
            "ndcg_cut_100             18      1.0000\n",
            "ndcg_cut_200             18      1.0000\n",
            "ndcg_cut_500             18      1.0000\n",
            "ndcg_cut_1000            18      1.0000\n",
            "map_cut_5                18      1.0000\n",
            "map_cut_10               18      1.0000\n",
            "map_cut_15               18      1.0000\n",
            "map_cut_20               18      1.0000\n",
            "map_cut_30               18      1.0000\n",
            "map_cut_100              18      1.0000\n",
            "map_cut_200              18      1.0000\n",
            "map_cut_500              18      1.0000\n",
            "map_cut_1000             18      1.0000\n",
            "relative_P_5             18      1.0000\n",
            "relative_P_10            18      1.0000\n",
            "relative_P_15            18      1.0000\n",
            "relative_P_20            18      1.0000\n",
            "relative_P_30            18      1.0000\n",
            "relative_P_100           18      1.0000\n",
            "relative_P_200           18      1.0000\n",
            "relative_P_500           18      1.0000\n",
            "relative_P_1000          18      1.0000\n",
            "success_1                18      1.0000\n",
            "success_5                18      1.0000\n",
            "success_10               18      1.0000\n",
            "set_P                    18      0.0312\n",
            "set_relative_P           18      1.0000\n",
            "set_recall               18      1.0000\n",
            "set_map                  18      0.0312\n",
            "set_F                    18      0.0606\n",
            "num_nonrel_judged_ret    18      30.0000\n",
            "num_q                    19      1.0000\n",
            "num_ret                  19      5.0000\n",
            "num_rel                  19      2.0000\n",
            "num_rel_ret              19      1.0000\n",
            "map                      19      0.5000\n",
            "gm_map                   19      -0.6931\n",
            "Rprec                    19      0.5000\n",
            "bpref                    19      0.5000\n",
            "recip_rank               19      1.0000\n",
            "iprec_at_recall_0.00     19      1.0000\n",
            "iprec_at_recall_0.10     19      1.0000\n",
            "iprec_at_recall_0.20     19      1.0000\n",
            "iprec_at_recall_0.30     19      1.0000\n",
            "iprec_at_recall_0.40     19      1.0000\n",
            "iprec_at_recall_0.50     19      1.0000\n",
            "iprec_at_recall_0.60     19      0.0000\n",
            "iprec_at_recall_0.70     19      0.0000\n",
            "iprec_at_recall_0.80     19      0.0000\n",
            "iprec_at_recall_0.90     19      0.0000\n",
            "iprec_at_recall_1.00     19      0.0000\n",
            "P_5                      19      0.2000\n",
            "P_10                     19      0.1000\n",
            "P_15                     19      0.0667\n",
            "P_20                     19      0.0500\n",
            "P_30                     19      0.0333\n",
            "P_100                    19      0.0100\n",
            "P_200                    19      0.0050\n",
            "P_500                    19      0.0020\n",
            "P_1000                   19      0.0010\n",
            "relstring                19      0.0000\n",
            "recall_5                 19      0.5000\n",
            "recall_10                19      0.5000\n",
            "recall_15                19      0.5000\n",
            "recall_20                19      0.5000\n",
            "recall_30                19      0.5000\n",
            "recall_100               19      0.5000\n",
            "recall_200               19      0.5000\n",
            "recall_500               19      0.5000\n",
            "recall_1000              19      0.5000\n",
            "infAP                    19      0.5000\n",
            "gm_bpref                 19      -0.6931\n",
            "Rprec_mult_0.20          19      1.0000\n",
            "Rprec_mult_0.40          19      1.0000\n",
            "Rprec_mult_0.60          19      0.5000\n",
            "Rprec_mult_0.80          19      0.5000\n",
            "Rprec_mult_1.00          19      0.5000\n",
            "Rprec_mult_1.20          19      0.3333\n",
            "Rprec_mult_1.40          19      0.3333\n",
            "Rprec_mult_1.60          19      0.2500\n",
            "Rprec_mult_1.80          19      0.2500\n",
            "Rprec_mult_2.00          19      0.2500\n",
            "utility                  19      -3.0000\n",
            "11pt_avg                 19      0.5455\n",
            "binG                     19      0.5000\n",
            "G                        19      0.5000\n",
            "ndcg                     19      0.6131\n",
            "ndcg_rel                 19      0.8066\n",
            "Rndcg                    19      0.6131\n",
            "ndcg_cut_5               19      0.6131\n",
            "ndcg_cut_10              19      0.6131\n",
            "ndcg_cut_15              19      0.6131\n",
            "ndcg_cut_20              19      0.6131\n",
            "ndcg_cut_30              19      0.6131\n",
            "ndcg_cut_100             19      0.6131\n",
            "ndcg_cut_200             19      0.6131\n",
            "ndcg_cut_500             19      0.6131\n",
            "ndcg_cut_1000            19      0.6131\n",
            "map_cut_5                19      0.5000\n",
            "map_cut_10               19      0.5000\n",
            "map_cut_15               19      0.5000\n",
            "map_cut_20               19      0.5000\n",
            "map_cut_30               19      0.5000\n",
            "map_cut_100              19      0.5000\n",
            "map_cut_200              19      0.5000\n",
            "map_cut_500              19      0.5000\n",
            "map_cut_1000             19      0.5000\n",
            "relative_P_5             19      0.5000\n",
            "relative_P_10            19      0.5000\n",
            "relative_P_15            19      0.5000\n",
            "relative_P_20            19      0.5000\n",
            "relative_P_30            19      0.5000\n",
            "relative_P_100           19      0.5000\n",
            "relative_P_200           19      0.5000\n",
            "relative_P_500           19      0.5000\n",
            "relative_P_1000          19      0.5000\n",
            "success_1                19      1.0000\n",
            "success_5                19      1.0000\n",
            "success_10               19      1.0000\n",
            "set_P                    19      0.2000\n",
            "set_relative_P           19      0.5000\n",
            "set_recall               19      0.5000\n",
            "set_map                  19      0.1000\n",
            "set_F                    19      0.2857\n",
            "num_nonrel_judged_ret    19      4.0000\n",
            "num_q                    22      1.0000\n",
            "num_ret                  22      107.0000\n",
            "num_rel                  22      1.0000\n",
            "num_rel_ret              22      1.0000\n",
            "map                      22      0.0500\n",
            "gm_map                   22      -2.9957\n",
            "Rprec                    22      0.0000\n",
            "bpref                    22      0.0000\n",
            "recip_rank               22      0.0500\n",
            "iprec_at_recall_0.00     22      0.0500\n",
            "iprec_at_recall_0.10     22      0.0500\n",
            "iprec_at_recall_0.20     22      0.0500\n",
            "iprec_at_recall_0.30     22      0.0500\n",
            "iprec_at_recall_0.40     22      0.0500\n",
            "iprec_at_recall_0.50     22      0.0500\n",
            "iprec_at_recall_0.60     22      0.0500\n",
            "iprec_at_recall_0.70     22      0.0500\n",
            "iprec_at_recall_0.80     22      0.0500\n",
            "iprec_at_recall_0.90     22      0.0500\n",
            "iprec_at_recall_1.00     22      0.0500\n",
            "P_5                      22      0.0000\n",
            "P_10                     22      0.0000\n",
            "P_15                     22      0.0000\n",
            "P_20                     22      0.0500\n",
            "P_30                     22      0.0333\n",
            "P_100                    22      0.0100\n",
            "P_200                    22      0.0050\n",
            "P_500                    22      0.0020\n",
            "P_1000                   22      0.0010\n",
            "relstring                22      0.0000\n",
            "recall_5                 22      0.0000\n",
            "recall_10                22      0.0000\n",
            "recall_15                22      0.0000\n",
            "recall_20                22      1.0000\n",
            "recall_30                22      1.0000\n",
            "recall_100               22      1.0000\n",
            "recall_200               22      1.0000\n",
            "recall_500               22      1.0000\n",
            "recall_1000              22      1.0000\n",
            "infAP                    22      0.0500\n",
            "gm_bpref                 22      -11.5129\n",
            "Rprec_mult_0.20          22      0.0000\n",
            "Rprec_mult_0.40          22      0.0000\n",
            "Rprec_mult_0.60          22      0.0000\n",
            "Rprec_mult_0.80          22      0.0000\n",
            "Rprec_mult_1.00          22      0.0000\n",
            "Rprec_mult_1.20          22      0.0000\n",
            "Rprec_mult_1.40          22      0.0000\n",
            "Rprec_mult_1.60          22      0.0000\n",
            "Rprec_mult_1.80          22      0.0000\n",
            "Rprec_mult_2.00          22      0.0000\n",
            "utility                  22      -105.0000\n",
            "11pt_avg                 22      0.0500\n",
            "binG                     22      0.2277\n",
            "G                        22      0.2277\n",
            "ndcg                     22      0.2277\n",
            "ndcg_rel                 22      0.2277\n",
            "Rndcg                    22      0.1138\n",
            "ndcg_cut_5               22      0.0000\n",
            "ndcg_cut_10              22      0.0000\n",
            "ndcg_cut_15              22      0.0000\n",
            "ndcg_cut_20              22      0.2277\n",
            "ndcg_cut_30              22      0.2277\n",
            "ndcg_cut_100             22      0.2277\n",
            "ndcg_cut_200             22      0.2277\n",
            "ndcg_cut_500             22      0.2277\n",
            "ndcg_cut_1000            22      0.2277\n",
            "map_cut_5                22      0.0000\n",
            "map_cut_10               22      0.0000\n",
            "map_cut_15               22      0.0000\n",
            "map_cut_20               22      0.0500\n",
            "map_cut_30               22      0.0500\n",
            "map_cut_100              22      0.0500\n",
            "map_cut_200              22      0.0500\n",
            "map_cut_500              22      0.0500\n",
            "map_cut_1000             22      0.0500\n",
            "relative_P_5             22      0.0000\n",
            "relative_P_10            22      0.0000\n",
            "relative_P_15            22      0.0000\n",
            "relative_P_20            22      1.0000\n",
            "relative_P_30            22      1.0000\n",
            "relative_P_100           22      1.0000\n",
            "relative_P_200           22      1.0000\n",
            "relative_P_500           22      1.0000\n",
            "relative_P_1000          22      1.0000\n",
            "success_1                22      0.0000\n",
            "success_5                22      0.0000\n",
            "success_10               22      0.0000\n",
            "set_P                    22      0.0093\n",
            "set_relative_P           22      1.0000\n",
            "set_recall               22      1.0000\n",
            "set_map                  22      0.0093\n",
            "set_F                    22      0.0185\n",
            "num_nonrel_judged_ret    22      72.0000\n",
            "num_q                    24      1.0000\n",
            "num_ret                  24      23.0000\n",
            "num_rel                  24      1.0000\n",
            "num_rel_ret              24      1.0000\n",
            "map                      24      1.0000\n",
            "gm_map                   24      0.0000\n",
            "Rprec                    24      1.0000\n",
            "bpref                    24      1.0000\n",
            "recip_rank               24      1.0000\n",
            "iprec_at_recall_0.00     24      1.0000\n",
            "iprec_at_recall_0.10     24      1.0000\n",
            "iprec_at_recall_0.20     24      1.0000\n",
            "iprec_at_recall_0.30     24      1.0000\n",
            "iprec_at_recall_0.40     24      1.0000\n",
            "iprec_at_recall_0.50     24      1.0000\n",
            "iprec_at_recall_0.60     24      1.0000\n",
            "iprec_at_recall_0.70     24      1.0000\n",
            "iprec_at_recall_0.80     24      1.0000\n",
            "iprec_at_recall_0.90     24      1.0000\n",
            "iprec_at_recall_1.00     24      1.0000\n",
            "P_5                      24      0.2000\n",
            "P_10                     24      0.1000\n",
            "P_15                     24      0.0667\n",
            "P_20                     24      0.0500\n",
            "P_30                     24      0.0333\n",
            "P_100                    24      0.0100\n",
            "P_200                    24      0.0050\n",
            "P_500                    24      0.0020\n",
            "P_1000                   24      0.0010\n",
            "relstring                24      0.0000\n",
            "recall_5                 24      1.0000\n",
            "recall_10                24      1.0000\n",
            "recall_15                24      1.0000\n",
            "recall_20                24      1.0000\n",
            "recall_30                24      1.0000\n",
            "recall_100               24      1.0000\n",
            "recall_200               24      1.0000\n",
            "recall_500               24      1.0000\n",
            "recall_1000              24      1.0000\n",
            "infAP                    24      1.0000\n",
            "gm_bpref                 24      0.0000\n",
            "Rprec_mult_0.20          24      1.0000\n",
            "Rprec_mult_0.40          24      1.0000\n",
            "Rprec_mult_0.60          24      1.0000\n",
            "Rprec_mult_0.80          24      1.0000\n",
            "Rprec_mult_1.00          24      1.0000\n",
            "Rprec_mult_1.20          24      0.5000\n",
            "Rprec_mult_1.40          24      0.5000\n",
            "Rprec_mult_1.60          24      0.5000\n",
            "Rprec_mult_1.80          24      0.5000\n",
            "Rprec_mult_2.00          24      0.5000\n",
            "utility                  24      -21.0000\n",
            "11pt_avg                 24      1.0000\n",
            "binG                     24      1.0000\n",
            "G                        24      1.0000\n",
            "ndcg                     24      1.0000\n",
            "ndcg_rel                 24      1.0000\n",
            "Rndcg                    24      1.0000\n",
            "ndcg_cut_5               24      1.0000\n",
            "ndcg_cut_10              24      1.0000\n",
            "ndcg_cut_15              24      1.0000\n",
            "ndcg_cut_20              24      1.0000\n",
            "ndcg_cut_30              24      1.0000\n",
            "ndcg_cut_100             24      1.0000\n",
            "ndcg_cut_200             24      1.0000\n",
            "ndcg_cut_500             24      1.0000\n",
            "ndcg_cut_1000            24      1.0000\n",
            "map_cut_5                24      1.0000\n",
            "map_cut_10               24      1.0000\n",
            "map_cut_15               24      1.0000\n",
            "map_cut_20               24      1.0000\n",
            "map_cut_30               24      1.0000\n",
            "map_cut_100              24      1.0000\n",
            "map_cut_200              24      1.0000\n",
            "map_cut_500              24      1.0000\n",
            "map_cut_1000             24      1.0000\n",
            "relative_P_5             24      1.0000\n",
            "relative_P_10            24      1.0000\n",
            "relative_P_15            24      1.0000\n",
            "relative_P_20            24      1.0000\n",
            "relative_P_30            24      1.0000\n",
            "relative_P_100           24      1.0000\n",
            "relative_P_200           24      1.0000\n",
            "relative_P_500           24      1.0000\n",
            "relative_P_1000          24      1.0000\n",
            "success_1                24      1.0000\n",
            "success_5                24      1.0000\n",
            "success_10               24      1.0000\n",
            "set_P                    24      0.0435\n",
            "set_relative_P           24      1.0000\n",
            "set_recall               24      1.0000\n",
            "set_map                  24      0.0435\n",
            "set_F                    24      0.0833\n",
            "num_nonrel_judged_ret    24      18.0000\n",
            "num_q                    26      1.0000\n",
            "num_ret                  26      52.0000\n",
            "num_rel                  26      3.0000\n",
            "num_rel_ret              26      2.0000\n",
            "map                      26      0.0698\n",
            "gm_map                   26      -2.6615\n",
            "Rprec                    26      0.0000\n",
            "bpref                    26      0.0000\n",
            "recip_rank               26      0.1429\n",
            "iprec_at_recall_0.00     26      0.1429\n",
            "iprec_at_recall_0.10     26      0.1429\n",
            "iprec_at_recall_0.20     26      0.1429\n",
            "iprec_at_recall_0.30     26      0.1429\n",
            "iprec_at_recall_0.40     26      0.0667\n",
            "iprec_at_recall_0.50     26      0.0667\n",
            "iprec_at_recall_0.60     26      0.0667\n",
            "iprec_at_recall_0.70     26      0.0667\n",
            "iprec_at_recall_0.80     26      0.0000\n",
            "iprec_at_recall_0.90     26      0.0000\n",
            "iprec_at_recall_1.00     26      0.0000\n",
            "P_5                      26      0.0000\n",
            "P_10                     26      0.1000\n",
            "P_15                     26      0.0667\n",
            "P_20                     26      0.0500\n",
            "P_30                     26      0.0667\n",
            "P_100                    26      0.0200\n",
            "P_200                    26      0.0100\n",
            "P_500                    26      0.0040\n",
            "P_1000                   26      0.0020\n",
            "relstring                26      0.0000\n",
            "recall_5                 26      0.0000\n",
            "recall_10                26      0.3333\n",
            "recall_15                26      0.3333\n",
            "recall_20                26      0.3333\n",
            "recall_30                26      0.6667\n",
            "recall_100               26      0.6667\n",
            "recall_200               26      0.6667\n",
            "recall_500               26      0.6667\n",
            "recall_1000              26      0.6667\n",
            "infAP                    26      0.0698\n",
            "gm_bpref                 26      -11.5129\n",
            "Rprec_mult_0.20          26      0.0000\n",
            "Rprec_mult_0.40          26      0.0000\n",
            "Rprec_mult_0.60          26      0.0000\n",
            "Rprec_mult_0.80          26      0.0000\n",
            "Rprec_mult_1.00          26      0.0000\n",
            "Rprec_mult_1.20          26      0.0000\n",
            "Rprec_mult_1.40          26      0.0000\n",
            "Rprec_mult_1.60          26      0.0000\n",
            "Rprec_mult_1.80          26      0.0000\n",
            "Rprec_mult_2.00          26      0.0000\n",
            "utility                  26      -48.0000\n",
            "11pt_avg                 26      0.0762\n",
            "binG                     26      0.1790\n",
            "G                        26      0.1790\n",
            "ndcg                     26      0.2511\n",
            "ndcg_rel                 26      0.2196\n",
            "Rndcg                    26      0.1256\n",
            "ndcg_cut_5               26      0.0000\n",
            "ndcg_cut_10              26      0.1564\n",
            "ndcg_cut_15              26      0.1564\n",
            "ndcg_cut_20              26      0.1564\n",
            "ndcg_cut_30              26      0.2511\n",
            "ndcg_cut_100             26      0.2511\n",
            "ndcg_cut_200             26      0.2511\n",
            "ndcg_cut_500             26      0.2511\n",
            "ndcg_cut_1000            26      0.2511\n",
            "map_cut_5                26      0.0000\n",
            "map_cut_10               26      0.0476\n",
            "map_cut_15               26      0.0476\n",
            "map_cut_20               26      0.0476\n",
            "map_cut_30               26      0.0698\n",
            "map_cut_100              26      0.0698\n",
            "map_cut_200              26      0.0698\n",
            "map_cut_500              26      0.0698\n",
            "map_cut_1000             26      0.0698\n",
            "relative_P_5             26      0.0000\n",
            "relative_P_10            26      0.3333\n",
            "relative_P_15            26      0.3333\n",
            "relative_P_20            26      0.3333\n",
            "relative_P_30            26      0.6667\n",
            "relative_P_100           26      0.6667\n",
            "relative_P_200           26      0.6667\n",
            "relative_P_500           26      0.6667\n",
            "relative_P_1000          26      0.6667\n",
            "success_1                26      0.0000\n",
            "success_5                26      0.0000\n",
            "success_10               26      1.0000\n",
            "set_P                    26      0.0385\n",
            "set_relative_P           26      0.6667\n",
            "set_recall               26      0.6667\n",
            "set_map                  26      0.0256\n",
            "set_F                    26      0.0727\n",
            "num_nonrel_judged_ret    26      36.0000\n",
            "num_q                    28      1.0000\n",
            "num_ret                  28      52.0000\n",
            "num_rel                  28      2.0000\n",
            "num_rel_ret              28      2.0000\n",
            "map                      28      0.2679\n",
            "gm_map                   28      -1.3173\n",
            "Rprec                    28      0.0000\n",
            "bpref                    28      0.0000\n",
            "recip_rank               28      0.2500\n",
            "iprec_at_recall_0.00     28      0.2857\n",
            "iprec_at_recall_0.10     28      0.2857\n",
            "iprec_at_recall_0.20     28      0.2857\n",
            "iprec_at_recall_0.30     28      0.2857\n",
            "iprec_at_recall_0.40     28      0.2857\n",
            "iprec_at_recall_0.50     28      0.2857\n",
            "iprec_at_recall_0.60     28      0.2857\n",
            "iprec_at_recall_0.70     28      0.2857\n",
            "iprec_at_recall_0.80     28      0.2857\n",
            "iprec_at_recall_0.90     28      0.2857\n",
            "iprec_at_recall_1.00     28      0.2857\n",
            "P_5                      28      0.2000\n",
            "P_10                     28      0.2000\n",
            "P_15                     28      0.1333\n",
            "P_20                     28      0.1000\n",
            "P_30                     28      0.0667\n",
            "P_100                    28      0.0200\n",
            "P_200                    28      0.0100\n",
            "P_500                    28      0.0040\n",
            "P_1000                   28      0.0020\n",
            "relstring                28      0.0000\n",
            "recall_5                 28      0.5000\n",
            "recall_10                28      1.0000\n",
            "recall_15                28      1.0000\n",
            "recall_20                28      1.0000\n",
            "recall_30                28      1.0000\n",
            "recall_100               28      1.0000\n",
            "recall_200               28      1.0000\n",
            "recall_500               28      1.0000\n",
            "recall_1000              28      1.0000\n",
            "infAP                    28      0.2679\n",
            "gm_bpref                 28      -11.5129\n",
            "Rprec_mult_0.20          28      0.0000\n",
            "Rprec_mult_0.40          28      0.0000\n",
            "Rprec_mult_0.60          28      0.0000\n",
            "Rprec_mult_0.80          28      0.0000\n",
            "Rprec_mult_1.00          28      0.0000\n",
            "Rprec_mult_1.20          28      0.0000\n",
            "Rprec_mult_1.40          28      0.0000\n",
            "Rprec_mult_1.60          28      0.2500\n",
            "Rprec_mult_1.80          28      0.2500\n",
            "Rprec_mult_2.00          28      0.2500\n",
            "utility                  28      -48.0000\n",
            "11pt_avg                 28      0.2857\n",
            "binG                     28      0.3934\n",
            "G                        28      0.3934\n",
            "ndcg                     28      0.4685\n",
            "ndcg_rel                 28      0.3663\n",
            "Rndcg                    28      0.2342\n",
            "ndcg_cut_5               28      0.2641\n",
            "ndcg_cut_10              28      0.4685\n",
            "ndcg_cut_15              28      0.4685\n",
            "ndcg_cut_20              28      0.4685\n",
            "ndcg_cut_30              28      0.4685\n",
            "ndcg_cut_100             28      0.4685\n",
            "ndcg_cut_200             28      0.4685\n",
            "ndcg_cut_500             28      0.4685\n",
            "ndcg_cut_1000            28      0.4685\n",
            "map_cut_5                28      0.1250\n",
            "map_cut_10               28      0.2679\n",
            "map_cut_15               28      0.2679\n",
            "map_cut_20               28      0.2679\n",
            "map_cut_30               28      0.2679\n",
            "map_cut_100              28      0.2679\n",
            "map_cut_200              28      0.2679\n",
            "map_cut_500              28      0.2679\n",
            "map_cut_1000             28      0.2679\n",
            "relative_P_5             28      0.5000\n",
            "relative_P_10            28      1.0000\n",
            "relative_P_15            28      1.0000\n",
            "relative_P_20            28      1.0000\n",
            "relative_P_30            28      1.0000\n",
            "relative_P_100           28      1.0000\n",
            "relative_P_200           28      1.0000\n",
            "relative_P_500           28      1.0000\n",
            "relative_P_1000          28      1.0000\n",
            "success_1                28      0.0000\n",
            "success_5                28      1.0000\n",
            "success_10               28      1.0000\n",
            "set_P                    28      0.0385\n",
            "set_relative_P           28      1.0000\n",
            "set_recall               28      1.0000\n",
            "set_map                  28      0.0385\n",
            "set_F                    28      0.0741\n",
            "num_nonrel_judged_ret    28      28.0000\n",
            "num_q                    all     15.0000\n",
            "num_ret                  all     519.0000\n",
            "num_rel                  all     35.0000\n",
            "num_rel_ret              all     16.0000\n",
            "map                      all     0.3734\n",
            "gm_map                   all     0.0403\n",
            "Rprec                    all     0.3000\n",
            "bpref                    all     0.3000\n",
            "recip_rank               all     0.4746\n",
            "iprec_at_recall_0.00     all     0.4770\n",
            "iprec_at_recall_0.10     all     0.4770\n",
            "iprec_at_recall_0.20     all     0.4770\n",
            "iprec_at_recall_0.30     all     0.4770\n",
            "iprec_at_recall_0.40     all     0.4719\n",
            "iprec_at_recall_0.50     all     0.4719\n",
            "iprec_at_recall_0.60     all     0.2819\n",
            "iprec_at_recall_0.70     all     0.2819\n",
            "iprec_at_recall_0.80     all     0.2675\n",
            "iprec_at_recall_0.90     all     0.2675\n",
            "iprec_at_recall_1.00     all     0.2675\n",
            "P_5                      all     0.1333\n",
            "P_10                     all     0.0867\n",
            "P_15                     all     0.0578\n",
            "P_20                     all     0.0500\n",
            "P_30                     all     0.0356\n",
            "P_100                    all     0.0107\n",
            "P_200                    all     0.0053\n",
            "P_500                    all     0.0021\n",
            "P_1000                   all     0.0011\n",
            "relstring                all     0.0000\n",
            "recall_5                 all     0.4667\n",
            "recall_10                all     0.5889\n",
            "recall_15                all     0.5889\n",
            "recall_20                all     0.6722\n",
            "recall_30                all     0.6944\n",
            "recall_100               all     0.6944\n",
            "recall_200               all     0.6944\n",
            "recall_500               all     0.6944\n",
            "recall_1000              all     0.6944\n",
            "infAP                    all     0.3734\n",
            "gm_bpref                 all     0.0009\n",
            "Rprec_mult_0.20          all     0.4000\n",
            "Rprec_mult_0.40          all     0.4000\n",
            "Rprec_mult_0.60          all     0.3111\n",
            "Rprec_mult_0.80          all     0.3000\n",
            "Rprec_mult_1.00          all     0.3000\n",
            "Rprec_mult_1.20          all     0.1711\n",
            "Rprec_mult_1.40          all     0.1667\n",
            "Rprec_mult_1.60          all     0.1690\n",
            "Rprec_mult_1.80          all     0.1667\n",
            "Rprec_mult_2.00          all     0.1667\n",
            "utility                  all     -32.4667\n",
            "11pt_avg                 all     0.3834\n",
            "binG                     all     0.4386\n",
            "G                        all     0.4386\n",
            "ndcg                     all     0.4746\n",
            "ndcg_rel                 all     0.5006\n",
            "Rndcg                    all     0.3994\n",
            "ndcg_cut_5               all     0.4009\n",
            "ndcg_cut_10              all     0.4472\n",
            "ndcg_cut_15              all     0.4472\n",
            "ndcg_cut_20              all     0.4683\n",
            "ndcg_cut_30              all     0.4746\n",
            "ndcg_cut_100             all     0.4746\n",
            "ndcg_cut_200             all     0.4746\n",
            "ndcg_cut_500             all     0.4746\n",
            "ndcg_cut_1000            all     0.4746\n",
            "map_cut_5                all     0.3439\n",
            "map_cut_10               all     0.3661\n",
            "map_cut_15               all     0.3661\n",
            "map_cut_20               all     0.3719\n",
            "map_cut_30               all     0.3734\n",
            "map_cut_100              all     0.3734\n",
            "map_cut_200              all     0.3734\n",
            "map_cut_500              all     0.3734\n",
            "map_cut_1000             all     0.3734\n",
            "relative_P_5             all     0.4667\n",
            "relative_P_10            all     0.5889\n",
            "relative_P_15            all     0.5889\n",
            "relative_P_20            all     0.6722\n",
            "relative_P_30            all     0.6944\n",
            "relative_P_100           all     0.6944\n",
            "relative_P_200           all     0.6944\n",
            "relative_P_500           all     0.6944\n",
            "relative_P_1000          all     0.6944\n",
            "success_1                all     0.4000\n",
            "success_5                all     0.6000\n",
            "success_10               all     0.7333\n",
            "set_P                    all     0.0422\n",
            "set_relative_P           all     0.6944\n",
            "set_recall               all     0.6944\n",
            "set_map                  all     0.0308\n",
            "set_F                    all     0.0731\n",
            "num_nonrel_judged_ret    all     332.0000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NPjJXXa_vX_7"
      },
      "source": [
        "\n",
        "(a) A clear list of all final modifications made.  \n",
        "\n",
        "*   Change each word to lower case\n",
        "*   Remove stopword\n",
        "*   Stem words using whoosh's stemmer\n",
        "*   Stem words using nltk's lancaster stemmer\n",
        "*   Change scoring parameter\n",
        "\n",
        "\n",
        "(b)  Why each modification was made – how did it help?  \n",
        "*   Some words in topic are in upper case, but the word in document may not necessarily be in upper case. Change everything to lower case can help searcher find relevant documents.\n",
        "*   Some topics have stop word 'and', this word is not informative and it appears in almost every document. By removing words like this, searcher can return result based on more informative words in query.\n",
        "*   Words in different form can mean the same thing, apply stemmer to both query and document can help searcher find relevant documents efficiently by reducing the effect of words' format. \n",
        "*   Lancaster stemmer has harsher rule of stemming so that words have same root but different in format can match now.\n",
        "*   Different corpus may have different characteristics, changing default parameter of BM25F so that we can find a better parameter set for this particular corpus. \n",
        "\n",
        "(c)  The  final  MAP  performance  that  these  modifications  attained.\n",
        "0.3734"
      ]
    }
  ]
}