diff --git a/_toc.yml b/_toc.yml
index dad8117..fd31798 100644
--- a/_toc.yml
+++ b/_toc.yml
@@ -27,6 +27,9 @@ subtrees:
     - file: ch-dask-dataframe/indexing
     - file: ch-dask-dataframe/map-partitions
     - file: ch-dask-dataframe/shuffle
+  - file: ch-dask-ml/index
+    entries:
+    - file: ch-dask-ml/distributed-training
   - file: ch-ray-core/index
     entries:
       - file: ch-ray-core/ray-intro
diff --git a/ch-dask-dataframe/shuffle.ipynb b/ch-dask-dataframe/shuffle.ipynb
index 964a9ed..9456359 100644
--- a/ch-dask-dataframe/shuffle.ipynb
+++ b/ch-dask-dataframe/shuffle.ipynb
@@ -11,9 +11,9 @@
     "\n",
     "## Shuffle 实现机制\n",
     "\n",
-    "{numref}`sec-dask-task-graph` 介绍了 Dask 主要基于 Task Graph 构建， Dask 的 Task Graph 是一个有向无环图。有向边表示下游 Partition 的输入依赖上游 Partition 的输出，任何数据移动都会在 Task Graph 上生成一条有向边。很多计算任务的 Shuffle 有大量的数据移动，有的场景下，所有数据都会打散，也意味着上游会有多条指向下游的边。这种基于 Task Graph 的 Shuffle 会使得 Task Graph 非常庞大，Task Graph 过大会使得 Dask Scheduler 的负载过重，进一步导致计算极其缓慢。如 {numref}`fig-dask-shuffle` 左侧所示，`tasks` 是基于 Task Graph 的机制，上游和下游之间建立了有向边，如果有中间层（通常因为上游流入的数据太大，需要将数据进一步切分成多个 Partition），那么中间层还会进一步增加 Task Graph 的复杂程度。\n",
+    "{numref}`sec-dask-task-graph` 介绍了 Dask 主要基于 Task Graph 构建， Dask 的 Task Graph 是一个有向无环图。有向边表示下游 Partition 的输入依赖上游 Partition 的输出，任何数据移动都会在 Task Graph 上生成一条有向边。很多计算任务的 Shuffle 有大量的数据移动，有的场景下，所有数据都会打散，也意味着上游会有多条指向下游的边。这种基于 Task Graph 的 Shuffle 会使得 Task Graph 非常庞大，Task Graph 过大会使得 Dask Scheduler 的负载过重，进一步导致计算极其缓慢。如 {numref}`fig-shuffle-tasks-p2p` 左侧所示，`tasks` 是基于 Task Graph 的机制，上游和下游之间建立了有向边，如果有中间层（通常因为上游流入的数据太大，需要将数据进一步切分成多个 Partition），那么中间层还会进一步增加 Task Graph 的复杂程度。\n",
     "\n",
-    "为解决 Task Graph 过大的问题，Dask 设计了一种点对点（Peer-to-peer）的 Shuffle 机制。如 {numref}`fig-dask-shuffle` 右侧所示，`p2p` 在 Task Graph 中引入了一个虚拟的障碍（Barrier）节点。Barrier 并不是一个真正的 Task，引入 Barrier 节点可以使 Task Graph 复杂度显著下降。\n",
+    "为解决 Task Graph 过大的问题，Dask 设计了一种点对点（Peer-to-peer）的 Shuffle 机制。如 {numref}`fig-shuffle-tasks-p2p` 右侧所示，`p2p` 在 Task Graph 中引入了一个虚拟的障碍（Barrier）节点。Barrier 并不是一个真正的 Task，引入 Barrier 节点可以使 Task Graph 复杂度显著下降。\n",
     "\n",
     "```{figure} ../img/ch-dask-dataframe/shuffle-tasks-p2p.png\n",
     "---\n",
@@ -26,7 +26,7 @@
     "目前，Dask 提供了两类 Shuffle 实现策略：单机和分布式。\n",
     "\n",
     "* 单机。如果数据大小超出了内存空间，可以将中间数据写到磁盘上。单机场景默认使用这种策略。\n",
-    "* 分布式。如 {numref}`fig-dask-shuffle` 所示，分布式场景提供了两种 Shuffle 策略，`tasks` 和 `p2p`。`tasks` 是基于 Task Graph 的 Shuffle 实现，很多场景效率比较低，会遇到刚提到的 Task Graph 过大的问题。`p2p` 基于点对点的 Shuffle 实现，Task Graph 的复杂性显著降低，性能也显著提升。Dask 会优先选择 `p2p`。\n",
+    "* 分布式。如 {numref}`fig-shuffle-tasks-p2p` 所示，分布式场景提供了两种 Shuffle 策略，`tasks` 和 `p2p`。`tasks` 是基于 Task Graph 的 Shuffle 实现，很多场景效率比较低，会遇到刚提到的 Task Graph 过大的问题。`p2p` 基于点对点的 Shuffle 实现，Task Graph 的复杂性显著降低，性能也显著提升。Dask 会优先选择 `p2p`。\n",
     "\n",
     "`dask.config.set({\"dataframe.shuffle.method\": \"p2p\"})` 对当前 Python 脚本的所有计算都使用 `p2p` 方式进行 Shuffle。也可以针对某个算子设置 Shuffle 策略，比如 `ddf.merge(shuffle_method=\"p2p\")`。\n",
     "\n",
diff --git a/ch-dask-ml/distributed-training.ipynb b/ch-dask-ml/distributed-training.ipynb
new file mode 100644
index 0000000..f70fbb3
--- /dev/null
+++ b/ch-dask-ml/distributed-training.ipynb
@@ -0,0 +1,3583 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "(sec-dask-ml-distributed-training)=\n",
+    "# 分布式机器学习\n",
+    "\n",
+    "如果训练数据量很大，Dask-ML 提供了分布式机器学习功能，可以在集群上对大数据进行训练。目前，Dask 提供了两类分布式机器学习 API：\n",
+    "\n",
+    "* scikit-learn：与 scikit-learn 的调用方式类似\n",
+    "* XGBoost 和 LightGBM：与 XGBoost 和 LightGBM 的调用方式类似\n",
+    "\n",
+    "## scikit-learn API\n",
+    "\n",
+    "基于 Dask Array、Dask DataFrame 和 Dask Delayed 提供的分布式计算能力，参考 scikit-learn，Dask-ML 对机器学习算法做了分布式的实现，比如 `dask_ml.linear_model` 中的线性回归 [`LinearRegression`](https://ml.dask.org/modules/generated/dask_ml.linear_model.LinearRegression.html)、逻辑回归 [`LogisticRegression`](https://ml.dask.org/modules/generated/dask_ml.linear_model.LogisticRegression.html)，`dask_ml.cluster` 中的 [`KMeans`](https://ml.dask.org/modules/generated/dask_ml.cluster.KMeans.html)。Dask-ML 尽量保持这些机器学习算法的使用方法与 scikit-learn 一致。\n",
+    "\n",
+    "在一个 2 节点组成的 Dask 集群上使用 `dask_ml.linear_model` 中线性模型。这个集群每个节点有 90GiB 内存，我们随机生成一个 37GiB 的数据集，并将其切分为训练集和测试集。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "tags": [
+     "hide-cell"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "%config InlineBackend.figure_format = 'svg'\n",
+    "import time\n",
+    "\n",
+    "import seaborn as sns\n",
+    "import pandas as pd\n",
+    "\n",
+    "from dask.distributed import Client, LocalCluster"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import dask_ml.datasets\n",
+    "import sklearn.linear_model\n",
+    "import dask_ml.linear_model\n",
+    "from dask_ml.model_selection import train_test_split"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "    <div style=\"width: 24px; height: 24px; background-color: #e1e1e1; border: 3px solid #9D9D9D; border-radius: 5px; position: absolute;\"> </div>\n",
+       "    <div style=\"margin-left: 48px;\">\n",
+       "        <h3 style=\"margin-bottom: 0px;\">Client</h3>\n",
+       "        <p style=\"color: #9D9D9D; margin-bottom: 0px;\">Client-ad77e682-0ae4-11ef-8730-000012e4fe80</p>\n",
+       "        <table style=\"width: 100%; text-align: left;\">\n",
+       "\n",
+       "        <tr>\n",
+       "        \n",
+       "            <td style=\"text-align: left;\"><strong>Connection method:</strong> Direct</td>\n",
+       "            <td style=\"text-align: left;\"></td>\n",
+       "        \n",
+       "        </tr>\n",
+       "\n",
+       "        \n",
+       "            <tr>\n",
+       "                <td style=\"text-align: left;\">\n",
+       "                    <strong>Dashboard: </strong> <a href=\"http://10.0.0.3:43549/status\" target=\"_blank\">http://10.0.0.3:43549/status</a>\n",
+       "                </td>\n",
+       "                <td style=\"text-align: left;\"></td>\n",
+       "            </tr>\n",
+       "        \n",
+       "\n",
+       "        </table>\n",
+       "\n",
+       "        \n",
+       "\n",
+       "        \n",
+       "            <details>\n",
+       "            <summary style=\"margin-bottom: 20px;\"><h3 style=\"display: inline;\">Scheduler Info</h3></summary>\n",
+       "            <div style=\"\">\n",
+       "    <div>\n",
+       "        <div style=\"width: 24px; height: 24px; background-color: #FFF7E5; border: 3px solid #FF6132; border-radius: 5px; position: absolute;\"> </div>\n",
+       "        <div style=\"margin-left: 48px;\">\n",
+       "            <h3 style=\"margin-bottom: 0px;\">Scheduler</h3>\n",
+       "            <p style=\"color: #9D9D9D; margin-bottom: 0px;\">Scheduler-c7851ab9-9963-4c85-b394-bb74e8e2967f</p>\n",
+       "            <table style=\"width: 100%; text-align: left;\">\n",
+       "                <tr>\n",
+       "                    <td style=\"text-align: left;\">\n",
+       "                        <strong>Comm:</strong> tcp://10.0.0.3:8786\n",
+       "                    </td>\n",
+       "                    <td style=\"text-align: left;\">\n",
+       "                        <strong>Workers:</strong> 2\n",
+       "                    </td>\n",
+       "                </tr>\n",
+       "                <tr>\n",
+       "                    <td style=\"text-align: left;\">\n",
+       "                        <strong>Dashboard:</strong> <a href=\"http://10.0.0.3:43549/status\" target=\"_blank\">http://10.0.0.3:43549/status</a>\n",
+       "                    </td>\n",
+       "                    <td style=\"text-align: left;\">\n",
+       "                        <strong>Total threads:</strong> 128\n",
+       "                    </td>\n",
+       "                </tr>\n",
+       "                <tr>\n",
+       "                    <td style=\"text-align: left;\">\n",
+       "                        <strong>Started:</strong> 5 hours ago\n",
+       "                    </td>\n",
+       "                    <td style=\"text-align: left;\">\n",
+       "                        <strong>Total memory:</strong> 180.00 GiB\n",
+       "                    </td>\n",
+       "                </tr>\n",
+       "            </table>\n",
+       "        </div>\n",
+       "    </div>\n",
+       "\n",
+       "    <details style=\"margin-left: 48px;\">\n",
+       "        <summary style=\"margin-bottom: 20px;\">\n",
+       "            <h3 style=\"display: inline;\">Workers</h3>\n",
+       "        </summary>\n",
+       "\n",
+       "        \n",
+       "        <div style=\"margin-bottom: 20px;\">\n",
+       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
+       "            <div style=\"margin-left: 48px;\">\n",
+       "            <details>\n",
+       "                <summary>\n",
+       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: tcp://10.0.0.2:46501</h4>\n",
+       "                </summary>\n",
+       "                <table style=\"width: 100%; text-align: left;\">\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Comm: </strong> tcp://10.0.0.2:46501\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Total threads: </strong> 64\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Dashboard: </strong> <a href=\"http://10.0.0.2:42539/status\" target=\"_blank\">http://10.0.0.2:42539/status</a>\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Memory: </strong> 90.00 GiB\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Nanny: </strong> tcp://10.0.0.2:40241\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\"></td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
+       "                            <strong>Local directory: </strong> /tmp/dask-scratch-space/worker-rxylv59_\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                    \n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Tasks executing: </strong> \n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Tasks in memory: </strong> \n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Tasks ready: </strong> \n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Tasks in flight: </strong>\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>CPU usage:</strong> 6.0%\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Last seen: </strong> Just now\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Memory usage: </strong> 301.68 MiB\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Spilled bytes: </strong> 0 B\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Read bytes: </strong> 572.9739612289254 B\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Write bytes: </strong> 1.71 kiB\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    \n",
+       "\n",
+       "                </table>\n",
+       "            </details>\n",
+       "            </div>\n",
+       "        </div>\n",
+       "        \n",
+       "        <div style=\"margin-bottom: 20px;\">\n",
+       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
+       "            <div style=\"margin-left: 48px;\">\n",
+       "            <details>\n",
+       "                <summary>\n",
+       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: tcp://10.0.0.3:39997</h4>\n",
+       "                </summary>\n",
+       "                <table style=\"width: 100%; text-align: left;\">\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Comm: </strong> tcp://10.0.0.3:39997\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Total threads: </strong> 64\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Dashboard: </strong> <a href=\"http://10.0.0.3:40955/status\" target=\"_blank\">http://10.0.0.3:40955/status</a>\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Memory: </strong> 90.00 GiB\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Nanny: </strong> tcp://10.0.0.3:34825\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\"></td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
+       "                            <strong>Local directory: </strong> /tmp/dask-scratch-space/worker-kdphx4zv\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                    \n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Tasks executing: </strong> \n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Tasks in memory: </strong> \n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Tasks ready: </strong> \n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Tasks in flight: </strong>\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>CPU usage:</strong> 4.0%\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Last seen: </strong> Just now\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Memory usage: </strong> 300.18 MiB\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Spilled bytes: </strong> 0 B\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Read bytes: </strong> 8.27 kiB\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Write bytes: </strong> 10.57 kiB\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    \n",
+       "\n",
+       "                </table>\n",
+       "            </details>\n",
+       "            </div>\n",
+       "        </div>\n",
+       "        \n",
+       "\n",
+       "    </details>\n",
+       "</div>\n",
+       "            </details>\n",
+       "        \n",
+       "\n",
+       "    </div>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "<Client: 'tcp://10.0.0.3:8786' processes=2 threads=128, memory=180.00 GiB>"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "client = Client(\"10.0.0.3:8786\")\n",
+    "client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/fs/fast/u20200002/envs/dispy/lib/python3.11/site-packages/dask/base.py:1462: UserWarning: Running on a single-machine scheduler when a distributed client is active might lead to unexpected results.\n",
+      "  warnings.warn(\n"
+     ]
+    },
+    {
+     "data": {
+      "text/html": [
+       "<table>\n",
+       "    <tr>\n",
+       "        <td>\n",
+       "            <table style=\"border-collapse: collapse;\">\n",
+       "                <thead>\n",
+       "                    <tr>\n",
+       "                        <td> </td>\n",
+       "                        <th> Array </th>\n",
+       "                        <th> Chunk </th>\n",
+       "                    </tr>\n",
+       "                </thead>\n",
+       "                <tbody>\n",
+       "                    \n",
+       "                    <tr>\n",
+       "                        <th> Bytes </th>\n",
+       "                        <td> 37.25 GiB </td>\n",
+       "                        <td> 381.47 MiB </td>\n",
+       "                    </tr>\n",
+       "                    \n",
+       "                    <tr>\n",
+       "                        <th> Shape </th>\n",
+       "                        <td> (10000000, 500) </td>\n",
+       "                        <td> (100000, 500) </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <th> Dask graph </th>\n",
+       "                        <td colspan=\"2\"> 100 chunks in 1 graph layer </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <th> Data type </th>\n",
+       "                        <td colspan=\"2\"> float64 numpy.ndarray </td>\n",
+       "                    </tr>\n",
+       "                </tbody>\n",
+       "            </table>\n",
+       "        </td>\n",
+       "        <td>\n",
+       "        <svg width=\"75\" height=\"170\" style=\"stroke:rgb(0,0,0);stroke-width:1\" >\n",
+       "\n",
+       "  <!-- Horizontal lines -->\n",
+       "  <line x1=\"0\" y1=\"0\" x2=\"25\" y2=\"0\" style=\"stroke-width:2\" />\n",
+       "  <line x1=\"0\" y1=\"6\" x2=\"25\" y2=\"6\" />\n",
+       "  <line x1=\"0\" y1=\"12\" x2=\"25\" y2=\"12\" />\n",
+       "  <line x1=\"0\" y1=\"18\" x2=\"25\" y2=\"18\" />\n",
+       "  <line x1=\"0\" y1=\"25\" x2=\"25\" y2=\"25\" />\n",
+       "  <line x1=\"0\" y1=\"31\" x2=\"25\" y2=\"31\" />\n",
+       "  <line x1=\"0\" y1=\"37\" x2=\"25\" y2=\"37\" />\n",
+       "  <line x1=\"0\" y1=\"43\" x2=\"25\" y2=\"43\" />\n",
+       "  <line x1=\"0\" y1=\"50\" x2=\"25\" y2=\"50\" />\n",
+       "  <line x1=\"0\" y1=\"56\" x2=\"25\" y2=\"56\" />\n",
+       "  <line x1=\"0\" y1=\"62\" x2=\"25\" y2=\"62\" />\n",
+       "  <line x1=\"0\" y1=\"68\" x2=\"25\" y2=\"68\" />\n",
+       "  <line x1=\"0\" y1=\"75\" x2=\"25\" y2=\"75\" />\n",
+       "  <line x1=\"0\" y1=\"81\" x2=\"25\" y2=\"81\" />\n",
+       "  <line x1=\"0\" y1=\"87\" x2=\"25\" y2=\"87\" />\n",
+       "  <line x1=\"0\" y1=\"93\" x2=\"25\" y2=\"93\" />\n",
+       "  <line x1=\"0\" y1=\"100\" x2=\"25\" y2=\"100\" />\n",
+       "  <line x1=\"0\" y1=\"106\" x2=\"25\" y2=\"106\" />\n",
+       "  <line x1=\"0\" y1=\"112\" x2=\"25\" y2=\"112\" />\n",
+       "  <line x1=\"0\" y1=\"120\" x2=\"25\" y2=\"120\" style=\"stroke-width:2\" />\n",
+       "\n",
+       "  <!-- Vertical lines -->\n",
+       "  <line x1=\"0\" y1=\"0\" x2=\"0\" y2=\"120\" style=\"stroke-width:2\" />\n",
+       "  <line x1=\"25\" y1=\"0\" x2=\"25\" y2=\"120\" style=\"stroke-width:2\" />\n",
+       "\n",
+       "  <!-- Colored Rectangle -->\n",
+       "  <polygon points=\"0.0,0.0 25.412616514582485,0.0 25.412616514582485,120.0 0.0,120.0\" style=\"fill:#8B4903A0;stroke-width:0\"/>\n",
+       "\n",
+       "  <!-- Text -->\n",
+       "  <text x=\"12.706308\" y=\"140.000000\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >500</text>\n",
+       "  <text x=\"45.412617\" y=\"60.000000\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(-90,45.412617,60.000000)\">10000000</text>\n",
+       "</svg>\n",
+       "        </td>\n",
+       "    </tr>\n",
+       "</table>"
+      ],
+      "text/plain": [
+       "dask.array<normal, shape=(10000000, 500), dtype=float64, chunksize=(100000, 500), chunktype=numpy.ndarray>"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "X, y = dask_ml.datasets.make_classification(n_samples=10_000_000, \n",
+    "        n_features=500, \n",
+    "        random_state=42,\n",
+    "        chunks=10_000_000 // 100\n",
+    ")\n",
+    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)\n",
+    "X"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "调用 `fit` 方法（与 scikit-learn 类似）："
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/fs/fast/u20200002/envs/dispy/lib/python3.11/site-packages/dask/config.py:789: FutureWarning: Dask configuration key 'fuse_ave_width' has been deprecated; please use 'optimization.fuse.ave-width' instead\n",
+      "  warnings.warn(\n"
+     ]
+    }
+   ],
+   "source": [
+    "lr = dask_ml.linear_model.LogisticRegression(solver=\"lbfgs\").fit(X_train, y_train)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "训练好的模型可以用来预测（`predict`），也可以计算准确度（`score`）。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([ True, False,  True,  True,  True])"
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "y_predicted = lr.predict(X_test)\n",
+    "y_predicted[:5].compute()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "0.668674"
+      ]
+     },
+     "execution_count": 14,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "lr.score(X_test, y_test).compute()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "如果在单机的 scikit-learn 上使用同样大小的数据训练模型，会因为内存不足而报错。\n",
+    "\n",
+    "尽管 Dask-ML 这种分布式训练的 API 与 scikit-learn 极其相似，scikit-learn 只能使用单核，Dask-ML 可以使用多核甚至集群，但并不意味着所有场景下都选择 Dask-ML，因为有些时候 Dask-ML 并非性能或性价比最优的选择。这一点与 Dask DataFrame 和 pandas 关系一样，如果数据量能放进单机内存，原生的 pandas 、NumPy 和 scikit-learn 的性能和兼容性总是最优的。\n",
+    "\n",
+    "下面的代码对不同规模的训练数据进行了性能分析，在单机多核且数据量较小的场景，Dask-ML 的性能并不比 scikit-learn 更快。主要因为：很多机器学习算法是迭代式的，scikit-learn 中，迭代式算法使用 Python 原生 `for` 循环来实现；Dask-ML 参考了这种 `for` 循环，但对于 Dask 的 Task Graph 来说，`for` 循环会使得 Task Graph 很臃肿，执行效率并不是很高。\n",
+    "\n",
+    "你也可以根据你所拥有的内存来测试一下性能。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "    <div style=\"width: 24px; height: 24px; background-color: #e1e1e1; border: 3px solid #9D9D9D; border-radius: 5px; position: absolute;\"> </div>\n",
+       "    <div style=\"margin-left: 48px;\">\n",
+       "        <h3 style=\"margin-bottom: 0px;\">Client</h3>\n",
+       "        <p style=\"color: #9D9D9D; margin-bottom: 0px;\">Client-b4f64c31-0ae4-11ef-8730-000012e4fe80</p>\n",
+       "        <table style=\"width: 100%; text-align: left;\">\n",
+       "\n",
+       "        <tr>\n",
+       "        \n",
+       "            <td style=\"text-align: left;\"><strong>Connection method:</strong> Cluster object</td>\n",
+       "            <td style=\"text-align: left;\"><strong>Cluster type:</strong> distributed.LocalCluster</td>\n",
+       "        \n",
+       "        </tr>\n",
+       "\n",
+       "        \n",
+       "            <tr>\n",
+       "                <td style=\"text-align: left;\">\n",
+       "                    <strong>Dashboard: </strong> <a href=\"http://127.0.0.1:8787/status\" target=\"_blank\">http://127.0.0.1:8787/status</a>\n",
+       "                </td>\n",
+       "                <td style=\"text-align: left;\"></td>\n",
+       "            </tr>\n",
+       "        \n",
+       "\n",
+       "        </table>\n",
+       "\n",
+       "        \n",
+       "\n",
+       "        \n",
+       "            <details>\n",
+       "            <summary style=\"margin-bottom: 20px;\"><h3 style=\"display: inline;\">Cluster Info</h3></summary>\n",
+       "            <div class=\"jp-RenderedHTMLCommon jp-RenderedHTML jp-mod-trusted jp-OutputArea-output\">\n",
+       "    <div style=\"width: 24px; height: 24px; background-color: #e1e1e1; border: 3px solid #9D9D9D; border-radius: 5px; position: absolute;\">\n",
+       "    </div>\n",
+       "    <div style=\"margin-left: 48px;\">\n",
+       "        <h3 style=\"margin-bottom: 0px; margin-top: 0px;\">LocalCluster</h3>\n",
+       "        <p style=\"color: #9D9D9D; margin-bottom: 0px;\">1872fd25</p>\n",
+       "        <table style=\"width: 100%; text-align: left;\">\n",
+       "            <tr>\n",
+       "                <td style=\"text-align: left;\">\n",
+       "                    <strong>Dashboard:</strong> <a href=\"http://127.0.0.1:8787/status\" target=\"_blank\">http://127.0.0.1:8787/status</a>\n",
+       "                </td>\n",
+       "                <td style=\"text-align: left;\">\n",
+       "                    <strong>Workers:</strong> 8\n",
+       "                </td>\n",
+       "            </tr>\n",
+       "            <tr>\n",
+       "                <td style=\"text-align: left;\">\n",
+       "                    <strong>Total threads:</strong> 64\n",
+       "                </td>\n",
+       "                <td style=\"text-align: left;\">\n",
+       "                    <strong>Total memory:</strong> 90.00 GiB\n",
+       "                </td>\n",
+       "            </tr>\n",
+       "            \n",
+       "            <tr>\n",
+       "    <td style=\"text-align: left;\"><strong>Status:</strong> running</td>\n",
+       "    <td style=\"text-align: left;\"><strong>Using processes:</strong> True</td>\n",
+       "</tr>\n",
+       "\n",
+       "            \n",
+       "        </table>\n",
+       "\n",
+       "        <details>\n",
+       "            <summary style=\"margin-bottom: 20px;\">\n",
+       "                <h3 style=\"display: inline;\">Scheduler Info</h3>\n",
+       "            </summary>\n",
+       "\n",
+       "            <div style=\"\">\n",
+       "    <div>\n",
+       "        <div style=\"width: 24px; height: 24px; background-color: #FFF7E5; border: 3px solid #FF6132; border-radius: 5px; position: absolute;\"> </div>\n",
+       "        <div style=\"margin-left: 48px;\">\n",
+       "            <h3 style=\"margin-bottom: 0px;\">Scheduler</h3>\n",
+       "            <p style=\"color: #9D9D9D; margin-bottom: 0px;\">Scheduler-897dca6c-6012-4df7-9a10-bd08f8810617</p>\n",
+       "            <table style=\"width: 100%; text-align: left;\">\n",
+       "                <tr>\n",
+       "                    <td style=\"text-align: left;\">\n",
+       "                        <strong>Comm:</strong> tcp://127.0.0.1:38477\n",
+       "                    </td>\n",
+       "                    <td style=\"text-align: left;\">\n",
+       "                        <strong>Workers:</strong> 8\n",
+       "                    </td>\n",
+       "                </tr>\n",
+       "                <tr>\n",
+       "                    <td style=\"text-align: left;\">\n",
+       "                        <strong>Dashboard:</strong> <a href=\"http://127.0.0.1:8787/status\" target=\"_blank\">http://127.0.0.1:8787/status</a>\n",
+       "                    </td>\n",
+       "                    <td style=\"text-align: left;\">\n",
+       "                        <strong>Total threads:</strong> 64\n",
+       "                    </td>\n",
+       "                </tr>\n",
+       "                <tr>\n",
+       "                    <td style=\"text-align: left;\">\n",
+       "                        <strong>Started:</strong> Just now\n",
+       "                    </td>\n",
+       "                    <td style=\"text-align: left;\">\n",
+       "                        <strong>Total memory:</strong> 90.00 GiB\n",
+       "                    </td>\n",
+       "                </tr>\n",
+       "            </table>\n",
+       "        </div>\n",
+       "    </div>\n",
+       "\n",
+       "    <details style=\"margin-left: 48px;\">\n",
+       "        <summary style=\"margin-bottom: 20px;\">\n",
+       "            <h3 style=\"display: inline;\">Workers</h3>\n",
+       "        </summary>\n",
+       "\n",
+       "        \n",
+       "        <div style=\"margin-bottom: 20px;\">\n",
+       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
+       "            <div style=\"margin-left: 48px;\">\n",
+       "            <details>\n",
+       "                <summary>\n",
+       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: 0</h4>\n",
+       "                </summary>\n",
+       "                <table style=\"width: 100%; text-align: left;\">\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Comm: </strong> tcp://127.0.0.1:44219\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Total threads: </strong> 8\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Dashboard: </strong> <a href=\"http://127.0.0.1:36081/status\" target=\"_blank\">http://127.0.0.1:36081/status</a>\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Memory: </strong> 11.25 GiB\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Nanny: </strong> tcp://127.0.0.1:34355\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\"></td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
+       "                            <strong>Local directory: </strong> /tmp/dask-scratch-space/worker-439c1uaa\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                </table>\n",
+       "            </details>\n",
+       "            </div>\n",
+       "        </div>\n",
+       "        \n",
+       "        <div style=\"margin-bottom: 20px;\">\n",
+       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
+       "            <div style=\"margin-left: 48px;\">\n",
+       "            <details>\n",
+       "                <summary>\n",
+       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: 1</h4>\n",
+       "                </summary>\n",
+       "                <table style=\"width: 100%; text-align: left;\">\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Comm: </strong> tcp://127.0.0.1:41549\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Total threads: </strong> 8\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Dashboard: </strong> <a href=\"http://127.0.0.1:44857/status\" target=\"_blank\">http://127.0.0.1:44857/status</a>\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Memory: </strong> 11.25 GiB\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Nanny: </strong> tcp://127.0.0.1:41265\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\"></td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
+       "                            <strong>Local directory: </strong> /tmp/dask-scratch-space/worker-hyxlvh30\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                </table>\n",
+       "            </details>\n",
+       "            </div>\n",
+       "        </div>\n",
+       "        \n",
+       "        <div style=\"margin-bottom: 20px;\">\n",
+       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
+       "            <div style=\"margin-left: 48px;\">\n",
+       "            <details>\n",
+       "                <summary>\n",
+       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: 2</h4>\n",
+       "                </summary>\n",
+       "                <table style=\"width: 100%; text-align: left;\">\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Comm: </strong> tcp://127.0.0.1:42877\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Total threads: </strong> 8\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Dashboard: </strong> <a href=\"http://127.0.0.1:40235/status\" target=\"_blank\">http://127.0.0.1:40235/status</a>\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Memory: </strong> 11.25 GiB\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Nanny: </strong> tcp://127.0.0.1:40939\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\"></td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
+       "                            <strong>Local directory: </strong> /tmp/dask-scratch-space/worker-e70v3hq2\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                </table>\n",
+       "            </details>\n",
+       "            </div>\n",
+       "        </div>\n",
+       "        \n",
+       "        <div style=\"margin-bottom: 20px;\">\n",
+       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
+       "            <div style=\"margin-left: 48px;\">\n",
+       "            <details>\n",
+       "                <summary>\n",
+       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: 3</h4>\n",
+       "                </summary>\n",
+       "                <table style=\"width: 100%; text-align: left;\">\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Comm: </strong> tcp://127.0.0.1:34321\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Total threads: </strong> 8\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Dashboard: </strong> <a href=\"http://127.0.0.1:40295/status\" target=\"_blank\">http://127.0.0.1:40295/status</a>\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Memory: </strong> 11.25 GiB\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Nanny: </strong> tcp://127.0.0.1:35007\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\"></td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
+       "                            <strong>Local directory: </strong> /tmp/dask-scratch-space/worker-udlmb2zo\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                </table>\n",
+       "            </details>\n",
+       "            </div>\n",
+       "        </div>\n",
+       "        \n",
+       "        <div style=\"margin-bottom: 20px;\">\n",
+       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
+       "            <div style=\"margin-left: 48px;\">\n",
+       "            <details>\n",
+       "                <summary>\n",
+       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: 4</h4>\n",
+       "                </summary>\n",
+       "                <table style=\"width: 100%; text-align: left;\">\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Comm: </strong> tcp://127.0.0.1:36039\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Total threads: </strong> 8\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Dashboard: </strong> <a href=\"http://127.0.0.1:45691/status\" target=\"_blank\">http://127.0.0.1:45691/status</a>\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Memory: </strong> 11.25 GiB\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Nanny: </strong> tcp://127.0.0.1:34883\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\"></td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
+       "                            <strong>Local directory: </strong> /tmp/dask-scratch-space/worker-g5h5ob4b\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                </table>\n",
+       "            </details>\n",
+       "            </div>\n",
+       "        </div>\n",
+       "        \n",
+       "        <div style=\"margin-bottom: 20px;\">\n",
+       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
+       "            <div style=\"margin-left: 48px;\">\n",
+       "            <details>\n",
+       "                <summary>\n",
+       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: 5</h4>\n",
+       "                </summary>\n",
+       "                <table style=\"width: 100%; text-align: left;\">\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Comm: </strong> tcp://127.0.0.1:35057\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Total threads: </strong> 8\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Dashboard: </strong> <a href=\"http://127.0.0.1:43309/status\" target=\"_blank\">http://127.0.0.1:43309/status</a>\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Memory: </strong> 11.25 GiB\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Nanny: </strong> tcp://127.0.0.1:43945\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\"></td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
+       "                            <strong>Local directory: </strong> /tmp/dask-scratch-space/worker-61hsl1ap\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                </table>\n",
+       "            </details>\n",
+       "            </div>\n",
+       "        </div>\n",
+       "        \n",
+       "        <div style=\"margin-bottom: 20px;\">\n",
+       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
+       "            <div style=\"margin-left: 48px;\">\n",
+       "            <details>\n",
+       "                <summary>\n",
+       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: 6</h4>\n",
+       "                </summary>\n",
+       "                <table style=\"width: 100%; text-align: left;\">\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Comm: </strong> tcp://127.0.0.1:36811\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Total threads: </strong> 8\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Dashboard: </strong> <a href=\"http://127.0.0.1:44197/status\" target=\"_blank\">http://127.0.0.1:44197/status</a>\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Memory: </strong> 11.25 GiB\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Nanny: </strong> tcp://127.0.0.1:44607\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\"></td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
+       "                            <strong>Local directory: </strong> /tmp/dask-scratch-space/worker-syjczr8e\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                </table>\n",
+       "            </details>\n",
+       "            </div>\n",
+       "        </div>\n",
+       "        \n",
+       "        <div style=\"margin-bottom: 20px;\">\n",
+       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
+       "            <div style=\"margin-left: 48px;\">\n",
+       "            <details>\n",
+       "                <summary>\n",
+       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: 7</h4>\n",
+       "                </summary>\n",
+       "                <table style=\"width: 100%; text-align: left;\">\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Comm: </strong> tcp://127.0.0.1:42081\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Total threads: </strong> 8\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Dashboard: </strong> <a href=\"http://127.0.0.1:35819/status\" target=\"_blank\">http://127.0.0.1:35819/status</a>\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Memory: </strong> 11.25 GiB\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Nanny: </strong> tcp://127.0.0.1:33971\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\"></td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
+       "                            <strong>Local directory: </strong> /tmp/dask-scratch-space/worker-1rw7_3km\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                </table>\n",
+       "            </details>\n",
+       "            </div>\n",
+       "        </div>\n",
+       "        \n",
+       "\n",
+       "    </details>\n",
+       "</div>\n",
+       "\n",
+       "        </details>\n",
+       "    </div>\n",
+       "</div>\n",
+       "            </details>\n",
+       "        \n",
+       "\n",
+       "    </div>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "<Client: 'tcp://127.0.0.1:38477' processes=8 threads=64, memory=90.00 GiB>"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "client = Client(LocalCluster())\n",
+    "client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/fs/fast/u20200002/envs/dispy/lib/python3.11/site-packages/dask/base.py:1462: UserWarning: Running on a single-machine scheduler when a distributed client is active might lead to unexpected results.\n",
+      "  warnings.warn(\n",
+      "/fs/fast/u20200002/envs/dispy/lib/python3.11/site-packages/dask/config.py:789: FutureWarning: Dask configuration key 'fuse_ave_width' has been deprecated; please use 'optimization.fuse.ave-width' instead\n",
+      "  warnings.warn(\n",
+      "/fs/fast/u20200002/envs/dispy/lib/python3.11/site-packages/dask/base.py:1462: UserWarning: Running on a single-machine scheduler when a distributed client is active might lead to unexpected results.\n",
+      "  warnings.warn(\n",
+      "/fs/fast/u20200002/envs/dispy/lib/python3.11/site-packages/dask/config.py:789: FutureWarning: Dask configuration key 'fuse_ave_width' has been deprecated; please use 'optimization.fuse.ave-width' instead\n",
+      "  warnings.warn(\n",
+      "/fs/fast/u20200002/envs/dispy/lib/python3.11/site-packages/dask/base.py:1462: UserWarning: Running on a single-machine scheduler when a distributed client is active might lead to unexpected results.\n",
+      "  warnings.warn(\n",
+      "/fs/fast/u20200002/envs/dispy/lib/python3.11/site-packages/dask/config.py:789: FutureWarning: Dask configuration key 'fuse_ave_width' has been deprecated; please use 'optimization.fuse.ave-width' instead\n",
+      "  warnings.warn(\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "<Axes: xlabel='# of samples', ylabel='time'>"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    },
+    {
+     "data": {
+      "image/svg+xml": [
+       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
+       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
+       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
+       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"404.923125pt\" height=\"310.86825pt\" viewBox=\"0 0 404.923125 310.86825\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
+       " <metadata>\n",
+       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
+       "   <cc:Work>\n",
+       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
+       "    <dc:date>2024-05-05T21:42:17.956299</dc:date>\n",
+       "    <dc:format>image/svg+xml</dc:format>\n",
+       "    <dc:creator>\n",
+       "     <cc:Agent>\n",
+       "      <dc:title>Matplotlib v3.8.4, https://matplotlib.org/</dc:title>\n",
+       "     </cc:Agent>\n",
+       "    </dc:creator>\n",
+       "   </cc:Work>\n",
+       "  </rdf:RDF>\n",
+       " </metadata>\n",
+       " <defs>\n",
+       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
+       " </defs>\n",
+       " <g id=\"figure_1\">\n",
+       "  <g id=\"patch_1\">\n",
+       "   <path d=\"M 0 310.86825 \n",
+       "L 404.923125 310.86825 \n",
+       "L 404.923125 0 \n",
+       "L 0 0 \n",
+       "z\n",
+       "\" style=\"fill: #ffffff\"/>\n",
+       "  </g>\n",
+       "  <g id=\"axes_1\">\n",
+       "   <g id=\"patch_2\">\n",
+       "    <path d=\"M 40.603125 273.312 \n",
+       "L 397.723125 273.312 \n",
+       "L 397.723125 7.2 \n",
+       "L 40.603125 7.2 \n",
+       "z\n",
+       "\" style=\"fill: #ffffff\"/>\n",
+       "   </g>\n",
+       "   <g id=\"patch_3\">\n",
+       "    <path d=\"M 52.507125 273.312 \n",
+       "L 100.123125 273.312 \n",
+       "L 100.123125 221.64475 \n",
+       "L 52.507125 221.64475 \n",
+       "z\n",
+       "\" clip-path=\"url(#pa659405b0a)\" style=\"fill: #3274a1\"/>\n",
+       "   </g>\n",
+       "   <g id=\"patch_4\">\n",
+       "    <path d=\"M 171.547125 273.312 \n",
+       "L 219.163125 273.312 \n",
+       "L 219.163125 181.551416 \n",
+       "L 171.547125 181.551416 \n",
+       "z\n",
+       "\" clip-path=\"url(#pa659405b0a)\" style=\"fill: #3274a1\"/>\n",
+       "   </g>\n",
+       "   <g id=\"patch_5\">\n",
+       "    <path d=\"M 290.587125 273.312 \n",
+       "L 338.203125 273.312 \n",
+       "L 338.203125 137.589854 \n",
+       "L 290.587125 137.589854 \n",
+       "z\n",
+       "\" clip-path=\"url(#pa659405b0a)\" style=\"fill: #3274a1\"/>\n",
+       "   </g>\n",
+       "   <g id=\"patch_6\">\n",
+       "    <path d=\"M 100.123125 273.312 \n",
+       "L 147.739125 273.312 \n",
+       "L 147.739125 178.425449 \n",
+       "L 100.123125 178.425449 \n",
+       "z\n",
+       "\" clip-path=\"url(#pa659405b0a)\" style=\"fill: #e1812c\"/>\n",
+       "   </g>\n",
+       "   <g id=\"patch_7\">\n",
+       "    <path d=\"M 219.163125 273.312 \n",
+       "L 266.779125 273.312 \n",
+       "L 266.779125 106.288398 \n",
+       "L 219.163125 106.288398 \n",
+       "z\n",
+       "\" clip-path=\"url(#pa659405b0a)\" style=\"fill: #e1812c\"/>\n",
+       "   </g>\n",
+       "   <g id=\"patch_8\">\n",
+       "    <path d=\"M 338.203125 273.312 \n",
+       "L 385.819125 273.312 \n",
+       "L 385.819125 19.872 \n",
+       "L 338.203125 19.872 \n",
+       "z\n",
+       "\" clip-path=\"url(#pa659405b0a)\" style=\"fill: #e1812c\"/>\n",
+       "   </g>\n",
+       "   <g id=\"patch_9\">\n",
+       "    <path d=\"M 100.123125 273.312 \n",
+       "L 100.123125 273.312 \n",
+       "L 100.123125 273.312 \n",
+       "L 100.123125 273.312 \n",
+       "z\n",
+       "\" clip-path=\"url(#pa659405b0a)\" style=\"fill: #3274a1\"/>\n",
+       "   </g>\n",
+       "   <g id=\"patch_10\">\n",
+       "    <path d=\"M 100.123125 273.312 \n",
+       "L 100.123125 273.312 \n",
+       "L 100.123125 273.312 \n",
+       "L 100.123125 273.312 \n",
+       "z\n",
+       "\" clip-path=\"url(#pa659405b0a)\" style=\"fill: #e1812c\"/>\n",
+       "   </g>\n",
+       "   <g id=\"matplotlib.axis_1\">\n",
+       "    <g id=\"xtick_1\">\n",
+       "     <g id=\"line2d_1\">\n",
+       "      <defs>\n",
+       "       <path id=\"m59f7e4950e\" d=\"M 0 0 \n",
+       "L 0 3.5 \n",
+       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
+       "      </defs>\n",
+       "      <g>\n",
+       "       <use xlink:href=\"#m59f7e4950e\" x=\"100.123125\" y=\"273.312\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "     <g id=\"text_1\">\n",
+       "      <!-- 500000 -->\n",
+       "      <g transform=\"translate(81.035625 287.910437) scale(0.1 -0.1)\">\n",
+       "       <defs>\n",
+       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
+       "L 3169 4666 \n",
+       "L 3169 4134 \n",
+       "L 1269 4134 \n",
+       "L 1269 2991 \n",
+       "Q 1406 3038 1543 3061 \n",
+       "Q 1681 3084 1819 3084 \n",
+       "Q 2600 3084 3056 2656 \n",
+       "Q 3513 2228 3513 1497 \n",
+       "Q 3513 744 3044 326 \n",
+       "Q 2575 -91 1722 -91 \n",
+       "Q 1428 -91 1123 -41 \n",
+       "Q 819 9 494 109 \n",
+       "L 494 744 \n",
+       "Q 775 591 1075 516 \n",
+       "Q 1375 441 1709 441 \n",
+       "Q 2250 441 2565 725 \n",
+       "Q 2881 1009 2881 1497 \n",
+       "Q 2881 1984 2565 2268 \n",
+       "Q 2250 2553 1709 2553 \n",
+       "Q 1456 2553 1204 2497 \n",
+       "Q 953 2441 691 2322 \n",
+       "L 691 4666 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
+       "Q 1547 4250 1301 3770 \n",
+       "Q 1056 3291 1056 2328 \n",
+       "Q 1056 1369 1301 889 \n",
+       "Q 1547 409 2034 409 \n",
+       "Q 2525 409 2770 889 \n",
+       "Q 3016 1369 3016 2328 \n",
+       "Q 3016 3291 2770 3770 \n",
+       "Q 2525 4250 2034 4250 \n",
+       "z\n",
+       "M 2034 4750 \n",
+       "Q 2819 4750 3233 4129 \n",
+       "Q 3647 3509 3647 2328 \n",
+       "Q 3647 1150 3233 529 \n",
+       "Q 2819 -91 2034 -91 \n",
+       "Q 1250 -91 836 529 \n",
+       "Q 422 1150 422 2328 \n",
+       "Q 422 3509 836 4129 \n",
+       "Q 1250 4750 2034 4750 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       </defs>\n",
+       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"190.869141\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"254.492188\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"318.115234\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "    <g id=\"xtick_2\">\n",
+       "     <g id=\"line2d_2\">\n",
+       "      <g>\n",
+       "       <use xlink:href=\"#m59f7e4950e\" x=\"219.163125\" y=\"273.312\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "     <g id=\"text_2\">\n",
+       "      <!-- 1000000 -->\n",
+       "      <g transform=\"translate(196.894375 287.910437) scale(0.1 -0.1)\">\n",
+       "       <defs>\n",
+       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
+       "L 1825 531 \n",
+       "L 1825 4091 \n",
+       "L 703 3866 \n",
+       "L 703 4441 \n",
+       "L 1819 4666 \n",
+       "L 2450 4666 \n",
+       "L 2450 531 \n",
+       "L 3481 531 \n",
+       "L 3481 0 \n",
+       "L 794 0 \n",
+       "L 794 531 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       </defs>\n",
+       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"190.869141\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"254.492188\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"318.115234\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"381.738281\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "    <g id=\"xtick_3\">\n",
+       "     <g id=\"line2d_3\">\n",
+       "      <g>\n",
+       "       <use xlink:href=\"#m59f7e4950e\" x=\"338.203125\" y=\"273.312\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "     <g id=\"text_3\">\n",
+       "      <!-- 1500000 -->\n",
+       "      <g transform=\"translate(315.934375 287.910437) scale(0.1 -0.1)\">\n",
+       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"190.869141\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"254.492188\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"318.115234\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"381.738281\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "    <g id=\"text_4\">\n",
+       "     <!-- # of samples -->\n",
+       "     <g transform=\"translate(186.192031 301.588562) scale(0.1 -0.1)\">\n",
+       "      <defs>\n",
+       "       <path id=\"DejaVuSans-23\" d=\"M 3272 2816 \n",
+       "L 2363 2816 \n",
+       "L 2100 1772 \n",
+       "L 3016 1772 \n",
+       "L 3272 2816 \n",
+       "z\n",
+       "M 2803 4594 \n",
+       "L 2478 3297 \n",
+       "L 3391 3297 \n",
+       "L 3719 4594 \n",
+       "L 4219 4594 \n",
+       "L 3897 3297 \n",
+       "L 4872 3297 \n",
+       "L 4872 2816 \n",
+       "L 3775 2816 \n",
+       "L 3519 1772 \n",
+       "L 4513 1772 \n",
+       "L 4513 1294 \n",
+       "L 3397 1294 \n",
+       "L 3072 0 \n",
+       "L 2572 0 \n",
+       "L 2894 1294 \n",
+       "L 1978 1294 \n",
+       "L 1656 0 \n",
+       "L 1153 0 \n",
+       "L 1478 1294 \n",
+       "L 494 1294 \n",
+       "L 494 1772 \n",
+       "L 1594 1772 \n",
+       "L 1856 2816 \n",
+       "L 850 2816 \n",
+       "L 850 3297 \n",
+       "L 1978 3297 \n",
+       "L 2297 4594 \n",
+       "L 2803 4594 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
+       "Q 1497 3097 1228 2736 \n",
+       "Q 959 2375 959 1747 \n",
+       "Q 959 1119 1226 758 \n",
+       "Q 1494 397 1959 397 \n",
+       "Q 2419 397 2687 759 \n",
+       "Q 2956 1122 2956 1747 \n",
+       "Q 2956 2369 2687 2733 \n",
+       "Q 2419 3097 1959 3097 \n",
+       "z\n",
+       "M 1959 3584 \n",
+       "Q 2709 3584 3137 3096 \n",
+       "Q 3566 2609 3566 1747 \n",
+       "Q 3566 888 3137 398 \n",
+       "Q 2709 -91 1959 -91 \n",
+       "Q 1206 -91 779 398 \n",
+       "Q 353 888 353 1747 \n",
+       "Q 353 2609 779 3096 \n",
+       "Q 1206 3584 1959 3584 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \n",
+       "L 2375 4384 \n",
+       "L 1825 4384 \n",
+       "Q 1516 4384 1395 4259 \n",
+       "Q 1275 4134 1275 3809 \n",
+       "L 1275 3500 \n",
+       "L 2222 3500 \n",
+       "L 2222 3053 \n",
+       "L 1275 3053 \n",
+       "L 1275 0 \n",
+       "L 697 0 \n",
+       "L 697 3053 \n",
+       "L 147 3053 \n",
+       "L 147 3500 \n",
+       "L 697 3500 \n",
+       "L 697 3744 \n",
+       "Q 697 4328 969 4595 \n",
+       "Q 1241 4863 1831 4863 \n",
+       "L 2375 4863 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
+       "L 2834 2853 \n",
+       "Q 2591 2978 2328 3040 \n",
+       "Q 2066 3103 1784 3103 \n",
+       "Q 1356 3103 1142 2972 \n",
+       "Q 928 2841 928 2578 \n",
+       "Q 928 2378 1081 2264 \n",
+       "Q 1234 2150 1697 2047 \n",
+       "L 1894 2003 \n",
+       "Q 2506 1872 2764 1633 \n",
+       "Q 3022 1394 3022 966 \n",
+       "Q 3022 478 2636 193 \n",
+       "Q 2250 -91 1575 -91 \n",
+       "Q 1294 -91 989 -36 \n",
+       "Q 684 19 347 128 \n",
+       "L 347 722 \n",
+       "Q 666 556 975 473 \n",
+       "Q 1284 391 1588 391 \n",
+       "Q 1994 391 2212 530 \n",
+       "Q 2431 669 2431 922 \n",
+       "Q 2431 1156 2273 1281 \n",
+       "Q 2116 1406 1581 1522 \n",
+       "L 1381 1569 \n",
+       "Q 847 1681 609 1914 \n",
+       "Q 372 2147 372 2553 \n",
+       "Q 372 3047 722 3315 \n",
+       "Q 1072 3584 1716 3584 \n",
+       "Q 2034 3584 2315 3537 \n",
+       "Q 2597 3491 2834 3397 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
+       "Q 1497 1759 1228 1600 \n",
+       "Q 959 1441 959 1056 \n",
+       "Q 959 750 1161 570 \n",
+       "Q 1363 391 1709 391 \n",
+       "Q 2188 391 2477 730 \n",
+       "Q 2766 1069 2766 1631 \n",
+       "L 2766 1759 \n",
+       "L 2194 1759 \n",
+       "z\n",
+       "M 3341 1997 \n",
+       "L 3341 0 \n",
+       "L 2766 0 \n",
+       "L 2766 531 \n",
+       "Q 2569 213 2275 61 \n",
+       "Q 1981 -91 1556 -91 \n",
+       "Q 1019 -91 701 211 \n",
+       "Q 384 513 384 1019 \n",
+       "Q 384 1609 779 1909 \n",
+       "Q 1175 2209 1959 2209 \n",
+       "L 2766 2209 \n",
+       "L 2766 2266 \n",
+       "Q 2766 2663 2505 2880 \n",
+       "Q 2244 3097 1772 3097 \n",
+       "Q 1472 3097 1187 3025 \n",
+       "Q 903 2953 641 2809 \n",
+       "L 641 3341 \n",
+       "Q 956 3463 1253 3523 \n",
+       "Q 1550 3584 1831 3584 \n",
+       "Q 2591 3584 2966 3190 \n",
+       "Q 3341 2797 3341 1997 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
+       "Q 3544 3216 3844 3400 \n",
+       "Q 4144 3584 4550 3584 \n",
+       "Q 5097 3584 5394 3201 \n",
+       "Q 5691 2819 5691 2113 \n",
+       "L 5691 0 \n",
+       "L 5113 0 \n",
+       "L 5113 2094 \n",
+       "Q 5113 2597 4934 2840 \n",
+       "Q 4756 3084 4391 3084 \n",
+       "Q 3944 3084 3684 2787 \n",
+       "Q 3425 2491 3425 1978 \n",
+       "L 3425 0 \n",
+       "L 2847 0 \n",
+       "L 2847 2094 \n",
+       "Q 2847 2600 2669 2842 \n",
+       "Q 2491 3084 2119 3084 \n",
+       "Q 1678 3084 1418 2786 \n",
+       "Q 1159 2488 1159 1978 \n",
+       "L 1159 0 \n",
+       "L 581 0 \n",
+       "L 581 3500 \n",
+       "L 1159 3500 \n",
+       "L 1159 2956 \n",
+       "Q 1356 3278 1631 3431 \n",
+       "Q 1906 3584 2284 3584 \n",
+       "Q 2666 3584 2933 3390 \n",
+       "Q 3200 3197 3328 2828 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
+       "L 1159 -1331 \n",
+       "L 581 -1331 \n",
+       "L 581 3500 \n",
+       "L 1159 3500 \n",
+       "L 1159 2969 \n",
+       "Q 1341 3281 1617 3432 \n",
+       "Q 1894 3584 2278 3584 \n",
+       "Q 2916 3584 3314 3078 \n",
+       "Q 3713 2572 3713 1747 \n",
+       "Q 3713 922 3314 415 \n",
+       "Q 2916 -91 2278 -91 \n",
+       "Q 1894 -91 1617 61 \n",
+       "Q 1341 213 1159 525 \n",
+       "z\n",
+       "M 3116 1747 \n",
+       "Q 3116 2381 2855 2742 \n",
+       "Q 2594 3103 2138 3103 \n",
+       "Q 1681 3103 1420 2742 \n",
+       "Q 1159 2381 1159 1747 \n",
+       "Q 1159 1113 1420 752 \n",
+       "Q 1681 391 2138 391 \n",
+       "Q 2594 391 2855 752 \n",
+       "Q 3116 1113 3116 1747 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
+       "L 1178 4863 \n",
+       "L 1178 0 \n",
+       "L 603 0 \n",
+       "L 603 4863 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
+       "L 3597 1613 \n",
+       "L 953 1613 \n",
+       "Q 991 1019 1311 708 \n",
+       "Q 1631 397 2203 397 \n",
+       "Q 2534 397 2845 478 \n",
+       "Q 3156 559 3463 722 \n",
+       "L 3463 178 \n",
+       "Q 3153 47 2828 -22 \n",
+       "Q 2503 -91 2169 -91 \n",
+       "Q 1331 -91 842 396 \n",
+       "Q 353 884 353 1716 \n",
+       "Q 353 2575 817 3079 \n",
+       "Q 1281 3584 2069 3584 \n",
+       "Q 2775 3584 3186 3129 \n",
+       "Q 3597 2675 3597 1894 \n",
+       "z\n",
+       "M 3022 2063 \n",
+       "Q 3016 2534 2758 2815 \n",
+       "Q 2500 3097 2075 3097 \n",
+       "Q 1594 3097 1305 2825 \n",
+       "Q 1016 2553 972 2059 \n",
+       "L 3022 2063 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "      </defs>\n",
+       "      <use xlink:href=\"#DejaVuSans-23\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-20\" x=\"83.789062\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"115.576172\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-66\" x=\"176.757812\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-20\" x=\"211.962891\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-73\" x=\"243.75\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-61\" x=\"295.849609\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"357.128906\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-70\" x=\"454.541016\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"518.017578\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-65\" x=\"545.800781\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-73\" x=\"607.324219\"/>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "   </g>\n",
+       "   <g id=\"matplotlib.axis_2\">\n",
+       "    <g id=\"ytick_1\">\n",
+       "     <g id=\"line2d_4\">\n",
+       "      <defs>\n",
+       "       <path id=\"m8a0f117d1b\" d=\"M 0 0 \n",
+       "L -3.5 0 \n",
+       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
+       "      </defs>\n",
+       "      <g>\n",
+       "       <use xlink:href=\"#m8a0f117d1b\" x=\"40.603125\" y=\"273.312\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "     <g id=\"text_5\">\n",
+       "      <!-- 0 -->\n",
+       "      <g transform=\"translate(27.240625 277.111219) scale(0.1 -0.1)\">\n",
+       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "    <g id=\"ytick_2\">\n",
+       "     <g id=\"line2d_5\">\n",
+       "      <g>\n",
+       "       <use xlink:href=\"#m8a0f117d1b\" x=\"40.603125\" y=\"237.289211\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "     <g id=\"text_6\">\n",
+       "      <!-- 10 -->\n",
+       "      <g transform=\"translate(20.878125 241.08843) scale(0.1 -0.1)\">\n",
+       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "    <g id=\"ytick_3\">\n",
+       "     <g id=\"line2d_6\">\n",
+       "      <g>\n",
+       "       <use xlink:href=\"#m8a0f117d1b\" x=\"40.603125\" y=\"201.266423\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "     <g id=\"text_7\">\n",
+       "      <!-- 20 -->\n",
+       "      <g transform=\"translate(20.878125 205.065642) scale(0.1 -0.1)\">\n",
+       "       <defs>\n",
+       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
+       "L 3431 531 \n",
+       "L 3431 0 \n",
+       "L 469 0 \n",
+       "L 469 531 \n",
+       "Q 828 903 1448 1529 \n",
+       "Q 2069 2156 2228 2338 \n",
+       "Q 2531 2678 2651 2914 \n",
+       "Q 2772 3150 2772 3378 \n",
+       "Q 2772 3750 2511 3984 \n",
+       "Q 2250 4219 1831 4219 \n",
+       "Q 1534 4219 1204 4116 \n",
+       "Q 875 4013 500 3803 \n",
+       "L 500 4441 \n",
+       "Q 881 4594 1212 4672 \n",
+       "Q 1544 4750 1819 4750 \n",
+       "Q 2544 4750 2975 4387 \n",
+       "Q 3406 4025 3406 3419 \n",
+       "Q 3406 3131 3298 2873 \n",
+       "Q 3191 2616 2906 2266 \n",
+       "Q 2828 2175 2409 1742 \n",
+       "Q 1991 1309 1228 531 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       </defs>\n",
+       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "    <g id=\"ytick_4\">\n",
+       "     <g id=\"line2d_7\">\n",
+       "      <g>\n",
+       "       <use xlink:href=\"#m8a0f117d1b\" x=\"40.603125\" y=\"165.243634\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "     <g id=\"text_8\">\n",
+       "      <!-- 30 -->\n",
+       "      <g transform=\"translate(20.878125 169.042853) scale(0.1 -0.1)\">\n",
+       "       <defs>\n",
+       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
+       "Q 3050 2419 3304 2112 \n",
+       "Q 3559 1806 3559 1356 \n",
+       "Q 3559 666 3084 287 \n",
+       "Q 2609 -91 1734 -91 \n",
+       "Q 1441 -91 1130 -33 \n",
+       "Q 819 25 488 141 \n",
+       "L 488 750 \n",
+       "Q 750 597 1062 519 \n",
+       "Q 1375 441 1716 441 \n",
+       "Q 2309 441 2620 675 \n",
+       "Q 2931 909 2931 1356 \n",
+       "Q 2931 1769 2642 2001 \n",
+       "Q 2353 2234 1838 2234 \n",
+       "L 1294 2234 \n",
+       "L 1294 2753 \n",
+       "L 1863 2753 \n",
+       "Q 2328 2753 2575 2939 \n",
+       "Q 2822 3125 2822 3475 \n",
+       "Q 2822 3834 2567 4026 \n",
+       "Q 2313 4219 1838 4219 \n",
+       "Q 1578 4219 1281 4162 \n",
+       "Q 984 4106 628 3988 \n",
+       "L 628 4550 \n",
+       "Q 988 4650 1302 4700 \n",
+       "Q 1616 4750 1894 4750 \n",
+       "Q 2613 4750 3031 4423 \n",
+       "Q 3450 4097 3450 3541 \n",
+       "Q 3450 3153 3228 2886 \n",
+       "Q 3006 2619 2597 2516 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       </defs>\n",
+       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "    <g id=\"ytick_5\">\n",
+       "     <g id=\"line2d_8\">\n",
+       "      <g>\n",
+       "       <use xlink:href=\"#m8a0f117d1b\" x=\"40.603125\" y=\"129.220846\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "     <g id=\"text_9\">\n",
+       "      <!-- 40 -->\n",
+       "      <g transform=\"translate(20.878125 133.020065) scale(0.1 -0.1)\">\n",
+       "       <defs>\n",
+       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
+       "L 825 1625 \n",
+       "L 2419 1625 \n",
+       "L 2419 4116 \n",
+       "z\n",
+       "M 2253 4666 \n",
+       "L 3047 4666 \n",
+       "L 3047 1625 \n",
+       "L 3713 1625 \n",
+       "L 3713 1100 \n",
+       "L 3047 1100 \n",
+       "L 3047 0 \n",
+       "L 2419 0 \n",
+       "L 2419 1100 \n",
+       "L 313 1100 \n",
+       "L 313 1709 \n",
+       "L 2253 4666 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       </defs>\n",
+       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "    <g id=\"ytick_6\">\n",
+       "     <g id=\"line2d_9\">\n",
+       "      <g>\n",
+       "       <use xlink:href=\"#m8a0f117d1b\" x=\"40.603125\" y=\"93.198057\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "     <g id=\"text_10\">\n",
+       "      <!-- 50 -->\n",
+       "      <g transform=\"translate(20.878125 96.997276) scale(0.1 -0.1)\">\n",
+       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "    <g id=\"ytick_7\">\n",
+       "     <g id=\"line2d_10\">\n",
+       "      <g>\n",
+       "       <use xlink:href=\"#m8a0f117d1b\" x=\"40.603125\" y=\"57.175269\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "     <g id=\"text_11\">\n",
+       "      <!-- 60 -->\n",
+       "      <g transform=\"translate(20.878125 60.974488) scale(0.1 -0.1)\">\n",
+       "       <defs>\n",
+       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
+       "Q 1688 2584 1439 2293 \n",
+       "Q 1191 2003 1191 1497 \n",
+       "Q 1191 994 1439 701 \n",
+       "Q 1688 409 2113 409 \n",
+       "Q 2538 409 2786 701 \n",
+       "Q 3034 994 3034 1497 \n",
+       "Q 3034 2003 2786 2293 \n",
+       "Q 2538 2584 2113 2584 \n",
+       "z\n",
+       "M 3366 4563 \n",
+       "L 3366 3988 \n",
+       "Q 3128 4100 2886 4159 \n",
+       "Q 2644 4219 2406 4219 \n",
+       "Q 1781 4219 1451 3797 \n",
+       "Q 1122 3375 1075 2522 \n",
+       "Q 1259 2794 1537 2939 \n",
+       "Q 1816 3084 2150 3084 \n",
+       "Q 2853 3084 3261 2657 \n",
+       "Q 3669 2231 3669 1497 \n",
+       "Q 3669 778 3244 343 \n",
+       "Q 2819 -91 2113 -91 \n",
+       "Q 1303 -91 875 529 \n",
+       "Q 447 1150 447 2328 \n",
+       "Q 447 3434 972 4092 \n",
+       "Q 1497 4750 2381 4750 \n",
+       "Q 2619 4750 2861 4703 \n",
+       "Q 3103 4656 3366 4563 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       </defs>\n",
+       "       <use xlink:href=\"#DejaVuSans-36\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "    <g id=\"ytick_8\">\n",
+       "     <g id=\"line2d_11\">\n",
+       "      <g>\n",
+       "       <use xlink:href=\"#m8a0f117d1b\" x=\"40.603125\" y=\"21.15248\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "     <g id=\"text_12\">\n",
+       "      <!-- 70 -->\n",
+       "      <g transform=\"translate(20.878125 24.951699) scale(0.1 -0.1)\">\n",
+       "       <defs>\n",
+       "        <path id=\"DejaVuSans-37\" d=\"M 525 4666 \n",
+       "L 3525 4666 \n",
+       "L 3525 4397 \n",
+       "L 1831 0 \n",
+       "L 1172 0 \n",
+       "L 2766 4134 \n",
+       "L 525 4134 \n",
+       "L 525 4666 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       </defs>\n",
+       "       <use xlink:href=\"#DejaVuSans-37\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "    <g id=\"text_13\">\n",
+       "     <!-- time -->\n",
+       "     <g transform=\"translate(14.798438 151.552094) rotate(-90) scale(0.1 -0.1)\">\n",
+       "      <defs>\n",
+       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
+       "L 1172 3500 \n",
+       "L 2356 3500 \n",
+       "L 2356 3053 \n",
+       "L 1172 3053 \n",
+       "L 1172 1153 \n",
+       "Q 1172 725 1289 603 \n",
+       "Q 1406 481 1766 481 \n",
+       "L 2356 481 \n",
+       "L 2356 0 \n",
+       "L 1766 0 \n",
+       "Q 1100 0 847 248 \n",
+       "Q 594 497 594 1153 \n",
+       "L 594 3053 \n",
+       "L 172 3053 \n",
+       "L 172 3500 \n",
+       "L 594 3500 \n",
+       "L 594 4494 \n",
+       "L 1172 4494 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
+       "L 1178 3500 \n",
+       "L 1178 0 \n",
+       "L 603 0 \n",
+       "L 603 3500 \n",
+       "z\n",
+       "M 603 4863 \n",
+       "L 1178 4863 \n",
+       "L 1178 4134 \n",
+       "L 603 4134 \n",
+       "L 603 4863 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "      </defs>\n",
+       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-69\" x=\"39.208984\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"66.992188\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-65\" x=\"164.404297\"/>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "   </g>\n",
+       "   <g id=\"line2d_12\">\n",
+       "    <path clip-path=\"url(#pa659405b0a)\" style=\"fill: none; stroke: #424242; stroke-width: 2.25; stroke-linecap: square\"/>\n",
+       "   </g>\n",
+       "   <g id=\"line2d_13\">\n",
+       "    <path clip-path=\"url(#pa659405b0a)\" style=\"fill: none; stroke: #424242; stroke-width: 2.25; stroke-linecap: square\"/>\n",
+       "   </g>\n",
+       "   <g id=\"line2d_14\">\n",
+       "    <path clip-path=\"url(#pa659405b0a)\" style=\"fill: none; stroke: #424242; stroke-width: 2.25; stroke-linecap: square\"/>\n",
+       "   </g>\n",
+       "   <g id=\"line2d_15\">\n",
+       "    <path clip-path=\"url(#pa659405b0a)\" style=\"fill: none; stroke: #424242; stroke-width: 2.25; stroke-linecap: square\"/>\n",
+       "   </g>\n",
+       "   <g id=\"line2d_16\">\n",
+       "    <path clip-path=\"url(#pa659405b0a)\" style=\"fill: none; stroke: #424242; stroke-width: 2.25; stroke-linecap: square\"/>\n",
+       "   </g>\n",
+       "   <g id=\"line2d_17\">\n",
+       "    <path clip-path=\"url(#pa659405b0a)\" style=\"fill: none; stroke: #424242; stroke-width: 2.25; stroke-linecap: square\"/>\n",
+       "   </g>\n",
+       "   <g id=\"patch_11\">\n",
+       "    <path d=\"M 40.603125 273.312 \n",
+       "L 40.603125 7.2 \n",
+       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
+       "   </g>\n",
+       "   <g id=\"patch_12\">\n",
+       "    <path d=\"M 397.723125 273.312 \n",
+       "L 397.723125 7.2 \n",
+       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
+       "   </g>\n",
+       "   <g id=\"patch_13\">\n",
+       "    <path d=\"M 40.603125 273.312 \n",
+       "L 397.723125 273.312 \n",
+       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
+       "   </g>\n",
+       "   <g id=\"patch_14\">\n",
+       "    <path d=\"M 40.603125 7.2 \n",
+       "L 397.723125 7.2 \n",
+       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
+       "   </g>\n",
+       "   <g id=\"legend_1\">\n",
+       "    <g id=\"patch_15\">\n",
+       "     <path d=\"M 47.603125 59.234375 \n",
+       "L 134.51875 59.234375 \n",
+       "Q 136.51875 59.234375 136.51875 57.234375 \n",
+       "L 136.51875 14.2 \n",
+       "Q 136.51875 12.2 134.51875 12.2 \n",
+       "L 47.603125 12.2 \n",
+       "Q 45.603125 12.2 45.603125 14.2 \n",
+       "L 45.603125 57.234375 \n",
+       "Q 45.603125 59.234375 47.603125 59.234375 \n",
+       "z\n",
+       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
+       "    </g>\n",
+       "    <g id=\"text_14\">\n",
+       "     <!-- method -->\n",
+       "     <g transform=\"translate(71.751563 23.798437) scale(0.1 -0.1)\">\n",
+       "      <defs>\n",
+       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
+       "L 3513 0 \n",
+       "L 2938 0 \n",
+       "L 2938 2094 \n",
+       "Q 2938 2591 2744 2837 \n",
+       "Q 2550 3084 2163 3084 \n",
+       "Q 1697 3084 1428 2787 \n",
+       "Q 1159 2491 1159 1978 \n",
+       "L 1159 0 \n",
+       "L 581 0 \n",
+       "L 581 4863 \n",
+       "L 1159 4863 \n",
+       "L 1159 2956 \n",
+       "Q 1366 3272 1645 3428 \n",
+       "Q 1925 3584 2291 3584 \n",
+       "Q 2894 3584 3203 3211 \n",
+       "Q 3513 2838 3513 2113 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \n",
+       "L 2906 4863 \n",
+       "L 3481 4863 \n",
+       "L 3481 0 \n",
+       "L 2906 0 \n",
+       "L 2906 525 \n",
+       "Q 2725 213 2448 61 \n",
+       "Q 2172 -91 1784 -91 \n",
+       "Q 1150 -91 751 415 \n",
+       "Q 353 922 353 1747 \n",
+       "Q 353 2572 751 3078 \n",
+       "Q 1150 3584 1784 3584 \n",
+       "Q 2172 3584 2448 3432 \n",
+       "Q 2725 3281 2906 2969 \n",
+       "z\n",
+       "M 947 1747 \n",
+       "Q 947 1113 1208 752 \n",
+       "Q 1469 391 1925 391 \n",
+       "Q 2381 391 2643 752 \n",
+       "Q 2906 1113 2906 1747 \n",
+       "Q 2906 2381 2643 2742 \n",
+       "Q 2381 3103 1925 3103 \n",
+       "Q 1469 3103 1208 2742 \n",
+       "Q 947 2381 947 1747 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "      </defs>\n",
+       "      <use xlink:href=\"#DejaVuSans-6d\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-65\" x=\"97.412109\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-74\" x=\"158.935547\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-68\" x=\"198.144531\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"261.523438\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-64\" x=\"322.705078\"/>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "    <g id=\"patch_16\">\n",
+       "     <path d=\"M 49.603125 38.476562 \n",
+       "L 69.603125 38.476562 \n",
+       "L 69.603125 31.476562 \n",
+       "L 49.603125 31.476562 \n",
+       "z\n",
+       "\" style=\"fill: #3274a1\"/>\n",
+       "    </g>\n",
+       "    <g id=\"text_15\">\n",
+       "     <!-- scikit-learn -->\n",
+       "     <g transform=\"translate(77.603125 38.476562) scale(0.1 -0.1)\">\n",
+       "      <defs>\n",
+       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
+       "L 3122 2828 \n",
+       "Q 2878 2963 2633 3030 \n",
+       "Q 2388 3097 2138 3097 \n",
+       "Q 1578 3097 1268 2742 \n",
+       "Q 959 2388 959 1747 \n",
+       "Q 959 1106 1268 751 \n",
+       "Q 1578 397 2138 397 \n",
+       "Q 2388 397 2633 464 \n",
+       "Q 2878 531 3122 666 \n",
+       "L 3122 134 \n",
+       "Q 2881 22 2623 -34 \n",
+       "Q 2366 -91 2075 -91 \n",
+       "Q 1284 -91 818 406 \n",
+       "Q 353 903 353 1747 \n",
+       "Q 353 2603 823 3093 \n",
+       "Q 1294 3584 2113 3584 \n",
+       "Q 2378 3584 2631 3529 \n",
+       "Q 2884 3475 3122 3366 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-6b\" d=\"M 581 4863 \n",
+       "L 1159 4863 \n",
+       "L 1159 1991 \n",
+       "L 2875 3500 \n",
+       "L 3609 3500 \n",
+       "L 1753 1863 \n",
+       "L 3688 0 \n",
+       "L 2938 0 \n",
+       "L 1159 1709 \n",
+       "L 1159 0 \n",
+       "L 581 0 \n",
+       "L 581 4863 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-2d\" d=\"M 313 2009 \n",
+       "L 1997 2009 \n",
+       "L 1997 1497 \n",
+       "L 313 1497 \n",
+       "L 313 2009 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
+       "Q 2534 3019 2420 3045 \n",
+       "Q 2306 3072 2169 3072 \n",
+       "Q 1681 3072 1420 2755 \n",
+       "Q 1159 2438 1159 1844 \n",
+       "L 1159 0 \n",
+       "L 581 0 \n",
+       "L 581 3500 \n",
+       "L 1159 3500 \n",
+       "L 1159 2956 \n",
+       "Q 1341 3275 1631 3429 \n",
+       "Q 1922 3584 2338 3584 \n",
+       "Q 2397 3584 2469 3576 \n",
+       "Q 2541 3569 2628 3553 \n",
+       "L 2631 2963 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
+       "L 3513 0 \n",
+       "L 2938 0 \n",
+       "L 2938 2094 \n",
+       "Q 2938 2591 2744 2837 \n",
+       "Q 2550 3084 2163 3084 \n",
+       "Q 1697 3084 1428 2787 \n",
+       "Q 1159 2491 1159 1978 \n",
+       "L 1159 0 \n",
+       "L 581 0 \n",
+       "L 581 3500 \n",
+       "L 1159 3500 \n",
+       "L 1159 2956 \n",
+       "Q 1366 3272 1645 3428 \n",
+       "Q 1925 3584 2291 3584 \n",
+       "Q 2894 3584 3203 3211 \n",
+       "Q 3513 2838 3513 2113 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "      </defs>\n",
+       "      <use xlink:href=\"#DejaVuSans-73\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-63\" x=\"52.099609\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-69\" x=\"107.080078\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-6b\" x=\"134.863281\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-69\" x=\"192.773438\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-74\" x=\"220.556641\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-2d\" x=\"259.765625\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"295.849609\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-65\" x=\"323.632812\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-61\" x=\"385.15625\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-72\" x=\"446.435547\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"485.798828\"/>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "    <g id=\"patch_17\">\n",
+       "     <path d=\"M 49.603125 53.154688 \n",
+       "L 69.603125 53.154688 \n",
+       "L 69.603125 46.154688 \n",
+       "L 49.603125 46.154688 \n",
+       "z\n",
+       "\" style=\"fill: #e1812c\"/>\n",
+       "    </g>\n",
+       "    <g id=\"text_16\">\n",
+       "     <!-- dask-ml -->\n",
+       "     <g transform=\"translate(77.603125 53.154688) scale(0.1 -0.1)\">\n",
+       "      <use xlink:href=\"#DejaVuSans-64\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-61\" x=\"63.476562\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-73\" x=\"124.755859\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-6b\" x=\"176.855469\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-2d\" x=\"234.765625\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"270.849609\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"368.261719\"/>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "   </g>\n",
+       "  </g>\n",
+       " </g>\n",
+       " <defs>\n",
+       "  <clipPath id=\"pa659405b0a\">\n",
+       "   <rect x=\"40.603125\" y=\"7.2\" width=\"357.12\" height=\"266.112\"/>\n",
+       "  </clipPath>\n",
+       " </defs>\n",
+       "</svg>\n"
+      ],
+      "text/plain": [
+       "<Figure size 640x480 with 1 Axes>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "num_sample = [500_000, 1_000_000, 1_500_000]\n",
+    "num_feature = 1_000\n",
+    "timings = []\n",
+    "\n",
+    "for n in num_sample:\n",
+    "    X, y = dask_ml.datasets.make_classification(n_samples=n, \n",
+    "        n_features=num_feature, \n",
+    "        random_state=42,\n",
+    "        chunks=n // 10\n",
+    "    )\n",
+    "    t1 = time.time()\n",
+    "    sklearn.linear_model.LogisticRegression(solver=\"lbfgs\").fit(X, y)\n",
+    "    timings.append(('scikit-learn', n, time.time() - t1))\n",
+    "    t1 = time.time()\n",
+    "    dask_ml.linear_model.LogisticRegression(solver=\"lbfgs\").fit(X, y)\n",
+    "    timings.append(('dask-ml', n, time.time() - t1))\n",
+    "\n",
+    "df = pd.DataFrame(timings, columns=['method', '# of samples', 'time'])\n",
+    "sns.barplot(data=df, x='# of samples', y='time', hue='method')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "可见，在逻辑回归这个场景上，比起 scikit-learn，Dask-ML 在单机多核上并无太多优势。而且很多传统机器学习算法对训练数据量的要求没那么高，随着训练数据的增加，传统的机器学习算法的性能不会显著增加。训练数据量和模型性能之间的关系可以通过学习曲线（Learning Curves）来可视化，随着训练数据量增加，像朴素贝叶斯等算法的性能提升十分有限。如果一些机器学习算法无法进行分布式训练或分布式训练成本很高，可以考虑对训练数据采样，数据大小能够放进单机内存，使用 scikit-learn 这种单机框架训练。\n",
+    "\n",
+    "综上，如果有一个超出单机内存的训练数据，要根据问题特点、所使用的算法和成本等多方面因素来决定使用何种方式处理。\n",
+    "\n",
+    "## XGBoost 和 LightGBM\n",
+    "\n",
+    "XGBoost 和 LightGBM 是两种决策树模型的实现，他们本身就对分布式训练友好，且集成了 Dask 的分布式能力。下面以 XGBoost 为例，介绍 XGBoost 如何基于 Dask 实现分布式训练，LightGBM 与之类似。\n",
+    "\n",
+    "在 XGBoost 中，训练一个模型既可以使用 `train` 方法，也可以使用 scikit-learn 式的 `fit` 方法。这两种方式都支持 Dask 分布式训练。\n",
+    "\n",
+    "下面的代码对单机的 XGBoost 和 Dask 分布式训练两种方式进行了性能对比。如果使用 Dask，需要将 [`xgboost.DMatrix`](https://xgboost.readthedocs.io/en/stable/python/python_api.html#xgboost.DMatrix) 修改为 [`xgboost.dask.DaskDMatrix`](https://xgboost.readthedocs.io/en/stable/python/python_api.html#xgboost.dask.DaskDMatrix)，将 [`xgboost.train`](https://xgboost.readthedocs.io/en/stable/python/python_api.html#xgboost.train) 修改为 [`xgboost.dask.train`](https://xgboost.readthedocs.io/en/stable/python/python_api.html#xgboost.dask.train)；并传入 Dask 集群客户端 `client`。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/fs/fast/u20200002/envs/dispy/lib/python3.11/site-packages/dask/base.py:1462: UserWarning: Running on a single-machine scheduler when a distributed client is active might lead to unexpected results.\n",
+      "  warnings.warn(\n",
+      "[22:13:43] task [xgboost.dask-0]:tcp://127.0.0.1:44219 got new rank 0\n",
+      "[22:13:43] task [xgboost.dask-1]:tcp://127.0.0.1:41549 got new rank 1\n",
+      "[22:13:43] task [xgboost.dask-2]:tcp://127.0.0.1:42877 got new rank 2\n",
+      "[22:13:43] task [xgboost.dask-3]:tcp://127.0.0.1:34321 got new rank 3\n",
+      "[22:13:43] task [xgboost.dask-4]:tcp://127.0.0.1:36039 got new rank 4\n",
+      "[22:13:43] task [xgboost.dask-5]:tcp://127.0.0.1:35057 got new rank 5\n",
+      "[22:13:43] task [xgboost.dask-6]:tcp://127.0.0.1:36811 got new rank 6\n",
+      "[22:13:43] task [xgboost.dask-7]:tcp://127.0.0.1:42081 got new rank 7\n",
+      "/fs/fast/u20200002/envs/dispy/lib/python3.11/site-packages/dask/base.py:1462: UserWarning: Running on a single-machine scheduler when a distributed client is active might lead to unexpected results.\n",
+      "  warnings.warn(\n",
+      "[22:16:27] task [xgboost.dask-0]:tcp://127.0.0.1:44219 got new rank 0\n",
+      "[22:16:27] task [xgboost.dask-1]:tcp://127.0.0.1:41549 got new rank 1\n",
+      "[22:16:27] task [xgboost.dask-2]:tcp://127.0.0.1:42877 got new rank 2\n",
+      "[22:16:27] task [xgboost.dask-3]:tcp://127.0.0.1:34321 got new rank 3\n",
+      "[22:16:27] task [xgboost.dask-4]:tcp://127.0.0.1:36039 got new rank 4\n",
+      "[22:16:27] task [xgboost.dask-5]:tcp://127.0.0.1:35057 got new rank 5\n",
+      "[22:16:28] task [xgboost.dask-6]:tcp://127.0.0.1:36811 got new rank 6\n",
+      "[22:16:28] task [xgboost.dask-7]:tcp://127.0.0.1:42081 got new rank 7\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "<Axes: xlabel='# of samples', ylabel='time'>"
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    },
+    {
+     "data": {
+      "image/svg+xml": [
+       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
+       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
+       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
+       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"411.285625pt\" height=\"310.86825pt\" viewBox=\"0 0 411.285625 310.86825\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
+       " <metadata>\n",
+       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
+       "   <cc:Work>\n",
+       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
+       "    <dc:date>2024-05-05T22:16:40.172826</dc:date>\n",
+       "    <dc:format>image/svg+xml</dc:format>\n",
+       "    <dc:creator>\n",
+       "     <cc:Agent>\n",
+       "      <dc:title>Matplotlib v3.8.4, https://matplotlib.org/</dc:title>\n",
+       "     </cc:Agent>\n",
+       "    </dc:creator>\n",
+       "   </cc:Work>\n",
+       "  </rdf:RDF>\n",
+       " </metadata>\n",
+       " <defs>\n",
+       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
+       " </defs>\n",
+       " <g id=\"figure_1\">\n",
+       "  <g id=\"patch_1\">\n",
+       "   <path d=\"M 0 310.86825 \n",
+       "L 411.285625 310.86825 \n",
+       "L 411.285625 0 \n",
+       "L 0 0 \n",
+       "z\n",
+       "\" style=\"fill: #ffffff\"/>\n",
+       "  </g>\n",
+       "  <g id=\"axes_1\">\n",
+       "   <g id=\"patch_2\">\n",
+       "    <path d=\"M 46.965625 273.312 \n",
+       "L 404.085625 273.312 \n",
+       "L 404.085625 7.2 \n",
+       "L 46.965625 7.2 \n",
+       "z\n",
+       "\" style=\"fill: #ffffff\"/>\n",
+       "   </g>\n",
+       "   <g id=\"patch_3\">\n",
+       "    <path d=\"M 64.821625 273.312 \n",
+       "L 136.245625 273.312 \n",
+       "L 136.245625 220.514797 \n",
+       "L 64.821625 220.514797 \n",
+       "z\n",
+       "\" clip-path=\"url(#p085c8fb7b0)\" style=\"fill: #3274a1\"/>\n",
+       "   </g>\n",
+       "   <g id=\"patch_4\">\n",
+       "    <path d=\"M 243.381625 273.312 \n",
+       "L 314.805625 273.312 \n",
+       "L 314.805625 19.872 \n",
+       "L 243.381625 19.872 \n",
+       "z\n",
+       "\" clip-path=\"url(#p085c8fb7b0)\" style=\"fill: #3274a1\"/>\n",
+       "   </g>\n",
+       "   <g id=\"patch_5\">\n",
+       "    <path d=\"M 136.245625 273.312 \n",
+       "L 207.669625 273.312 \n",
+       "L 207.669625 259.656931 \n",
+       "L 136.245625 259.656931 \n",
+       "z\n",
+       "\" clip-path=\"url(#p085c8fb7b0)\" style=\"fill: #e1812c\"/>\n",
+       "   </g>\n",
+       "   <g id=\"patch_6\">\n",
+       "    <path d=\"M 314.805625 273.312 \n",
+       "L 386.229625 273.312 \n",
+       "L 386.229625 196.491772 \n",
+       "L 314.805625 196.491772 \n",
+       "z\n",
+       "\" clip-path=\"url(#p085c8fb7b0)\" style=\"fill: #e1812c\"/>\n",
+       "   </g>\n",
+       "   <g id=\"patch_7\">\n",
+       "    <path d=\"M 136.245625 273.312 \n",
+       "L 136.245625 273.312 \n",
+       "L 136.245625 273.312 \n",
+       "L 136.245625 273.312 \n",
+       "z\n",
+       "\" clip-path=\"url(#p085c8fb7b0)\" style=\"fill: #3274a1\"/>\n",
+       "   </g>\n",
+       "   <g id=\"patch_8\">\n",
+       "    <path d=\"M 136.245625 273.312 \n",
+       "L 136.245625 273.312 \n",
+       "L 136.245625 273.312 \n",
+       "L 136.245625 273.312 \n",
+       "z\n",
+       "\" clip-path=\"url(#p085c8fb7b0)\" style=\"fill: #e1812c\"/>\n",
+       "   </g>\n",
+       "   <g id=\"matplotlib.axis_1\">\n",
+       "    <g id=\"xtick_1\">\n",
+       "     <g id=\"line2d_1\">\n",
+       "      <defs>\n",
+       "       <path id=\"m06bf63f022\" d=\"M 0 0 \n",
+       "L 0 3.5 \n",
+       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
+       "      </defs>\n",
+       "      <g>\n",
+       "       <use xlink:href=\"#m06bf63f022\" x=\"136.245625\" y=\"273.312\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "     <g id=\"text_1\">\n",
+       "      <!-- 100000 -->\n",
+       "      <g transform=\"translate(117.158125 287.910437) scale(0.1 -0.1)\">\n",
+       "       <defs>\n",
+       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
+       "L 1825 531 \n",
+       "L 1825 4091 \n",
+       "L 703 3866 \n",
+       "L 703 4441 \n",
+       "L 1819 4666 \n",
+       "L 2450 4666 \n",
+       "L 2450 531 \n",
+       "L 3481 531 \n",
+       "L 3481 0 \n",
+       "L 794 0 \n",
+       "L 794 531 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
+       "Q 1547 4250 1301 3770 \n",
+       "Q 1056 3291 1056 2328 \n",
+       "Q 1056 1369 1301 889 \n",
+       "Q 1547 409 2034 409 \n",
+       "Q 2525 409 2770 889 \n",
+       "Q 3016 1369 3016 2328 \n",
+       "Q 3016 3291 2770 3770 \n",
+       "Q 2525 4250 2034 4250 \n",
+       "z\n",
+       "M 2034 4750 \n",
+       "Q 2819 4750 3233 4129 \n",
+       "Q 3647 3509 3647 2328 \n",
+       "Q 3647 1150 3233 529 \n",
+       "Q 2819 -91 2034 -91 \n",
+       "Q 1250 -91 836 529 \n",
+       "Q 422 1150 422 2328 \n",
+       "Q 422 3509 836 4129 \n",
+       "Q 1250 4750 2034 4750 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       </defs>\n",
+       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"190.869141\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"254.492188\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"318.115234\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "    <g id=\"xtick_2\">\n",
+       "     <g id=\"line2d_2\">\n",
+       "      <g>\n",
+       "       <use xlink:href=\"#m06bf63f022\" x=\"314.805625\" y=\"273.312\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "     <g id=\"text_2\">\n",
+       "      <!-- 500000 -->\n",
+       "      <g transform=\"translate(295.718125 287.910437) scale(0.1 -0.1)\">\n",
+       "       <defs>\n",
+       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
+       "L 3169 4666 \n",
+       "L 3169 4134 \n",
+       "L 1269 4134 \n",
+       "L 1269 2991 \n",
+       "Q 1406 3038 1543 3061 \n",
+       "Q 1681 3084 1819 3084 \n",
+       "Q 2600 3084 3056 2656 \n",
+       "Q 3513 2228 3513 1497 \n",
+       "Q 3513 744 3044 326 \n",
+       "Q 2575 -91 1722 -91 \n",
+       "Q 1428 -91 1123 -41 \n",
+       "Q 819 9 494 109 \n",
+       "L 494 744 \n",
+       "Q 775 591 1075 516 \n",
+       "Q 1375 441 1709 441 \n",
+       "Q 2250 441 2565 725 \n",
+       "Q 2881 1009 2881 1497 \n",
+       "Q 2881 1984 2565 2268 \n",
+       "Q 2250 2553 1709 2553 \n",
+       "Q 1456 2553 1204 2497 \n",
+       "Q 953 2441 691 2322 \n",
+       "L 691 4666 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       </defs>\n",
+       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"190.869141\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"254.492188\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"318.115234\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "    <g id=\"text_3\">\n",
+       "     <!-- # of samples -->\n",
+       "     <g transform=\"translate(192.554531 301.588562) scale(0.1 -0.1)\">\n",
+       "      <defs>\n",
+       "       <path id=\"DejaVuSans-23\" d=\"M 3272 2816 \n",
+       "L 2363 2816 \n",
+       "L 2100 1772 \n",
+       "L 3016 1772 \n",
+       "L 3272 2816 \n",
+       "z\n",
+       "M 2803 4594 \n",
+       "L 2478 3297 \n",
+       "L 3391 3297 \n",
+       "L 3719 4594 \n",
+       "L 4219 4594 \n",
+       "L 3897 3297 \n",
+       "L 4872 3297 \n",
+       "L 4872 2816 \n",
+       "L 3775 2816 \n",
+       "L 3519 1772 \n",
+       "L 4513 1772 \n",
+       "L 4513 1294 \n",
+       "L 3397 1294 \n",
+       "L 3072 0 \n",
+       "L 2572 0 \n",
+       "L 2894 1294 \n",
+       "L 1978 1294 \n",
+       "L 1656 0 \n",
+       "L 1153 0 \n",
+       "L 1478 1294 \n",
+       "L 494 1294 \n",
+       "L 494 1772 \n",
+       "L 1594 1772 \n",
+       "L 1856 2816 \n",
+       "L 850 2816 \n",
+       "L 850 3297 \n",
+       "L 1978 3297 \n",
+       "L 2297 4594 \n",
+       "L 2803 4594 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
+       "Q 1497 3097 1228 2736 \n",
+       "Q 959 2375 959 1747 \n",
+       "Q 959 1119 1226 758 \n",
+       "Q 1494 397 1959 397 \n",
+       "Q 2419 397 2687 759 \n",
+       "Q 2956 1122 2956 1747 \n",
+       "Q 2956 2369 2687 2733 \n",
+       "Q 2419 3097 1959 3097 \n",
+       "z\n",
+       "M 1959 3584 \n",
+       "Q 2709 3584 3137 3096 \n",
+       "Q 3566 2609 3566 1747 \n",
+       "Q 3566 888 3137 398 \n",
+       "Q 2709 -91 1959 -91 \n",
+       "Q 1206 -91 779 398 \n",
+       "Q 353 888 353 1747 \n",
+       "Q 353 2609 779 3096 \n",
+       "Q 1206 3584 1959 3584 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \n",
+       "L 2375 4384 \n",
+       "L 1825 4384 \n",
+       "Q 1516 4384 1395 4259 \n",
+       "Q 1275 4134 1275 3809 \n",
+       "L 1275 3500 \n",
+       "L 2222 3500 \n",
+       "L 2222 3053 \n",
+       "L 1275 3053 \n",
+       "L 1275 0 \n",
+       "L 697 0 \n",
+       "L 697 3053 \n",
+       "L 147 3053 \n",
+       "L 147 3500 \n",
+       "L 697 3500 \n",
+       "L 697 3744 \n",
+       "Q 697 4328 969 4595 \n",
+       "Q 1241 4863 1831 4863 \n",
+       "L 2375 4863 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
+       "L 2834 2853 \n",
+       "Q 2591 2978 2328 3040 \n",
+       "Q 2066 3103 1784 3103 \n",
+       "Q 1356 3103 1142 2972 \n",
+       "Q 928 2841 928 2578 \n",
+       "Q 928 2378 1081 2264 \n",
+       "Q 1234 2150 1697 2047 \n",
+       "L 1894 2003 \n",
+       "Q 2506 1872 2764 1633 \n",
+       "Q 3022 1394 3022 966 \n",
+       "Q 3022 478 2636 193 \n",
+       "Q 2250 -91 1575 -91 \n",
+       "Q 1294 -91 989 -36 \n",
+       "Q 684 19 347 128 \n",
+       "L 347 722 \n",
+       "Q 666 556 975 473 \n",
+       "Q 1284 391 1588 391 \n",
+       "Q 1994 391 2212 530 \n",
+       "Q 2431 669 2431 922 \n",
+       "Q 2431 1156 2273 1281 \n",
+       "Q 2116 1406 1581 1522 \n",
+       "L 1381 1569 \n",
+       "Q 847 1681 609 1914 \n",
+       "Q 372 2147 372 2553 \n",
+       "Q 372 3047 722 3315 \n",
+       "Q 1072 3584 1716 3584 \n",
+       "Q 2034 3584 2315 3537 \n",
+       "Q 2597 3491 2834 3397 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
+       "Q 1497 1759 1228 1600 \n",
+       "Q 959 1441 959 1056 \n",
+       "Q 959 750 1161 570 \n",
+       "Q 1363 391 1709 391 \n",
+       "Q 2188 391 2477 730 \n",
+       "Q 2766 1069 2766 1631 \n",
+       "L 2766 1759 \n",
+       "L 2194 1759 \n",
+       "z\n",
+       "M 3341 1997 \n",
+       "L 3341 0 \n",
+       "L 2766 0 \n",
+       "L 2766 531 \n",
+       "Q 2569 213 2275 61 \n",
+       "Q 1981 -91 1556 -91 \n",
+       "Q 1019 -91 701 211 \n",
+       "Q 384 513 384 1019 \n",
+       "Q 384 1609 779 1909 \n",
+       "Q 1175 2209 1959 2209 \n",
+       "L 2766 2209 \n",
+       "L 2766 2266 \n",
+       "Q 2766 2663 2505 2880 \n",
+       "Q 2244 3097 1772 3097 \n",
+       "Q 1472 3097 1187 3025 \n",
+       "Q 903 2953 641 2809 \n",
+       "L 641 3341 \n",
+       "Q 956 3463 1253 3523 \n",
+       "Q 1550 3584 1831 3584 \n",
+       "Q 2591 3584 2966 3190 \n",
+       "Q 3341 2797 3341 1997 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
+       "Q 3544 3216 3844 3400 \n",
+       "Q 4144 3584 4550 3584 \n",
+       "Q 5097 3584 5394 3201 \n",
+       "Q 5691 2819 5691 2113 \n",
+       "L 5691 0 \n",
+       "L 5113 0 \n",
+       "L 5113 2094 \n",
+       "Q 5113 2597 4934 2840 \n",
+       "Q 4756 3084 4391 3084 \n",
+       "Q 3944 3084 3684 2787 \n",
+       "Q 3425 2491 3425 1978 \n",
+       "L 3425 0 \n",
+       "L 2847 0 \n",
+       "L 2847 2094 \n",
+       "Q 2847 2600 2669 2842 \n",
+       "Q 2491 3084 2119 3084 \n",
+       "Q 1678 3084 1418 2786 \n",
+       "Q 1159 2488 1159 1978 \n",
+       "L 1159 0 \n",
+       "L 581 0 \n",
+       "L 581 3500 \n",
+       "L 1159 3500 \n",
+       "L 1159 2956 \n",
+       "Q 1356 3278 1631 3431 \n",
+       "Q 1906 3584 2284 3584 \n",
+       "Q 2666 3584 2933 3390 \n",
+       "Q 3200 3197 3328 2828 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
+       "L 1159 -1331 \n",
+       "L 581 -1331 \n",
+       "L 581 3500 \n",
+       "L 1159 3500 \n",
+       "L 1159 2969 \n",
+       "Q 1341 3281 1617 3432 \n",
+       "Q 1894 3584 2278 3584 \n",
+       "Q 2916 3584 3314 3078 \n",
+       "Q 3713 2572 3713 1747 \n",
+       "Q 3713 922 3314 415 \n",
+       "Q 2916 -91 2278 -91 \n",
+       "Q 1894 -91 1617 61 \n",
+       "Q 1341 213 1159 525 \n",
+       "z\n",
+       "M 3116 1747 \n",
+       "Q 3116 2381 2855 2742 \n",
+       "Q 2594 3103 2138 3103 \n",
+       "Q 1681 3103 1420 2742 \n",
+       "Q 1159 2381 1159 1747 \n",
+       "Q 1159 1113 1420 752 \n",
+       "Q 1681 391 2138 391 \n",
+       "Q 2594 391 2855 752 \n",
+       "Q 3116 1113 3116 1747 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
+       "L 1178 4863 \n",
+       "L 1178 0 \n",
+       "L 603 0 \n",
+       "L 603 4863 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
+       "L 3597 1613 \n",
+       "L 953 1613 \n",
+       "Q 991 1019 1311 708 \n",
+       "Q 1631 397 2203 397 \n",
+       "Q 2534 397 2845 478 \n",
+       "Q 3156 559 3463 722 \n",
+       "L 3463 178 \n",
+       "Q 3153 47 2828 -22 \n",
+       "Q 2503 -91 2169 -91 \n",
+       "Q 1331 -91 842 396 \n",
+       "Q 353 884 353 1716 \n",
+       "Q 353 2575 817 3079 \n",
+       "Q 1281 3584 2069 3584 \n",
+       "Q 2775 3584 3186 3129 \n",
+       "Q 3597 2675 3597 1894 \n",
+       "z\n",
+       "M 3022 2063 \n",
+       "Q 3016 2534 2758 2815 \n",
+       "Q 2500 3097 2075 3097 \n",
+       "Q 1594 3097 1305 2825 \n",
+       "Q 1016 2553 972 2059 \n",
+       "L 3022 2063 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "      </defs>\n",
+       "      <use xlink:href=\"#DejaVuSans-23\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-20\" x=\"83.789062\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"115.576172\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-66\" x=\"176.757812\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-20\" x=\"211.962891\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-73\" x=\"243.75\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-61\" x=\"295.849609\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"357.128906\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-70\" x=\"454.541016\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"518.017578\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-65\" x=\"545.800781\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-73\" x=\"607.324219\"/>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "   </g>\n",
+       "   <g id=\"matplotlib.axis_2\">\n",
+       "    <g id=\"ytick_1\">\n",
+       "     <g id=\"line2d_3\">\n",
+       "      <defs>\n",
+       "       <path id=\"m5d0aed10b6\" d=\"M 0 0 \n",
+       "L -3.5 0 \n",
+       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
+       "      </defs>\n",
+       "      <g>\n",
+       "       <use xlink:href=\"#m5d0aed10b6\" x=\"46.965625\" y=\"273.312\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "     <g id=\"text_4\">\n",
+       "      <!-- 0 -->\n",
+       "      <g transform=\"translate(33.603125 277.111219) scale(0.1 -0.1)\">\n",
+       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "    <g id=\"ytick_2\">\n",
+       "     <g id=\"line2d_4\">\n",
+       "      <g>\n",
+       "       <use xlink:href=\"#m5d0aed10b6\" x=\"46.965625\" y=\"228.339493\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "     <g id=\"text_5\">\n",
+       "      <!-- 20 -->\n",
+       "      <g transform=\"translate(27.240625 232.138712) scale(0.1 -0.1)\">\n",
+       "       <defs>\n",
+       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
+       "L 3431 531 \n",
+       "L 3431 0 \n",
+       "L 469 0 \n",
+       "L 469 531 \n",
+       "Q 828 903 1448 1529 \n",
+       "Q 2069 2156 2228 2338 \n",
+       "Q 2531 2678 2651 2914 \n",
+       "Q 2772 3150 2772 3378 \n",
+       "Q 2772 3750 2511 3984 \n",
+       "Q 2250 4219 1831 4219 \n",
+       "Q 1534 4219 1204 4116 \n",
+       "Q 875 4013 500 3803 \n",
+       "L 500 4441 \n",
+       "Q 881 4594 1212 4672 \n",
+       "Q 1544 4750 1819 4750 \n",
+       "Q 2544 4750 2975 4387 \n",
+       "Q 3406 4025 3406 3419 \n",
+       "Q 3406 3131 3298 2873 \n",
+       "Q 3191 2616 2906 2266 \n",
+       "Q 2828 2175 2409 1742 \n",
+       "Q 1991 1309 1228 531 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       </defs>\n",
+       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "    <g id=\"ytick_3\">\n",
+       "     <g id=\"line2d_5\">\n",
+       "      <g>\n",
+       "       <use xlink:href=\"#m5d0aed10b6\" x=\"46.965625\" y=\"183.366986\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "     <g id=\"text_6\">\n",
+       "      <!-- 40 -->\n",
+       "      <g transform=\"translate(27.240625 187.166205) scale(0.1 -0.1)\">\n",
+       "       <defs>\n",
+       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
+       "L 825 1625 \n",
+       "L 2419 1625 \n",
+       "L 2419 4116 \n",
+       "z\n",
+       "M 2253 4666 \n",
+       "L 3047 4666 \n",
+       "L 3047 1625 \n",
+       "L 3713 1625 \n",
+       "L 3713 1100 \n",
+       "L 3047 1100 \n",
+       "L 3047 0 \n",
+       "L 2419 0 \n",
+       "L 2419 1100 \n",
+       "L 313 1100 \n",
+       "L 313 1709 \n",
+       "L 2253 4666 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       </defs>\n",
+       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "    <g id=\"ytick_4\">\n",
+       "     <g id=\"line2d_6\">\n",
+       "      <g>\n",
+       "       <use xlink:href=\"#m5d0aed10b6\" x=\"46.965625\" y=\"138.394479\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "     <g id=\"text_7\">\n",
+       "      <!-- 60 -->\n",
+       "      <g transform=\"translate(27.240625 142.193698) scale(0.1 -0.1)\">\n",
+       "       <defs>\n",
+       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
+       "Q 1688 2584 1439 2293 \n",
+       "Q 1191 2003 1191 1497 \n",
+       "Q 1191 994 1439 701 \n",
+       "Q 1688 409 2113 409 \n",
+       "Q 2538 409 2786 701 \n",
+       "Q 3034 994 3034 1497 \n",
+       "Q 3034 2003 2786 2293 \n",
+       "Q 2538 2584 2113 2584 \n",
+       "z\n",
+       "M 3366 4563 \n",
+       "L 3366 3988 \n",
+       "Q 3128 4100 2886 4159 \n",
+       "Q 2644 4219 2406 4219 \n",
+       "Q 1781 4219 1451 3797 \n",
+       "Q 1122 3375 1075 2522 \n",
+       "Q 1259 2794 1537 2939 \n",
+       "Q 1816 3084 2150 3084 \n",
+       "Q 2853 3084 3261 2657 \n",
+       "Q 3669 2231 3669 1497 \n",
+       "Q 3669 778 3244 343 \n",
+       "Q 2819 -91 2113 -91 \n",
+       "Q 1303 -91 875 529 \n",
+       "Q 447 1150 447 2328 \n",
+       "Q 447 3434 972 4092 \n",
+       "Q 1497 4750 2381 4750 \n",
+       "Q 2619 4750 2861 4703 \n",
+       "Q 3103 4656 3366 4563 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       </defs>\n",
+       "       <use xlink:href=\"#DejaVuSans-36\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "    <g id=\"ytick_5\">\n",
+       "     <g id=\"line2d_7\">\n",
+       "      <g>\n",
+       "       <use xlink:href=\"#m5d0aed10b6\" x=\"46.965625\" y=\"93.421972\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "     <g id=\"text_8\">\n",
+       "      <!-- 80 -->\n",
+       "      <g transform=\"translate(27.240625 97.221191) scale(0.1 -0.1)\">\n",
+       "       <defs>\n",
+       "        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n",
+       "Q 1584 2216 1326 1975 \n",
+       "Q 1069 1734 1069 1313 \n",
+       "Q 1069 891 1326 650 \n",
+       "Q 1584 409 2034 409 \n",
+       "Q 2484 409 2743 651 \n",
+       "Q 3003 894 3003 1313 \n",
+       "Q 3003 1734 2745 1975 \n",
+       "Q 2488 2216 2034 2216 \n",
+       "z\n",
+       "M 1403 2484 \n",
+       "Q 997 2584 770 2862 \n",
+       "Q 544 3141 544 3541 \n",
+       "Q 544 4100 942 4425 \n",
+       "Q 1341 4750 2034 4750 \n",
+       "Q 2731 4750 3128 4425 \n",
+       "Q 3525 4100 3525 3541 \n",
+       "Q 3525 3141 3298 2862 \n",
+       "Q 3072 2584 2669 2484 \n",
+       "Q 3125 2378 3379 2068 \n",
+       "Q 3634 1759 3634 1313 \n",
+       "Q 3634 634 3220 271 \n",
+       "Q 2806 -91 2034 -91 \n",
+       "Q 1263 -91 848 271 \n",
+       "Q 434 634 434 1313 \n",
+       "Q 434 1759 690 2068 \n",
+       "Q 947 2378 1403 2484 \n",
+       "z\n",
+       "M 1172 3481 \n",
+       "Q 1172 3119 1398 2916 \n",
+       "Q 1625 2713 2034 2713 \n",
+       "Q 2441 2713 2670 2916 \n",
+       "Q 2900 3119 2900 3481 \n",
+       "Q 2900 3844 2670 4047 \n",
+       "Q 2441 4250 2034 4250 \n",
+       "Q 1625 4250 1398 4047 \n",
+       "Q 1172 3844 1172 3481 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       </defs>\n",
+       "       <use xlink:href=\"#DejaVuSans-38\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "    <g id=\"ytick_6\">\n",
+       "     <g id=\"line2d_8\">\n",
+       "      <g>\n",
+       "       <use xlink:href=\"#m5d0aed10b6\" x=\"46.965625\" y=\"48.449465\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "     <g id=\"text_9\">\n",
+       "      <!-- 100 -->\n",
+       "      <g transform=\"translate(20.878125 52.248684) scale(0.1 -0.1)\">\n",
+       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
+       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
+       "      </g>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "    <g id=\"text_10\">\n",
+       "     <!-- time -->\n",
+       "     <g transform=\"translate(14.798438 151.552094) rotate(-90) scale(0.1 -0.1)\">\n",
+       "      <defs>\n",
+       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
+       "L 1172 3500 \n",
+       "L 2356 3500 \n",
+       "L 2356 3053 \n",
+       "L 1172 3053 \n",
+       "L 1172 1153 \n",
+       "Q 1172 725 1289 603 \n",
+       "Q 1406 481 1766 481 \n",
+       "L 2356 481 \n",
+       "L 2356 0 \n",
+       "L 1766 0 \n",
+       "Q 1100 0 847 248 \n",
+       "Q 594 497 594 1153 \n",
+       "L 594 3053 \n",
+       "L 172 3053 \n",
+       "L 172 3500 \n",
+       "L 594 3500 \n",
+       "L 594 4494 \n",
+       "L 1172 4494 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
+       "L 1178 3500 \n",
+       "L 1178 0 \n",
+       "L 603 0 \n",
+       "L 603 3500 \n",
+       "z\n",
+       "M 603 4863 \n",
+       "L 1178 4863 \n",
+       "L 1178 4134 \n",
+       "L 603 4134 \n",
+       "L 603 4863 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "      </defs>\n",
+       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-69\" x=\"39.208984\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"66.992188\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-65\" x=\"164.404297\"/>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "   </g>\n",
+       "   <g id=\"line2d_9\">\n",
+       "    <path clip-path=\"url(#p085c8fb7b0)\" style=\"fill: none; stroke: #424242; stroke-width: 2.25; stroke-linecap: square\"/>\n",
+       "   </g>\n",
+       "   <g id=\"line2d_10\">\n",
+       "    <path clip-path=\"url(#p085c8fb7b0)\" style=\"fill: none; stroke: #424242; stroke-width: 2.25; stroke-linecap: square\"/>\n",
+       "   </g>\n",
+       "   <g id=\"line2d_11\">\n",
+       "    <path clip-path=\"url(#p085c8fb7b0)\" style=\"fill: none; stroke: #424242; stroke-width: 2.25; stroke-linecap: square\"/>\n",
+       "   </g>\n",
+       "   <g id=\"line2d_12\">\n",
+       "    <path clip-path=\"url(#p085c8fb7b0)\" style=\"fill: none; stroke: #424242; stroke-width: 2.25; stroke-linecap: square\"/>\n",
+       "   </g>\n",
+       "   <g id=\"patch_9\">\n",
+       "    <path d=\"M 46.965625 273.312 \n",
+       "L 46.965625 7.2 \n",
+       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
+       "   </g>\n",
+       "   <g id=\"patch_10\">\n",
+       "    <path d=\"M 404.085625 273.312 \n",
+       "L 404.085625 7.2 \n",
+       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
+       "   </g>\n",
+       "   <g id=\"patch_11\">\n",
+       "    <path d=\"M 46.965625 273.312 \n",
+       "L 404.085625 273.312 \n",
+       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
+       "   </g>\n",
+       "   <g id=\"patch_12\">\n",
+       "    <path d=\"M 46.965625 7.2 \n",
+       "L 404.085625 7.2 \n",
+       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
+       "   </g>\n",
+       "   <g id=\"legend_1\">\n",
+       "    <g id=\"patch_13\">\n",
+       "     <path d=\"M 325.102813 59.234375 \n",
+       "L 397.085625 59.234375 \n",
+       "Q 399.085625 59.234375 399.085625 57.234375 \n",
+       "L 399.085625 14.2 \n",
+       "Q 399.085625 12.2 397.085625 12.2 \n",
+       "L 325.102813 12.2 \n",
+       "Q 323.102813 12.2 323.102813 14.2 \n",
+       "L 323.102813 57.234375 \n",
+       "Q 323.102813 59.234375 325.102813 59.234375 \n",
+       "z\n",
+       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
+       "    </g>\n",
+       "    <g id=\"text_11\">\n",
+       "     <!-- method -->\n",
+       "     <g transform=\"translate(341.784844 23.798437) scale(0.1 -0.1)\">\n",
+       "      <defs>\n",
+       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
+       "L 3513 0 \n",
+       "L 2938 0 \n",
+       "L 2938 2094 \n",
+       "Q 2938 2591 2744 2837 \n",
+       "Q 2550 3084 2163 3084 \n",
+       "Q 1697 3084 1428 2787 \n",
+       "Q 1159 2491 1159 1978 \n",
+       "L 1159 0 \n",
+       "L 581 0 \n",
+       "L 581 4863 \n",
+       "L 1159 4863 \n",
+       "L 1159 2956 \n",
+       "Q 1366 3272 1645 3428 \n",
+       "Q 1925 3584 2291 3584 \n",
+       "Q 2894 3584 3203 3211 \n",
+       "Q 3513 2838 3513 2113 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \n",
+       "L 2906 4863 \n",
+       "L 3481 4863 \n",
+       "L 3481 0 \n",
+       "L 2906 0 \n",
+       "L 2906 525 \n",
+       "Q 2725 213 2448 61 \n",
+       "Q 2172 -91 1784 -91 \n",
+       "Q 1150 -91 751 415 \n",
+       "Q 353 922 353 1747 \n",
+       "Q 353 2572 751 3078 \n",
+       "Q 1150 3584 1784 3584 \n",
+       "Q 2172 3584 2448 3432 \n",
+       "Q 2725 3281 2906 2969 \n",
+       "z\n",
+       "M 947 1747 \n",
+       "Q 947 1113 1208 752 \n",
+       "Q 1469 391 1925 391 \n",
+       "Q 2381 391 2643 752 \n",
+       "Q 2906 1113 2906 1747 \n",
+       "Q 2906 2381 2643 2742 \n",
+       "Q 2381 3103 1925 3103 \n",
+       "Q 1469 3103 1208 2742 \n",
+       "Q 947 2381 947 1747 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "      </defs>\n",
+       "      <use xlink:href=\"#DejaVuSans-6d\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-65\" x=\"97.412109\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-74\" x=\"158.935547\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-68\" x=\"198.144531\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"261.523438\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-64\" x=\"322.705078\"/>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "    <g id=\"patch_14\">\n",
+       "     <path d=\"M 327.102813 38.476562 \n",
+       "L 347.102813 38.476562 \n",
+       "L 347.102813 31.476562 \n",
+       "L 327.102813 31.476562 \n",
+       "z\n",
+       "\" style=\"fill: #3274a1\"/>\n",
+       "    </g>\n",
+       "    <g id=\"text_12\">\n",
+       "     <!-- xgboost -->\n",
+       "     <g transform=\"translate(355.102813 38.476562) scale(0.1 -0.1)\">\n",
+       "      <defs>\n",
+       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
+       "L 2247 1797 \n",
+       "L 3578 0 \n",
+       "L 2900 0 \n",
+       "L 1881 1375 \n",
+       "L 863 0 \n",
+       "L 184 0 \n",
+       "L 1544 1831 \n",
+       "L 300 3500 \n",
+       "L 978 3500 \n",
+       "L 1906 2253 \n",
+       "L 2834 3500 \n",
+       "L 3513 3500 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \n",
+       "Q 2906 2416 2648 2759 \n",
+       "Q 2391 3103 1925 3103 \n",
+       "Q 1463 3103 1205 2759 \n",
+       "Q 947 2416 947 1791 \n",
+       "Q 947 1169 1205 825 \n",
+       "Q 1463 481 1925 481 \n",
+       "Q 2391 481 2648 825 \n",
+       "Q 2906 1169 2906 1791 \n",
+       "z\n",
+       "M 3481 434 \n",
+       "Q 3481 -459 3084 -895 \n",
+       "Q 2688 -1331 1869 -1331 \n",
+       "Q 1566 -1331 1297 -1286 \n",
+       "Q 1028 -1241 775 -1147 \n",
+       "L 775 -588 \n",
+       "Q 1028 -725 1275 -790 \n",
+       "Q 1522 -856 1778 -856 \n",
+       "Q 2344 -856 2625 -561 \n",
+       "Q 2906 -266 2906 331 \n",
+       "L 2906 616 \n",
+       "Q 2728 306 2450 153 \n",
+       "Q 2172 0 1784 0 \n",
+       "Q 1141 0 747 490 \n",
+       "Q 353 981 353 1791 \n",
+       "Q 353 2603 747 3093 \n",
+       "Q 1141 3584 1784 3584 \n",
+       "Q 2172 3584 2450 3431 \n",
+       "Q 2728 3278 2906 2969 \n",
+       "L 2906 3500 \n",
+       "L 3481 3500 \n",
+       "L 3481 434 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-62\" d=\"M 3116 1747 \n",
+       "Q 3116 2381 2855 2742 \n",
+       "Q 2594 3103 2138 3103 \n",
+       "Q 1681 3103 1420 2742 \n",
+       "Q 1159 2381 1159 1747 \n",
+       "Q 1159 1113 1420 752 \n",
+       "Q 1681 391 2138 391 \n",
+       "Q 2594 391 2855 752 \n",
+       "Q 3116 1113 3116 1747 \n",
+       "z\n",
+       "M 1159 2969 \n",
+       "Q 1341 3281 1617 3432 \n",
+       "Q 1894 3584 2278 3584 \n",
+       "Q 2916 3584 3314 3078 \n",
+       "Q 3713 2572 3713 1747 \n",
+       "Q 3713 922 3314 415 \n",
+       "Q 2916 -91 2278 -91 \n",
+       "Q 1894 -91 1617 61 \n",
+       "Q 1341 213 1159 525 \n",
+       "L 1159 0 \n",
+       "L 581 0 \n",
+       "L 581 4863 \n",
+       "L 1159 4863 \n",
+       "L 1159 2969 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "      </defs>\n",
+       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-67\" x=\"59.179688\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-62\" x=\"122.65625\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"186.132812\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"247.314453\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-73\" x=\"308.496094\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-74\" x=\"360.595703\"/>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "    <g id=\"patch_15\">\n",
+       "     <path d=\"M 327.102813 53.154688 \n",
+       "L 347.102813 53.154688 \n",
+       "L 347.102813 46.154688 \n",
+       "L 327.102813 46.154688 \n",
+       "z\n",
+       "\" style=\"fill: #e1812c\"/>\n",
+       "    </g>\n",
+       "    <g id=\"text_13\">\n",
+       "     <!-- dask-ml -->\n",
+       "     <g transform=\"translate(355.102813 53.154688) scale(0.1 -0.1)\">\n",
+       "      <defs>\n",
+       "       <path id=\"DejaVuSans-6b\" d=\"M 581 4863 \n",
+       "L 1159 4863 \n",
+       "L 1159 1991 \n",
+       "L 2875 3500 \n",
+       "L 3609 3500 \n",
+       "L 1753 1863 \n",
+       "L 3688 0 \n",
+       "L 2938 0 \n",
+       "L 1159 1709 \n",
+       "L 1159 0 \n",
+       "L 581 0 \n",
+       "L 581 4863 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "       <path id=\"DejaVuSans-2d\" d=\"M 313 2009 \n",
+       "L 1997 2009 \n",
+       "L 1997 1497 \n",
+       "L 313 1497 \n",
+       "L 313 2009 \n",
+       "z\n",
+       "\" transform=\"scale(0.015625)\"/>\n",
+       "      </defs>\n",
+       "      <use xlink:href=\"#DejaVuSans-64\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-61\" x=\"63.476562\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-73\" x=\"124.755859\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-6b\" x=\"176.855469\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-2d\" x=\"234.765625\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"270.849609\"/>\n",
+       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"368.261719\"/>\n",
+       "     </g>\n",
+       "    </g>\n",
+       "   </g>\n",
+       "  </g>\n",
+       " </g>\n",
+       " <defs>\n",
+       "  <clipPath id=\"p085c8fb7b0\">\n",
+       "   <rect x=\"46.965625\" y=\"7.2\" width=\"357.12\" height=\"266.112\"/>\n",
+       "  </clipPath>\n",
+       " </defs>\n",
+       "</svg>\n"
+      ],
+      "text/plain": [
+       "<Figure size 640x480 with 1 Axes>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "import xgboost as xgb\n",
+    "\n",
+    "num_sample = [100_000, 500_000]\n",
+    "num_feature = 1_000\n",
+    "xgb_timings = []\n",
+    "\n",
+    "for n in num_sample:\n",
+    "    X, y = dask_ml.datasets.make_classification(n_samples=n, \n",
+    "        n_features=num_feature, \n",
+    "        random_state=42,\n",
+    "        chunks=n // 10\n",
+    "    )\n",
+    "    dtrain = xgb.DMatrix(X, y)\n",
+    "    t1 = time.time()\n",
+    "    xgb.train(\n",
+    "        {\"tree_method\": \"hist\", \"objective\": \"binary:hinge\"},\n",
+    "        dtrain,\n",
+    "        num_boost_round=4,\n",
+    "        evals=[(dtrain, \"train\")],\n",
+    "        verbose_eval=False,\n",
+    "    )\n",
+    "    xgb_timings.append(('xgboost', n, time.time() - t1))\n",
+    "    dtrain_dask = xgb.dask.DaskDMatrix(client, X, y)\n",
+    "    t1 = time.time()\n",
+    "    xgb.dask.train(\n",
+    "        client,\n",
+    "        {\"tree_method\": \"hist\", \"objective\": \"binary:hinge\"},\n",
+    "        dtrain_dask,\n",
+    "        num_boost_round=4,\n",
+    "        evals=[(dtrain_dask, \"train\")],\n",
+    "        verbose_eval=False,\n",
+    "    )\n",
+    "    xgb_timings.append(('dask-ml', n, time.time() - t1))\n",
+    "\n",
+    "df = pd.DataFrame(xgb_timings, columns=['method', '# of samples', 'time'])\n",
+    "sns.barplot(data=df, x='# of samples', y='time', hue='method')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "如果是 XGBoost 的 scikit-learn 式 API，需要将 [`xgboost.XGBClassifier`](https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.XGBClassifier) 修改为 [`xgboost.dask.DaskXGBClassifier`](https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.dask.DaskXGBClassifier) 或者 [`xgboost.XGBRegressor`](https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.XGBRegressor) 修改为 [`xgboost.dask.DaskXGBRegressor`](https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.XGBRegressor)。\n",
+    "\n",
+    "### 分布式 GPU 训练\n",
+    "\n",
+    "Dask 可以管理多块 GPU，XGBoost 可以基于 Dask 进行多 GPU 训练，我们需要安装 Dask-CUDA 以启动一个多 GPU 的 Dask 集群。Dask 可以将 XGBoost 分布到多张 GPU 卡上进行训练。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/fs/fast/u20200002/envs/dispy/lib/python3.11/site-packages/distributed/deploy/spec.py:324: UserWarning: Port 8787 is already in use.\n",
+      "Perhaps you already have a cluster running?\n",
+      "Hosting the HTTP server on port 44607 instead\n",
+      "  self.scheduler = cls(**self.scheduler_spec.get(\"options\", {}))\n"
+     ]
+    },
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "    <div style=\"width: 24px; height: 24px; background-color: #e1e1e1; border: 3px solid #9D9D9D; border-radius: 5px; position: absolute;\"> </div>\n",
+       "    <div style=\"margin-left: 48px;\">\n",
+       "        <h3 style=\"margin-bottom: 0px;\">Client</h3>\n",
+       "        <p style=\"color: #9D9D9D; margin-bottom: 0px;\">Client-7c3ce804-0aef-11ef-98d2-000012e4fe80</p>\n",
+       "        <table style=\"width: 100%; text-align: left;\">\n",
+       "\n",
+       "        <tr>\n",
+       "        \n",
+       "            <td style=\"text-align: left;\"><strong>Connection method:</strong> Cluster object</td>\n",
+       "            <td style=\"text-align: left;\"><strong>Cluster type:</strong> dask_cuda.LocalCUDACluster</td>\n",
+       "        \n",
+       "        </tr>\n",
+       "\n",
+       "        \n",
+       "            <tr>\n",
+       "                <td style=\"text-align: left;\">\n",
+       "                    <strong>Dashboard: </strong> <a href=\"http://127.0.0.1:44607/status\" target=\"_blank\">http://127.0.0.1:44607/status</a>\n",
+       "                </td>\n",
+       "                <td style=\"text-align: left;\"></td>\n",
+       "            </tr>\n",
+       "        \n",
+       "\n",
+       "        </table>\n",
+       "\n",
+       "        \n",
+       "\n",
+       "        \n",
+       "            <details>\n",
+       "            <summary style=\"margin-bottom: 20px;\"><h3 style=\"display: inline;\">Cluster Info</h3></summary>\n",
+       "            <div class=\"jp-RenderedHTMLCommon jp-RenderedHTML jp-mod-trusted jp-OutputArea-output\">\n",
+       "    <div style=\"width: 24px; height: 24px; background-color: #e1e1e1; border: 3px solid #9D9D9D; border-radius: 5px; position: absolute;\">\n",
+       "    </div>\n",
+       "    <div style=\"margin-left: 48px;\">\n",
+       "        <h3 style=\"margin-bottom: 0px; margin-top: 0px;\">LocalCUDACluster</h3>\n",
+       "        <p style=\"color: #9D9D9D; margin-bottom: 0px;\">e461dd92</p>\n",
+       "        <table style=\"width: 100%; text-align: left;\">\n",
+       "            <tr>\n",
+       "                <td style=\"text-align: left;\">\n",
+       "                    <strong>Dashboard:</strong> <a href=\"http://127.0.0.1:44607/status\" target=\"_blank\">http://127.0.0.1:44607/status</a>\n",
+       "                </td>\n",
+       "                <td style=\"text-align: left;\">\n",
+       "                    <strong>Workers:</strong> 4\n",
+       "                </td>\n",
+       "            </tr>\n",
+       "            <tr>\n",
+       "                <td style=\"text-align: left;\">\n",
+       "                    <strong>Total threads:</strong> 4\n",
+       "                </td>\n",
+       "                <td style=\"text-align: left;\">\n",
+       "                    <strong>Total memory:</strong> 90.00 GiB\n",
+       "                </td>\n",
+       "            </tr>\n",
+       "            \n",
+       "            <tr>\n",
+       "    <td style=\"text-align: left;\"><strong>Status:</strong> running</td>\n",
+       "    <td style=\"text-align: left;\"><strong>Using processes:</strong> True</td>\n",
+       "</tr>\n",
+       "\n",
+       "            \n",
+       "        </table>\n",
+       "\n",
+       "        <details>\n",
+       "            <summary style=\"margin-bottom: 20px;\">\n",
+       "                <h3 style=\"display: inline;\">Scheduler Info</h3>\n",
+       "            </summary>\n",
+       "\n",
+       "            <div style=\"\">\n",
+       "    <div>\n",
+       "        <div style=\"width: 24px; height: 24px; background-color: #FFF7E5; border: 3px solid #FF6132; border-radius: 5px; position: absolute;\"> </div>\n",
+       "        <div style=\"margin-left: 48px;\">\n",
+       "            <h3 style=\"margin-bottom: 0px;\">Scheduler</h3>\n",
+       "            <p style=\"color: #9D9D9D; margin-bottom: 0px;\">Scheduler-a6b71eff-839c-4686-9316-a886dc1da17a</p>\n",
+       "            <table style=\"width: 100%; text-align: left;\">\n",
+       "                <tr>\n",
+       "                    <td style=\"text-align: left;\">\n",
+       "                        <strong>Comm:</strong> tcp://127.0.0.1:33619\n",
+       "                    </td>\n",
+       "                    <td style=\"text-align: left;\">\n",
+       "                        <strong>Workers:</strong> 4\n",
+       "                    </td>\n",
+       "                </tr>\n",
+       "                <tr>\n",
+       "                    <td style=\"text-align: left;\">\n",
+       "                        <strong>Dashboard:</strong> <a href=\"http://127.0.0.1:44607/status\" target=\"_blank\">http://127.0.0.1:44607/status</a>\n",
+       "                    </td>\n",
+       "                    <td style=\"text-align: left;\">\n",
+       "                        <strong>Total threads:</strong> 4\n",
+       "                    </td>\n",
+       "                </tr>\n",
+       "                <tr>\n",
+       "                    <td style=\"text-align: left;\">\n",
+       "                        <strong>Started:</strong> Just now\n",
+       "                    </td>\n",
+       "                    <td style=\"text-align: left;\">\n",
+       "                        <strong>Total memory:</strong> 90.00 GiB\n",
+       "                    </td>\n",
+       "                </tr>\n",
+       "            </table>\n",
+       "        </div>\n",
+       "    </div>\n",
+       "\n",
+       "    <details style=\"margin-left: 48px;\">\n",
+       "        <summary style=\"margin-bottom: 20px;\">\n",
+       "            <h3 style=\"display: inline;\">Workers</h3>\n",
+       "        </summary>\n",
+       "\n",
+       "        \n",
+       "        <div style=\"margin-bottom: 20px;\">\n",
+       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
+       "            <div style=\"margin-left: 48px;\">\n",
+       "            <details>\n",
+       "                <summary>\n",
+       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: 0</h4>\n",
+       "                </summary>\n",
+       "                <table style=\"width: 100%; text-align: left;\">\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Comm: </strong> tcp://127.0.0.1:45305\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Total threads: </strong> 1\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Dashboard: </strong> <a href=\"http://127.0.0.1:46261/status\" target=\"_blank\">http://127.0.0.1:46261/status</a>\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Memory: </strong> 22.50 GiB\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Nanny: </strong> tcp://127.0.0.1:37589\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\"></td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
+       "                            <strong>Local directory: </strong> /tmp/dask-scratch-space/worker-qo8pr3rx\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                </table>\n",
+       "            </details>\n",
+       "            </div>\n",
+       "        </div>\n",
+       "        \n",
+       "        <div style=\"margin-bottom: 20px;\">\n",
+       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
+       "            <div style=\"margin-left: 48px;\">\n",
+       "            <details>\n",
+       "                <summary>\n",
+       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: 1</h4>\n",
+       "                </summary>\n",
+       "                <table style=\"width: 100%; text-align: left;\">\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Comm: </strong> tcp://127.0.0.1:38835\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Total threads: </strong> 1\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Dashboard: </strong> <a href=\"http://127.0.0.1:38961/status\" target=\"_blank\">http://127.0.0.1:38961/status</a>\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Memory: </strong> 22.50 GiB\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Nanny: </strong> tcp://127.0.0.1:40985\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\"></td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
+       "                            <strong>Local directory: </strong> /tmp/dask-scratch-space/worker-vhjea3dv\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                </table>\n",
+       "            </details>\n",
+       "            </div>\n",
+       "        </div>\n",
+       "        \n",
+       "        <div style=\"margin-bottom: 20px;\">\n",
+       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
+       "            <div style=\"margin-left: 48px;\">\n",
+       "            <details>\n",
+       "                <summary>\n",
+       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: 2</h4>\n",
+       "                </summary>\n",
+       "                <table style=\"width: 100%; text-align: left;\">\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Comm: </strong> tcp://127.0.0.1:46315\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Total threads: </strong> 1\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Dashboard: </strong> <a href=\"http://127.0.0.1:42153/status\" target=\"_blank\">http://127.0.0.1:42153/status</a>\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Memory: </strong> 22.50 GiB\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Nanny: </strong> tcp://127.0.0.1:39945\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\"></td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
+       "                            <strong>Local directory: </strong> /tmp/dask-scratch-space/worker-5uebhi4w\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                </table>\n",
+       "            </details>\n",
+       "            </div>\n",
+       "        </div>\n",
+       "        \n",
+       "        <div style=\"margin-bottom: 20px;\">\n",
+       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
+       "            <div style=\"margin-left: 48px;\">\n",
+       "            <details>\n",
+       "                <summary>\n",
+       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: 3</h4>\n",
+       "                </summary>\n",
+       "                <table style=\"width: 100%; text-align: left;\">\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Comm: </strong> tcp://127.0.0.1:38331\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Total threads: </strong> 1\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Dashboard: </strong> <a href=\"http://127.0.0.1:42005/status\" target=\"_blank\">http://127.0.0.1:42005/status</a>\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Memory: </strong> 22.50 GiB\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td style=\"text-align: left;\">\n",
+       "                            <strong>Nanny: </strong> tcp://127.0.0.1:42591\n",
+       "                        </td>\n",
+       "                        <td style=\"text-align: left;\"></td>\n",
+       "                    </tr>\n",
+       "                    <tr>\n",
+       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
+       "                            <strong>Local directory: </strong> /tmp/dask-scratch-space/worker-zylz7yva\n",
+       "                        </td>\n",
+       "                    </tr>\n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                    \n",
+       "\n",
+       "                </table>\n",
+       "            </details>\n",
+       "            </div>\n",
+       "        </div>\n",
+       "        \n",
+       "\n",
+       "    </details>\n",
+       "</div>\n",
+       "\n",
+       "        </details>\n",
+       "    </div>\n",
+       "</div>\n",
+       "            </details>\n",
+       "        \n",
+       "\n",
+       "    </div>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "<Client: 'tcp://127.0.0.1:33619' processes=4 threads=4, memory=90.00 GiB>"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from dask_cuda import LocalCUDACluster\n",
+    "import xgboost as xgb\n",
+    "client = Client(LocalCUDACluster())\n",
+    "client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "clf = xgb.dask.DaskXGBClassifier(verbosity=1)\n",
+    "clf.set_params(tree_method=\"hist\", device=\"cuda\")\n",
+    "clf.client = client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/fs/fast/u20200002/envs/dispy/lib/python3.11/site-packages/dask_ml/datasets.py:373: UserWarning: Running on a single-machine scheduler when a distributed client is active might lead to unexpected results.\n",
+      "  informative_idx, beta = dask.compute(\n",
+      "[23:01:19] task [xgboost.dask-0]:tcp://127.0.0.1:45305 got new rank 0\n",
+      "[23:01:19] task [xgboost.dask-1]:tcp://127.0.0.1:38835 got new rank 1\n",
+      "[23:01:19] task [xgboost.dask-2]:tcp://127.0.0.1:46315 got new rank 2\n",
+      "[23:01:20] task [xgboost.dask-3]:tcp://127.0.0.1:38331 got new rank 3\n"
+     ]
+    }
+   ],
+   "source": [
+    "X, y = dask_ml.datasets.make_classification(n_samples=100_000, \n",
+    "        n_features=1_000, \n",
+    "        random_state=42,\n",
+    "        chunks=100_000 // 100\n",
+    ")\n",
+    "clf.fit(X, y, eval_set=[(X, y)], verbose=False)\n",
+    "prediction = clf.predict(X)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/ch-dask-ml/index.md b/ch-dask-ml/index.md
new file mode 100644
index 0000000..7068513
--- /dev/null
+++ b/ch-dask-ml/index.md
@@ -0,0 +1,4 @@
+# Dask 机器学习
+
+```{tableofcontents}
+```
\ No newline at end of file