{"cells": [{"cell_type": "markdown", "metadata": {}, "source": ["# 2A.ml - Clustering\n", "\n", "Ce notebook utilise les donn\u00e9es des v\u00e9los de Chicago [Divvy Data](https://www.divvybikes.com/system-data). Il s'inspire du challenge cr\u00e9\u00e9e pour d\u00e9couvrir les habitudes des habitantes de la ville [City Bike](http://www.xavierdupre.fr/app/ensae_projects/helpsphinx/challenges/city_bike.html). L'id\u00e9e est d'explorer plusieurs algorithmes de clustering et de voire comment trafiquer les donn\u00e9es pour les faire marcher et en tirer quelques apprentissages."]}, {"cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [{"data": {"text/html": ["<div id=\"my_id_menu_nb\">run previous cell, wait for 2 seconds</div>\n", "<script>\n", "function repeat_indent_string(n){\n", "    var a = \"\" ;\n", "    for ( ; n > 0 ; --n)\n", "        a += \"    \";\n", "    return a;\n", "}\n", "// look up into all sections and builds an automated menu //\n", "var update_menu_string = function(begin, lfirst, llast, sformat, send, keep_item, begin_format, end_format) {\n", "    var anchors = document.getElementsByClassName(\"section\");\n", "    if (anchors.length == 0) {\n", "        anchors = document.getElementsByClassName(\"text_cell_render rendered_html\");\n", "    }\n", "    var i,t;\n", "    var text_menu = begin;\n", "    var text_memo = \"<pre>\\nlength:\" + anchors.length + \"\\n\";\n", "    var ind = \"\";\n", "    var memo_level = 1;\n", "    var href;\n", "    var tags = [];\n", "    var main_item = 0;\n", "    var format_open = 0;\n", "    for (i = 0; i <= llast; i++)\n", "        tags.push(\"h\" + i);\n", "\n", "    for (i = 0; i < anchors.length; i++) {\n", "        text_memo += \"**\" + anchors[i].id + \"--\\n\";\n", "\n", "        var child = null;\n", "        for(t = 0; t < tags.length; t++) {\n", "            var r = anchors[i].getElementsByTagName(tags[t]);\n", "            if (r.length > 0) {\n", "child = r[0];\n", "break;\n", "            }\n", "        }\n", "        if (child == null) {\n", "            text_memo += \"null\\n\";\n", "            continue;\n", "        }\n", "        if (anchors[i].hasAttribute(\"id\")) {\n", "            // when converted in RST\n", "            href = anchors[i].id;\n", "            text_memo += \"#1-\" + href;\n", "            // passer \u00e0 child suivant (le chercher)\n", "        }\n", "        else if (child.hasAttribute(\"id\")) {\n", "            // in a notebook\n", "            href = child.id;\n", "            text_memo += \"#2-\" + href;\n", "        }\n", "        else {\n", "            text_memo += \"#3-\" + \"*\" + \"\\n\";\n", "            continue;\n", "        }\n", "        var title = child.textContent;\n", "        var level = parseInt(child.tagName.substring(1,2));\n", "\n", "        text_memo += \"--\" + level + \"?\" + lfirst + \"--\" + title + \"\\n\";\n", "\n", "        if ((level < lfirst) || (level > llast)) {\n", "            continue ;\n", "        }\n", "        if (title.endsWith('\u00b6')) {\n", "            title = title.substring(0,title.length-1).replace(\"<\", \"&lt;\")\n", "         .replace(\">\", \"&gt;\").replace(\"&\", \"&amp;\");\n", "        }\n", "        if (title.length == 0) {\n", "            continue;\n", "        }\n", "\n", "        while (level < memo_level) {\n", "            text_menu += end_format + \"</ul>\\n\";\n", "            format_open -= 1;\n", "            memo_level -= 1;\n", "        }\n", "        if (level == lfirst) {\n", "            main_item += 1;\n", "        }\n", "        if (keep_item != -1 && main_item != keep_item + 1) {\n", "            // alert(main_item + \" - \" + level + \" - \" + keep_item);\n", "            continue;\n", "        }\n", "        while (level > memo_level) {\n", "            text_menu += \"<ul>\\n\";\n", "            memo_level += 1;\n", "        }\n", "        text_menu += repeat_indent_string(level-2);\n", "        text_menu += begin_format + sformat.replace(\"__HREF__\", href).replace(\"__TITLE__\", title);\n", "        format_open += 1;\n", "    }\n", "    while (1 < memo_level) {\n", "        text_menu += end_format + \"</ul>\\n\";\n", "        memo_level -= 1;\n", "        format_open -= 1;\n", "    }\n", "    text_menu += send;\n", "    //text_menu += \"\\n\" + text_memo;\n", "\n", "    while (format_open > 0) {\n", "        text_menu += end_format;\n", "        format_open -= 1;\n", "    }\n", "    return text_menu;\n", "};\n", "var update_menu = function() {\n", "    var sbegin = \"\";\n", "    var sformat = '<a href=\"#__HREF__\">__TITLE__</a>';\n", "    var send = \"\";\n", "    var begin_format = '<li>';\n", "    var end_format = '</li>';\n", "    var keep_item = -1;\n", "    var text_menu = update_menu_string(sbegin, 2, 4, sformat, send, keep_item,\n", "       begin_format, end_format);\n", "    var menu = document.getElementById(\"my_id_menu_nb\");\n", "    menu.innerHTML=text_menu;\n", "};\n", "window.setTimeout(update_menu,2000);\n", "            </script>"], "text/plain": ["<IPython.core.display.HTML object>"]}, "execution_count": 2, "metadata": {}, "output_type": "execute_result"}], "source": ["from jyquickhelper import add_notebook_menu\n", "add_notebook_menu()"]}, {"cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": ["%matplotlib inline"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## Les donn\u00e9es\n", "\n", "Elles ont \u00e9t\u00e9 pr\u00e9trait\u00e9es selon le notebook [Bike Pattern 2](http://www.xavierdupre.fr/app/ensae_projects/helpsphinx/notebooks/city_bike_solution_cluster_start.html). Elles repr\u00e9sentent la distribution du nombre de v\u00e9los partant (*startdist*) et arrivant (*stopdist*). On utilise le clustering pour d\u00e9couvrir les diff\u00e9rents usages des habitants de Chicago avec pour intuition le fait que les habitants de Chicago utilise majoritairement les v\u00e9los pour aller et venir entre leur appartement et leur lieu de travail. Cette m\u00eame id\u00e9e mais \u00e0 Paris est illustr\u00e9e par ce billet de blog : [Busy areas in Paris](http://www.xavierdupre.fr/blog/2013-09-26_nojs.html)."]}, {"cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [{"data": {"text/plain": ["['.\\\\features_bike_chicago.txt']"]}, "execution_count": 4, "metadata": {}, "output_type": "execute_result"}], "source": ["from pyensae.datasource import download_data\n", "file = download_data(\"features_bike_chicago.zip\")\n", "file"]}, {"cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [{"data": {"text/html": ["<div>\n", "<style scoped>\n", "    .dataframe tbody tr th:only-of-type {\n", "        vertical-align: middle;\n", "    }\n", "\n", "    .dataframe tbody tr th {\n", "        vertical-align: top;\n", "    }\n", "\n", "    .dataframe thead th {\n", "        text-align: right;\n", "    }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", "  <thead>\n", "    <tr style=\"text-align: right;\">\n", "      <th></th>\n", "      <th>station_id</th>\n", "      <th>station_name</th>\n", "      <th>weekday</th>\n", "      <th>(startdist, 00:00:00)</th>\n", "      <th>(startdist, 00:10:00)</th>\n", "      <th>(startdist, 00:20:00)</th>\n", "      <th>(startdist, 00:30:00)</th>\n", "      <th>(startdist, 00:40:00)</th>\n", "      <th>(startdist, 00:50:00)</th>\n", "      <th>(startdist, 01:00:00)</th>\n", "      <th>...</th>\n", "      <th>(stopdist, 22:20:00)</th>\n", "      <th>(stopdist, 22:30:00)</th>\n", "      <th>(stopdist, 22:40:00)</th>\n", "      <th>(stopdist, 22:50:00)</th>\n", "      <th>(stopdist, 23:00:00)</th>\n", "      <th>(stopdist, 23:10:00)</th>\n", "      <th>(stopdist, 23:20:00)</th>\n", "      <th>(stopdist, 23:30:00)</th>\n", "      <th>(stopdist, 23:40:00)</th>\n", "      <th>(stopdist, 23:50:00)</th>\n", "    </tr>\n", "  </thead>\n", "  <tbody>\n", "    <tr>\n", "      <th>0</th>\n", "      <td>2.0</td>\n", "      <td>Michigan Ave &amp; Balbo Ave</td>\n", "      <td>0.0</td>\n", "      <td>0.003756</td>\n", "      <td>0.000939</td>\n", "      <td>0.002817</td>\n", "      <td>0.000000</td>\n", "      <td>0.000000</td>\n", "      <td>0.003756</td>\n", "      <td>0.000000</td>\n", "      <td>...</td>\n", "      <td>0.004381</td>\n", "      <td>0.002191</td>\n", "      <td>0.004381</td>\n", "      <td>0.002191</td>\n", "      <td>0.004381</td>\n", "      <td>0.004381</td>\n", "      <td>0.005476</td>\n", "      <td>0.002191</td>\n", "      <td>0.000000</td>\n", "      <td>0.005476</td>\n", "    </tr>\n", "    <tr>\n", "      <th>1</th>\n", "      <td>2.0</td>\n", "      <td>Michigan Ave &amp; Balbo Ave</td>\n", "      <td>1.0</td>\n", "      <td>0.000000</td>\n", "      <td>0.000000</td>\n", "      <td>0.001106</td>\n", "      <td>0.001106</td>\n", "      <td>0.001106</td>\n", "      <td>0.002212</td>\n", "      <td>0.000000</td>\n", "      <td>...</td>\n", "      <td>0.009371</td>\n", "      <td>0.012048</td>\n", "      <td>0.006693</td>\n", "      <td>0.004016</td>\n", "      <td>0.005355</td>\n", "      <td>0.006693</td>\n", "      <td>0.002677</td>\n", "      <td>0.000000</td>\n", "      <td>0.000000</td>\n", "      <td>0.000000</td>\n", "    </tr>\n", "    <tr>\n", "      <th>2</th>\n", "      <td>2.0</td>\n", "      <td>Michigan Ave &amp; Balbo Ave</td>\n", "      <td>2.0</td>\n", "      <td>0.001357</td>\n", "      <td>0.002714</td>\n", "      <td>0.000000</td>\n", "      <td>0.001357</td>\n", "      <td>0.000000</td>\n", "      <td>0.005427</td>\n", "      <td>0.000000</td>\n", "      <td>...</td>\n", "      <td>0.002907</td>\n", "      <td>0.002907</td>\n", "      <td>0.015988</td>\n", "      <td>0.005814</td>\n", "      <td>0.001453</td>\n", "      <td>0.001453</td>\n", "      <td>0.011628</td>\n", "      <td>0.000000</td>\n", "      <td>0.000000</td>\n", "      <td>0.007267</td>\n", "    </tr>\n", "    <tr>\n", "      <th>3</th>\n", "      <td>2.0</td>\n", "      <td>Michigan Ave &amp; Balbo Ave</td>\n", "      <td>3.0</td>\n", "      <td>0.000000</td>\n", "      <td>0.004144</td>\n", "      <td>0.000000</td>\n", "      <td>0.000000</td>\n", "      <td>0.002762</td>\n", "      <td>0.004144</td>\n", "      <td>0.000000</td>\n", "      <td>...</td>\n", "      <td>0.009274</td>\n", "      <td>0.003091</td>\n", "      <td>0.003091</td>\n", "      <td>0.007728</td>\n", "      <td>0.001546</td>\n", "      <td>0.003091</td>\n", "      <td>0.009274</td>\n", "      <td>0.001546</td>\n", "      <td>0.007728</td>\n", "      <td>0.001546</td>\n", "    </tr>\n", "    <tr>\n", "      <th>4</th>\n", "      <td>2.0</td>\n", "      <td>Michigan Ave &amp; Balbo Ave</td>\n", "      <td>4.0</td>\n", "      <td>0.000000</td>\n", "      <td>0.000000</td>\n", "      <td>0.000000</td>\n", "      <td>0.002846</td>\n", "      <td>0.000000</td>\n", "      <td>0.000000</td>\n", "      <td>0.000949</td>\n", "      <td>...</td>\n", "      <td>0.008214</td>\n", "      <td>0.001027</td>\n", "      <td>0.006160</td>\n", "      <td>0.004107</td>\n", "      <td>0.015400</td>\n", "      <td>0.006160</td>\n", "      <td>0.002053</td>\n", "      <td>0.006160</td>\n", "      <td>0.007187</td>\n", "      <td>0.000000</td>\n", "    </tr>\n", "  </tbody>\n", "</table>\n", "<p>5 rows \u00d7 291 columns</p>\n", "</div>"], "text/plain": ["   station_id              station_name  weekday  (startdist, 00:00:00)  \\\n", "0         2.0  Michigan Ave & Balbo Ave      0.0               0.003756   \n", "1         2.0  Michigan Ave & Balbo Ave      1.0               0.000000   \n", "2         2.0  Michigan Ave & Balbo Ave      2.0               0.001357   \n", "3         2.0  Michigan Ave & Balbo Ave      3.0               0.000000   \n", "4         2.0  Michigan Ave & Balbo Ave      4.0               0.000000   \n", "\n", "   (startdist, 00:10:00)  (startdist, 00:20:00)  (startdist, 00:30:00)  \\\n", "0               0.000939               0.002817               0.000000   \n", "1               0.000000               0.001106               0.001106   \n", "2               0.002714               0.000000               0.001357   \n", "3               0.004144               0.000000               0.000000   \n", "4               0.000000               0.000000               0.002846   \n", "\n", "   (startdist, 00:40:00)  (startdist, 00:50:00)  (startdist, 01:00:00)  \\\n", "0               0.000000               0.003756               0.000000   \n", "1               0.001106               0.002212               0.000000   \n", "2               0.000000               0.005427               0.000000   \n", "3               0.002762               0.004144               0.000000   \n", "4               0.000000               0.000000               0.000949   \n", "\n", "           ...           (stopdist, 22:20:00)  (stopdist, 22:30:00)  \\\n", "0          ...                       0.004381              0.002191   \n", "1          ...                       0.009371              0.012048   \n", "2          ...                       0.002907              0.002907   \n", "3          ...                       0.009274              0.003091   \n", "4          ...                       0.008214              0.001027   \n", "\n", "   (stopdist, 22:40:00)  (stopdist, 22:50:00)  (stopdist, 23:00:00)  \\\n", "0              0.004381              0.002191              0.004381   \n", "1              0.006693              0.004016              0.005355   \n", "2              0.015988              0.005814              0.001453   \n", "3              0.003091              0.007728              0.001546   \n", "4              0.006160              0.004107              0.015400   \n", "\n", "   (stopdist, 23:10:00)  (stopdist, 23:20:00)  (stopdist, 23:30:00)  \\\n", "0              0.004381              0.005476              0.002191   \n", "1              0.006693              0.002677              0.000000   \n", "2              0.001453              0.011628              0.000000   \n", "3              0.003091              0.009274              0.001546   \n", "4              0.006160              0.002053              0.006160   \n", "\n", "   (stopdist, 23:40:00)  (stopdist, 23:50:00)  \n", "0              0.000000              0.005476  \n", "1              0.000000              0.000000  \n", "2              0.000000              0.007267  \n", "3              0.007728              0.001546  \n", "4              0.007187              0.000000  \n", "\n", "[5 rows x 291 columns]"]}, "execution_count": 5, "metadata": {}, "output_type": "execute_result"}], "source": ["import pandas\n", "features = pandas.read_csv(\"features_bike_chicago.txt\", sep=\"\\t\", encoding=\"utf-8\", low_memory=False, header=[0,1])\n", "features.columns = [\"station_id\", \"station_name\", \"weekday\"] + list(features.columns[3:])\n", "features.head()"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## Exercice 1 : petits clusters\n", "\n", "Que faire des petits clusters ?"]}, {"cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": []}, {"cell_type": "markdown", "metadata": {}, "source": ["## Exercice 2 : autres types de clustering\n", "\n", "On essaye des algorithmes de clustering qui n'imposent pas de choisir un nombre de clusters initial.\n", "\n", "1. On essaye [DBScan](http://scikit-learn.org/stable/modules/generated/sklearn.cluster.DBSCAN.html). Est-ce que cela fonctionne ? Si non pourquoi ?\n", "2. Et si vous savez pourquoi, vous trouverez une solution d'y rem\u00e9dier."]}, {"cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": []}], "metadata": {"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.0"}}, "nbformat": 4, "nbformat_minor": 2}