.. _rocexamplerst:

===
ROC
===


.. only:: html

    **Links:** :download:`notebook <roc_example.ipynb>`, :downloadlink:`html <roc_example2html.html>`, :download:`PDF <roc_example.pdf>`, :download:`python <roc_example.py>`, :downloadlink:`slides <roc_example.slides.html>`, :githublink:`GitHub|_doc/notebooks/metric/roc_example.ipynb|*`


A few graphs about ROC on the iris datasets.

.. code:: ipython3

    %matplotlib inline
    import matplotlib.pyplot as plt
    plt.style.use('ggplot')

.. code:: ipython3

    from jyquickhelper import add_notebook_menu
    add_notebook_menu()






.. contents::
    :local:





Iris datasets
-------------

.. code:: ipython3

    from sklearn import datasets
    iris = datasets.load_iris()
    X = iris.data[:, :2]
    y = iris.target

.. code:: ipython3

    from sklearn.model_selection import train_test_split
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)

.. code:: ipython3

    from sklearn.linear_model import LogisticRegression
    clf = LogisticRegression()
    clf.fit(X_train, y_train)




.. parsed-literal::
    LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
              intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
              penalty='l2', random_state=None, solver='liblinear', tol=0.0001,
              verbose=0, warm_start=False)



.. code:: ipython3

    import numpy
    ypred = clf.predict(X_test)
    yprob = clf.predict_proba(X_test)
    score = numpy.array(list(yprob[i,ypred[i]] for i in range(len(ypred))))

.. code:: ipython3

    data = numpy.zeros((len(ypred), 2))
    data[:,0] = score.ravel()
    data[ypred==y_test,1] = 1
    data[:5]




.. parsed-literal::
    array([[ 0.70495209,  1.        ],
           [ 0.56148737,  0.        ],
           [ 0.56148737,  1.        ],
           [ 0.77416227,  1.        ],
           [ 0.58631799,  0.        ]])



ROC with scikit-learn
---------------------

We use the following example `Receiver Operating Characteristic
(ROC) <http://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html#sphx-glr-auto-examples-model-selection-plot-roc-py>`__.

.. code:: ipython3

    from sklearn.metrics import roc_curve
    fpr, tpr, th = roc_curve(y_test == ypred, score)

.. code:: ipython3

    import matplotlib.pyplot as plt
    plt.plot(fpr, tpr, label='ROC curve')
    plt.plot([0, 1], [0, 1], linestyle='--')
    plt.xlim([0.0, 1.0])
    plt.ylim([0.0, 1.0])
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.legend(loc="lower right")




.. parsed-literal::
    <matplotlib.legend.Legend at 0x268373a2128>




.. image:: roc_example_11_1.png


.. code:: ipython3

    import pandas
    df = pandas.DataFrame(dict(fpr=fpr, tpr=tpr, threshold=th))
    df






.. raw:: html

    <div>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>fpr</th>
          <th>threshold</th>
          <th>tpr</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0.000000</td>
          <td>0.910712</td>
          <td>0.032258</td>
        </tr>
        <tr>
          <th>1</th>
          <td>0.000000</td>
          <td>0.869794</td>
          <td>0.096774</td>
        </tr>
        <tr>
          <th>2</th>
          <td>0.000000</td>
          <td>0.863174</td>
          <td>0.161290</td>
        </tr>
        <tr>
          <th>3</th>
          <td>0.000000</td>
          <td>0.805864</td>
          <td>0.258065</td>
        </tr>
        <tr>
          <th>4</th>
          <td>0.000000</td>
          <td>0.790909</td>
          <td>0.387097</td>
        </tr>
        <tr>
          <th>5</th>
          <td>0.000000</td>
          <td>0.650510</td>
          <td>0.612903</td>
        </tr>
        <tr>
          <th>6</th>
          <td>0.052632</td>
          <td>0.634499</td>
          <td>0.612903</td>
        </tr>
        <tr>
          <th>7</th>
          <td>0.052632</td>
          <td>0.620319</td>
          <td>0.709677</td>
        </tr>
        <tr>
          <th>8</th>
          <td>0.105263</td>
          <td>0.615015</td>
          <td>0.709677</td>
        </tr>
        <tr>
          <th>9</th>
          <td>0.210526</td>
          <td>0.607975</td>
          <td>0.741935</td>
        </tr>
        <tr>
          <th>10</th>
          <td>0.210526</td>
          <td>0.604496</td>
          <td>0.774194</td>
        </tr>
        <tr>
          <th>11</th>
          <td>0.263158</td>
          <td>0.586318</td>
          <td>0.774194</td>
        </tr>
        <tr>
          <th>12</th>
          <td>0.263158</td>
          <td>0.584172</td>
          <td>0.806452</td>
        </tr>
        <tr>
          <th>13</th>
          <td>0.315789</td>
          <td>0.561487</td>
          <td>0.838710</td>
        </tr>
        <tr>
          <th>14</th>
          <td>0.421053</td>
          <td>0.556499</td>
          <td>0.838710</td>
        </tr>
        <tr>
          <th>15</th>
          <td>0.578947</td>
          <td>0.525449</td>
          <td>0.838710</td>
        </tr>
        <tr>
          <th>16</th>
          <td>0.578947</td>
          <td>0.522579</td>
          <td>0.870968</td>
        </tr>
        <tr>
          <th>17</th>
          <td>0.631579</td>
          <td>0.522551</td>
          <td>0.870968</td>
        </tr>
        <tr>
          <th>18</th>
          <td>0.684211</td>
          <td>0.520835</td>
          <td>0.903226</td>
        </tr>
        <tr>
          <th>19</th>
          <td>0.736842</td>
          <td>0.516687</td>
          <td>0.903226</td>
        </tr>
        <tr>
          <th>20</th>
          <td>0.736842</td>
          <td>0.453826</td>
          <td>0.967742</td>
        </tr>
        <tr>
          <th>21</th>
          <td>0.842105</td>
          <td>0.417941</td>
          <td>0.967742</td>
        </tr>
        <tr>
          <th>22</th>
          <td>0.842105</td>
          <td>0.412703</td>
          <td>1.000000</td>
        </tr>
        <tr>
          <th>23</th>
          <td>1.000000</td>
          <td>0.375573</td>
          <td>1.000000</td>
        </tr>
      </tbody>
    </table>
    </div>



ROC - TPR / FPR
---------------

We do the same with the class this module provides
`ROC <http://www.xavierdupre.fr/app/mlstatpy/helpsphinx/mlstatpy/ml/roc.html?highlight=roc#module-mlstatpy.ml.roc>`__.


-  TPR = True Positive Rate
-  FPR = False Positive Rate

You can see as TPR the distribution function of a score for a positive
example and the FPR the same for a negative example.

.. code:: ipython3

    from mlstatpy.ml.roc import ROC

.. code:: ipython3

    roc = ROC(df=data)

.. code:: ipython3

    roc




.. parsed-literal::
    Overall precision: 0.63 - AUC=0.850594
    --------------
          score  label  weight
    0  0.375573    0.0     1.0
    1  0.385480    0.0     1.0
    2  0.412314    0.0     1.0
    3  0.412703    1.0     1.0
    4  0.417941    0.0     1.0
    --------------
           score  label  weight
    45  0.863174    1.0     1.0
    46  0.863174    1.0     1.0
    47  0.869794    1.0     1.0
    48  0.903335    1.0     1.0
    49  0.910712    1.0     1.0
    --------------
        False Positive Rate  True Positive Rate  threshold
    0              0.000000            0.032258   0.910712
    1              0.000000            0.193548   0.828617
    2              0.000000            0.354839   0.790909
    3              0.000000            0.516129   0.737000
    4              0.052632            0.645161   0.627589
    5              0.157895            0.741935   0.607975
    6              0.263158            0.838710   0.561487
    7              0.526316            0.838710   0.542211
    8              0.684211            0.903226   0.520835
    9              0.842105            0.967742   0.417941
    10             1.000000            1.000000   0.375573
    --------------
           error  recall  threshold
    0   0.000000    0.02   0.910712
    1   0.000000    0.12   0.828617
    2   0.000000    0.22   0.790909
    3   0.000000    0.32   0.737000
    4   0.047619    0.42   0.627589
    5   0.115385    0.52   0.607975
    6   0.161290    0.62   0.561487
    7   0.277778    0.72   0.542211
    8   0.317073    0.82   0.520835
    9   0.347826    0.92   0.417941
    10  0.380000    1.00   0.375573



.. code:: ipython3

    roc.auc()




.. parsed-literal::
    0.85059422750424452



.. code:: ipython3

    roc.plot(nb=10)




.. parsed-literal::
    <matplotlib.axes._subplots.AxesSubplot at 0x2683ff2b668>




.. image:: roc_example_18_1.png


This function draws the curve with only 10 points but we can ask for
more.

.. code:: ipython3

    roc.plot(nb=100)




.. parsed-literal::
    <matplotlib.axes._subplots.AxesSubplot at 0x2683feba240>




.. image:: roc_example_20_1.png


We can also ask to draw bootstropped curves to get a sense of the
confidence.

.. code:: ipython3

    roc.plot(nb=10, bootstrap=10)




.. parsed-literal::
    <matplotlib.axes._subplots.AxesSubplot at 0x26840008748>




.. image:: roc_example_22_1.png


ROC - score distribution
------------------------

This another representation for the metrics FPR and TPR. :math:`P(x<s)`
is the probability that a score for a positive example to be less than
:math:`s`. :math:`P(->s)` is the probability that a score for a negative
example to be higher than :math:`s`. We assume in this case that the
higher the better for the score.

.. code:: ipython3

    roc.plot(curve=ROC.CurveType.PROBSCORE, thresholds=True)




.. parsed-literal::
    <matplotlib.axes._subplots.AxesSubplot at 0x268410618d0>




.. image:: roc_example_24_1.png


When curves intersects at score :math:`s^*`, error rates for positive
and negative examples are equal. If we show the confusion matrix for
this particular score :math:`s^*`, it gives:

.. code:: ipython3

    conf = roc.confusion()
    conf["P(+<s)"] = 1 - conf["True Positive"] / conf.loc[len(conf)-1,"True Positive"]
    conf["P(->s)"] = 1 - conf["True Negative"] / conf.loc[0,"True Negative"]
    conf






.. raw:: html

    <div>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>True Positive</th>
          <th>False Positive</th>
          <th>False Negative</th>
          <th>True Negative</th>
          <th>threshold</th>
          <th>P(+&lt;s)</th>
          <th>P(-&gt;s)</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0.0</td>
          <td>0.0</td>
          <td>31.0</td>
          <td>19.0</td>
          <td>0.910712</td>
          <td>1.000000</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>1</th>
          <td>1.0</td>
          <td>0.0</td>
          <td>30.0</td>
          <td>19.0</td>
          <td>0.910712</td>
          <td>0.967742</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>2</th>
          <td>6.0</td>
          <td>0.0</td>
          <td>25.0</td>
          <td>19.0</td>
          <td>0.828617</td>
          <td>0.806452</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>3</th>
          <td>11.0</td>
          <td>0.0</td>
          <td>20.0</td>
          <td>19.0</td>
          <td>0.790909</td>
          <td>0.645161</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>4</th>
          <td>16.0</td>
          <td>0.0</td>
          <td>15.0</td>
          <td>19.0</td>
          <td>0.737000</td>
          <td>0.483871</td>
          <td>0.000000</td>
        </tr>
        <tr>
          <th>5</th>
          <td>20.0</td>
          <td>1.0</td>
          <td>11.0</td>
          <td>18.0</td>
          <td>0.627589</td>
          <td>0.354839</td>
          <td>0.052632</td>
        </tr>
        <tr>
          <th>6</th>
          <td>23.0</td>
          <td>3.0</td>
          <td>8.0</td>
          <td>16.0</td>
          <td>0.607975</td>
          <td>0.258065</td>
          <td>0.157895</td>
        </tr>
        <tr>
          <th>7</th>
          <td>26.0</td>
          <td>5.0</td>
          <td>5.0</td>
          <td>14.0</td>
          <td>0.561487</td>
          <td>0.161290</td>
          <td>0.263158</td>
        </tr>
        <tr>
          <th>8</th>
          <td>26.0</td>
          <td>10.0</td>
          <td>5.0</td>
          <td>9.0</td>
          <td>0.542211</td>
          <td>0.161290</td>
          <td>0.526316</td>
        </tr>
        <tr>
          <th>9</th>
          <td>28.0</td>
          <td>13.0</td>
          <td>3.0</td>
          <td>6.0</td>
          <td>0.520835</td>
          <td>0.096774</td>
          <td>0.684211</td>
        </tr>
        <tr>
          <th>10</th>
          <td>30.0</td>
          <td>16.0</td>
          <td>1.0</td>
          <td>3.0</td>
          <td>0.417941</td>
          <td>0.032258</td>
          <td>0.842105</td>
        </tr>
        <tr>
          <th>11</th>
          <td>31.0</td>
          <td>19.0</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>0.375573</td>
          <td>0.000000</td>
          <td>1.000000</td>
        </tr>
      </tbody>
    </table>
    </div>



ROC - recall / precision
------------------------

In this representation, we show the score.

.. code:: ipython3

    import matplotlib.pyplot as plt
    fig, axes = plt.subplots(ncols=2, nrows=1, figsize=(14,4))
    roc.plot(curve=ROC.CurveType.RECPREC, thresholds=True, ax=axes[0])
    roc.plot(curve=ROC.CurveType.RECPREC, ax=axes[1])




.. parsed-literal::
    <matplotlib.axes._subplots.AxesSubplot at 0x2684151d3c8>




.. image:: roc_example_28_1.png