.. _pythonhadooppigrst:

=================
Python Hadoop Pig
=================


.. only:: html

    **Links:** :download:`notebook <python_hadoop_pig.ipynb>`, :downloadlink:`html <python_hadoop_pig2html.html>`, :download:`PDF <python_hadoop_pig.pdf>`, :download:`python <python_hadoop_pig.py>`, :downloadlink:`slides <python_hadoop_pig.slides.html>`, :githublink:`GitHub|_doc/notebooks/python_hadoop_pig.ipynb|*`


This notebook aims at showing how to submit a PIG job to remote hadoop
cluster (tested with Cloudera). It works better if you know Hadoop
otherwise I recommend reading `Map/Reduce avec
PIG <http://www.xavierdupre.fr/app/ensae_teaching_cs/helpsphinx2/notebooks/td3a_cenonce_session6.html#td3acenoncesession6rst>`__
(French). First, we download data. We are going to upload that data to
the remote cluster. The Hadoop distribution tested here is
`Cloudera <http://www.cloudera.com/>`__.

.. code:: ipython3

    import pyensae
    %load_ext pyensae
    %load_ext pyenbc
    pyensae.download_data("ConfLongDemo_JSI.txt", website="https://archive.ics.uci.edu/ml/machine-learning-databases/00196/")




.. parsed-literal::
    'ConfLongDemo_JSI.txt'



We open a SSH connection to the bridge which can communicate to the
cluster.

.. code:: ipython3

    import pyquickhelper.ipythonhelper as ipy
    params={"server":"", "username":"", "password":""}
    ipy.open_html_form(params=params,title="credentials",key_save="ssh_remote_hadoop")






.. raw:: html

    <div style="background-color:gainsboro; padding:2px; border:0px;"><b>credentials</b>
    <br />password <input type="password" id="ssh_remote_hadooppassword" value="" size="80" />
    <br />server <input type="text" id="ssh_remote_hadoopserver" value="" size="80" />
    <br />username <input type="text" id="ssh_remote_hadoopusername" value="" size="80" />
    <br /><button onclick="set_valuessh_remote_hadoop()">Ok</button></div>
    <script type="text/Javascript">
    function set_valuessh_remote_hadoop(){
       command='ssh_remote_hadoop = {' ;
       var ssh_remote_hadooppasswordvar_value = document.getElementById('ssh_remote_hadooppassword').value;
       command += '"password":"' + ssh_remote_hadooppasswordvar_value + '",';
       var ssh_remote_hadoopservervar_value = document.getElementById('ssh_remote_hadoopserver').value;
       command += '"server":"' + ssh_remote_hadoopservervar_value + '",';
       var ssh_remote_hadoopusernamevar_value = document.getElementById('ssh_remote_hadoopusername').value;
       command += '"username":"' + ssh_remote_hadoopusernamevar_value + '",';
       command += '}';
       var kernel = IPython.notebook.kernel;
       kernel.execute(command);
    }
    </script>



.. code:: ipython3

    password = ssh_remote_hadoop["password"]
    server = ssh_remote_hadoop["server"]
    username = ssh_remote_hadoop["username"]

We open the SSH connection:

.. code:: ipython3

    %remote_open




.. parsed-literal::
    <pyensae.remote.ssh_remote_connection.ASSHClient at 0xa2422e8>



We check the content of the remote machine:

.. code:: ipython3

    %remote_cmd ls -l






.. raw:: html

    <pre>
    total 3404
    -rw-rw-r-- 1 xavierdupre xavierdupre    1043 Jul 14 23:40 centrer_reduire.pig
    -rw-r--r-- 1 xavierdupre xavierdupre       2 Jul 15 00:22 diff_cluster
    -rw-rw-r-- 1 xavierdupre xavierdupre       0 Sep 27 00:21 dummy
    -rw-rw-r-- 1 xavierdupre xavierdupre     290 Jul 14 23:48 init_random.pig
    -rw-rw-r-- 1 xavierdupre xavierdupre    1654 Jul 15 00:20 iteration_complete.pig
    -rw-rw-r-- 1 xavierdupre xavierdupre     235 Jul 14 23:37 nb_obervations.pig
    -rw-rw-r-- 1 xavierdupre xavierdupre    1778 Jul 14 23:57 pig_1436911046432.log
    -rw-rw-r-- 1 xavierdupre xavierdupre    4570 Jul 15 00:45 pig_1436913856496.log
    -rw-rw-r-- 1 xavierdupre xavierdupre    4570 Jul 15 23:52 pig_1436997076356.log
    -rw-rw-r-- 1 xavierdupre xavierdupre     574 Jul 15 23:51 post_traitement.pig
    -rw-rw-r-- 1 xavierdupre xavierdupre     659 Sep 27 00:21 pystream.pig
    -rw-rw-r-- 1 xavierdupre xavierdupre     382 Sep 27 00:21 pystream.py
    -rw-rw-r-- 1 xavierdupre xavierdupre   26186 Jul 15 23:52 redirection.err
    -rw-rw-r-- 1 xavierdupre xavierdupre       0 Jul 15 23:51 redirection.out
    -rw-rw-r-- 1 xavierdupre xavierdupre 3400818 Jul 15 23:48 Skin_NonSkin.txt

    </pre>



.. code:: ipython3

    %remote_ls .






.. raw:: html

    <div>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th></th>
          <th>attributes</th>
          <th>code</th>
          <th>alias</th>
          <th>folder</th>
          <th>size</th>
          <th>unit</th>
          <th>name</th>
          <th>isdir</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>-rw-rw-r--</th>
          <th>1</th>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>1043</td>
          <td>Jul</td>
          <td>14</td>
          <td>23:40</td>
          <td>centrer_reduire.pig</td>
          <td>False</td>
        </tr>
        <tr>
          <th>-rw-r--r--</th>
          <th>1</th>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>2</td>
          <td>Jul</td>
          <td>15</td>
          <td>00:22</td>
          <td>diff_cluster</td>
          <td>False</td>
        </tr>
        <tr>
          <th rowspan="13" valign="top">-rw-rw-r--</th>
          <th>1</th>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>Sep</td>
          <td>27</td>
          <td>00:21</td>
          <td>dummy</td>
          <td>False</td>
        </tr>
        <tr>
          <th>1</th>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>290</td>
          <td>Jul</td>
          <td>14</td>
          <td>23:48</td>
          <td>init_random.pig</td>
          <td>False</td>
        </tr>
        <tr>
          <th>1</th>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>1654</td>
          <td>Jul</td>
          <td>15</td>
          <td>00:20</td>
          <td>iteration_complete.pig</td>
          <td>False</td>
        </tr>
        <tr>
          <th>1</th>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>235</td>
          <td>Jul</td>
          <td>14</td>
          <td>23:37</td>
          <td>nb_obervations.pig</td>
          <td>False</td>
        </tr>
        <tr>
          <th>1</th>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>1778</td>
          <td>Jul</td>
          <td>14</td>
          <td>23:57</td>
          <td>pig_1436911046432.log</td>
          <td>False</td>
        </tr>
        <tr>
          <th>1</th>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>4570</td>
          <td>Jul</td>
          <td>15</td>
          <td>00:45</td>
          <td>pig_1436913856496.log</td>
          <td>False</td>
        </tr>
        <tr>
          <th>1</th>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>4570</td>
          <td>Jul</td>
          <td>15</td>
          <td>23:52</td>
          <td>pig_1436997076356.log</td>
          <td>False</td>
        </tr>
        <tr>
          <th>1</th>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>574</td>
          <td>Jul</td>
          <td>15</td>
          <td>23:51</td>
          <td>post_traitement.pig</td>
          <td>False</td>
        </tr>
        <tr>
          <th>1</th>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>659</td>
          <td>Sep</td>
          <td>27</td>
          <td>00:21</td>
          <td>pystream.pig</td>
          <td>False</td>
        </tr>
        <tr>
          <th>1</th>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>382</td>
          <td>Sep</td>
          <td>27</td>
          <td>00:21</td>
          <td>pystream.py</td>
          <td>False</td>
        </tr>
        <tr>
          <th>1</th>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>26186</td>
          <td>Jul</td>
          <td>15</td>
          <td>23:52</td>
          <td>redirection.err</td>
          <td>False</td>
        </tr>
        <tr>
          <th>1</th>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>Jul</td>
          <td>15</td>
          <td>23:51</td>
          <td>redirection.out</td>
          <td>False</td>
        </tr>
        <tr>
          <th>1</th>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>3400818</td>
          <td>Jul</td>
          <td>15</td>
          <td>23:48</td>
          <td>Skin_NonSkin.txt</td>
          <td>False</td>
        </tr>
      </tbody>
    </table>
    </div>



We check the content on the cluster:

.. code:: ipython3

    %remote_cmd hdfs dfs -ls






.. raw:: html

    <pre>
    Found 33 items
    drwx------   - xavierdupre xavierdupre          0 2015-09-27 02:00 .Trash
    drwx------   - xavierdupre xavierdupre          0 2015-09-27 00:22 .staging
    -rw-r--r--   3 xavierdupre xavierdupre     132727 2014-11-16 02:37 ConfLongDemo_JSI.small.example.txt
    drwxr-xr-x   - xavierdupre xavierdupre          0 2014-11-16 02:38 ConfLongDemo_JSI.small.example2.walking.txt
    -rw-r--r--   3 xavierdupre xavierdupre    3400818 2015-07-14 23:35 Skin_NonSkin.txt
    drwxr-xr-x   - xavierdupre xavierdupre          0 2015-07-15 00:22 diff_cluster
    drwxr-xr-x   - xavierdupre xavierdupre          0 2015-07-14 23:44 donnees_normalisees
    drwxr-xr-x   - xavierdupre xavierdupre          0 2015-07-14 23:43 ecartstypes
    drwxr-xr-x   - xavierdupre xavierdupre          0 2015-07-14 23:49 init_random
    drwxr-xr-x   - xavierdupre xavierdupre          0 2015-07-14 23:41 moyennes
    drwxr-xr-x   - xavierdupre xavierdupre          0 2015-07-14 23:38 nb_obervations
    drwxr-xr-x   - xavierdupre xavierdupre          0 2015-07-15 00:05 output_iter1
    drwxr-xr-x   - xavierdupre xavierdupre          0 2015-07-15 00:22 output_iter10
    drwxr-xr-x   - xavierdupre xavierdupre          0 2015-07-15 00:07 output_iter2
    drwxr-xr-x   - xavierdupre xavierdupre          0 2015-07-15 00:09 output_iter3
    drwxr-xr-x   - xavierdupre xavierdupre          0 2015-07-15 00:11 output_iter4
    drwxr-xr-x   - xavierdupre xavierdupre          0 2015-07-15 00:13 output_iter5
    drwxr-xr-x   - xavierdupre xavierdupre          0 2015-07-15 00:15 output_iter6
    drwxr-xr-x   - xavierdupre xavierdupre          0 2015-07-15 00:17 output_iter7
    drwxr-xr-x   - xavierdupre xavierdupre          0 2015-07-15 00:18 output_iter8
    drwxr-xr-x   - xavierdupre xavierdupre          0 2015-07-15 00:20 output_iter9
    -rw-r--r--   3 xavierdupre xavierdupre     461444 2014-11-20 01:33 paris.2014-11-11_22-00-18.331391.txt
    drwxr-xr-x   - xavierdupre xavierdupre          0 2014-11-23 22:03 python_info.txt
    drwxr-xr-x   - xavierdupre xavierdupre          0 2014-11-23 22:07 python_info2.txt
    drwxr-xr-x   - xavierdupre xavierdupre          0 2014-12-03 22:55 random
    drwxr-xr-x   - xavierdupre xavierdupre          0 2014-11-20 23:43 unitest2
    drwxr-xr-x   - xavierdupre xavierdupre          0 2015-09-27 00:23 unittest
    drwxr-xr-x   - xavierdupre xavierdupre          0 2015-09-27 00:22 unittest2
    drwxr-xr-x   - xavierdupre xavierdupre          0 2014-11-20 01:53 velib_1hjs
    drwxr-xr-x   - xavierdupre xavierdupre          0 2014-11-21 01:17 velib_py
    drwxr-xr-x   - xavierdupre xavierdupre          0 2014-11-23 21:34 velib_py_results
    drwxr-xr-x   - xavierdupre xavierdupre          0 2014-11-23 21:51 velib_py_results_3days
    drwxr-xr-x   - xavierdupre xavierdupre          0 2014-11-21 11:08 velib_several_days

    </pre>



.. code:: ipython3

    %dfs_ls .






.. raw:: html

    <div>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>attributes</th>
          <th>code</th>
          <th>alias</th>
          <th>folder</th>
          <th>size</th>
          <th>date</th>
          <th>time</th>
          <th>name</th>
          <th>isdir</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>drwx------</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2015-09-27</td>
          <td>02:00</td>
          <td>.Trash</td>
          <td>True</td>
        </tr>
        <tr>
          <th>1</th>
          <td>drwx------</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2015-09-27</td>
          <td>00:22</td>
          <td>.staging</td>
          <td>True</td>
        </tr>
        <tr>
          <th>2</th>
          <td>-rw-r--r--</td>
          <td>3</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>132727</td>
          <td>2014-11-16</td>
          <td>02:37</td>
          <td>ConfLongDemo_JSI.small.example.txt</td>
          <td>False</td>
        </tr>
        <tr>
          <th>3</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2014-11-16</td>
          <td>02:38</td>
          <td>ConfLongDemo_JSI.small.example2.walking.txt</td>
          <td>True</td>
        </tr>
        <tr>
          <th>4</th>
          <td>-rw-r--r--</td>
          <td>3</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>3400818</td>
          <td>2015-07-14</td>
          <td>23:35</td>
          <td>Skin_NonSkin.txt</td>
          <td>False</td>
        </tr>
        <tr>
          <th>5</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2015-07-15</td>
          <td>00:22</td>
          <td>diff_cluster</td>
          <td>True</td>
        </tr>
        <tr>
          <th>6</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2015-07-14</td>
          <td>23:44</td>
          <td>donnees_normalisees</td>
          <td>True</td>
        </tr>
        <tr>
          <th>7</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2015-07-14</td>
          <td>23:43</td>
          <td>ecartstypes</td>
          <td>True</td>
        </tr>
        <tr>
          <th>8</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2015-07-14</td>
          <td>23:49</td>
          <td>init_random</td>
          <td>True</td>
        </tr>
        <tr>
          <th>9</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2015-07-14</td>
          <td>23:41</td>
          <td>moyennes</td>
          <td>True</td>
        </tr>
        <tr>
          <th>10</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2015-07-14</td>
          <td>23:38</td>
          <td>nb_obervations</td>
          <td>True</td>
        </tr>
        <tr>
          <th>11</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2015-07-15</td>
          <td>00:05</td>
          <td>output_iter1</td>
          <td>True</td>
        </tr>
        <tr>
          <th>12</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2015-07-15</td>
          <td>00:22</td>
          <td>output_iter10</td>
          <td>True</td>
        </tr>
        <tr>
          <th>13</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2015-07-15</td>
          <td>00:07</td>
          <td>output_iter2</td>
          <td>True</td>
        </tr>
        <tr>
          <th>14</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2015-07-15</td>
          <td>00:09</td>
          <td>output_iter3</td>
          <td>True</td>
        </tr>
        <tr>
          <th>15</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2015-07-15</td>
          <td>00:11</td>
          <td>output_iter4</td>
          <td>True</td>
        </tr>
        <tr>
          <th>16</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2015-07-15</td>
          <td>00:13</td>
          <td>output_iter5</td>
          <td>True</td>
        </tr>
        <tr>
          <th>17</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2015-07-15</td>
          <td>00:15</td>
          <td>output_iter6</td>
          <td>True</td>
        </tr>
        <tr>
          <th>18</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2015-07-15</td>
          <td>00:17</td>
          <td>output_iter7</td>
          <td>True</td>
        </tr>
        <tr>
          <th>19</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2015-07-15</td>
          <td>00:18</td>
          <td>output_iter8</td>
          <td>True</td>
        </tr>
        <tr>
          <th>20</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2015-07-15</td>
          <td>00:20</td>
          <td>output_iter9</td>
          <td>True</td>
        </tr>
        <tr>
          <th>21</th>
          <td>-rw-r--r--</td>
          <td>3</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>461444</td>
          <td>2014-11-20</td>
          <td>01:33</td>
          <td>paris.2014-11-11_22-00-18.331391.txt</td>
          <td>False</td>
        </tr>
        <tr>
          <th>22</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2014-11-23</td>
          <td>22:03</td>
          <td>python_info.txt</td>
          <td>True</td>
        </tr>
        <tr>
          <th>23</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2014-11-23</td>
          <td>22:07</td>
          <td>python_info2.txt</td>
          <td>True</td>
        </tr>
        <tr>
          <th>24</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2014-12-03</td>
          <td>22:55</td>
          <td>random</td>
          <td>True</td>
        </tr>
        <tr>
          <th>25</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2014-11-20</td>
          <td>23:43</td>
          <td>unitest2</td>
          <td>True</td>
        </tr>
        <tr>
          <th>26</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2015-09-27</td>
          <td>00:23</td>
          <td>unittest</td>
          <td>True</td>
        </tr>
        <tr>
          <th>27</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2015-09-27</td>
          <td>00:22</td>
          <td>unittest2</td>
          <td>True</td>
        </tr>
        <tr>
          <th>28</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2014-11-20</td>
          <td>01:53</td>
          <td>velib_1hjs</td>
          <td>True</td>
        </tr>
        <tr>
          <th>29</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2014-11-21</td>
          <td>01:17</td>
          <td>velib_py</td>
          <td>True</td>
        </tr>
        <tr>
          <th>30</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2014-11-23</td>
          <td>21:34</td>
          <td>velib_py_results</td>
          <td>True</td>
        </tr>
        <tr>
          <th>31</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2014-11-23</td>
          <td>21:51</td>
          <td>velib_py_results_3days</td>
          <td>True</td>
        </tr>
        <tr>
          <th>32</th>
          <td>drwxr-xr-x</td>
          <td>-</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2014-11-21</td>
          <td>11:08</td>
          <td>velib_several_days</td>
          <td>True</td>
        </tr>
      </tbody>
    </table>
    </div>



We upload the file on the bridge (we should zip it first, it would
reduce the uploading time).

.. code:: ipython3

    %remote_up ConfLongDemo_JSI.txt ConfLongDemo_JSI.txt




.. parsed-literal::
    'ConfLongDemo_JSI.txt'



We check it got there:

.. code:: ipython3

    %remote_cmd ls Conf*JSI.txt






.. raw:: html

    <pre>
    ConfLongDemo_JSI.txt

    </pre>



We put it on the cluster:

.. code:: ipython3

    %remote_cmd hdfs dfs -put ConfLongDemo_JSI.txt ConfLongDemo_JSI.txt






.. raw:: html

    <pre>

    </pre>



We check it was put on the cluster:

.. code:: ipython3

    %remote_cmd hdfs dfs -ls Conf*JSI.txt






.. raw:: html

    <pre>
    Found 1 items
    -rw-r--r--   3 xavierdupre xavierdupre   21546346 2015-09-27 11:33 ConfLongDemo_JSI.txt

    </pre>



.. code:: ipython3

    dfs_ls Conf*JSI.txt






.. raw:: html

    <div>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>attributes</th>
          <th>code</th>
          <th>alias</th>
          <th>folder</th>
          <th>size</th>
          <th>date</th>
          <th>time</th>
          <th>name</th>
          <th>isdir</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>-rw-r--r--</td>
          <td>3</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>21546346</td>
          <td>2015-09-27</td>
          <td>11:33</td>
          <td>ConfLongDemo_JSI.txt</td>
          <td>False</td>
        </tr>
      </tbody>
    </table>
    </div>



We create a simple PIG program:

.. code:: ipython3

    %%PIG filter_example.pig
    
    myinput = LOAD 'ConfLongDemo_JSI.txt' USING PigStorage(',') AS
        (index:long, sequence, tag, timestamp:long, dateformat, x:double,y:double, z:double, activity) ;
    filt = FILTER myinput BY activity == 'walking' ;
    STORE filt INTO 'ConfLongDemo_JSI.walking.txt' USING PigStorage() ;

.. code:: ipython3

    %pig_submit filter_example.pig -r=filter_example.redirect






.. raw:: html

    <pre>

    </pre>



We check the redirected files were created:

.. code:: ipython3

    %remote_cmd ls f*redirect*






.. raw:: html

    <pre>
    filter_example.redirect.err
    filter_example.redirect.out

    </pre>



We check the tail on a regular basis to see the job running (some other
commands can be used to monitor jobs, ``%remote_cmd mapred --help``).

.. code:: ipython3

    %remote_cmd tail filter_example.redirect.err






.. raw:: html

    <pre>
    Spillable Memory Manager spill count : 0
    Total bags proactively spilled: 0
    Total records proactively spilled: 0

    Job DAG:
    job_1435583503337_0055


    2015-09-27 11:38:56,436 [main] WARN  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning ACCESSING_NON_EXISTENT_FIELD 164860 time(s).
    2015-09-27 11:38:56,436 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!

    </pre>



.. code:: ipython3

    %remote_cmd hdfs dfs -ls Conf*JSI.walking.txt






.. raw:: html

    <pre>
    Found 2 items
    -rw-r--r--   3 xavierdupre xavierdupre          0 2015-09-27 11:38 ConfLongDemo_JSI.walking.txt/_SUCCESS
    -rw-r--r--   3 xavierdupre xavierdupre          0 2015-09-27 11:38 ConfLongDemo_JSI.walking.txt/part-m-00000

    </pre>



.. code:: ipython3

    %dfs_ls Conf*JSI.walking.txt






.. raw:: html

    <div>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>attributes</th>
          <th>code</th>
          <th>alias</th>
          <th>folder</th>
          <th>size</th>
          <th>date</th>
          <th>time</th>
          <th>name</th>
          <th>isdir</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>-rw-r--r--</td>
          <td>3</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2015-09-27</td>
          <td>11:38</td>
          <td>ConfLongDemo_JSI.walking.txt/_SUCCESS</td>
          <td>False</td>
        </tr>
        <tr>
          <th>1</th>
          <td>-rw-r--r--</td>
          <td>3</td>
          <td>xavierdupre</td>
          <td>xavierdupre</td>
          <td>0</td>
          <td>2015-09-27</td>
          <td>11:38</td>
          <td>ConfLongDemo_JSI.walking.txt/part-m-00000</td>
          <td>False</td>
        </tr>
      </tbody>
    </table>
    </div>



After that, the stream has to downloaded to the bridge and then to the
local machine with ``%remote_down``. We finally close the connection.

.. code:: ipython3

    %remote_close




.. parsed-literal::
    True



**END**