.. _f-magicazure: module ``remote.magic_azure`` ============================= .. inheritance-diagram:: pyenbc.remote.magic_azure Short summary +++++++++++++ module ``pyenbc.remote.magic_azure`` Magic command to run PIG script with Azure. :githublink:`%|py|6` Classes +++++++ +------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+ | class | truncated documentation | +============================================================+=================================================================================================================================================+ | :class:`MagicAzure ` | Defines magic commands to access `blob storage `_ ... | +------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+ Functions +++++++++ +---------------------------------------------------------------------------------+---------------------------------------------------------+ | function | truncated documentation | +=================================================================================+=========================================================+ | :func:`register_azure_magics ` | register magics function, can be called from a notebook | +---------------------------------------------------------------------------------+---------------------------------------------------------+ Properties ++++++++++ +--------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ | property | truncated documentation | +============================================================================================+===============================================================================================================+ | :meth:`Context ` | return the context or None | +--------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ | :meth:`cross_validation_lock ` | A contextmanager for running a block with our cross validation lock set to True. At the end of the block, ... | +--------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ Static Methods ++++++++++++++ +--------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------+ | staticmethod | truncated documentation | +==================================================================================================+=======================================================================================================+ | :meth:`blob_copy_parser ` | defines the way to parse the magic command ``%blob_copy`` | +--------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------+ | :meth:`blob_delete_parser ` | defines the way to parse the magic command ``%blob_delete`` | +--------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------+ | :meth:`blob_down_parser ` | Defines the way to parse the magic command ``%blob_down``. | +--------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------+ | :meth:`blob_downmerge_parser ` | defines the way to parse the magic command ``%blob_downmerge`` | +--------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------+ | :meth:`blob_head_parser ` | defines the way to parse the magic command ``%blob_head`` | +--------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------+ | :meth:`blob_ls_parser ` | defines the way to parse the magic command ``%blob_ls`` | +--------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------+ | :meth:`blob_lsl_parser ` | defines the way to parse the magic command ``%blob_lsl`` | +--------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------+ | :meth:`blob_open_parser ` | defines the way to parse the magic command ``%blob_open`` | +--------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------+ | :meth:`blob_path_parser ` | defines the magic command ``%blob_path``, checks the path used in commands ``blob_down``, ``blob_up`` | +--------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------+ | :meth:`blob_rmr_parser ` | defines the way to parse the magic command ``%blob_rmr`` | +--------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------+ | :meth:`blob_up_parser ` | Defines the way to parse the magic command ``%blob_up``. | +--------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------+ | :meth:`hd_job_kill_parser ` | defines the way to parse the magic command ``%hd_job_kill`` | +--------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------+ | :meth:`hd_job_status_parser ` | defines the way to parse the magic command ``%hd_job_status`` | +--------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------+ | :meth:`hd_open_parser ` | defines the way to parse the magic command ``%hd_open`` | +--------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------+ | :meth:`hd_pig_submit_parser ` | Defines the way to parse the magic command ``%hd_pig_submit``. | +--------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------+ | :meth:`hd_queue_parser ` | defines the way to parse the magic command ``%hd_queue`` | +--------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------+ | :meth:`hd_tail_stderr_parser ` | defines the way to parse the magic command ``%hd_tail_stderr`` | +--------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------+ | :meth:`HIVE_azure_parser ` | defines the way to parse the magic command ``%HIVE_azure`` | +--------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------+ | :meth:`HIVE_azure_submit_parser ` | Defines the way to parse the magic command ``%HIVE_azure_submit``. | +--------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------+ | :meth:`jython_parser ` | defines the way to parse the magic command ``%%jython`` | +--------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------+ | :meth:`PIG_azure_parser ` | defines the way to parse the magic command ``%%PIG_azure`` | +--------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------+ | :meth:`runjython_parser ` | defines the way to parse the magic command ``%%runjython`` | +--------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------+ Methods +++++++ +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | method | truncated documentation | +========================================================================================+====================================================================================================================================+ | :py:meth:`_interpret_path ` | interpret a path | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :py:meth:`_replace_params ` | replaces parameter such ``__PASSWORD__`` by variable in the notebook environment | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :py:meth:`_run_jython ` | run a jython script | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`azureclient ` | returns the AzureClient object | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`blob_close ` | close the connection and remove the connection from the notebook workspace | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`blob_containers ` | returns the list of containers | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`blob_copy ` | copy a blob storage, see :ref:`l-magic-path-container` | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`blob_delete ` | deletes a blob, see :ref:`l-magic-path-container` | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`blob_down ` | download a file from the blob storage, see :ref:`l-magic-path-container` Example | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`blob_downmerge ` | download all files from a folder, see :ref:`l-magic-path-container` Example | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`blob_head ` | download a file from the blob storage and display its head, see :ref:`l-magic-path-container` Example | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`blob_ls ` | defines command %blob_ls, see :ref:`l-magic-path-container` | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`blob_lsl ` | defines command %blob_lsl (extended version of blob_lsl), see :ref:`l-magic-path-container` | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`blob_open ` | | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`blob_path ` | checks the path used in commands ``blob_down``, ``blob_up``, see :meth:`_interpret_path`, :ref:`l-magic-path-container` ... | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`blob_rm ` | calls :meth:`blob_delete` | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`blob_rmr ` | deletes a folder, see :ref:`l-magic-path-container` | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`blob_up ` | upload a file to the blob storage, we assume the container is the first element of the path, see :ref:`l-magic-path-container` ... | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`blobcontainer ` | returns the Blob Storage container | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`blobservice ` | returns the BlobService object | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`create_client ` | Create a :class:`AzureClient` and stores in the workspace. | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`get_blob_connection ` | returns the connection stored in the workspace | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`hd_job_kill ` | defines ``%hd_job_kill`` | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`hd_job_status ` | defines ``%hd_job_status`` | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`hd_open ` | Opens a connection to blob service. | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`hd_pig_submit ` | Defines command ``%hd_pig_submit``. | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`hd_queue ` | defines ``%hd_queue`` | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`hd_tail_stderr ` | defines ``%hd_tail_stderr`` | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`hd_wasb_prefix ` | defines ``%hd_wasb_prefix``, returns the prefix used to connect to the blob storage, it includes the *container* ... | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`HIVE_azure ` | defines command ``%%HIVE_azure`` | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`HIVE_azure_submit ` | Defines command ``%HIVE_azure_submit``. | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`jython ` | Defines command ``%%runjython``. run a jython script used for streaming in HDInsight, the function appends ... | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`PIG_azure ` | defines command ``%%PIG_azure`` | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ | :meth:`runjpython ` | Defines command ``%%runjython``. | +----------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+ Documentation +++++++++++++ .. automodule:: pyenbc.remote.magic_azure :members: :special-members: __init__ :show-inheritance: