module `remote.magic_azure`¶

Short summary¶

module pyenbc.remote.magic_azure

Magic command to run PIG script with Azure.

Classes¶

class	truncated documentation
`MagicAzure`	Defines magic commands to access blob storage …

Functions¶

function	truncated documentation
`register_azure_magics`	register magics function, can be called from a notebook

Properties¶

property	truncated documentation
`Context`	return the context or None
`cross_validation_lock`	A contextmanager for running a block with our cross validation lock set to True. At the end of the block, …

Static Methods¶

staticmethod	truncated documentation
`blob_copy_parser`	defines the way to parse the magic command `%blob_copy`
`blob_delete_parser`	defines the way to parse the magic command `%blob_delete`
`blob_down_parser`	Defines the way to parse the magic command `%blob_down`.
`blob_downmerge_parser`	defines the way to parse the magic command `%blob_downmerge`
`blob_head_parser`	defines the way to parse the magic command `%blob_head`
`blob_ls_parser`	defines the way to parse the magic command `%blob_ls`
`blob_lsl_parser`	defines the way to parse the magic command `%blob_lsl`
`blob_open_parser`	defines the way to parse the magic command `%blob_open`
`blob_path_parser`	defines the magic command `%blob_path`, checks the path used in commands `blob_down`, `blob_up`
`blob_rmr_parser`	defines the way to parse the magic command `%blob_rmr`
`blob_up_parser`	Defines the way to parse the magic command `%blob_up`.
`hd_job_kill_parser`	defines the way to parse the magic command `%hd_job_kill`
`hd_job_status_parser`	defines the way to parse the magic command `%hd_job_status`
`hd_open_parser`	defines the way to parse the magic command `%hd_open`
`hd_pig_submit_parser`	Defines the way to parse the magic command `%hd_pig_submit`.
`hd_queue_parser`	defines the way to parse the magic command `%hd_queue`
`hd_tail_stderr_parser`	defines the way to parse the magic command `%hd_tail_stderr`
`HIVE_azure_parser`	defines the way to parse the magic command `%HIVE_azure`
`HIVE_azure_submit_parser`	Defines the way to parse the magic command `%HIVE_azure_submit`.
`jython_parser`	defines the way to parse the magic command `%%jython`
`PIG_azure_parser`	defines the way to parse the magic command `%%PIG_azure`
`runjython_parser`	defines the way to parse the magic command `%%runjython`

Methods¶

method	truncated documentation
`_interpret_path`	interpret a path
`_replace_params`	replaces parameter such `__PASSWORD__` by variable in the notebook environment
`_run_jython`	run a jython script
`azureclient`	returns the AzureClient object
`blob_close`	close the connection and remove the connection from the notebook workspace
`blob_containers`	returns the list of containers
`blob_copy`	copy a blob storage, see l-magic-path-container
`blob_delete`	deletes a blob, see l-magic-path-container
`blob_down`	download a file from the blob storage, see l-magic-path-container Example
`blob_downmerge`	download all files from a folder, see l-magic-path-container Example
`blob_head`	download a file from the blob storage and display its head, see l-magic-path-container Example
`blob_ls`	defines command %blob_ls, see l-magic-path-container
`blob_lsl`	defines command %blob_lsl (extended version of blob_lsl), see l-magic-path-container
`blob_open`
`blob_path`	checks the path used in commands `blob_down`, `blob_up`, see `_interpret_path()`, l-magic-path-container …
`blob_rm`	calls `blob_delete()`
`blob_rmr`	deletes a folder, see l-magic-path-container
`blob_up`	upload a file to the blob storage, we assume the container is the first element of the path, see l-magic-path-container …
`blobcontainer`	returns the Blob Storage container
`blobservice`	returns the BlobService object
`create_client`	Create a `AzureClient` and stores in the workspace.
`get_blob_connection`	returns the connection stored in the workspace
`hd_job_kill`	defines `%hd_job_kill`
`hd_job_status`	defines `%hd_job_status`
`hd_open`	Opens a connection to blob service.
`hd_pig_submit`	Defines command `%hd_pig_submit`.
`hd_queue`	defines `%hd_queue`
`hd_tail_stderr`	defines `%hd_tail_stderr`
`hd_wasb_prefix`	defines `%hd_wasb_prefix`, returns the prefix used to connect to the blob storage, it includes the container …
`HIVE_azure`	defines command `%%HIVE_azure`
`HIVE_azure_submit`	Defines command `%HIVE_azure_submit`.
`jython`	Defines command `%%runjython`. run a jython script used for streaming in HDInsight, the function appends …
`PIG_azure`	defines command `%%PIG_azure`
`runjpython`	Defines command `%%runjython`.

Documentation¶

Magic command to run PIG script with Azure.

source on GitHub

class pyenbc.remote.magic_azure.MagicAzure(**kwargs: Any)¶

Bases: MagicClassWithHelpers

Defines magic commands to access blob storage and HDInsight.

When the container is not specified, it will take the default one.

Magic command %blob_open does not work

Try this:

%load_ext pyenbc

The exception tells more about what goes wrong. Usually a module is missing.

Incorrect padding

The following crypted message happens sometimes:

Error: Incorrect padding

It is usually due to an incorrect password. Some notebooks uses:

import pyquickhelper.ipythonhelper as ipy
params={"blob_storage":"hdblobstorage", "password":""}
ipy.open_html_form(params=params,title="credentials",key_save="blobservice")

blobstorage = blobservice["blob_storage"]
blobpassword = blobservice["password"]

%load_ext pyenbc
%blob_open

This code avoids the author letting password in a notebook but you can just replace everything by:

blobstorage = "<username>"
blobpassword = "****long*key*******=="

%load_ext pyenbc
%blob_open

source on GitHub

Create a configurable given a config config.

configConfig: If this is empty, default values are used. If config is a Config instance, it will be used to configure the instance.
parentConfigurable instance, optional: The parent Configurable instance of this object.

Subclasses of Configurable must call the __init__() method of Configurable before doing anything else and using super():

class MyConfigurable(Configurable):
    def __init__(self, config=None):
        super(MyConfigurable, self).__init__(config=config)
        # Then any other code you need to finish initialization.

This ensures that instances will be configured properly.

HIVE_azure(line, cell=None)¶

defines command %%HIVE_azure

HIVE_azure

The code for magic command %HIVE_azure is equivalent to:

with open(filename, "w", encoding="utf8") as f:
    f.write(script)

source on GitHub

static HIVE_azure_parser()¶

defines the way to parse the magic command %HIVE_azure

source on GitHub

HIVE_azure_submit(line)¶

Defines command %HIVE_azure_submit.

HIVE_azure_submit

The code for magic command %HIVE_azure_submit is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.hive_submit(bs, cl.account_name, hive_file_name, dependencies, **options)

source on GitHub

static HIVE_azure_submit_parser()¶

Defines the way to parse the magic command %HIVE_azure_submit.

source on GitHub

PIG_azure(line, cell=None)¶

defines command %%PIG_azure

PIG_azure

The code for magic command %PIG_azure is equivalent to:

with open(filename, "w", encoding="utf8") as f:
    f.write(script)

source on GitHub

static PIG_azure_parser()¶

defines the way to parse the magic command %%PIG_azure

source on GitHub

_cross_validation_lock: bool¶

_interpret_path(line, cl, bs, empty_is_value=False)¶

interpret a path

Parameters:

line – line (see l-magic-path-container)
cl – AzureClient
bs – blob service
empty_is_value – if True, do not raise an exception

Returns:

container, remotepath

source on GitHub

_replace_params(cell)¶

replaces parameter such __PASSWORD__ by variable in the notebook environment

Parameters:: cell – string
Returns:: modified string

source on GitHub

_run_jython(cell, filename, func_name, args, true_jython=None)¶

run a jython script

Parameters:

cell – content of the cell
filename – filename used to store the content of the cell
func_name – function name
args – list of arguments to run
true_jython – jython (True) or this Python (False)

Returns:

out, err

source on GitHub

_trait_default_generators = {}¶

_trait_notifiers: Dict[str, Any]¶

_trait_validators: Dict[str, Any]¶

_trait_values: Dict[str, Any]¶

azureclient(line)¶

returns the AzureClient object

source on GitHub

blob_close(line)¶

close the connection and remove the connection from the notebook workspace

blob_close

Does nothing.

source on GitHub

blob_containers(line)¶

returns the list of containers

source on GitHub

blob_copy(line)¶

copy a blob storage, see l-magic-path-container

blob_copy

The code for magic command %blob_copy is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.copy_blob(bs, container, dest, src)

source on GitHub

static blob_copy_parser()¶

defines the way to parse the magic command %blob_copy

source on GitHub

blob_delete(line)¶

deletes a blob, see l-magic-path-container

blob_delete

The code for magic command %blob_delete is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.delete_blob(bs, container, remotepath)

source on GitHub

static blob_delete_parser()¶

defines the way to parse the magic command %blob_delete

source on GitHub

blob_down(line)¶

download a file from the blob storage, see l-magic-path-container

Example:

%blob_down remotepath localfile

the command does not allow spaces in file names

blob_down

The code for magic command %blob_down is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.download(bs, container, remotepath, localfile)

source on GitHub

static blob_down_parser()¶

Defines the way to parse the magic command %blob_down.

source on GitHub

blob_downmerge(line)¶

download all files from a folder, see l-magic-path-container

Example:

%blob_downmerge remotepath localfile

the command does not allow spaces in file names

blob_downmerge

The code for magic command %blob_downmerge is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.download_merge(bs, container, remotepath, localfile)

New in version 1.1.

source on GitHub

static blob_downmerge_parser()¶

defines the way to parse the magic command %blob_downmerge

source on GitHub

blob_head(line)¶

download a file from the blob storage and display its head, see l-magic-path-container

Example:

%blob_head remotepath

the command does not allow spaces in file names

blob_head

The code for magic command %blob_head is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
df = cl.df_head(bs, container, remotepath, localfile)

source on GitHub

static blob_head_parser()¶

defines the way to parse the magic command %blob_head

source on GitHub

blob_ls(line)¶

defines command %blob_ls, see l-magic-path-container

blob_ls

The code for magic command %blob_ls is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
df = cl.ls(bs, container, remotepath)

source on GitHub

static blob_ls_parser()¶

defines the way to parse the magic command %blob_ls

source on GitHub

blob_lsl(line)¶

defines command %blob_lsl (extended version of blob_lsl), see l-magic-path-container

blob_lsl

The code for magic command %blob_lsl is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
df = cl.ls(bs, container, remotepath, add_metadata=True)

source on GitHub

static blob_lsl_parser()¶

defines the way to parse the magic command %blob_lsl

source on GitHub

blob_open(line)¶

blob_open

Opens a connection to blob service. It returns objects AzureClient and BlobService.

The code for magic command %blob_open is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()

Changed in version 1.1.

source on GitHub

static blob_open_parser()¶

defines the way to parse the magic command %blob_open

source on GitHub

blob_path(line)¶

checks the path used in commands blob_down, blob_up, see _interpret_path(), l-magic-path-container

blob_path

The code for magic command %blob_path is equivalent to:

if line.startswith("/"):
    container = account_name
    remotepath = remotepath.lstrip("/")
else:
    spl = line.split("/")
    container = spl[0]
    remotepath = None if len(spl) == 1 else "/".join(spl[1:])

result = container, remotepath

source on GitHub

static blob_path_parser()¶

defines the magic command %blob_path, checks the path used in commands blob_down, blob_up

source on GitHub

blob_rm(line)¶

calls blob_delete

New in version 1.1.

source on GitHub

blob_rmr(line)¶

deletes a folder, see l-magic-path-container

blob_rmr

The code for magic command %blob_rmr is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.delete_folder(bs, container, remotepath)

source on GitHub

static blob_rmr_parser()¶

defines the way to parse the magic command %blob_rmr

source on GitHub

blob_up(line)¶

upload a file to the blob storage, we assume the container is the first element of the path, see l-magic-path-container

Example:

%blob_up localfile remotepath

the command does not allow spaces in files

blob_up

The code for magic command %blob_up is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.upload(bs, container, remotepath, localfile)

source on GitHub

static blob_up_parser()¶

Defines the way to parse the magic command %blob_up.

source on GitHub

blobcontainer(line)¶

returns the Blob Storage container

source on GitHub

blobservice(line)¶

returns the BlobService object

source on GitHub

create_client(account_name, account_key, hadoop_server=None, hadoop_password=None, username=None)¶

Create a AzureClient and stores in the workspace.

Parameters:

account_name – login
account_key – password
hadoop_server – hadoop server
hadoop_password – hadoop password
username – username

Returns:

instance of AzureClient

source on GitHub

get_blob_connection()¶

returns the connection stored in the workspace

source on GitHub

hd_job_kill(line)¶

defines %hd_job_kill

hd_job_kill

The code for magic command %hd_job_kill is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.job_kill(jobid)

source on GitHub

static hd_job_kill_parser()¶

defines the way to parse the magic command %hd_job_kill

source on GitHub

hd_job_status(line)¶

defines %hd_job_status

hd_job_status

The code for magic command %hd_job_status is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.job_status(jobid)

source on GitHub

static hd_job_status_parser()¶

defines the way to parse the magic command %hd_job_status

source on GitHub

hd_open(line)¶

Opens a connection to blob service.

hd_open

Opens a connection to blob service. It returns objects AzureClient and BlobService.

The code for magic command %hd_open is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()

source on GitHub

static hd_open_parser()¶

defines the way to parse the magic command %hd_open

source on GitHub

hd_pig_submit(line)¶

Defines command %hd_pig_submit.

hd_pig_submit

The code for magic command %hd_pig_submit is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.pig_submit(bs, cl.account_name, pig_file_name, dependencies, **options)

source on GitHub

static hd_pig_submit_parser()¶

Defines the way to parse the magic command %hd_pig_submit.

source on GitHub

hd_queue(line)¶

defines %hd_queue

hd_queue

The code for magic command %hd_queue is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.job_queue(showall=showall)

source on GitHub

static hd_queue_parser()¶

defines the way to parse the magic command %hd_queue

source on GitHub

hd_tail_stderr(line)¶

defines %hd_tail_stderr

Warning

This function gets the status of the job to get the script name. But the rediction uses the script name and not the job id. As a consequence, if the same script name is run multiple times, the redirection will contain the output of multiples jobs.

hd_tail_stderr

The code for magic command %hd_tail_stderr is equivalent to:

from pyenbc.remote import AzureClient
cl = AzureClient(account_name, account_key, hadoop_server, hadoop_password, pseudo=username)
bs = cl.open_blob_service()
cl.standard_outputs(job_id, bs, cl.account_name, ".")