Methods

Summary

method	class parent	truncated documentation
HIVE	MagicRemoteSSH	defines command %%HIVE
HIVE_azure	MagicAzure	defines command %%HIVE_azure
HIVE_azure_submit	MagicAzure	Defines command %HIVE_azure_submit.
PIG	MagicRemoteSSH	defines command %%PIG
PIG_azure	MagicAzure	defines command %%PIG_azure
__init__	AzureClient	constructor
__init__	AzureDrive	constructor
__init__	AzureException	store more information than a regular exception
__init__	AzureTransferAPI
__init__	CloudTransfer	constructor
__init__	ASSHClient	constructor
__str__	AzureClient	usual
__str__	AzureException	usual
__str__	ASSHClient	usual
_interpret_path	AzureClient	replace variavble such as $PSEUDO, $USERNAME
_interpret_path	MagicAzure	interpret a path
_replace_params	MagicAzure	replaces parameter such __PASSWORD__ by variable in the notebook environment
_replace_params	MagicRemoteSSH	replaces parameter such __PASSWORD__ by variable in the notebook environment
_run_jython	MagicAzure	run a jython script
azureclient	MagicAzure	returns the AzureClient object
blob_close	MagicAzure	close the connection and remove the connection from the notebook workspace
blob_containers	MagicAzure	returns the list of containers
blob_copy	MagicAzure	copy a blob storage, see l-magic-path-container
blob_delete	MagicAzure	deletes a blob, see l-magic-path-container
blob_down	MagicAzure	download a file from the blob storage, see l-magic-path-container Example
blob_downmerge	MagicAzure	download all files from a folder, see l-magic-path-container Example
blob_head	MagicAzure	download a file from the blob storage and display its head, see l-magic-path-container Example
blob_ls	MagicAzure	defines command %blob_ls, see l-magic-path-container
blob_lsl	MagicAzure	defines command %blob_lsl (extended version of blob_lsl), see l-magic-path-container
blob_open	MagicAzure
blob_path	MagicAzure	checks the path used in commands blob_down, blob_up, see _interpret_path(), l-magic-path-container …
blob_rm	MagicAzure	calls blob_delete()
blob_rmr	MagicAzure	deletes a folder, see l-magic-path-container
blob_up	MagicAzure	upload a file to the blob storage, we assume the container is the first element of the path, see l-magic-path-container …
blobcontainer	MagicAzure	returns the Blob Storage container
blobservice	MagicAzure	returns the BlobService object
close	AzureDrive	close the connection
close	CloudTransfer	close the connection
close	ASSHClient	close the connection
close_remote_shell	MagicRemoteSSH	Defines %close_remote_shell
close_session	ASSHClient	close a session
connect	AzureDrive	connect
connect	CloudTransfer	connect
connect	ASSHClient	connect
copy_blob	AzureClient	copy a blob
create_client	MagicAzure	Create a AzureClient and stores in the workspace.
delete_blob	AzureClient	delete a blob
delete_folder	AzureClient	delete a folder and its content
df_head	AzureClient	Downloads the beginning of a stream and displays as a DataFrame.
dfs_exists	ASSHClient	tells if a file exists on the cluster
dfs_ls	MagicRemoteSSH	returns the content of a folder on the cluster as a dataframe Example
dfs_ls	ASSHClient	return the content of a folder on the cluster as a DataFrame
dfs_mkdir	MagicRemoteSSH	Creates a folder on the cluster. Example
dfs_mkdir	ASSHClient	creates a directory on the cluster
dfs_rm	MagicRemoteSSH	remove a file on the cluster Example
dfs_rm	ASSHClient	removes a file on the cluster
download	AzureClient	Downloads data from a blob storage to a file. No more than 64Mb can be downloaded at the same, it needs to be split …
download	ASSHClient	download a file from the remote machine (not on the cluster)
download_cluster	ASSHClient	download a file directly from the cluster to the local machine
download_data	AzureClient	Downloads data from a blob storage and return bytes. No more than 64Mb can be downloaded at the same, it needs …
download_data	AzureDrive	download binary data
download_data	CloudTransfer	download binary data
download_merge	AzureClient	Downloads all files from a folder in a blob storage to a single local file. Files will be merged. No more …
execute_command	ASSHClient	execute a command line, it raises an error if there is an error
exists	AzureClient	test the existence of a path on the blob storage
exists	ASSHClient	tells if a file exists on the bridge
get_blob_connection	MagicAzure	returns the connection stored in the workspace
get_connection	MagicRemoteSSH	returns the connection stored in the workspace
get_status	AzureClient	return the status of the webHCatUrl server
get_version	AzureClient	return the status of the WebHCat version
hd_job_kill	MagicAzure	defines %hd_job_kill
hd_job_status	MagicAzure	defines %hd_job_status
hd_open	MagicAzure	Opens a connection to blob service.
hd_pig_submit	MagicAzure	Defines command %hd_pig_submit.
hd_queue	MagicAzure	defines %hd_queue
hd_tail_stderr	MagicAzure	defines %hd_tail_stderr
hd_wasb_prefix	MagicAzure	defines %hd_wasb_prefix, returns the prefix used to connect to the blob storage, it includes the container …
hive_submit	AzureClient	Submits a :epkg:`HIVE` job, the function uploads it to the cluster as well as the dependencies. The code …
hive_submit	MagicRemoteSSH	defines command %hive_submit
hive_submit	ASSHClient	submits a PIG script, it first upload the script to the default folder and submit it
job_kill	AzureClient	kills a job see Delete Job — DELETE queue/:jobid …
job_queue	AzureClient	returns the list of jobs It uses the API Job Information — GET queue/:jobid. …
job_status	AzureClient	return the status of a job see List Versions — GET version …
job_syntax	MagicRemoteSSH	defines command %job_syntax
jython	MagicAzure	Defines command %%runjython. run a jython script used for streaming in HDInsight, the function appends …
ls	AzureClient	return the content of a blob storage
ls	ASSHClient	return the content of a folder on the bridge as a DataFrame
open_blob_service	AzureClient	open a blob service
open_remote_shell	MagicRemoteSSH	Defines %open_remote_shell
open_session	ASSHClient	Opens a session with method invoke_shell. …
pig_submit	AzureClient	Submits a :epkg:`PIG` job, the function uploads it to the cluster as well as the dependencies. The code …
pig_submit	MagicRemoteSSH	defines command %pig_submit
pig_submit	ASSHClient	Submits a :epkg:`PIG` script, it first upload the script to the default folder and submits it.
remote_close	MagicRemoteSSH	close a SSH connection and store the connection into the notebook workspace
remote_cmd	MagicRemoteSSH	run a command on the remote machine Example
remote_cmd_text	MagicRemoteSSH	run a command on the remote machine and returns raw text (not HTML) Example
remote_down	MagicRemoteSSH	download a file from the remote machine, Example
remote_down_cluster	MagicRemoteSSH	download a file from the cluster to the local machine through the bridge Example
remote_ls	MagicRemoteSSH	returns the content of a folder on the remote machine as a dataframe Example
remote_open	MagicRemoteSSH	open a SSH connection and store the connection into the notebook workspace
remote_py	MagicRemoteSSH	defines command %remote_py
remote_up	MagicRemoteSSH	upload a file to the remote machine, Example
remote_up_cluster	MagicRemoteSSH	upload a file to the remote machine and then to the remote cluster, Example
retrieve	AzureTransferAPI	retrieve data from path
runjpython	MagicAzure	Defines command %%runjython.
send_recv_session	ASSHClient	Send something through a session, the function is supposed to return when the execute of the given command is done, …
shell_remote	MagicRemoteSSH	Defines %shell_remote and %%shell_remote
shell_remote_text	MagicRemoteSSH	Defines %shell_remote_text and %%shell_remote_text
standard_outputs	AzureClient	returns the standard output and error for a specific job id
transfer	AzureTransferAPI	we assume a data holds in memory, tansfer data to path
upload	AzureClient	Uploads data from a file to a blob storage. No more than 64Mb can be uploaded at the same, it needs to be split …
upload	ASSHClient	upload a file to the remote machine (not on the cluster)
upload_cluster	ASSHClient	the function directly uploads the file to the cluster, it first goes to the bridge, uploads it to the cluster and …
upload_data	AzureClient	Uploads data (bytes) to a blob storage. No more than 64Mb can be uploaded at the same, it needs to be split into …
upload_data	AzureDrive	upload binary data
upload_data	CloudTransfer	upload binary data
url_blob	AzureClient	returns an url for a blob file name
url_webHCatUrl	AzureClient	returns an url to the cluster
wait_job	AzureClient	wait until a job has run or failed
wasb_prefix	AzureClient	when using an instruction LOAD in a PIG script, file blob name must be reference using a wasb syntax. …
wasb_to_file	AzureClient	return something like wasb://demo@myblobstorage.blob...