module remote.magic_remote_ssh
¶
Short summary¶
module pyenbc.remote.magic_remote_ssh
Magic command to communicate with an Hadoop cluster (Linux).
Classes¶
class |
truncated documentation |
---|---|
Defines commands to access a remote machine (bridge) through SSH, for the time being, all the command do not accept … |
Functions¶
function |
truncated documentation |
---|---|
register magics function, can be called from a notebook |
Properties¶
property |
truncated documentation |
---|---|
|
return the context or None |
|
A contextmanager for running a block with our cross validation lock set to True. At the end of the block, … |
Static Methods¶
staticmethod |
truncated documentation |
---|---|
defines the way to parse the magic command |
|
defines the way to parse the magic command |
|
defines the way to parse the magic command |
|
defines the way to parse the magic command |
|
defines the way to parse the magic command |
|
defines the way to parse the magic command |
|
defines the way to parse the magic command |
|
defines the way to parse the magic command |
|
defines the way to parse the magic command |
|
defines the way to parse the magic command |
|
defines the way to parse the magic command |
|
defines the way to parse the magic command |
|
defines the way to parse the magic command |
|
defines the way to parse the magic command |
|
defines the way to parse the magic command |
|
defines the way to parse the magic command |
Methods¶
method |
truncated documentation |
---|---|
replaces parameter such |
|
Defines |
|
returns the content of a folder on the cluster as a dataframe Example |
|
Creates a folder on the cluster. Example |
|
remove a file on the cluster Example |
|
returns the connection stored in the workspace |
|
defines command |
|
defines command |
|
defines command |
|
Defines |
|
defines command |
|
defines command |
|
close a SSH connection and store the connection into the notebook workspace |
|
run a command on the remote machine Example |
|
run a command on the remote machine and returns raw text (not HTML) Example |
|
download a file from the remote machine, Example |
|
download a file from the cluster to the local machine through the bridge Example |
|
returns the content of a folder on the remote machine as a dataframe Example |
|
open a SSH connection and store the connection into the notebook workspace |
|
defines command |
|
upload a file to the remote machine, Example |
|
upload a file to the remote machine and then to the remote cluster, Example |
|
Defines |
|
Defines |
Documentation¶
Magic command to communicate with an Hadoop cluster (Linux).
- class pyenbc.remote.magic_remote_ssh.MagicRemoteSSH(**kwargs: Any)¶
Bases:
MagicClassWithHelpers
Defines commands to access a remote machine (bridge) through SSH, for the time being, all the command do not accept another parameters such as a SSH client which means only one connection can be opened at the same time.
The notebooks Python Hadoop Pig and Communication with a remote Linux machine through SSH show how these commands can be used.
Create a configurable given a config config.
- configConfig
If this is empty, default values are used. If config is a
Config
instance, it will be used to configure the instance.- parentConfigurable instance, optional
The parent Configurable instance of this object.
Subclasses of Configurable must call the
__init__()
method ofConfigurable
before doing anything else and usingsuper()
:class MyConfigurable(Configurable): def __init__(self, config=None): super(MyConfigurable, self).__init__(config=config) # Then any other code you need to finish initialization.
This ensures that instances will be configured properly.
- HIVE(line, cell=None)¶
defines command
%%HIVE
HIVE
The code for magic command
%HIVE
is equivalent to:with open(filename, "w", encoding="utf8") as f: f.write(script)
- static HIVE_parser()¶
defines the way to parse the magic command
%%HIVE
- PIG(line, cell=None)¶
defines command
%%PIG
PIG
The code for magic command
%PIG
is equivalent to:with open(filename, "w", encoding="utf8") as f: f.write(script)
- static PIG_parser()¶
defines the way to parse the magic command
%%PIG
- _replace_params(cell)¶
replaces parameter such
__PASSWORD__
by variable in the notebook environment- Parameters:
cell – string
- Returns:
modified string
- _trait_default_generators = {}¶
- close_remote_shell(line)¶
Defines
%close_remote_shell
- dfs_ls(line)¶
returns the content of a folder on the cluster as a dataframe
Example:
%dfs_ls .
dfs_ls
The code for magic command
%dfs_ls
is equivalent to:ssh = ASSHClient(server, username, password) ssh.connect() df = ssh.dfs_ls(args.path) ssh.close()
New in version 1.1.
- static dfs_ls_parser()¶
defines the way to parse the magic command
%dfs_ls
- dfs_mkdir(line)¶
Creates a folder on the cluster.
Example:
%dfs_mkdir afolder
dfs_mkdir
The code for magic command
%dfs_mkdir
is equivalent to:ssh = ASSHClient(server, username, password) ssh.connect() df = ssh.dfs_mkdir(path) ssh.close()
New in version 1.1.
- static dfs_mkdir_parser()¶
defines the way to parse the magic command
%dfs_mkdir
- dfs_rm(line)¶
remove a file on the cluster
Example:
%dfs_rm .
dfs_rm
The code for magic command
%dfs_rm
is equivalent to:ssh = ASSHClient(server, username, password) ssh.connect() df = ssh.dfs_rm(path, recursive=recursive) ssh.close()
New in version 1.1.
- static dfs_rm_parser()¶
defines the way to parse the magic command
%dfs_rm
- get_connection()¶
returns the connection stored in the workspace
- hive_submit(line)¶
defines command
%hive_submit
hive_submit
The code for magic command
%hive_submit
is equivalent to:ssh = ASSHClient(server, username, password) ssh.connect() out, err = ssh.hive_submit( pig, redirection=redirection, local=local) ssh.close()
- static hive_submit_parser()¶
defines the way to parse the magic command
%hive_submit
- job_syntax(line)¶
defines command
%job_syntax
- static job_syntax_parser()¶
defines the way to parse the magic command
%job_syntax
- open_remote_shell(line)¶
Defines
%open_remote_shell
open_remote_shell
The code for magic command
%open_remote_shell
is equivalent to:ssh = ASSHClient(server, username, password) ssh.connect() ssh.open_session(out_format=format) ssh.close()
- static open_remote_shell_parser()¶
defines the way to parse the magic command
%open_remote_shell
- pig_submit(line)¶
defines command
%pig_submit
pig_submit
The code for magic command
%pig_submit
is equivalent to:ssh = ASSHClient(server, username, password) ssh.connect() out, err = ssh.pig_submit( pig, dependencies=dependencies, redirection=redirection, local=local, stop_on_failure=stop_on_failure) ssh.close()
- static pig_submit_parser()¶
defines the way to parse the magic command
%pig_submit
- remote_close(line)¶
close a SSH connection and store the connection into the notebook workspace
remote_close
The code for magic command
%remote_close
is equivalent to:ssh = ASSHClient(server, username, password) # ... ssh.connect() ssh.close()
- remote_cmd(line, cell=None)¶
run a command on the remote machine
Example:
%remote_cmd ls
Or:
%%remote_cmd <something> anything going to stdin
In the second case, if __PASSWORD__ is found, it will be replaced by the password stored in workspace.
remote_cmd
The code for magic command
%remote_cmd
is equivalent to:ssh = ASSHClient(server, username, password) ssh.connect() out, err = ssh.execute_command( line, no_exception=True, fill_stdin=cell) ssh.close()
- remote_cmd_text(line, cell=None)¶
run a command on the remote machine and returns raw text (not HTML)
Example:
%remote_cmd_text ls
Or:
%%remote_cmd_text <something> anything going to stdin
In the second case, if __PASSWORD__ is found, it will be replaced by the password stored in workspace.
- remote_down(line)¶
download a file from the remote machine,
Example:
%remote_down remotepath localfile
the command does not allow spaces in files
remote_down
The code for magic command
%remote_down
is equivalent to:ssh = ASSHClient(server, username, password) ssh.connect() ssh.download(remotepath, localfile) ssh.close()
- remote_down_cluster(line)¶
download a file from the cluster to the local machine through the bridge
Example:
%remote_down_cluster remotepath localfile
the command does not allow spaces in files
remote_down_cluster
The code for magic command
%remote_down_cluster
is equivalent to:ssh = ASSHClient(server, username, password) ssh.connect() ssh.download_cluster(remotepath, localfile, merge=merge) ssh.close()
New in version 1.1.
- static remote_down_cluster_parser()¶
defines the way to parse the magic command
%remote_down_cluster
- static remote_down_parser()¶
defines the way to parse the magic command
%remote_down
- remote_ls(line)¶
returns the content of a folder on the remote machine as a dataframe
Example:
%remote_ls .
remote_ls
The code for magic command
%remote_ls
is equivalent to:ssh = ASSHClient(server, username, password) ssh.connect() df = ssh.ls(path) ssh.close()
New in version 1.1.
- static remote_ls_parser()¶
defines the way to parse the magic command
%remote_ls
- remote_open(line)¶
open a SSH connection and store the connection into the notebook workspace
remote_open
The code for magic command
%remote_open
is equivalent to:ssh = ASSHClient(server, username, password) ssh.connect()
- static remote_open_parser()¶
defines the way to parse the magic command
%remote_open
- remote_py(line)¶
defines command
%remote_py
remote_py
The code for magic command
%remote_py
is equivalent to:ssh = ASSHClient(server, username, password) ssh.connect() ssh.upload(filename, dest) args = " ".join('"{}"'.format(_) for _ in args.args) if args.args is not None else "" out, err = ssh.execute_command(exe + " " + dest + " " + args, no_exception=True) ssh.close()
- static remote_py_parser()¶
defines the way to parse the magic command
%remote_py
- remote_up(line)¶
upload a file to the remote machine,
Example:
%remote_up localfile remotepath
the command does not allow spaces in files
remote_up
The code for magic command
%remote_up
is equivalent to:ssh = ASSHClient(server, username, password) ssh.connect() ssh.upload(localfile, remotepath) ssh.close()
- remote_up_cluster(line)¶
upload a file to the remote machine and then to the remote cluster,
Example:
%remote_up_cluster localfile remotepath
the command does not allow spaces in files
remote_up_cluster
The code for magic command
%remote_up_cluster
is equivalent to:ssh = ASSHClient(server, username, password) ssh.connect() ssh.upload_cluster(localfile, remotepath) ssh.close()
New in version 1.1.
- static remote_up_cluster_parser()¶
defines the way to parse the magic command
%remote_up_cluster
- static remote_up_parser()¶
defines the way to parse the magic command
%remote_up
- shell_remote(line, cell=None)¶
Defines
%shell_remote
and%%shell_remote
- shell_remote_text(line, cell=None)¶
Defines
%shell_remote_text
and%%shell_remote_text
- pyenbc.remote.magic_remote_ssh.register_magics_ssh(ip=None)¶
register magics function, can be called from a notebook
- Parameters:
ip – from
get_ipython()