Examples¶
A python script which generates documentation
The following code prints the version of Python on the standard output. It is added to the documentation:
.. runpython::
:showcode:
import sys
print("sys.version_info=", str(sys.version_info))
If give the following results:
sys.version_info= sys.version_info(major=3, minor=9, micro=1, releaselevel='final', serial=0)
Options showcode can be used to display the code. The option rst will assume the output is in RST format and must be interpreted. showout will complement the RST output with the raw format.
(original entry : sphinx_runpython_extension.py:docstring of pyquickhelper.sphinxext.sphinx_runpython_extension.RunPythonDirective, line 4)
Clone a git repository
clone("local_folder", "github.com", "sdpython", "pyquickhelper")
(original entry : pygit_helper.py:docstring of pyquickhelper.loghelper.repositories.pygit_helper.clone, line 15)
Convert a dataframe into RST
<<<
from pandas import DataFrame
from pyquickhelper.pandashelper import df2rst
df = DataFrame([{'A': 0, 'B': 'text'},
{'A': 1e-5, 'C': 'longer text'}])
print(df2rst(df))
>>>
+-------+------+-------------+
| A | B | C |
+=======+======+=============+
| 0.0 | text | |
+-------+------+-------------+
| 1e-05 | | longer text |
+-------+------+-------------+
(original entry : tblformat.py:docstring of pyquickhelper.pandashelper.tblformat.df2rst, line 63)
Convert a dataframe into markdown
<<<
from io import StringIO
from textwrap import dedent
import pandas
from_excel = dedent('''
Op;axes;shape;SpeedUp
ReduceMax;(3,);(8, 24, 48, 8);2.96
ReduceMax;(3,);(8, 24, 48, 16);2.57
ReduceMax;(3,);(8, 24, 48, 32);2.95
ReduceMax;(3,);(8, 24, 48, 64);3.28
ReduceMax;(3,);(8, 24, 48, 100);3.05
ReduceMax;(3,);(8, 24, 48, 128);3.11
ReduceMax;(3,);(8, 24, 48, 200);2.86
ReduceMax;(3,);(8, 24, 48, 256);2.50
ReduceMax;(3,);(8, 24, 48, 400);2.48
ReduceMax;(3,);(8, 24, 48, 512);2.90
ReduceMax;(3,);(8, 24, 48, 1024);2.76
ReduceMax;(0,);(8, 24, 48, 8);19.29
ReduceMax;(0,);(8, 24, 48, 16);11.83
ReduceMax;(0,);(8, 24, 48, 32);5.69
ReduceMax;(0,);(8, 24, 48, 64);5.49
ReduceMax;(0,);(8, 24, 48, 100);6.13
ReduceMax;(0,);(8, 24, 48, 128);6.27
ReduceMax;(0,);(8, 24, 48, 200);5.46
ReduceMax;(0,);(8, 24, 48, 256);4.76
ReduceMax;(0,);(8, 24, 48, 400);2.21
ReduceMax;(0,);(8, 24, 48, 512);4.52
ReduceMax;(0,);(8, 24, 48, 1024);4.38
ReduceSum;(3,);(8, 24, 48, 8);1.79
ReduceSum;(3,);(8, 24, 48, 16);0.79
ReduceSum;(3,);(8, 24, 48, 32);1.67
ReduceSum;(3,);(8, 24, 48, 64);1.19
ReduceSum;(3,);(8, 24, 48, 100);2.08
ReduceSum;(3,);(8, 24, 48, 128);2.96
ReduceSum;(3,);(8, 24, 48, 200);1.66
ReduceSum;(3,);(8, 24, 48, 256);2.26
ReduceSum;(3,);(8, 24, 48, 400);1.76
ReduceSum;(3,);(8, 24, 48, 512);2.61
ReduceSum;(3,);(8, 24, 48, 1024);2.21
ReduceSum;(0,);(8, 24, 48, 8);2.56
ReduceSum;(0,);(8, 24, 48, 16);2.05
ReduceSum;(0,);(8, 24, 48, 32);3.04
ReduceSum;(0,);(8, 24, 48, 64);2.57
ReduceSum;(0,);(8, 24, 48, 100);2.41
ReduceSum;(0,);(8, 24, 48, 128);2.77
ReduceSum;(0,);(8, 24, 48, 200);2.02
ReduceSum;(0,);(8, 24, 48, 256);1.61
ReduceSum;(0,);(8, 24, 48, 400);1.59
ReduceSum;(0,);(8, 24, 48, 512);1.48
ReduceSum;(0,);(8, 24, 48, 1024);1.50
''')
df = pandas.read_csv(StringIO(from_excel), sep=";")
print(df.columns)
sub = df[["Op", "axes", "shape", "SpeedUp"]]
piv = df.pivot_table(values="SpeedUp", index=['axes', "shape"], columns="Op")
piv = piv.reset_index(drop=False)
print(piv.to_markdown(index=False))
>>>
Index(['Op', 'axes', 'shape', 'SpeedUp'], dtype='object')
| axes | shape | ReduceMax | ReduceSum |
|:-------|:------------------|------------:|------------:|
| (0,) | (8, 24, 48, 100) | 6.13 | 2.41 |
| (0,) | (8, 24, 48, 1024) | 4.38 | 1.5 |
| (0,) | (8, 24, 48, 128) | 6.27 | 2.77 |
| (0,) | (8, 24, 48, 16) | 11.83 | 2.05 |
| (0,) | (8, 24, 48, 200) | 5.46 | 2.02 |
| (0,) | (8, 24, 48, 256) | 4.76 | 1.61 |
| (0,) | (8, 24, 48, 32) | 5.69 | 3.04 |
| (0,) | (8, 24, 48, 400) | 2.21 | 1.59 |
| (0,) | (8, 24, 48, 512) | 4.52 | 1.48 |
| (0,) | (8, 24, 48, 64) | 5.49 | 2.57 |
| (0,) | (8, 24, 48, 8) | 19.29 | 2.56 |
| (3,) | (8, 24, 48, 100) | 3.05 | 2.08 |
| (3,) | (8, 24, 48, 1024) | 2.76 | 2.21 |
| (3,) | (8, 24, 48, 128) | 3.11 | 2.96 |
| (3,) | (8, 24, 48, 16) | 2.57 | 0.79 |
| (3,) | (8, 24, 48, 200) | 2.86 | 1.66 |
| (3,) | (8, 24, 48, 256) | 2.5 | 2.26 |
| (3,) | (8, 24, 48, 32) | 2.95 | 1.67 |
| (3,) | (8, 24, 48, 400) | 2.48 | 1.76 |
| (3,) | (8, 24, 48, 512) | 2.9 | 2.61 |
| (3,) | (8, 24, 48, 64) | 3.28 | 1.19 |
| (3,) | (8, 24, 48, 8) | 2.96 | 1.79 |
(original entry : tblformat.py:docstring of pyquickhelper.pandashelper.tblformat.df2rst, line 76)
Convert a notebook into multiple formats
from pyquickhelper.ipythonhelper import process_notebooks
process_notebooks("td1a_correction_session7.ipynb",
"dest_folder", "dest_folder",
formats=("ipynb", "html", "python", "rst", "slides", "pdf",
"docx", "github")])
(original entry : process_notebooks.py:docstring of pyquickhelper.helpgen.process_notebooks.process_notebooks, line 44)
Convert a notebook into slides
By default, the function automatically adds sections if there is none and it copies the javascript from reveal.js at the right place.
from pyquickhelper.helpgen import nb2slides
nb2slides("nb.ipynb", "convert.slides.html")
(original entry : process_notebook_api.py:docstring of pyquickhelper.helpgen.process_notebook_api.nb2slides, line 12)
Display the call stack
<<<
from pyquickhelper.pycode import get_call_stack
print(get_call_stack())
>>>
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/src/pyquickhelper/helpgen/process_sphinx_cmd.py", line 23, in <module>
main()
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/src/pyquickhelper/helpgen/process_sphinx_cmd.py", line 19, in main
run_sphinx_build(sys.argv)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/src/pyquickhelper/helpgen/process_sphinx_cmd.py", line 15, in run_sphinx_build
build_main(argv=argv[1:])
File "somewhere/.local/lib/python3.9/site-packages/sphinx/cmd/build.py", line 290, in main
return build_main(argv)
File "somewhere/.local/lib/python3.9/site-packages/sphinx/cmd/build.py", line 276, in build_main
app.build(args.force_all, filenames)
File "somewhere/.local/lib/python3.9/site-packages/sphinx/application.py", line 329, in build
self.builder.build_update()
File "somewhere/.local/lib/python3.9/site-packages/sphinx/builders/__init__.py", line 288, in build_update
self.build(to_build,
File "somewhere/.local/lib/python3.9/site-packages/sphinx/builders/__init__.py", line 302, in build
updated_docnames = set(self.read())
File "somewhere/.local/lib/python3.9/site-packages/sphinx/builders/__init__.py", line 409, in read
self._read_serial(docnames)
File "somewhere/.local/lib/python3.9/site-packages/sphinx/builders/__init__.py", line 430, in _read_serial
self.read_doc(docname)
File "somewhere/.local/lib/python3.9/site-packages/sphinx/builders/__init__.py", line 483, in read_doc
publisher.publish()
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/core.py", line 217, in publish
self.document = self.reader.read(self.source, self.parser,
File "somewhere/.local/lib/python3.9/site-packages/sphinx/io.py", line 103, in read
self.parse()
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/readers/__init__.py", line 78, in parse
self.parser.parse(self.input, document)
File "somewhere/.local/lib/python3.9/site-packages/sphinx/parsers.py", line 78, in parse
self.statemachine.run(inputlines, document, inliner=self.inliner)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 170, in run
results = StateMachineWS.run(self, input_lines, input_offset,
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/statemachine.py", line 239, in run
context, next_state, result = self.check_line(
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/statemachine.py", line 451, in check_line
return method(match, context, next_state)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2769, in underline
self.section(title, source, style, lineno - 1, messages)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 327, in section
self.new_subsection(title, lineno, messages)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 393, in new_subsection
newabsoffset = self.nested_parse(
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 281, in nested_parse
state_machine.run(block, input_offset, memo=self.memo,
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 196, in run
results = StateMachineWS.run(self, input_lines, input_offset)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/statemachine.py", line 239, in run
context, next_state, result = self.check_line(
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/statemachine.py", line 451, in check_line
return method(match, context, next_state)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2769, in underline
self.section(title, source, style, lineno - 1, messages)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 327, in section
self.new_subsection(title, lineno, messages)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 393, in new_subsection
newabsoffset = self.nested_parse(
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 281, in nested_parse
state_machine.run(block, input_offset, memo=self.memo,
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 196, in run
results = StateMachineWS.run(self, input_lines, input_offset)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/statemachine.py", line 239, in run
context, next_state, result = self.check_line(
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/statemachine.py", line 451, in check_line
return method(match, context, next_state)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2342, in explicit_markup
nodelist, blank_finish = self.explicit_construct(match)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2354, in explicit_construct
return method(self, expmatch)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2096, in directive
return self.run_directive(
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2146, in run_directive
result = directive_instance.run()
File "somewhere/.local/lib/python3.9/site-packages/sphinx/ext/autodoc/directive.py", line 159, in run
result = parse_generated_content(self.state, params.result, documenter)
File "somewhere/.local/lib/python3.9/site-packages/sphinx/ext/autodoc/directive.py", line 102, in parse_generated_content
nested_parse_with_titles(state, content, node)
File "somewhere/.local/lib/python3.9/site-packages/sphinx/util/nodes.py", line 340, in nested_parse_with_titles
return state.nested_parse(content, 0, node, match_titles=1)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 281, in nested_parse
state_machine.run(block, input_offset, memo=self.memo,
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 196, in run
results = StateMachineWS.run(self, input_lines, input_offset)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/statemachine.py", line 239, in run
context, next_state, result = self.check_line(
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/statemachine.py", line 451, in check_line
return method(match, context, next_state)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2344, in explicit_markup
self.explicit_list(blank_finish)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2369, in explicit_list
newline_offset, blank_finish = self.nested_list_parse(
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 318, in nested_list_parse
state_machine.run(block, input_offset, memo=self.memo,
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 196, in run
results = StateMachineWS.run(self, input_lines, input_offset)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/statemachine.py", line 239, in run
context, next_state, result = self.check_line(
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/statemachine.py", line 451, in check_line
return method(match, context, next_state)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2647, in explicit_markup
nodelist, blank_finish = self.explicit_construct(match)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2354, in explicit_construct
return method(self, expmatch)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2096, in directive
return self.run_directive(
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2146, in run_directive
result = directive_instance.run()
File "somewhere/.local/lib/python3.9/site-packages/sphinx/domains/__init__.py", line 281, in run
return super().run()
File "somewhere/.local/lib/python3.9/site-packages/sphinx/directives/__init__.py", line 199, in run
self.state.nested_parse(self.content, self.content_offset, contentnode)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 281, in nested_parse
state_machine.run(block, input_offset, memo=self.memo,
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 196, in run
results = StateMachineWS.run(self, input_lines, input_offset)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/statemachine.py", line 239, in run
context, next_state, result = self.check_line(
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/statemachine.py", line 451, in check_line
return method(match, context, next_state)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2342, in explicit_markup
nodelist, blank_finish = self.explicit_construct(match)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2354, in explicit_construct
return method(self, expmatch)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2096, in directive
return self.run_directive(
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2146, in run_directive
result = directive_instance.run()
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_doc/sphinxdoc/source/pyquickhelper/sphinxext/sphinx_exref_extension.py", line 100, in run
return BlocRef.run(self)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_doc/sphinxdoc/source/pyquickhelper/sphinxext/sphinx_blocref_extension.py", line 137, in run
return self.private_run()
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_doc/sphinxdoc/source/pyquickhelper/sphinxext/sphinx_blocref_extension.py", line 165, in private_run
(blocref,) = super(BlocRef, self).run()
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/directives/admonitions.py", line 46, in run
self.state.nested_parse(self.content, self.content_offset,
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 281, in nested_parse
state_machine.run(block, input_offset, memo=self.memo,
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 196, in run
results = StateMachineWS.run(self, input_lines, input_offset)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/statemachine.py", line 239, in run
context, next_state, result = self.check_line(
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/statemachine.py", line 451, in check_line
return method(match, context, next_state)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2342, in explicit_markup
nodelist, blank_finish = self.explicit_construct(match)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2354, in explicit_construct
return method(self, expmatch)
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2096, in directive
return self.run_directive(
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_venv/lib/python3.9/site-packages/docutils/parsers/rst/states.py", line 2146, in run_directive
result = directive_instance.run()
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_doc/sphinxdoc/source/pyquickhelper/sphinxext/sphinx_runpython_extension.py", line 596, in run
out, err, context = run_python_script(script, comment=comment, setsysvar=p['setsysvar'],
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_doc/sphinxdoc/source/pyquickhelper/sphinxext/sphinx_runpython_extension.py", line 246, in run_python_script
exec(obj, globs, loc)
File "", line 5, in <module>
File "", line 4, in run_python_script_140023386919104
File "somewhere/workspace/pyquickhelper/pyquickhelper_UT_39_std/_doc/sphinxdoc/source/pyquickhelper/pycode/trace_execution.py", line 39, in get_call_stack
traceback.print_stack(file=s)
(original entry : trace_execution.py:docstring of pyquickhelper.pycode.trace_execution.get_call_stack, line 4)
Encrypted and compressed backup
Here is an example which stores everything on hard drive. A second run only modifies files updated between the two processes. A modified file does not remove the previous version, it creates a new file. Example:
from pyquickhelper.loghelper import fLOG
from pyquickhelper.filehelper import FileTreeNode, EncryptedBackup
from pyensae.remote import TransferAPIFile
key_crypt = "crypt"
local = os.path.normpath(os.path.join(os.path.dirname(__file__), ".."))
this = os.path.normpath(os.path.dirname(__file__))
file_status=os.path.join(this, "backup_status.txt")
file_map=os.path.join(this, "backup_mapping.txt")
backup = True
if backup:
# code to backup
root = os.path.normpath(os.path.join(os.path.dirname(__file__)))
api = TransferAPIFile("f:\\mycryptedbackup")
ft = FileTreeNode(root, repository=True)
enc = EncryptedBackup(
key=key_crypt,
file_tree_node=ft,
transfer_api=api,
root_local=local,
file_status=file_status,
file_map=file_map,
fLOG=print)
enc.start_transfering()
restore = not backup
if restore:
# code to restore
root = os.path.normpath(os.path.join(os.path.dirname(__file__)))
api = TransferAPIFile("f:\\mycryptedbackup")
enc = EncryptedBackup(
key=key_crypt,
file_tree_node=None,
transfer_api=api,
root_local=local,
file_status=file_status,
file_map=file_map,
fLOG=print)
dest=os.path.join(this, "_temp")
enc.retrieve_all(dest)
(original entry : encrypted_backup.py:docstring of pyquickhelper.filehelper.encrypted_backup.EncryptedBackup, line 6)
File autocompletion in IPython
The following code:
from pyquickhelper.ipythonhelper import AutoCompletionFile
d = AutoCompletionFile(".")
Will produce the following auto completion picture:
(original entry : kindofcompletion.py:docstring of pyquickhelper.ipythonhelper.kindofcompletion.AutoCompletionFile, line 4)
Hierarchical display for a profiling
pyinstrument has a nice display to show
time spent and call stack at the same time. This function
tries to replicate that display based on the results produced
by module cProfile. Here is an example.
<<<
import time
from pyquickhelper.pycode.profiling import profile, profile2graph
def fct0(t):
time.sleep(t)
def fct1(t):
time.sleep(t)
def fct2():
fct1(0.1)
fct1(0.01)
def fct3():
fct0(0.2)
fct1(0.5)
def fct4():
fct2()
fct3()
ps = profile(fct4)[0]
root, nodes = profile2graph(ps, clean_text=lambda x: x.split('/')[-1])
text = root.to_text()
print(text)
>>>
fct1 -- 3 3 -- 0.00002 0.61090 -- :11:fct1 (fct1)
<built-in method time.sleep> -- 3 3 -- 0.61088 0.61088 -- ~:0:<built-in method time.sleep> (<built-in method time.sleep>) +++
fct4 -- 1 1 -- 0.00001 0.81120 -- :25:fct4 (fct4)
fct2 -- 1 1 -- 0.00001 0.11034 -- :15:fct2 (fct2)
fct1 -- 2 2 -- 0.00002 0.11034 -- :11:fct1 (fct1) +++
fct3 -- 1 1 -- 0.00001 0.70085 -- :20:fct3 (fct3)
fct0 -- 1 1 -- 0.00000 0.20027 -- :7:fct0 (fct0)
<built-in method time.sleep> -- 1 1 -- 0.20027 0.20027 -- ~:0:<built-in method time.sleep> (<built-in method time.sleep>) +++
fct1 -- 1 1 -- 0.00001 0.50057 -- :11:fct1 (fct1) +++
<built-in method time.sleep> -- 4 4 -- 0.81115 0.81115 -- ~:0:<built-in method time.sleep> (<built-in method time.sleep>)
(original entry : profiling.py:docstring of pyquickhelper.pycode.profiling.profile2graph, line 10)
How to display a formula
We want to check this formula to successfully converted.
Brackets and backslashes might be an issue.
(original entry : utils_sphinx_doc_helpers.py:docstring of pyquickhelper.helpgen.utils_sphinx_doc_helpers.example_function_latex, line 4)
How to test a Sphinx directive?
The following code defines a simple directive definedbased on an existing one. It also defined what to do if a new node is inserted in the documentation.
from docutils import nodes
from pyquickhelper.helpgen import rst2html
class runpythonthis_node(nodes.Structural, nodes.Element):
pass
class RunPythonThisDirective (RunPythonDirective):
runpython_class = runpythonthis_node
def visit_node(self, node):
self.body.append("<p><b>visit_node</b></p>")
def depart_node(self, node):
self.body.append("<p><b>depart_node</b></p>")
content = '''
test a directive
================
.. runpythonthis::
print("this code shoud appear" + "___")
'''.replace(" ", "")
# to remove spaces at the beginning of the line
tives = [ ("runpythonthis", RunPythonThisDirective,
runpythonthis_node, visit_node, depart_node) ]
html = rst2html(content, writer="html", keep_warnings=True,
directives=tives)
Unfortunately, this functionality is only tested on Python 3. It might not work on Python 2.7. The function produces files if the document contains latex converted into image.
(original entry : rst_converters.py:docstring of pyquickhelper.helpgen.rst_converters.rst2html, line 64)
List files from FTP site
from pyquickhelper.filehelper import TransferFTP
ftp = TransferFTP("ftp....", "login", "password")
res = ftp.ls("path")
for v in res:
print(v["name"])
ftp.close()
(original entry : ftp_transfer.py:docstring of pyquickhelper.filehelper.ftp_transfer.TransferFTP.ls, line 8)
Open a add a form in a notebook to ask parameters to a user
Cell 1:
from pyquickhelper.ipythonhelper import open_html_form
params = { "module":, "version":"v..." }
open_html_form (params, title="try the password *", key_save="form1")
Cell 2:
print(form1)
We can execute a simple action after the button Ok is pressed. This second trick
comes from this notebook.
The code displays whatever comes from function custom_action in this case.
You should return "" to display nothing.
def custom_action(x):
x["combined"] = x["first_name"] + " " + x["last_name"]
return x
params = { "first_name":"", "last_name":"" }
open_html_form(params, title="enter your name", key_save="my_address",
hook="custom_action(my_address)")
(original entry : html_forms.py:docstring of pyquickhelper.ipythonhelper.html_forms.open_html_form, line 19)
Produce HTML documentation for a function or class
The following code can display the dosstring in HTML format to display it in a notebook.
from pyquickhelper.helpgen import docstring2html
import sklearn.linear_model
docstring2html(sklearn.linear_model.LogisticRegression)
(original entry : rst_converters.py:docstring of pyquickhelper.helpgen.rst_converters.docstring2html, line 25)
Run a notebook end to end
from pyquickhelper.ipythonhelper import run_notebook
run_notebook("source.ipynb", working_dir="temp",
outfilename="modified.ipynb",
additional_path=["custom_path"] )
(original entry : run_notebook.py:docstring of pyquickhelper.ipythonhelper.run_notebook.run_notebook, line 39)
Run a program using the command line
from pyquickhelper.loghelper import run_cmd
out, err = run_cmd("python setup.py install", wait=True)
(original entry : run_cmd.py:docstring of pyquickhelper.loghelper.run_cmd.run_cmd, line 34)
Run help generation
# from the main folder which contains folder src or the sources
generate_help_sphinx("pyquickhelper")
(original entry : sphinx_main.py:docstring of pyquickhelper.helpgen.sphinx_main.generate_help_sphinx, line 59)
Simple configuration file for Sphinx
We assume a module is configurated using the same
structure as pyquickhelper.
The file conf.py could just contain:
# -*- coding: utf-8 -*-
import sys, os, datetime, re
import solar_theme
from pyquickhelper.helpgen.default_conf import set_sphinx_variables
sys.path.insert(0, os.path.abspath(os.path.join(os.path.split(__file__)[0])))
set_sphinx_variables(__file__, "pyquickhelper", "Xavier Dupré", 2014,
"solar_theme", solar_theme.theme_path, locals())
# custom settings
...
setup.py must contain a string such as __version__ = 3.4.
Close to the setup, there must be a file version.txt.
You overwrite a value by giving a variable another value after the fucntion is called.
Some parts of the code can be disabled before generating the documentation. Those parts are surrounded by:
# -- HELP BEGIN EXCLUDE --
import module
# -- HELP END EXCLUDE --
If enable_disabled_parts is set to a string, these sections will become:
# -- HELP BEGIN EXCLUDE --
if hasattr(sys, <enable_disabled_parts>) and sys.<enable_disabled_parts>:
import module
# -- HELP END EXCLUDE --
(original entry : default_conf.py:docstring of pyquickhelper.helpgen.default_conf.set_sphinx_variables, line 43)
Transfer files to webste through FTP
Simple sketch to transfer a list of files to
a website through FTP
ftp = TransferFTP('ftp.<website>', alias, password, fLOG=print)
issues = [ ]
done = [ ]
notdone = [ ]
for file in files :
try :
r = ftp.transfer (file, path)
if r : done.append( (file, path) )
else : notdone.append ( (file, path) )
except Exception as e :
issues.append( (file, e) )
try :
ftp.close()
except Exception as e :
print ("unable to close FTP connection using ftp.close")
(original entry : ftp_transfer.py:docstring of pyquickhelper.filehelper.ftp_transfer.TransferFTP, line 4)
Transfer updated files to a website
The following code shows how to transfer the content of a folder to website through FTP protocol.
ftn = FileTreeNode("c:/somefolder")
ftp = TransferFTP("ftp.website.fr", "login", "password", fLOG=print)
fftp = FolderTransferFTP (ftn, ftp, "status_file.txt",
root_web = "/www/htdocs/app/pyquickhelper/helpsphinx")
fftp.start_transfering()
ftp.close()
(original entry : ftp_transfer_files.py:docstring of pyquickhelper.filehelper.ftp_transfer_files.FolderTransferFTP, line 5)
Visualize the difference between two text files or strings
with open("file1.txt","r",encoding="utf8") as f:
text1 = f.read()
with open("file2.txt","r",encoding="utf8") as f:
text2 = f.read()
pg = create_visual_diff_through_html(text1,text2)
with open("page.html","w",encoding="utf8") as f:
f.write(pg)
import webbrowser
webbrowser.open("page.html")
(original entry : visual_sync.py:docstring of pyquickhelper.filehelper.visual_sync.create_visual_diff_through_html, line 20)
synchronize two folders
The following function synchronizes a folder with another one
on a USB drive or a network drive. To minimize the number of access
to the other location, it stores the status of the previous
synchronization in a file (status_copy.txt in the below example).
Next time, the function goes through the directory and sub-directories
to synchronize and only propagates the modifications which happened
since the last modification.
The function filter_copy defines what file to synchronize or not.
def filter_copy(file):
return "_don_t_synchronize_" not in file
synchronize_folder( "c:/mydata",
"g:/mybackup",
hash_size = 0,
filter_copy = filter_copy,
file_date = "c:/status_copy.txt")
The function is able to go through 90.000 files and 90 Gb in 12 minutes (for an update).
(original entry : synchelper.py:docstring of pyquickhelper.filehelper.synchelper.synchronize_folder, line 41)
