module helpgen.process_notebooks

Short summary

module pyquickhelper.helpgen.process_notebooks

Contains the main function to generate the documentation for a module designed the same way as this one, generate_help_sphinx().

source on GitHub

Functions

function

truncated documentation

_preprocess_notebook

Preprocesses the content of a notebook.

_process_notebooks_in

The notebook conversion does not handle images from url for pdf and docx. They could be downloaded first …

_process_notebooks_in_private

This function fails in nbconvert 6.0 when the conversion is called more than once. The conversion probably changes the …

_process_notebooks_in_private_cmd

add_link_to_notebook

Adds a link to the notebook in HTML format and does a little bit of cleaning for various format.

add_tag_for_slideshow

Modifies a notebook to add tag for a slideshow.

build_all_notebooks_coverage

Creates a rst page (gallery) with links to all notebooks and information about coverage. It relies on function …

build_notebooks_gallery

Creates a rst page (gallery) with links to all notebooks. For each notebook, it creates a snippet.

build_thumbail_in_gallery

Returns rst code for a notebook.

find_pdflatex

Returns the executable for latex.

process_notebooks

Converts notebooks into html, rst, latex, pdf, python, docx using …

Documentation

Contains the main function to generate the documentation for a module designed the same way as this one, generate_help_sphinx.

source on GitHub

pyquickhelper.helpgen.process_notebooks._preprocess_notebook(notebook_content)[source]

Preprocesses the content of a notebook.

Parameters:

notebook_content – notebook content

Returns:

modified content

source on GitHub

pyquickhelper.helpgen.process_notebooks._process_notebooks_in(notebooks, outfold, build, latex_path=None, pandoc_path=None, formats=('ipynb', 'html', 'python', 'rst', 'slides', 'pdf', 'github'), fLOG=<function fLOG>, exc=True, nblinks=None, remove_unicode_latex=False, notebook_replacements=None)[source]

The notebook conversion does not handle images from url for pdf and docx. They could be downloaded first and replaced by local files.

Note

nbconvert introduced a commit which breaks the conversion of notebooks in latex if they have a cell outputting svg (see PR 910).

Use xelatex if possible.

source on GitHub

pyquickhelper.helpgen.process_notebooks._process_notebooks_in_private(fnbcexe, list_args, options_args)[source]

This function fails in nbconvert 6.0 when the conversion is called more than once. The conversion probably changes the initial state.

source on GitHub

pyquickhelper.helpgen.process_notebooks._process_notebooks_in_private_cmd(fnbcexe, list_args, options_args, fLOG)[source]

Adds a link to the notebook in HTML format and does a little bit of cleaning for various format.

Parameters:

  • file – notebook.html

  • nb – notebook (.ipynb)

  • pdf – if True, add a link to the PDF, assuming it will exists at the same location

  • html – if True, add a link to the HTML conversion

  • python – if True, add a link to the Python conversion

  • slides – if True, add a link to the HTML slides

  • exc – raises an exception (True) or a warning (False)

  • github – add a link to the notebook on github

  • notebook – location of the notebook (file might be a copy)

  • nblinks – dictionary {ref: url}

  • notebook_replacements – stirng replacement in notebooks

  • fLOG – logging function

Returns:

list of generated files

The function does some cleaning too in the files.

source on GitHub

pyquickhelper.helpgen.process_notebooks.add_tag_for_slideshow(ipy, folder, encoding='utf8')[source]

Modifies a notebook to add tag for a slideshow.

Parameters:

  • ipy – notebook file

  • folder – where to write the new notebook

  • encoding – encoding

Returns:

written file

source on GitHub

pyquickhelper.helpgen.process_notebooks.build_all_notebooks_coverage(nbs, fileout, module_name, dump=None, badge=True, too_old=30, fLOG=<function noLOG>)[source]

Creates a rst page (gallery) with links to all notebooks and information about coverage. It relies on function notebook_coverage.

Parameters:

  • nbs – list of notebooks to consider or tuple(full path, rst),

  • fileout – file to create

  • module_name – module name

  • dump – dump containing information about notebook execution (or None for the default one)

  • badge – builds an image with the notebook coverage

  • too_old – drop executions older than too_old days from now

  • fLOG – logging function

Returns:

dataframe which contains the data

source on GitHub

Creates a rst page (gallery) with links to all notebooks. For each notebook, it creates a snippet.

Parameters:

  • nbs – list of notebooks to consider or tuple(full path, rst),

  • fileout – file to create

  • layout – 'classic' or 'table'

  • neg_pattern – do not consider notebooks matching this regular expression

  • snippet_folder – folder where to find custom snippet for notebooks, the snippet should have the same name as the notebook itself, snippet must have extension .png

  • fLOG – logging function

Returns:

created file name

Example for parameter nbs:

('challenges\city_tour\city_tour_1.ipynb',
    'ensae_projects\_doc\notebooks\challenges\city_tour\city_tour_1.ipynb')
('challenges\city_tour\city_tour_1_solution.ipynb',
    'ensae_projects\_doc\notebooks\challenges\city_tour\city_tour_1_solution.ipynb')
('challenges\city_tour\city_tour_data_preparation.ipynb',
    'ensae_projects\_doc\notebooks\challenges\city_tour\city_tour_data_preparation.ipynb')
('challenges\city_tour\city_tour_long.ipynb',
    'ensae_projects\_doc\notebooks\challenges\city_tour\city_tour_long.ipynb')
('cheat_sheets\chsh_files.ipynb',
    'ensae_projects\_doc\notebooks\cheat_sheets\chsh_files.ipynb')
('cheat_sheets\chsh_geo.ipynb',
    'ensae_projects\_doc\notebooks\cheat_sheets\chsh_geo.ipynb')

nbs can be a folder, in that case, the function will build the list of all notebooks in that folder. nbs can be a list of tuple. the function adds a thumbnail, organizes the list of notebook as a galley, it adds a link on notebook coverage. The function bypasses the generation of a snippet if a custom one was found.

source on GitHub

Returns rst code for a notebook.

Parameters:

  • nbfile – notebook file

  • folder_snippet – where to store the snippet

  • relative – the path to the snippet will be relative to this folder

  • rst_link – rst link

  • layout – 'classic' or 'table'

  • snippet_folder – folder where to find custom snippet for notebooks, the snippet should have the same name as the notebook itself, snippet must have extension .png

Returns:

RST

Modifies the function to bypass the generation of a snippet if a custom one was found. Parameter snippet_folder was added.

source on GitHub

pyquickhelper.helpgen.process_notebooks.find_pdflatex(latex_path)[source]

Returns the executable for latex.

Parameters:

latex_path – path to look (only on Windows)

Returns:

executable

source on GitHub

pyquickhelper.helpgen.process_notebooks.process_notebooks(notebooks, outfold, build, latex_path=None, pandoc_path=None, formats='ipynb, html, python, rst, slides, pdf, github', fLOG=<function fLOG>, exc=True, remove_unicode_latex=False, nblinks=None, notebook_replacements=None)[source]

Converts notebooks into html, rst, latex, pdf, python, docx using nbconvert.

Parameters:

  • notebooks – list of notebooks or comma separated values

  • outfold – folder which will contains the outputs

  • build – temporary folder which contains all produced files

  • pandoc_path – path to pandoc

  • formats – list of formats to convert into (pdf format means latex then compilation), or comma separated values

  • latex_path – path to the latex compiler

  • fLOG – logging function

  • exc – raises an exception (True) or a warning (False) if an error happens

  • nblinks – dictionary {ref: url} or a string in json format

  • remove_unicode_latex – remove unicode characters for latex (to avoid failing)

  • notebook_replacements – string replacement in a notebook before conversion or a string in json format

Returns:

list of tuple [(file, created or skipped)]

This function relies on pandoc. It also needs modules pywin32, pygments.

pywin32 might have some issues to find its DLL, look import_pywin32.

The latex compilation uses MiKTeX. The conversion into Word document directly uses pandoc. It still has an issue with table.

Some latex templates (for nbconvert) uses [commandchars=\\\{\}]{\|} which allows commands \\ and it does not compile. The one used here is report. Some others bugs can be found at: schlichtanders/latex_test.html. For example, you must not let spaces between symbol $ and the formulas it indicates.

If pandoc_path is None, uses find_pandoc_path to guess it. If latex_path is None, uses find_latex_path to guess it.

Convert a notebook into multiple formats

from pyquickhelper.ipythonhelper import process_notebooks
process_notebooks("td1a_correction_session7.ipynb",
                  "dest_folder", "dest_folder",
                  formats=("ipynb", "html", "python", "rst", "slides", "pdf",
                           "docx", "github")])

For latex and pdf, a custom processor was added to handle raw data and add \begin{verbatim} and \end{verbatim}. Format github adds a link to file on github.

Todo(bug): check differences between _process_notebooks_in_private and _process_notebooks_in_private_cmd

For latex and pdf, the custom preprocessor is not taken into account. by function _process_notebooks_in_private.

source on GitHub