module profiling.event_profiler

Short summary

module cpyquickhelper.profiling.event_profiler

Profiling class.

source on GitHub

Classes

class	truncated documentation
EventProfiler	This profiler profiles both memory and function calls. It stores events and produces a timeline. See EventProfiler …
EventProfilerDebug	One class to measure time wasted by profiling.
WithEventProfiler	Enables profiling with the following syntax:

Properties

property	truncated documentation
n_columns	Returns the number of stored informations by the profiler in memory. This corresponds to the number of columns returned …
n_columns	Returns the number of stored informations by the profiler in memory. This corresponds to the number of columns returned …
report	Returns the profiling report as a dataframe.

Methods

method	truncated documentation
__enter__
__exit__
__init__
__init__
__init__
_choose	Returns a string which represents the called function. When the function is built-in, frame contains the function …
_choose	Returns a string which represents the called function. When the function is built-in, frame contains the function …
_choose_mod	Returns a string which represents the module of the called function. When the function is built-in, frame contains …
_choose_mod	Returns a string which represents the module of the called function. When the function is built-in, frame contains …
_empty_cache	Empties the cache. This function logs a couple of events. The cache must contains enough place to log them. …
_empty_cache	Empties the cache. This function logs a couple of events. The cache must contains enough place to log them. …
_restore_profiler	This relies on sys.setprofile() and sys.getprofile().
_restore_profiler	This relies on sys.setprofile() and sys.getprofile().
_setup_profiler	This relies on sys.setprofile() and sys.getprofile().
_setup_profiler	This relies on sys.setprofile() and sys.getprofile().
log_event	Logs an event in the database.
log_event	Logs an event in the database.
retrieve_raw_results	Retrieves the raw results. Difficult to interpret.
retrieve_raw_results	Retrieves the raw results. Difficult to interpret.
retrieve_results	Retrieves the raw results. Difficult to interpret.
retrieve_results	Retrieves the raw results. Difficult to interpret.
start	Starts the profiling.
start	Starts the profiling without enabling it.
stop	Stops the profiling.
stop	Stops the unstarted profiling.

Documentation

@file @brief Profiling class.

class cpyquickhelper.profiling.event_profiler.EventProfiler(size=1000000, impl='python')

Bases: object

This profiler profiles both memory and function calls. It stores events and produces a timeline. See EventProfiler for an example of use.

Parameters:

• size – size of the buffer to store events

• impl – different implementation of the same function (‘python’, ‘pybind11’, ‘c’)

The profiler stores every event about function calls and returns, and memory allocation. It does not give the time spent in every function but a timeline with the series of events which occured. There are two pieces:

• A C++ class which logs every event into a bug buffer, every event is described by five int64 (id_frame, id_arg, event, timestamp, value1, value2).

• A python class which wraps the first one.

Function calls are caught by using sys.setprofile(). Memory calls are caught by using PyMem_SetAllocator. Once the profiled code has ended, results are returned by @see meth retrieve_results. This function extends the information stored in the C++ class with file names, function names, line numbers, memory deallocated.

Note

The program crashes if the profiled code raises an exception. The default memory allocator is not restored and any allocation is logged into a buffer which was deleted. The exception must be caught or class @see cl WithEventProfiler must be used.

About parameter impl, the question is related to the implementation of method log_event. A call to the python implementation takes 2-3 microseconds, a call to the pybind11 implementation takes 1-2 microseconds.

__init__(size=1000000, impl='python')

_choose(frame, arg, f_back=False)

Returns a string which represents the called function. When the function is built-in, frame contains the function calling the built-in, arg is then a pointer on this function. This method returns its name.

_choose_mod(frame, arg, clean_name, f_back=False)

Returns a string which represents the module of the called function. When the function is built-in, frame contains the function calling the built-in, arg is then a pointer on this function. This method returns its module.

_empty_cache()

Empties the cache. This function logs a couple of events. The cache must contains enough place to log them.

_event_mapping = {'c_call': 3, 'c_exception': 6, 'c_return': 4, 'call': 1, 'calloc': 1001, 'exception': 5, 'free': 1003, 'malloc': 1000, 'profiler_call': 10, 'profiler_return': 11, 'profiler_sort': 101, 'profiler_start': 100, 'realloc': 1002, 'realloc_free': 1004, 'return': 2}

_restore_profiler()

This relies on sys.setprofile() and sys.getprofile().

_setup_profiler()

This relies on sys.setprofile() and sys.getprofile().

log_event(frame, event, arg)

Logs an event in the database.

Parameters:

• frame – (frame), see inspect

• event – (str) kind of event

• arg – None or…

property n_columns

Returns the number of stored informations by the profiler in memory. This corresponds to the number of columns returned by @see meth retrieve_raw_results.

retrieve_raw_results(empty_cache=True)

Retrieves the raw results. Difficult to interpret.

Parameters:

empty_cache – retrives the data from the C++ container and stores it in a numpy array before

Returns:

numpy array

retrieve_results(empty_cache=True, clean_file_name=None)

Retrieves the raw results. Difficult to interpret.

Parameters:

• empty_cache – retrives the data from the C++ container and stores it in a numpy array before

• clean_file_name – function which cleans the file name

Returns:

numpy array

start()

Starts the profiling.

stop()

Stops the profiling.

class cpyquickhelper.profiling.event_profiler.EventProfilerDebug(size=1000000, impl='python')

Bases: EventProfiler

One class to measure time wasted by profiling.

start()

Starts the profiling without enabling it.

stop()

Stops the unstarted profiling.

class cpyquickhelper.profiling.event_profiler.WithEventProfiler(size=1000000, impl='python', clean_file_name=None)

Bases: object

Enables profiling with the following syntax:

prof = WithEventProfiler()
with prof:
    # code to profile
    # may raise an exception

The class restores the memory allocator and stops logging any event even if an exception was raised.

Parameters:

• size – size of the buffer to store events

• impl – different implementation of the same function (‘python’, ‘pybind11’)

• clean_file_name – function uses to clean or shorten the file name saved in the report.

__enter__()

__exit__(typ, value, traceback)

__init__(size=1000000, impl='python', clean_file_name=None)

property report

Returns the profiling report as a dataframe.