Measure ONNX runtime performances#

The following example shows how to use the command line to compare one or two runtimes with scikit-learn. It relies on function validate_runtime which can be called from python or through a command line described in page Command lines.

Run the benchmark#

The following line creates a folder used to dump information about models which failed during the benchmark.

import os
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
import pandas
if not os.path.exists("dump_errors"):
    os.mkdir("dump_errors")

The benchmark can be run with a python instruction or a command line:

python -m mlprodict validate_runtime -v 1 --out_raw data.csv --out_summary summary.csv
           -b 1 --dump_folder dump_errors --runtime python,onnxruntime1
           --models LinearRegression,DecisionTreeRegressor
           --n_features 4,10 --out_graph bench_png
           -t "{\"1\":{\"number\":10,\"repeat\":10},\"10\":{\"number\":5,\"repeat\":5}}"

We use the python instruction in this example.

from mlprodict.cli import validate_runtime

validate_runtime(
    verbose=1,
    out_raw="data.csv", out_summary="summary.csv",
    benchmark=True, dump_folder="dump_errors",
    runtime=['python', 'onnxruntime1'],
    models=['LinearRegression', 'DecisionTreeRegressor'],
    n_features=[4, 10], dtype="32",
    out_graph="bench.png",
    time_kwargs={
        1: {"number": 100, "repeat": 100},
        10: {"number": 50, "repeat": 50},
        100: {"number": 40, "repeat": 50},
        1000: {"number": 40, "repeat": 40},
        10000: {"number": 20, "repeat": 20},
    }
)
RT/SKL-N=1, N=10, N=100, N=1000, N=10000
time_kwargs={1: {'number': 100, 'repeat': 100}, 10: {'number': 50, 'repeat': 50}, 100: {'number': 40, 'repeat': 50}, 1000: {'number': 40, 'repeat': 40}, 10000: {'number': 20, 'repeat': 20}}
[enumerate_validated_operator_opsets] opset in [17, None].

  0%|          | 0/2 [00:00<?, ?it/s]
LinearRegression            :   0%|          | 0/2 [00:00<?, ?it/s][enumerate_compatible_opset] opset in [17, None].

LinearRegression            :  50%|#####     | 1/2 [03:11<03:11, 191.39s/it]
DecisionTreeRegressor       :  50%|#####     | 1/2 [03:11<03:11, 191.39s/it][enumerate_compatible_opset] opset in [17, None].

DecisionTreeRegressor       : 100%|##########| 2/2 [10:35<00:00, 340.33s/it]
DecisionTreeRegressor       : 100%|##########| 2/2 [10:35<00:00, 317.99s/it]
Saving raw_data into 'data.csv'.
Saving summary into 'summary.csv'.
Saving graph into 'bench.png'.
findfont: Font family ['STIXGeneral'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXGeneral'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXGeneral'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXNonUnicode'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXNonUnicode'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXNonUnicode'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXSizeOneSym'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXSizeTwoSym'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXSizeThreeSym'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXSizeFourSym'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXSizeFiveSym'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmsy10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmr10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmtt10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmmi10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmb10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmss10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmex10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['DejaVu Sans Display'] not found. Falling back to DejaVu Sans.

Let’s show the results.

df = pandas.read_csv("summary.csv")
df.head(n=2).T
0 1
name DecisionTreeRegressor DecisionTreeRegressor
problem b-reg b-reg
scenario default default
optim NaN NaN
method_name predict predict
output_index 0 0
conv_options {} {}
inst {"random_state": 42} {"random_state": 42}
n_features 4 4
runtime onnxruntime1 python
skl_version 1.2.1 1.2.1
skl_nop 1 1
skl_ncoef -1.0 -1.0
skl_nlin -1.0 -1.0
skl_nnodes 223.0 223.0
skl_ntrees 1.0 1.0
skl_max_depth 13.0 13.0
onx_size 8943 8943
onx_nnodes 1 1
onx_ninits 0 0
onx_producer_name skl2onnx skl2onnx
onx_producer_version 1.13.1 1.13.1
onx_ai.onnx.ml 3 3
onx_size_optim 8943 8943
onx_nnodes_optim 1 1
onx_ninits_optim 0 0
opset17 OK 17/3 OK 17/3
RT/SKL-N=1 0.724179 0.311947
N=10 0.730205 0.316873
N=100 0.762137 31.396165
N=1000 0.586674 70.842404
N=10000 0.317246 122.252102
RT/SKL-N=1-min 0.673482 0.303278
RT/SKL-N=1-max 1.126067 0.316214
N=10-min 0.714238 0.311731
N=10-max 0.740292 0.322754
N=100-min 0.700777 0.395965
N=100-max 2.601827 75.713904
N=1000-min 0.56944 0.728227
N=1000-max 0.667581 311.160683
N=10000-min 0.312533 40.025484
N=10000-max 0.32857 587.093982


Let’s display the graph generated by the function.

img = mpimg.imread('bench.png')
fig = plt.imshow(img)
fig.axes.get_xaxis().set_visible(False)
fig.axes.get_yaxis().set_visible(False)
plt.show()
plot onnx benchmark

Total running time of the script: ( 10 minutes 43.597 seconds)

Gallery generated by Sphinx-Gallery