Benchmark inference for scikit-learn models#

This short code compares the execution of a couple of runtime for inference including onnxruntime. It uses examples Measure ONNX runtime performances. It is an automated process to compare the performance of a model against scikit-learn. This model is a simple model taken from all implemented by scikit-learn.

Linear Regression#

from pandas import read_csv
from mlprodict.cli import validate_runtime
from mlprodict.plotting.plotting import plot_validate_benchmark

res = validate_runtime(
    verbose=1,
    out_raw="data.csv", out_summary="summary.csv",
    benchmark=True, dump_folder="dump_errors",
    runtime=['python', 'onnxruntime1'],
    models=['LinearRegression'],
    skip_models=['LinearRegression[m-reg]'],
    n_features=[10, 50], dtype="32",
    out_graph="bench.png",
    opset_min=15, opset_max=15,
    time_kwargs={
        1: {"number": 50, "repeat": 50},
        10: {"number": 25, "repeat": 25},
        100: {"number": 20, "repeat": 20},
        1000: {"number": 20, "repeat": 20},
        10000: {"number": 10, "repeat": 10},
    }
)

results = read_csv('summary.csv')
results
RT/SKL-N=1, N=10, N=100, N=1000, N=10000

Out:

time_kwargs={1: {'number': 50, 'repeat': 50}, 10: {'number': 25, 'repeat': 25}, 100: {'number': 20, 'repeat': 20}, 1000: {'number': 20, 'repeat': 20}, 10000: {'number': 10, 'repeat': 10}}
[enumerate_validated_operator_opsets] opset in [15, 15].

  0%|          | 0/1 [00:00<?, ?it/s]
LinearRegression            :   0%|          | 0/1 [00:00<?, ?it/s][enumerate_compatible_opset] opset in [15, 15].

LinearRegression            : 100%|##########| 1/1 [00:43<00:00, 43.45s/it]
LinearRegression            : 100%|##########| 1/1 [00:43<00:00, 43.46s/it]
Saving raw_data into 'data.csv'.
Saving summary into 'summary.csv'.
Saving graph into 'bench.png'.
findfont: Font family ['STIXGeneral'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXGeneral'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXGeneral'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXNonUnicode'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXNonUnicode'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXNonUnicode'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXSizeOneSym'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXSizeTwoSym'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXSizeThreeSym'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXSizeFourSym'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXSizeFiveSym'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmsy10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmr10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmtt10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmmi10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmb10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmss10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmex10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['DejaVu Sans Display'] not found. Falling back to DejaVu Sans.
name problem scenario optim method_name output_index conv_options inst n_features runtime skl_version skl_nop skl_ncoef skl_nlin onx_size onx_nnodes onx_ninits onx_producer_name onx_producer_version onx_ai.onnx.ml onx_size_optim onx_nnodes_optim onx_ninits_optim opset15 RT/SKL-N=1 N=10 N=100 N=1000 N=10000 RT/SKL-N=1-min RT/SKL-N=1-max N=10-min N=10-max N=100-min N=100-max N=1000-min N=1000-max N=10000-min N=10000-max
0 LinearRegression b-reg default NaN predict 0 {} NaN 10 onnxruntime1 1.0.2 1 10 1 290 1 0 skl2onnx 1.11.1 1 290 1 0 OK 15/1 0.871490 0.883441 0.890904 0.847754 2.017820 0.810446 1.035661 0.854918 1.465509 0.883105 0.935423 0.329009 1.140110 1.898464 2.051557
1 LinearRegression b-reg default NaN predict 0 {} NaN 10 python 1.0.2 1 10 1 290 1 0 skl2onnx 1.11.1 1 290 1 0 OK 15/1 0.411075 0.408217 0.415163 0.339389 0.562617 0.397127 0.419834 0.405607 0.410690 0.412036 0.420237 0.129449 0.497009 0.528236 0.583922
2 LinearRegression b-reg default NaN predict 0 {} NaN 50 onnxruntime1 1.0.2 1 50 1 491 1 0 skl2onnx 1.11.1 1 491 1 0 OK 15/1 0.874621 0.851003 0.955369 2.200768 2.575946 0.842020 1.002503 0.816183 1.416143 0.945761 0.999240 2.029203 2.248088 2.502052 2.629870
3 LinearRegression b-reg default NaN predict 0 {} NaN 50 python 1.0.2 1 50 1 491 1 0 skl2onnx 1.11.1 1 491 1 0 OK 15/1 0.404682 0.401703 0.413525 0.470683 0.591081 0.401021 0.410244 0.398743 0.404658 0.410266 0.417043 0.423785 0.570792 0.575156 0.610485


Graph.

_, ax = plot_validate_benchmark(results)
ax

# import matplotlib.pyplot as plt
# plt.show()
RT/SKL-N=1, N=10, N=100, N=1000, N=10000

Out:

array([<AxesSubplot:title={'center':'RT/SKL-N=1'}>,
       <AxesSubplot:title={'center':'N=10'}>,
       <AxesSubplot:title={'center':'N=100'}>,
       <AxesSubplot:title={'center':'N=1000'}>,
       <AxesSubplot:title={'center':'N=10000'}>], dtype=object)

Total running time of the script: ( 0 minutes 59.739 seconds)

Gallery generated by Sphinx-Gallery