Benchmark inference for scikit-learn models

This short code compares the execution of a couple of runtime for inference including onnxruntime. It uses examples Measure ONNX runtime performances. It is an automated process to compare the performance of a model against scikit-learn. This model is a simple model taken from all implemented by scikit-learn.

Linear Regression

from pandas import read_csv
from mlprodict.cli import validate_runtime
from mlprodict.plotting.plotting import plot_validate_benchmark

res = validate_runtime(
    verbose=1,
    out_raw="data.csv", out_summary="summary.csv",
    benchmark=True, dump_folder="dump_errors",
    runtime=['python', 'onnxruntime1'],
    models=['LinearRegression'],
    skip_models=['LinearRegression[m-reg]'],
    n_features=[10, 50], dtype="32",
    out_graph="bench.png",
    opset_min=15, opset_max=15,
    time_kwargs={
        1: {"number": 50, "repeat": 50},
        10: {"number": 25, "repeat": 25},
        100: {"number": 20, "repeat": 20},
        1000: {"number": 20, "repeat": 20},
        10000: {"number": 10, "repeat": 10},
    }
)

results = read_csv('summary.csv')
results

time_kwargs={1: {'number': 50, 'repeat': 50}, 10: {'number': 25, 'repeat': 25}, 100: {'number': 20, 'repeat': 20}, 1000: {'number': 20, 'repeat': 20}, 10000: {'number': 10, 'repeat': 10}}
[enumerate_validated_operator_opsets] opset in [15, 15].

  0%|          | 0/1 [00:00<?, ?it/s]
LinearRegression            :   0%|          | 0/1 [00:00<?, ?it/s][enumerate_compatible_opset] opset in [15, 15].

LinearRegression            : 100%|##########| 1/1 [00:34<00:00, 34.87s/it]
LinearRegression            : 100%|##########| 1/1 [00:34<00:00, 34.87s/it]
Saving raw_data into 'data.csv'.
Saving summary into 'summary.csv'.
Saving graph into 'bench.png'.
findfont: Font family ['STIXGeneral'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXGeneral'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXGeneral'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXNonUnicode'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXNonUnicode'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXNonUnicode'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXSizeOneSym'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXSizeTwoSym'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXSizeThreeSym'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXSizeFourSym'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXSizeFiveSym'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmsy10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmr10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmtt10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmmi10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmb10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmss10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmex10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['DejaVu Sans Display'] not found. Falling back to DejaVu Sans.

	name	problem	scenario	optim	method_name	output_index	conv_options	inst	n_features	runtime	skl_version	skl_nop	skl_ncoef	skl_nlin	onx_size	onx_nnodes	onx_ninits	onx_producer_name	onx_producer_version	onx_ai.onnx.ml	onx_size_optim	onx_nnodes_optim	onx_ninits_optim	opset15	RT/SKL-N=1	N=10	N=100	N=1000	N=10000	RT/SKL-N=1-min	RT/SKL-N=1-max	N=10-min	N=10-max	N=100-min	N=100-max	N=1000-min	N=1000-max	N=10000-min	N=10000-max
0	LinearRegression	b-reg	default	NaN	predict	0	{}	NaN	10	onnxruntime1	1.2.0	1	10	1	290	1	0	skl2onnx	1.13.1	1	290	1	0	OK 15/1	0.785002	0.804724	0.801732	1.020297	1.928560	0.723856	1.012242	0.761097	1.683360	0.793499	0.813619	0.896072	1.056262	1.847040	1.962342
1	LinearRegression	b-reg	default	NaN	predict	0	{}	NaN	10	python	1.2.0	1	10	1	290	1	0	skl2onnx	1.13.1	1	290	1	0	OK 15/1	0.417037	0.415814	0.421362	0.460073	0.608036	0.402423	0.431028	0.412353	0.419395	0.416551	0.427191	0.397862	0.533997	0.575217	0.638020
2	LinearRegression	b-reg	default	NaN	predict	0	{}	NaN	50	onnxruntime1	1.2.0	1	50	1	491	1	0	skl2onnx	1.13.1	1	491	1	0	OK 15/1	0.788376	0.772780	0.873732	2.039609	1.348183	0.756350	0.978361	0.730118	1.584541	0.859194	0.903624	1.918849	2.087055	0.653299	3.101425
3	LinearRegression	b-reg	default	NaN	predict	0	{}	NaN	50	python	1.2.0	1	50	1	491	1	0	skl2onnx	1.13.1	1	491	1	0	OK 15/1	0.411969	0.410149	0.422061	0.465076	0.336590	0.409540	0.414435	0.407995	0.412918	0.417069	0.439684	0.434929	0.478493	0.279825	0.512767

Graph.

_, ax = plot_validate_benchmark(results)
ax

# import matplotlib.pyplot as plt
# plt.show()

array([<AxesSubplot: title={'center': 'RT/SKL-N=1'}>,
       <AxesSubplot: title={'center': 'N=10'}>,
       <AxesSubplot: title={'center': 'N=100'}>,
       <AxesSubplot: title={'center': 'N=1000'}>,
       <AxesSubplot: title={'center': 'N=10000'}>], dtype=object)