.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "gyexamples/plot_orttraining_benchmark_torch.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        Click :ref:`here <sphx_glr_download_gyexamples_plot_orttraining_benchmark_torch.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_gyexamples_plot_orttraining_benchmark_torch.py:


.. _l-orttraining-benchmark-torch:

Benchmark, comparison torch - forward-backward
==============================================

The benchmark compares the processing time between :epkg:`pytorch`
and :epkg:`onnxruntime-training` on a linear regression and a neural network.
This example starts from :ref:`l-orttraining-linreg-fwbw`
but uses :epkg:`pytorch` to replace the parts updating the gradients
and computing the error gradient. The training algorithm becomes:

.. image:: images/onnxfwbwtorch.png

Class :epkg:`TrainingAgent` (from :epkg:`onnxruntime-training`) is still
used and wrapped into :epkg:`ORTModule`. This script then
follows the same instructions as :ref:`l-orttraining-benchmark-fwbw`
to compare :epkg:`pytorch` only against :epkg:`pytorch` and
:epkg:`onnxruntime-training`.


.. contents::
    :local:

First comparison: neural network
++++++++++++++++++++++++++++++++

.. GENERATED FROM PYTHON SOURCE LINES 30-47

.. code-block:: default

    import time
    import numpy
    from pandas import DataFrame
    import matplotlib.pyplot as plt
    import torch
    from onnxruntime import get_device
    from onnxruntime.training.ortmodule import ORTModule
    from pyquickhelper.pycode.profiling import profile, profile2graph
    from sklearn.datasets import make_regression
    from sklearn.model_selection import train_test_split


    X, y = make_regression(1000, n_features=100, bias=2)
    X = X.astype(numpy.float32)
    y = y.astype(numpy.float32)
    X_train, X_test, y_train, y_test = train_test_split(X, y)








.. GENERATED FROM PYTHON SOURCE LINES 48-50

Common parameters and training algorithm
++++++++++++++++++++++++++++++++++++++++

.. GENERATED FROM PYTHON SOURCE LINES 50-64

.. code-block:: default



    def from_numpy(v, device=None, requires_grad=False):
        """
        Convers a numpy array into a torch array and
        sets *device* and *requires_grad*.
        """
        v = torch.from_numpy(v)
        if device is not None:
            v = v.to(device)
        v.requires_grad_(requires_grad)
        return v









.. GENERATED FROM PYTHON SOURCE LINES 65-67

Training, two functions with same code but it is easier
to distinguish between in the profiling.

.. GENERATED FROM PYTHON SOURCE LINES 67-128

.. code-block:: default


    def train_model_torch(model, device, x, y, n_iter=100, learning_rate=1e-5,
                          profiler=None):

        def forward_torch(model, x):
            return model(x)

        model = model.to(device)
        x = from_numpy(x, requires_grad=True, device=device)
        y = from_numpy(y, requires_grad=True, device=device)

        criterion = torch.nn.MSELoss(reduction='sum')
        optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)
        losses = []
        for t in range(n_iter):

            def step_train_torch():
                y_pred = forward_torch(model, x)
                loss = criterion(y_pred, y)
                optimizer.zero_grad()
                loss.backward()
                optimizer.step()
                return loss

            loss = step_train_torch()
            losses.append(loss)
            if profiler is not None:
                profiler.step()

        return losses


    def train_model_ort(model, device, x, y, n_iter=100, learning_rate=1e-5,
                        profiler=None):
        def forward_ort(model, x):
            return model(x)

        model = model.to(device)
        x = from_numpy(x, requires_grad=True, device=device)
        y = from_numpy(y, requires_grad=True, device=device)

        criterion = torch.nn.MSELoss(reduction='sum')
        optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)
        losses = []
        for t in range(n_iter):

            def step_train_ort():
                y_pred = forward_ort(model, x)
                loss = criterion(y_pred, y)
                optimizer.zero_grad()
                loss.backward()
                optimizer.step()
                return loss

            loss = step_train_ort()
            losses.append(loss)
            if profiler is not None:
                profiler.step()

        return losses








.. GENERATED FROM PYTHON SOURCE LINES 129-130

Benchmark function

.. GENERATED FROM PYTHON SOURCE LINES 130-186

.. code-block:: default



    def benchmark(model_torch, model_ort, device, name, verbose=True, max_iter=100):

        print(f"[benchmark] {name}")
        begin = time.perf_counter()
        losses = train_model_torch(
            model_torch, device, X_train, y_train, n_iter=200)
        duration_torch = time.perf_counter() - begin
        length_torch = len(losses)
        print(
            f"[benchmark] torch={length_torch!r} iterations - {duration_torch!r} seconds")

        if model_ort is None:
            length_ort = 0
            duration_ort = 0
        else:
            begin = time.perf_counter()
            losses = train_model_ort(model_ort, device, X_train,
                                     y_train, n_iter=max_iter)
            duration_ort = time.perf_counter() - begin
            length_ort = len(losses)
        print(
            f"[benchmark] onxrt={length_ort!r} iteration - {duration_ort!r} seconds")

        return dict(torch=duration_torch, ort=duration_ort, name=name,
                    iter_torch=length_torch, iter_ort=length_ort)


    class MLPNet(torch.nn.Module):
        def __init__(self, D_in, D_out):
            super(MLPNet, self).__init__()
            self.linear1 = torch.nn.Linear(D_in, 50)
            self.linear2 = torch.nn.Linear(50, 10)
            self.linear3 = torch.nn.Linear(10, D_out)

        def forward(self, x):
            o1 = torch.sigmoid(self.linear1(x))
            o2 = torch.sigmoid(self.linear2(o1))
            return self.linear3(o2)


    d_in, d_out, N = X.shape[1], 1, X.shape[0]
    model_torch = MLPNet(d_in, d_out)
    try:
        model_ort = ORTModule(MLPNet(d_in, d_out))
    except Exception as e:
        model_ort = None
        print("ERROR: installation of torch extension for onnxruntime "
              "probably failed due to: ", e)
    max_iter = 100

    device = torch.device('cpu')
    benches = [benchmark(model_torch, model_ort, device, name='NN-CPU',
                         max_iter=max_iter)]





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    ERROR: installation of torch extension for onnxruntime probably failed due to:  ORTModule's extensions were not detected at 'somewhere/workspace/onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/onnxruntime/training/ortmodule/torch_cpp_extensions' folder. Run `python -m torch_ort.configure` before using `ORTModule` frontend.
    [benchmark] NN-CPU
    somewhere/workspace/onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/loss.py:536: UserWarning: Using a target size (torch.Size([750])) that is different to the input size (torch.Size([750, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
      return F.mse_loss(input, target, reduction=self.reduction)
    [benchmark] torch=200 iterations - 1.58295133698266 seconds
    [benchmark] onxrt=0 iteration - 0 seconds




.. GENERATED FROM PYTHON SOURCE LINES 187-189

Profiling
+++++++++

.. GENERATED FROM PYTHON SOURCE LINES 189-213

.. code-block:: default



    def clean_name(text):
        pos = text.find('onnxruntime')
        if pos >= 0:
            return text[pos:]
        pos = text.find('onnxcustom')
        if pos >= 0:
            return text[pos:]
        pos = text.find('torch')
        if pos >= 0:
            return text[pos:]
        pos = text.find('site-packages')
        if pos >= 0:
            return text[pos:]
        return text


    ps = profile(lambda: benchmark(
        model_torch, model_ort, device, name='NN-CPU', max_iter=max_iter))[0]
    root, nodes = profile2graph(ps, clean_text=clean_name)
    text = root.to_text()
    print(text)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    [benchmark] NN-CPU
    somewhere/workspace/onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/loss.py:536: UserWarning: Using a target size (torch.Size([750])) that is different to the input size (torch.Size([750, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
      return F.mse_loss(input, target, reduction=self.reduction)
    [benchmark] torch=200 iterations - 1.3074407509993762 seconds
    [benchmark] onxrt=0 iteration - 0 seconds
    filter                                                       --    18    18 -- 0.00006 0.00015 -- /usr/local/lib/python3.9/logging/__init__.py:787:filter (filter)
        filter                                                   --    12    12 -- 0.00001 0.00001 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/sphinx/util/logging.py:351:filter (filter)
        filter                                                   --     6     6 -- 0.00004 0.00005 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/sphinx/util/logging.py:484:filter (filter)
            <built-in method builtins.isinstance>                --    18    18 -- 0.00001 0.00001 -- ~:0:<built-in method builtins.isinstance> (<built-in method builtins.isinstance>) +++
        <built-in method builtins.hasattr>                       --    18    18 -- 0.00002 0.00002 -- ~:0:<built-in method builtins.hasattr> (<built-in method builtins.hasattr>) +++
    acquire                                                      --    30    30 -- 0.00005 0.00009 -- /usr/local/lib/python3.9/logging/__init__.py:892:acquire (acquire)
        <method 'acquire' of '_thread.RLock' objects>            --    30    30 -- 0.00003 0.00003 -- ~:0:<method 'acquire' of '_thread.RLock' objects> (<method 'acquire' of '_thread.RLock' objects>)
    release                                                      --    30    30 -- 0.00005 0.00006 -- /usr/local/lib/python3.9/logging/__init__.py:899:release (release)
        <method 'release' of '_thread.RLock' objects>            --    30    30 -- 0.00001 0.00001 -- ~:0:<method 'release' of '_thread.RLock' objects> (<method 'release' of '_thread.RLock' objects>)
    emit                                                         --    12    12 -- 0.00007 0.00112 -- /usr/local/lib/python3.9/logging/__init__.py:1067:emit (emit)
        format                                                   --    12    12 -- 0.00003 0.00045 -- /usr/local/lib/python3.9/logging/__init__.py:912:format (format)
            format                                               --    12    12 -- 0.00008 0.00042 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/sphinx/util/logging.py:531:format (format)
                format                                           --    12    12 -- 0.00007 0.00031 -- /usr/local/lib/python3.9/logging/__init__.py:646:format (format)
                    usesTime                                     --    12    12 -- 0.00002 0.00006 -- /usr/local/lib/python3.9/logging/__init__.py:624:usesTime (usesTime)
                        usesTime                                 --    12    12 -- 0.00003 0.00005 -- /usr/local/lib/python3.9/logging/__init__.py:417:usesTime (usesTime)
                            <method 'find' of 'str' objects>     --    12    12 -- 0.00002 0.00002 -- ~:0:<method 'find' of 'str' objects> (<method 'find' of 'str' objects>)
                    formatMessage                                --    12    12 -- 0.00002 0.00007 -- /usr/local/lib/python3.9/logging/__init__.py:630:formatMessage (formatMessage)
                        format                                   --    12    12 -- 0.00002 0.00006 -- /usr/local/lib/python3.9/logging/__init__.py:428:format (format)
                            _format                              --    12    12 -- 0.00004 0.00004 -- /usr/local/lib/python3.9/logging/__init__.py:425:_format (_format)
                    getMessage                                   --    12    12 -- 0.00005 0.00010 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/sphinx/util/logging.py:89:getMessage (getMessage)
                        getMessage                               --    12    12 -- 0.00004 0.00004 -- /usr/local/lib/python3.9/logging/__init__.py:354:getMessage (getMessage)
                        <built-in method builtins.getattr>       --    12    12 -- 0.00001 0.00001 -- ~:0:<built-in method builtins.getattr> (<built-in method builtins.getattr>) +++
                colorize                                         --     2     2 -- 0.00001 0.00002 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/sphinx/util/console.py:72:colorize (colorize)
                    escseq                                       --     4     4 -- 0.00001 0.00001 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/sphinx/util/console.py:73:escseq (escseq)
                <built-in method builtins.getattr>               --    12    12 -- 0.00001 0.00001 -- ~:0:<built-in method builtins.getattr> (<built-in method builtins.getattr>) +++
        flush                                                    --    12    12 -- 0.00006 0.00056 -- /usr/local/lib/python3.9/logging/__init__.py:1056:flush (flush)
            acquire                                              --    12    12 -- 0.00002 0.00003 -- /usr/local/lib/python3.9/logging/__init__.py:892:acquire (acquire) +++
            release                                              --    12    12 -- 0.00002 0.00003 -- /usr/local/lib/python3.9/logging/__init__.py:899:release (release) +++
            flush                                                --     6     6 -- 0.00002 0.00044 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/sphinx/util/logging.py:557:flush (flush)
                <method 'flush' of '_io.TextIOWrapper' objects>  --     6     6 -- 0.00041 0.00041 -- ~:0:<method 'flush' of '_io.TextIOWrapper' objects> (<method 'flush' of '_io.TextIOWrapper' objects>)
            <built-in method builtins.hasattr>                   --    12    12 -- 0.00001 0.00001 -- ~:0:<built-in method builtins.hasattr> (<built-in method builtins.hasattr>) +++
        write                                                    --     6     6 -- 0.00002 0.00002 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/sphinx/util/logging.py:549:write (write)
        write                                                    --     6     6 -- 0.00002 0.00003 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/sphinx/util/logging.py:567:write (write)
    isEnabledFor                                                 --    12    12 -- 0.00002 0.00002 -- /usr/local/lib/python3.9/logging/__init__.py:1677:isEnabledFor (isEnabledFor)
    __init__                                                     --     2     2 -- 0.00001 0.00001 -- /usr/local/lib/python3.9/warnings.py:403:__init__ (__init__)
    <lambda>                                                     --     1     1 -- 0.00318 1.31255 -- onnxcustom/onnxcustom_UT_39_std/_doc/examples/plot_orttraining_benchmark_torch.py:207:<lambda> (<lambda>)
        benchmark                                                --     1     1 -- 0.00007 1.30937 -- onnxcustom/onnxcustom_UT_39_std/_doc/examples/plot_orttraining_benchmark_torch.py:132:benchmark (benchmark)
            train_model_torch                                    --     1     1 -- 0.00267 1.30741 -- onnxcustom/onnxcustom_UT_39_std/_doc/examples/plot_orttraining_benchmark_torch.py:68:train_model_torch (train_model_torch)
                from_numpy                                       --     2     2 -- 0.00002 0.00007 -- onnxcustom/onnxcustom_UT_39_std/_doc/examples/plot_orttraining_benchmark_torch.py:52:from_numpy (from_numpy)
                step_train_torch                                 --   200   200 -- 0.00742 1.30321 -- onnxcustom/onnxcustom_UT_39_std/_doc/examples/plot_orttraining_benchmark_torch.py:83:step_train_torch (step_train_torch)
                    forward_torch                                --   200   200 -- 0.00106 0.22978 -- onnxcustom/onnxcustom_UT_39_std/_doc/examples/plot_orttraining_benchmark_torch.py:71:forward_torch (forward_torch)
                        _call_impl                               --   200   200 -- 0.00227 0.22872 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1188:_call_impl (_call_impl) +++
                    backward                                     --   200   200 -- 0.00337 0.78665 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/_tensor.py:429:backward (backward)
                        backward                                 --   200   200 -- 0.00496 0.78307 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/autograd/__init__.py:103:backward (backward)
                            _make_grads                          --   200   200 -- 0.00367 0.01017 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/autograd/__init__.py:47:_make_grads (_make_grads)
                                <built-in metho...rch.ones_like> --   200   200 -- 0.00568 0.00568 -- ~:0:<built-in method torch.ones_like> (<built-in method torch.ones_like>)
                                <method 'numel'...Base' objects> --   200   200 -- 0.00027 0.00027 -- ~:0:<method 'numel' of 'torch._C._TensorBase' objects> (<method 'numel' of 'torch._C._TensorBase' objects>)
                                <method 'append...list' objects> --   200   200 -- 0.00025 0.00025 -- ~:0:<method 'append' of 'list' objects> (<method 'append' of 'list' objects>) +++
                                <built-in metho...ns.isinstance> --   200   200 -- 0.00030 0.00030 -- ~:0:<built-in method builtins.isinstance> (<built-in method builtins.isinstance>) +++
                            _tensor_or_tensors_to_tuple          --   200   200 -- 0.00061 0.00061 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/autograd/__init__.py:95:_tensor_or_tensors_to_tuple (_tensor_or_tensors_to_tuple)
                            <built-in method ...ansforms_active> --   200   200 -- 0.00046 0.00046 -- ~:0:<built-in method torch._C._are_functorch_transforms_active> (<built-in method torch._C._are_functorch_transforms_active>)
                            <method 'run_back...neBase' objects> --   200   200 -- 0.76613 0.76613 -- ~:0:<method 'run_backward' of 'torch._C._EngineBase' objects> (<method 'run_backward' of 'torch._C._EngineBase' objects>)
                            <built-in method ...tins.isinstance> --   400   400 -- 0.00053 0.00053 -- ~:0:<built-in method builtins.isinstance> (<built-in method builtins.isinstance>) +++
                            <built-in method builtins.len>       --   200   200 -- 0.00020 0.00020 -- ~:0:<built-in method builtins.len> (<built-in method builtins.len>) +++
                        <built-in method to...ch_function_unary> --   200   200 -- 0.00021 0.00021 -- ~:0:<built-in method torch._C._has_torch_function_unary> (<built-in method torch._C._has_torch_function_unary>)
                    _call_impl                                   --   200   200 -- 0.00225 0.12697 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1188:_call_impl (_call_impl) +++
                    wrapper                                      --   200   200 -- 0.00840 0.09830 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/optim/optimizer.py:135:wrapper (wrapper)
                        __init__                                 --   200   200 -- 0.00172 0.01074 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/autograd/profiler.py:478:__init__ (__init__) +++
                        __enter__                                --   200   200 -- 0.00335 0.01291 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/autograd/profiler.py:487:__enter__ (__enter__) +++
                        __exit__                                 --   200   200 -- 0.00129 0.00428 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/autograd/profiler.py:491:__exit__ (__exit__) +++
                        _use_grad                                --   200   200 -- 0.00451 0.06053 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/optim/optimizer.py:19:_use_grad (_use_grad)
                            __init__                             --   400   400 -- 0.00183 0.00245 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py:227:__init__ (__init__) +++
                            step                                 --   200   200 -- 0.01762 0.05328 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/optim/sgd.py:119:step (step)
                                __hash__                         --  2406  2406 -- 0.00186 0.00333 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/_tensor.py:928:__hash__ (__hash__) +++
                                sgd                              --   200   200 -- 0.00066 0.03071 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/optim/sgd.py:171:sgd (sgd)
                                    _single_tensor_sgd           --   200   200 -- 0.00556 0.03005 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/optim/sgd.py:213:_single_tensor_sgd (_single_tensor_sgd)
                                        <method 'ad...' objects> --  1200  1200 -- 0.02449 0.02449 -- ~:0:<method 'add_' of 'torch._C._TensorBase' objects> (<method 'add_' of 'torch._C._TensorBase' objects>)
                                <method 'append...list' objects> --  3600  3600 -- 0.00162 0.00162 -- ~:0:<method 'append' of 'list' objects> (<method 'append' of 'list' objects>) +++
                            <built-in method ...is_grad_enabled> --   200   200 -- 0.00029 0.00029 -- ~:0:<built-in method torch.is_grad_enabled> (<built-in method torch.is_grad_enabled>) +++
                        _optimizer_step_code                     --   200   200 -- 0.00012 0.00012 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/optim/optimizer.py:117:_optimizer_step_code (_optimizer_step_code)
                        <method 'format' of 'str' objects>       --   200   200 -- 0.00134 0.00134 -- ~:0:<method 'format' of 'str' objects> (<method 'format' of 'str' objects>) +++
                    zero_grad                                    --   200   200 -- 0.01680 0.05410 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/optim/optimizer.py:246:zero_grad (zero_grad)
                        __init__                                 --   200   200 -- 0.00216 0.00997 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/autograd/profiler.py:478:__init__ (__init__) +++
                        __enter__                                --   200   200 -- 0.00354 0.01363 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/autograd/profiler.py:487:__enter__ (__enter__) +++
                        __exit__                                 --   200   200 -- 0.00137 0.00418 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/autograd/profiler.py:491:__exit__ (__exit__) +++
                        <method 'requires_g...nsorBase' objects> --  1200  1200 -- 0.00284 0.00284 -- ~:0:<method 'requires_grad_' of 'torch._C._TensorBase' objects> (<method 'requires_grad_' of 'torch._C._TensorBase' objects>) +++
                        <method 'zero_' of ...nsorBase' objects> --  1200  1200 -- 0.00619 0.00619 -- ~:0:<method 'zero_' of 'torch._C._TensorBase' objects> (<method 'zero_' of 'torch._C._TensorBase' objects>)
                        <method 'get' of 'dict' objects>         --   200   200 -- 0.00022 0.00022 -- ~:0:<method 'get' of 'dict' objects> (<method 'get' of 'dict' objects>) +++
                        <built-in method builtins.hasattr>       --   200   200 -- 0.00027 0.00027 -- ~:0:<built-in method builtins.hasattr> (<built-in method builtins.hasattr>) +++
                __init__                                         --     1     1 -- 0.00001 0.00013 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/loss.py:532:__init__ (__init__)
                    __init__                                     --     1     1 -- 0.00001 0.00012 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/loss.py:20:__init__ (__init__)
                        __init__                                 --     1     1 -- 0.00003 0.00004 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:309:__init__ (__init__)
                        __setattr__                              --     1     1 -- 0.00001 0.00007 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1272:__setattr__ (__setattr__)
                to                                               --     1     1 -- 0.00001 0.00089 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:883:to (to)
                    _apply                                       --     1     1 -- 0.00002 0.00086 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:639:_apply (_apply) +++
                __init__                                         --     1     1 -- 0.00001 0.00030 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/optim/sgd.py:93:__init__ (__init__)
                    __init__                                     --     1     1 -- 0.00003 0.00029 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/optim/optimizer.py:45:__init__ (__init__)
                        parameters                               --     7     7 -- 0.00001 0.00014 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1688:parameters (parameters)
                            named_parameters                     --     7     7 -- 0.00001 0.00013 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1713:named_parameters (named_parameters)
                                _named_members                   --     7     7 -- 0.00004 0.00012 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1675:_named_members (_named_members)
                                    __hash__                     --     6     6 -- 0.00001 0.00001 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/_tensor.py:928:__hash__ (__hash__) +++
                                    <lambda>                     --     4     4 -- 0.00001 0.00001 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1735:<lambda> (<lambda>)
                                    named_modules                --     5     5 -- 0.00002 0.00004 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1847:named_modules (named_modules) +++
                        _hook_for_profile                        --     1     1 -- 0.00001 0.00002 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/optim/optimizer.py:130:_hook_for_profile (_hook_for_profile)
                        add_param_group                          --     1     1 -- 0.00007 0.00010 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/optim/optimizer.py:300:add_param_group (add_param_group)
                            __hash__                             --    12    12 -- 0.00001 0.00001 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/_tensor.py:928:__hash__ (__hash__) +++
                <method 'append' of 'list' objects>              --   200   200 -- 0.00015 0.00015 -- ~:0:<method 'append' of 'list' objects> (<method 'append' of 'list' objects>) +++
            <built-in method builtins.print>                     --     3     3 -- 0.00003 0.00188 -- ~:0:<built-in method builtins.print> (<built-in method builtins.print>)
                write                                            --     6     6 -- 0.00007 0.00185 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/sphinx_gallery/gen_rst.py:83:write (write) +++
    write                                                        --     7     7 -- 0.00009 0.00274 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/sphinx_gallery/gen_rst.py:83:write (write)
        verbose                                                  --     6     6 -- 0.00004 0.00263 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/sphinx/util/logging.py:128:verbose (verbose)
            log                                                  --     6     6 -- 0.00006 0.00259 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/sphinx/util/logging.py:121:log (log)
                log                                              --     6     6 -- 0.00007 0.00253 -- /usr/local/lib/python3.9/logging/__init__.py:1825:log (log)
                    log                                          --     6     6 -- 0.00006 0.00239 -- /usr/local/lib/python3.9/logging/__init__.py:1485:log (log)
                        _log                                     --     6     6 -- 0.00004 0.00232 -- /usr/local/lib/python3.9/logging/__init__.py:1553:_log (_log)
                            findCaller                           --     6     6 -- 0.00007 0.00013 -- /usr/local/lib/python3.9/logging/__init__.py:1502:findCaller (findCaller)
                                <lambda>                         --     6     6 -- 0.00002 0.00002 -- /usr/local/lib/python3.9/logging/__init__.py:156:<lambda> (<lambda>)
                                normcase                         --    12    12 -- 0.00002 0.00003 -- /usr/local/lib/python3.9/posixpath.py:52:normcase (normcase)
                                    <built-in met...osix.fspath> --    12    12 -- 0.00001 0.00001 -- ~:0:<built-in method posix.fspath> (<built-in method posix.fspath>) +++
                                <built-in metho...ltins.hasattr> --    12    12 -- 0.00001 0.00001 -- ~:0:<built-in method builtins.hasattr> (<built-in method builtins.hasattr>) +++
                            makeRecord                           --     6     6 -- 0.00004 0.00061 -- /usr/local/lib/python3.9/logging/__init__.py:1538:makeRecord (makeRecord)
                                __init__                         --     6     6 -- 0.00024 0.00057 -- /usr/local/lib/python3.9/logging/__init__.py:278:__init__ (__init__)
                                    getLevelName                 --     6     6 -- 0.00004 0.00005 -- /usr/local/lib/python3.9/logging/__init__.py:119:getLevelName (getLevelName)
                                        <method 'ge...' objects> --    12    12 -- 0.00001 0.00001 -- ~:0:<method 'get' of 'dict' objects> (<method 'get' of 'dict' objects>) +++
                                    current_process              --     6     6 -- 0.00000 0.00000 -- /usr/local/lib/python3.9/multiprocessing/process.py:37:current_process (current_process)
                                    name                         --     6     6 -- 0.00001 0.00001 -- /usr/local/lib/python3.9/multiprocessing/process.py:189:name (name)
                                    splitext                     --     6     6 -- 0.00003 0.00008 -- /usr/local/lib/python3.9/posixpath.py:117:splitext (splitext)
                                        _splitext                --     6     6 -- 0.00003 0.00004 -- /usr/local/lib/python3.9/genericpath.py:121:_splitext (_splitext)
                                            <method '...objects> --    12    12 -- 0.00001 0.00001 -- ~:0:<method 'rfind' of 'str' objects> (<method 'rfind' of 'str' objects>) +++
                                    basename                     --     6     6 -- 0.00004 0.00007 -- /usr/local/lib/python3.9/posixpath.py:140:basename (basename)
                                        _get_sep                 --     6     6 -- 0.00001 0.00001 -- /usr/local/lib/python3.9/posixpath.py:41:_get_sep (_get_sep)
                                    name                         --     6     6 -- 0.00001 0.00001 -- /usr/local/lib/python3.9/threading.py:1053:name (name)
                                    current_thread               --     6     6 -- 0.00001 0.00002 -- /usr/local/lib/python3.9/threading.py:1318:current_thread (current_thread)
                            handle                               --     6     6 -- 0.00002 0.00154 -- /usr/local/lib/python3.9/logging/__init__.py:1579:handle (handle)
                                filter                           --     6     6 -- 0.00001 0.00001 -- /usr/local/lib/python3.9/logging/__init__.py:787:filter (filter) +++
                                callHandlers                     --     6     6 -- 0.00006 0.00151 -- /usr/local/lib/python3.9/logging/__init__.py:1633:callHandlers (callHandlers)
                                    handle                       --    12    12 -- 0.00006 0.00146 -- /usr/local/lib/python3.9/logging/__init__.py:935:handle (handle)
                                        filter                   --    12    12 -- 0.00005 0.00014 -- /usr/local/lib/python3.9/logging/__init__.py:787:filter (filter) +++
                                        acquire                  --    12    12 -- 0.00003 0.00005 -- /usr/local/lib/python3.9/logging/__init__.py:892:acquire (acquire) +++
                                        release                  --    12    12 -- 0.00001 0.00002 -- /usr/local/lib/python3.9/logging/__init__.py:899:release (release) +++
                                        emit                     --     6     6 -- 0.00003 0.00030 -- /usr/local/lib/python3.9/logging/__init__.py:1067:emit (emit) +++
                                        emit                     --     6     6 -- 0.00005 0.00090 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/sphinx/util/logging.py:151:emit (emit)
                                            acquire              --     6     6 -- 0.00001 0.00001 -- /usr/local/lib/python3.9/logging/__init__.py:892:acquire (acquire) +++
                                            release              --     6     6 -- 0.00001 0.00001 -- /usr/local/lib/python3.9/logging/__init__.py:899:release (release) +++
                                            emit                 --     6     6 -- 0.00004 0.00082 -- /usr/local/lib/python3.9/logging/__init__.py:1067:emit (emit) +++
                        isEnabledFor                             --     6     6 -- 0.00001 0.00001 -- /usr/local/lib/python3.9/logging/__init__.py:1677:isEnabledFor (isEnabledFor) +++
                    isEnabledFor                                 --     6     6 -- 0.00002 0.00003 -- /usr/local/lib/python3.9/logging/__init__.py:1834:isEnabledFor (isEnabledFor)
                        isEnabledFor                             --     6     6 -- 0.00001 0.00001 -- /usr/local/lib/python3.9/logging/__init__.py:1677:isEnabledFor (isEnabledFor) +++
                    process                                      --     6     6 -- 0.00004 0.00005 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/sphinx/util/logging.py:131:process (process)
    __call__                                                     --   800   800 -- 0.00331 0.02544 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/_ops.py:437:__call__ (__call__)
        <built-in method torch._ops...er._record_function_enter> --   400   400 -- 0.01742 0.01742 -- ~:0:<built-in method torch._ops.profiler._record_function_enter> (<built-in method torch._ops.profiler._record_function_enter>)
        <built-in method torch._ops...ler._record_function_exit> --   400   400 -- 0.00471 0.00471 -- ~:0:<built-in method torch._ops.profiler._record_function_exit> (<built-in method torch._ops.profiler._record_function_exit>)
    __hash__                                                     --  2430  2430 -- 0.00188 0.00336 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/_tensor.py:928:__hash__ (__hash__)
        <built-in method builtins.id>                            --  2430  2430 -- 0.00148 0.00148 -- ~:0:<built-in method builtins.id> (<built-in method builtins.id>)
    __init__                                                     --   424   424 -- 0.00190 0.00255 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py:227:__init__ (__init__)
        <built-in method torch._C._set_grad_enabled>             --   424   424 -- 0.00029 0.00029 -- ~:0:<built-in method torch._C._set_grad_enabled> (<built-in method torch._C._set_grad_enabled>)
        <built-in method torch.is_grad_enabled>                  --   424   424 -- 0.00036 0.00036 -- ~:0:<built-in method torch.is_grad_enabled> (<built-in method torch.is_grad_enabled>) +++
    __init__                                                     --   400   400 -- 0.00388 0.02071 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/autograd/profiler.py:478:__init__ (__init__)
        <built-in method torch.zeros>                            --   400   400 -- 0.01683 0.01683 -- ~:0:<built-in method torch.zeros> (<built-in method torch.zeros>)
    __enter__                                                    --   400   400 -- 0.00689 0.02653 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/autograd/profiler.py:487:__enter__ (__enter__)
        __call__                                                 --   400   400 -- 0.00222 0.01965 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/_ops.py:437:__call__ (__call__) +++
    __exit__                                                     --   400   400 -- 0.00266 0.00846 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/autograd/profiler.py:491:__exit__ (__exit__)
        __call__                                                 --   400   400 -- 0.00109 0.00580 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/_ops.py:437:__call__ (__call__) +++
    _apply                                                       --     1     4 -- 0.00028 0.00086 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:639:_apply (_apply)
        __init__                                                 --    12    12 -- 0.00007 0.00007 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py:126:__init__ (__init__)
            is_scripting                                         --    12    12 -- 0.00000 0.00000 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/_jit_internal.py:1082:is_scripting (is_scripting)
        __enter__                                                --    12    12 -- 0.00006 0.00012 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py:131:__enter__ (__enter__)
            __init__                                             --    12    12 -- 0.00003 0.00005 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py:227:__init__ (__init__) +++
            <built-in method torch.is_grad_enabled>              --    12    12 -- 0.00001 0.00001 -- ~:0:<built-in method torch.is_grad_enabled> (<built-in method torch.is_grad_enabled>) +++
        __exit__                                                 --    12    12 -- 0.00004 0.00010 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py:135:__exit__ (__exit__)
            __init__                                             --    12    12 -- 0.00003 0.00005 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py:227:__init__ (__init__) +++
        _apply                                                   --     3     3 -- 0.00026 0.00080 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:639:_apply (_apply) +++
        compute_should_use_set_data                              --    12    12 -- 0.00005 0.00010 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:643:compute_should_use_set_data (compute_should_use_set_data)
            get_overwrite_module_params_on_conversion            --    12    12 -- 0.00001 0.00001 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/__future__.py:20:get_overwrite_module_params_on_conversion (get_overwrite_module_params_on_conversion)
            <built-in method torch._h...tible_shallow_copy_type> --    12    12 -- 0.00004 0.00004 -- ~:0:<built-in method torch._has_compatible_shallow_copy_type> (<built-in method torch._has_compatible_shallow_copy_type>)
        convert                                                  --    12    12 -- 0.00005 0.00013 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:983:convert (convert)
            <method 'to' of 'torch._C._TensorBase' objects>      --    12    12 -- 0.00006 0.00006 -- ~:0:<method 'to' of 'torch._C._TensorBase' objects> (<method 'to' of 'torch._C._TensorBase' objects>) +++
            <method 'is_floating_poin..._C._TensorBase' objects> --    12    12 -- 0.00002 0.00002 -- ~:0:<method 'is_floating_point' of 'torch._C._TensorBase' objects> (<method 'is_floating_point' of 'torch._C._TensorBase' objects>)
        children                                                 --     7     7 -- 0.00002 0.00005 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1790:children (children)
            named_children                                       --     7     7 -- 0.00003 0.00004 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1799:named_children (named_children)
    _call_impl                                                   --   400  1000 -- 0.00964 0.35569 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1188:_call_impl (_call_impl)
        forward                                                  --   200   200 -- 0.01106 0.22546 -- onnxcustom/onnxcustom_UT_39_std/_doc/examples/plot_orttraining_benchmark_torch.py:166:forward (forward)
            _call_impl                                           --   600   600 -- 0.00511 0.12028 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1188:_call_impl (_call_impl) +++
            __getattr__                                          --   600   600 -- 0.00303 0.00303 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1256:__getattr__ (__getattr__) +++
            <built-in method torch.sigmoid>                      --   400   400 -- 0.09109 0.09109 -- ~:0:<built-in method torch.sigmoid> (<built-in method torch.sigmoid>)
        forward                                                  --   600   600 -- 0.00597 0.11288 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/linear.py:113:forward (forward)
            __getattr__                                          --  1200  1200 -- 0.00199 0.00199 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1256:__getattr__ (__getattr__) +++
            <built-in method torch._C._nn.linear>                --   600   600 -- 0.10493 0.10493 -- ~:0:<built-in method torch._C._nn.linear> (<built-in method torch._C._nn.linear>)
        forward                                                  --   200   200 -- 0.00191 0.12420 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/loss.py:535:forward (forward)
            mse_loss                                             --   200   200 -- 0.00598 0.12228 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/functional.py:3264:mse_loss (mse_loss)
                broadcast_tensors                                --   200   200 -- 0.00348 0.01386 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/functional.py:45:broadcast_tensors (broadcast_tensors)
                    __getattr__                                  --   200   200 -- 0.00069 0.00102 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/_VF.py:26:__getattr__ (__getattr__)
                        <built-in method builtins.getattr>       --   200   200 -- 0.00032 0.00032 -- ~:0:<built-in method builtins.getattr> (<built-in method builtins.getattr>) +++
                    <built-in method torc..._has_torch_function> --   200   200 -- 0.00024 0.00024 -- ~:0:<built-in method torch._C._has_torch_function> (<built-in method torch._C._has_torch_function>)
                    <built-in method torch.broadcast_tensors>    --   200   200 -- 0.00913 0.00913 -- ~:0:<built-in method torch.broadcast_tensors> (<built-in method torch.broadcast_tensors>)
                get_enum                                         --   200   200 -- 0.00043 0.00043 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/_reduction.py:7:get_enum (get_enum)
                <built-in method torch....rch_function_variadic> --   200   200 -- 0.00029 0.00029 -- ~:0:<built-in method torch._C._has_torch_function_variadic> (<built-in method torch._C._has_torch_function_variadic>)
                <method 'size' of 'torc...._TensorBase' objects> --   800   800 -- 0.00214 0.00214 -- ~:0:<method 'size' of 'torch._C._TensorBase' objects> (<method 'size' of 'torch._C._TensorBase' objects>)
                <built-in method torch._C._nn.mse_loss>          --   200   200 -- 0.08758 0.08758 -- ~:0:<built-in method torch._C._nn.mse_loss> (<built-in method torch._C._nn.mse_loss>)
                <method 'format' of 'str' objects>               --   200   200 -- 0.00746 0.00746 -- ~:0:<method 'format' of 'str' objects> (<method 'format' of 'str' objects>) +++
                <built-in method _warnings.warn>                 --   200   200 -- 0.00356 0.00453 -- ~:0:<built-in method _warnings.warn> (<built-in method _warnings.warn>)
                    _showwarnmsg                                 --     1     1 -- 0.00001 0.00096 -- /usr/local/lib/python3.9/warnings.py:96:_showwarnmsg (_showwarnmsg)
                        _showwarning                             --     1     1 -- 0.00001 0.00095 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/sphinx_gallery/gen_rst.py:557:_showwarning (_showwarning)
                            formatwarning                        --     1     1 -- 0.00001 0.00005 -- /usr/local/lib/python3.9/warnings.py:15:formatwarning (formatwarning)
                                _formatwarnmsg_impl              --     1     1 -- 0.00003 0.00004 -- /usr/local/lib/python3.9/warnings.py:35:_formatwarnmsg_impl (_formatwarnmsg_impl)
                                    getline                      --     1     1 -- 0.00000 0.00001 -- /usr/local/lib/python3.9/linecache.py:26:getline (getline)
                                        getlines                 --     1     1 -- 0.00000 0.00001 -- /usr/local/lib/python3.9/linecache.py:36:getlines (getlines)
                                __init__                         --     1     1 -- 0.00000 0.00000 -- /usr/local/lib/python3.9/warnings.py:403:__init__ (__init__) +++
                            write                                --     1     1 -- 0.00002 0.00089 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/sphinx_gallery/gen_rst.py:83:write (write) +++
                    __init__                                     --     1     1 -- 0.00001 0.00001 -- /usr/local/lib/python3.9/warnings.py:403:__init__ (__init__) +++
        <built-in method torch._C._get_tracing_state>            --  1000  1000 -- 0.00379 0.00379 -- ~:0:<built-in method torch._C._get_tracing_state> (<built-in method torch._C._get_tracing_state>)
    __getattr__                                                  --  1800  1800 -- 0.00502 0.00502 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1256:__getattr__ (__getattr__)
    named_modules                                                --     5    11 -- 0.00004 0.00004 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1847:named_modules (named_modules)
        named_modules                                            --     6     6 -- 0.00001 0.00002 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1847:named_modules (named_modules) +++
    <built-in method builtins.len>                               --   212   212 -- 0.00021 0.00021 -- ~:0:<built-in method builtins.len> (<built-in method builtins.len>)
    <method 'append' of 'list' objects>                          --  4001  4001 -- 0.00202 0.00202 -- ~:0:<method 'append' of 'list' objects> (<method 'append' of 'list' objects>)
    <method 'requires_grad_' of 'torch._C._TensorBase' objects>  --  1202  1202 -- 0.00285 0.00285 -- ~:0:<method 'requires_grad_' of 'torch._C._TensorBase' objects> (<method 'requires_grad_' of 'torch._C._TensorBase' objects>)
    <method 'to' of 'torch._C._TensorBase' objects>              --    14    14 -- 0.00006 0.00006 -- ~:0:<method 'to' of 'torch._C._TensorBase' objects> (<method 'to' of 'torch._C._TensorBase' objects>)
    <method 'format' of 'str' objects>                           --   401   401 -- 0.00881 0.00881 -- ~:0:<method 'format' of 'str' objects> (<method 'format' of 'str' objects>)
    <method 'get' of 'dict' objects>                             --   241   241 -- 0.00024 0.00024 -- ~:0:<method 'get' of 'dict' objects> (<method 'get' of 'dict' objects>)
    <built-in method builtins.hasattr>                           --   254   254 -- 0.00032 0.00032 -- ~:0:<built-in method builtins.hasattr> (<built-in method builtins.hasattr>)
    <built-in method builtins.isinstance>                        --   661   662 -- 0.00091 0.00097 -- ~:0:<built-in method builtins.isinstance> (<built-in method builtins.isinstance>)
        __instancecheck__                                        --     6     6 -- 0.00001 0.00003 -- /usr/local/lib/python3.9/abc.py:96:__instancecheck__ (__instancecheck__)
        __instancecheck__                                        --     1     1 -- 0.00002 0.00002 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/parameter.py:8:__instancecheck__ (__instancecheck__)
    <method 'items' of 'collections.OrderedDict' objects>        --    20    20 -- 0.00001 0.00001 -- ~:0:<method 'items' of 'collections.OrderedDict' objects> (<method 'items' of 'collections.OrderedDict' objects>)
    <built-in method torch.is_grad_enabled>                      --   636   636 -- 0.00066 0.00066 -- ~:0:<built-in method torch.is_grad_enabled> (<built-in method torch.is_grad_enabled>)
    <built-in method builtins.getattr>                           --   237   237 -- 0.00035 0.00035 -- ~:0:<built-in method builtins.getattr> (<built-in method builtins.getattr>)
    <method 'setdefault' of 'dict' objects>                      --    14    14 -- 0.00001 0.00001 -- ~:0:<method 'setdefault' of 'dict' objects> (<method 'setdefault' of 'dict' objects>)
    <method 'add' of 'set' objects>                              --    13    13 -- 0.00001 0.00002 -- ~:0:<method 'add' of 'set' objects> (<method 'add' of 'set' objects>)
        __hash__                                                 --     6     6 -- 0.00000 0.00001 -- onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/_tensor.py:928:__hash__ (__hash__) +++
    <built-in method posix.fspath>                               --    24    24 -- 0.00001 0.00001 -- ~:0:<built-in method posix.fspath> (<built-in method posix.fspath>)
    <built-in method _thread.get_ident>                          --    12    12 -- 0.00001 0.00001 -- ~:0:<built-in method _thread.get_ident> (<built-in method _thread.get_ident>)
    <method 'rfind' of 'str' objects>                            --    18    18 -- 0.00003 0.00003 -- ~:0:<method 'rfind' of 'str' objects> (<method 'rfind' of 'str' objects>)




.. GENERATED FROM PYTHON SOURCE LINES 214-216

if GPU is available
+++++++++++++++++++

.. GENERATED FROM PYTHON SOURCE LINES 216-223

.. code-block:: default


    if get_device().upper() == 'GPU':

        device = torch.device('cuda:0')
        benches.append(benchmark(model_torch, model_ort, device, name='NN-GPU',
                                 max_iter=max_iter))








.. GENERATED FROM PYTHON SOURCE LINES 224-226

Linear Regression
+++++++++++++++++

.. GENERATED FROM PYTHON SOURCE LINES 226-269

.. code-block:: default



    class LinearRegressionNet(torch.nn.Module):
        def __init__(self, D_in, D_out):
            super(LinearRegressionNet, self).__init__()
            self.linear = torch.nn.Linear(D_in, D_out)

        def forward(self, x):
            return self.linear(x)


    d_in, d_out, N = X.shape[1], 1, X.shape[0]
    model_torch = LinearRegressionNet(d_in, d_out)
    try:
        model_ort = ORTModule(LinearRegressionNet(d_in, d_out))
    except Exception as e:
        model_ort = None
        print("ERROR: installation of torch extension for onnxruntime "
              "probably failed due to: ", e)

    device = torch.device('cpu')
    benches.append(benchmark(model_torch, model_ort, device, name='LR-CPU',
                             max_iter=max_iter))


    if get_device().upper() == 'GPU':

        device = torch.device('cuda:0')
        benches.append(benchmark(model_torch, model_ort, device, name='LR-GPU',
                                 max_iter=max_iter))

        ######################################
        # GPU profiling
        # +++++++++++++

        if get_device().upper() == 'GPU':
            ps = profile(lambda: benchmark(
                model_torch, model_ort, device, name='LR-GPU',
                max_iter=max_iter))[0]
            root, nodes = profile2graph(ps, clean_text=clean_name)
            text = root.to_text()
            print(text)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    ERROR: installation of torch extension for onnxruntime probably failed due to:  ORTModule's extensions were not detected at 'somewhere/workspace/onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/onnxruntime/training/ortmodule/torch_cpp_extensions' folder. Run `python -m torch_ort.configure` before using `ORTModule` frontend.
    [benchmark] LR-CPU
    somewhere/workspace/onnxcustom/onnxcustom_UT_39_std/_venv/lib/python3.9/site-packages/torch/nn/modules/loss.py:536: UserWarning: Using a target size (torch.Size([750])) that is different to the input size (torch.Size([750, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
      return F.mse_loss(input, target, reduction=self.reduction)
    [benchmark] torch=200 iterations - 0.8049870599061251 seconds
    [benchmark] onxrt=0 iteration - 0 seconds




.. GENERATED FROM PYTHON SOURCE LINES 270-274

Graphs
++++++

Dataframe first.

.. GENERATED FROM PYTHON SOURCE LINES 274-278

.. code-block:: default


    df = DataFrame(benches).set_index('name')
    df






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>torch</th>
          <th>ort</th>
          <th>iter_torch</th>
          <th>iter_ort</th>
        </tr>
        <tr>
          <th>name</th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>NN-CPU</th>
          <td>1.582951</td>
          <td>0</td>
          <td>200</td>
          <td>0</td>
        </tr>
        <tr>
          <th>LR-CPU</th>
          <td>0.804987</td>
          <td>0</td>
          <td>200</td>
          <td>0</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 279-280

text output

.. GENERATED FROM PYTHON SOURCE LINES 280-283

.. code-block:: default


    print(df)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

               torch  ort  iter_torch  iter_ort
    name                                       
    NN-CPU  1.582951    0         200         0
    LR-CPU  0.804987    0         200         0




.. GENERATED FROM PYTHON SOURCE LINES 284-285

Graphs.

.. GENERATED FROM PYTHON SOURCE LINES 285-293

.. code-block:: default


    print(df.columns)
    fig, ax = plt.subplots(1, 1, figsize=(4, 4))
    df[['torch', 'ort']].plot.bar(title="Processing time", ax=ax)
    ax.tick_params(axis='x', rotation=30)
    fig.savefig("plot_orttraining_benchmark_torch.png")

    # plt.show()



.. image-sg:: /gyexamples/images/sphx_glr_plot_orttraining_benchmark_torch_001.png
   :alt: Processing time
   :srcset: /gyexamples/images/sphx_glr_plot_orttraining_benchmark_torch_001.png
   :class: sphx-glr-single-img


.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Index(['torch', 'ort', 'iter_torch', 'iter_ort'], dtype='object')





.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 0 minutes  4.319 seconds)


.. _sphx_glr_download_gyexamples_plot_orttraining_benchmark_torch.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example


    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_orttraining_benchmark_torch.py <plot_orttraining_benchmark_torch.py>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_orttraining_benchmark_torch.ipynb <plot_orttraining_benchmark_torch.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_