OrtValue

onnxruntime implements tensors with class OrtValue. It has the same properties as a numpy.array, shape and type but only represents a contiguous array. The current implementation is just a container and does not allow standard operators such as addition, substraction. onnxruntime has a C implementation wrapped into a Python class with the same.

Python Wrapper for OrtValue

Note

Method ortvalue_from_numpy does not copy data, it borrows the data pointer. The numpy array must remain alive while the instance of OrtValue is in use.

class onnxruntime.OrtValue(ortvalue, numpy_obj=None)

A data structure that supports all ONNX data formats (tensors and non-tensors) that allows users to place the data backing these on a device, for example, on a CUDA supported device. This class provides APIs to construct and deal with OrtValues.

__init__(ortvalue, numpy_obj=None)

_get_c_value()

as_sparse_tensor()

The function will return SparseTensor contained in this OrtValue

data_ptr()

Returns the address of the first element in the OrtValue’s data buffer

data_type()

Returns the data type of the data in the OrtValue

device_name()

Returns the name of the device where the OrtValue’s data buffer resides e.g. cpu, cuda

element_type()

Returns the proto type of the data in the OrtValue if the OrtValue is a tensor.

has_value()

Returns True if the OrtValue corresponding to an optional type contains data, else returns False

is_sparse_tensor()

Returns True if the OrtValue contains a SparseTensor, else returns False

is_tensor()

Returns True if the OrtValue contains a Tensor, else returns False

is_tensor_sequence()

Returns True if the OrtValue contains a Tensor Sequence, else returns False

numpy()

Returns a Numpy object from the OrtValue. Valid only for OrtValues holding Tensors. Throws for OrtValues holding non-Tensors. Use accessors to gain a reference to non-Tensor objects such as SparseTensor

static ort_value_from_sparse_tensor(sparse_tensor)

The function will construct an OrtValue instance from a valid SparseTensor The new instance of OrtValue will assume the ownership of sparse_tensor

static ortvalue_from_numpy(numpy_obj, device_type='cpu', device_id=0)

Factory method to construct an OrtValue (which holds a Tensor) from a given Numpy object A copy of the data in the Numpy object is held by the OrtValue only if the device is NOT cpu

Parameters:

• numpy_obj – The Numpy object to construct the OrtValue from

• device_type – e.g. cpu, cuda, cpu by default

• device_id – device id, e.g. 0

static ortvalue_from_shape_and_type(shape=None, element_type=None, device_type='cpu', device_id=0)

Factory method to construct an OrtValue (which holds a Tensor) from given shape and element_type

Parameters:

• shape – List of integers indicating the shape of the OrtValue

• element_type – The data type of the elements in the OrtValue (numpy type)

• device_type – e.g. cpu, cuda, cpu by default

• device_id – device id, e.g. 0

shape()

Returns the shape of the data in the OrtValue

update_inplace(np_arr)

Update the OrtValue in place with a new Numpy array. The numpy contents are copied over to the device memory backing the OrtValue. It can be used to update the input valuess for an InferenceSession with CUDA graph enabled or other scenarios where the OrtValue needs to be updated while the memory address can not be changed.

C Class OrtValue or C_OrtValue

class onnxruntime.capi._pybind_state.OrtValue

__dlpack__(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValue, stream: object = None) → object

Returns a DLPack representing the tensor (part of __dlpack__ protocol). This method does not copy the pointer shape, instead, it copies the pointer value. The OrtValue must persist until the dlpack structure is consumed.

__dlpack_device__(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValue) → tuple

Returns a tuple of integers, (device, device index) (part of __dlpack__ protocol).

__init__(*args, **kwargs)

as_sparse_tensor(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValue) → onnxruntime::python::PySparseTensor

data_ptr(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValue) → int

data_type(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValue) → str

device_name(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValue) → str

element_type(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValue) → int

Returns an integer equal to the ONNX tensor proto type of the tensor or sequence. This integer is one type defined by ONNX TensorProto_DataType (such as onnx.TensorProto.FLOAT).Raises an exception in any other case.

static from_dlpack(data: object, is_bool_tensor: bool = False) → onnxruntime.capi.onnxruntime_pybind11_state.OrtValue

Converts a tensor from a external library into an OrtValue by means of the __dlpack__ protocol.

has_value(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValue) → bool

is_sparse_tensor(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValue) → bool

is_tensor(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValue) → bool

is_tensor_sequence(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValue) → bool

numpy(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValue) → object

static ort_value_from_sparse_tensor(arg0: onnxruntime::python::PySparseTensor) → onnxruntime.capi.onnxruntime_pybind11_state.OrtValue

static ortvalue_from_numpy(arg0: object, arg1: onnxruntime.capi.onnxruntime_pybind11_state.OrtDevice) → onnxruntime.capi.onnxruntime_pybind11_state.OrtValue

static ortvalue_from_shape_and_type(arg0: List[int], arg1: object, arg2: onnxruntime.capi.onnxruntime_pybind11_state.OrtDevice) → onnxruntime.capi.onnxruntime_pybind11_state.OrtValue

shape(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValue) → list

to_dlpack(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValue) → object

Returns a DLPack representing the tensor. This method does not copy the pointer shape, instead, it copies the pointer value. The OrtValue must be persist until the dlpack structure is consumed.

update_inplace(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValue, arg0: numpy.ndarray) → None

C Class OrtValueVector

class onnxruntime.capi._pybind_state.OrtValueVector(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValueVector)

__getitem__(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValueVector, arg0: int) → onnxruntime.capi.onnxruntime_pybind11_state.OrtValue

__init__(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValueVector) → None

__iter__(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValueVector) → Iterator

__len__(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValueVector) → int

bool_tensor_indices(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValueVector) → List[int]

Returns the indices of every boolean tensor in this vector of OrtValue. In case of a boolean tensor, method to_dlpacks returns a uint8 tensor instead of a boolean tensor. If torch consumes the dlpack structure, .to(torch.bool) must be applied to the torch tensor to get a boolean tensor.

dlpack_at(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValueVector, arg0: int) → object

element_type_at(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValueVector, idx: int) → int

Returns an integer equal to the ONNX proto type of the tensor at position i. This integer is one type defined by ONNX TensorProto_DataType (such as onnx.TensorProto.FLOAT).Raises an exception in any other case.

push_back(*args, **kwargs)

Overloaded function.

1. push_back(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValueVector, arg0: onnxruntime.capi.onnxruntime_pybind11_state.OrtValue) -> None

2. push_back(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValueVector, dlpack_tensor: object, is_bool_tensor: bool = False) -> None

Add a new OrtValue after being ownership was transferred from the DLPack structure.

push_back_batch(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValueVector, arg0: List[object], arg1: List[int], arg2: List[object], arg3: List[List[int]], arg4: List[onnxruntime.capi.onnxruntime_pybind11_state.OrtDevice]) → None

Add a batch of OrtValue’s by wrapping PyTorch tensors.

reserve(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValueVector, arg0: int) → None

shrink_to_fit(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValueVector) → None

to_dlpacks(self: onnxruntime.capi.onnxruntime_pybind11_state.OrtValueVector, to_tensor: object) → list

Converts all OrtValue into tensors through DLPack protocol, the method creates a DLPack structure for every tensors, then calls python function to_tensor to a new object consuming the DLPack structure or return a list of capsule if this function is None.

Parameters:

to_tensor – this function takes a capsule holding a pointer onto a DLPack structure and returns a new tensor which becomes the new owner of the data. This function takes one python object and returns a new python object. It fits the same signature as torch.utils.from_dlpack, if None, the method returns a capsule for every new DLPack structure.

Returns:

a list containing the new tensors or a the new capsules if to_tensor is None

This method is used to replace tuple(torch._C._from_dlpack(ov.to_dlpack()) for ov in ort_values) by a faster instruction tuple(ort_values.to_dlpack(torch._C._from_dlpack)). This loop is difficult to parallelize as it goes through the GIL many times. It creates many tensors acquiring ownership of existing OrtValue. This method saves one object creation and an C++ allocation for every transferred tensor.