NegativeLogLikelihoodLoss#

NegativeLogLikelihoodLoss - 13#

Version

This version of the operator has been available since version 13.

Summary

A NegativeLogLikelihoodLoss operator computes (weighted) negative log likelihood loss. Its “input” tensor has the shape of (N, C, d1, d2, …, dk) where k >= 0. The “input” tensor contains log-probabilities for input[n, :, d_1, d_2,…, d_k] being in a class of [0, C). The operator’s “target” input tensor has the shape of (N, d1, d2, …, dk). It encodes class labels (one of C classes) or it may contain a special value (indicated by an attribute ignore_index) for N x d1 x d2 x … x dk samples. The loss value for input[n, :, d_1, d_2,…d_k] being classified as class c = target[n][d_1][d_2]…[d_k] is computed as:

loss[n][d_1][d_2]…[d_k] = -input[n][c][d_1][d_2]…[d_k].

When an optional “weight” is provided, the sample loss is calculated as:

loss[n][d_1][d_2]…[d_k] = -input[n][c][d_1][d_2]…[d_k] * weight[c].

loss is zero for the case when target-value equals ignore_index.

loss[n][d_1][d_2]…[d_k] = 0, when target[n][d_1][d_2]…[d_k] = ignore_index

If “reduction” attribute is set to “none”, the operator’s output will be the above loss with shape (N, d1, d2, …, dk). If “reduction” attribute is set to “mean” (the default attribute value), the output loss is (weight) averaged:

mean(loss), if “weight” is not provided,

or if weight is provided,

sum(loss) / sum(weight[target[n][d_1][d_2]…[d_k]]]), for all samples.

If “reduction” attribute is set to “sum”, the output is a scalar:

sum(loss).

See also https://pytorch.org/docs/stable/nn.html#torch.nn.NLLLoss.

Example 1:

// negative log likelihood loss, “none” reduction N, C, d1 = 2, 3, 2 input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]],

[[0.0, 1.0], [2.0, 2.0], [1.0, 2]]]

target = [[2, 1], [0, 2]]

loss = np.zeros((N, d1)) for n in range(N):

for d_1 in range(d1):

c = target[n][d_1] loss[n][d_1] = -input[n][c][d_1]

// print(loss) // [[-3. -2.] // [-0. -2.]]

Example 2:

// weighted negative log likelihood loss, sum reduction N, C, d1 = 2, 3, 2 input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]],

[[0.0, 1.0], [2.0, 2.0], [1.0, 2]]]

target = [[2, 1], [0, 2]] weight = [0.2, 0.3, 0.1] loss = np.zeros((N, d1)) for n in range(N):

for d_1 in range(d1):

c = target[n][d_1] loss[n][d_1] = -input[n][c][d_1] * weight[c]

loss = np.sum(loss) // print(loss) // -1.1

Example 3:

// weighted negative log likelihood loss, mean reduction N, C, d1 = 2, 3, 2 input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]],

[[0.0, 1.0], [2.0, 2.0], [1.0, 2]]]

target = [[2, 1], [0, 2]] weight = [0.2, 0.3, 0.1] loss = np.zeros((N, d1)) weight_total = 0 for n in range(N):

for d_1 in range(d1):

c = target[n][d_1] loss[n][d_1] = -input[n][c][d_1] * weight[c] weight_total = weight_total + weight[c]

loss = np.sum(loss) / weight_total // print(loss) // -1.57

Attributes

  • ignore_index: Specifies a target value that is ignored and does not contribute to the input gradient. It’s an optional value.

  • reduction: Type of reduction to apply to loss: none, sum, mean (default). ‘none’: the output is the loss for each sample. ‘sum’: the output will be summed. ‘mean’: the sum of the output will be divided by the sum of applied weights. Default value is 'mean'.

Inputs

Between 2 and 3 inputs.

  • input (heterogeneous) - T: Input tensor of shape (N, C) or (N, C, d1, d2, …, dk).

  • target (heterogeneous) - Tind: Target tensor of shape (N) or (N, d1, d2, …, dk). Target element value shall be in range of [0, C). If ignore_index is specified, it may have a value outside [0, C) and the target values should either be in the range [0, C) or have the value ignore_index.

  • weight (optional, heterogeneous) - T: Optional rescaling weight tensor. If given, it has to be a tensor of size C. Otherwise, it is treated as if having all ones.

Outputs

  • loss (heterogeneous) - T: The negative log likelihood loss

Type Constraints

  • T in ( tensor(double), tensor(float), tensor(float16) ): Constrain input, weight, and output types to floating-point tensors.

  • Tind in ( tensor(int32), tensor(int64) ): Constrain target to integer types

Examples

_input_shape_is_NC

reduction = 'none'
node = onnx.helper.make_node(
    'NegativeLogLikelihoodLoss',
    inputs=['input', 'target'],
    outputs=['loss'],
    reduction=reduction
)

N, C = 3, 5
np.random.seed(0)
input = np.random.rand(N, C).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, )).astype(np.int64)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(input, target, weight=None, reduction=reduction)

expect(node, inputs=[input, target], outputs=[negative_log_likelihood_loss],
    name='test_nllloss_NC')

_input_shape_is_NCd1d2

reduction = 'none'
node = onnx.helper.make_node(
    'NegativeLogLikelihoodLoss',
    inputs=['input', 'target'],
    outputs=['loss'],
    reduction=reduction
)

N, C, dim1, dim2 = 3, 5, 6, 6
np.random.seed(0)
input = np.random.rand(N, C, dim1, dim2).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, dim1, dim2)).astype(np.int64)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(input, target, weight=None, reduction=reduction)

expect(node, inputs=[input, target], outputs=[negative_log_likelihood_loss],
    name='test_nllloss_NCd1d2')

_input_shape_is_NCd1d2_reduction_mean

reduction = 'mean'
node = onnx.helper.make_node(
    'NegativeLogLikelihoodLoss',
    inputs=['input', 'target'],
    outputs=['loss'],
    reduction=reduction
)

N, C, dim1, dim2 = 3, 5, 6, 6
np.random.seed(0)
input = np.random.rand(N, C, dim1, dim2).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, dim1, dim2)).astype(np.int64)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(input, target, weight=None, reduction=reduction)

expect(node, inputs=[input, target], outputs=[negative_log_likelihood_loss],
    name='test_nllloss_NCd1d2_reduction_mean')

_input_shape_is_NCd1d2_reduction_sum

reduction = 'sum'
node = onnx.helper.make_node(
    'NegativeLogLikelihoodLoss',
    inputs=['input', 'target'],
    outputs=['loss'],
    reduction=reduction
)

N, C, dim1, dim2 = 3, 5, 6, 6
np.random.seed(0)
input = np.random.rand(N, C, dim1, dim2).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, dim1, dim2)).astype(np.int64)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(input, target, weight=None, reduction=reduction)

expect(node, inputs=[input, target], outputs=[negative_log_likelihood_loss],
    name='test_nllloss_NCd1d2_reduction_sum')

_input_shape_is_NCd1d2_with_weight

reduction = 'none'
node = onnx.helper.make_node(
    'NegativeLogLikelihoodLoss',
    inputs=['input', 'target', 'weight'],
    outputs=['loss'],
    reduction=reduction
)

N, C, dim1, dim2 = 3, 5, 6, 6
np.random.seed(0)
input = np.random.rand(N, C, dim1, dim2).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, dim1, dim2)).astype(np.int64)
weight = np.random.rand(C).astype(np.float32)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(input, target, weight=weight, reduction=reduction)

expect(node, inputs=[input, target, weight], outputs=[negative_log_likelihood_loss],
    name='test_nllloss_NCd1d2_with_weight')

_input_shape_is_NCd1d2_with_weight_reduction_mean

reduction = 'mean'
node = onnx.helper.make_node(
    'NegativeLogLikelihoodLoss',
    inputs=['input', 'target', 'weight'],
    outputs=['loss'],
    reduction=reduction
)

N, C, dim1, dim2 = 3, 5, 6, 6
np.random.seed(0)
input = np.random.rand(N, C, dim1, dim2).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, dim1, dim2)).astype(np.int64)
weight = np.random.rand(C).astype(np.float32)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(input, target, weight=weight, reduction=reduction)

expect(node, inputs=[input, target, weight], outputs=[negative_log_likelihood_loss],
    name='test_nllloss_NCd1d2_with_weight_reduction_mean')

_input_shape_is_NCd1d2_with_weight_reduction_sum

reduction = 'sum'
node = onnx.helper.make_node(
    'NegativeLogLikelihoodLoss',
    inputs=['input', 'target', 'weight'],
    outputs=['loss'],
    reduction=reduction
)

N, C, dim1, dim2 = 3, 5, 6, 6
np.random.seed(0)
input = np.random.rand(N, C, dim1, dim2).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, dim1, dim2)).astype(np.int64)
weight = np.random.rand(C).astype(np.float32)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(input, target, weight=weight, reduction=reduction)

expect(node, inputs=[input, target, weight], outputs=[negative_log_likelihood_loss],
    name='test_nllloss_NCd1d2_with_weight_reduction_sum')

_input_shape_is_NCd1d2_with_weight_reduction_sum_ii

reduction = 'sum'
ignore_index = np.int64(0)
node = onnx.helper.make_node(
    'NegativeLogLikelihoodLoss',
    inputs=['input', 'target', 'weight'],
    outputs=['loss'],
    reduction=reduction,
    ignore_index=ignore_index
)

N, C, dim1, dim2 = 3, 5, 6, 6
np.random.seed(0)
input = np.random.rand(N, C, dim1, dim2).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, dim1, dim2)).astype(np.int64)
target[0][0][0] = np.int64(0)
weight = np.random.rand(C).astype(np.float32)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(input, target, weight=weight, reduction=reduction, ignore_index=ignore_index)

expect(node, inputs=[input, target, weight], outputs=[negative_log_likelihood_loss],
    name='test_nllloss_NCd1d2_with_weight_reduction_sum_ii')

_input_shape_is_NCd1d2_no_weight_reduction_mean_ii

reduction = 'mean'
ignore_index = np.int64(1)
node = onnx.helper.make_node(
    'NegativeLogLikelihoodLoss',
    inputs=['input', 'target'],
    outputs=['loss'],
    reduction=reduction,
    ignore_index=ignore_index
)

N, C, dim1, dim2 = 3, 5, 6, 6
np.random.seed(0)
input = np.random.rand(N, C, dim1, dim2).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, dim1, dim2)).astype(np.int64)
target[0][0][0] = np.int64(1)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(input, target, reduction=reduction, ignore_index=ignore_index)

expect(node, inputs=[input, target], outputs=[negative_log_likelihood_loss],
    name='test_nllloss_NCd1d2_no_weight_reduction_mean_ii')

_input_shape_is_NCd1

reduction = 'mean'
node = onnx.helper.make_node(
    'NegativeLogLikelihoodLoss',
    inputs=['input', 'target'],
    outputs=['loss'],
    reduction=reduction
)

N, C, d1 = 3, 5, 2
np.random.seed(0)
input = np.random.rand(N, C, d1).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, d1)).astype(np.int64)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(input, target, weight=None, reduction=reduction)

expect(node, inputs=[input, target], outputs=[negative_log_likelihood_loss],
    name='test_nllloss_NCd1')

_input_shape_is_NCd1_weight

reduction = 'mean'
node = onnx.helper.make_node(
    'NegativeLogLikelihoodLoss',
    inputs=['input', 'target', 'weight'],
    outputs=['loss'],
    reduction=reduction
)

N, C, d1 = 3, 5, 2
np.random.seed(0)
input = np.random.rand(N, C, d1).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, d1)).astype(np.int64)
weight = np.random.rand(C).astype(np.float32)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(input, target, weight=weight, reduction=reduction)

expect(node, inputs=[input, target, weight], outputs=[negative_log_likelihood_loss],
    name='test_nllloss_NCd1_weight')

_input_shape_is_NCd1_ii

reduction = 'mean'
ignore_index = np.int64(1)
node = onnx.helper.make_node(
    'NegativeLogLikelihoodLoss',
    inputs=['input', 'target'],
    outputs=['loss'],
    reduction=reduction,
    ignore_index=ignore_index
)

N, C, d1 = 3, 5, 2
np.random.seed(0)
input = np.random.rand(N, C, d1).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, d1)).astype(np.int64)
target[0][0] = np.int64(1)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(input, target, weight=None, reduction=reduction, ignore_index=ignore_index)

expect(node, inputs=[input, target], outputs=[negative_log_likelihood_loss],
    name='test_nllloss_NCd1_ii')

_input_shape_is_NCd1_weight_ii

reduction = 'mean'
ignore_index = np.int64(1)
node = onnx.helper.make_node(
    'NegativeLogLikelihoodLoss',
    inputs=['input', 'target', 'weight'],
    outputs=['loss'],
    reduction=reduction,
    ignore_index=ignore_index
)

N, C, d1 = 3, 5, 2
np.random.seed(0)
input = np.random.rand(N, C, d1).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, d1)).astype(np.int64)
target[0][0] = np.int64(1)
weight = np.random.rand(C).astype(np.float32)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(input, target, weight=weight, reduction=reduction, ignore_index=ignore_index)

expect(node, inputs=[input, target, weight], outputs=[negative_log_likelihood_loss],
    name='test_nllloss_NCd1_weight_ii')

_input_shape_is_NCd1d2d3d4d5_mean_weight

reduction = 'mean'

node = onnx.helper.make_node(
    'NegativeLogLikelihoodLoss',
    inputs=['input', 'target', 'weight'],
    outputs=['loss'],
    reduction=reduction)

N, C, dim1, dim2, dim3, dim4, dim5 = 3, 5, 6, 6, 5, 3, 4
np.random.seed(0)
input = np.random.rand(N, C, dim1, dim2, dim3, dim4, dim5).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, dim1, dim2, dim3, dim4, dim5)).astype(np.int64)
weight = np.random.rand(C).astype(np.float32)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(input,
                                                                    target,
                                                                    weight=weight,
                                                                    reduction=reduction)

expect(node, inputs=[input, target, weight], outputs=[negative_log_likelihood_loss],
    name='test_nllloss_NCd1d2d3d4d5_mean_weight')

_input_shape_is_NCd1d2d3d4d5_none_no_weight

reduction = 'none'

node = onnx.helper.make_node(
    'NegativeLogLikelihoodLoss',
    inputs=['input', 'target'],
    outputs=['loss'],
    reduction=reduction)

N, C, dim1, dim2, dim3, dim4, dim5 = 3, 5, 6, 6, 5, 3, 4
np.random.seed(0)
input = np.random.rand(N, C, dim1, dim2, dim3, dim4, dim5).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, dim1, dim2, dim3, dim4, dim5)).astype(np.int64)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(input,
                                                                    target,
                                                                    reduction=reduction)

expect(node, inputs=[input, target], outputs=[negative_log_likelihood_loss],
    name='test_nllloss_NCd1d2d3d4d5_none_no_weight')

_input_shape_is_NCd1_mean_weight_negative_ii

reduction = 'mean'
ignore_index = np.int64(-1)

node = onnx.helper.make_node(
    'NegativeLogLikelihoodLoss',
    inputs=['input', 'target', 'weight'],
    outputs=['loss'],
    reduction=reduction,
    ignore_index=ignore_index)

N, C, dim1 = 3, 5, 6
np.random.seed(0)
input = np.random.rand(N, C, dim1).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, dim1)).astype(np.int64)
target[0][0] = -1
weight = np.random.rand(C).astype(np.float32)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(input,
                                                                    target,
                                                                    weight=weight,
                                                                    reduction=reduction,
                                                                    ignore_index=ignore_index)

expect(node, inputs=[input, target, weight], outputs=[negative_log_likelihood_loss],
    name='test_nllloss_NCd1_mean_weight_negative_ii')

_input_shape_is_NCd1d2d3_none_no_weight_negative_ii

reduction = 'none'
ignore_index = np.int64(-5)

node = onnx.helper.make_node(
    'NegativeLogLikelihoodLoss',
    inputs=['input', 'target'],
    outputs=['loss'],
    reduction=reduction,
    ignore_index=ignore_index)

N, C, dim1, dim2, dim3 = 3, 5, 6, 6, 5
np.random.seed(0)
input = np.random.rand(N, C, dim1, dim2, dim3).astype(np.float32)
target = np.random.randint(0, high=C, size=(N, dim1, dim2, dim3)).astype(np.int64)
target[0][0][0][0] = -5

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(input,
                                                                    target,
                                                                    reduction=reduction,
                                                                    ignore_index=ignore_index)

expect(node, inputs=[input, target], outputs=[negative_log_likelihood_loss],
    name='test_nllloss_NCd1d2d3_none_no_weight_negative_ii')

_input_shape_is_NCd1d2d3_sum_weight_high_ii

reduction = 'sum'
ignore_index = np.int64(10)

node = onnx.helper.make_node(
    'NegativeLogLikelihoodLoss',
    inputs=['input', 'target', 'weight'],
    outputs=['loss'],
    reduction=reduction,
    ignore_index=ignore_index)

N, C = 3, 5
np.random.seed(0)
input = np.random.rand(N, C).astype(np.float32)
target = np.random.randint(0, high=C, size=(N)).astype(np.int64)
target[0] = 10
weight = np.random.rand(C).astype(np.float32)

negative_log_likelihood_loss = compute_negative_log_likelihood_loss(input,
                                                                    target,
                                                                    weight=weight,
                                                                    reduction=reduction,
                                                                    ignore_index=ignore_index)

expect(node, inputs=[input, target, weight], outputs=[negative_log_likelihood_loss],
    name='test_nllloss_NCd1d2d3_sum_weight_high_ii')

Differences

00A NegativeLogLikelihoodLoss operator computes (weighted) negative log likelihood loss.A NegativeLogLikelihoodLoss operator computes (weighted) negative log likelihood loss.
11Its "input" tensor has the shape of (N, C, d1, d2, ..., dk) where k >= 0.Its "input" tensor has the shape of (N, C, d1, d2, ..., dk) where k >= 0.
22The "input" tensor contains log-probabilities for input[n, :, d_1, d_2,..., d_k] being in a class of [0, C).The "input" tensor contains log-probabilities for input[n, :, d_1, d_2,..., d_k] being in a class of [0, C).
33The operator's "target" input tensor has the shape of (N, d1, d2, ..., dk). It encodes class labels (one of C classes)The operator's "target" input tensor has the shape of (N, d1, d2, ..., dk). It encodes class labels (one of C classes)
44or it may contain a special value (indicated by an attribute ignore_index) for N x d1 x d2 x ... x dk samples.or it may contain a special value (indicated by an attribute ignore_index) for N x d1 x d2 x ... x dk samples.
55The loss value for input[n, :, d_1, d_2,...d_k] being classified as class c = target[n][d_1][d_2]...[d_k] is computed as:The loss value for input[n, :, d_1, d_2,...d_k] being classified as class c = target[n][d_1][d_2]...[d_k] is computed as:
6
67 loss[n][d_1][d_2]...[d_k] = -input[n][c][d_1][d_2]...[d_k]. loss[n][d_1][d_2]...[d_k] = -input[n][c][d_1][d_2]...[d_k].
8
79When an optional "weight" is provided, the sample loss is calculated as:When an optional "weight" is provided, the sample loss is calculated as:
10
811 loss[n][d_1][d_2]...[d_k] = -input[n][c][d_1][d_2]...[d_k] * weight[c]. loss[n][d_1][d_2]...[d_k] = -input[n][c][d_1][d_2]...[d_k] * weight[c].
12
913loss is zero for the case when target-value equals ignore_index.loss is zero for the case when target-value equals ignore_index.
1014
1115 loss[n][d_1][d_2]...[d_k] = 0, when target[n][d_1][d_2]...[d_k] = ignore_index loss[n][d_1][d_2]...[d_k] = 0, when target[n][d_1][d_2]...[d_k] = ignore_index
16
1217If "reduction" attribute is set to "none", the operator's output will be the above loss with shape (N, d1, d2, ..., dk).If "reduction" attribute is set to "none", the operator's output will be the above loss with shape (N, d1, d2, ..., dk).
1318If "reduction" attribute is set to "mean" (the default attribute value), the output loss is (weight) averaged:If "reduction" attribute is set to "mean" (the default attribute value), the output loss is (weight) averaged:
19
1420 mean(loss), if "weight" is not provided, mean(loss), if "weight" is not provided,
21
1522or if weight is provided,or if weight is provided,
23
1624 sum(loss) / sum(weight[target[n][d_1][d_2]...[d_k]]]), for all samples. sum(loss) / sum(weight[target[n][d_1][d_2]...[d_k]]]), for all samples.
25
1726If "reduction" attribute is set to "sum", the output is a scalar:If "reduction" attribute is set to "sum", the output is a scalar:
1827 sum(loss). sum(loss).
28
1929See also https://pytorch.org/docs/stable/nn.html#torch.nn.NLLLoss.See also https://pytorch.org/docs/stable/nn.html#torch.nn.NLLLoss.
30
2031Example 1:Example 1:
32
2133 // negative log likelihood loss, "none" reduction // negative log likelihood loss, "none" reduction
2234 N, C, d1 = 2, 3, 2 N, C, d1 = 2, 3, 2
2335 input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]], input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]],
2436 [[0.0, 1.0], [2.0, 2.0], [1.0, 2]]] [[0.0, 1.0], [2.0, 2.0], [1.0, 2]]]
2537 target = [[2, 1], [0, 2]] target = [[2, 1], [0, 2]]
38
2639 loss = np.zeros((N, d1)) loss = np.zeros((N, d1))
2740 for n in range(N): for n in range(N):
2841 for d_1 in range(d1): for d_1 in range(d1):
2942 c = target[n][d_1] c = target[n][d_1]
3043 loss[n][d_1] = -input[n][c][d_1] loss[n][d_1] = -input[n][c][d_1]
44
3145 // print(loss) // print(loss)
3246 // [[-3. -2.] // [[-3. -2.]
3347 // [-0. -2.]] // [-0. -2.]]
48
3449Example 2:Example 2:
50
3551 // weighted negative log likelihood loss, sum reduction // weighted negative log likelihood loss, sum reduction
3652 N, C, d1 = 2, 3, 2 N, C, d1 = 2, 3, 2
3753 input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]], input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]],
3854 [[0.0, 1.0], [2.0, 2.0], [1.0, 2]]] [[0.0, 1.0], [2.0, 2.0], [1.0, 2]]]
3955 target = [[2, 1], [0, 2]] target = [[2, 1], [0, 2]]
4056 weight = [0.2, 0.3, 0.1] weight = [0.2, 0.3, 0.1]
4157 loss = np.zeros((N, d1)) loss = np.zeros((N, d1))
4258 for n in range(N): for n in range(N):
4359 for d_1 in range(d1): for d_1 in range(d1):
4460 c = target[n][d_1] c = target[n][d_1]
4561 loss[n][d_1] = -input[n][c][d_1] * weight[c] loss[n][d_1] = -input[n][c][d_1] * weight[c]
62
4663 loss = np.sum(loss) loss = np.sum(loss)
4764 // print(loss) // print(loss)
4865 // -1.1 // -1.1
66
4967Example 3:Example 3:
68
5069 // weighted negative log likelihood loss, mean reduction // weighted negative log likelihood loss, mean reduction
5170 N, C, d1 = 2, 3, 2 N, C, d1 = 2, 3, 2
5271 input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]], input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]],
5372 [[0.0, 1.0], [2.0, 2.0], [1.0, 2]]] [[0.0, 1.0], [2.0, 2.0], [1.0, 2]]]
5473 target = [[2, 1], [0, 2]] target = [[2, 1], [0, 2]]
5574 weight = [0.2, 0.3, 0.1] weight = [0.2, 0.3, 0.1]
5675 loss = np.zeros((N, d1)) loss = np.zeros((N, d1))
5776 weight_total = 0 weight_total = 0
5877 for n in range(N): for n in range(N):
5978 for d_1 in range(d1): for d_1 in range(d1):
6079 c = target[n][d_1] c = target[n][d_1]
6180 loss[n][d_1] = -input[n][c][d_1] * weight[c] loss[n][d_1] = -input[n][c][d_1] * weight[c]
6281 weight_total = weight_total + weight[c] weight_total = weight_total + weight[c]
82
6383 loss = np.sum(loss) / weight_total loss = np.sum(loss) / weight_total
6484 // print(loss) // print(loss)
6585 // -1.57 // -1.57
6686
6787**Attributes****Attributes**
6888
6989* **ignore_index**:* **ignore_index**:
7090 Specifies a target value that is ignored and does not contribute to Specifies a target value that is ignored and does not contribute to
7191 the input gradient. It's an optional value. the input gradient. It's an optional value.
7292* **reduction**:* **reduction**:
7393 Type of reduction to apply to loss: none, sum, mean (default). Type of reduction to apply to loss: none, sum, mean (default).
7494 'none': the output is the loss for each sample. 'sum': the output 'none': the output is the loss for each sample. 'sum': the output
7595 will be summed. 'mean': the sum of the output will be divided by the will be summed. 'mean': the sum of the output will be divided by the
7696 sum of applied weights. Default value is 'mean'. sum of applied weights. Default value is 'mean'.
7797
7898**Inputs****Inputs**
7999
80100Between 2 and 3 inputs.Between 2 and 3 inputs.
81101
82102* **input** (heterogeneous) - **T**:* **input** (heterogeneous) - **T**:
83103 Input tensor of shape (N, C) or (N, C, d1, d2, ..., dk). Input tensor of shape (N, C) or (N, C, d1, d2, ..., dk).
84104* **target** (heterogeneous) - **Tind**:* **target** (heterogeneous) - **Tind**:
85105 Target tensor of shape (N) or (N, d1, d2, ..., dk). Target element Target tensor of shape (N) or (N, d1, d2, ..., dk). Target element
86106 value shall be in range of [0, C). If ignore_index is specified, it value shall be in range of [0, C). If ignore_index is specified, it
87107 may have a value outside [0, C) and the target values should either may have a value outside [0, C) and the target values should either
88108 be in the range [0, C) or have the value ignore_index. be in the range [0, C) or have the value ignore_index.
89109* **weight** (optional, heterogeneous) - **T**:* **weight** (optional, heterogeneous) - **T**:
90110 Optional rescaling weight tensor. If given, it has to be a tensor of Optional rescaling weight tensor. If given, it has to be a tensor of
91111 size C. Otherwise, it is treated as if having all ones. size C. Otherwise, it is treated as if having all ones.
92112
93113**Outputs****Outputs**
94114
95115* **loss** (heterogeneous) - **T**:* **loss** (heterogeneous) - **T**:
96116 The negative log likelihood loss The negative log likelihood loss
97117
98118**Type Constraints****Type Constraints**
99119
100120* **T** in (* **T** in (
101121 tensor(double), tensor(double),
102122 tensor(float), tensor(float),
103123 tensor(float16) tensor(float16)
104124 ): ):
105125 Constrain input, weight, and output types to floating-point tensors. Constrain input, weight, and output types to floating-point tensors.
106126* **Tind** in (* **Tind** in (
107127 tensor(int32), tensor(int32),
108128 tensor(int64) tensor(int64)
109129 ): ):
110130 Constrain target to integer types Constrain target to integer types

NegativeLogLikelihoodLoss - 12#

Version

This version of the operator has been available since version 12.

Summary

A NegativeLogLikelihoodLoss operator computes (weighted) negative log likelihood loss. Its “input” tensor has the shape of (N, C, d1, d2, …, dk) where k >= 0. The “input” tensor contains log-probabilities for input[n, :, d_1, d_2,…, d_k] being in a class of [0, C). The operator’s “target” input tensor has the shape of (N, d1, d2, …, dk). It encodes class labels (one of C classes) or it may contain a special value (indicated by an attribute ignore_index) for N x d1 x d2 x … x dk samples. The loss value for input[n, :, d_1, d_2,…d_k] being classified as class c = target[n][d_1][d_2]…[d_k] is computed as:

loss[n][d_1][d_2]…[d_k] = -input[n][c][d_1][d_2]…[d_k].

When an optional “weight” is provided, the sample loss is calculated as:

loss[n][d_1][d_2]…[d_k] = -input[n][c][d_1][d_2]…[d_k] * weight[c].

loss is zero for the case when target-value equals ignore_index.

loss[n][d_1][d_2]…[d_k] = 0, when target[n][d_1][d_2]…[d_k] = ignore_index

If “reduction” attribute is set to “none”, the operator’s output will be the above loss with shape (N, d1, d2, …, dk). If “reduction” attribute is set to “mean” (the default attribute value), the output loss is (weight) averaged:

mean(loss), if “weight” is not provided,

or if weight is provided,

sum(loss) / sum(weight[target[n][d_1][d_2]…[d_k]]]), for all samples.

If “reduction” attribute is set to “sum”, the output is a scalar:

sum(loss).

See also https://pytorch.org/docs/stable/nn.html#torch.nn.NLLLoss. Example 1:

// negative log likelihood loss, “none” reduction N, C, d1 = 2, 3, 2 input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]],

[[0.0, 1.0], [2.0, 2.0], [1.0, 2]]]

target = [[2, 1], [0, 2]] loss = np.zeros((N, d1)) for n in range(N):

for d_1 in range(d1):

c = target[n][d_1] loss[n][d_1] = -input[n][c][d_1]

// print(loss) // [[-3. -2.] // [-0. -2.]]

Example 2:

// weighted negative log likelihood loss, sum reduction N, C, d1 = 2, 3, 2 input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]],

[[0.0, 1.0], [2.0, 2.0], [1.0, 2]]]

target = [[2, 1], [0, 2]] weight = [0.2, 0.3, 0.1] loss = np.zeros((N, d1)) for n in range(N):

for d_1 in range(d1):

c = target[n][d_1] loss[n][d_1] = -input[n][c][d_1] * weight[c]

loss = np.sum(loss) // print(loss) // -1.1

Example 3:

// weighted negative log likelihood loss, mean reduction N, C, d1 = 2, 3, 2 input = [[[1.0, 2.0], [2.0, 2.0], [3.0, 2.0]],

[[0.0, 1.0], [2.0, 2.0], [1.0, 2]]]

target = [[2, 1], [0, 2]] weight = [0.2, 0.3, 0.1] loss = np.zeros((N, d1)) weight_total = 0 for n in range(N):

for d_1 in range(d1):

c = target[n][d_1] loss[n][d_1] = -input[n][c][d_1] * weight[c] weight_total = weight_total + weight[c]

loss = np.sum(loss) / weight_total // print(loss) // -1.57

Attributes

  • ignore_index: Specifies a target value that is ignored and does not contribute to the input gradient. It’s an optional value.

  • reduction: Type of reduction to apply to loss: none, sum, mean (default). ‘none’: the output is the loss for each sample. ‘sum’: the output will be summed. ‘mean’: the sum of the output will be divided by the sum of applied weights. Default value is 'mean'.

Inputs

Between 2 and 3 inputs.

  • input (heterogeneous) - T: Input tensor of shape (N, C) or (N, C, d1, d2, …, dk).

  • target (heterogeneous) - Tind: Target tensor of shape (N) or (N, d1, d2, …, dk). Target element value shall be in range of [0, C). If ignore_index is specified, it may have a value outside [0, C) and the target values should either be in the range [0, C) or have the value ignore_index.

  • weight (optional, heterogeneous) - T: Optional rescaling weight tensor. If given, it has to be a tensor of size C. Otherwise, it is treated as if having all ones.

Outputs

  • loss (heterogeneous) - T: The negative log likelihood loss

Type Constraints

  • T in ( tensor(double), tensor(float), tensor(float16) ): Constrain input, weight, and output types to floating-point tensors.

  • Tind in ( tensor(int32), tensor(int64) ): Constrain target to integer types