How to create Sparse Tensors in TensorFlow?

There are two ways to create a sparse tensor. Both of the ways are discussed in detail with an example below:

By Directly Specifying Indices and Values

As we discussed earlier, you can represent sparse tensor by the tf.sparse.SparseTensor object. Now, we will see how we can create sparse tensors in TensorFlow. When you are creating a sparse tensor, you need to specify the following three components:

Values: These are the non-zero values, represented in 1D tensor.
Indices: These are the indices of the non-zero values in the tensor, represented in 2D tensor.
Dense Shape: It specifies the overall shape of the tensor in 1D tensor.

Python3

import tensorflow as tf
 
# Define non-zero values and their indices
values = tf.constant([41, 30], dtype=tf.int32)
indices = tf.constant([[1, 2], [3, 3]], dtype=tf.int64)
dense_shape = tf.constant([4, 8], dtype=tf.int64)
 
# Create a sparse tensor
sparse_tensor = tf.sparse.SparseTensor(indices=indices, values=values, dense_shape=dense_shape)
 
# Print the sparse tensor
print(sparse_tensor)

Output:

SparseTensor(indices=tf.Tensor(
[[1 2]
 [3 3]], shape=(2, 2), dtype=int64), values=tf.Tensor([41 30], shape=(2,), dtype=int32), dense_shape=tf.Tensor([4 8], shape=(2,), dtype=int64))

The above code shows that we have created a sparse tensor using with two non-zero values 41 and 30 which are at indices (1, 2) and (3, 3), respectively. The tf.sparse.SparseTensor constructor is then used to create the sparse tensor by providing the values, indices, and dense_shape. When we print the sparse vector we get the structure of indices, values, and dense_shape.

Creating from Dense Tensor

In case you are working on a large dataset, basically comprising all tensors. However, you notice that most of the values in the tensor are zero. Then, you can use tf.sparse module to work with sparse tensors. Here, you will have to choose tf.sparse.from_dense method.

Python

import tensorflow as tf
 
# Create a dense tensor
dense_tensor = tf.constant([[1, 0, 0], [0, 0, 2], [0, 3, 0]], dtype=tf.float32)
 
# Convert the dense tensor to a sparse tensor
sparse_tensor = tf.sparse.from_dense(dense_tensor)
 
# Print the dense and sparse tensors
print("Dense Tensor:")
print(dense_tensor)
 
print("\nSparse Tensor:")
print(sparse_tensor)

Output:

Dense Tensor:
tf.Tensor(
[[1. 0. 0.]
 [0. 0. 2.]
 [0. 3. 0.]], shape=(3, 3), dtype=float32)
Sparse Tensor:
SparseTensor(indices=tf.Tensor(
[[0 0]
 [1 2]
 [2 1]], shape=(3, 2), dtype=int64), values=tf.Tensor([1. 2. 3.], shape=(3,), dtype=float32), dense_shape=tf.Tensor([3 3], shape=(2,), dtype=int64))

In the above code, we have our dense tensor stored in the dense_tensor variable. Then we simply use tf.sparse.from_dense method with dense_tensor and get our sparse tensor. You can see both the dense and the sparse vector in the output.

Sparse tensors in Tensorflow

Imagine you are working with a massive dataset which is represented by multi-dimensional arrays called tensors. In simple terms, tensors are the building blocks of mathematical operations on the data. However, sometimes, tensors can have majority of values as zero. Such a tensor with a lot of zero values is called as sparse tensor.

Sparse tensors are mostly encountered in the fields of computer vision and natural language processing. These can be pretty overwhelming at times. Therefore, in this article we will be discussing various aspect related to sparse tensors. You will have the following concepts cleared when you read this article: