What are Sparse Tensors?

Sparse tensor is a term specifically defining those vectors which have a multitude of zero values. Unlike other tensors which holds majority non-zero values, sparse vectors have different approach. Sparse vector smartly optimizes the storage and computation by keeping track of non-zero values only. Hence, they are idea for the scenarios of sparsity.

For example, you have a gigantic chessboard which has most of its squares empty (zero values). A sparse vector is like a clever agent which focuses more on the pieces (non-zero values) and their precise positions. It doesn’t bother about the empty squares and thereby, reduces storage requirements and computational load associated with unnecessary zero values.

In TensorFlow, you can represent a sparse tensor by using tf.sparse.SparseTensor object. They are currently encoded using the coordinate list (COO) format. It neatly organizes the non-zero values, their corresponding indices, and the overall shape of the tensor. A special thing about sparse tensor is that it maintains the essence of the data while discarding the noise. Thus, we finally have an efficient way of storing and processing large datasets.

Imagine you are working with a massive dataset which is represented by multi-dimensional arrays called tensors. In simple terms, tensors are the building blocks of mathematical operations on the data. However, sometimes, tensors can have majority of values as zero. Such a tensor with a lot of zero values is called as sparse tensor.

Sparse tensors are mostly encountered in the fields of computer vision and natural language processing. These can be pretty overwhelming at times. Therefore, in this article we will be discussing various aspect related to sparse tensors. You will have the following concepts cleared when you read this article: