Implementation of Sequence Padding and Sequence Packing

Handling Sequence Padding and Packing in PyTorch for RNNs

The code imports necessary modules from PyTorch: torch and torch.nn.utils.rnn defining a list of example sequences with variable lengths (sequences). Each sequence in the list is converted to a PyTorch tensor using a list comprehension (sequences_tensor).
The pad_sequence function from torch.nn.utils.rnn is used to pad the sequences to the maximum length with zeros, ensuring that all sequences have the same length. The batch_first=True argument specifies that the batch dimension should be the first dimension in the resulting tensor.
Then, the code calculates the actual lengths of sequences, and finally demonstrates how to pack sequences using pack_padded_sequence().

Python3

import torch
import torch.nn.utils.rnn as rnn_utils
# Define sequences
sequences = [  
    [1, 2, 3],
    [4, 5],
    [6, 7, 8, 9],
    [10]
]
sequences_tensor = [torch.tensor(seq) for seq in sequences] # Convert sequences to PyTorch tensors

# Padding
padded_sequences = rnn_utils.pad_sequence(sequences_tensor, batch_first=True)
print("Padded sequences:","\n",padded_sequences)

# Packing 
sequence_lengths = torch.tensor([len(seq) for seq in sequences]) # Calculating actual lengths of sequences
# Pack padded sequences
packed_sequences = rnn_utils.pack_padded_sequence(padded_sequences, sequence_lengths, batch_first=True, enforce_sorted=False)
print("\nPacked sequences:",packed_sequences)

Output:

Padded sequences: 
 tensor([[ 1,  2,  3,  0],
        [ 4,  5,  0,  0],
        [ 6,  7,  8,  9],
        [10,  0,  0,  0]])

Packed sequences: PackedSequence(data=tensor([ 6,  1,  4, 10,  7,  2,  5,  8,  3,  9]), batch_sizes=tensor([4, 3, 2, 1]), sorted_indices=tensor([2, 0, 1, 3]), unsorted_indices=tensor([1, 2, 0, 3]))

The output consists of a 2-dimensional PyTorch tensor, representing the padded sequences. Each row in the tensor corresponds to a sequence, and columns represent elements within each sequence. For example,

First Sequence (Row 1):
- Original sequence: [1, 2, 3]
- Padded sequence: [1, 2, 3, 0]
- The original sequence had three elements, so it was padded with a zero to match the length of the longest sequence in the batch (which is four).

Second Sequence (Row 2):
- Original sequence: [4, 5]
- Padded sequence: [4, 5, 0, 0]
- The original sequence had two elements, so it was padded with two zeros to match the length of the longest sequence in the batch.

Same is done for all the rows.

In the packed sequence:

data: contains the flattened non-padded elements from the padded sequences.
batch_sizes: indicates how many elements are present at each time step, reflecting the varying sequence lengths within the batch.

This packed sequence is feed into your recurrent neural network (RNN) model during training, allowing it to efficiently process variable-length sequences.

How to handle sequence padding and packing in PyTorch for RNNs?

There are many dataset that have sequences with variable lengths and recurrent neural networks (RNNs) require fixed-length inputs. To address this challenge, sequence padding and packing techniques are used, particularly in PyTorch, a popular deep learning framework. The article demonstrates how sequence padding ensures uniformity in sequence lengths by adding zeros to shorter sequences, while sequence packing compresses padded sequences for efficient processing in RNNs.

Table of Content

Sequence Padding and Packing for RNNs
Implementation of Sequence Padding and Sequence Packing
Handling Sequence Padding and Packing in PyTorch for RNNs

Implementation of Sequence Padding and Sequence Packing

How to handle sequence padding and packing in PyTorch for RNNs?

Similar Reads