How Bigrams are generated?

Let’s take an example sentence “You are learning from Geeks for Geeks”, generating bigrams involves taking two adjacent words at a time to form pairs. Let’s break down the process and the purpose of each bigram:

Step 1: Tokenization

The first step is to split the sentence into individual words (tokens). For the sentence “You are learning from Geeks for Geeks”, the tokens would be:

 ['You', 'are', 'learning', 'from', 'Geeks', 'for', 'Geeks']

Step 2: Creating Bigrams

After tokenization, bigrams are formed by pairing each word with the next word in the sequence. Here’s how each bigram is constructed from the tokens:

1. (‘You’, ‘are’): This bigram pairs the first word “You” with the second word “are”. It helps in understanding the use of the pronoun “You” in a command or statement form, indicating the subject of the sentence.
2. (‘are’, ‘learning’): This bigram links “are” with “learning”, forming a verb phrase that indicates an ongoing action. It’s crucial for capturing the progressive tense in the sentence.
3. (‘learning’, ‘from’): Connecting “learning” with “from” helps in identifying the prepositional phrase that specifies the source or method of learning.
4. (‘from’, ‘Geeks’): This bigram pairs “from” with “Geeks”, which indicates the starting point or the source of learning, in this case, the entity “Geeks”.
5. (‘Geeks’, ‘for’): By pairing “Geeks” with “for”, it sets up another phrase, hinting at a purpose or reason which is about to be explained further.
6. (‘for’, ‘Geeks’): This final bigram closes the loop by linking “for” back to “Geeks”, suggesting a repetitive or cyclical learning process from the same source, or could imply that the learning is intended for “Geeks”, depending on additional context that might be present in a longer text.

Each of these bigrams captures a small piece of the syntactic and semantic structure of the sentence. Analyzing these pairs helps in understanding how words combine to form meaningful phrases that contribute to the overall meaning of the sentence.

Bigrams, or pairs of consecutive words, are an essential concept in natural language processing (NLP) and computational linguistics. Their utility spans various applications, from enhancing machine learning models to improving language understanding in AI systems. In this article, we are going to learn how bigrams are generated using NLTK library.