Language Modeling & N-grams in NLP
Language Modeling is a fundamental concept in Natural Language Processing (NLP). It helps computers understand how likely a sentence is and predict the next word in a sequence.
What is a Language Model?
A Language Model (LM) calculates the probability of a sequence of words.
It helps machines:
- Understand sentences
- Predict next words
Examples of Language Model
- “I am going to school” ✔
- “I am going to moon” ❌
- “She is reading a book” ✔
- “She is reading a banana” ❌
- “They are playing cricket” ✔
Probability Concept
A sentence probability is calculated using conditional probability:
P(w1, w2, …, wn) = Product of probabilities
What are N-grams?
N-grams are sequences of words used to simplify language modeling.
Types of N-grams
- Unigram → Single word
- Bigram → Two-word sequence
- Trigram → Three-word sequence
Examples of N-grams (5 Examples)
Example 1:
“I love NLP”
- Bigram → I love, love NLP
Example 2:
“Machine learning is fun”
- Bigram → Machine learning, learning is
Example 3:
“Deep learning models are powerful”
- Trigram → Deep learning models
Example 4:
“I am going home”
- Trigram → I am going
Example 5:
“Natural language processing is amazing”
- Bigram → Natural language, language processing
Bigram Probability
P(word₂ | word₁) = Count(word₁, word₂) / Count(word₁)
Probability Examples
- P(love | I) = 1
- P(likes | He) = 1
- P(eats | She) = 1
- P(play | They) = 1
- P(watch | We) = 1
Applications of Language Models
- Auto-complete systems
- Chatbots
- Speech recognition
- Machine translation
- Search engines
Limitations
- Cannot understand long context
- Data sparsity problem
- High memory usage
- Rare words get zero probability
- Limited understanding
Conclusion
Language Modeling and N-grams are essential for predicting word sequences in NLP. They form the foundation for many modern AI applications like chatbots and translation systems.