Sasi Kiran | AI Engineer

N-Gram Models: The Basics That Kickstart Your NLP Journey Toward LLMs N-Gram Models: The Basics That Kickstart Your NLP Journey Toward LLMs N-gram models are fundamental building blocks in natural language processing that capture the sequential nature of human language. Unlike simpler approaches that ignore word order, N-grams preserve local context and help us model how language flows naturally from one word to the next. What Are N-gram Models? An N-gram model is a probabilistic language model based on the Markov chain assumption, which states that the probability of a word depends only on the previous N-1 words. The "N" in N-gram refers to the number of consecutive words grouped together as a single unit. Types of N-grams N-gram Type Description Example (from "data science is fascinating") Unigram Single word units "data", "science", "is", "fascinating...

Tokenization: How AI Understands Language Tokenization: How AI Understands Language The fundamental process that enables AI systems to comprehend and generate human language These days, we're all interacting with generative AI models like ChatGPT and Gemini. But have you ever wondered how these systems actually understand your questions and generate relevant responses? The answer lies in a sophisticated process that transforms human language into a format that machines can comprehend and manipulate. While massive training datasets containing billions of words are crucial for AI performance, the secret lies in how we prepare this data for AI systems. Raw text, as humans write it, is messy and inconsistent. It contains spaces, punctuation, capitalization variations, and countless linguistic nuances that machines struggle to process directly. This brings us to the core concept that bridges human lang...

Understanding Morphology in NLP: Morphemes, Stemming, Lemmatization & Lexicon Explained Morphology in NLP: Breaking Words to Make Machines Smarter A Deep Dive by Sasi | Explaining Morphemes, Stemming, Lemmatization, and Lexicons Hey everyone 👋! Let’s dive deeper into something we all take for granted in language: word structure . In NLP, we call this study morphology . This post is all about how computers can break down and understand words—just like humans do—with the help of four key tools: Morphemes Stemming Lemmatization Lexicon 1. Morphemes – The Building Blocks 🧱 A morpheme is the smallest unit in a word that carries meaning. These include roots like “run”, prefixes like “un-”, and suffixes like “-ing”. Let’s look at an example: "unbelievably" → "un" (negation) + "believe" (root) + "able" (capable of) + "ly" (manner) Understanding morphe...

Understanding Morphology in NLP: The Key to Word-Level Language Intelligence Understanding Morphology in NLP A Deep Dive into Telugu vs English Morphological Patterns TL;DR This post introduces Morphology in NLP using real language examples (Telugu vs English). Learn how word forms change and how machines understand them. Welcome to the fascinating world of morphology in Natural Language Processing! Today, we're diving deep into how different languages structure their words, and why this matters immensely for building intelligent language systems. What is Morphology? Morphology is the study of word structure - how words are formed and how they change their forms to express different meanings. In NLP, understanding morphology is crucial because it helps machines recognize relationships between different word forms and extract meaningful information from text. Think of morpheme...

Bag of Words Explained: Mastering the Fundamentals of NLP Welcome to Part 2 of the ' NLP Engineering ' series where I’ll guide you through essential NLP concepts—from theory to practice—with clear explanations and hands-on code examples. Perfect for beginners and seasoned engineers alike!" Natural Language Processing (NLP) requires converting text into numerical formats that machines can understand. One of the fundamental techniques for this transformation is the Bag of Words (BoW) model. In this blog post, I'll explain how BoW works, why it's useful, and demonstrate it with Python code. What is Bag of Words? Bag of Words is a method of representing text data as numerical features. At its core, BoW involves creating a meaningful vocabulary from a corpus of text, which can then be used for various NLP tasks like sentiment analysis, text classification, or document clustering. The name "Bag of...

  Have you ever played a game with your friends and noticed that your performance can vary depending on the level or challenge? Machine learning models can also have a similar issue with performance variability, known as bias and variance. Bias: Bias is the overall difference between what the model predicts and what the correct answer should be, averaged across all examples. A model with high bias consistently makes the same types of errors, regardless of the specific input features or examples. To illustrate this concept, let's consider a regression model that predicts the price of a house based on its square footage, number of bedrooms, and location. When evaluating this model on a test dataset, we calculate the mean squared error (MSE) between the predicted prices and the actual prices across all examples in the test dataset. If the MSE is high, this suggests that the model is biased and consistently predicts prices that are too high or too low compared to the true values. To ca...