{"id":2937,"date":"2025-12-17T15:30:49","date_gmt":"2025-12-17T15:30:49","guid":{"rendered":"https:\/\/blog.samarthya.me\/wps\/?p=2937"},"modified":"2025-12-17T15:30:59","modified_gmt":"2025-12-17T15:30:59","slug":"mastering-tf-idf-a-gamified-journey","status":"publish","type":"post","link":"https:\/\/blog.samarthya.me\/wps\/2025\/12\/17\/mastering-tf-idf-a-gamified-journey\/","title":{"rendered":"Mastering TF-IDF: A Gamified Journey!"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

Understanding how computers “read” and understand text is a fascinating field. One of the most fundamental techniques for identifying important keywords in a document, relative to a collection of documents, is TF-IDF (Term Frequency-Inverse Document Frequency)<\/strong>.<\/p>\n\n\n\n

I recently embarked on a gamified learning challenge to demystify TF-IDF, breaking it down into its core components. This post summarizes my adventure!<\/p>\n\n\n\n

\n\n\n\n

The Core Idea: Why TF-IDF Matters<\/strong><\/h3>\n\n\n\n

Imagine you have a huge library of books. If you want to find out what a specific book is really<\/em> about, just looking at words that appear frequently isn’t enough. “The” or “is” might appear often, but they tell you nothing unique. TF-IDF helps us find words that are:<\/p>\n\n\n\n

    \n
  1. Frequent within a specific document (TF)<\/strong><\/li>\n\n\n\n
  2. Rare across the entire collection of documents (IDF)<\/strong><\/li>\n<\/ol>\n\n\n\n

    When a word meets both criteria, it’s likely a powerful keyword that defines that document’s content.<\/p>\n\n\n\n

    \n\n\n\n

    Level 1: Term Frequency (TF) \u2013 How Often Does a Word Appear?<\/h3>\n\n\n\n

    Term Frequency is the simplest part. It’s just a count of how often a word (term) appears in a document, normalized by the total number of words in that document.<\/p>\n\n\n\n

    Formula:<\/p>\n\n\n\n

    TF<\/mtext>(<\/mo>t<\/mi>,<\/mo>d<\/mi>)<\/mo>=<\/mo>Number of times term <\/mtext>t<\/mi> appears in document <\/mtext>d<\/mi><\/mrow>Total number of words in document <\/mtext>d<\/mi><\/mrow><\/mfrac><\/mrow>\\text{TF}(t, d) = \\frac{\\text{Number of times term } t \\text{ appears in document } d}{\\text{Total number of words in document } d}<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

    Example:<\/strong><\/h4>\n\n\n\n

    Let’s say we have Document A<\/strong>:<\/p>\n\n\n\n

    \n

    “The cat sat on the mat. The cat is black.”<\/p>\n<\/blockquote>\n\n\n\n

      \n
    • Total words in Document A:<\/strong> 10<\/li>\n\n\n\n
    • Term ‘cat’ appears:<\/strong> 2 times<\/li>\n<\/ul>\n\n\n\n

      Calculation:<\/p>\n\n\n\n

      TF<\/mtext>(<\/mo>cat<\/mtext>,<\/mo>A<\/mtext>)<\/mo>=<\/mo>2<\/mn>10<\/mn><\/mfrac>=<\/mo>\ud835\udfce.\ud835\udfd0<\/mn><\/mrow>\\text{TF}(\\text{cat}, \\text{A}) = \\frac{2}{10} = \\mathbf{0.2}<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

      A higher TF means the word is more important to that specific document<\/em>.<\/p>\n\n\n\n

      \n\n\n\n

      Level 2: Inverse Document Frequency (IDF) \u2013 How Unique is a Word Across Documents?<\/h3>\n\n\n\n

      Here’s where TF-IDF gets clever. IDF helps us ignore common words (like “the”) that appear in almost every document. It assigns a higher score to words that are rare across our entire collection of documents (our “corpus”).<\/p>\n\n\n\n

      Formula (General form):<\/p>\n\n\n\n

      IDF<\/mtext>(<\/mo>t<\/mi>,<\/mo>D<\/mi>)<\/mo>=<\/mo>log<\/mi>\u2061<\/mo><\/mrow>(<\/mo>Total number of documents <\/mtext>N<\/mi><\/mrow>Number of documents containing term <\/mtext>t<\/mi><\/mrow><\/mfrac>)<\/mo><\/mrow><\/mrow>\\text{IDF}(t, D) = \\log \\left( \\frac{\\text{Total number of documents } N}{\\text{Number of documents containing term } t} \\right)<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

      Our Corpus (4 Documents):<\/strong><\/p>\n\n\n\n

        \n
      • D1:<\/strong> “The cat<\/strong> sat on the mat.”<\/li>\n\n\n\n
      • D2:<\/strong> “The dog chased the frisbee<\/strong>.”<\/li>\n\n\n\n
      • D3:<\/strong> “I like my cat<\/strong>.”<\/li>\n\n\n\n
      • D4:<\/strong> “My neighbor’s cat<\/strong> is cute.”<\/li>\n<\/ul>\n\n\n\n

        Calculations (using natural logarithm):<\/strong><\/p>\n\n\n\n

          \n
        • For ‘cat’: Appears in D1, D3, D4 (3 documents out of 4) <\/li>\n<\/ul>\n\n\n\n
          IDF<\/mtext>(<\/mo>cat<\/mtext>,<\/mo>D<\/mi>)<\/mo>=<\/mo>ln<\/mi>\u2061<\/mo><\/mrow>(<\/mo>4<\/mn>\/<\/mi>3<\/mn>)<\/mo>\u2248<\/mo>\ud835\udfce.\ud835\udfd0\ud835\udfd6\ud835\udfd5<\/mn><\/mrow>\\text{IDF}(\\text{cat}, D) = \\ln(4\/3) \\approx \\mathbf{0.287}<\/annotation><\/semantics><\/math><\/div>\n\n\n\n
            \n
          • For ‘frisbee’: Appears in D2 (1 document out of 4)<\/li>\n<\/ul>\n\n\n\n
            IDF<\/mtext>(<\/mo>frisbee<\/mtext>,<\/mo>D<\/mi>)<\/mo>=<\/mo>ln<\/mi>\u2061<\/mo><\/mrow>(<\/mo>4<\/mn>\/<\/mi>1<\/mn>)<\/mo>\u2248<\/mo>\ud835\udfcf.\ud835\udfd1\ud835\udfd6\ud835\udfd4<\/mn><\/mrow>\\text{IDF}(\\text{frisbee}, D) = \\ln(4\/1) \\approx \\mathbf{1.386}<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

            Notice: ‘frisbee’ (rare) gets a much higher IDF than ‘cat’ (relatively common).<\/p>\n\n\n\n

            \n\n\n\n

            Level 3: The TF-IDF Score \u2013 Combining Frequency and Uniqueness<\/h3>\n\n\n\n

            The final TF-IDF score is simply the product of TF and IDF. This score highlights words that are both frequent in a document and<\/em> unique across the corpus.<\/p>\n\n\n\n

            Formula:<\/p>\n\n\n\n

            TF-IDF<\/mtext>(<\/mo>t<\/mi>,<\/mo>d<\/mi>,<\/mo>D<\/mi>)<\/mo>=<\/mo>TF<\/mtext>(<\/mo>t<\/mi>,<\/mo>d<\/mi>)<\/mo>\u00d7<\/mo>IDF<\/mtext>(<\/mo>t<\/mi>,<\/mo>D<\/mi>)<\/mo><\/mrow>\\text{TF-IDF}(t, d, D) = \\text{TF}(t, d) \\times \\text{IDF}(t, D)<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

            Putting it Together:<\/strong><\/h4>\n\n\n\n
              \n
            • TF for ‘cat’ in Document A:<\/strong> 0.2<\/li>\n\n\n\n
            • IDF for ‘cat’ in Corpus:<\/strong> 0.287 <\/li>\n<\/ul>\n\n\n\n
              TF-IDF<\/mtext>(<\/mo>cat<\/mtext>,<\/mo>A<\/mtext>,<\/mo>D<\/mi>)<\/mo>=<\/mo>0.2<\/mn>\u00d7<\/mo>0.287<\/mn>\u2248<\/mo>\ud835\udfce.\ud835\udfce\ud835\udfd3\ud835\udfd5\ud835\udfd2<\/mn><\/mrow>\\text{TF-IDF}(\\text{cat}, \\text{A}, D) = 0.2 \\times 0.287 \\approx \\mathbf{0.0574}<\/annotation><\/semantics><\/math><\/div>\n\n\n\n
                \n
              • Hypothetical TF for ‘frisbee’ in a document:<\/strong> 0.1 (e.g., if ‘frisbee’ appeared once in a 10-word document)<\/li>\n\n\n\n
              • IDF for ‘frisbee’ in Corpus:<\/strong> 1.386<\/li>\n<\/ul>\n\n\n\n
                TF-IDF<\/mtext>(<\/mo>frisbee<\/mtext>,<\/mo>hypothetical doc<\/mtext>,<\/mo>D<\/mi>)<\/mo>=<\/mo>0.1<\/mn>\u00d7<\/mo>1.386<\/mn>\u2248<\/mo>\ud835\udfce.\ud835\udfcf\ud835\udfd1\ud835\udfd6\ud835\udfd4<\/mn><\/mrow>\\text{TF-IDF}(\\text{frisbee}, \\text{hypothetical doc}, D) = 0.1 \\times 1.386 \\approx \\mathbf{0.1386}<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

                Result:<\/strong> Even with a lower term frequency, ‘frisbee’ has a higher overall TF-IDF score because its uniqueness (high IDF) boosts its importance. <\/p>\n\n\n\n

                This is the magic of TF-IDF!<\/p>\n","protected":false},"excerpt":{"rendered":"

                Understanding how computers “read” and understand text is a fascinating field. One of the most fundamental techniques for identifying important keywords in a document, relative to a collection of documents, is TF-IDF (Term Frequency-Inverse Document Frequency). I recently embarked on a gamified learning challenge to demystify TF-IDF, breaking it down into its core components. This […]<\/p>\n","protected":false},"author":2,"featured_media":2941,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[34],"tags":[],"class_list":["post-2937","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technical"],"_links":{"self":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts\/2937","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/comments?post=2937"}],"version-history":[{"count":2,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts\/2937\/revisions"}],"predecessor-version":[{"id":2942,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts\/2937\/revisions\/2942"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/media\/2941"}],"wp:attachment":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/media?parent=2937"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/categories?post=2937"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/tags?post=2937"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}