How to vectorize text with TF-IDF in scikit-learn

Text columns need a numerical representation before most scikit-learn estimators can train or compare samples. TfidfVectorizer turns raw documents into a sparse feature matrix where each column is a token or n-gram and each value reflects how important that term is within a document and across the fitted corpus.

The vectorizer learns its vocabulary and inverse-document-frequency weights during fit_transform(). Later transform() calls reuse that learned mapping, so new documents keep the same column order and shape as the training matrix.

A short support-ticket corpus keeps the output small enough to inspect. The script uses word unigrams and bigrams, removes generic English stop words, and prints the learned terms plus the non-zero weights for one new message.

1. Save a Python script that fits TfidfVectorizer on training documents.

   vectorize_tfidf.py

   from sklearn.feature_extraction.text import TfidfVectorizer
    
    
   documents = [
       "reset password token expired",
       "reset account password link",
       "invoice payment receipt delayed",
       "payment reminder invoice sent",
   ]
    
   vectorizer = TfidfVectorizer(
       lowercase=True,
       stop_words="english",
       ngram_range=(1, 2),
   )
    
   matrix = vectorizer.fit_transform(documents)
   feature_names = vectorizer.get_feature_names_out()
    
   new_documents = ["password reset link sent"]
   new_matrix = vectorizer.transform(new_documents)
    
   print(f"training documents: {matrix.shape[0]}")
   print(f"tf-idf matrix shape: {matrix.shape}")
   print("first features:", ", ".join(feature_names[:8]))
   print("learned vocabulary contains:", "password" in vectorizer.vocabulary_)
   print(f"new document shape: {new_matrix.shape}")
   print("new document non-zero weights:")
    
   row = new_matrix[0]
   for column_index in row.nonzero()[1]:
       print(f"  {feature_names[column_index]}: {row[0, column_index]:.3f}")

   fit_transform() learns the vocabulary and IDF weights from documents. Use transform() for later documents so prediction or similarity inputs keep the same feature columns.

2. Run the script.

   $ python3 vectorize_tfidf.py
   training documents: 4
   tf-idf matrix shape: (4, 24)
   first features: account, account password, delayed, expired, invoice, invoice payment, invoice sent, link
   learned vocabulary contains: True
   new document shape: (1, 24)
   new document non-zero weights:
     link: 0.555
     password: 0.438
     reset: 0.438
     sent: 0.555

3. Check the training matrix dimensions.

   tf-idf matrix shape: (4, 24) means four training documents became four rows and the fitted vocabulary produced 24 token or bigram columns.

4. Confirm that the fitted vocabulary contains an expected token.

   learned vocabulary contains: True verifies that password survived preprocessing and was assigned a column in the fitted matrix.

5. Verify that new text uses the same feature columns.

   new document shape: (1, 24) confirms that transform() reused the 24-column training vocabulary. The non-zero weights show only the matched terms from the new message.