Feature Extraction for Text - Feature Extraction | Data Preparation

Machine learning models need numbers, so text must first be converted into numeric vectors.

Text preprocessing (before extraction)

Lowercase the text.
Remove punctuation, numbers or noise if not needed.
Remove stop words ("the", "is", "and").
Tokenize into words.
Stem or lemmatize (running → run).

1. Bag of Words (BoW)

Counts how many times each word appears in each document. Word order is ignored.

from sklearn.feature_extraction.text import CountVectorizer

corpus = [
    "Machine learning needs clean data",
    "Data preparation improves machine learning models",
    "Feature extraction reduces dimensionality",
]

cv = CountVectorizer(stop_words="english")
X_bow = cv.fit_transform(corpus)

print(cv.get_feature_names_out())
print(X_bow.toarray())

2. TF-IDF

Term Frequency-Inverse Document Frequency down-weights words that appear in many documents and up-weights words that are distinctive.

TF = how often a word appears in a document.
IDF = log(total documents / documents containing the word).
TF-IDF = TF x IDF.

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(stop_words="english")
X_tfidf = tfidf.fit_transform(corpus)

import pandas as pd
print(pd.DataFrame(X_tfidf.toarray(), columns=tfidf.get_feature_names_out()).round(2))

3. N-grams

Sequences of n consecutive words capture short phrases such as "not good", which a single-word model misses.

tfidf_ng = TfidfVectorizer(stop_words="english", ngram_range=(1, 2), min_df=1)
X_ng = tfidf_ng.fit_transform(corpus)
print(X_ng.shape)
print(tfidf_ng.get_feature_names_out()[:10])

4. Hashing trick

HashingVectorizer maps words to a fixed number of columns using a hash function. It needs no vocabulary and is memory-efficient for huge text streams.

from sklearn.feature_extraction.text import HashingVectorizer

hv = HashingVectorizer(n_features=2**10, alternate_sign=False)
X_hash = hv.transform(corpus)
print(X_hash.shape)

5. Word and sentence embeddings

Dense vectors that place similar meanings close together.

Embedding	Notes
Word2Vec / GloVe / FastText	One fixed vector per word
Sentence-BERT	Vector for a full sentence
Transformer embeddings (BERT-style)	Contextual vectors: the same word has different vectors in different sentences

Comparison

Method	Captures meaning?	Vector size	Notes
Bag of Words	No	Vocabulary size (sparse)	Simple baseline
TF-IDF	Weak	Vocabulary size (sparse)	Strong baseline for classification
N-grams	Local phrases	Very large (sparse)	Use `min_df` to limit size
Embeddings	Yes	100-1024 (dense)	Best quality, more compute

Tips

Fit vectorizers on the training text only.
Limit vocabulary with max_features, min_df and max_df.
For Bangla text, use a suitable tokenizer and, ideally, multilingual embeddings.