Feature Selection

What is Feature Selection?

When we build a Machine Learning model, we use a dataset that has many columns. In ML language, each column (except the target/output column) is called a feature.

But here is the problem — not every column is useful. Some columns help the model predict better. Some columns are just noise and confuse the model. Feature Selection means:

Picking only the important columns and removing the useless ones, before training the model.

Simple Example

Suppose we want to predict the Price of a House. We have this data:

Look at the column Owner_Favorite_Color. Does the owner's favorite color decide the price of the house? No! It has nothing to do with price. This column is useless for our model.

But House_Size and Bedrooms clearly affect the price — bigger house, more bedrooms, more price.

So a good Feature Selection process would say:

• Keep: House_Size, Bedrooms
• Remove: Owner_Favorite_Color

This makes the model simpler, faster, and more accurate, because it is not getting confused by useless information.

There are 3 major types of Feature Selection methods:

1. Filter Method
2. Wrapper Method
3. Embedded Method

In this post, we explore both the Filter Method and the Wrapper Method in detail. The Embedded Method is introduced below (with a full walkthrough coming in the next post).

1. Filter Method

Filter Method checks each feature separately, using statistics — it does not involve training any ML model. We just look at each column and ask:

"Does this column have a real relationship with the target column, or not?"

If a feature has a strong relationship with the target, we keep it. If it has almost no relationship, we drop it.

Think of it like a first-round filter — like when a company checks resumes before calling people for an interview. Filter Method quickly removes the useless features before we even build the model. That's why it's fast and simple.

The two most common statistical tests used in Filter Method are:

1. ANOVA (Analysis of Variance) — used when feature is a Number and target is a Category
2. Chi-Square Test — used when both feature and target are Categories

Let's understand both, one by one, with simple math and pandas code.

1.1 ANOVA (Analysis of Variance)

When to use it?

Use ANOVA when:

• Your feature (input column) has numbers — like Age, Salary, Marks
• Your target (output column) has categories — like Pass/Fail, Yes/No

The Idea (in simple words)

ANOVA checks: "If I split the numeric feature into groups based on the target category, are the group averages very different from each other, or almost the same?"

• If the group averages are very different → the feature is useful, keep it.
• If the group averages are almost the same → the feature is useless, drop it.

Small Example

We want to predict if a student Passes or Fails an exam, using Study_Hours as a feature.

Split Study_Hours by Result:

• Fail group: 1, 2 → Average = 1.5
• Pass group: 8, 9 → Average = 8.5

The averages (1.5 vs 8.5) are very far apart. This tells us Study_Hours clearly separates Pass and Fail students. So ANOVA will give this feature a high score — meaning it is important, keep it.

The Formula (simplified)

ANOVA calculates something called the F-value:

F = (Variance BETWEEN groups) / (Variance WITHIN groups)

• Variance between groups = how different the group averages are from the overall average.
• Variance within groups = how spread out the values are inside each group.

Simple rule: Bigger F-value = feature is more useful. Smaller F-value (close to 0 or 1) = feature is not useful.

You don't need to calculate this by hand every time — Python does it for us. Let's see how.

Doing ANOVA with Pandas + Sklearn

import pandas as pd
from sklearn.feature_selection import f_classif

# Step 1: Create the dataframe
data = {
    'Study_Hours': [1, 2, 3, 8, 9, 7, 2, 8],
    'Result':      ['Fail', 'Fail', 'Fail', 'Pass', 'Pass', 'Pass', 'Fail', 'Pass']
}
df = pd.DataFrame(data)

# Step 2: Separate feature (X) and target (y)
X = df[['Study_Hours']]   # feature must be numeric
y = df['Result']          # target is category

# Step 3: Apply ANOVA (f_classif)
f_score, p_value = f_classif(X, y)

print("F-score:", f_score)
print("p-value:", p_value)

How to read the result:

• F-score — higher value means the feature is more useful (as we discussed above).
• p-value — this tells us if the result is trustworthy.
  • If p-value is less than 0.05 → the feature is important, keep it.
  • If p-value is more than 0.05 → the feature is not important, you can drop it.

In our example, Study_Hours will get a high F-score and a very small p-value, confirming it is a strong feature for predicting Pass/Fail.

1.2 Chi-Square Test (χ²)

When to use it?

Use Chi-Square when:

• Your feature (input column) is a category — like Gender, City, Color
• Your target (output column) is also a category — like Yes/No, Pass/Fail

The Idea (in simple words)

Chi-Square checks: "Is there a real connection between two category columns, or are they just showing random, unrelated numbers by chance?"

It compares:

• Observed values — what we actually see in our data
• Expected values — what we would see if there was no relationship at all between the feature and target

If Observed and Expected are very different, it means there is a real relationship — keep the feature. If they are almost the same, there is no relationship — drop the feature.

Small Example

We want to check if Gender affects whether a person Buys a Product.

First, make a contingency table (a simple count table):

Expected value formula (if Gender had NO effect on buying):

Expected = (Row Total × Column Total) / Grand Total

For Male & Yes: Expected = (3 × 2) / 6 = 1
But Observed Male & Yes = 2

Since Observed (2) is different from Expected (1), this hints Gender might actually be related to buying behaviour.

The Formula (simplified)

Chi-Square (χ²) = Sum of [ (Observed − Expected)² / Expected ]

We do this calculation for every cell in the table, then add them all up. Bigger Chi-Square value = stronger relationship = keep the feature.

Doing Chi-Square with Pandas + Sklearn

import pandas as pd
from sklearn.feature_selection import chi2
from sklearn.preprocessing import LabelEncoder

# Step 1: Create the dataframe
data = {
    'Gender': ['Male', 'Male', 'Male', 'Female', 'Female', 'Female'],
    'Bought_Product': ['Yes', 'Yes', 'No', 'No', 'No', 'No']
}
df = pd.DataFrame(data)

# Step 2: Convert categories into numbers (Chi-square needs numeric input)
le = LabelEncoder()
df['Gender_encoded'] = le.fit_transform(df['Gender'])   # Male=1, Female=0 (example)
df['Target_encoded'] = le.fit_transform(df['Bought_Product'])

X = df[['Gender_encoded']]
y = df['Target_encoded']

# Step 3: Apply Chi-Square
chi_score, p_value = chi2(X, y)

print("Chi-Square score:", chi_score)
print("p-value:", p_value)

How to read the result:

• Same rule as before — higher Chi-Square score and p-value < 0.05 means the feature has a real relationship with the target, so keep it.

Quick Recap

Filter Method is usually the first step in Feature Selection — quick, cheap, and done before any model training.

2. Wrapper Method

In the Filter Method, we judged each column using pure statistics without touching a Machine Learning model.

Wrapper Method works differently: It actually trains an actual Machine Learning model on different combinations of features, checks the accuracy score, and picks the combination that gives the best results.

Think of it like a sports team tryout:
• Filter Method was checking player height and weight on paper before inviting them to practice.
• Wrapper Method is putting players together on the field in practice matches to see which combination actually wins games.

Because it uses a real model to evaluate features, it discovers something Filter Method cannot: feature interaction. Two columns might look weak on their own, but when combined together, they might give the model huge predictive power.

The 3 Common Wrapper Techniques

1. Forward Feature Selection (Start empty, add one by one)
   Start with 0 features. Try each feature individually, pick the best one. Then test remaining features paired with your first pick. Repeat until adding more features stops improving accuracy.
2. Backward Elimination (Start with all, remove one by one)
   Start with all features and train the model. Identify the weakest feature that hurts or doesn't help the score, and drop it. Repeat until removing any more features hurts performance.
3. Recursive Feature Elimination (RFE)
   The most popular industry-standard version of Backward Elimination in scikit-learn. It trains the model, checks feature weights or coefficients, prunes the weakest features, and repeats until it reaches the exact number of top features you requested.

Step-by-Step Example with Python (RFE)

Suppose we are predicting whether a student passes an exam. We have useful columns (Study_Hours, Past_Score, Attendance_Pct) and random noise columns (Lucky_Shirt_Color, Birth_Day_Number). Let's see how RFE finds the top 3 features:

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.feature_selection import RFE

# Step 1: Create a simple dataset
data = {
    'Study_Hours':       [2, 5, 8, 1, 6, 9, 3, 7],
    'Past_Score':        [45, 70, 88, 40, 75, 92, 50, 80],
    'Attendance_Pct':    [60, 85, 95, 50, 90, 98, 65, 88],
    'Lucky_Shirt_Color': [1, 3, 2, 1, 3, 2, 1, 2],       # noise
    'Birth_Day_Number':  [14, 2, 29, 18, 5, 23, 11, 7],   # noise
    'Passed':            [0, 1, 1, 0, 1, 1, 0, 1]
}
df = pd.DataFrame(data)

X = df.drop(columns=['Passed'])
y = df['Passed']

# Step 2: Choose the base model
model = LogisticRegression()

# Step 3: Set up RFE to pick the top 3 features
rfe = RFE(estimator=model, n_features_to_select=3)
rfe.fit(X, y)

# Step 4: View selected features and their rankings
results = pd.DataFrame({
    'Feature': X.columns,
    'Selected': rfe.support_,     # True if kept, False if dropped
    'Rank': rfe.ranking_          # 1 means top selected feature
})

print(results.sort_values(by='Rank'))

How to read the result:

• rfe.support_ returns True for the columns that made the cut.
• rfe.ranking_ gives Rank 1 to all chosen features. Dropped features get Rank 2, 3, etc. based on how early they were eliminated.

Filter vs Wrapper Method: Quick Comparison

3. Embedded Method

(Coming in the next post — full explanation with examples.)