This article is detailing about an important concept in Machine Learning and enhances the modelling process, particularly in binary classification problems.
- What is the Weight of Evidence (WOE)?
- Use case of WOE
- How to Calculate WOE?
- How it is useful in Logistic Regression
- What is Information Value (IV)?
- How to Calculate IV?
- Interpretation of IV
- Needs of WOW & IV
- How to calculate WOE and IV in Python
What is the Weight of Evidence (WOE)?
Weight of Evidence (WOE) and Information Value (IV) are often used as preprocessing techniques in logistic regression, specifically in the context of binary classification problems where the dependent variable is categorical with two outcomes (0 or 1).
WOE is applied to transform predictor variables, especially categorical or continuous ones, into a format suitable for logistic regression.
Use case of WOE
Weight of Evidence is often used in credit scoring, fraud detection, Risk Modelling, and any binary classification problem where understanding the impact of predictor variables on the target variable is crucial.
How to Calculate WOE?
Step 1. Data Binning/Grouping
For continuous variables, divide the variable into bins or intervals. For categorical variables, each category is treated as a separate bin.
Step 2. Calculate WOE
For each bin, calculate the percentage of events and non-events.
Calculate the Weight of Evidence (WOE) using the formula.
WOE=ln(%of Events / %of Non-events)
How is it useful in Logistic Regression?
- WoE technique is commonly used for transforming variables in Logistic Regression. WOE helps in treating missing values logically for both types of variables — categorical and continuous. The original variable values in the dataset can be replaced with the Calculated WoE values, and the dataset with WOE-transformed variables can be used for logistic regression modeling.
- WOE reflects group identity, which means it captures the general trend of the distribution of good and bad.
What is Information Value (IV)?
Information Value (IV) is a metric used to measure the predictive power of a variable in a binary classification model, such as logistic regression. The IV is often calculated in conjunction with the Weight of Evidence (WOE) for each category or bin of a predictor variable.
How to Calculate IV?
IV=∑(%of Non-events−%of Events)×WOE
- Calculate the WOE for each bin or category of a variable.
- Calculate the percentage of events and non-events in each bin.
- Compute the difference between the percentage of non-events and events in each bin.
- Multiply the difference by the WOE for each bin.
- Sum up the results across all bins to get the Information Value
Interpretation of IV
Higher IV values indicate a stronger predictive power of the variable.
IV can be interpreted as follows.
| Information Value |
Predictive power |
| Less than 0.02 |
Not useful for prediction |
| 0.02 to 0.1 |
Weak predictive power |
| 0.1 to 0.3 |
Moderate predictive power |
| 0.3 to 0.5 |
Strong predictive power |
| Greater than 0.5 |
Suspicious predictive power |
Needs of WOW & IV
- Variable Transformation: WOE helps in transforming continuous variables into a form that is more suitable for modelling, especially for logistic regression.
- Handling non-linearity: Logistic regression assumes a linear relationship between the independent variables and the log odds of the dependent variable. WOE allows for capturing non-linear relationships and improves the model's ability to handle complex patterns.
- Addresses Outliers: WOE is less sensitive to outliers compared to other encoding methods. It places more emphasis on the distribution of events and non-events rather than specific data points.
- Feature Selection: IV provides a measure of the predictive power of each variable. High IV values indicate strong predictive power and variables with low or zero IV may be considered for removal from the model, reducing complexity and potential overfitting.
- Ranking Variables: IV allows the ranking of variables based on their importance in predicting the target variable.
- Handling Missing Values: WOE can handle missing values by creating a separate category for them during the binning process, ensuring that information from missing values is not lost.
- Model Performance Enhancement: WOE-driven feature selection improves model accuracy, stability, and interpretability.
How to calculate WOE and IV in Python?
You can use the below function in Python by passing a dataset, the feature of interest (feature_b), the target variable (target), and the target categories (target_category). The function will return a DataFrame (dset) containing WoE-related information and the calculated IV.
def calculate_woe_iv(dataset, feature_b, target,target_category):
dataset[feature_b].replace([np.inf, -np.inf], np.nan, inplace = True)
if dataset[feature_b].dtype in ['float64', 'int64'] and 'flag' not in feature_b.lower():
dataset['feature'] = pd.qcut(dataset[feature_b], q = [0, .05, .1, .2,.4, .6, .8,0.9,0.95, 1],
duplicates = 'drop', precision = 0).values.add_categories(['Missing', 'Others'])
else:
dataset['feature'] = dataset[feature_b]
dataset['feature'].fillna('Missing', inplace = True)
lst = []
num_rows = dataset.shape[0]
val_list = dataset['feature'].unique()
for val in val_list:
occ_val = dataset[dataset['feature'] == val].count()['feature']
if occ_val < 0.005 * num_rows :
dataset['feature'] = np.where(dataset['feature'] == val, 'Others', dataset['feature'])
val_list = dataset['feature'].unique()
for val in val_list:
lst.append({
'Feature': feature_b,
'Value': val,
'All': dataset[dataset['feature'] == val].count()['feature'],
'Good': dataset[(dataset['feature'] == val) & (dataset[target] == 0)].count()['feature'],
'Bad': dataset[(dataset['feature'] == val) & (dataset[target] == 1)].count()['feature']
})
dset = pd.DataFrame(lst)
dset['Event_rate'] = dset['Bad'] / dset['All']
dset['Prop'] = dset['All'] / dset['All'].sum()
dset['Prop_Good'] = dset['Good'] / dset['Good'].sum()
dset['Prop_Bad'] = dset['Bad'] / dset['Bad'].sum()
dset['WoE'] = np.log(dset['Prop_Good'] / dset['Prop_Bad'])
dset = dset.replace({'WoE': {np.inf: 0, -np.inf: 0}})
dset['IV_component'] = (dset['Prop_Good'] - dset['Prop_Bad']) * dset['WoE']
iv = dset['IV_component'].sum()
dset = dset.sort_values(by = 'Event_rate')
return dset, iv
I hope you find this information useful.
Happy Learning!