In the classification model, let us assume that there are two classes: class 0 and class 1, and two raters who will rate each data point that belongs to a particular class. One of the raters can be considered as a classification model and the other as a real-world observer. Â The Cohen Kappa score can be used to compare the agreement of these raters on a variety of different tasks.
What is Cohen’s Kappa Score or Kappa Coefficient?
Cohen Kappa Score is a statistical measurement of the performance of the classification machine learning model. It is based on the agreement between the two raters. The agreement between the two raters (real-world observer and the classification model) can be perfect or it can be by chance.
How to Calculate Cohen’s Kappa Score?
Cohen’s Kappa Score can be calculated using the Confusion Matrix. Since the cohen kappa is based on the concept of raters. Let us assume actual values are values by rater 1 and predicted values are values by rater 2.
1. True Positive
These are the correctly predicted positive values i.e. the value which is predicted as positive and is true.
2. True Negative
These are the correctly predicted negative values i.e. the value which is predicted as negative and is true.
3. False Positive
These are the incorrectly predicted positive values i.e. the value which is predicted as positive and is false.
4. False Negative
These are the incorrectly predicted negative values i.e. the value which is predicted as negative and is false.
Number of Agreements
The number of agreements (where both raters are in agreement with each other) is True Positive and True Negative.
Number of Disagreements
The number of disagreements (where both raters are in disagreement with each other) is False Positive and False Negative.
Cohen’s Kappa Formula
The below formula represents Cohen’s Kappa:
where
- Po= observed probability, the probability that rater 1 and rater 2 are in perfect agreement.
- Pe = expected probability, probability (expected) both the raters are in agreement by chance or randomly.
Interpretation of Kappa Score or Coefficient
The value of the Cohen Kappa Score or Kappa Coefficient can range between -1 and +1.
- Kappa’s score of -1 is considered to be the worst possible performance.
- Kappa’s score of +1 is considered to be the best possible performance.
- Kappa’s score of 0 means that the model is randomly guessing.
Advantage of Cohen Kappa
Cohen Kappa or Kappa Coefficient is a very useful performance evaluation metric when dealing with imbalanced data. Therefore in highly imbalanced data where the Accuracy metric might fail, the Cohen Kappa coefficient can be considered.