Adversarial debiasing workflow and blog #359
Conversation
| longExcerpt = "This blog post focuses on the Adversarial Debiasing model in Orange, a tool for enhancing fairness in your machine learning algorithms. We will walk through how to use it and explain the trade-offs that come with using fairness algorithms." | ||
| +++ | ||
|
|
||
| In the [previous blog post]({{< ref "2023-08-25-fairness-reweighing-preprocessor.md" >}}), we talked about how to use the Reweighing widget as a preprocessor for a model. This blog post will discuss the Adversarial Debiasing model, a bias-aware model. We will also show how to use it in Orange. |
There was a problem hiding this comment.
As before, never link to .mds.
|
|
||
| [Adversarial Debiasing](https://arxiv.org/abs/1801.07593) is an in-processing type of fairness mitigation algorithm. It is a technique that uses adversarial training to mitigate bias. It involves simultaneous training of a predictor and a discriminator. The goal of the predictor is to predict the target variable accurately. At the same time, the discriminator aims to predict the protected variable (such as gender or race) based on the predictor's predictions. The main goal is to maximize the predictor's ability to predict the target variable while reducing the discriminator's ability to predict the protected variable based on those predictions. Because of the Adversarial Debiasing implementation in the [AIF360](https://aif360.res.ibm.com/) package we are using, the algorithm focuses on Disparate Impact and Statistical Parity Difference fairness metrics. | ||
|
|
||
| Like most model widgets in Orange, the Adversarial Debiasing widget allows users to tune the model's hyperparameters to achieve the best possible results. The widget's interface is similar to other model widgets, allowing the user to set the model's hyperparameters. The widget's interface is shown in the image below. |
There was a problem hiding this comment.
"Like most model widgets in Orange, the Adversarial Debiasing widget allows users to tune the model's hyperparameters to achieve the best possible results" -> delete.
There was a problem hiding this comment.
Actually, delete the whole paragraphs but the last sentence. What would you gain by so much generic speak? If you'd want to describe some parameters, fine, but just writing that some exist is useless.
| - Number of epochs: The number of epochs to train the neural network. This is the number of times the model will see the entire dataset. | ||
| - Batch size: The number of instances to be used in each training batch. The model will be trained on batches of instances instead of the entire dataset simultaneously. | ||
| - Use Debiasing: Whether to use the debiasing or not. The widget will function as a regular neural network model if this option is not selected. | ||
| - Adversary loss weight: The weight of the adversary loss in the total loss function. The adversary loss is the loss function of the discriminator. The higher the weight, the more the model will try to reduce the discriminator's ability to predict the protected variable at the expense of the predictor's ability to predict the target variable. |
There was a problem hiding this comment.
Each description starts with a : . . Avoid useless rephrasing. The following style is, for me, better:
- Number of epochs: the number of times the model will see the entire dataset during training.
Also, if you do not talk of NN weights don't use "hyperparameters". From user perspective (and we write blogs for them), these are just parameters or settings. Also, there is no need to enumerate all of them here, I think you should only introduce the ones you need within your analyses. Or, if you really want to emphasize some, move this to a less prominent place.
|
|
||
| - Adversarial Debiasing implementation: The current algorithm implementation aims to make predictions independent of protected attributes, focusing on balancing outcomes. This approach inherently focuses on balancing outcomes (reflected in the Disparate Impact and Statistical Parity Difference metrics). However, the nuances of other fairness measures might not be as effectively addressed. | ||
|
|
||
| When we use debiasing, there can also be a slight decrease in accuracy. This happens because the model is now not only trying to be accurate but also fair. This balance is a necessary trade-off we accept when we want to remove bias. However, it is worth noting that in this example, the accuracy remains comparable to the scenario without debiasing. |
There was a problem hiding this comment.
I like the trade-off description.
|
|
||
| {{< window-screenshot src="/blog_img/2023/2023-08-28-fairness-adversarial-debiasing-box-plot-debias.png" >}} | ||
|
|
||
| From the box plot, we can see that the model with debiasing enabled gives more balanced results between the unprivileged and privileged groups. This is expected because it is precisely what the Disparate Impact and Statistical Parity Difference metrics measure. |
There was a problem hiding this comment.
"we can see that the model" - how do we see that? What do we see and what would we see instead if we wouldn't use debiasing. Reference actual values.
|
|
||
| ## Orange use case | ||
|
|
||
| Now that we know how the Adversarial Debiasing widget works and how to use it, let us look at a real-world example for a classification task. For this example, we will use the [Adult dataset](https://archive.ics.uci.edu/ml/datasets/adult), which we have used before. Unlike before, we will not use the As Fairness widget to select fairness attributes; instead, we will keep the default ones, "sex" for the protected attribute and "male" for the privileged protected attribute value. In this example, we will train two Adversarial Debiasing models, one with and one without debiasing. We will also train a Random Forest model for comparison. We will then compare the results of the three models. |
There was a problem hiding this comment.
Describe the data set in one or two sentences.
"We will then compare the results of the three models." - delete, you already wrote this in previous sentence. Or else, you can shorten:
"In this example, we will train two Adversarial Debiasing models, one with and one without debiasing. We will also train a Random Forest model for comparison. We will then compare the results of the three models." -> "We will train two Adversarial Debiasing models, one with and one without debiasing, and will compare them to Random Forests."
|
|
||
| {{< window-screenshot src="/blog_img/2023/2023-08-28-fairness-adversarial-debiasing-scores.png" >}} | ||
|
|
||
| The Adversarial Debiasing model results show that fairness metrics, particularly Disparate Impact and Statistical Parity Difference, have improved when debiasing. When not using debiasing, the results are very similar to that of the Random Forest model. When using debiasing, their values moved closer to a fair baseline, signifying reduced biases for these metrics. The Disparate Impact moved from 0.294 to 1.051, suggesting that the model's predictions became more balanced regarding favorable outcomes for privileged and unprivileged groups. The Statistical Parity Difference went from -0.180 to 0.006, indicating a near-zero bias between these groups regarding favorable outcomes. |
There was a problem hiding this comment.
The Adversarial Debiasing model results show that fairness metrics, particularly Disparate Impact and Statistical Parity Difference, have improved when debiasing. -> Test and Score shows that debiasing improved fairness metrics, particularly Disparate Impact and Statistical Parity Difference.
When not using debiasing -> Without debiasing
very similar -> similar
When using debiasing, their values moved closer to a fair baseline, signifying reduced biases for these metrics. -> delete, you are just repeating yourself
Continue with "Debiasing improved Disparate Impact from 0.294 to 1.051." And delete "suggesting that...." because we see the improvement and do not need yet another tedious definition.
|
|
||
| The Adversarial Debiasing model results show that fairness metrics, particularly Disparate Impact and Statistical Parity Difference, have improved when debiasing. When not using debiasing, the results are very similar to that of the Random Forest model. When using debiasing, their values moved closer to a fair baseline, signifying reduced biases for these metrics. The Disparate Impact moved from 0.294 to 1.051, suggesting that the model's predictions became more balanced regarding favorable outcomes for privileged and unprivileged groups. The Statistical Parity Difference went from -0.180 to 0.006, indicating a near-zero bias between these groups regarding favorable outcomes. | ||
|
|
||
| However, this does not mean that all forms of bias were addressed. Equal Opportunity Difference and Average Odds Difference metrics worsened from -0.097 to 0.358 and -0.087 to 0.197, respectively. This suggests that even though the algorithm has been optimized for specific fairness criteria, other biases have emerged or become more pronounced. |
There was a problem hiding this comment.
'this does not mean that" -> "not"
| weight = 1003 | ||
| +++ | ||
|
|
||
| The easiest method to address bias in machine learning is to use a bias-aware model. This approach eliminates the need for fairness preprocessing or postprocessing, simplifying the workflow. In this workflow, we will delve into exactly how to achieve this. We will employ a bias-aware model named Adversarial Debasing for classification. We will train two versions of this model: one with debiasing activated and another with debiasing deactivated. Finally, we will compare and display the fairness metrics using a box plot widget. |
There was a problem hiding this comment.
simplifying the workflow -> delete
"In this workflow, we will delve into exactly how to achieve this. We will employ a bias-aware model named Adversarial Debasing for classification." -> 'This workflow employ a bias-aware model named Adversarial Debasing."
"one with debiasing activated and another with debiasing deactivated." -> "with or without debiasing"
ZanMervic
force-pushed
the
fairness-adversarial-debiasing
branch
from
490e8dd
to
07395ad
Compare
Fourth of fairness workflows and blogs.