Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
jacquelinekclee authored Sep 9, 2022
1 parent 4373d7f commit edc4c9f
Showing 1 changed file with 1 addition and 2 deletions.
3 changes: 1 addition & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,9 +77,8 @@ With the original model, a singular threshold was used. Under this model, both d

To try and improve these metrics for fairness, 2 different threshold tests were conducted to find better thresholds based on groups (complainant ethnicity in this case). Firstly, the thresholds that maximize utility for the individual group were used. These thresholds satisfied demographic parity and equalized odds, but not predictive value parity. Secondly, the thresholds that satisfy the 3 parity measures (demographic parity, equalized odds, and predictive value parity) were found using the training data and then tested with a separate dataset. See the paper for a more in depth breakdown on how these tests were conducted.

Depending on what is most important to the decision-making body (CCRB), different threshold(s) would be chosen. Maximizing utility overall enforces equality for all complainants, while maximizing utility for each group is more equitable. Satisfying demographic parity ensures that substantiation doesn’t depend on complainant ethnicity. Equality of odds would focus on ensuring that the chance of obtaining the benefit of substantiation is equal across groups, while predictive value parity focuses on ensuring deserving complainants receive substantiation and underserving complainants don’t.
Depending on what is most important to the decision-making body (CCRB), different threshold(s) would be chosen. Maximizing utility overall enforces equality for all complainants, while maximizing utility for each group is more equitable. Satisfying demographic parity ensures that substantiation doesn’t depend on complainant ethnicity. Equality of odds would focus on ensuring that the chance of obtaining the benefit of substantiation is equal across groups, while predictive value parity focuses on ensuring deserving complainants receive substantiation and underserving complainants don’t. The table below demonstrates different group thresholds that could be used, depending on the goal (maximizing utility or enforcing a parity measure).

Threshold for Black Complainants Threshold for non-Black Complainants Test Demographic Parity Equality of Odds Predictive Value Parity
| | Threshold for Black Complainants | Threshold for non-Black Complainants | Test | Demographic Parity | Equality of Odds | Predictive Value Parity |
|---:|-----------------------------------:|---------------------------------------:|:--------------------------------|:---------------------|:---------------------|:--------------------------|
| 0 | 0.527 | 0.527 | Max utility overall | Not satisfied | Not satisfied | Satisfied |
Expand Down

0 comments on commit edc4c9f

Please sign in to comment.