Political Stats rant
Nov. 25th, 2020 07:55 am![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
eve11Signature matching (software algorithms or people) is not good nor accurate enough to use to disqualify ballots. The people who want to use it need a good lesson in the Base Rate Fallacy, aka Bayes Rule.
The problem is that enacting a giant sweeping measure for all persons, in order to try and catch a miniscule amount of actual cheating, will do nothing more than disenfranchise large amounts of perfectly legitimate voters. Even if the software is very accurate. Which it is not, I will say. My signature changes from time to time, and from circumstance to circumstance. I just got signature-rejected earlier this year from my Fidelity account when I tried to write a large check to my contractor. They disallowed it because "the signatures didn't match." But they did. Both were my signature. I ended up having to perform direct wire transfers (at $15 a pop) because they wouldn't change their verification methods.
Here is an example of the math.
Suppose that you have a signature matching machine that is 99.9 percent accurate for both matches and non-matches. Actually that's unlikely. It's probably way better at detecting true mismatches.
Let's instead assume that it is 99.99 percent accurate in detecting when two signatures truly don't match. And let's assume it is 99.9 percent accurate in detecting when two signature truly do match. Eg, it's very good and very accurate. It could be an algorithm, a person, etc.
Suppose the population that we send to this machine is a collection of voting ballots. In this collection, let's say a whopping 0.5% of the ballots were fraudulently cast by someone forging a signature. This is likely WAAAAY larger than what you would actually see in person. Eg that would mean that 1 out of every 200 ballots was fraud. But the effect gets only more lopsided the lower the fraud rate is, so let's start with that.
Suppose you get a ballot back and it has been flagged by your machine as a non-match. What is the probability that this ballot truly is fraudulent?
The problem here is that the initial accuracy of the machine is taken with respect to two different populations: we know it is 99.99% accurate on anything that is truly fraud, and 99.9% accurate on anything that is truly honest. But when we get a ballot back with a mark, we don't know whether it is fraudulent or honest. The marks are a mix of true positives and false positives. And the chances of it being fraudulent or honest given that it was marked as such depend on the overall rate of fraud in the population.
The math looks like this (where "|" means "conditioned on", eg Pr(A | B) means Probability of A given we know B):
Pr( Fraudulent | Marked by our machine ) =
Pr(Fraudulent) * Pr(Marked | Fraudulent)
------------------------------------------------------------------------------------------
Pr(Fraudulent) * Pr(Marked | Fraudulent) + Pr(Not Fraudulent) * Pr(Marked | Not Fraudulent)
The numerator is an expression that gives us the percent of ballots that are both Fraudulent and marked as such. The denominator is the expression for the the percent of ballots that are marked as fraudulent, including both the true positives from the numerator (the first term in the sum), and the non-fraudulent ones that are accidentally marked (the second term in the sum).
The numbers for our very accurate machine are:
= 0.005*0.9999 / (0.005*0.9999 + 0.995*0.001)
= 0.834 or 83.4% chance of being fraudulent given it was marked.
Is 83.4 percent a good number? I mean it's pretty high. But let's examine it further. What it means is that 17% of these marked ballots that we are throwing away are wrong. How many marked ballots are there? That's the denominator. It is about 0.06% of all ballots. So now I've thrown away 100 good votes to try and catch 500 bad ones. Is that a good tradeoff?
What happens if the machines aren't nearly as accurate as 0.999 on true matches? Suppose they will as before, almost always flag a true mismatch, but they are only 99% accurate at discovering true matches. Maybe 1% of the population just doesn't have that consistent of a signature. (Apparently I am in that 1% if my bank is concerned. Wrong 1% to be in wrt a bank account, *sigh*)
Then the numbers are:
= 0.005*0.9999 / (0.005*0.9999 + 0.995*0.01)
= 33% chance of truly being fraudulent if it is marked.
And now, to try and catch the 1 in 200 people who are cheating, we've gone and disenfranchised 1 in 100 who aren't. We've flagged 1.5% of ballots, and two thirds of them were perfectly honest.
What if the population of cheaters is way less than 1 in every 200 ballots? What if it's 1 in every 2000? For our more accurate machine that is 99.9% accurate on non-fraudulent ballots we have
= 0.0005*0.9999 / (0.0005*0.9999 + 0.9995*0.001)
= 8% chance of truly being fraudulent when it is marked. And we have disenfranchised 0.5% of voters trying to catch ten times fewer than that.
Dare I ask what the numbers are for our less accurate algorithm? The one that is only 99% good at marking honest ballots as honest? Those numbers are:
= 0.0005*0.9999 / (0.0005*0.9999 + 0.9995*0.01)
= 4.7% chance of being fraudulent if it is marked. Almost 96% of flagged ballots will be honest in this case, and we are marking 1% of ballots. We are disenfranchising 1 out of 100 honest voters to try and catch 1 in 2000 who are cheating.
Those numbers just don't add up.
The problem is that enacting a giant sweeping measure for all persons, in order to try and catch a miniscule amount of actual cheating, will do nothing more than disenfranchise large amounts of perfectly legitimate voters. Even if the software is very accurate. Which it is not, I will say. My signature changes from time to time, and from circumstance to circumstance. I just got signature-rejected earlier this year from my Fidelity account when I tried to write a large check to my contractor. They disallowed it because "the signatures didn't match." But they did. Both were my signature. I ended up having to perform direct wire transfers (at $15 a pop) because they wouldn't change their verification methods.
Here is an example of the math.
Suppose that you have a signature matching machine that is 99.9 percent accurate for both matches and non-matches. Actually that's unlikely. It's probably way better at detecting true mismatches.
Let's instead assume that it is 99.99 percent accurate in detecting when two signatures truly don't match. And let's assume it is 99.9 percent accurate in detecting when two signature truly do match. Eg, it's very good and very accurate. It could be an algorithm, a person, etc.
Suppose the population that we send to this machine is a collection of voting ballots. In this collection, let's say a whopping 0.5% of the ballots were fraudulently cast by someone forging a signature. This is likely WAAAAY larger than what you would actually see in person. Eg that would mean that 1 out of every 200 ballots was fraud. But the effect gets only more lopsided the lower the fraud rate is, so let's start with that.
Suppose you get a ballot back and it has been flagged by your machine as a non-match. What is the probability that this ballot truly is fraudulent?
The problem here is that the initial accuracy of the machine is taken with respect to two different populations: we know it is 99.99% accurate on anything that is truly fraud, and 99.9% accurate on anything that is truly honest. But when we get a ballot back with a mark, we don't know whether it is fraudulent or honest. The marks are a mix of true positives and false positives. And the chances of it being fraudulent or honest given that it was marked as such depend on the overall rate of fraud in the population.
The math looks like this (where "|" means "conditioned on", eg Pr(A | B) means Probability of A given we know B):
Pr( Fraudulent | Marked by our machine ) =
Pr(Fraudulent) * Pr(Marked | Fraudulent)
------------------------------------------------------------------------------------------
Pr(Fraudulent) * Pr(Marked | Fraudulent) + Pr(Not Fraudulent) * Pr(Marked | Not Fraudulent)
The numerator is an expression that gives us the percent of ballots that are both Fraudulent and marked as such. The denominator is the expression for the the percent of ballots that are marked as fraudulent, including both the true positives from the numerator (the first term in the sum), and the non-fraudulent ones that are accidentally marked (the second term in the sum).
The numbers for our very accurate machine are:
= 0.005*0.9999 / (0.005*0.9999 + 0.995*0.001)
= 0.834 or 83.4% chance of being fraudulent given it was marked.
Is 83.4 percent a good number? I mean it's pretty high. But let's examine it further. What it means is that 17% of these marked ballots that we are throwing away are wrong. How many marked ballots are there? That's the denominator. It is about 0.06% of all ballots. So now I've thrown away 100 good votes to try and catch 500 bad ones. Is that a good tradeoff?
What happens if the machines aren't nearly as accurate as 0.999 on true matches? Suppose they will as before, almost always flag a true mismatch, but they are only 99% accurate at discovering true matches. Maybe 1% of the population just doesn't have that consistent of a signature. (Apparently I am in that 1% if my bank is concerned. Wrong 1% to be in wrt a bank account, *sigh*)
Then the numbers are:
= 0.005*0.9999 / (0.005*0.9999 + 0.995*0.01)
= 33% chance of truly being fraudulent if it is marked.
And now, to try and catch the 1 in 200 people who are cheating, we've gone and disenfranchised 1 in 100 who aren't. We've flagged 1.5% of ballots, and two thirds of them were perfectly honest.
What if the population of cheaters is way less than 1 in every 200 ballots? What if it's 1 in every 2000? For our more accurate machine that is 99.9% accurate on non-fraudulent ballots we have
= 0.0005*0.9999 / (0.0005*0.9999 + 0.9995*0.001)
= 8% chance of truly being fraudulent when it is marked. And we have disenfranchised 0.5% of voters trying to catch ten times fewer than that.
Dare I ask what the numbers are for our less accurate algorithm? The one that is only 99% good at marking honest ballots as honest? Those numbers are:
= 0.0005*0.9999 / (0.0005*0.9999 + 0.9995*0.01)
= 4.7% chance of being fraudulent if it is marked. Almost 96% of flagged ballots will be honest in this case, and we are marking 1% of ballots. We are disenfranchising 1 out of 100 honest voters to try and catch 1 in 2000 who are cheating.
Those numbers just don't add up.
