This was emailed to me by a student earlier today. Thanks to years of training in the mystical arts of the statistik, I am able to pinpoint where exactly these values (the final marks of Polish high-schools students for the year) seem to come unstuck. You might find it more difficult, of course. Just bear in mind that 30% is the pass mark here.
(This does rather make me wonder if I can get away with something similar in this week's exam board, but I suspect this kind of of shenanigans is harder to pull off when surrounded by mathematicians.)
Showing posts with label Statistics Is Your Friend. Show all posts
Showing posts with label Statistics Is Your Friend. Show all posts
Tuesday, 30 June 2015
Tuesday, 24 June 2014
Statistics Is Alarming And Depressing
So apparently this May was the 351st month in a row to have a global temperate higher than the 20th century mean temperature for the equivalent month. I figured it was worth doing a little noodling here. Let's assume global warming isn't real, and test how likely we would be to see these results if that were true.
First, let's assume that each month has an equal chance of being above the average temperature and of being below the average temperature. That's actually a pretty reasonable assumption, by definition. But we'll also include the much less likely assumption that each month's temperature is independent of the temperatures of the months before and after it. Obviously this is problematic - a hot June implies a hot July - but we'll run with that for now.
Under these assumptions, the chance of 351 above average months in a row is equal to 0.5 to the power 351, or, expressed as a percentage, a probability of:
0.000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000022%
Winning the National Lottery seven weeks in a row is more likely.
But what about that terrible independence assumption? Well, we can compensate. Let's assume that if the month before was above average temperature-wise, there's a 95% chance this month will be above average too. That's a number plucked entirely out of thin air, but it's deliberately high. Obviously I'm not a climatologist or a meteorologist, but from personal experience I'd be surprised if that were still too low - comments naturally welcomed.
So what do we do now? Well, since we know every month was above average, there was a 50% chance of month 1 being above average, and a 95% chance every other month was. That gives us a probability equal to 0.5 multiplied by 0.95 to the power 350, which is 0.00000080%.
The true chance almost certainly lies between those two extremes, but at the very best, the chance of seeing what we've seen without global warming being real is smaller than the chance of phoning three people at random and finding out they all share your birthday.
Your move, George Will.
(And since I've invoked the name of the bow-tied charlatan and rape apologist, I should really share this piece by Amanda Marcotte, which really got me chuckling. Remember kids, if you write a column in which you worry aloud that men who "only" grope a girl and don't rape her might be getting too bad a rap, it's really unbecoming and evidence of simple-mindedness if you get upset about it.)
First, let's assume that each month has an equal chance of being above the average temperature and of being below the average temperature. That's actually a pretty reasonable assumption, by definition. But we'll also include the much less likely assumption that each month's temperature is independent of the temperatures of the months before and after it. Obviously this is problematic - a hot June implies a hot July - but we'll run with that for now.
Under these assumptions, the chance of 351 above average months in a row is equal to 0.5 to the power 351, or, expressed as a percentage, a probability of:
0.000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000022%
Winning the National Lottery seven weeks in a row is more likely.
But what about that terrible independence assumption? Well, we can compensate. Let's assume that if the month before was above average temperature-wise, there's a 95% chance this month will be above average too. That's a number plucked entirely out of thin air, but it's deliberately high. Obviously I'm not a climatologist or a meteorologist, but from personal experience I'd be surprised if that were still too low - comments naturally welcomed.
So what do we do now? Well, since we know every month was above average, there was a 50% chance of month 1 being above average, and a 95% chance every other month was. That gives us a probability equal to 0.5 multiplied by 0.95 to the power 350, which is 0.00000080%.
The true chance almost certainly lies between those two extremes, but at the very best, the chance of seeing what we've seen without global warming being real is smaller than the chance of phoning three people at random and finding out they all share your birthday.
Your move, George Will.
(And since I've invoked the name of the bow-tied charlatan and rape apologist, I should really share this piece by Amanda Marcotte, which really got me chuckling. Remember kids, if you write a column in which you worry aloud that men who "only" grope a girl and don't rape her might be getting too bad a rap, it's really unbecoming and evidence of simple-mindedness if you get upset about it.)
Thursday, 1 May 2014
FOXFix
This is why I hate them. This is why I always introduced my GCSE stats lessons as "lessons in lying". This is why I'm going back to statistics teaching (undergraduate level this time) full time in September.
FOX News:
Correct graph:
Note that they didn't just invert the y-axis. Any fool can do that. Note that they rearranged the x-axis to put events out of chronological order. That's some high level bullshitting right there.
(Important note: just because I've corrected this doesn't mean I think the correct graph is actually particularly useful; I don't. But at least the information is honestly displayed.)
Update: forgot to h/t JJ and There's Coffee In That Nebula. My apologies, folks!
Update 2: Clay points out in comments quite persuasively that it's a fake, though a fake based on screenshots from another spectacularly bad FOX graphic. Clearly I should have checked thoroughly before throwing this up. We regret the error.
FOX News:
Correct graph:
Note that they didn't just invert the y-axis. Any fool can do that. Note that they rearranged the x-axis to put events out of chronological order. That's some high level bullshitting right there.
(Important note: just because I've corrected this doesn't mean I think the correct graph is actually particularly useful; I don't. But at least the information is honestly displayed.)
Update: forgot to h/t JJ and There's Coffee In That Nebula. My apologies, folks!
Update 2: Clay points out in comments quite persuasively that it's a fake, though a fake based on screenshots from another spectacularly bad FOX graphic. Clearly I should have checked thoroughly before throwing this up. We regret the error.
Thursday, 23 January 2014
"Make Me A Coffee And Swallow This Pill"
One of the main roles in my job is to meet people who want to put together a medical trial, and help them to construct in such a way that the analysis will be sound, and the process will be ethical. The resulting meetings vary wildly in how easily they go, of course, depending on both the experience and the temperament of those involved.
Every now and again truly ridiculous ideas are thrown up, like the idea of forming a control group out of people who refuse to be included in the trial (let's just steal their info without their knowledge) or creating them from people who can't speak English to consent in the first place. No obvious statistical or ethical problems there!
As crazy as some of these ideas are, they always come from small cogs in larger machines. If I didn't shoot them down, someone else would; that's why they come to me in the first place. So when I see governors of entire states buying tickets for the Ethical Vacuum Express, I get worried.
There are hundreds of thousands of people in America and across the world who would say the biggest problem with this idea is that Virginia hasn't done enough to crush its unions to ensure the skids are greased enough for medical experimentation on your underlings. An awful lot of them spend extraordinary amounts of money to acquire the ears of the people who write the law. This should be more of a worry than it is currently being allowed to be.
(h/t Rising Hegemon)
Every now and again truly ridiculous ideas are thrown up, like the idea of forming a control group out of people who refuse to be included in the trial (let's just steal their info without their knowledge) or creating them from people who can't speak English to consent in the first place. No obvious statistical or ethical problems there!
As crazy as some of these ideas are, they always come from small cogs in larger machines. If I didn't shoot them down, someone else would; that's why they come to me in the first place. So when I see governors of entire states buying tickets for the Ethical Vacuum Express, I get worried.
In August 2011, following an email from Bob McDonnell to Virginia's secretary of health, Maureen McDonnell met at the Executive Mansion with Williams and one of the secretary's senior policy advisors. At that meeting, according to the indictment, Williams discussed the idea of having Virginia government employees use Anatabloc, Star Scientific's anti-inflammatory dietary supplement, "as a control group for research studies."
This wasn't the only time this kind of idea came up. In October 2011, according to the indictment, Maureen McDonnell accompanied Williams and a research scientist who consulted for Star Scientific to a company event in Grand Blanc, Mich... The scientist later emailed Maureen McDonnell a summary of their discussions. In it, he suggested it might be useful "to perform a study of Virginia government employees… to determine the prevalences [sic] of autoimmune and inflammatory conditions."This, of course, is two tastes of conservative thought tasting great together. You've got the idea that you shouldn't feel bad about assuming the people working under can be taken advantage of above and beyond the fact they do what you want them to for fairly indifferent pay, and you have the idea (not stated, but doubtless ready to be deployed at a moment's notice) that if this scheme were made voluntary, all considerations about inappropriate arm-twisting would suddenly disappear. An employee taking on a role they really shouldn't be expected to for fear of rocking the boat? Unpossible!
There are hundreds of thousands of people in America and across the world who would say the biggest problem with this idea is that Virginia hasn't done enough to crush its unions to ensure the skids are greased enough for medical experimentation on your underlings. An awful lot of them spend extraordinary amounts of money to acquire the ears of the people who write the law. This should be more of a worry than it is currently being allowed to be.
(h/t Rising Hegemon)
Wednesday, 9 October 2013
I'm A Doctor As Well, You Know. Doctor Too, Is What I'm Saying.
With tomorrow being D-Day for those excited about newly rediscovered Doctor Who episodes - a group that most certainly includes me - I have a golden opportunity to bust some basic probability out on y'all. What are the chances of either finishing a partly-complete story, or of unearthing an episode from an entirely absent story?
If we ignore "Mission to the Unknown" (this was Abi's idea, so complain to her if you don't like it; personally I went for it mainly because it simplifies the maths), there are nine entirely absent stories. There are six stories missing two episodes, and one story ("The Tenth Planet") missing one. The BBC have referred to missing episodes, plural, so if we take the worst case scenario, the chances of completing a full story are around 2%, and the chances of seeing part of a hitherto entirely unavailabe story rests at 60%.
But maybe things are less bleak. Maybe the BBC found three episodes. At that point, we're looking at a 3% chance of a complete story, and a 74% chance something entirely new has turned up.
Except, though, that those calculations assume independent finds, which seems pretty unlikely. Multiple finds don't necessarily mean episodes from a single story, but that's a definite possibility (see "Cybermen, Tomb of"), and the contrary idea - finding an episode from one story means the next find is actually less likely to come from that story - seems on its face to be foolish.
That means we can treat the figures for the probability of a complete story as minimum values, but the probabilities of "new" stories as maximums.
Anyway, here's the results of a few hours noodling during a day of boring talks:
(Apparently there might be as many as nine unearthed episodes, but I ran out of talks.)
Cheers to tweeps Abi, JJ, and Fonz for helping me out with the numbers of missing episodes/stories whilst I was out of reach of my laptop.
If we ignore "Mission to the Unknown" (this was Abi's idea, so complain to her if you don't like it; personally I went for it mainly because it simplifies the maths), there are nine entirely absent stories. There are six stories missing two episodes, and one story ("The Tenth Planet") missing one. The BBC have referred to missing episodes, plural, so if we take the worst case scenario, the chances of completing a full story are around 2%, and the chances of seeing part of a hitherto entirely unavailabe story rests at 60%.
But maybe things are less bleak. Maybe the BBC found three episodes. At that point, we're looking at a 3% chance of a complete story, and a 74% chance something entirely new has turned up.
Except, though, that those calculations assume independent finds, which seems pretty unlikely. Multiple finds don't necessarily mean episodes from a single story, but that's a definite possibility (see "Cybermen, Tomb of"), and the contrary idea - finding an episode from one story means the next find is actually less likely to come from that story - seems on its face to be foolish.
That means we can treat the figures for the probability of a complete story as minimum values, but the probabilities of "new" stories as maximums.
Anyway, here's the results of a few hours noodling during a day of boring talks:
(Apparently there might be as many as nine unearthed episodes, but I ran out of talks.)
Cheers to tweeps Abi, JJ, and Fonz for helping me out with the numbers of missing episodes/stories whilst I was out of reach of my laptop.
Friday, 26 July 2013
Friday Randomness
Randomness indeed. I don't know what's weirder; the fact that you can buy this:
or that inquisitive penguins were exactly the medium I've frequently used to describe my probability-based PhD thesis to inquisitive lay-people.
or that inquisitive penguins were exactly the medium I've frequently used to describe my probability-based PhD thesis to inquisitive lay-people.
Friday, 31 May 2013
"Get Into That Kitchen And Make Me A Natural BLT, Woman!"
I'm feeling lazy today (as well as stressed; I've got a meeting coming up with the second-worst PhD student I've ever met because I'm at the shitty end of an extremely long list of people who've passed the buck on him), so how about I just pick on an obvious idiot: Erick Erickson
It's the invocation of science that's truly wonderful, however. Science, we now learn, simply means that the most common occurrences are the right occurrences. That shit be biology, yo!
In celebration of this brave new face of the scientific method, I propose the following list of Erickson's Laws, each of which details behaviour or concepts that only those dabbling in anti-science would even consider:
After a Pew Research report found that mothers were the sole breadwinners in 40% of American households with children, Erick Erickson said on Fox Business that it is “anti-science” to suggest that’s acceptable.
Said Erickson: “I’m so used to liberals telling conservatives that they’re anti-science. But liberals who defend this and say it is not a bad thing are very anti-science. When you look at biology — when you look at the natural world — the roles of a male and a female in society and in other animals, the male typically is the dominant role. The female, it’s not antithesis, or it’s not competing, it’s a complementary role.”This is so hilarious a misunderstanding of the words "natural" and "science" it almost qualifies as art. When Erickson looks around this society we've artificially created to look inside the houses we've artificially constructed, he's horrified to learn that the person gaining the money we've artificially given meaning to isn't being collected by the natural people? "Natural breadwinner" is no less an oxymoron than "natural shuttle-pilot".
It's the invocation of science that's truly wonderful, however. Science, we now learn, simply means that the most common occurrences are the right occurrences. That shit be biology, yo!
In celebration of this brave new face of the scientific method, I propose the following list of Erickson's Laws, each of which details behaviour or concepts that only those dabbling in anti-science would even consider:
- Women making money
- Cooked food
- Objecting to people fucking in public
- DVDs
- Antibiotics
- Giving birth
- Spines
Friday, 24 May 2013
Cheatassonne
Ah, X-Box Carcassonne, you is so pretty. But why you hurt me so?
Let's leave aside the question of whose bright idea it was to emulate a turn-based game and require each player to use a separate controller, and get to the real meat of the problem; the games utter inability to even plausibly mimic random tile allocation.
For those who haven't played this game before, the idea is that each player in turn is given a tile, on which can be one or more of three features: part of a city, a length of road, or a cloister. The tile is then placed next to the tiles already on the board, so that e.g. a city piece is adjacent to another city piece. Completing cities and roads gets you points; cloisters give you points when you surround them, monks being agoraphobics, presumably. You can also place farmers in the fields, who get points if they have nearby cities to sell corn to.
It's all pretty simple, but it rather relies on the idea that each player has the same chance (more or less; the tiles don't always quite divide equally) of getting each tile. Specifically, cloisters are often the most sought after items, because they can be quite valuable, and you can sort of leave them to themselves whilst you build other structures around them.
In my last three River games against four AI players, I have received precisely zero cloisters of the seven per game available. The chances of this are less than 2%. In my last game I came second, losing to an AI by 19 points. 26 of his points came from the three cloisters he acquired, of the seven available to five players. The chances of this are less than 15%. Yes, I confess that falls short of significant, but I figure if I whine about this now, I'll either collect more evidence in later games, or the universe will try to spite me by proving me wrong, and actually allow me to win a gorram game.
Those of you who suspect I'm simply applying basic statistics to a case of sore losing may not be entirely devoid of a point. This doesn't mean you can't all piss off, though.
Monday, 20 May 2013
A) Aggravating, Or B) Aggravating And Enraging
Here's an interesting study dug up by bspencer over at Lawyers, Guns and Money, who right now is embroiled in a massive sprawling multi-thread discussion on what it means when men complain about feminists being dismissive of them.
On that more general point, I don't want to say too much (short version: there is a big difference between recognising some people can use "check your privilege" as a way to shut down male comments and thinking it's so major and commonplace a problem as to make it worth mentioning in your introductory talk for a secular conference for women, because they'll totally never have thought about that), but the secularist issue has spawned some interesting tangents, including bspenser's suggestion that, per the link above, Americans dislike atheists more than they do Muslims.
Again, this isn't something I want to dwell on - there's all sorts of reasons why labelling of a hypothetical person as a Muslim does not translate into how actual Muslims (for which one can frequently simple read: brown people) are considered, and of course there's a greater distance still between how people feel about a minority and how oppressed that minority is (I should note that bspenser isn't claiming otherwise on that score).
I'm mentioning all of this because the suggestion that atheists are more reviled than Muslims struck me as so ludicrous on the face of it that the Patheos link proves strikes me as very important (though it really demonstrates a lack of trust than hatred, I think). Even with - or maybe because of - the amount of time I spend reading up on American culture and politics, I have to continually remind myself how very differently atheism is viewed in (some or much of) the United States to my own experiences in Britain, especially since it's so completely counter to how one would expect things to be given which of those two countries has Christianity as its official state religion.
Let's break down the Patheos piece a little. Firstly, a few sentences on the conjunction fallacy, which basically goes like this: "a figure bounds out of a TARDIS and saves Earth from the Daleks. Is it more likely our saviour is a) a Time Lord, or b) a Time Lord and the Doctor". Much is made of the fact that most people will choose b), even though by its very nature, a) is more plausible, since if b) is true, a) is true as well.
To simplify significantly, the idea here is that the fallacy is more seductive the more the question seems to describe the subset of people in the second group. If the above was re-written to have answers "a) a Time Lord" and "b) a Time Lord who is under 170cm tall", people would be more likely to realise what's going on. There is, in other words, a correlation between people's inherent assumptions - and by extension their prejudices - and how likely they are to commit the fallacy. Whether this is because the added assumptions make it harder for someone to not read option a) as say, "a Time Lord but not the Doctor", or if people's grasp of probability is being overloaded, I don't know [1].
So, here the idea in this research was to see how often the fallacy was applied when considering different types of people. If the question is re-written as "Somebody does some pretty shitty things, is he a) a teacher, or b) a teacher and some quality X", where X is Christian, Muslim, rapist, or atheist.
You can see the results at the link. Even when you offer the idea that the man is a rapist - that is a man who's already committed criminal acts worse than the ones described in the study - fewer people make the intuitive leap. I suppose one could make the argument that it's easier to spot the fallacy when given other unrelated crimes than when considering a persons' (lack of) belief system, but even so, when "just because he's a rapist doesn't make him a thief" carries more rhetorical power than "just because he's an atheist doesn't make him a thief", something strange is definitely going on. [2]
All of which is a sobering reminder of how lucky I really am. Not only am I a heterosexual white cis man with a middle class background and a decent job, but I also get to be an atheist in a country and in a societal circle in which the only reason people would have to think me likely to commit acts of grotesque immorality is having met me. Thank the lack of God for that!
[1] There's an alternative version of the question that goes like this: "Emma is a sexually confident, assertive woman, who likes to wear high heels and tops that show off her cleavage. Is she most likely to be a) a librarian or b) a stripper? The idea here is that there are (allegedly) fewer strippers in the UK than librarians, and therefore if the other information offered to us is irrelevant, a) is the correct answer. You can learn a great deal about people by seeing how much of the additional information they will claim isn't irrelevant at all.
[2] Admittedly, that's just from looking at the means. Using the confidence intervals, we can only say that people consider lawbreaking to be no less common a feature of atheists than they do of rapists.
On that more general point, I don't want to say too much (short version: there is a big difference between recognising some people can use "check your privilege" as a way to shut down male comments and thinking it's so major and commonplace a problem as to make it worth mentioning in your introductory talk for a secular conference for women, because they'll totally never have thought about that), but the secularist issue has spawned some interesting tangents, including bspenser's suggestion that, per the link above, Americans dislike atheists more than they do Muslims.
Again, this isn't something I want to dwell on - there's all sorts of reasons why labelling of a hypothetical person as a Muslim does not translate into how actual Muslims (for which one can frequently simple read: brown people) are considered, and of course there's a greater distance still between how people feel about a minority and how oppressed that minority is (I should note that bspenser isn't claiming otherwise on that score).
I'm mentioning all of this because the suggestion that atheists are more reviled than Muslims struck me as so ludicrous on the face of it that the Patheos link proves strikes me as very important (though it really demonstrates a lack of trust than hatred, I think). Even with - or maybe because of - the amount of time I spend reading up on American culture and politics, I have to continually remind myself how very differently atheism is viewed in (some or much of) the United States to my own experiences in Britain, especially since it's so completely counter to how one would expect things to be given which of those two countries has Christianity as its official state religion.
Let's break down the Patheos piece a little. Firstly, a few sentences on the conjunction fallacy, which basically goes like this: "a figure bounds out of a TARDIS and saves Earth from the Daleks. Is it more likely our saviour is a) a Time Lord, or b) a Time Lord and the Doctor". Much is made of the fact that most people will choose b), even though by its very nature, a) is more plausible, since if b) is true, a) is true as well.
To simplify significantly, the idea here is that the fallacy is more seductive the more the question seems to describe the subset of people in the second group. If the above was re-written to have answers "a) a Time Lord" and "b) a Time Lord who is under 170cm tall", people would be more likely to realise what's going on. There is, in other words, a correlation between people's inherent assumptions - and by extension their prejudices - and how likely they are to commit the fallacy. Whether this is because the added assumptions make it harder for someone to not read option a) as say, "a Time Lord but not the Doctor", or if people's grasp of probability is being overloaded, I don't know [1].
So, here the idea in this research was to see how often the fallacy was applied when considering different types of people. If the question is re-written as "Somebody does some pretty shitty things, is he a) a teacher, or b) a teacher and some quality X", where X is Christian, Muslim, rapist, or atheist.
You can see the results at the link. Even when you offer the idea that the man is a rapist - that is a man who's already committed criminal acts worse than the ones described in the study - fewer people make the intuitive leap. I suppose one could make the argument that it's easier to spot the fallacy when given other unrelated crimes than when considering a persons' (lack of) belief system, but even so, when "just because he's a rapist doesn't make him a thief" carries more rhetorical power than "just because he's an atheist doesn't make him a thief", something strange is definitely going on. [2]
All of which is a sobering reminder of how lucky I really am. Not only am I a heterosexual white cis man with a middle class background and a decent job, but I also get to be an atheist in a country and in a societal circle in which the only reason people would have to think me likely to commit acts of grotesque immorality is having met me. Thank the lack of God for that!
[1] There's an alternative version of the question that goes like this: "Emma is a sexually confident, assertive woman, who likes to wear high heels and tops that show off her cleavage. Is she most likely to be a) a librarian or b) a stripper? The idea here is that there are (allegedly) fewer strippers in the UK than librarians, and therefore if the other information offered to us is irrelevant, a) is the correct answer. You can learn a great deal about people by seeing how much of the additional information they will claim isn't irrelevant at all.
[2] Admittedly, that's just from looking at the means. Using the confidence intervals, we can only say that people consider lawbreaking to be no less common a feature of atheists than they do of rapists.
Thursday, 25 April 2013
Exercises In Maths Smackdown
Just passing this along from Kevin Drum. Like him, and an awful lot of other people, I don't believe anyone promising to cut government waste or demanding others cut it should be given the time of day unless and until they can give a single example of actual, you know, waste.
Pissing away one four millionth of the budget isn't going to cut it. Drum's example gets the job done expertly; a more cider-related analogy (and we're all about the cider here, Gods know) would be demanding a serving wench be replaced because she spills one millilitre for every ten thousand pints she pulls.
Pissing away one four millionth of the budget isn't going to cut it. Drum's example gets the job done expertly; a more cider-related analogy (and we're all about the cider here, Gods know) would be demanding a serving wench be replaced because she spills one millilitre for every ten thousand pints she pulls.
Monday, 25 February 2013
Sorkin Who? or Show Me The Moneyball!
Statistics and Sorkin? There was no way I wasn't going to get round to seeing this.
Except... it's not very Sorkiny, is it? Maybe it's Zaillian's influence - an entirely solid writer, but not a particularly flashy pen - but Moneyball's dialogue manages to be economical and engaging, but never particularly rhythmic or funny. Not that the film's trying to be funny, in fairness, but the end result would be difficult to recognise as involving Sorkin had his name not been in the credits. Even the occasional scene that bears a familiar Sorkin structure - the multiple rapid-fire calls to baseball managers Beane and Brand make as they shuffle players around and off the board being the most obvious one - seem strangely muted. This might just be a function of Miller's direction, of course. That said, Sorkin's writing has been getting increasingly austere of late in any case; you can draw a straight line through Studio 60... to Charlie Wilson's War to The Social Network to The Newsroom, with the former being indistinguishable from The West Wing except in whether anybody liked it, and each successive product drifting further and further from rapid-fire banter spat out in corridors.
Which is a shame, because as Sorkin himself has claimed, he's neither a top tier dramatist nor a top tier comedy writer, and his writing works as well as it does because of how good he is at mixing the two. Studio 60...'s biggest failing was both that the set-up relied to much on the comedy, and that his resulting instinct to up the drama didn't work at all in context, particularly after the West Wing (the question of how things would have turned out if you swapped Studio 60... and Sportsnight around in the chronology is an interesting one). I don't know if the critical and commercial failure of Studio 60... (which continues to be underrated; it's not great, but it's frequently pretty good) led to a deliberate shift in tone - one which did no damage to The Social Network, but made The Newsroom feel a little too cold and distant at times - but there seems to be no sign of it stopping, and it ends up hurting Moneyball.
Or so it seems to me. Perhaps I'm just ill-suited to judge the film, having little interest in baseball and plenty of interest in the sort of statistical juggling employed by Brand. Emphasising the former and skirting around the particulars of the latter is unquestionably the right choice for the film to have made, and exactly the wrong way round for me to really appreciate it. Like a foot fetishist watching Top Heavy Sluts 4, I understand utterly why the camera is focusing in the area it is, but I which they'd point us downward a little more often.
(Spoilers - by which I mean mention of historical events - below the fold)
Wednesday, 20 February 2013
What David Brooks Can't Do (Because He's Staggeringly Ignorant)
In the name of all things good and true, New York Times, will you keep fucking idiots like David Brooks out of my wheelhouse? It's just embarrassing, is what it is.
One would have thought that after Brooks went on national TV to explain how Nate Silver's election model was an example of cupidity and intellectual overreach, only for said model to end up not so much correct as spot on to a legitimately terrifying extent, he'd have the good sense to shut his pie-hole regarding the nature of data manipulation.
But no! Not this man! Not this man who's teaching a university class on humility and making his students read his own fucking book on humility (next semester: Silvio Berlusconi will show up to fuck as many co-eds as necessary before they understand the importance of virginity). Once again it's critically important that someone - who always happens to be David Brooks - must stand up and argue that only the arrogant believe the discipline they've dedicated their lives to can do what they say it does. One might think the true arrogance lies in a man with no detailed knowledge of a field lecturing us on what that field's limitations are, but since Brooks is an expert on humility, you'd look pretty stupid trying to point that out to him. It's not like humility is an easy concept to grasp without training, after all; it's not statistical analysis or anything.
I could rebut "What Data Can't Do" point by point, starting with the idea that the human mind is bad at maths because you can't calculate an irrational number in your head - roughly equivalent to arguing the human mind is bad at language because lot's of people can't spell chrysanthemum - and moving on to more serious points, such as his entire paragraph on data haystacks is countered using exactly five words: "Bonferroni correction, you smug prick".
Mainly, though, it's not worth the effort, because Brooks thinking regarding statistics suffers from the same problem as his thinking on everything else; a tireless desire to paint intellectual laziness as a principled position. At least 80% of the Brooks pieces I've read (and I admit that's not a huge sample, life being too short and all) are based on or at least involve the same maddening logical fallacy: "I don't know how you would do this so I don't believe it can be done". That in itself, of course, is just a gussied-up version of "I don't have the knowledge needed to rebut this argument I dislike, so I'll insist you don't have the knowledge either". Brooks course on humility sounds like the worst use of university time since my alma mater let Tony Blair in to get bloodstains all over our new building, but if he wanted to teach a course in using ignorance as a weapon, I'd sign up for it like a shot.
The aforementioned haranguing of Nate Silver (in which Brooks referred to him and statisticians like him as "wizards" living in "silly-land") is just the most obvious example of this particular brand of strategic ignorance, but his latest article comes close. Brooks knows sweet fuck-all about how to construct, test, improve and implement a statistical model (seriously, he doesn't think we have ways of dealing with the lack of a control group?), but that doesn't mean he should have to feel uncomfortable slapping together the thoughts he came up with in the shower and showing it in exchange for money to millions of people, many of whom could have put him right in seconds had he asked.
How fucking humble of him.
One would have thought that after Brooks went on national TV to explain how Nate Silver's election model was an example of cupidity and intellectual overreach, only for said model to end up not so much correct as spot on to a legitimately terrifying extent, he'd have the good sense to shut his pie-hole regarding the nature of data manipulation.
But no! Not this man! Not this man who's teaching a university class on humility and making his students read his own fucking book on humility (next semester: Silvio Berlusconi will show up to fuck as many co-eds as necessary before they understand the importance of virginity). Once again it's critically important that someone - who always happens to be David Brooks - must stand up and argue that only the arrogant believe the discipline they've dedicated their lives to can do what they say it does. One might think the true arrogance lies in a man with no detailed knowledge of a field lecturing us on what that field's limitations are, but since Brooks is an expert on humility, you'd look pretty stupid trying to point that out to him. It's not like humility is an easy concept to grasp without training, after all; it's not statistical analysis or anything.
I could rebut "What Data Can't Do" point by point, starting with the idea that the human mind is bad at maths because you can't calculate an irrational number in your head - roughly equivalent to arguing the human mind is bad at language because lot's of people can't spell chrysanthemum - and moving on to more serious points, such as his entire paragraph on data haystacks is countered using exactly five words: "Bonferroni correction, you smug prick".
Mainly, though, it's not worth the effort, because Brooks thinking regarding statistics suffers from the same problem as his thinking on everything else; a tireless desire to paint intellectual laziness as a principled position. At least 80% of the Brooks pieces I've read (and I admit that's not a huge sample, life being too short and all) are based on or at least involve the same maddening logical fallacy: "I don't know how you would do this so I don't believe it can be done". That in itself, of course, is just a gussied-up version of "I don't have the knowledge needed to rebut this argument I dislike, so I'll insist you don't have the knowledge either". Brooks course on humility sounds like the worst use of university time since my alma mater let Tony Blair in to get bloodstains all over our new building, but if he wanted to teach a course in using ignorance as a weapon, I'd sign up for it like a shot.
The aforementioned haranguing of Nate Silver (in which Brooks referred to him and statisticians like him as "wizards" living in "silly-land") is just the most obvious example of this particular brand of strategic ignorance, but his latest article comes close. Brooks knows sweet fuck-all about how to construct, test, improve and implement a statistical model (seriously, he doesn't think we have ways of dealing with the lack of a control group?), but that doesn't mean he should have to feel uncomfortable slapping together the thoughts he came up with in the shower and showing it in exchange for money to millions of people, many of whom could have put him right in seconds had he asked.
How fucking humble of him.
Monday, 5 November 2012
Autumn Branches
After thinking about it for a while, I decided that utterly witless Dylan Byers column I linked to last week needed more rubbishing, and that it might be worth breathing life into my It All Adds Up column over at GeekPlanetOnline to do it.
So I did that, then.
So I did that, then.
Tuesday, 30 October 2012
Questions Answered
Shorter Dylan Byers: Nate Silver offers multiple predictions with appropriate caveats and explanation of uncertainty. The only comment on the only prediction I was able to concentrate on doesn't agree with my gut. Now a word from people with no statistical training who agree with me.
Shorter Byers next week: why do liberals always think they're smarter than me?
Thursday, 31 May 2012
Illegal Truths
I'd planned on not posting anything else today, but then this was flagged on LGM, and I concluded it could not pass without comment. Various panjandrums in North Carolina are hoping to pass a bill that outlaws various statistical methods for the prediction of sea-level change.
OK, "outlaws" is a bit strong. Scientists can still use the methods, they just can't use them in anything the state will actually officially look at. And what are these deeply questionable, pseudo-sorcerous approaches that must be refuted as the bunkum they are?
Well, let me ask you a question: what comes next in the following sequence:
1, 2, 4, 8, 16, 32, 64, 128, ...
OK, "outlaws" is a bit strong. Scientists can still use the methods, they just can't use them in anything the state will actually officially look at. And what are these deeply questionable, pseudo-sorcerous approaches that must be refuted as the bunkum they are?
Well, let me ask you a question: what comes next in the following sequence:
1, 2, 4, 8, 16, 32, 64, 128, ...
Monday, 23 April 2012
Prior Data Conflict
My friend Gero (pictured here - and I still don't have my prize money, BTW) works in the field of prior data conflict. To simplify extraordinarily, this deals with the situation in which you've begun collecting data under certain assumptions about what you're going to find, and your first few results look like that theory is going to be blown out of the water.
I mention this purely because I just heard a snippet of Front Row on Radio 4 in which I received my first information about Joss Whedon's The Avengers: it's an embarrassment to feminism with terrible dialogue and a strongly right-wing message. So claims John Wilson, at least, about which I know nothing (save to say that anyone who precedes quoting MacBeth by pointing out he's about to use Shakespeare's words is either condescending, foolish or, on Shakespeare's "birthday", being entirely too cute).
I could phone Gero and ask him how to handle the exceptionally tricky task of figuring out exactly how to alter my preconceptions. Or, I could just read more reviews. It's almost as though all the stuff Gero (and I, and BigHead, and Ibb) work on turn out to not be useful in daily life. How strange...
I mention this purely because I just heard a snippet of Front Row on Radio 4 in which I received my first information about Joss Whedon's The Avengers: it's an embarrassment to feminism with terrible dialogue and a strongly right-wing message. So claims John Wilson, at least, about which I know nothing (save to say that anyone who precedes quoting MacBeth by pointing out he's about to use Shakespeare's words is either condescending, foolish or, on Shakespeare's "birthday", being entirely too cute).
I could phone Gero and ask him how to handle the exceptionally tricky task of figuring out exactly how to alter my preconceptions. Or, I could just read more reviews. It's almost as though all the stuff Gero (and I, and BigHead, and Ibb) work on turn out to not be useful in daily life. How strange...
Friday, 6 April 2012
A Tale Of Cocktails: These Are The New Facts
Top Ten
1. Brain Hemmorhage
2. Fuzzy Shark
3. Choc Berry
=4. Baby Guiness
=4. Dennis the Menace
6. Malibu Pop
=7. Angel Delight
=7. Kir Royale
=7. After Six
=10. Ume Royale
=10. Midori Sour
Worst Cocktail
Champagne Cocktail
Statistics
Mean = 6.80
Median = 6.8
Range = 3.1
Standard deviation = 0.70
Histogram
(Not too far from normally distributed, is it?)
Supplies Consumed
Booze
Advocaat
Amaretto
Blue Curacao
Brandy
Chambord
Champagne
Creme de Cassis
Creme de Menthe
Elderflower liquor
Gin
Irish Cream (Baileys)
Kahlua
Malibu
Midori
Peach Schnapps
Plum wine
Port
Rum (dark)
Sloe Gin
Tia Maria
Triple Sec
Vodka
Mixers
Bitters
Cocoa
Cranberry Juice
Cream
Grenadine
Lemon Juice
Lemonade
Lime Cordial
Milk
Orange Juice
Pineapple Juice
Sugar
Vanilla syrup
Garnish
Cherry
Chocolate
Lemon
Marshmallow
Mint Matchmaker
Orange
Whipped cream
A shark
Glasses
Champagne
Cocktail
Collins
Cordial
Highball
Shot
Estimated amount of ice used: 305 cubic centimetres.
1. Brain Hemmorhage
2. Fuzzy Shark
3. Choc Berry
=4. Baby Guiness
=4. Dennis the Menace
6. Malibu Pop
=7. Angel Delight
=7. Kir Royale
=7. After Six
=10. Ume Royale
=10. Midori Sour
Worst Cocktail
Champagne Cocktail
Statistics
Mean = 6.80
Median = 6.8
Range = 3.1
Standard deviation = 0.70
Histogram
(Not too far from normally distributed, is it?)
Supplies Consumed
Booze
Advocaat
Amaretto
Blue Curacao
Brandy
Chambord
Champagne
Creme de Cassis
Creme de Menthe
Elderflower liquor
Gin
Irish Cream (Baileys)
Kahlua
Malibu
Midori
Peach Schnapps
Plum wine
Port
Rum (dark)
Sloe Gin
Tia Maria
Triple Sec
Vodka
Mixers
Bitters
Cocoa
Cranberry Juice
Cream
Grenadine
Lemon Juice
Lemonade
Lime Cordial
Milk
Orange Juice
Pineapple Juice
Sugar
Vanilla syrup
Garnish
Cherry
Chocolate
Lemon
Marshmallow
Mint Matchmaker
Orange
Whipped cream
A shark
Glasses
Champagne
Cocktail
Collins
Cordial
Highball
Shot
Estimated amount of ice used: 305 cubic centimetres.
Saturday, 4 February 2012
A Favour To Fools
I'm still having a little trouble adjusting to being a statistician. Nominally, I'm half-statistician, half whatever-I-want-so-long-as-I-keep-publishing, but that was an arrangement worked out before it was known (or at least, known by me) that my first major project would require a degree of un-fucking so extensive it almost makes me sympathetic for whomever is running the Kormen Foundation's PR wing right now.
Still, as little as statistics grabs me as compared to probability theory, it's clearly a very important field. After all, it's thanks to the power of statistics that I can see through high-level political analyses like this one:
Oh, how I pity you people, scrabbling around in the dirt, waiting for someone with Godlike powers of deduction - I'm not saying my skills are Godlike, that's for history to decide - to peel away the layers of reality and reveal the truth, nestled in some hyperdimensional pocket reality that you would have no hope of reaching were it not for the warp-drives of the statistical mind.
Let me lay down some truth on you. 60% of Hispanics say they'll vote for Obama. 54% of registered Hispanic Republicans voted for Mitt Romney. The missing number that none of you will have thought of, because of how only someone as smart as me could possibly think of it?
31%.
Only 31% of registered Hispanic voters in Florida are Republican!
Are the deep complexities of this baffling situation slowly swimming into focus? Perhaps you require further hand-holding. No, don't apologise: I don't mind. It's the only way you'll learn. Obama current carries 60% of the Hispanic vote, which we'll assume, hardly unreasonably, means 60% of the registered Hispanic vote. Mitt Romney carried 54% of the 31% of registered Hispanic voters who identify as Republicans. That's less than 17% of the total registered Hispanic population (Lopez could have worked this out from the numbers he included in his own colum. Alas! If only CNN had access to calculators!).
Clearly we can assume many if not all of those Republican Hispanics who chose not to vote for Romney when offered Gingrich, Paul or Santorum instead will choose Romney when their only other viable option is Barack Obama. Indeed, if they all choose Romney then - let me just check my figures using the super-charged Mathematron that is my mind - he could carry 31% of the vote from Republicans alone! Against a mere 60% for Obama! To reiterate: ZOMG!
Oh, but what about the independent voters, Lopez wonders? They, after all, are the "real prize" [1]. And if 54% of Republicans like Romney, then how can we know how popular he is with the the constantly obsessed-over swing voter?
Well, maybe we can do some approximating using actual numbers that actual Lopez put into his actual column, like an undergrad who knows they have to include their sources even though they can't wrap their feeble minds around what their betters were saying in the first place. 38% of the registered Hispanics in the state are Democrats, leaving 31% of them unaffiliated. Let's assume every Democratically registered Hispanic will vote for Obama - that means that if the election were held today, 71% of unaffiliated voters would go for the 44th President.
That number again: 71%.
Of course, that number might be too high, but that would only be possible if registered Republicans were planning to vote for Obama as well, which ain't exactly bad news for the Democrats.
Tell me again how the Hispanic vote might be a hurdle for Obama, would you please?
You're welcome, the internet. Man, but do I need a cigarette...
(h/t Balloon Juice)
[1] It's been said before, but it's a source of continual irritation to me that so much time and energy goes in to working out how best to court people too fucking stupid to have decided whether they want to vote for the guy who'll slash taxes on the rich, dismantle the safety net for the very poor, and announce his arrival in the Oval Office by bombing every brown person between Tunisia and Turkmenistan the instant he's sat down, or alternatively some or none of those things as soon as he's told not to by Donald Trump. I realise there are people out there who really like all of those fucking bullshit alternatives of shittiness, but I can't understand how there are people who just aren't sure.
Still, as little as statistics grabs me as compared to probability theory, it's clearly a very important field. After all, it's thanks to the power of statistics that I can see through high-level political analyses like this one:
As Mitt Romney dominated the Florida Republican primary Tuesday night, he also captured the bulk of the votes from Latinos in the state, with 54% of their ballots... [T]he independents there who voted for [Obama] in 2008...will be the prize in the November election... Obama -- who starts with a 60% lead among all Latinos in state polls -- may end up battling Romney over the growing Latino vote.Now, I know what you're thinking, with your pathetic, unscientific minds. You're all like: ZOMG! 54% is close to 60%, oh noes! I mean, this shit seems airtight, doesn't it?
Oh, how I pity you people, scrabbling around in the dirt, waiting for someone with Godlike powers of deduction - I'm not saying my skills are Godlike, that's for history to decide - to peel away the layers of reality and reveal the truth, nestled in some hyperdimensional pocket reality that you would have no hope of reaching were it not for the warp-drives of the statistical mind.
Let me lay down some truth on you. 60% of Hispanics say they'll vote for Obama. 54% of registered Hispanic Republicans voted for Mitt Romney. The missing number that none of you will have thought of, because of how only someone as smart as me could possibly think of it?
31%.
Only 31% of registered Hispanic voters in Florida are Republican!
Are the deep complexities of this baffling situation slowly swimming into focus? Perhaps you require further hand-holding. No, don't apologise: I don't mind. It's the only way you'll learn. Obama current carries 60% of the Hispanic vote, which we'll assume, hardly unreasonably, means 60% of the registered Hispanic vote. Mitt Romney carried 54% of the 31% of registered Hispanic voters who identify as Republicans. That's less than 17% of the total registered Hispanic population (Lopez could have worked this out from the numbers he included in his own colum. Alas! If only CNN had access to calculators!).
Clearly we can assume many if not all of those Republican Hispanics who chose not to vote for Romney when offered Gingrich, Paul or Santorum instead will choose Romney when their only other viable option is Barack Obama. Indeed, if they all choose Romney then - let me just check my figures using the super-charged Mathematron that is my mind - he could carry 31% of the vote from Republicans alone! Against a mere 60% for Obama! To reiterate: ZOMG!
Oh, but what about the independent voters, Lopez wonders? They, after all, are the "real prize" [1]. And if 54% of Republicans like Romney, then how can we know how popular he is with the the constantly obsessed-over swing voter?
Well, maybe we can do some approximating using actual numbers that actual Lopez put into his actual column, like an undergrad who knows they have to include their sources even though they can't wrap their feeble minds around what their betters were saying in the first place. 38% of the registered Hispanics in the state are Democrats, leaving 31% of them unaffiliated. Let's assume every Democratically registered Hispanic will vote for Obama - that means that if the election were held today, 71% of unaffiliated voters would go for the 44th President.
That number again: 71%.
Of course, that number might be too high, but that would only be possible if registered Republicans were planning to vote for Obama as well, which ain't exactly bad news for the Democrats.
Tell me again how the Hispanic vote might be a hurdle for Obama, would you please?
You're welcome, the internet. Man, but do I need a cigarette...
(h/t Balloon Juice)
[1] It's been said before, but it's a source of continual irritation to me that so much time and energy goes in to working out how best to court people too fucking stupid to have decided whether they want to vote for the guy who'll slash taxes on the rich, dismantle the safety net for the very poor, and announce his arrival in the Oval Office by bombing every brown person between Tunisia and Turkmenistan the instant he's sat down, or alternatively some or none of those things as soon as he's told not to by Donald Trump. I realise there are people out there who really like all of those fucking bullshit alternatives of shittiness, but I can't understand how there are people who just aren't sure.
Tuesday, 20 December 2011
The Day Job
Gosh. This place is falling dangerously quiet, huh?
I do have an excuse, though. I'm currently in the middle of a hideous chain-reaction of ever more horrifying catastrophes at the office. I haven't been this dumbfounded by the degree to which things can go off the rails since I was asked to teach algebra to kids who recoiled in horror at the thought of multiplication.
It looks like I've put a lid on the chaos, though. Just. Who knows what will burst into flames once more the instant I've left for Christmas. Surely it can't be as bad as what happened last week, when our external expert revealed a shift in position meant he was no longer external, and that on careful reflection his field is too far from ours for him to constitute an expert. Like hiring a stripper for a stag do (I hear stories), and finding out she won't take off her underwear. And she's your sister.
Oh, and you only have three days to find another stripper, or the wedding's off. Some people have the strangest pre-nups...
I do have an excuse, though. I'm currently in the middle of a hideous chain-reaction of ever more horrifying catastrophes at the office. I haven't been this dumbfounded by the degree to which things can go off the rails since I was asked to teach algebra to kids who recoiled in horror at the thought of multiplication.
It looks like I've put a lid on the chaos, though. Just. Who knows what will burst into flames once more the instant I've left for Christmas. Surely it can't be as bad as what happened last week, when our external expert revealed a shift in position meant he was no longer external, and that on careful reflection his field is too far from ours for him to constitute an expert. Like hiring a stripper for a stag do (I hear stories), and finding out she won't take off her underwear. And she's your sister.
Oh, and you only have three days to find another stripper, or the wedding's off. Some people have the strangest pre-nups...
Monday, 21 November 2011
Statistics Is Fun And Easy
This caught my eye (from multiple sources; original here) today: a handy graph proving that you can underline some of the most serious obstacles to economic justice without even having to reach for a protractor.
Could the above have some connection to some US billionaires having a de facto 1% tax rate? We report, you decide!
Could the above have some connection to some US billionaires having a de facto 1% tax rate? We report, you decide!
Subscribe to:
Comments (Atom)








