[A] judgement about whether one subject is ‘harder’ than another depends very much on who happened to take those subjects. And if the characteristics of the entry change, so would the supposed difficulties.This to me is the root of the problem; the very idea of two tasks being "equally hard" fails to take into account how "hard" is measured as regards the people selected for those tasks. There's also a brilliant point made by Goldstein and Cresswell that for any A-level with an average mark of X%, one could construct a sufficiently easy (or difficult) spelling test that would have an X% pass mark as well, but it would not be sensible to refer to them as "equally hard"; the differences in the two tasks make the comparison ridiculous.
Tuesday, 18 August 2009
Teach Them HARD! Redux
Since Chemie was kind enough to offer some links on the subject of A-level difficulty/desirability, I have a quick peruse of the TSR page on the matter. More specifically, I looked up the paper it referred to, which can be found here. It's worth a quick look, if you have time. Page 111 has the graph of their results (short analysis: science subjects are amongst the hardest, Spielbergo's sister can be rest assured that music is really tough too, Gooder might be annoyed to find that almost the only thing claimed to be easier than sociology is film studies), but it's also worth looking at Section 9.1 (beginning p115), which describes some of the criticisms of the method, that match and expand upon some of my concerns going into this. Section 91.4. in particular reminds us that:
Subscribe to:
Post Comments (Atom)
5 comments:
Why is that ridiculous? I think it would be perfectly correct to describe those two tests as equally hard (assuming the exams are taken by a sufficiently large random sample of the population).
On brazenly skimming the paper their position appears to be that "It is meaningless to say, for example, that ‘art is easier than physics’; they are just different.". This is of course true, but it is not meaningless to say that Art Exam A is easier than Physics Exam B, which is what we're actually trying to determine. For example, it is surely true to say that for a person of average abilities, a Grade 1 music exam is easier than a final year degree exam in physics, because it takes less effort on the part of this person to get a pass in the music exam. It may be difficult to prove this with statistics, but that doesn't make it a ridiculous concept. In fact the whole system of examinations rests on it being a real concept.
Oh at A-level Sociolgy and Film Studies are both laughably easy it's true.
The difficulty curve however is akin to the cliffs of Dover when you move to degree and then masters level
"Why is that ridiculous? I think it would be perfectly correct to describe those two tests as equally hard (assuming the exams are taken by a sufficiently large random sample of the population)."
Because "hard" is not simply a measure of the chance of success. The time taken to prepare and the hurdles to be cleared across the process are relevant factors as well. The fact that both a two-year course and a (say) ten minute spelling test has the same proportion of passes completely ignores those factors (note that I said "an A-level", not "an A-level exam"; if we're going to talk seriously about respective difficulties we need to consider the courses as a whole).
As to the paper's position, whilst what you point to certainly features, I'm not sure I'd say it was their "position", exactly, though since I've only skimmed it too my reading might be wrong. Regardless, I agree that this particular argument needs to be fleshed out.
In general, I'm not sure how much stock I'd put in the argument that min(X)is less than max(Y) implies the sets X and Y can be sensibly compared. This is partially my maths brain kicking in, if you have two intervals you want to compare, there are many different ways to do it, all with their own advantages and problems, so whilst you can argue it can be done in specific cases, it does not follow that it makes sense to do it in general, especially when they are being compared with regard to so fuzzy a metric as "hard". I'd much rather look at Scarlett Johansson than a bisected sheep suspended in chemicals, but that doesn't mean I think it's sensible to compare the aesthetics of women with that of art.
The counter to the above is that "Grade 1 piano is less than General Relativity II" is more objectively true than "attractive blonde is greater than mutilated ovine", but even then we run into problems. Would Stephen Hawking find Grade 1 piano easier? Which Grades could be compared to a degree-level physics module? Were the writers of the paper to have included the phrase "impossible to compare within the bounds of two year courses designed for 16-18 year olds with a roughly equal level of experience", then it becomes harder to refute their position.
Still don't agree about spelling tests: it's an illusion that there's no preparation involved for one, as in fact many years of (mainly reading) practice are required to become a good speller. If someone scored poorly in such a test, it would require a lot of effort on their part to raise their standard up to the point of scoring well. It may not be as worthwhile an effort as a broader subject, but that doesn't mean it isn't equally hard.
I take your point about comparing min(X) with max(Y), but I purposely chose an extreme example to demonstrate that real differences in difficulty between particular levels of subjects must exist. Smaller differences are harder to judge, of course, but I don't think that means it's not sensible to try.
"Would Stephen Hawking find Grade 1 piano easier?"
I've always accepted that we need to average over a large sample so that outliers aren't important.
"Which Grades could be compared to a degree-level physics module?"
Yep, that's a tough question, because I expect the set of people who do both contains only high-achievers, and it's probably not a large set either. So contrary to your final point, comparison of A-level difficult should be much easier that this because there is a large sample with roughly equal levels of experience. I do concede that there's no absolute level of hardness - the average Frenchman would find A-level French a doddle, for example - but in the context of the population of A-level students, a rough measurement ought to be possible
"Still don't agree about spelling tests: it's an illusion that there's no preparation involved for one, as in fact many years of (mainly reading) practice are required to become a good speller. If someone scored poorly in such a test, it would require a lot of effort on their part to raise their standard up to the point of scoring well. It may not be as worthwhile an effort as a broader subject, but that doesn't mean it isn't equally hard."
I'm not convinced, for several reasons. Firstly, inflexibility of grade (i.e. the work needed to improve one's score) is a different beast to average grade (the most ludicrous example can demonstrate this; I could offer an A-Level in breathing that gave a B to everyone who could breathe, and an A to anyone who could go without breathing for four minutes). The task of upping one's grade may be hard, or easy, but it shouldn't be conflated with the difficulty of the course itself. Secondly, we return to the other part of the discussion; the passive preparation of becoming better at spelling is a very different beast to active study, once again making comparison difficult.
Mainly, though, I don't agree because any argument that spelling tests are equally hard as A-level exams if they have the same average score can also be used to imply General Studies is harder than Maths, since the latter has a higher average score. General Studies is at heart a spelling test across many different subjects, a reflection (essentially) on how well-informed you are across the board. The passive preparation for the exam (which was most certainly the only preparation I did) is comparable to the passive prep for a spelling test.
"Yep, that's a tough question, because I expect the set of people who do both contains only high-achievers, and it's probably not a large set either."
That wasn't really the intent of my question, though the fault is probably mine. A much better one would be "which grade is equivalent to a Physics A-level?" A large sample size doesn't seem particularly useful in determining that equivalence, because it's not a lack of data causing the problem; it's not having a sensible method of analysing that data. As mentioned, two intervals cannot necessarily be compared, irrespective of how many data points were used to calculate them.
Post a Comment