Leaving Cert Standardisation

[Duke of Marmalade]

I attach the report on Leaving Cert standardisation. It is not bedtime reading. But it does give some insight into the algorithm (they don't call it that). Appendix G gives the technical details. It is really wonkish, I suspect deliberately so; it would be a brave soul who would challenge the math.

There follows my broad understanding of the approach, but I would welcome clarification/correction.

There are basically two inputs:
(1) The Teachers' Assessments
(2) Predictions from Junior cycle for that school and subject based on a regressed fitting of past Leaving Cert results to past Junior cycle performance. (Note: regression is based on results from all schools and subjects, and includes other predictors but the Junior Cert results correlation with Leaving Cert results dominates).

These provide two probability distributions for each school and subject combination (cell).
The distribution used for standardisation is a mixture of the two. The proportion of that mixture which is Junior Cycle Prediction is decided by how many students are in that cell. The formula for deriving that mixed distribution is where it gets really wonkish but it gives the example that if there were only 6 in the cell no credibility would be given to Junior Cycle Prediction (i.e. the Teachers' Assessments would be accepted without adjustment) and vice versa, if the cell was large a greater credibility would be given to the Junior Cycle Prediction. Unfortunately we are not told the limit of this credibility but I doubt it exceeds 50%. Having combined the distributions in this way the students are fitted into this standardised distribution (for that cell) based on the marks given by the Teachers' Assessment. So if someone is middle in the class they would be given the mid point of this standardised distribution. Note that the individual's own Junior Cycle performance has no role whatever in his/her final mark. The Junior Cycle Prediction is applied at the school level.

Now it is recognised that Teachers assessments were over optimistic and I assume the Junior Cycle Prediction was quite accurate in producing the historical averages. So the less credibility given to the Junior Cycle Prediction the better the chances of enjoying the optimism of the Teachers' Assessment.
These are some examples:
Maths
21,552 sitting
Historical Average H1s 5.8%
Teachers' Assessment 11.6%
Standardised 8.4%
So that sort of stacks up with an average 50% credibility being given to the maths Junior Cycle Prediction

Arabic
155 sitting
Historical Average H1s 17.1%
Teachers' Assessment 34.8%
Standardised 34.8%
Suggesting no credibility given to Junior Cycle Prediction

Latin
48 sitting
Historical Average H1s 19.2%
Teachers' Assessment 43.8%
Standardised 41.7%

Overall, I think any ambulance chasers will find it difficult to pick holes in this, though there may be a role for math expert witnesses

 

[Colm Fagan]

Hi Duke

I haven't looked at the documentation yet, but it might be worth checking if the standardisation process could have caused what appears to be a freak result with an Irish-German school (got from the RTE website):

"A Dublin school which places a focus on the teaching of German has expressed shock at lower than expected marks awarded to its students in this year's Leaving Cert Calculated Grades process.

14% of Leaving Certificate students at St Kilian's Deutsche Schule received H1 grades in German this year, compared to 41% last year.

The schools said it expected around half of its students to receive a H1."

 

[Duke of Marmalade]

Hi Duke

I haven't looked at the documentation yet, but it might be worth checking if the standardisation process could have caused what appears to be a freak result with an Irish-German school (got from the RTE website):

"A Dublin school which places a focus on the teaching of German has expressed shock at lower than expected marks awarded to its students in this year's Leaving Cert Calculated Grades process.

14% of Leaving Certificate students at St Kilian's Deutsche Schule received H1 grades in German this year, compared to 41% last year.

The schools said it expected around half of its students to receive a H1."

That is strange. The national position for German was as follows:
6,772 sitting
5.7% get H1s historically
Teachers' Assessment 13.9%
Standardised result 9.0%

If St Killians expected 50% H1s, I interpret that as the Teachers' Assessment.
So they must have got a very significant adjustment from the Junior Cycle Prediction.
But that would suggest that the Junior Cycle Prediction was way below the historic result of 41%, in fact way below the final result of 14%.
Definitely something not hanging together in this picture.

 

[Duke of Marmalade]

To whom it may concern,

I wish, on behalf of the LC Students of 2020, to signal our deep concern at the very flawed process applied to the calculation of grades in German at Higher level for this school. The results we have received bear no relationship to the ability, past performance or calculated mark awarded to the students from this school. I wish to have answered to the following questions.

• How come some students awarded the same calculated mark can be given 2 entirely different grades where there is a deviation of 2 grade levels?
• How come students given a calculated mark mid-way through a grade range can be down-graded by one or in some cases 2 grade levels?
• How come mother tongue students were down-graded from a H1 to H2 or in some cases a H3?
• How come students in our school who completed the German language curricular stream and achieved a level C1 in the German Sprachdiplom exams (far more challenging than the LC) were awarded H2 or H3 contrary to the marks calculated by their teachers? These pupils have reached the language proficiency level to study at a German University. The European Language Framework level for Sprachdiplom 2 is C1/C2, for Leaving Certificate it is B1/B2. Please explain to me the fairness of an algorithm that can get things so utterly wrong.
• How have the Gaelscoileanna fared in Irish? Is the same mismatch evident?

We calculated 19 H1 marks. This is not an inflation of grades in German for our school. We were awarded 6 H1 grades.

• 9 pupils got a grade in line with their calculated mark.
• 24 Pupils were downgraded from their calculated mark by one grade level.
• 8 Pupils were down-graded by two grade levels.

I want this investigated as it cannot be justified in any context given the nature of the school curriculum, students’ performances in German language exams from Germany and the performance of the school in the Leaving Certificate German exams every year.

The students in St Kilian’s have been seriously disadvantaged because of a warped bias based on the outcry in the UK. Where is the justice here? How is this wrong to be righted? Guidelines given to teachers stated that the school’s past performance would be considered as part of the national standardisation. I challenge you to go back and use the past performance for students from this school in German to come up with a fair and equitable assessment of their grades in this subject. We will seek information on the statistics for downgrades in Irish from Gealscoileanna to compare and establish bias.

Yours sincerely,
Alice Lynch, Principal

She's got a point. Nasty swipes at the Gaelscoileanna and our bending to the UK example.

 

[Duke of Marmalade]

I see this thread has been promoted from the depths. I understand that it is more important than the depths but unfortunately it will probably get less attention for that promotion. Any chance of a demotion?

 

[SPC100]

My guess: Their national model which predicts performance in leaving cert german, given junior cert outcome (in all subjects) doesn't have a way to understand the advantage this cohort of students have in german. So they are strongly pulled towards national distribution.

Edit: Note that their achieved percentage of h1 in german is very close to the national teachers assessment of h1s (14%)

 

[SPC100]

I wonder how exam focussed school like the institute will have fared. Based on my skim of this they will be (unfairly?) pulled back towards national distribution.

(I'm assuming that they did outperform in the past, as their marketing/reputation implies)

 

[SPC100]

Duke I think a key thing underspecified in your overview of 2) is that the regression is based on national past performance, and not pupils who have attended that school.

 

[Itchy]

A quick look at the guide DoM posted. Outliers are set aside in assessing the Teacher estimates. There were no outliers in this class as the teachers had a large cluster of high grade students. So when they were put in rank order, the lowest was pulled right down. The outliers are then brought back in to the mix and the gaps between the highest/lowest outlier and the non-outliers were further adjusted. So depending on the performance of the highest and lowest outliers, they would have compressed the grades of the non-outliers group. In reality, these student were the outliers in the national sense.

Isnt there a French Lycee also. I wonder did they have the same experience. Far more data in Irish than in German so probably not best to compare to the Gaelscoileanna.

 

[SPC100]

When I skimmed it was unclear if the school had to note students as outliers or if the department identified them as such based on data.

 

[Duke of Marmalade]

Duke I think a key thing underspecified in your overview of 2) is that the regression is based on national past performance, and not pupils who have attended that school.

Yes, good spot I have edited my post. It is clear that the Junior Cycle Prediction has gone seriously awry for St Kilians probably because it is so ahead of the average. This could expose the hubris in the model yet.

 

[Duke of Marmalade]

A quick look at the guide DoM posted. Outliers are set aside in assessing the Teacher estimates. There were no outliers in this class as the teachers had a large cluster of high grade students. So when they were put in rank order, the lowest was pulled right down. The outliers are then brought back in to the mix and the gaps between the highest/lowest outlier and the non-outliers were further adjusted. So depending on the performance of the highest and lowest outliers, they would have compressed the grades of the non-outliers group. In reality, these student were the outliers in the national sense.

Isnt there a French Lycee also. I wonder did they have the same experience. Far more data in Irish than in German so probably not best to compare to the Gaelscoileanna.

Not sure that outliers explains the St Kilians syndrome. The fact is that the proportion of H1s will be an interpolation of Teachers Assessment and the modelled Junior Cycle Prediction. The Teachers Assessment was 50% so the contribution from JCP must have been hugely less than that to bring the answer down to 14%.
A possible interpolation would be:
JCP result 6% (historical national average) JCP/TA weighting 4:1
That just can’t be right

 

[SPC100]

For this school it looks like it is a high achieving school.

Comparing past results on their blog to National points. It looks like in the past they have 6x more students who get 600 and greater (10p.c vs 1.4) and 3x (39p.c. vs 13) more students who get 500 or greater compared to National distribution.

The german unfairness is very clear to see, as we can identify a trait that predicts higher results. And see the algorithm was blind to this.

But, There is growing noise that this may be more widespread, and it is the high performing schools that are losing out.

Might be we over corrected based on the uk experience.

 

[Duke of Marmalade]

You would have thought that the results from this overly complicated model would have been thoroughly desk checked.
Surely "Historical 41%, Teachers Assessment 50%, Standardised 14%" would jump right off the page".

 

[Sadim]

That is strange. The national position for German was as follows:
6,772 sitting
5.7% get H1s historically
Teachers' Assessment 13.9%
Standardised result 9.0%

If St Killians expected 50% H1s, I interpret that as the Teachers' Assessment.
So they must have got a very significant adjustment from the Junior Cycle Prediction.
But that would suggest that the Junior Cycle Prediction was way below the historic result of 41%, in fact way below the final result of 14%.
Definitely something not hanging together in this picture.

Saw them interviewed on the TV yesterday. Most of them are German natives or first generation Germans whose first tongue at home was in German!

 

[Sadim]

For this school it looks like it is a high achieving school.

Comparing past results on their blog to National points. It looks like in the past they have 6x more students who get 600 and greater (10p.c vs 1.4) and 3x (39p.c. vs 13) more students who get 500 or greater compared to National distribution.

The german unfairness is very clear to see, as we can identify a trait that predicts higher results. And see the algorithm was blind to this.

But, There is growing noise that this may be more widespread, and it is the high performing schools that are losing out.

Might be we over corrected based on the uk experience.

Certainly my daughter was at a high achieving school and it worked out badly for her. Principal said to me informally before Christmas they had a target for her of 550-560, she got 520. By the way, it was not a fee paying school.

 

[SPC100]

You would have thought that the results from this overly complicated model would have been thoroughly desk checked.
Surely "Historical 41%, Teachers Assessment 50%, Standardised 14%" would jump right off the page".

I agree.

Like all good ideas when you see them written down, we say Of course, they should have done that. It's so obvious.

But that doesn't mean it was figured out in advance.

 

[Duke of Marmalade]

I think I now understand how St Kilians were cheated, and they were cheated.
The Junior Cert prediction was a composite of Irish, English, Maths and the next best two subjects per student,
Thus the fact that St Kilians are superb at German was greatly diluted and so their prediction was only slightly above the average and this was applied to all subjects including German. They got very minimal recognition for the fact that they are especially good at German.
In most cases this would not matter, a school gets good or bad results across the board.
This would not have happened if the correct procedure of standardising according to school historical results had been applied, but we were so keen to avoid the UK debacle that we abandoned the correct approach and St Kilians are big losers as a result.

 

[SPC100]

You would have thought that the results from this overly complicated model would have been thoroughly desk checked.
Surely "Historical 41%, Teachers Assessment 50%, Standardised 14%" would jump right off the page".

There are something like 900 schools and maybe 20 subjects, so maybe 20k tuples too look at. But a quick sort by max delta would rise the problematic ones.

Logical possible conclusions
1. Ineptitude - they never examined the data like this (seems unlikely given this is a simple test and the backstory, and all schools will be looking at how much their grades changed)
2. Assuming they did actually do this review, there were two many cases to manually investigate, or there were many discrepancies bigger than this one, so they never got this far.
3. They decided these discrepancies were better than the ones witch included schools historic marks.

 

[SPC100]

The Irish english maths and two best world appear to limit over performance/reduce dynamic range e.g. a 10a1 jc student would have the same uplifting effect on their cohort as a 5a student. But clearly a cohort of students with all As in jc would be expected to outperform one with half as many As.

Also reliance on Irish might pull back schools like this one where presumably many students don't do Irish, or those that do might have less exposure to it, due to foreign background.

Also apparently some 'high point chasing' students strategically choose ordinary Irish knowing they will focus more on their top 6 and Irish will have no effect on their points. If this happens in some cohorts, This would likely result in the model predicting lower than expected outcome.