Re: Explain this Gobbly Gook

[kber <kber@EARTHLINK.NET>]

bill

while normally you are right to be paranoid, it may not be appropriate
in this case.  Apparently they scaled based on a sample, and when people
complained, decided rather than using a sample (which inevitably has
some margin of error) they were going to use the entire population.
Then there is no sampling error.

Of course, the issue is still -scaled to what?  Are they presuming a
normal distribution, with the mean at some C level, which of course
would mean that at least /3 would be found unsatisfactory.  There is
also the question how one justifies scaling if this is supposed to be a
criterion referenced test - the criterion is supposed to be set before
the test is administered on the full scale , perhaps modified by a
sample application of the questions?

I am not a testing expert - no psychometrician be me.  Nevertheless, I
do so a problem - is the contractor (CTB McGraw Hill) also the developer
of the test?  If so, are they also selling test prep materials for that
test?  If so, then isn't there an inevitable conflict of interest -  if
people are doing well, who the heck needs test prep materials.


Just a random thought.

Ken Bernstein

William Cala wrote:

>   On Sept. 10th our 4th and 8th grade ELA and Math Scores were sent to
> all NY State Schools.  The 4th grade ELA scores miraculously improved
> at the highest level by 35% and the highest 8th grade ELA scores
> decreased by around 11%.  The scores are embargoed at this point.
> Superintendents vehemently complained that results such as these are
> highly improbable and suspect.  We all believe that the scores were
> scaled to make 4th grade look good and 8th look bad.  The commissioner
> has gone on record as saying the increase in dropouts is due to poor
> middle level educationn NOT the tests, ergo new low scores in 8th
> grade to prove his hypothesis.  Well the scores are a bit too out of
> wack for anyone to believe that nonesense so they are scrambling to
> "check some discrepancies". In an urgent fax to all of us today we
> were told that the reporting of the scores are going to be changed.
> This proves to me that you can scale the results to get ANY results
> you wish.  Now, here is the question for all of you on the list
> smarter than I.  What does the following quote I am taking from the
> Deputy commissioner actually say in English.  I AM SERIOUS.   I know
> what is happening, they are changing scaling of the scores so we don't
> go storm their office, but someone put this crap in English for me....
> PLEASE! "One procedure is to compare the scale scores we originally
> developed based on a large sample to those developed based on the
> total state data.  This procedure, conducted with our contractor CTB,
> identified a discrepancy between the results for the intermediate ELA
> test based on the sample data and the actual results based on the
> total statewide performance.  We have confirmed that the first reports
> sent to you for the intermediate ELA test were innacurate, and we will
> release new score reports to the schools" I see this as the ability to
> scale a score to yield the results one desires! Thoughts? Bill
> Cala "Scaling is for Fish"