Vern Edwards

19 minutes ago, joel hoffman said:

Yes. Well said. But not to overlook the basis for the ratings when making comparisons between competitors.

I often saw selection decisions made solely or primarily on numerical scores or solely price/quality (score) or quality/price ratios.

IMO that was assuming too much precision in scoring systems. Both in allocating points among the various factors and subfactors and in assigning points during the evaluation.

Plus it tended to obscure WHERE (which factors/subfactors) the points were assigned to.

I believe that those all were some major reasons for the Army banning numerical rating back in the early 2000’s.

I'm glad you brought that up. The following is from the book Decision Analysis and Behavioral Research, by Ward Edwards (no relation) and Detloff von Winterfeldtn (1986), page 20:

For reasons we do not fully understand, numerical subjectivity can produce considerable discomfort and resistance among those not used to it. We suspect this is because people are taught in school that numbers are precise, know from experience that judgments are rarely precise, and so hesitate to express judgments in a way that carries an aura of spurious precision. Judgments indeed are seldom precise 𑁋 but the precision of numbers is illusory. Almost all numbers that describe the physical world, as well as those that describe judgments , are imprecise to some degree. When it is important to do so, one can describe the extent of that imprecision by using more numbers. Very often quite imprecise numbers can lead to firm and unequivocal conclusions. The advantage of numerical subjectivity is simply that expressing judgments in numerical form makes it easy to use arithmetical tools to aggregate them. The aggregation of variou kinds of judgments is the essential step in every meaningful decision.

Italics added.

Note that in that formulation the numbers do not represent facts, but judgments based on facts.

When I entered government service with the Air Force, source selection teams used numerical scoring. That was abandoned in about 1980 after a number of bid protest decisiond based on faulty use of numerical scores. We went to the adjectival/color raing system still in use today by DOD and some other agencies. But adjectives and colors do not facilitate aggregation of judgements.

Note that the bid protest decisions were not based on the impropriety of using numerical ratings, but on the lack of know-how and the incompetence of the people who used them.

No matter what rating system you use, adjectival, color, or numerical, you must be able to explain the basis for the rating assigned, and you must assign ratings consistently.

The purpose of assigning ratings is simplification of complex information, and when you are evaluating on the basis of multiple factors, numerical ratings provide the greatest degree of simplification.

I have made that point many times in this forum, but only the thinkers seem to get it.

@joel hoffman

50 minutes ago, joel hoffman said:

It’s essentially a roll up summary level description or label for the supporting documentation.

So could I say that you used ratings to summarize and simplify more complex information?

3 hours ago, joel hoffman said:

I always made the SSEB team develop and provide the above (the MEAT of the evaluation) during the CONSENSUS evaluation, and THEN we assigned the appropriate, corresponding “rating”, again by consensus.

@joel hoffman

Why did you assign ratings to the MEAT of the evaluation? What did the ratings do for you that the MEAT didn't do??

Don, This blog post is incomplete.

You say why you wouldn't use ratings, but you don't say why you would.

Why would you use ratings, if ever, and what kind of ratings would you use?

And if you would ever use ratings, how would you use them?