Jump to content

Myth-Information:

You have to rate proposals in a source selection.

When discussing the evaluation of competitive proposals with my students, I make a point of asking the following two questions (in order):

1. Are agencies required to evaluate proposals?

2. Are agencies required to rate proposals?

Usually, students respond affirmatively to question #1 and are able to support their answers by citing FAR 15.305(a), which states "An agency shall evaluate competitive proposals and then assess their relative qualities solely on the factors and subfactors specified in the solicitation." However, confusion sets in when I follow with question #2 and students read the very next sentence of FAR 15.305(a), which states "Evaluations may be conducted using any rating method or combination of methods, including color or adjectival ratings, numerical weights, and ordinal rankings." Clearly, the language regarding use of a rating method in conjunction with an evaluation is permissive, not mandatory.

"What's the difference?", "Why wouldn't you rate proposals?", "How do you decide who is the better value if you don't rate the proposals?" are typical student responses. These are all good questions.

Evaluation v. Rating

A good way to understand the difference between evaluation and rating is to look at a typical article in Consumer Reports (CR). Here?s an example of a summary evaluation of a new car?s ?Driving Experience? (model name omitted):

The ride is steady and composed. It absorbs bumps smoothly but is firm. Road noise is reduced, but the tires still rumble noticeably and slap over pavement joints. Routine handling is responsive and fairly agile. Body lean is suppressed, and the quick steering has good weight and feedback. It displayed good grip and balance in emergency maneuvers, and its standard electronic stability control is well calibrated. The [car] posted a commendable speed in our avoidance maneuver. The smooth 166-hp, 2.4-liter, four-cylinder engine provides adequate acceleration. The five-speed automatic transmission is very smooth and responsive. We measured 21 mpg overall on regular fuel. The all-wheel-drive system sends power to the rear wheels when needed more quickly than in the previous [model]. The brakes provided short, straight stops on wet and dry pavement. Low-beam headlights reached only a fair distance, and high beams reached a good distance.

?Driving Experience? was one evaluation factor under the heading ?Road Test.? CR also evaluated ?Reliability?, ?Safety?, and ?Owner Satisfaction?, to name a few. According to the Web site, there were over 50 different tests and evaluations performed on the car. Presumably, this produced a mountain of data. However, the typical car buyer does not have the time to peruse the data, nor do they fully understand it. As such, CR established a 100-point scale and a set of predetermined criteria to translate test and evaluation results into scores on the scale. In addition, they partitioned the scale into quintiles and assigned an adjective to each (Poor, Fair, Good, Very Good, and Excellent). Using this rating method, the car described above received a score of 74 and an adjectival rating of ?Very Good.? In this case, CR used a combination of rating methods (numerical scoring and adjectival rating) to translate complex evaluation results into an easily consumable format for its readers.

But Teach, Why Wouldn?t you Rate Proposals?

First, it's not required. Besides that, the results of the evaluation may not be particularly complex. For example, let?s say I used price and performance risk as my evaluation factors in a source selection. Performance risk had two subfactors?past performance and experience. In the solicitation, I instructed offerors to submit a one-page write-up and customer point of contact for each of their relevant contracts. The evaluation of performance risk consisted of an assessment of the write-ups as well as interviews with the customer points of contact to validate the offeror?s claimed experience as well as to ascertain how well the offeror performed. The evaluators then wrote an evaluation of each offeror?s performance risk, documenting the relative strengths and weaknesses of each. Why would it be necessary to translate this information into a rating? How would this aid my decision-making? I?m not going to be faced with volumes of information.

Another reason I would avoid the use of ratings is when I was dealing with evaluators that didn?t understand them. In my experience, when ratings are used, ratings are all you get. I can recall receiving technical evaluations that had nothing more than the word ?Excellent? (when I used adjectival ratings) or ?95? (when I used a numerical rating). I wanted an evaluation and I got a rating.

How do you decide who is the better value if you don't rate the proposals?

The answer is the same way that you would if you did rate proposals?by performing a comparative assessment of proposals against all source selection criteria in the solicitation. A source selection authority (SSA) relies on ratings to make their source selection decision at their peril. See, for example, Si-Nor, Inc., B-282064, 25 May 1999, where the source selection authority based her decision to award to a higher-priced offeror on the fact that the offeror had a higher past performance rating. One of the reasons the protest was sustained was because the SSA did not describe the benefits associated with the additional costs, as required by FAR 15.308. ?Because they had a higher rating? will typically fail to meet this requirement.

So we shouldn?t use ratings?

Not necessarily. The point is that you have discretion to use or not use ratings. Most people don?t know why they use ratings other than the fact that it?s traditional where they work. The decision to use (or not use) ratings should result from thoughtful deliberation, not a successful copy and paste from your office mate?s old source selection plan. A wise man once said ?Tradition is the hobgoblin of mediocre minds.?

18 Comments

Recommended Comments

jljordan

Members

comment_31

Don,

Interesting points that you pose. They tie in nicely with Mr. Edwards writings of simplifying the evaluation process. I am going to present this paper (along with your other myth debunkers and Mr. Edwards thoughts) to my office. I hope that the ideas are embraced and (gasp!!!) even put to use in a few solicitations to see how they work out.

~ JJ

dgm

Members

comment_32

I love these myth buster blog entries you are doing. They are very useful both in improving my understanding and my ability to better communicate and explain things to some of the program offices I work with.

Guest RU.Sure?

Guest RU.Sure?

Guests

comment_1984

"(a) Proposal evaluation is an assessment of the proposal and the offeror’s ability to perform the prospective contract successfully. An agency shall evaluate competitive proposals and then assess their relative qualities solely on the factors and subfactors specified in the solicitation. Evaluations may be conducted using any rating method or combination of methods, including color or adjectival ratings, numerical weights, and ordinal rankings. The relative strengths, deficiencies, significant weaknesses, and risks supporting proposal evaluation shall be documented in the contract file."

Your interpretation of this language is not entirely unreasonable, but I do not think ratings are "clearly" permissive and not mandatory. An agency shall evaluate proposals. Then the next sentence describes a presumptive means by which agency [shall?] do so: ratings. The permissive aspect, as I read this, applies to the type of ratings used in evaluation, not whether ratings have to be used at all.

If that were the case, I would expect that sentence to caveat 'if ratings are used' in some way. It clearly does not, thus implies ratings will be used, but the type of rating is discretionary. In other words, it says "may be conducted using any rating method or combination of methods...". It does not say "may be conducted using any or no rating method or combination of methods...".

joel hoffman

Members

(edited)

comment_1995

“An agency shall evaluate competitive proposals and then assess “their relative qualities” solely on the factors and subfactors specified in the solicitation."

Does assess “their relative qualities” mean

“compare the quality between the different proposals” - solely based upon the specified factors and subfactors ?

Oxford Dictionary: Relative as an adjective:

…”considered in relation or in proportion to something else.”

  1. "the relative effectiveness of the various mechanisms is not known"

—-—————————-

Or does it mean “assess their [each proposal’s] relative qualities” under the specified factors and subfactors using some type of rating system [e.g., is it relatively weak, meets, exceeds, excels under the factor or subfactor quality requirements]?

In other words what is the contextual meaning of “their” [singular or plural] and “relative” [compared to each other or compared to the evaluation criteria]?

Edited by joel hoffman

Don, This blog post is incomplete.

You say why you wouldn't use ratings, but you don't say why you would.

Why would you use ratings, if ever, and what kind of ratings would you use?

And if you would ever use ratings, how would you use them?

On 4/3/2025 at 1:26 PM, Guest RU.Sure? said:

The relative strengths, deficiencies, significant weaknesses, and risks supporting proposal evaluation shall be documented in the contract file."

Don, the assigned ratings must be supported by documenting “the relative strengths, deficiencies, significant weaknesses, and risks resulting from the proposal evaluation. I always made the SSEB team develop and provide the above (the MEAT of the evaluation) during the CONSENSUS evaluation, and THEN we assigned the appropriate, corresponding “rating”, again by consensus. Non-numerical ratings naturally fall out based upon the supporting data.

The underlying, supporting data were also used in the comparative analysis and cost-technical trade-off when applicable.

3 hours ago, joel hoffman said:

I always made the SSEB team develop and provide the above (the MEAT of the evaluation) during the CONSENSUS evaluation, and THEN we assigned the appropriate, corresponding “rating”, again by consensus.

@joel hoffman

Why did you assign ratings to the MEAT of the evaluation? What did the ratings do for you that the MEAT didn't do??

1 hour ago, Vern Edwards said:

@joel hoffman

Why did you assign ratings to the MEAT of the evaluation? What did the ratings do for you that the MEAT didn't do??

It’s essentially a roll up summary level description or label for the supporting documentation. Why would one write down every detail supporting the ratings in a matrix of all the proposals? When comparing proposals, one can start at the rating level but then would distinguish by examining the underlying details.

@joel hoffman

50 minutes ago, joel hoffman said:

It’s essentially a roll up summary level description or label for the supporting documentation.

So could I say that you used ratings to summarize and simplify more complex information?

joel hoffman

Members

(edited)

comment_2008
6 hours ago, Vern Edwards said:

@joel hoffman

So could I say that you used ratings to summarize and simplify more complex information?

Yes. Well said. It’s to avoid not being able to see the forest for the trees or however that idiom goes.

But we must not overlook the basis for the ratings when making comparisons between competitors.

I often saw selection decisions made solely or primarily on numerical scores or solely price/quality (score) or quality/price ratios.

IMO that was assuming too much precision in scoring systems. Both in allocating points among the various factors and subfactors and in assigning points during the evaluation.

Plus it tended to obscure WHERE (which factors/subfactors) the points were assigned to.

I believe that those all were some major reasons for the Army banning numerical rating back in the early 2000’s.

Edited by joel hoffman

joel hoffman

Members

(edited)

comment_2009

I once taught a design-build construction class in Portland, Oregon with a Contracting Officer who was involved in that decision, when she worked at Army level.

I remember her confirming those reasons being involved in the decision. She said there were numerous protests involving scoring schemes and that the agencies tended to overlook real differences between proposals,, instead relying on scores. Plus Adjectival rating systems allowed more subjectivity in the selection decisions.

I’m not sure that many or all of the evaluation teams and contracting personnel have learned to justify the basis (the meat ) first, before assigning ratings or tend to assign ratings first, then find ways to justify the rating..

Edited by joel hoffman

19 minutes ago, joel hoffman said:

Yes. Well said. But not to overlook the basis for the ratings when making comparisons between competitors.

I often saw selection decisions made solely or primarily on numerical scores or solely price/quality (score) or quality/price ratios.

IMO that was assuming too much precision in scoring systems. Both in allocating points among the various factors and subfactors and in assigning points during the evaluation.

Plus it tended to obscure WHERE (which factors/subfactors) the points were assigned to.

I believe that those all were some major reasons for the Army banning numerical rating back in the early 2000’s.

I'm glad you brought that up. The following is from the book Decision Analysis and Behavioral Research, by Ward Edwards (no relation) and Detloff von Winterfeldtn (1986), page 20:

For reasons we do not fully understand, numerical subjectivity can produce considerable discomfort and resistance among those not used to it. We suspect this is because people are taught in school that numbers are precise, know from experience that judgments are rarely precise, and so hesitate to express judgments in a way that carries an aura of spurious precision. Judgments indeed are seldom precise 𑁋 but the precision of numbers is illusory. Almost all numbers that describe the physical world, as well as those that describe judgments , are imprecise to some degree. When it is important to do so, one can describe the extent of that imprecision by using more numbers. Very often quite imprecise numbers can lead to firm and unequivocal conclusions. The advantage of numerical subjectivity is simply that expressing judgments in numerical form makes it easy to use arithmetical tools to aggregate them. The aggregation of variou kinds of judgments is the essential step in every meaningful decision.

Italics added.

Note that in that formulation the numbers do not represent facts, but judgments based on facts.

When I entered government service with the Air Force, source selection teams used numerical scoring. That was abandoned in about 1980 after a number of bid protest decisiond based on faulty use of numerical scores. We went to the adjectival/color raing system still in use today by DOD and some other agencies. But adjectives and colors do not facilitate aggregation of judgements.

Note that the bid protest decisions were not based on the impropriety of using numerical ratings, but on the lack of know-how and the incompetence of the people who used them.

No matter what rating system you use, adjectival, color, or numerical, you must be able to explain the basis for the rating assigned, and you must assign ratings consistently.

The purpose of assigning ratings is simplification of complex information, and when you are evaluating on the basis of multiple factors, numerical ratings provide the greatest degree of simplification.

I have made that point many times in this forum, but only the thinkers seem to get it.

Thank you, Vern. I was personally able to successfully use the numerical system in source selections but observed and noted the problems mentioned herein at other Districts and agencies.

formerfed

Members

comment_2014

We had an interesting experience in my last government job doing a source selection without use of ratings. The chief issue was getting the evaluation team to focus and succinctly state relative strengths, weaknesses, etc. of each offeror. From experiences they wanted to start with assigning a rating and work backwards with supporting narrative. The contracting officer spent a lot of time going over the evaluator findings and actually worked full time with them to ensure what was produced by the evaluators was what was needed.

Two unusual reactions. The lawyer reviewing the files asked where the ratings were? The CO had to walk him through the process. The Source Selection Official was very pleased and said it made his job very easy.

Guest Lori Shultz

Guest Lori Shultz

Guests

comment_2021
Hi. We run a YouTube growth service, which increases your number of subscribers both safety and practically. - We guarantee to gain you new 700+ subscribers per month - People subscribe because they are interested in your videos/channel, increasing video likes, comments and interaction. - All actions are made manually by our team. We do not use any bots. The price is just $60 (USD) per month, and we can start immediately. If you are interested and would like to see some of our previous work, let me know and we can discuss further. Kind Regards, To Unsubscribe, reply with the word unsubscribe in the subject.
Guest Albert Das

Guest Albert Das

Guests

comment_2022
We are currently seeking companies like yours for a potential long-term partnership. Could you kindly share your product offerings along with pricing details? Please contact me on WhatsApp: +44 758 789 0515
Guest Claire Willson

Guest Claire Willson

Guests

comment_2036
Hi, I hope this message finds you well. I’m reaching out because I have a keen interest in contributing a guest post to your website. As a seasoned writer, I specialize in creating content that drives traffic and captivates readers. I’d be happy to provide a guest post, tailored specifically to your audience’s interests and needs. If you’re open to this idea, I can share some topic suggestions that I think would resonate well with your readers. Looking forward to your response. Best, Claire.
Guest
Add a comment...