Numerical Scoring In Source Selection: Lessons To Be Learned

[Moderator]

The Nash & Cibinic Report, Volume 36, Issue 12

By Vernon J. Edwards

Addendum by Ralph C. Nash

Numerical (“point”) scoring/rating systems have long been used as a proposal evaluation technique. Once upon a time the use of such systems in the source selection process was common. But after a number of bid protest decisions in the 1970s and 1980s involving numerical scores, some agencies began to restrict or even prohibit their use and mandated the use of adjectival and colorrating schemes instead of numbers.

Please Read:  Numerical Scoring In Source Selection: Lessons To Be Learned

[here_2_help]

Quote

Surely, one of the basic tenets for attorneys advising and representing a Government agency is to prevent the publication of a protest decision that tells the public that the agency is totally incompetent in carrying out its mission of procuring goods and services to meet what often are urgent needs.

Sick burn, bro.

To be clear, the quote above did not come from Mr. Edwards.

[govt2310]

The DoD Source Selection Procedures at 2.3.2.1 prohibited numerical scoring of proposals.  This DAU article says that DoD changed that rule in August 2022, https://www.dau.edu/library/damag/july-aug2024/distinct-numeric-scoring-protocol.  However, when I look at the revised DoD Source Selection Procedures dated August 2022, https://www.acq.osd.mil/dpap/policy/policyvault/USA000740-22-DPC.pdf, it still prohibits numerical scoring of proposals.  It appears the DAU article author is wrong.  Am I missing something?

[formerfed]

I don’t agree with Ralph Nash’s Addendum comment concerning attorneys.  He states he understands how a Contracting Officer including those above him or her might not understand fundamental rules of the competitive negotiation process.  Then he adds he doesn’t see why no Air Force lawyer didn’t see the flaws or stop the procurement is discouraging.  He’s assuming government contract attorneys have the knowledge, expertise and experience to do so across the board.  In my opinion that thinking is flawed in today’s times.  The government has a difficult time recruiting, training and retaining experienced people regardless of their job series.  Some agencies do better than others but his rational seems flawed to me that the Air Force can do better with attorneys than Contracting Officers.

[formerfed]

Wow.  I’m posting to a year old thread!  Maybe too much coffee this morning 😁

[govt2310]

Thanks, formerfed!

 

[shikakenin]

On 9/9/2024 at 10:10 AM, formerfed said:

I don’t agree with Ralph Nash’s Addendum comment concerning attorneys.  He states he understands how a Contracting Officer including those above him or her might not understand fundamental rules of the competitive negotiation process.  Then he adds he doesn’t see why no Air Force lawyer didn’t see the flaws or stop the procurement is discouraging.  He’s assuming government contract attorneys have the knowledge, expertise and experience to do so across the board.  In my opinion that thinking is flawed in today’s times.  The government has a difficult time recruiting, training and retaining experienced people regardless of their job series.  Some agencies do better than others but his rational seems flawed to me that the Air Force can do better with attorneys than Contracting Officers.

I respectfully disagree with your disagree.  RCN was perfectly properly critical!!! 😁  Actually, you do have a point.  Especially if the lawyer is a NEWB to contracting.  Smart Lawyer NEWBS rely on Seasoned Contracting Officers for pragmatic advice (provided said "seasoning" is a fine wine of wisdom and not simply tortured time).   Or, those NEWB Lawyers should simply cruise WIFCON!  I learn a hellava lot from you intellectuals doing occasional drive byes over the years. If I would just read a book every now and then to up me vocabulary, maybe I could debate y'all better.  All I can muster is a little sarcasm with my longshoreman-like dialect.  Gotta have FUN man!  CMON!!! 

Ok, back to Addendum. One thing to add I do disagree with in RCN's Addendum is this: 

"But when they see the agency violating a long-established rule, as in R&K Enterprise, they should go to the head of the agency if necessary to warn of the embarrassment that a published decision would inflict on the agency. That’s what I call giving sound legal advice. RCN"

I don't disagree with the direction, but I surmise the warning is of no consequence.  The "embarrassment infliction principle" is over-rated.   Agencies feel nothing.... 

[RJ_Walther]

 

On 9/9/2024 at 12:13 PM, govt2310 said:

The DoD Source Selection Procedures at 2.3.2.1 prohibited numerical scoring of proposals.  This DAU article says that DoD changed that rule in August 2022, https://www.dau.edu/library/damag/july-aug2024/distinct-numeric-scoring-protocol.  However, when I look at the revised DoD Source Selection Procedures dated August 2022, https://www.acq.osd.mil/dpap/policy/policyvault/USA000740-22-DPC.pdf, it still prohibits numerical scoring of proposals.  It appears the DAU article author is wrong.  Am I missing something?

It appears that the DoD Source Selection Procedures says that quantitative evaluation can be used, and that the relative importance of factors cannot be numerical, but it never explicitly prohibits numerical scoring of factors (although it does mandate the use of a specific set of adjectival ratings, which I guess implicitly prohibits use of any other rating system, including numerical scoring). This does not appear to be a change from the 2016 or 2011 Procedures which have identical language, maybe he is referring to some change in the DFARS, but the one pre-2016 DFARS I looked at didn't mention it. I'll admit I had previously taken the Procedures to mean that numerical scoring was prohibited, until today's close read to try to interpret the article you referenced.

Quote

2.3.2.1. Evaluation Factor/Subfactor Weighting. The evaluation of factors and sub-factors may be quantitative, qualitative, or a combination of both. However, numerical or percentage weighting of the relative importance of evaluation factors and sub-factors shall not be used.

So according to the article if you are in the DoD you can currently claim to be able to quantitatively assess the value of risk and a project plan (two examples from the article) on a unitless ordinal scale of 1 to 100 (I guess with a waiver to not use the mandatory adjectival ratings), but are prohibited by the DoD procedures from saying that the risk assessment is twice as important as the project plan. The author says that if this restriction were lifted it would reduce exposure to procurement fraud, but never explains how moving the obvious subjectivity in assessing risk and a project plan one step earlier in the process (from the trade-off analysis to the assignment of a quantitative value to an obviously qualitative assessment) would make a difference.

I do buy the author's logic that some of the problems with numerical scoring are reduced by using a bigger scale (e.g. 1 to 100 instead of 0 to 4) to allow more differentiation of scores, but in the end the relative weighting of evaluation factors just isn't a very useful concept for source selection (e.g., it is meaningless to say risk is twice as important as price in making a source selection decision) but rather is really useful for letting potential offerors know where to focus their efforts (e.g., offerors can choose to submit a high-risk/low-priced proposal or a low-risk/high priced proposal or something in between, and it improves the process if the government indicates which it would prefer), and I'm not sure that using numerical relative weighting of non-numerical things is any better than a simple narrative description.

[Vern Edwards]

All that need be said about the use of numbers in the presentation of evaluation results was said by Edwards and Von Winterfeldt in the quote in my article, which I quote again below:

Quote

The fundamental principle might be called numerical subjectivity, the idea that subjective judgments are often most useful if expressed as numbers. For reasons we do not fully understand, numerical subjectivity can produce considerable discomfort and resistance among those not used to it. We suspect this is because people are taught in school that numbers are precise, know from experience that judgments are rarely precise, and so hesitate to express judgments in a way that carries an aura of spurious precision. Judgments indeed are seldom precise—but the precision of numbers is illusory. Almost all numbers that describe the physical world, as well as those that describe judgments, are imprecise to some degree. When it is important to do so, one can describe the extent of that imprecision by using more numbers. Very often, quite imprecise numbers can lead to firm and unequivocal conclusions. The advantage of numerical subjectivity is simply that expressing judgments in numerical form makes it easy to use arithmetical tools to aggregate them. The aggregation of various kinds of judgments is the essential step in every meaningful decision.

Bold added.

Numerical scores are just thinking tools. If you say much more them than that you risk looking foolish.

Use numerical scores to help you think, but never justify a source selection decision on the basis of those scores.

Why is it so difficult for policymakers, evaluators, and contracting personnel to understand this simple principle?

It is a wretched failure of public, professional, and personal education and training.

[formerfed]

53 minutes ago, Vern Edwards said:

Numerical scores are just thinking tools. If you say much more them than that you risk looking foolish.

Use numerical scores to help you think, but never justify a source selection decision on the basis of those scores.

Why is it so difficult for policymakers, evaluators, and contracting personnel to understand this simple principle?

It is a wretched failure of public, professional, and personal education and training.

Showing my age here but in the mid-1970s when GSA was the centralized authority for buying ADP, GSA promoted use of a formula based scoring in source selection.  Numerical ratings from technical factors were added together to come up with an overall technical score.  Then proposed prices were assigned points based on a formula where the low priced offeror received maximum points and higher priced offerors received proportionally less.  The offeror with the most points, technical and price combined, was the winner.  

[Vern Edwards]

When competitive negotiated procurement started to become dominant in the 1950s, many agencies used numerical scoring in decision making. The method was widely used by engineers as an analytical technique. The old textbooks about systems engineering included coverage of numerical scoring and weighting techniques in the making of engineering decisions.

What happened was conjunction of two sources of ignorance. The contracting people did not understand the engineer's use of numbers as tools of analysis, and the engineering personnel did not understand the way lawyers think. So the GAO began sustaining protests when contracting officers explained their selection decisions as matters of which offeror had the most "points", which the GAO considered unsound.

Instead of educating and training their personnel in the use of proper methods and terchniques, agency policy makers began discouraging the use of numerical scoring and weighting and encouraging the adoption of adjectival scoring and weighting. The Air Force was the first to ban the use of numerical scoring and weighting, adopting "color coding" instead, which is still use by DOD. Meanwhile, decision theorists like Ward Edwards and Detlof von Winterfeldt still used and taught the use of numbers. All modern textbooks about decisionmaking at least describe the use of numbers. But most people don't read books about decisionmaking.

I'm deeply discouraged by the lack of sound thinking and the lack of quality education and training in our business.

There are books that explain these things, but I think reading has gone out of style.

[formerfed]

Yes, reading seems to be out of style.  The trend now is quick information to solve a problem or answer a question.  Those immediate responses are then used in a situation even if they aren’t correct.

I’m amazed on how many think source selection can be easily learned and applied from just taking a brief course or looking at examples.   

[Vern Edwards]

36 minutes ago, formerfed said:

I’m amazed on how many think source selection can be easily learned and applied from just taking a brief course or looking at examples.

So am I.

Not too long ago I posted this question:

Quote

Here's a question I want you to think about... Given the day and age we live in, if I were to hire [a smart person] straight out of [a good college]... Could that individual develop and learn the [contracting] business in a pure virtual environment?

I was surprised at how many persons, including long-term members of this forum, answered, "Yes."

If you cannot learn source selection in a short course devoted to the topic, how can you learn the contracting business virtually?

Of course, what do we mean by "learn source selection" and "learn the contracting business".

I have been in the contracting business now for more than 50 years. i still don't know all I should know, and I have worked hard at knowing.