Jump to content

Relative Importance of Factors


joel hoffman

Recommended Posts

cs123 - I've been with a few agencies, and every one of them stress not using numerical ratings. Apparently sometime in the past there was a protest or something for an acquisition that used numerical ratings, and (as the tale goes) a CO was asked something like, "What is the difference between a 7 and an 8?" Or a 6 and an 8... And the CO could not provide a well thought-out answer, and thus push for non-numerical ratings (whether colors or words).

Link to comment
Share on other sites

As has been described in other threads, there are a couple of major problems with numerical scoring systems.

1) many acquisition teams and selection officials have relied on the scores rather than on the underlying basis for the ratings, which is what is really important and

2) Many teams/officials tend to think that the scores denote some type of precision, as Desparado pointed out.

Another problem that I have witnessed was that rating officials would tend to back into the rating by first assigning a score, then develop a narrative to support their pre-determined rating or just ignore that requirement altogether.

The Army banned numerical weights and numerical scoring systems in 2004. The lead for the committee that revised the AFARS to ban numbers taught a class with me shortly after that and confirmed that the problems that pointed ought were behind the ban. The Air Force was already using colors that corresponded to adjectival ratings rather than numbers. I don't know what Navy was doing.

Link to comment
Share on other sites

Guest Vern Edwards

Joel:

The problems that you describe are user problems, not system problems. If you put poorly educated and trained people to work in source selection you get stupid behavior and poor decisions.

I am sick of reading comments about problems with numerical scoring/weighting in source selection. If you take a college course in decision analysis you will learn numerical methods. There is a reason for that. They work better at what they are supposed to do, which is to simplify complex information and make it more easily digestible. Only the government would forbid the use of a perfectly workable and practically superior system because it cannot figure out how to train its people. Adjectival systems are fine if you have only two or three criteria, say: experience, past performance, and price. Otherwise, a more powerful system better facilitates analysis and integration.

Our workforce is intelligent, but it's pathetic in terms of its professional education and training. To people who think that adjectival systems are better than numerical systems, I say: You're ignorant. You don't understand the purpose of scoring systems. You don't understand what they're supposed to do and how they're supposed to work. Read a bleeping book, dummy. Don't wait for DAU or FAI to teach you. And don't wait for your boss to explain, because if he or she knew they would have explained it to you by now.

To people who want to do better than just get by and who want to be real professionals, I say: Go to Amazon.com and search in books for "decision analysis." Buy one. Read it. Think. Here are some of my favorites:

Skinner, Introduction to Decision Analysis, 3d ed.

Goodwin & Wright, Decision Analysis for Management Judgement, 5th ed. (very, very good)

Keeney & Raiffa, Decisions with Multiple Objectives: Preferences and Value Tradeoffs

Edwards & Von Winterfeldt, Decision Analysis and Behavioral Research (a magnum opus of the highest order, but only for those willing to think deeply)

Keeney, Value-Focused Thinking: A Path to Creative Decisionmaking

If you have a college education, you should be able to understand those, although you may actually have to think hard from time to time.

To everyone reading this, I say: If you're not willing to spend your money and do the hard work of reading and thinking, then go do what you can be good at -- micro-purchases. You'll have a long happy career, and you'll hardly have to think at all. If you want to be a source selection guru, then read, think, learn, and advance. To hell with DAU, FAI, and policy pukes who say numbers are bad.

Link to comment
Share on other sites

Vern and Don, I don't necessarily disagree with you. I didn't personally have problems with numerical scoring system.

However, I observed much ignorance and mis-use of scoring systems. The Army directed all of its organizations to cease use of numerical scoring systems or assigning numerical weights to distinguish relative importance of factors and sub factors in April 2004 . We were also told not to debate or question the Army level decision.

I have recently observed some State Highway Departments using mechanical basis of award selection processes such as "lowest $/point" for their design-build highway projects. Those goofy systems were used by some Corps offices and were discarded in the late 1980's.

Link to comment
Share on other sites

Guest Vern Edwards

Dumb professional behavior must be changed through training or firing. Personally, I prefer training. People are smart enough, but too many agencies are letting amateurs do what is supposed to be done by pros. They just don't know how to do their jobs. Read this decision:

http://wifcon.com/cgen/4086947.pdf

Massive incompetence. Prof. Nash is writing about it in the January 2015 issue of The Nash & Cibinic Report. We were both shocked. This was a big acquisition. First the air tanker source selection and now this. It appears that some parts of the Air Force just can't do source selections. Whoever was in charge of that one should be relieved or reassigned. It took them longer than U.S. involvement in WWII, from December 7, 1941 to V-J Day on September 2, 1945, just to lose a protest decision. One day shy of four years from release of the RFP to the GAO decision. Four years, five if you count the draft RFP, and with what result? The GAO has recommended that they do a new proposal evaluation. Read that decision. Adjectival scoring didn't save those people.

How can something like that happen?

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...