Jump to content

Justifying award to higher priced


Freyr

Recommended Posts

1 hour ago, Freyr said:

@Don Mansfieldreading what you just put logically makes sense but why then do COs always put "is significantly more important" when what it really comes down to is the basic question of whether or not the technical superiority is worth the price? Is it just to comply with the FAR? Is it just a clumsy way of saying "we'll pay more for a better technical solution if it's worth it"? 

COs are probably doing that to comply with FAR. What do those COs do when including a statement of relative importance is not required (e.g., SAP, FSS ordering, orders under multiple-award contracts < $6 million)?

Link to comment
Share on other sites

On 7/16/2021 at 9:17 PM, Freyr said:

Every evaluation I've seen starts, as the FAR requires, with an eval against started criteria. Why doesn't the FAR allow us to just cut the chase and compare each proposal to one another? 

There is no prohibition in FAR against evaluating proposals by direct comparison. If you are familiar with decision analysis techniques, you know that evaluation by direct comparison can be a valid method and that there are suitable ways to do that.

The idea that agencies may not evaluate proposals on the basis of direct comparisons dates back to agency source selection regulations, especially those for research and development, that required evaluation of proposals on the basis of comparisons to agency established standards, rather than comparison to each other. See, e.g., Air Force Regulation 70-15, 16 April 1976, p. A-37:

Quote

The evaluation of proposals is conducted at the factor level under each. item and compared against measurable objective standards, and not by comparing proposals against each other... All standards must be prepared before proposals are received.

Proposals were evaluated and rated on the basis of whether they met, failed to meet, or beneficially exceeded the standards.

In other words, to refer to ji20874's example, if the pillow evaluation factor was to be softness, then the agency had to have a scale and standard of softness, and each proposal would be rated (excellent, good, fair, etc.) on the basis of how soft its pillow is compared to the standard. However, another way to evaluate offerors would be to determine how soft each competing pillow is and rank the offerors by direct comparison, with the softest pillow getting the highest rating and the others getting proportionately lower ratings.

The prohibition against evaluation by direct comparison was intended to ensure that the winner met the agency requirement for softness. In theory, evaluation by direct comparison could result in a winner that is best among the competitors, but that does not meet the agency's needs.

To the best of my  recollection, FAR has never prohibited evaluation by direct  comparison. Today, FAR 15.305 states:

Quote

Evaluations may be conducted using any rating method or combination of methods, including color or adjectival ratings, numerical weights, and ordinal rankings. 

Some agency-specific policies or procedures may still prohibit evaluation by direct comparison, but others do not. See, for example, the Department of Commerce Acquisition Manual, Section 5, Source Selection Process, para. 5.2.3:

Quote

5.2.3 Evaluation Standards To assist evaluators in determining the relative merit of each proposal with respect to the evaluation factors, evaluation standards/descriptions may be developed to allow for the evaluation of proposals against a uniform baseline rather than against each other. The use of evaluation standards promote consistency and potentially minimize bias that can result from direct comparison of proposals.

Although optional, the use of evaluation standards is strongly encouraged as they further detail the relationship between the statement of work and evaluation factors. Depending on the nature of the acquisition, the evaluation standard may either be quantitative, where an element can be defined in a readily measurable form (i.e. speed, accuracy rate, etc.,) or qualitative, where attributes or questions are used to assist in assessing whether the proposal meets the requirements of the solicitation. When used, evaluation standards shall be developed by the SSEB and reviewed by the Contracting Officer.

Like Freyr, many in the workforce think that FAR prohibits evaluation by direct comparison. It doesn't, but there are pros and cons.

If it were up to me, only specially trained contracting officers would be permitted to conduct source selections. It could save the taxpayers millions.

Link to comment
Share on other sites

Many agencies have operational policies saying the evaluation teams must only evaluate proposals using the stated standards established for factors and not make comparisons.  Then comparison occur after evaluation in source selection. 
 

Edit:  I didn’t see Verns post when I made mine.  It wasn’t there.  Strange but the times are two hours apart.

Link to comment
Share on other sites

1 hour ago, formerfed said:

I didn’t see Verns post when I made mine.

To which "Vern's post" are you referring? I have made several.

1 hour ago, formerfed said:

Many agencies have operational policies saying the evaluation teams must only evaluate proposals using the stated factors and not make comparisons.

Emphasis added.

The issue is not evaluation against the stated factors versus comparisons. Offerors and their offers must always be evaluated against the specified factors. The issue is the method of factor evaluation.

Factor Evaluation by Direct Comparison

A panel lays their heads on each pillow and subjectively decides which is the softest. The one identified as the softest is given 100 points. The second softest is given fewer points, say 80, assigned subjectively by comparison with the softest. The third gets even fewer points, say 60, again based on subjective comparison with the softest. And so on.

Factor Evaluation by Comparison to a Common Standard

A panel is given a descriptive scale of softness, such as the five point scale used by US Mattress: 1-firm, 2-medium firm, 3-medium, 4-medium soft, 5-soft. (The categories come with descriptions.) They lay their heads on each pillow and assign it a rating based on the descriptive scale, not based on direct comparison.

In both cases the pillows are rated "using the stated factors"; but in each case the ratings are assigned based by a different method. 

 

Link to comment
Share on other sites

19 minutes ago, Vern Edwards said:

To which "Vern's post" are you referring? I have made several.

I left out wording in my prior post.  I went back and edited it.  What I meant is some agency policies say use standards for factors rather than comparison.  

After your explanation, I didn’t really need to say more.

Link to comment
Share on other sites

3 hours ago, Vern Edwards said:

Factor Evaluation by Direct Comparison

A panel lays their heads on each pillow and subjectively decides which is the softest. The one identified as the softest is given 100 points. The second softest is given fewer points, say 80, assigned subjectively by comparison with the softest. The third gets even fewer points, say 60, again based on subjective comparison with the softest. And so on.

Factor Evaluation by Comparison to a Common Standard

A panel is given a descriptive scale of softness, such as the five point scale used by US Mattress: 1-firm, 2-medium firm, 3-medium, 4-medium soft, 5-soft. (The categories come with descriptions.) They lay their heads on each pillow and assign it a rating based on the descriptive scale, not based on direct comparison.

To be defensible in court, does the Common Standard need to be written in detail in the Source Selection Plan, or is it advisable that the SEB stick to the templated, counsel-approved, vague enough standards instead, e.g., "'Strength' means Likely to lead to successful contract performance blah blah".  I ask because an SEB is currently tearing its hair out trying to develop standards to implement those definitions against proposal content - during the evaluation.  Perhaps Section L bit off more than it could chew with no clear plan of what quantitative or qualitative standard its drafters might use.

Link to comment
Share on other sites

33 minutes ago, WifWaf said:

To be defensible in court, does the Common Standard need to be written in detail in the Source Selection Plan...

I am not aware of any such requirement. Moreover, FAR Part 15 makes no mention of source selection plans. The requirement for an SSP is an agency level requirement.

Moreover both the GAO and COFC have said that failure to adhere to a source selection plan is not grounds for sustaining a protest. See Epsilon Systems Solutions, Inc., GAO B-409720, 2014 CPD ¶ 230, July 21, 2014:

Quote

Source selection plans provide internal agency guidelines and, as such, do not give outside parties any rights. E.g., Walsh Investors, LLC, B–407717, B–407717.2, Jan. 28, 2013, 2013 CPD ¶57 at 8 n.6; Mid Pacific Envtl., B–283309.2, Jan. 10, 2000, 2000 CPD ¶40 at 6. It is the evaluation scheme in the RFP, not internal agency documents, such as the source selection plan, to which an agency is required to adhere in evaluating proposals and in making the source selection. All Points Logistics, Inc., B–407273.53, June 10, 2014, 2014 CPD ¶174 at 10 n.10; Meadowgate Techs., LLC, B–405989, B–405989.3, Jan. 17, 2012, 2012 CPD ¶27 at 6 n.7; Synergetics, Inc., B–299904, Sept. 14, 2007, 2007 CPD ¶168 at 8.

The position of the COFC is generally along the same lines, but not so unequivocal. See "Postscript II: Source Selection Plans," The Nash & Cibinic Report, (Sept, 2010), which discusses the somewhat inconsistence stances of the judges at the court.

33 minutes ago, WifWaf said:

Perhaps Section L bit off more than it could chew with no clear plan of what quantitative or qualitative standard its drafters might use.

Perhaps the CO and the SSEB have not been properly educated and trained by their superiors. Perhaps they haven't bothered to educate and train themselves. Perhaps for one or both of those reasons they are not qualified to conduct or participate in a source selection.

Link to comment
Share on other sites

4 hours ago, Vern Edwards said:

In both cases the pillows are rated "using the stated factors"; but in each case the ratings are assigned based by a different method. 

And there are other possible methods not using point scores or adjectival ratings -- for example, direct comparison and rank ordering such as A is soft and B is softer than A.

I suppose some readers are reading thoughts in this thread that they have never imagined.

Here's maybe another one:  A source selection evaluation need not be based on identifying strengths, weaknesses, significant weaknesses, and deficiencies in proposals -- it is not required to evaluate specifically for these things, and yet so many evaluation plans do this.  And then assigning ratings based on counting these adds stupidity to stupidity (speaking broadly).  Yes, the FAR mentions them, but it does not tell us to evaluate for them.  We should evaluate for quality or other attributes, and then the strengths, weaknesses, significant weaknesses, and deficiencies will become self-apparent without specifically looking for them -- and we then use these, or at least the significant weaknesses and deficiencies, to help shape discussions and in debriefings (but not for the evaluation).

Link to comment
Share on other sites

So many acquisition professionals carry around so much baggage in their minds -- I hope this exchange will help some of our readers shake off their errant and encrusted baggage.  Learning and honoring simple and correct principles is important.

Link to comment
Share on other sites

47 minutes ago, ji20874 said:

A source selection evaluation need not be based on identifying strengths, weaknesses, significant weaknesses, and deficiencies in proposals -- it is not required to evaluate specifically for these things, and yet so many evaluation plans do this. 

@ji20874I agree. In the June 2018 edition of The Nash & Cibinic Report, I wrote:

Quote

 

What is the purpose of identifying parts of a proposal as strengths, weaknesses, or deficiencies? It strikes us that if the evaluation factors are well established, as described above, then it is unnecessary to add the intermediate step of labeling particular statements or sets of statements as “strengths,” “weaknesses,” or “deficiencies.” Such ratings—while perhaps helpful to a Source Selection Authority in setting the stage for detailed consideration of the proposals based on the proposal facts and evaluation findings—become one more thing to explain, justify, document, and argue about if cited as bases for the source selection decision. And we think that the bid protest decisions we cited above, and the many others like it, prove our point.

Why be inefficient? Why get bogged down in intermediate ratings that are not essential to decisionmaking? Why not just reason from evaluation factor definitions and standards (major premises), through particular proposal facts (minor premises), to evaluation findings about proposal value? We realize that FAR 15.305(a) states: “The relative strengths, deficiencies, significant weaknesses, and risks supporting proposal evaluation shall be documented in the contract file.” But we do not read that sentence as requiring the rating of proposal content as strengths, deficiencies, and weaknesses. Our interpretation of that sentence is that if a source selection team chooses to apply such ratings it must explain (“document”) them.

 

Unfortunately, the DOD Source Selection Procedures (April 2016) effectively require that agencies make those findings in acquisitions to which the procedures apply:

Quote

These procedures are applicable to all acquisitions conducted as part of a major system acquisition program, as defined in Federal Acquisition Regulation (FAR) 2.101, and all competitively negotiated FAR part 15 acquisitions with an estimated value greater than $10 million.

However, waivers may be requested.

Link to comment
Share on other sites

On 7/15/2021 at 10:14 PM, Don Mansfield said:

@ji20874's pillow example is great. It illustrates that a tradeoff decision is a matter of judgment. The only thing I'd add is that when he decides whether to pay the extra $2 or not, what he said in his solicitation about the relative importance of price and softness is irrelevant.

After Don posted that, I told him that while I agreed with him, the statement that relative importance is "irrelevant" to the tradeoff decision requires an explanation. He responded on 7/16 at 06:10 pm in part as follows:

On 7/16/2021 at 6:10 PM, Don Mansfield said:

Fair enough. Assume ji's pillow solicitation stated that softness was significantly more important than price, but he decides Pillow A is a better value. He trades off extra softness for a lower price. The offeror of Pillow B protests because their pillow was softer, the solicitation said softness was significantly more important than price, and the difference in price is only $2.

In deciding such protests, the GAO has historically focused on whether the tradeoff decision is rational--not whether enough emphasis was placed on a particular factor in making the tradeoff decision. 

In "Relative Importance of Evaluation Factors: Much Ado About Nothing, 18 N&CR ¶ 30", Professor Nash analyzed several protest decisions where the protestor unsuccessfully tried to make an issue of the statement of relative importance not being consistent with the tradeoff decision.

While  I agree with what Don has said, I still think we need a rationale for saying that the relative importance of evaluation factors is "irrelevant"  to the tradeoff decision. If GAO has ruled that way—and it has, although it has not expressly said "irrelevant—we need to understand why. I think that as professionals we can't just say that the GAO has consistently ruled that way. I think we must understand why it has ruled that way, so we can explain it to others. And I don't think we should cite an article by Professor Nash that we can't find.

Link to comment
Share on other sites

On 7/17/2021 at 12:17 AM, Freyr said:

why then do COs always put "is significantly more important" when what it really comes down to is the basic question of whether or not the technical superiority is worth the price? Is it just to comply with the FAR?

I think it's worth considering that offerors make choices in their proposals, and if the government can tell them what they value the most it can help them make the choices needed to submit offers that the government wants. For example, if I want to buy a house I don't just tell the realtor "show me every house with at least three bedrooms" and then try to sort through the pile, I might instead say "A low price is most important to me, show me low-priced three bedroom houses" or alternatively "I want as much room as possible and I'm willing to pay for it" and now the houses the realtor shows me are more likely to be what I'm looking for, and I should have some similar houses to compare. When it comes down to choosing which house to buy I'll still decide whether the extra money is worth the extra space in the end, but I likely won't be deciding between a tiny foreclosure and a million-dollar mansion.

Same thing when you buy a car, you don't go to a Ford dealer and drive a Focus, a Mustang, and an F-150; you tell them you are interested in a economy, or speed, or towing capacity, and then you drive some cars that meet your preferred criteria. You still make tradeoffs (is the additional towing capacity of an F-250 worth the price compared to an F-150) but they are showing you the vehicles that they think are most in line with what you are looking for.

In the same way, if a solicitation says past performance is the most important factor and non-price factors are significantly more important than price potential offerors may decide to invest in a teaming arrangement with someone currently performing similar work, even if it is more expensive, rather than leaning their process to try to propose the lowest price they can. Now, the government will (ideally) get several proposals from offerors with solid past performance to make tradeoffs among, rather than a random collection of high and low priced offerors with widely varying past performance. 

 

Link to comment
Share on other sites

I wonder if GAO went a bit too far in B-416076 EFS Ebrex SARL saying that that 80% unsated evaluation factor was not proper, minus the misleading discussions part, I agree that discussions were misleading. 

But I kind of disagree with the fact that it was not proper for the agency to use the 80% cut-off for Acceptable even if it was not stated in the solicitation that that would happen in the evaluation stage. Perhaps not making it a cut-off and instead making it part of the subjective analysis that Ebrex just deserves an Acceptable and not a Good might have worked. As it relates to this thread, the agency did state in their RFP that "those aspects of relevancy include experience performing deliveries as a full line food service distributor, dollar value, and number of customers." FAR 15.304(d) says "[t]he rating method need not be disclosed in the solicitation" doesn't it?

Apart from that if the agency wanted to enter into discussions with Coastal they should have eliminated Ebrex from the competitive range because Ebrex didn't have a chance of winning even if discussions were to be held with them anyways, eliminating the "discussions were not meaningful" problem.

Link to comment
Share on other sites

On 7/19/2021 at 7:10 AM, Vern Edwards said:

After Don posted that, I told him that while I agreed with him, the statement that relative importance is "irrelevant" to the tradeoff decision requires an explanation. He responded on 7/16 at 06:10 pm in part as follows:

While  I agree with what Don has said, I still think we need a rationale for saying that the relative importance of evaluation factors is "irrelevant"  to the tradeoff decision. If GAO has ruled that way—and it has, although it has not expressly said "irrelevant—we need to understand why. I think that as professionals we can't just say that the GAO has consistently ruled that way. I think we must understand why it has ruled that way, so we can explain it to others. And I don't think we should cite an article by Professor Nash that we can't find.

The statement of relative importance is commonly misunderstood to mean the relative importance the factors will have when making a tradeoff. The problem with this belief is that the factors that will be most important when making a tradeoff is unknowable until after proposals have been evaluated. Although the solicitation may state something like "nonprice factors are significantly more important than price", the evaluation could result in proposals that are equally valuable on nonprice factors--in which case price or cost would become the most significant factor when making the tradeoff decision.

The statement of relative importance communicates the importance of factors in meeting the Government's acquisition objectives. It is supposed to help offerors in preparing responses to solicitations. (See @Witty_Username's post above).

Link to comment
Share on other sites

On 7/20/2021 at 1:57 PM, Don Mansfield said:

The statement of relative importance is commonly misunderstood to mean the relative importance the factors will have when making a tradeoff. The problem with this belief is that the factors that will be most important when making a tradeoff is unknowable until after proposals have been evaluated. Although the solicitation may state something like "nonprice factors are significantly more important than price", the evaluation could result in proposals that are equally valuable on nonprice factors--in which case price or cost would become the most significant factor when making the tradeoff decision.

The statement of relative importance communicates the importance of factors in meeting the Government's acquisition objectives. It is supposed to help offerors in preparing responses to solicitations. (See @Witty_Username's post above).

I used to add the comment to the effect that, as differences between non-price factors become minimal, differences between prices become more important and as differences between prices become minimal, differences between non- price factors become more important.

I think that the statement that “price is equal to the non-price factors, when combined” is somewhat misleading. What it means to industry is that price is the most important factor, if it is equal to all of the non-price factors combined. 

Link to comment
Share on other sites

Benjamin Franklin is considered by many to be the founder of decision analysis. Here is the text of his letter to his friend Joseph Priestly about how to make important decisions, dated Dec. 19, 1772. Note the mention of "weights" and the discussion of their application.

Spelling and punctuation as in the original.

To Joseph Priestley

London, September 19, 1772

Quote

 

Dear Sir,

In the Affair of so much Importance to you, wherein you ask my Advice, I cannot for want of sufficient Premises, advise you what to determine, but if you please I will tell you how.

When these difficult Cases occur, they are difficult chiefly because while we have them under Consideration all the Reasons pro and con are not present to the Mind at the same time; but sometimes one Set present themselves, and at other times another, the first being out of Sight. Hence the various Purposes or Inclinations that alternately prevail, and the Uncertainty that perplexes us.

To get over this, my Way is, to divide half a Sheet of Paper by a Line into two Columns, writing over the one Pro, and over the other Con. Then during three or four Days Consideration I put down under the different Heads short Hints of the different Motives that at different Times occur to me for or against the Measure. When I have thus got them all together in one View, I endeavour to estimate their respective Weights; and where I find two, one on each side, that seem equal, I strike them both out: If I find a Reason pro equal to some two Reasons con, I strike out the three. If I judge some two Reasons con equal to some three Reasons pro, I strike out the five; and thus proceeding I find at length where the Ballance lies; and if after a Day or two of farther Consideration nothing new that is of Importance occurs on either side, I come to a Determination accordingly.

And tho' the Weight of Reasons cannot be taken with the Precision of Algebraic Quantities, yet when each is thus considered separately and comparatively, and the whole lies before me, I think I can judge better, and am less likely to take a rash Step; and in fact I have found great Advantage from this kind of Equation, in what may be called Moral or Prudential Algebra.

Wishing sincerely that you may determine for the best, I am ever, my dear Friend,

Yours most affectionately

B. Franklin

 

You could use that method to select a contractor, citing the advice of one of the greatest of the Founders.

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...