I'm curious to know whether anyone has successfully done, or knows of anyone who has successfully done, a comparative evaluation pursuant to 13.106-2(b)(3). Specifically, I'm curious about the following:

• What did/would this look like in practice? For instance, beyond the evaluation factors themselves, would I need to specify the types of information that would lead to a superior proposal?
• Are there real efficiencies to be gained by using this method?
• In your opinion, if there are efficiencies, are they significant enough to be worth the risk of using a non-traditional evaluation method (particularly if the value exceeds the SAT)?

I've searched message boards, GAO opinions, and reference books, and talked to colleagues, supervisors, and legal counsel for any precedent on this, but have come up empty. Thanks for your time.

 

Take a look at Monsieur Edwards timeless essay here: http://www.wifcon.com/anal/analcomproc.htm.

Read his entire essay, but focus on these two paragraphs: 1) Simplified Acquisitions, and 2) Do not follow the FAR Part 15 Process Model when making simplified acquisitions.

Many times!

Why do you call this "a non-traditional evaluation method"?  I think of this as standard, basic, everyday material for a GS-1102.

A simplified acquisition under FAR Part 13 either uses price only or it used price and other factors.  For every acquisition, you take your pick.

If you want a bag of coal of a certain specified quality delivered on a certain fixed date (or window), and you aren't willing to pay a higher price for better quality or earlier delivery, you can do the acquisition using price only.  Among all quoters who promise to meet the quality level and delivery date, you select the lowest price for award.

But if you want a bag of coal at a minimum quality level (but you're willing to pay a higher price for higher quality) delivered no later than a certain date (but you're willing to pay a higher price for earlier delivery), you can do the acquisition using price and other factors -- here, the other factors are quality and delivery.  Among all quoters who meet or exceed your quality level and who meet or beat the delivery date, you subjectively select the quoter that provides the best value to the Government, considering price, quality, and delivery.  And, as stated in FAR 13.106-2(b)(3), you do all of this without a formal evaluation plan or the procedural processes of FAR Subpart 15.3, Source Selection.  And you don't have to compare quotes against a standard and then select the best value -- you can immediately compare quotes against each other.  It is easy.  You can use these procedures for all FAR Part 13 acquisitions, including commercial item acquisitions up to $7 million under FAR Subpart 13.5.   It really is easy.

napolik - Thank you for sharing the article. I'm aware of the "looks like a duck" FAR 15 trap. I don't think this article quite drills down to the level specificity I am asking for.

ji20874 - I appreciate you painting that picture for me (the "subjectively select the quoter" bit). The fact that you are the first person that I've ever heard of doing this is why I would describe it as "non-traditional." I've been contracting in the DC region for 5.5 years now across two agencies, and I never seen nor heard of this. In fact, this question originated from an old colleague, who's also worked at DHS at Vermont and VA in Dallas. Right or wrong, I don't get the impression that the practice is very widespread.

I'm curious, are there instances where you would have favored trade-offs when this method was available to you?

3 hours ago, FrankJon said:

I'm curious to know whether anyone has successfully done, or knows of anyone who has successfully done, a comparative evaluation pursuant to 13.106-2(b)(3).

All competitive acquisitions, simplified or otherwise, entail comparative evaluation. If the only factor is price, then you compare prices. If the factor include price and others, such as past performance, then you compare prices and past performance.

FrankJon must clarify his inquiry. Maybe what he wants to know is whether anyone has used a tradeoff process of evaluation when conducting a simplified acquisition, as opposed to simply lowest price or lowest price technically acceptable.

This is a tradeoff process, involving price and the other factors (quality and delivery in my example; price and past performance in Vern's example).  You don't have to (and probably shouldn't) assign ratings (ratings against a standard) (although you may if you insist).  In my example, if both quoters are acceptable but in comparison to each other, the first quoter has the better (lower) price, the better (higher) quality, and the better (earlier) delivery, and the second quoter has the worse (higher) price, worse (lower) quality, and worse (later) delivery, the selection decision is a very easy one in favor of the first quoter -- if it is a mixed bag, then the contracting officer has to make a tradeoff decision and FAR 13.106-3( b )( 3 )( ii ) will apply.  Tradeoffs in simplified acquisitions are very traditional, and they may be done without the FAR Part 15 baggage.    

 

Vern - I am not asking about a trade-off as defined by FAR 15, which allows for comparison only after each proposal has been independently evaluated against fixed criteria. As I understand the meaning of "comparative evaluation" as used in 13.106-2(b)(3), the Government may directly compare the strengths and weaknesses of proposals as they relate *to each other*, as opposed to comparing them to fixed criteria.

ji20874's response seems like a plausible real-world application to me: Once proposals have been deemed acceptable, they are compared side-by-side against one another, and the "best" one is chosen. This seems like a flexibility that customers would really appreciate.

Thank you, ji20874. The only caveat that I can see is that supporting the decision could become tricky, particularly for complex services, higher dollar values, and/or multiple proposals. I imagine that the CO and COR would really need to be on the same page to substantiate every distinction between proposals (perhaps more so than in the case of a FAR 15 trade-off).

FrankJon, have you done a tradeoff under any FAR procurement method (i.e. 8.4, 13, 15 or 16.5)?

38 minutes ago, FrankJon said:

Vern - I am not asking about a trade-off as defined by FAR 15, which allows for comparison only after each proposal has been independently evaluated against fixed criteria.

FrankJon:

That is a myth.

 

Fine, so your conclusion is...FAR 13 is using the precise term "comparative evaluation" to be synonymous with "trade-offs"? Or do you just not know the answer to my original question (which used a precise term and a precise citation)?

I'm genuinely asking. 

19 hours ago, FrankJon said:

I'm curious to know whether anyone has successfully done, or knows of anyone who has successfully done, a comparative evaluation pursuant to 13.106-2(b)(3). Specifically, I'm curious about the following:

• What did/would this look like in practice? For instance, beyond the evaluation factors themselves, would I need to specify the types of information that would lead to a superior proposal?
• Are there real efficiencies to be gained by using this method?
• In your opinion, if there are efficiencies, are they significant enough to be worth the risk of using a non-traditional evaluation method (particularly if the value exceeds the SAT)?

I've searched message boards, GAO opinions, and reference books, and talked to colleagues, supervisors, and legal counsel for any precedent on this, but have come up empty. Thanks for your time.

 

13 hours ago, FrankJon said:

Fine, so your conclusion is...FAR 13 is using the precise term "comparative evaluation" to be synonymous with "trade-offs"? Or do you just not know the answer to my original question (which used a precise term and a precise citation)?

I do not consider comparative evaluation to be synonymous with trade-offs.

I presume that your first post was referring to the following sentence in FAR 13.106-2(b)(3): "Contracting offices may conduct comparative evaluations of offers." If so, then nobody really knows what the FAR councils meant by "comparative evaluations." The sentence was added to FAR by FAC 97-03, 62 FR 64916, Dec. 9, 1997, without any explanation in the final rule or in the previous proposed and interim rules. You presume that it means that COs can evaluate offers directly, without using evaluation standards and ratings. Okay, let's assume that is true. Here is how you do it:

1. We'll choose and define our evaluation factors. For this example I'll choose experience, past performance, and price. We can define those things any way we like, and we'll include our definitions in our solicitation so quoters will know what we're looking for. We won't use "standards." Instead, we're going to compare quoters directly, and we're going to consider the factors to be equally important. We're not going to make tradeoffs.
2. Now we'll issue our RFQ. We'll check the quotes we receive for conformity with the solicitation. A quote is acceptable if it conforms to all material requirements of the RFQ, otherwise it's unacceptable. We'll evaluate only acceptable quotes.

Assume that we get four quotes, from companies A, B, C. and D. We'll evaluate using a method that is sometimes called "pairwise comparisons" and we'll assume transitivity.

First, we'll evaluate for experience. We'll compare A's description of its experience to B's and decide, subjectively, which has the better experience by taking note of asserted facts, identifying differences, determining their significance to us, and documenting our conclusions. Let's say we decide that A is better than B. We'll then compare A to C. This time we think C is better than A. Since C is better than A and A is better than B, we assume that C is also better than B. We'll then compare C to D. We decide that D is better than C. Since D is better than C and C is better than A and B, D is also beter than A and B. So D is best on experience. Since there were four offerors, and since D is best, we'll give D four points. Since C is better than A and B we'll give C three points. Since A is better than B we'll give A two points. Finally, we'll give B one point.

Experience: D = 4, C = 3, A = 2. and B = 1.

Second, we'll evaluate for past performance, using the same procedure as we did for experience. This time the result is as follows:

Past performance: D = 4, A = 3, B = 2, and C = 1.

Third, we now compare the four quoters' prices. This is easy. The lowest price gets four points and the highest gets 1 point. The result is:

Price: B = 4, A = 3, C = 2, and D = 1.

Fourth, we total the points.

A = 8, B = 7, C = 6, D - 9.

Fifth, D is best overall, so we award to D.

We evaluated on the basis of direct comparisons, based on subjective assessments and without standards. We made no tradeoffs, so we have none to document.

That's one way to do it. There are other ways. Is it a good way to do it? That depends on what you're buying and on your notions of value.

The hardest part is writing up the rationale for your subjective assessments. It takes some thinking and word-smithing. If you don't know how to do that, then don't try this method.

Do you want to weight the factors differently? If so, assign each a decimal weight, such as 0.5 for experience, 0.4 for past performance, and 0.1 for price, so that they add up to 1.0. Then multiply the points by the weights before totaling the points.

The fact that you and your colleagues have not heard of this approach should not be surprising. Government agencies make millions of simplified acquisitions every single fiscal year. Millions. No one knows, much less keeps track of, how they do them. They're rarely protested or the subject of explanatory articles. You've worked for the government for five and one-half years and in two agencies. In the words of BoB Dylan, your experience "is limited and underfed." Your lack of experience, knowledge, or passing familiarity with the method does not make the method nontraditional.

Are there any real efficiencies? Compared with what other methods? What I described might take a skilled buyer no more than a couple of days so to complete, depending on what you're buying and assuming that the buyer can write and won't have to go to the boss to ask questions every hour or so. That's longer than a price-only approach but shorter than a tradeoff process type approach. Forget efficiency if you're going to create an "evaluation board."

Risk? There is no appreciable risk for any but the brain dead. Less risk than a tradeoff approach.

Did I answer your questions?

FrankJon,

I'm glad you are here and that you're asking questions.  I hope to impress on you that FAR Part 13 (including Subpart 13.5) is intended to be simplified -- that word is even included in the part's title -- however, anyone can make it as complicated as he or she chooses, and many people do insist on making it too complicated.  I described an easy system using tradeoffs, and Vern described a simple system using scorings.  You absolutely err in thinking you have to "substantiate every distinction between proposals (perhaps more so than in the case of a FAR 15 trade-off)."

http://www.gao.gov/assets/680/674446.pdf

http://www.gao.gov/assets/670/668038.pdf

The bid protest cases linked above might be helpful in understanding how easy this can be.  Too many contracting professionals (and attorneys and reviewers and everyone else) want to deny the "simplified" in simplified acquisition procedures.

Again, I'm glad you're here and that you're asking questions.  I  hope this exchange is helpful to you.

The trick to evaluating based on direct comparisons is to be able to write a sentence about each pairwise comparison on each factor. Suppose that you think A is better than B on the factor of experience. Here's a simple test. Try writing a sentence that begins:

A is better than B because_____________________________________________________.

If you cannot finish that in a way that is consistent with the information that you have about the offerors and makes sense to an intelligent reader, then you'd better stop what you're doing and think things through.

Vern - Thank you for the thorough response.

ji20874 - Thank you for the GAO decisions, which I will review. You are absolutely correct that historically, I have not worked with many COs fully comfortable with using SAP, particularly when over the SAT. When I say that this approach seems like it could be trickier to write to than a traditional trade-off approach, it could just be due to my lack of familiarity with a comparative approach. When I've done trade-offs in the past, I initially compare each proposal to fixed criteria specified in the solicitation. It seems like there would be less risk for error in that situation than when directly comparing proposals to one another, since the proposals may not align and may not be using the same language to discuss the same things. The latter seems like a moving target. But maybe not. Maybe it's just a matter of familiarizing myself and trying it. Thanks again.

Purchasing agents for municipalities and for businesses make trade-offs and comparative analyses every day. I guess most people often make some type of trade-off comparisons when they go shopping for most things. I love to grocery shop but end up taking a long time - drives Libby nuts. The actual shopping process doesn't seem too complicated; the justification documentation for government simplified acquisition purchases will take some effort and time.

1 hour ago, FrankJon said:

I have not worked with many COs fully comfortable with using SAP, particularly when over the SAT.

Out of curiosity, are these COs typically "older" ones that have been in the career field for 20+ years?

I have seen some folks use sealed bidding for simple acquisitions under the SAT. 

Some folks resist change, which is human nature.  However, I think that resistance becomes irrational when folks are given "simplified" methods to do their job, but resist using them or want to over complicate things unnecessarily (e.g., unnecessary or unnecessarily complex evaluation criteria).

I've been doing this for 20+ years, but I resist being resistant to change and welcome relief from the onerous regulations placed on the contracting workforce.

Todd - I would say it's a mix of older and younger. My agency is pretty close to the flagpole, as they say. COs are generally leery of running afoul of OGC and PMR guidance, which is generally incongruous with exploiting efficiencies and experimenting. 

I'm not sure that everyone understands the distinction between (1) a tradeoff approach, which ji20874 has mentioned, and (2) a direct comparison approach without tradeoffs, which I described in my long post.

A tradeoff approach can be based on either (a) the assignment of ratings based on assessments of proposals against objective standards followed by comparisons of strengths and weaknesses in that regard or (b) assignment of ratings based on direct comparisons of proposals without objective standards. What makes for a tradeoff approach is the additional step of  identifying the marginal nonprice/price differences among proposals and determining whether they are worth the marginal differences in price. The tradeoff approach is only one way to select contractors, whether under FAR Part 13 or 15.

See: "Tradeoff Analysis vs. Cost-Benefit Analysis in Source Selection," The Nash & Cibinic Report (July 2010).

A direct comparison approach without tradeoffs is based on simple additive assessment. You compare and rank proposals in light of each of the various factors and additively determine which is best overall. You do not take the additional step of determining whether marginal nonprice differences are worth marginal price differences. I described that approach in detail above. An agency is not required to make nonprice/price tradeoffs unless it tells offerors that it's using a tradeoff process. The direct comparison approach without tradeoffs can also be used under FAR Part 13 or 15. It is generally simpler than the tradeoff approach (although you can make any process more complex), but it may not be the best approach, depending on what you're buying.

For many years people believed that evaluation based on direct comparisons was improper. it was not formally forbidden, except in those agencies, like the Air Force, which expressly forbade its use. It was never forbidden by the FAR itself.

Thank you again, Vern. I think this clarification was needed. I'm going to take some time to digest all of this and see if I can't convince a bold CO to try something new...