Jump to content

Evaluating Past Performance Pass/Fail


Recommended Posts

Hi All,

Is it possible to evaluate past performance on a pass/fail basis in a Part 15 competition (trade-off, not LPTA)? I don't see anything wrong with it as long as it is disclosed in the RFP, but some of our policy folks are queasy about it.

Link to comment
Share on other sites

Guest Vern Edwards

Yes, it is permissible ("possible") to evaluate past performance on a pass/fail basis in a trade-off process source selection. However, if you evaluate past performance on a pass/fail (go/no go, acceptable/unacceptable, satisfactory/unsatisfactory) basis instead of on a comparative scale, and if you find that a small business fails on past performance, then you will have made what amounts to a determination of nonresponsibility and so must refer the matter to the Small Business Administration for certificate of competency consideration. This is a very old case law rule of the GAO. It rarely comes up these days because it is widely known that if you use past performance as an evaluation factor in a trade-off process you should rate it on a comparative scale, not pass/fail. See J. Womack Enterprises, Inc., B-299344, Apr. 4, 2007, 2007 CPD para. 69:

An agency may use traditional responsibility factors, such as past performance, as technical evaluation factors where, as here, a comparative evaluation of those areas is to be performed. Advanced Res. Int’l, Inc.--Recon., B‑249679.2, Apr. 29, 1993, 93-1 CPD para. 348 at 2. A comparative evaluation means that competing proposals will be rated on a scale relative to each other, rather than on a pass/fail basis. Dynamic Aviation--Helicopters, B-274122, Nov. 1, 1996, 96-2 CPD para. 166 at 3. No SBA referral is required where a small business offeror’s proposal is not evaluated as unacceptable and is not selected for award because another offeror’s proposal is evaluated as superior under a comparative analysis or because of a cost/technical tradeoff analysis. Capitol CREAG LLC, B-294958.4, Jan. 31, 2005, 2005 CPD para. 31 at 6-8.

FAR 15.101-2(B)(1) refers to this rule in its discussion of LPTA, but the rule is the same in trade-off source selections.

Link to comment
Share on other sites

Guest Vern Edwards
Suggest a search of wifcon. There was a recent piece on point. I forget whether it was a discussion topic or a CompGen decision, or both. (They say memory is the second thing to go. I forget what was first.) The ruling was against pass/fail.

"The ruling was against pass/fail"? For pete's sake, Cajuncharlie, what the heck does that mean? You write an unclear response and don't give a definite cite to the information that you forgot. Just how useful was that?

It is not illegal or improper to evaluate past performance on a pass/fail basis in a tradeoff analysis source selection. However, using it that way has procedural implications with respect to small businesses and certificates of competency. That's all.

Link to comment
Share on other sites

Charlie might be thinking of the Department of Defense Source Selection Procedures, dated 4/4/2011. In its quest for uniformity, the DOD Guide prescribes use of uniform rating systems for the non-price factors and subfactors when using the Trade-Off Process. The DoD's procedures for the trade-off process don't allow the use of a Go/No-Go or Pass/Fail rating system for Past Performance.

Although use of acceptable/unacceptable criteria might be allowed for some technical factors with the Trade-off process, the Past Performance factor is further distinguished from the other non-price factors. It is a "confidence assessment" of the offeror’s probability of meeting the solicitation requirements, using a two step evaluation procedure. The first step considers how relevant the past performance information is to the instant acquisition. More relevant past performance ratings have "more influence on the past performance confidence assessment than past performance of lesser relevance." The second aspect of the past performance evaluation is to determine how well the contractor performed on the contracts. Both steps use prescribing rating systems. See the Manual at: http://www.jacksonke...df/S0327133.PDF

Link to comment
Share on other sites

I agree that it is permissible, but just to play the devil's advocate here, how would you evaluate offerors with no relevant past performance history? We're required to evaluate them neither favorably nor unfavorably, so how could that occur using just a go/no go decision? I think I know the answer (this isn't a comparative evaluation factor), but just want to flesh out the argument.

Again, I agree that it's permissible.

Link to comment
Share on other sites

See Appendix A, pages A-2 through A-3 at the reference I provided above for example. http://www.jacksonkelly.com/jk/pdf/S0327133.PDF

Link to comment
Share on other sites

Guest Vern Edwards

Joel:

The DOD procedure for tradeoff source selection does not provide for any rating of past performance per se, except to evaluate its relevancy. Instead of establishing a rating scheme for past performance, it provides for using past performance information to establish the agency's degree of confidence in the offeror's ability to perform. What the agency rates is not the offeror's past performance, but its own level of confidence in the offeror's ability to perform based upon its past performance. It's called a "performance confidence assessment," not a past performance confidence assessment.

Link to comment
Share on other sites

[EDIT: Please note that I didnt make any mention above of "rating" past performance during the source selection. The past performance ratings that I referred above to are those associated with evaluations of contractor performance and in the performance reports. We use such previously prepared reports, as well as evaluations by references or other documentation of past performance assessment in the private sector to evaluate the relevancy and quality of past performance to be used in step 2 confidence assessment. I don't know if service and supply contracts use "ratings" in their contractor performance evaluations. The evaluation of relevancy of past performance uses "Past Performance Relevancy Ratings"]

It is DoD's form of a past performance evaluation system. It is based upon a level of confidence that the contractor will successfully perform the contract. Both the Trade-Off and LPTA processes use a 2 strep process, which are described (though not in depth or clearly in my opinion) on pages 16-19 of the Manual . For step 1, the evaluators determine the relevancy of the past performance information to the instant contract scope.

In the LPTA method, the first step only uses the Relevant or Non-Relevant ratings. From page 17: "The SSP shall clearly identify the treatment of

relevancy within past performance evaluation."

From page A-2: ,"The criteria to establish what is recent and relevant shall be unique to each LPTA source selection. Therefore, the solicitation shall establish the criteria for recency and relevancy in relation to the specific requirement being procured. In establishing what is relevant for the acquisition, consideration should be given to what aspects of an offeror’s contract history would give the most confidence that the offeror will satisfy the current procurement."

In the second step, only an "acceptable" or "unacceptable" level of confidence is assigned for past performance per Appendix A. Also refer back to the body of the document for more background.

That's what DoD agencies are now required to uss for their PP evaluation. I cited it as an example in response to FSCO's question. The acceptable rating in Appendix A states: "Based on the offeror’s performance record, the Government has a reasonable expectation that the offeror will successfully perform the required effort, or the offeror’s performance record is unknown. (See note below.)" And:

"Note: In the case of an offeror without a record of relevant past performance or for whom information on past performance is not available or so sparse that no meaningful past performance rating can be reasonably assigned, the offeror may not be evaluated favorably or unfavorably on past performance (see FAR 15.305 (a)(2)(iv)). Therefore, the offeror shall be determined to have unknown past performance. In the context of acceptability/unacceptability, “unknown” shall be considered “acceptable.”

Link to comment
Share on other sites

Guest Vern Edwards

The DOD system is not a past performance rating system, because it does not rate past performance. See page 18:

The Past Performance Evaluation Team will review this past performance information and determine the quality and usefulness as it applies to performance confidence assessment.

And see Table 5 on page 19. As for LPTA, this thread is not about LPTA.

The DOD system for tradeoff source selection does not rate past performance at all, either pass/fail or on a comparative scale. It does that deliberately to avoid certain issues associated with rating past performance. Instead, it rates an agency's confidence in an offeror's ability to perform based on information about its past performance. DOD simply avoids the issue. If you must work within such a system the question on which this thread is based should not come up.

However, nothing in the DOD system prohibits an evaluation team from characterizing an offeror's past performance as unacceptable or "fail." It's just that such a characterization would not show up in the assigned rating, since the rating does not rate past performance.

Link to comment
Share on other sites

The thread initially concerns rating Past performance on a pass/fail basis under a trade-off source selection.

The DoD would only do that under an LPTA approach. DoD would otherwise rate PP under a 2 step comparative approach for a Trade-off approach, as described in the DoD Manual.

The second question asked how a firm with no past performance is rated wnder a pass/fail system. That is described in the 2 step LPTA acceptable/unacceptable approach.

Link to comment
Share on other sites

Thank God only DoD has to use that system for their Past Performance evaluations. I only described them as examples in response to 2 different questions. In DoD the pass/fail evaluation of PP iis limited to LPTA. The DoD guide describes how they rate a firm with no PP.

Link to comment
Share on other sites

Guest Vern Edwards

DoD would otherwise rate PP under a comparative approach for a Trade-off approach, as described in the DoD Manual.

You are wrong. DOD does not rate past performance. It rates its confidence in an offeror based on information about the offeror's past performance. And DOD did not issue a "manual." It issued a policy memo.

By the way, DOD's scheme for evaluating past performance is okay. It makes perfect sense, even though I would change some things.

Link to comment
Share on other sites

Guest Vern Edwards
By the way, DOD's scheme for evaluating past performance is okay. It makes perfect sense, even though I would change some things.

See my article in Wifcon readings: "The Level of Confidence Assessment Rating Technique: A Tool for Source Selection." I think it dates back to the mid-1990s, I can't remember. It describes a system conceptually similar to the DOD system adopted in 2011, except that mine uses numbers and is more complex. If you search GAO decisions you'll find some in which the agency used my LOCAR system. See Moore Medical Corp., B-261758, Oct. 26, 1995. You can get it by Googling the "B" number. Here is how I describe the LOCAR system in my article:

A LOCAR is not a score for an offeror’s capability; it is not an expression of the value of an offeror’s experience, past performance, or key personnel, etc. A LOCAR is an expression of an agency’s confidence in an offeror, based on the agency’s assessment of certain qualities of the offeror as an organization—the nature, depth and breadth of its experience; the quality of its past performance; the qualifications of its key employees, etc.

I wrote that more than 15 years ago. The DOD system is conceptually similar, except that it considers only past performance.

Link to comment
Share on other sites

You are wrong. DOD does not rate past performance. It rates its confidence in an offeror based on information about the offeror's past performance. And DOD did not issue a "manual." It issued a policy memo..It makes perfect sense, even though I would change some things.

Had to check out of hotel and drive to work... Yes, I agree. [EDIT: As I Did say and agree] DoD uses an evaluation of past performance in its assessment of their level of confidence that the contractor will successfully perform the contract.

The DoD agencies cant "rate" past performance on a pass/fail basis when using the trade-off approach. The DoD policy discusses evaluation of offerors with no relevant past performance history.

[EDIT: Please See my edit to post #10 above. My initial reference to PP ratings was with respect to "Contractor Performance Evaluations" , not to source selection performance evaluations.

There is a rating system used for evaluating the relevance of the past performance evaluations/ratings to the instant acquisition in step one.

Step Two is a confidence assessment, which takes into account the quality of past performance as well as the relevancy of the information.]

Link to comment
Share on other sites

Guest Vern Edwards

They can't rate past performance on a pass/fail basis because they don't rate past performance in any way at all. They rate only their own level of confidence in the offeror.

As you pointed out, there are two steps: first, to determine the relevancy of past performance, and second, to determine the agency's level of confidence in the offeror's ability to perform based on the quality of its past performance. Here is how the DOD policy memo describes that second step, on page 18:

The second aspect of the past performance evaluation is to determine how well the contractor performed on the contracts. The past performance evaluation performed in support of a current source selection does not establish, create, or change the existing record and history of the offeror’s past performance on past contracts; rather, the past performance evaluation process gathers information from customers on how well the offeror performed those past contracts. Requirements for considering history of small business utilization are outlined at FAR 15.304©(3)(ii) and DFARS 215.305(a)(2).... The Past Performance Evaluation Team will review this past performance information and determine the quality and usefulness as it applies to performance confidence assessment.

Nothing in the memo prevents the PPET from describing how well an offeror performed on a pass/fail or acceptable/unacceptable basis in its documentation. Nothing prevents the agency from using a questionnaire that asks references to evaluate the offeror's past performance on a pass/fail or acceptable/unacceptable basis.

Link to comment
Share on other sites

Guest Vern Edwards

P.S. Let's take our analysis of the DOD policy a step further. The performance confidence rating scale is:

Substantial confidence

Satisfactory confidence

Limited confidence

No confidence

Unknown confidence

The description of satisfactory confidence is:

Based on the offeror’s recent/relevant performance record, the Government has a reasonable expectation that the offeror will successfully perform the required effort.

I find nothing in the memo which would prevent a PPET from assigning a rating of satisfactory confidence based on a pass/fail or acceptable/unacceptable test of its past performance. If past performance "passes" the offeror gets at least a rating of satisfactory confidence. If it "fails" the offeror gets a rating of limited confidence or less.

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...