Jump to content
The Wifcon Forums and Blogs

FARmer

Interpreting what "Neutral" means in the eyes of GAO- Past Performance-

Recommended Posts

Hello WIFCON folks,

I had an interesting conversation today in regards to past performance evaluations (in the context of FAR part 15). The question is that if an offeror submits past performance that, when evaluated, is determined to be NOT relevant in size, scope, and complexity should this Offeror's past performance be looked at in the context of "Neutral" or not rated favorably or unfavorably per FAR13.305(a)(2)(iv)?

I believe that their was a misinterpretation of the FAR and GAO cases today in my discussion. I believe that if an offeror submits past performance to a RFP and that past performance submission was evaluated and determined NOT relevant in size, scope, and complexity, then the past performance would be something less than a default to a "neutral". Maybe a low confidence, no confidence, or even not acceptable (depending adjectival rating). It's fair game in the context of the evaluation. Whereas, an Offeror that has no past performance and submits a proposal that states it has no past performance...... This instance is what I believe the FAR is referring to in FAR 15.305(a)(2)(iv) as an offeror without a record of relevant past performance or for whom information on past performance is not available. Commonsense would lead me to believe that latter was true, then what is the point of even evaluating past performance. It's perfectly OK to allow a company that only has grass cutting experience to build a air craft carrier, right?

I have combed through some GAO cases that point to this direction, but nothing that specifically jumps out to say that the if an Offerors past performance, and it was not relevant, than an unacceptable rating is correct. I have looked at Menendez-Donnell & assoc File: B-286599, DA Defense Logistics HQ File: B-411153.3, Zolon Tech, Inc B-299904.2   I thought all of these cases were pretty clear in terms of past performance, but obviously not clear enough for my audience.

 

Thoughts? Any GOA cause that might help?

 

 

 

Share this post


Link to post
Share on other sites

"In the case of an offeror without a record of relevant past performance or for whom information on past performance is not available, the offeror may not be evaluated favorably or unfavorably on past performance."

Read FAR 15.305(a)(2)(iv) again, slowly: "without a record of relevant past performance."

On what do you base your stated beliefs? You mentioned "Commonsense" (sic) but we have the plain language.

 

Share this post


Link to post
Share on other sites

In the MDA case GAO states

Quote
"We note initially that, since the record confirms that the prior contracts MDA listed to establish its past performance were not similar to the current requirement, MDA Proposal, Tab 2, at 1-3, the agency reasonably determined that MDA's prior contracts did not provide a basis for assessing the firm's past performance. However, it appears this aspect of MDA's past performance warranted a neutral, rather than an unacceptable, rating; under FAR Sec. 15.305(a)(2)(iv), where an offeror does not have a record of relevant past performance, the offeror may not be evaluated either favorably or unfavorably. On the other hand, even if this is the case, and even if we also agreed with MDA that the agency should have clarified its experience or obtained additional information from its own records, as discussed below, GSA reasonably rated the proposal unacceptable based on MDA's failure to establish adequate experience and past performance for its subcontractors and key employees............
 
To the extent MDA contends that GSA was required to assign a neutral rating to its proposal based on the absence of information relating to its key subcontractors, we disagree. Although FAR Sec. 15.305(a)(2)(iv) requires an agency to assign a neutral rating where past performance information is not "available," here, the protester's proposal represented that its proposed subcontractors are engaged in projects that would illustrate their performance capability. The information thus was available, but MDA chose not to present the information in its proposal, in direct contravention of the terms of the RFP. In our view, an offeror cannot simply choose to withhold past performance information--and thereby obtain a neutral rating--where the solicitation expressly requires that the information be furnished, and where the information is readily available to the offeror."
 

Based on what I have read,  it's the offeror's burden to submit an adequately written proposal for the agency to evaluate and further noted that an agency may reject a proposal for information deficiencies that prevent the agency from fully evaluating the proposal. Based on all my reading, there is a difference between having no relevant past performance (FAR 15.305(a)(2)) (therefore the offeror submitted no past performance) vs. submitting a proposal that has past performance that is not even relevant in size scope or complexity.

Share this post


Link to post
Share on other sites
On ‎4‎/‎26‎/‎2016 at 2:37 PM, FARmer said:

I believe that if an offeror submits past performance to a RFP and that past performance submission was evaluated and determined NOT relevant in size, scope, and complexity, then the past performance would be something less than a default to a "neutral".

I agree with Pepe.  The FAR is clear.  If past performance is not relevant or is not available, they can not be evaluated favorably or unfavorably.  In your question, the determination was already made that the past performance was not relevant, so there does not appear to be a debate regarding whether it is relevant or not.  You may want to look at these other resources on Wifcon if you haven't already.


Blog entry from Emptor Cautus:

Protest decisions and court opinions:

   http://www.wifcon.com/pd15305a2ii.htm

Share this post


Link to post
Share on other sites
5 minutes ago, FARmer said:

Based on what I have read,  it's the offeror's burden to submit an adequately written proposal for the agency to evaluate and further noted that an agency may reject a proposal for information deficiencies that prevent the agency from fully evaluating the proposal. Based on all my reading, there is a difference between having no relevant past performance (FAR 15.305(a)(2)) (therefore the offeror submitted no past performance) vs. submitting a proposal that has past performance that is not even relevant in size scope or complexity.

PepeTheFrog does not know where to start, and is hop, hop, hopping away from this discussion.

Share this post


Link to post
Share on other sites
18 minutes ago, FARmer said:

Based on what I have read,  it's the offeror's burden to submit an adequately written proposal for the agency to evaluate and further noted that an agency may reject a proposal for information deficiencies that prevent the agency from fully evaluating the proposal. Based on all my reading, there is a difference between having no relevant past performance (FAR 15.305(a)(2)) (therefore the offeror submitted no past performance) vs. submitting a proposal that has past performance that is not even relevant in size scope or complexity.

In the case you cited the issue was that they did have relevant past performance which they withheld in an attempt to gain a "neutral" rating.  This is different than having no past performance or past performance that was not relevant.  Your initial question did not state they had relevant past performance that they withheld.

Believe it or not, the firm that only cut grass, but wanted to build an aircraft carrier would not receive an unfavorable rating with regard to a past performance element.  However, I suspect they would receive an unacceptable rating on an experience or technical approach factor. 

Share this post


Link to post
Share on other sites
DA Defense Logistics HQ  File: B-411153.3  Date: December 2, 2015 (http://www.gao.gov/products/D12323#mt=e-report) (This case lead me to the MDA case). 
 
An excerpt form the case, GAO states: "Furthermore, we find no basis to conclude that the Army violated FAR § 15.305(a)(2)(iv) by “unfavorably” evaluating the protester based on its lack of relevant past performance as a prime contractor.  The agency did not “unfavorably” evaluate the protester for its lack of relevant past performance as a prime contractor; rather, the Army reasonably determined that the protester had not demonstrated sufficiently relevant past performance to warrant the highest possible confidence assessment under the RFP’s qualitative evaluation criteria. "
 
Todd- Thank you for the link to Emptor's blog entry link I am reading it now.

Share this post


Link to post
Share on other sites

FARmer,

In the case you cited (and in the text you quoted above) the agency determined the offeror did have relevant past performance (the GAO agreed), it just was not sufficiently relevant (experience they had was not as a prime).  The requirement to not rate favorable or unfavorable applies when past performance does not exist or is not relevant at all. 

"The Army found that DA Defense submitted relevant past performance demonstrating its prior experience as a subcontractor performing the performance work statement’s maintenance, supply, and transportation requirements. The agency, however, found that DA Defense did not demonstrate relevant past performance with the overarching program management tasks that the protester, as the prime, would have to fulfill on the resulting order."

Share this post


Link to post
Share on other sites

Did the solicitation distinguish between experience and past performance? Sometimes the terms are used interchangeably and sometimes they have specific meanings.

Experience - opportunity to learn by doing

Past Performance - Quality assessment of recent, relevant experience

Share this post


Link to post
Share on other sites

First, "neutral rule" is an obsolete term.

Second, the easiest way to think about the past performance rule at FAR 15.305(a)(2)(iv) is this: You cannot evaluate something that does not exist or that you have no information about.

Third, evaluate experience and past performance separately.

  • Relevant experience is the number of chances that a firm has had in the past to do the kind of work that they will be required to do under the prospective contract and to learn about the kinds of problems that will be involved in performance and how best to handle them. Evaluation of relevant experience does not entail any assessment of how well the firm performed.
  • There are two facets to experience: depth and breadth. Depth is reflected in the number of times a firm has done the work. Breadth is reflected in the extent of the work that it has done. If a prospective contract entails performance of five distinct tasks, the more times a firm has performed a particular task, the deeper its experience with that task. The more of the five tasks the contractor has performed, the broader its experience with those tasks. See the matrix below. (Crude, but the best I could do here.) You can use a number of different rating schemes with that matrix and make it as complicated as you like.

 

 

 

 

 

BREADTH

 

 

 

 

TASKS

 

 

 

 

 

 

 

 

 

 

 

DEPTH

REFERENCES

1

2

3

4

5

A

 

 

 

 

 

 

B

 

 

 

 

 

 

C

 

 

 

 

 

 

 

 

 

D

 

 

 

 

 

 

 

 

E

 

 

 

 

 

  • Relevant past performance is the record of how well a firm performed doing the kind of work that it will have to do under the prospective contract and that confronted the firm with the kinds of problems that it is likely to encounter in the future. Evaluation of relevant past perfomance does not entail any assessment of how many times the firm has done that kind of work.

Always assign more importance to experience than to past performance, because experience is easier to verify than the quality of performance. You can't evaluate past performance that doesn't exist or for which you have no information, but you can evaluate the lack of verifiable experience.

Share this post


Link to post
Share on other sites

Ok, so Past Performance, FAR 15.305(a)(2)(iv) has been a hot topic. I truly understand that. So in laymans terms and using Vern's example above as well as what Todd Davis cited right before Vern's post-

Offeror A submitted past performance for the requirement. Offeror A's past performance specifically states that they have similar work that is relevant (using Vern's definition above) in all facets. When evaluated, the evaluators determine that the past performance submission is only relevant to Task 1, and is completely not relevant to any of the other tasks. As a matter of fact, base on the proposal, if Offeror A is considering their past performance relevant to Tasks 2, 3, 4, and 5 then they clearly don't understand the requirement they are proposing to.  To the evaluators, this past performance submission was unacceptable...... It wasn't that there was no information to provided to access, it was just completely irrelevant to the requirements of the RFP. 

Now Offeror B submitted NO past performance. They clearly stated in their proposal that they have no past performance and therefore Offeror B was not rated favorably or unfavorably.... they received a neutral per the DoD source selection guide.

 

So based on everything above, are you telling me that Offeror A should be rated the same as Offeror B?

As quoted from my earlier post in regards to the MDA case, GAO stated:

Quote

In our view, an offeror cannot simply choose to withhold past performance information--and thereby obtain a neutral rating--where the solicitation expressly requires that the information be furnished, and where the information is readily available to the offeror.

  Is it then unreasonable to state that an offeror cannot simply choose to submit irrelevant past performance in order to obtain a neutral rating?

 

Share this post


Link to post
Share on other sites

So I am going to post again because after reading many more cases, it appears that in my example above, both Offerors A and B should/would be treated the same........... So I guess what I thought is common sense isn't necessarily the case. :huh:

Share this post


Link to post
Share on other sites

FARmer:

The evaluation of past performance is supposed to be an evaluation of how well an offeror performed in the past. An RFP can limit consideration of past performance to "relevant" past performance, which means that the agency is interested only in how well offerors performed when doing "relevant" work. If an offeror describes performance history that the agency determines is not relevant, then there is nothing for it to evaluate. If there is nothing for it to evaluate, then the agency cannot say anything good or bad about nothing. But if the agency finds out on its own that there really was something, and that the offeror didn't report it, then the agency can evaluate the something that it found.

It really is quite simple.

Lesson? Don't rely on offerors to tell you about things that they didn't do well. Do some checking on your own.

Share this post


Link to post
Share on other sites

Vern, et al. Thanks once again for clearly differentiating between "relevant experience" and "past performance". This is one of my pet peeves.

A construction firm can excel at building houses, which is light, residential construction. But if the instant solicitation is for construction of an aircraft hangar (medium to heavy industrial), the firm's experience is not really relevant.

It's past performance record, if using the DOD's evaluation criteria for assigning a "confidence rating" would be of low relevance of those type projects to the instant project, thus would not lead one to assign the firm a high level of confidence rating.

I believe that certain organizations have contributed to the confusion by devising evaluation schemes that don't clearly differentiate between PP and level of experience, such as the Air Force "Past Performance- Price Trade-off" scheme, at least the one they were using a few years ago.  That was one of my pet peeves.

Indeed, I couldn't find in the Wifcon Protest pages, a separate category for experience, either.  I may have missed it.

Share this post


Link to post
Share on other sites

I believe all past performance is relevant -- but some of it is more relevant than others.

For example, the grass-cutting company has past performance in managing work and managing schedules and being committed to success.  Its past performance might be of little relevance in an aircraft carrier acquisition, but little relevance is not irrelevant.  If I were drafting an aircraft carrier solicitation, and if I were evaluating past performance, I would want to include text in the solicitation inviting the offeror to describe its past performance without limiting relevance to previous aircraft carrier contracts in the last year.  Then, I could do a comparative assessment of past performance information among all the offerors and identify those offerors in whom I have the highest confidence of future success, based on past performance.  In such a comparative assessment based on past performance, it is likely that the grass-cutting company's past performance (maybe very good, but of little relevance) would give me less confidence than a firm with more relevant past performance.

I think we err when we treat a past performance evaluation (FAR 15.305( a )( 2 )) like a technical evaluation (FAR 15.305( a )( 3 )).  The past performance is supposed to be a comparative assessment, in contrast to the technical evaluation which is supposed to assess each offeror's ability to accomplish the technical requirements (the word "comparative" is not used in the FAR text on technical evaluations).  We religiously make sure that our technical evaluations are not comparative, and that is good; but in our zeal we also make sure that our past performance evaluations are not comparative, and we err in doing so.  Even so, the source selection authority will also make a comparative assessment of proposals as part of the decision-making process (FAR 15.308).

If we did our past performance evaluations as comparative assessments, as the FAR tells us we should do, we would avoid some of the problems we seem to create for ourselves.

Share this post


Link to post
Share on other sites

The biggest problem is experience, not past performance. The first mistake is measuring experience in terms of years. Years are not a valid unit of experience, because the passage of time is not what educates a firm. The second problem is that people have not thought deeply enough about the nature and value of experience. What educates firms is the encounter with specific work. More specifically, the encounter with problems that must be solved in order to complete the work as specified, on time, and within budget, if possible. An experience is an instance of such an encounter, and the proper measure of experience is the number of such encounters.

When contracting, the evaluation team must think about the work that they want done and then think clearly about how that work will challenge a contractor. What problems will have to be solved in order to finish satisfactorily? By "problem" I don't just mean trouble, but also those circumstances, conditions, and situations that must be dealt with through the effective application of know-how or development of new know-how in order to make progress. Every performance of particular task is unique to some degree due to variations in customer, workforce, environment, initial conditions, events, etc. The more times a firm performs a particular task the more opportunities it will have to encounter those variations and the more it will learn about what works and what doesn't.

Past performance is nothing more than an assessment of how well the contractor has done that work in the past. Once you know what experience is relevant to the work to be done, you know what past performance is relevant.

The main problems associated with experience and past performance is that information about them is based on memories and subject to distortion and exaggeration. Past performance has an additional problem -- evaluation depends more heavily on opinion and is more vulnerable to bias, pro and con.

Share this post


Link to post
Share on other sites
36 minutes ago, ji20874 said:

 The past performance is supposed to be a comparative assessment, in contrast to the technical evaluation which is supposed to assess each offeror's ability to accomplish the technical requirements (the word "comparative" is not used in the FAR text on technical evaluations).  We religiously make sure that our technical evaluations are not comparative, and that is good; but in our zeal we also make sure that our past performance evaluations are not comparative, and we err in doing so.  Even so, the source selection authority will also make a comparative assessment of proposals as part of the decision-making process (FAR 15.308).

That is a misdescription of evaluation. All evaluations--except those conducted on a pass/fail, acceptable/unacceptable basis--entail a two-step process. In the first step you measure or assess each proposal on the basis of the applicable evaluation factors and factor standards. That's what you do when you determine a proposal to be exceptional, acceptable, etc., with respect to a factor. In the second step you compare each proposal to the others on the basis of the results of the first step in order to rank it based on its relative value. That is true for both the technical facet of proposals and for past performance when past performance is used as an evaluation factor as opposed to a responsibility consideration. In short, we do compare technical proposals, but only after we have evaluated them on the basis of factor standards.

Share this post


Link to post
Share on other sites

Vern,

Are your two steps the evaluation described in FAR 15.305 (done by the evaluators) and the selection decision in FAR 15.308 (done by the source selection authority)?  I am keeping them purposefully separate.  It is not a universal practice for evaluators to do a comparative assessment for the technical evaluation -- FAR 15.305( a )( 3 ) doesn't call for a comparative assessment or a qualitative ranking for the technical evaluation.  However, and in contrast, FAR 15.305( a )( 2 ) does call for a comparative assessment just for the past performance evaluation.  After all of this is done by the evaluators, the selecting official then does a comparative assessment of the proposals under FAR 15.308.

(edits made)

Share this post


Link to post
Share on other sites

My two steps are:

  1. Assigning value to each proposal based on factor standards.
  2. Comparing and ranking proposals based on the results of Step 1.

I consider both steps to be part of what FAR describes in 15.305, "Proposal evaluation." Under DOD procedures, Step 1 is performed by the SSET and Step 2  is performed by the SSAC. The results are then given to the SSA, who makes the source selection.

What FAR are you using? In the one at acquisition.gov, FAR 15.305(c) does not describe technical evaluation. It states restrictions on the use of support contractor personnel. FAR 15.305(a)(3), which is entitled "Technical evaluation," states only what must be included in the source selection records. It does not describe any process. But 15.305(a)(3)(ii) expressly states that the records of technical evaluation must include some form of ranking, which is produced in Step 2.

Share this post


Link to post
Share on other sites

FAR 15.305( a )( 3 ) doesn't require a comparative assessment or a qualitative ranking for the technical evaluation.  It allows for, but does not require, a quantitative ranking.

In contrast, FAR 15.305( a )( 2 ) explicitly calls for a comparative assessment for the past performance evaluation.

Sometimes, in my observation, past performance evaluations do not include a comparative assessment.  This comparative assessment asked for by FAR 15.305( a )( 2 ) is part of the past performance evaluation, and is separate from the comparative assessment of the proposals as a whole required by FAR 15.308.

Share this post


Link to post
Share on other sites

ji20874:

It appears that what I have said has caused you to experience some degree of cognitive dissonance. But I can assure you, on the basis of my experience and past performance in having conducted (for DOD and DOE), observed and studied, taught, consulted, and publicly written about source selection in several books and who knows how many articles, that I know what I'm talking about. I'll brag and say that there is only one person alive who has written more about this or been cited more frequently about this than me.

Your reading of FAR 15.305 is simply wrong. Here is 15.305(a)(3) in its entirety:

 

Quote

(3) Technical evaluation. When tradeoffs are performed (see 15.101-1), the source selection records shall include

(i) An assessment of each offeror’s ability to accomplish the technical requirements; and

(ii) A summary, matrix, or quantitative ranking, along with appropriate supporting narrative, of each technical proposal using the evaluation factors.

There shall be a ranking, and it shall be either "summary, matrix, or quantitative." Most agencies use summary or matrix rankings.

I know of no way to rank proposals other than by comparing them. See also FAR 15.305(a):

Quote

Proposal evaluation is an assessment of the proposal and the offeror’s ability to perform the prospective contract successfully.An agency shall evaluate competitive proposals and then assess their relative qualities solely on the factors and subfactors specified in the solicitation. Evaluations may be conducted using any rating method or combination of methods, including color or adjectival ratings, numerical weights, and ordinal rankings. The relative strengths, deficiencies, significant weaknesses, and risks supporting proposal evaluation shall be documented in the contract file.

There are two steps in each technical evaluation: 1) determination of the technical merits of each proposal on the basis of evaluation factor standards and 2) ranking of proposals from best to worst on the basis of the results of Step 1. Perhaps in your experience "technical evaluation" is Step 1 and Step 2 is called something else. I don't know. As I said, DOD organizationally separates Step 1 and Step 2 in some cases, but does not require that separation in all cases.

By the way, from a legal standpoint, there is no law against evaluating and rating proposals by direct comparison on the basis of the factors without the use of factor standards. In fact, that method is commonly taught in courses in decision analysis and is probably best for many commercial products.

 

Share this post


Link to post
Share on other sites

Vern,

This thread is about past performance evaluations.  My point was that we don't always do past performance evaluations as a comparative assessment.  If we did always do past performance evaluations as a comparative assessment, rather than evaluating against a factor standard, and if we didn't define relevance so narrowly in our solicitations, then a lot of the angst about neutral ratings in past performance evaluations would disappear.

What is true from an academic perspective doesn't always show itself in practice in all the contracting offices around the country.

I've been encouraging direct comparisons (rather than evaluation against a standard) for technical evaluations for a while.  I hope the practice spreads.  Part of the hindrance is a notion in the community, as a result of all the "training" that has occurred, that we absolutely muist evaluate against the standard first, before the comparative assessment of the selecting official, even in the Subpart 8.4 world.  That notion isn't true, but the perception exists.  The perception also exists that past performance evaluations have to be done against a factor standard, rather than as a comparative assessment -- but as I have been trying to point out, FAR 15.305( a )( 2 ) clearly calls for a comparative assessment for the past performance evaluation.  Notwithstanding all the "training" that occurs, many contracting officers do not know this.

Share this post


Link to post
Share on other sites
Guest
This topic is now closed to further replies.

×