In my agency, standard practice for past performance (PP) evaluation involves first determining the relevancy of past performance information to the proposed scope of work. We do this by evaluating the size of the PP contract, it's scope (by reviewing proposal information or finding the PWS), and the PP contract's "complexity". Size and scope have their own definitions, but I want to focus on complexity. Complexity is generally defined in the RFP as "performance challenges", which vary from RFP to RFP and could include subcontractor management, management of large complex contracts in highly regulated industries, cost efficiencies, etc. This results in separate documentation in our evaluation report speaking specifically to how the scope may or may not be relevant, and then how the complexity may or may not be relevant.

Background: My agency requests that offerors provide a few pages of information on two or three selected past performance (or "reference") contracts to aid in evaluation. This includes discussion of scope relevance and complexity relevance, which are separate entries on the provided form. We then evaluate other past performance information that is available to us.

My question to the forum is whether other agencies regularly evaluate the complexity of past performance contracts separate from scope (from my readings of GAO and COFC decisions, it seems to be a fairly standard practice), and whether or not evaluating complexity adds value. In my experience: 

• Lesson learned: Complexity is generally not a discriminator in relevancy determinations
• Root Cause: The RFP definition of complexity as “performance challenges” nearly always comes in the context of scope, making it difficult to distinguish between the two.
  • Offerors can struggle with the concept and implementation of complexity into reference contract information forms, when separated from a discussion on how a given contract is relevant to the scope of the proposed PWS.
  • Complexity is difficult and time-consuming to evaluate when reviewing non-reference contracts.
  • The acquisition community has used the phrase "size, scope, and complexity" for decades without carefully considering what they mean and how they each aid evaluation.
• Recommendation: Study the potential effects of removing complexity as an independently evaluated item in the past performance relevancy evaluation.
  • SEB’s can still include complexity as a part of the scope relevancy evaluation. For example: “Contract relevance will be determined based on size and scope, including complexity.”
• Conclusion: SEB’s can safely rely solely on size and scope to determine relevancy. RFP’s can still solicit examples performance challenges in the scope description. Eliminating separate complexity determinations would streamline the relevancy evaluation process.

Is "size, scope, and complexity" standard language in your RFPs? How do you approach complexity? Do you agree or disagree with my points and why? 

Thank you in advance for your thoughts. Note that I am not an 1102, but can pick my way around the FAR when I need to. Please be gentle. :) 

First, think in terms of (A) experience and (B) past performance. Experience is the record of what an offeror has done in the past. Past performance is an assessment of how well the offeror has performed. You should always evaluate both. They are related, but they are different things.

All assessments of past performance are associated with particular instances of experience. Let's call those instances jobs.

What you should want to know about an offeror is how many jobs it has had in the recent past that were like the one you will want it to do for you. You want a contractor to have experience doing jobs that were similar in terms of the type of work that was done, the size of the job, and the complexity of the job.

Complexity is the product of (a) what the work entailed, its "scope", and (b) the conditions under which the work was done. Complicated work performed under variable and unpredictable conditions can produce unexpected states of affairs. Such states may present themselves in the form of unexpected difficulties that the contractor will have to overcome in order to complete the job as promised. That is why the complexity of past jobs is an important consideration.

We presume that experience teaches a company about what can go wrong, what are the warning signs that something is going wrong, what if anything can be done to prevent them from going further wrong, what remedial actions can be taken to get things back on track, and what can be done to recover from unpreventable disaster. That's why experience is valuable.

What you want is a contractor (a) whose experiences should have prepared them for the work to come, and (B) whose past performance shows that they have learned from those experiences.

Me_Box, in addition to what Vern said - which is spot on - the  overall past performance rating would also take into consideration how recent and relevant the experience is to the instant project. One can be the best at what they do but if it isn’t very recent and/or relevant experience, the overall “past performance” rating might not be as strong as another [edit: solid] performer with more recent, relevant experience.

By chance are you with DoD? The DoD source selection procedures combine experience and past performance under one “confidence assessment”. 

Edited May 4, 20223 yr by joel hoffman
Changed “top” to “solid” to differentiate “the best” at something else from a solid performer with highly relevant experience.

👍Vern.  The best explanation I’ve seen on the differences between experience and past performance. 

1 hour ago, Vern Edwards said:

First, think in terms of (A) experience and (B) past performance. Experience is the record of what an offeror has done in the past. Past performance is an assessment of how well the offeror has performed. You should always evaluate both. They are related, but they are different things.

All assessments of past performance are associated with particular instances of experience. Let's call those instances jobs.

What you should want to know about an offeror is how many jobs it has had in the recent past that were like the one you will want it to do for you. You want a contractor to have experience doing jobs that were similar in terms of the type of work that was done, the size of the job, and the complexity of the job.

Complexity is the product of (a) what the work entailed, its "scope", and (b) the conditions under which the work was done. Complicated work performed under variable and unpredictable conditions can produce unexpected states of affairs. Such states may present themselves in the form of unexpected difficulties that the contractor will have to overcome in order to complete the job as promised. That is why the complexity of past jobs is an important consideration.

We presume that experience teaches a company about can go wrong, what are the warning signs that something is going wrong, what if anything can be done to prevent them from going further wrong, what remedial actions can be taken to get things back on track, and what can be done to recover from unpreventable disaster. That's why experience is valuable.

What you want is a contractor (a) whose experiences should have prepared them for the work to come, and (B) whose past performance shows that they have learned from those experiences.

Thank you for your insight, Vern. Much better put than various legal articles and cases I've been reading. 

I think my agency is trying to have their cake and eat it too. A few years ago they nixed "Experience" as a separate evaluation factor en lieu of Past Performance as a stand-alone factor comprising both assessments you reference. So it becomes a multi-step process where we first determine "relevance", and then determine whether there are Strengths or Weaknesses associated with that relevant performance. So the first step would be (A) experience, and the second step would be (B) past performance, even though Step (B) only happens when a contract is rated well in step (A). 

What I'm hearing from you is a suggestion that complexity should be evaluated hand in hand with scope, and space should be provided for offerors to describe those unpredictable conditions or events, along with their responses/corrective actions, to demonstrate how those conditions might be similar to the proposed scope of work/environment. 

 

1 hour ago, joel hoffman said:

Me_Box, in addition to what Vern said - which is spot on - the overall past performance rating would also take into consideration how recent and relevant the experience is to the instant project. One can be the best at what they do but if it isn’t very recent and/or relevant experience, the overall “past performance” rating might not be as strong as another top performer with more recent, relevant experience.

By chance are you with DoD? The DoD source selection procedures combine experience and past performance under one “confidence assessment”. 

I'm with DOE. We had selection procedures that included confidence assessments but changed those recently in favor of simple strengths and weaknesses. We took care of recency by only requesting/evaluating the past X years of past performance information and sum up with an adjectival rating for the offeror that combines relevancy and strengths/weaknesses. 

1 minute ago, Me_BOX_Me said:

What I'm hearing from you is a suggestion that complexity should be evaluated hand in hand with scope, and space should be provided for offerors to describe those unpredictable conditions or events, along with their responses/corrective actions, to demonstrate how those conditions might be similar to the proposed scope of work/environment.

What I think you should ask for is (1) a description of each job and the conditions of performance, (2) descriptions of the performance challenges of each job, and (3) descriptions of lessons-learned that will be of help during performance of your job.

11 minutes ago, Vern Edwards said:

What I think you should ask for is (1) a description of each job and the conditions of performance, (2) descriptions of the performance challenges of each job, and (3) descriptions of lessons-learned that will be of help during performance of your job.

Loud and clear, thanks again!