Jump to content

Selection of Factors Question


Voyager

Recommended Posts

This is a request for an example of when a CO should document the file not to use past performance.  It is not related to any ongoing procurement of mine but it does, in general, pertain to systems contracting.  It is a simple question looking for a simple answer, inspired by reading the first few pages of a recent COFC case wherein the lack of a PP factor seemed like just another background fact, stated in passing.

When an agency is leaving the technology development phase, e.g., after procuring a “Demonstration of Existing Technology” via BAA, and runs a competition for the design, development, and manufacture of prototypes, would either of the facts that 1) only the BAA demonstrators will compete or 2) the past performance would not be relevant to new technology, be a logical (and defensible), “reason past performance is not an appropriate evaluation factor for the acquisition,” IAW FAR 15.304(c)(3)(iii)?  If you think yes, please answer whether or not you also think the principle behind this reason could apply outside of systems contracting, and why.  If you think not, give another reason.

I am not looking for expert-level advice here.  I do not have an active procurement.  This is purely a discussion topic to spur thoughts and talk shop about past experiences.

Link to comment
Share on other sites

If you're conducting a FAR 15.3 competitive source selection, and you are limiting participation to a short list of companies with known good past performance, then YES, it may make sense to not use past performance as an evaluation factor.  Whether or not the procurement is for "systems contracting" is irrelevant.

Now, why don't you answer your own question and ask if anyone agrees with you?  You have an opinion, right?

Link to comment
Share on other sites

The post is also frustrating.  The link doesn’t work, it refers to a decision dated July 22 while all I could find is a decision of Jan 31, and that decision is very long and doesn’t mention past performance in the first few pages.  

Link to comment
Share on other sites

5 hours ago, Voyager said:

This is a request for an example

How about this to assist yourself.  A little more research beyond your cited case.  By example I did two internet searches and found GAO COFC cases that discuss the very topic.  I might add that in the world of "it depends" the specific facts of any specific procurement will guide the decision on past performance as a factor or not.   There is not a cookie cutter example but darn sure examples of experiences of others that will help in any specific instance you will have.

Hint - "Justification not to use past performance as an evaluation factor + GAO" or "COFC".

Link to comment
Share on other sites

13 hours ago, C Culham said:

Hint - "Justification not to use past performance as an evaluation factor + GAO" or "COFC".

Searching Westlaw for "15.304(c)(3)(iii)", I found only three GAO decisions about protests challenging an agency's decision, in accordance with FAR 15.304(c)(3)(iii), to not evaluate past performance. All three protests were denied. See:

  • Inuksuk A-S, Comp.Gen.May 26, 2022, B-420527.22022 CPD P 132
  • Inmarsat Government, Inc., Comp.Gen.May 21, 2021, B-419583 B-419583.2
  • Pathfinder Consultants, LLC, Comp.Gen.March 15, 2021, B-419509, 2021 CPD P 124

It is clear from those decisions that an agency must be diligent in developing and documenting its rationale.

I found no COFC decisions in which a decision pursuant to FAR 15.304(c)(3)(iii) was an issue.

19 hours ago, Voyager said:

This is a request for an example of when a CO should document the file not to use past performance. 

The request for an "example" of when a CO should document the file does not make sense. According to FAR 15.304(c)(3)(iii), COs must document the file whenever they decide that the evaluation of past performance would not be appropriate.

The OP also asked:

  1. whether, in a particular case, either of two possible justifications for not evaluating past performance were logical and defensible,
  2. whether the principle underlying those justifications would apply outside of system contracting and, if so
  3. why?

Unfortunately, he provided only scant facts and did not state the underlying principle as he understands it to be, so anyone wanting to answer his questions would have to expand upon the facts and identify the underlying principle on their own initiative, which means that a response might not reflect his understanding of the case.

Link to comment
Share on other sites

22 hours ago, Voyager said:

This is purely a discussion topic to spur thoughts and talk shop about past experiences.

In the spirit of discussion to spur thoughts...

FAR 15.304(c)(3)(iii) says: "Past performance need not be evaluated if the contracting officer documents the reason past performance is not an appropriate evaluation factor for the acquisition."

First, why wouldn't past performance be an "appropriate" evaluation factor?

  • Might it be because (1) the only companies that could perform are the usual suspects, (2) they've all screwed up majorly under many, many contracts, and (3) you know the SSA would never reject a major U.S. corporation with political influence (Boeing, for instance) on grounds of poor past performance?
  • Might it be because you are contracting for something that has never been produced or done before and so the past is not relevant?
  • Might it be because you want to give start-ups a chance based on new ideas and most of them do not have relevant experience or past performance?
  • Might it be that the acquisition entails very little performance risk; that evaluating past performance is about managing performance risk, not about rewarding contractors with good past performance; that evaluating past performance takes time and resources; and that evaluating it in the case in question is not worth the time and effort?
  • Some other rationale? (See ji20874's post.)

When do you make the decision?

  • Must you make it before you issue the RFP? Before you know for sure who the offerors will be?
  • If so, may you tell offerors that you might change your mind after you have received proposals?
  • May you tell offerors that you do not plan to evaluate past performance, but might change your mind after you receive proposals? If so, what would be an acceptable reason for changing your mind?

Does "evaluate" past performance refer only to comparative assessments, or does it also refer to pass/fail assessments?

If you say you are not going to evaluate past performance, may you still consider it as a responsibility factor pursuant to FAR 9.104-1(c)?

If you say you are not going to evaluate past performance, must you still consider it when determining offeror responsibility pursuant to FAR Subpart 9.1?

Note that FAR 15.304(c)(3)(iii) says that the contracting officer must document the reason that past performance is not an appropriate factor. It does not say that the contracting officer shall make the decision not to evaluate. Does "document the reason" mean make the case?

What if the SSA decides not to and the CO thinks it's a bad call? Must the CO make the case for the SSA or just record the SSA's explanation? What do you professionals say about that?

Link to comment
Share on other sites

One more example of when to not evaluate PP is when it's not meaningful, for example, evaluating the PP of the Best Buy corporation when buying 2 monitors for the conference room from the store down the street (slight permutation of bullet 4 above).   

Link to comment
Share on other sites

I queried Wifcon and its archives, and obtained only one old forum post that fully cites “FAR 15.304(c)(3)(iii).”  It was in regards to an RFQ for commercial locksmith services, and being “GSA certified” meant something.  This lack of discourse tells me we have fertile ground for discussion about improving the contract award process via the stated exception to the rule.

On 7/28/2022 at 11:20 PM, Vern Edwards said:

All three protests were denied. See:

  • Inuksuk A-S, Comp.Gen.May 26, 2022, B-420527.22022 CPD P 132

Thank you, @Vern Edwards.  At this part of the Inuksuk GAO case, I am intrigued:

“Here, the contracting officer documented the reasons that past performance was not an appropriate evaluation factor for this solicitation, concluding:
“Past performance will not be evaluated as part of this acquisition in accordance with (IAW) FAR 15.304(c)(3)(iii). This acquisition is restricted to Danish and/or Greenlandic companies IAW an agreement between the U.S. and The Kingdom of Denmark. The number of Danish/Greenlandic companies that meet these eligibility criteria and that have actual past performance experience for a base operations and maintenance contract is limited to a single company. The requirements of the Thule Base Maintenance Contract are very mature and explicit when it comes to critical utilities. The functional area experts have indicated that evaluation of the staffing plan from offerors will be sufficient to identify if the offeror can adequately execute the requirements of the contract. The infrastructure at Thule is aging and innovation to operate and maintain, given the explicit requirements is minimal. All offerors shall identify in Factor 2, Management, how they plan on retaining the current workforce. Historically, the retention ratio from one contract to the follow-on contract has been in the high 90th percentile. The latest retention ratio was ~98% in 2017 with some individuals working at Thule for decades. The retention of the incumbent workforce minimizes risk of not conducting past performance. Additionally, FAR 9.104 requires the contracting officer to determine responsibility by looking at several specific aspects of a contractor’s operation.”

My emphases added.  Who else knew the services contracts award process could be streamlined, eliminating weeks of work and whole PP sub-teams, if, among other reasons, a CO uses existing rules to document that the incumbent staff will be retained? 

Link to comment
Share on other sites

@Voyager

On 7/28/2022 at 11:21 AM, Voyager said:

I am not looking for expert-level advice here.  I do not have an active procurement.  This is purely a discussion topic to spur thoughts and talk shop about past experiences.

You started this thread to prompt a discussion, but you don't know what discussion is, do you?

Read up: https://teach.its.uiowa.edu/sites/teach.its.uiowa.edu/files/docs/docs/The_Process_of_a_Good_Discussion_ed.pdf

 

Link to comment
Share on other sites

@Vern Edwards

I don’t know how to lead a discussion, but I’ve got to try.  Wifcon is 7,000 members according to a recent post.  Perhaps my topic selection will embolden one to speak up. 

I have a favorite post from the archives to show you.  I read it recently and noticed participation levels were through the roof: Should Program Managers Be Contracting Officers?

Now, that was a theoretical discussion, whereas this one is practical and should lead to immediate improved job satisfaction.  And not the cheap kind resulting from some new OPM benefit, but one derived from actually studying and applying yourself.  I know I already learned something, so that makes one of us.

Link to comment
Share on other sites

On 7/28/2022 at 11:20 PM, Vern Edwards said:

The OP also asked:

  1. whether, in a particular case, either of two possible justifications for not evaluating past performance were logical and defensible,
  2. whether the principle underlying those justifications would apply outside of system contracting and, if so
  3. why?

 

On 7/29/2022 at 6:41 PM, ji20874 said:

Voyager,

You said you wanted a discussion, but you haven't said peep since the original posting.  What is your own answer to your question?  

I think the first reason I’ve bolded above in the example GAO case proves that ji20874’s reason and my OP’s proffered reason no. 1 was valid.  Market research, if done well, can apparently lead to a logical and defensible reason to not select a PP evaluation factor.  As the MR shows a smaller and smaller industry pool is available, the returns from examining PP are less and less.

The principle underlying this reason is ubiquitous as well.  It is obviously not limited to systems, as the real-life example came from base contracting at Thule.

As for the other reasons I’ve bolded, and the reasons I’ve not bolded (as well as the reasons in the other GAO cases), their analyses may not be so easy.  Who wants to tackle the next one?

Link to comment
Share on other sites

I don’t really see advantages in most instances to avoid evaluating past performance.  Regardless of your upfront assessment, the ability to consider it is still there, especially for surprises.  The manner which most agencies conduct assessments, the commitment of time and resources is minimal.  You ask companies for favorable experiences, a questionnaire gets sent to contacts, few responses are received, and the team looks at them as well as CPARS and assigns relevancy and sometimes recency to the ratings.  Few agencies dig further.  Even if they do, a small team of evaluators can come up with conclusions in a few days.

Link to comment
Share on other sites

I’m in general agreement with formerfed. Once we started evaluating past performance during source selections and moved away from IFB’s in the early 1990’s, we noticed a definite improvement in relations during performance with those firms that had been difficult to deal with over the years. And the complete dirtbags stopped competing because we finally had a way to weed them out.

To me, it would need to be a very specific and rare occurrence not to include PP as a selection factor. We need to reinforce the idea that quality of performance, including maintaining a good working relationship is and will be important to the government. Thus, it will also be important to the performing firm for their future prospects for exercise of options, for follow on or for other work.

Sure, it will require some effort to evaluate PP of the proposers. But the above described value of stressing quality performance including working relationships should justify the effort.

One other point. Even and perhaps especially when most or all of the current staff will be utilized on the new contract, knowing that PP, including how the contractor treats its employees, can be a discriminator and a motivator for the selectee.

Continuity and institutional memory of the technical and engineering staff for base operating contracts and for advisory contracts were of great value to the Installations that the USACE worked with during my career running source selections for those contracts and construction contracts as well as for the installation’s role during A/E and construction contract execution. Those subject matter experts were dedicated, loyal and invaluable for their advice and experience. 

Edited by joel hoffman
Added opening sentence…
Link to comment
Share on other sites

Well I suppose my thoughts here are not commensurate with good discussion but what the heck -

2 hours ago, joel hoffman said:

evaluating past performance during source selections and moved away from IFB’s

 

2 hours ago, joel hoffman said:

a way to weed them out

Was it really past performance or just the ability to evaluate rather than the required accept low bid or otherwise depend on a responsibility determination that no one really knew how to do? 

I would pose why  "past performance" at all?   Informally I offer offer this.

Beauty is in the eye of the beholder.   I want to build a house, I have lots of folks tell me about this or that contractor.  I choose a contractor that some say is a dirtbag but my success with the contractor is exquisite.  Heck GAO might have even told me I should have picked them!  Finding a grocery store, car repair shop, my hay guy, cow hauler, my trailer repair shop, my heavy equipment repair shop, the list goes on and on.   Could it be relationships are a greater indicator about success than just "How did they do?"  

And as I thought about this I think back to comments found on WIFCON and that are heard and found elsewhere about the value of CPARS specifically related to junk in junk out.  Hand in glove with comments about  getting responses to questionnaires that are glowing.

More and more the initialization of "evaluation" of past performance seems to have taken the same route of the IFB, almost!

 

Link to comment
Share on other sites

formerfed and Joel, I think you are clouding the discussion.  The original poster wanted to know if it would be possible to not evaluate past performance in a particular situation, and the answer is a simple YES.  You might choose to evaluate past performance in that situation, but that was not the question.

I make this note because I want to support the original poster in his or her seemingly new discovery that past performance evaluation in a 15.3 source selection may appropriately be waived -- he or she likely already has plenty of office co-workers, reviewers, and policy people who are saying past performance must or always should be evaluated.  Vern shared the case law that a contracting officer's decision to waive past performance evaluation has never been overturned on protest.  So I want to support the principle, and encourage contracting officers to appropriately use the flexibility they are given. 

Link to comment
Share on other sites

12 hours ago, C Culham said:

Well I suppose my thoughts here are not commensurate with good discussion but what the heck -

 

Was it really past performance or just the ability to evaluate rather than the required accept low bid or otherwise depend on a responsibility determination that no one really knew how to do? 

I would pose why  "past performance" at all?   Informally I offer offer this.

Beauty is in the eye of the beholder.   I want to build a house, I have lots of folks tell me about this or that contractor.  I choose a contractor that some say is a dirtbag but my success with the contractor is exquisite.  Heck GAO might have even told me I should have picked them!  Finding a grocery store, car repair shop, my hay guy, cow hauler, my trailer repair shop, my heavy equipment repair shop, the list goes on and on.   Could it be relationships are a greater indicator about success than just "How did they do?"  

And as I thought about this I think back to comments found on WIFCON and that are heard and found elsewhere about the value of CPARS specifically related to junk in junk out.  Hand in glove with comments about  getting responses to questionnaires that are glowing.

More and more the initialization of "evaluation" of past performance seems to have taken the same route of the IFB, almost!

 

Yes, it certainly used to be “junk in junk out in the IFB days.” We couldn’t get rid of the dirt bags. Contracting kept making awards to them. But once we got a chance to consider PP in RFP source selections we taught our offices that their ratings could be meaningful and to use the opportunity to honestly rate their contractors. Our offices made a greater effort to reward the good firms and to rate the lesser firms accordingly. 

Link to comment
Share on other sites

13 hours ago, ji20874 said:

formerfed and Joel, I think you are clouding the discussion.  The original poster wanted to know if it would be possible to not evaluate past performance in a particular situation, and the answer is a simple YES.  You might choose to evaluate past performance in that situation, but that was not the question.

I make this note because I want to support the original poster in his or her seemingly new discovery that past performance evaluation in a 15.3 source selection may appropriately be waived -- he or she likely already has plenty of office co-workers, reviewers, and policy people who are saying past performance must or always should be evaluated.  Vern shared the case law that a contracting officer's decision to waive past performance evaluation has never been overturned on protest.  So I want to support the principle, and encourage contracting officers to appropriately use the flexibility they are given. 

I understand your position, JI. I want to let the OP know to look past the policy and efforts to avoid considering PP, from the perspective of someone who had to endure the same POS contractors.  

Link to comment
Share on other sites

We certainly do not control which way the pendulum swings.  I don’t think there’s any harm in just exploring the mechanics of the rule’s allowance here.  As a then-Representative Abraham Lincoln once aptly said, “Determine that the thing can and shall be done, and then we shall find the way.”

@formerfed or @joel hoffmanHave you personally ever excluded the PP factor from a source selection?

Link to comment
Share on other sites

3 hours ago, Voyager said:

@formerfed or @joel hoffmanHave you personally ever excluded the PP factor from a source selection?

From a “source selection”, no. I don’t remember not including a PP factor or subfactors, especially since we included recent, relative experience as a factor or subfactor. Both were relevant discriminators.

We provided a standardized format for the experience factor/subfactor which required the proposer to identify its performance rating-if any and a reference that we reserved the right contact to verify, if necessary. We also reserved the right to use CPARS or other sources, if necessary for other recent past performance. It wasn’t a major exercise to evaluate.

Now - I will say that we didn’t get huge numbers of proposals for the types of contracts we were sourcing: construction, design-build, JOC, MATOC’s and Hospital O&M  or Base O&M contracts. They were Single award or MATOCs.

From a task order competition, I’d say we probably didn’t bother with experience or past performance early on in the ordering period. The qualifications of the pool members had been vetted for award of the IDIQ. Down the road, there may be situations where a pool member or members might be performing less than desired and/or we wanted to incentivize good performance. It was 15 years ago since I personally led or participated on a SS panel.

After that I retired.  I worked as a rehired annuitant on some Corps-Wide programs and maintaining various Model RFP’s that I had helped create for the Army Transformation Design-Build Construction Program (one step and two-phase) single award and MATOC’s and for Task Order competitions. Individual Districts used the Model RFP’s and could tailor the task order model.
I also performed other duties as a rehired annuitant. 

Link to comment
Share on other sites

Oh, no -- now the discussion is being clouded even more with talk about task order competitions.

The original poster is talking about FAR 15.3 source selections.  Any talk here about task order competitions is inapt, as the crucial citation from FAR 15.3 is wholly irrelevant to task order competitions.

The answer to the original poster's question is a simple YES -- a contracting officer may waive the evaluation of past performance in a FAR 15.3 source selection under the circumstances described in the original posting with an appropriate determination.  All the demurring going on is not contributing support or clarity and is clouding the simple YES answer by trying to change the YES [the contracting officer may waive...] to a NO [the contracting officer should not waive...].

Voyager, the FAR says what it says.  Do what is right for your circumstances.  Do not be afraid to prudently and appropriately use the flexibility the FAR gives you.

Link to comment
Share on other sites

ji, I was responding to Voyager’s question. I’m not saying that there may not be valid reason(s) for not needing to evaluate past performance for a specific acquisition. I previously explained some added values that our organization achieved by using past performance as a discriminator. Since recent relative experience is usually important to us, the quality of that performance and a contractor’s reputation goes hand in hand with it. 

Link to comment
Share on other sites

On 7/31/2022 at 7:48 AM, ji20874 said:

formerfed and Joel, I think you are clouding the discussion.  The original poster wanted to know if it would be possible to not evaluate past performance in a particular situation, and the answer is a simple YES.  You might choose to evaluate past performance in that situation, but that was not the question.

I make this note because I want to support the original poster in his or her seemingly new discovery that past performance evaluation in a 15.3 source selection may appropriately be waived -- he or she likely already has plenty of office co-workers, reviewers, and policy people who are saying past performance must or always should be evaluated.  Vern shared the case law that a contracting officer's decision to waive past performance evaluation has never been overturned on protest.  So I want to support the principle, and encourage contracting officers to appropriately use the flexibility they are given. 

I agree with ji20874.  But I think C_Culham has made a very good point--one that I think supports ji20874 in what he's trying to do.

On 7/31/2022 at 7:11 AM, C Culham said:

I would pose why  "past performance" at all?   Informally I offer offer this.

Beauty is in the eye of the beholder.   I want to build a house, I have lots of folks tell me about this or that contractor.  I choose a contractor that some say is a dirtbag but my success with the contractor is exquisite.  Heck GAO might have even told me I should have picked them!  Finding a grocery store, car repair shop, my hay guy, cow hauler, my trailer repair shop, my heavy equipment repair shop, the list goes on and on.   Could it be relationships are a greater indicator about success than just "How did they do?"  

And as I thought about this I think back to comments found on WIFCON and that are heard and found elsewhere about the value of CPARS specifically related to junk in junk out.  Hand in glove with comments about  getting responses to questionnaires that are glowing.

More and more the initialization of "evaluation" of past performance seems to have taken the same route of the IFB, almost!

In the late 1980s I aggressively promoted the evaluation of past performance in source selection, primarily as a way around the SBA's certificate of competency program. In 1994 I wrote a monograph about evaluation past performance for The George Washington University Law School: How to Evaluate Past Performance; A Best-Value Approach, The Monograph Series, 2d. However, by 2016 I was concerned that bureaucratization (and professional incompetence) had undermined the evaluation of past performance. See "Past Performance: Has Bureaucratization Crippled A Useful Policy?", The Nash & Cibinic Report, January 2016. I reviewed two GAO decisions sustaining protests against an agency evaluation of past performance, Al Raha Group for Technical Services, Inc., Comp. Gen. Dec. B-411015.2, 2015 CPD ¶ 134, 2015 WL 2089065, and Logistics Management International, Inc., Comp. Gen. Dec. B-411015.4, 2015 CPD ¶ 356, 2015 WL 7450343. I concluded:

Quote

 

The Air Force behaved like what it is—a bureaucracy. When bureaucracies perceive a procedural risk, they perversely resort to greater procedural detail and formality, i.e., to greater bureaucratization. The more detailed and formal a procedure becomes, especially when mandated by statute, regulation, or policy memo, and the greater its potential effect on the fortunes of others, the greater the potential for procedural error and challenge.

While red tape is an inevitable product of bureaucracy, people at the working level should not create more of it than absolutely necessary. Nothing in the FAR required the Air Force to adopt such a complicated method. The agency’s needlessly complicated formula for “relevancy” blew up in its face. Twice. Let that be a lesson to others: Know the law, keep your recipe as simple as the law allows (which, generally, is pretty simple), and follow the recipe to the letter.

 

I think that COs should really think about whether the evaluation of past performance will yield information that will be worth the effort to obtain it. I'm no longer convinced that it's always worth the effort. In fact, I've begun to think that evaluation of past performance should be the exception, not the rule.

BUT CAUTION: If you are going to take advantage of FAR 15.304(c)(3)(iii), and I think you should whenever it makes sense, write a good justification, one that makes sense, and document the file.

As for what makes sense...

Think!

Link to comment
Share on other sites

Our organization moved away from IFB and low bid mentality in order to obtain improved performance and quality.

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...