Jump to content
The Wifcon Forums and Blogs
Sign in to follow this  
Jamaal Valentine

Sources of Past Performance Information

Recommended Posts

Provided the FAR compels contracting officers to use the Government Past Performance Information Retrieval System (PPIRS), what are some useful methods of sourcing information 'you' would want to make a performance confidence assessment (not contractor rating like the BBB does)?

- Better Business Bureau (BBB), Angie's List, Yelp

- Duns and Bradstreet (when I was in industry and had a subscription)

- State Board websites for licensed or bonded contractors

- Call a friend/associate

- dreaded Past Performance Questionnaire

- implied endorsements through recognition of brand strength/popularity/business size/financials

This is for discussion, not necessarily right or wrong answers, so feel free to chime in or make that first post.

Share this post


Link to post
Share on other sites

Jamaal,

Your query is slightly confusing. The FAR doesn't compel contracting officers to use PPIRS. FAR 9.105-1(c ) requires ("shall") contracting officer to consider information in FAPIIS when making a determination of responsibility. It also encourages ("should") contracting officers to use additional sources of information to support their responsibility determinations. Many of the examples you list would fall into the sources of information listed in the FAR at 9.105-1(c ). It's not a matter of what information a contracting officer "would want" to make a responsibility determination, its a matter of what sources a contracting officer "should" consider when making a responsibility determination. Also, terms of art are important when asking questions. I've never seen the FAR or an agency supplement refer to a determination of responsibility as a performance confidence assessment.

Share this post


Link to post
Share on other sites

PTaylor, the DOD uses the term "confidence assessment" when performing past performance evaluations in DOD source selections. It is described in the DFAR Supplement. It isn't part of the responsibility determination.

Share this post


Link to post
Share on other sites

PTaylor:

Although related to responsibility, in the sense that some of the same information goes into both assessments, that is not what the post is about.

As for your belief about PPIRS see FAR 42.1503( g ). If I recall correctly, there is some case law to this. I'm on my phone now but will search later unless someone has information, for or against, readily available.

Share this post


Link to post
Share on other sites

Past Performance evaluation is a mine field. It would appear to be the most or nearly the most protested evaluation criteria in the last few years. Why? Because it is unfair to contractors many times and is subjective which allows contractors to attack its accuracy. My office constantly debates performance with our CORs. The COR wants to rate $500M in work marginal because of one or two inaccurate invoices. Most of these get upgraded but not always, depends on the CO.

The contractor gets to provide his comments and ask for a review. But the orginal bad report is still on file when it goes to PPIRS with the comments of the contractor and the Reviewer. So now what do you do with this report as an evaluator. You call them. They all stick to thier stories. Who do you believe? Once again subjective and depends on the point of view point of the evaluator. One of our large contractors said they may go to court to insist that revised reports not show the original bad information. They know that when they complain to the Reviewer (Division Chief usually) he would remove the bad report and replace it with a good one to avoid conflict if the system let him. Wrong but true.

OMB has suggested we should use Yelp, Angies list, Google, etc but guess what, they are starting to get law suits for inaccurate reports too. In the Government we have to give the contractor a chance to respond to bad information. Most of the COs I talk to find using this information to be too risky.

Then you have the huge corporations that have some great reports and some terrible reports. It depends on the Progam Manager they had available at the time and the viewpoint of the COR. While a small company with a bad report might indicate they would fail again, a bad report on one the large Beltway Bandits means nothing in my opinion. With hundreds of contracts and Billions in Government revenue there is bound to be a few bad ones and little incentive to exceed satisfactory on the rest. Higher ratings equal less profit. Even CPARs states that exceeding Satisfactory may be impossible in some contracts.

So to finish my diatribe, while past performace sounds like a great evaluation factor, if is frought with dangers. I believe that if the contractor was not so bad that they are in FAPIIS or debarred, we may want to keep them on a level playing field. The past performance information is just too inaccurate and I don't see any great improvement soon. When you use the information, you better dig deep and document, document, document. While the neighbors opinion of car dealers may influence you and not cause a issue, we are using taxpayer dollars and must base our evaluations on facts and not just opinions.

Just the opinion of an old battle weary CO fighting in the trenches.

Share this post


Link to post
Share on other sites

As another old battle weary CO fighting in the trenches, I will add that I avoid past performance evaluations wherever possible. They are required for FAR Part 15 acquisitions, but can be waived. They are not required for FAR Subpart 8.4 acquisitions or for fair opportunity considerations under FAR 16.505( b )( 1 ) (although for fair opportunity considerations, we may use past performance of other orders issued under the multiple-award contracts). I believe in past performance, and I believe that past performance is the best indicator of future performance, but it seems that we do past performance wrong.

As I tell my staff, a past performance evaluation is not a report card of past performance, and poor past performance is not an automatic disqualifier in an acquisition. Rather, we are supposed to use past performance as one of the factors in selecting the best value, and we're supposed to use past performance information to make a subjective assessment of the future risks to the Government of awarding a contract to that offeror.

But for fellow acquisition professionals who can't discern nuance, including many reviewers and some managers, a past performance evaluation is a report card of past performance and poor past performance is an automatic disqualifier in an acquisition -- in addition to the problems shared by Boof above, this is another reason why past performance is so troublesome in protests, and why I avoid past performance evaluations where I can (but I still believe in effective past performance evaluations as part of a source selection).

Here's a primer for our readers. Factors are in descending order of importance, and the non-price factors when combined are significantly more important than price. The reasons for the Excellent ratings in Factor 1 are essentially the same, so Factor 1 is not a discriminator in the selection. Which offer provides the best value?

. Offer A Offer B

Factor 1 Technical Excellent Excellent

Factor 2 Past Performance Low Risk Some Risk

Factor 3 Price $100 $95

One more question for thought. Imagine a contractor performed poorly (almost disastrously) on a recent and very relevant (very similar in all respects) contract, and the poor performance is reflected in CPARS/PPIRS. Can the past performance evaluation for a source selection have knowledge of this information and still assign that contractor a low risk (or high confidence) rating for past performance?

Share this post


Link to post
Share on other sites

Hello, Ji,

I also consider whether the people at that company involved in the negative past performance will even be working under my contract. If you are fishing nuance, there's a little more . . .

I'm still waiting for the study on PPIRS data which tells us how predictive these ratings are of actual performance. Has anyone ever done this? If the whole idea is flawed it would be good to find out if we are wasting our time with PPIRS.

Hello, Jamaal,

Away from work, most larger purchases are rated on some variation of the five-star system, usually displayed on a computer screen, with narrative reviews under the rating. (There is often no opportunity for the seller to respond, and when there is it is usually an apology and an offer to help fix the problem)

Share this post


Link to post
Share on other sites
Guest Vern Edwards

Boof and ji20874 have posted thoughtful comments.

Boof thinks that past performance is a "minefield." It appears that he bases that assessment on protest risk.

In November 2006 I did an in-depth analysis for The Nash & Cibinic Report of 120 past performance decisions by the GAO since 2001, following up on an earlier analysis by Professor Nash. My article was in the November 2006 issue. We both found sustain rates of about 14 percent, and he and I both concluded that "[A]gencies have so much discretion about how they evaluate past performance that protesters unhappy about an evaluation have little chance of success."

After reading Boof's comment I did a quick and dirty analysis of GAO protests.During the years 2001 though the present, past performance has been mentioned in 528 decision digests, which indicates that past performance was a basis for protest in those decisions. There were roughly 6,000 protest decisions during that period, calculating at 400 per year. (According to the gAO, there were 556 decisions in 2014.)

I did a quick and dirty analysis of those 528 decisions. I searched for the words "we sustain" and found them in 75 decisions. Now, some of those would have been sustained on other grounds, but if we assume that they were all sustained on the basis of past performance, that's rate of 14 percent. The overall sustain rate on all issues in 2014 was 13 percent. The overall sustain rate in 2010 was 19 percent.

I searched the 556 protests decided by GAO in 2014. Past performance was an issue in about 57, which is a little more than 10 percent of the total. Nine of those were sustained, about 16 percent (15.8). Since some were probably sustained for other reasons, the sustain rate for past performance in 2014 was not much different than the overall sustain rate of 13 percent.

According to the GAO's protest report for 2014 the most prevalent bases for sustaining a protest were, in rank order: (1) failure to follow the evaluation criteria, (2) flawed selection decision, (3) unreasonable technical evaluation, and (4) unequal treatment. "Unreasonable technical evaluation" might include past performance evaluations, but that's not clear from the report.

I did the same quick and dirty analysis for "discussions" and found 401 decisions with 72 sustained, a rate of 19 percent. I did the same for "cost realism" and "price realism" combined and found 184 decisions with 36 sustained, a rate of 19 percent. And I did the same for "tradeoff" and found 131 decisions with 31 sustained, a rate of 25 percent.

My conclusion is that past performance is a frequent basis for protest, but not the most frequent. I think that the chances of losing a past performance protest are no greater than for any other bases, and less than for some others. The people who have lost past performance protests have generally behaved stupidly. Past performance is not inherently risky with respect to protests. I wandered into a minefield in Vietnam, and that's not an apt metaphor for past performance. Past performance is dangerous only to the dumb.

Boof's best comment was this one:

Then you have the huge corporations that have some great reports and some terrible reports. It depends on the Progam Manager they had available at the time and the viewpoint of the COR. While a small company with a bad report might indicate they would fail again, a bad report on one the large Beltway Bandits means nothing in my opinion.

I wouldn't say it "means nothing," but I will say that it's hard to meaningfully assess the past performance of an entity as large as Lockheed.

My own guess is that past performance can work to identify a real loser or a real winner, but that most offerors fall into a "muddled middle" zone between the two extremes.

I agree with ji20874 that some agencies do past performance wrong. We should use it as a risk or confidence factor with which to adjust our initial assessments of value based on an offeror's promises and apparent capability. Remember, past performance is not something that is being offered to the Government or that the Government will get. It is something that somebody else got and that might be indicative of what the Government is likely to get if it choses that firm for the prospective contract. How reliable a factor or consideration it is depends on the quality of the evaluation team's information gathering and assessment efforts. If we were serious about past performance we'd consider it a kind of due diligence assessment, similar to what is done in mergers and acquisitions.

However, I wish that ji20874 would explain why he avoids evaluating past performance "wherever possible," since he says he believes in it and thinks it's the best indicator of future performance.

I think that reputation would have been a better term than past performance. To the extent that the words past performance suggest an objective and factual analysis, I think was ill-chosen. An objective, factual analysis is much too difficult, if not impossible. But reputation was rejected because industry feared it allowed too much subjectivity.

Share this post


Link to post
Share on other sites

CPARS could be reduced to a single question for me - would you award another contract to them for similar services or supplies? yes, no, maybe.

Share this post


Link to post
Share on other sites

One more question for thought. Imagine a contractor performed poorly (almost disastrously) on a recent and very relevant (very similar in all respects) contract, and the poor performance is reflected in CPARS/PPIRS. Can the past performance evaluation for a source selection have knowledge of this information and still assign that contractor a low risk (or high confidence) rating for past performance?

Yes, while it may increase the risk of a protest, but if the evaluators documents their rationale as to why the performance does not impact the overall risk, then yes they can. It is not just a simple yes or no, the evaluator needs to look at the facts and the information and provide a rationale for their judgement as to how that performance will impact the risk in their opnion.

Factors are in descending order of importance, and the non-price factors when combined are significantly more important than price. The reasons for the Excellent ratings in Factor 1 are essentially the same, so Factor 1 is not a discriminator in the selection. Which offer provides the best value?

. Offer A Offer B

Factor 1 Technical Excellent Excellent

Factor 2 Past Performance Low Risk Some Risk

Factor 3 Price $100 $95

One more question for thought. Imagine a contractor performed poorly (almost disastrously) on a recent and very relevant (very similar in all respects) contract, and the poor performance is reflected in CPARS/PPIRS. Can the past performance evaluation for a source selection have knowledge of this information and still assign that contractor a low risk (or high confidence) rating for past performance?

Since it appears that you are doing a trade off, there is not enough information available here to make a judgement. What is the make up of the factors that rate them as excellent in technical. what are the factors that make up the ratings in past performance. If offeror B is offering something that A is not, and the Government determines that proposal is the best value, then why can' t they award to B.

Share this post


Link to post
Share on other sites

I also consider whether the people at that company involved in the negative past performance will even be working under my contract.

That's fair, because in a past performance evaluation we're supposed to use past performance information to make a subjective assessment of the future risks to the Government of awarding a contract to that offeror -- if the people who caused the poor performance aren't part of the proposed contract effort, then there might not be a risk for the future contract.

Vern,

I try to avoid past performance evaluations because my agency makes it too hard -- so many reviewers, some of whom don't understand the purpose of a past performance evaluation, and so many people afraid to make a hard decision, and so many people afraid of a protest, and so many people so very cautious, and so many people who care much more about the process and the file documentation than they about results or performance of the awarded contract, so much second-guessing -- all of this combines to make it too troublesome. If they understood your comment and actually cared about the results we deliver to users and line managers acting on behalf of the American taxpayer, well, then I would be more eager about past performance evaluations. I'm dealing with too many other fires and I don't have time to re-educate the entire agency contracting workforce -- but I wish DAU and FAI and so forth would do much better teaching on the correct principles of past performance evaluations.

Share this post


Link to post
Share on other sites

airborne373,

It is not just a simple yes or no, the evaluator needs to look at the facts and the information and provide a rationale for their judgement as to how that performance will impact the risk in their opnion.

That's right! The contractor with a poor performance history on a recent and very relevant contract can still get a low risk (or high confidence) rating for past performance.

On the other question, assume that the reasons for the Excellent ratings in Factor 1 are essentially the same, so Factor 1 is not a discriminator in the selection.

Share this post


Link to post
Share on other sites
Guest Vern Edwards

ji20874:

Thanks. Your reasons make sense to me.

All:

For an excellent illustration of the appropriate use of past performance as a confidence factor, see Moore Medical Corp., B-261758, Oct. 26, 1995, 95-2 CPD ¶ 204.

Just Google "B-261758".

Share this post


Link to post
Share on other sites

Vern: thanks for the data

All: Vern hits at what I am most concerned with. Our information gathering and assessment efforts leave quite a bit to be desired.

I was hoping the discussion would go into that a bit but overall the posts have been interesting.

Share this post


Link to post
Share on other sites

ji20874:

DAU and FAI need to have a mandatory class on current principles 101 for key positions. The tenured policy folks and leaders are directly responsible for the state of affairs.

Sending junior personnel to another class to learn principles they can't implement upon returning to the office simply isn't working fast enough.

Share this post


Link to post
Share on other sites

I try to avoid past performance evaluations because my agency makes it too hard -- so many reviewers, some of whom don't understand the purpose of a past performance evaluation, and so many people afraid to make a hard decision, and so many people afraid of a protest, and so many people so very cautious, and so many people who care much more about the process and the file documentation than they about results or performance of the awarded contract, so much second-guessing -- all of this combines to make it too troublesome. If they understood your comment and actually cared about the results we deliver to users and line managers acting on behalf of the American taxpayer, well, then I would be more eager about past performance evaluations. I'm dealing with too many other fires and I don't have time to re-educate the entire agency contracting workforce -- but I wish DAU and FAI and so forth would do much better teaching on the correct principles of past performance evaluations.

True statement, Too many see the role of evaluating PP as going through the motions. sometimes i fear that it is not just PP evaluations that people dont understand the purpose of what they are doing.

Share this post


Link to post
Share on other sites

The case Vern cited is a unicorn.

You mean to tell me an agency used numerical ratings and didn't award to a company with the highest technical rating and lowest price?

LOCAR - sounds a bit how we, in DoD, do it on paper. I haven't seen it in practice.

LOCAR, I don't know who you are. I don't know what you want. If you are looking for ransom I can tell you I don't have money, but what I do have are a very particular set of skills. Skills I have acquired over a very long career. Skills that make me a nightmare for people like you...I will pursue you, I will find I you, and I will master you.

Share this post


Link to post
Share on other sites
Guest Vern Edwards

For a complete description of the LOCAR technique, see Source Selection Answer Book, 2nd ed. (Management Concepts, 2006), Appendix D.

Share this post


Link to post
Share on other sites

Thanks Vern. I actually requested my office to order this resource and was turned down. The fact that LOCAR is detailed within those pages makes me want the information inside this book even more.

I have my privately funded library that covers an array of work and life related t topics from Steps to Writing Well, 12th ed. (I should probably revisit that one) to Government Contract Law: The Deskbook for Procurement Professionals, 3rd ed. and everything in between. One of my favorites is How to Become a Leader by Myles Munroe. Very simple read, but proven.

Being relatively junior, but entrusted with a team of 13, the Source Selection Answer Book should be at the right level for where I am. Once my library fund can support it I'll add it to the desk, the shelf is reserved for books that get opened less frequently.

Share this post


Link to post
Share on other sites

Having posed the questions, let me close out the thought process--

Here's a primer for our readers. Factors are in descending order of importance, and the non-price factors when combined are significantly more important than price. The reasons for the Excellent ratings in Factor 1 are essentially the same, so Factor 1 is not a discriminator in the selection. Which offer provides the best value?

. Offer A Offer B

Factor 1 Technical Excellent Excellent

Factor 2 Past Performance Low Risk Some Risk

Factor 3 Price $100 $95

Answer: Either A or B could provide the best value -- it depends on the source selection authority's independent judgment of the risks in Factor 2 balanced against the price in Factor 3. One SSA could select Offer A, saying he or she is not willing to accept the risks associated with Offer B and that the benefit associated with A's low risk merits the additional $5. Another SSA (or the same SSA in a different mood) could select Offer B, saying he or she is willing to accept the risks associated with Offer B and that the benefit associated with A's low risk does not merit the additional $5.

One more question for thought. Imagine a contractor performed poorly (almost disastrously) on a recent and very relevant (very similar in all respects) contract, and the poor performance is reflected in CPARS/PPIRS. Can the past performance evaluation for a source selection have knowledge of this information and still assign that contractor a low risk (or high confidence) rating for past performance?

airborne373 already answered this question. Yes, a source selection can have knowledge of an offeror's poor (almost disastrous) performance on a recent and very relevant contract and still assign that contractor a low risk (or high confidence) rating for past performance. Maybe it was another division -- maybe the contractor learned its lesson and has taken corrective action -- there are all sorts of maybes.

Share this post


Link to post
Share on other sites
Guest
This topic is now closed to further replies.
Sign in to follow this  

×
×
  • Create New...