Jump to content

Past Performance Evaluation


Prosperity

Recommended Posts

6 hours ago, Sam101 said:

But isn't it best practice to explicitly state in Section M for Past Performance that this is the case? Otherwise offerors might think the "relevancy is a threshold" way is being used... I'm not too convinced about the thought that the more recent the experience the better it is when the cut-off is three or five years for the definition of recency anyways, I'm not sure that a one-year-old project is better than a two-year-old project, but if somehow it is I think it's best to state that explicitly in Section M.

@Sam101Why say anything about recent and relevant? Why include the words in Section M?

 

 

Link to comment
Share on other sites

  • Replies 50
  • Created
  • Last Reply

Top Posters In This Topic

14 minutes ago, Vern Edwards said:

Why say anything about recent and relevant? Why include the words in Section M?

Because it's not fair to have the offerors guess as to whether or not the Government is using the "relevancy is a threshold" method or "there are different levels of relevancy" method. If I don't say anything about relevancy in Section M then I'd be forced to evaluate the "relevancy is a threshold" way and end up having a document in my contract file saying "we have substantial confidence that Taco Bell can make cheeseburgers" just because I see in CPARS or on a past performance questionnaire that Taco Bell got rated Outstanding for making food. If I rate Taco Bell as satisfactory confidence and Burger King as substantial confidence (Burger King also has a CPARS that says they are outstanding, for making cheeseburgers), Taco Bell can protest and say "making any kind of fast food is a relevant past contract and my past customer says I'm Outstanding, so I should be rating substantial confidence, you never said in Section M that you would be looking at degree of relevancy, so me and Burger King should have an equal rating since we both have Outstanding CPARS for making food."

But after typing all that I can see how degrees of receny and relevancy can be seen as common sense not worth mentioning in Section M, but more so for relevancy, if we're talking about degrees of recency I would definitely mention that in Section M, since recency is not as common sense as relevancy.

Link to comment
Share on other sites

@Sam101 Wow. You seem awfully opinionated for someone who showed up in the beginner forum asking for advice just last Tuesday. And then you end with a sentence like this:

12 minutes ago, Sam101 said:

But after typing all that I can see how degrees of receny and relevancy can be seen as common sense not worth mentioning in Section M, but more so for relevancy, if we're talking about degrees of recency I would definitely mention that in Section M, since recency is not as common sense as relevancy.

What do you know about common sense in acquisition?

Good luck, kid. I got nothing more for you.

Link to comment
Share on other sites

1 hour ago, Vern Edwards said:

You seem awfully opinionated

Is this:

1 hour ago, Sam101 said:

But after typing all that I can see how degrees of receny and relevancy can be seen as common sense not worth mentioning in Section M, but more so for relevancy, if we're talking about degrees of recency I would definitely mention that in Section M, since recency is not as common sense as relevancy.

More opinionated than this?:

1 hour ago, Sam101 said:

Because it's not fair to have the offerors guess as to whether or not the Government is using the "relevancy is a threshold" method or "there are different levels of relevancy" method. If I don't say anything about relevancy in Section M then I'd be forced to evaluate the "relevancy is a threshold" way and end up having a document in my contract file saying "we have substantial confidence that Taco Bell can make cheeseburgers" just because I see in CPARS or on a past performance questionnaire that Taco Bell got rated Outstanding for making food. If I rate Taco Bell as satisfactory confidence and Burger King as substantial confidence (Burger King also has a CPARS that says they are outstanding, for making cheeseburgers), Taco Bell can protest and say "making any kind of fast food is a relevant past contract and my past customer says I'm Outstanding, so I should be rating substantial confidence, you never said in Section M that you would be looking at degree of relevancy, so me and Burger King should have an equal rating since we both have Outstanding CPARS for making food."

If relevancy and recency is too common sense for Section M than someone can argue that everything in Section M is common sense, to the point where all Section M would say is "the offeror's proposal will be evaluated." I say recency because it was mentioned in this thread but I don't see degree of recency addressed in the DoD SSP, it says recency is a threshold type question and only mentions levels of relevancy, that's even more reason to state in Section M what the levels of recency are (if the Government for whatever reason wants to evaluate degrees of recency like, 1 year ago is better than 2 years ago) is it not?

Link to comment
Share on other sites

On 3/26/2021 at 8:08 AM, ji20874 said:

The Government will assess its confidence in the likelihood of the offeror's success based on its past performance.

 

On 3/26/2021 at 8:08 AM, ji20874 said:

The Government will assess its confidence in the likelihood of the offeror's success based on its experience.

Is the Government really going to assess its own confidence? Why not just say that the Government is going to evaluate an offeror's experience (or past performance) and develop a confidence rating that reflects the Government's opinion of the offeror's likelihood of successful performance? 

Link to comment
Share on other sites

Maybe you could say that the government is going to determine its level of confidence in each offeror on the basis of its assessment of each offeror's experience.

In my Chambers dictionary, assess means "evaluate or estimate the nature, ability, or quality of [something). Determine means, among other things, "to decide."

Link to comment
Share on other sites

Sam101, In reviewing my comments herein concerning the use of relevant experience as a separate factor, I should clarify that I don’t have a problem with evaluating the relevancy of recent project experience in the past performance/confidence assessment per the DoD source selection procedures. In fact, if you are in DoD, the methodology and rating schema are standardized for uniform application when evaluation of past performance is applicable.

If the EXTENT of recent (recent is defined as the age limit of projects allowed to be submitted), relevant corporate experience is important as a discriminator, then also include it as a separate factor.

Sam101, the sample submission and evaluation language that I sent you includes the separate experience factor and reflects the standardized DoD method for past performance/confidence assessment.

If the format of the samples confused you (there is no mentions of Sections “L” or “M”), it’s because  we don’t use the Uniform Contract Format (UCF) for Design-Build construction or straight construction for single award or basic ID/IQ solicitations/contracts*. It’s combined in one section.

We first list all factors and sub factors, if applicable in their relative order of importance, including price and describe their relative importance. We also describe the relative importance of price to the combined non-price factors. It’s pretty easy to see that, unless price is listed as the number one factor, that it can’t be “of equal importance to” or “more important than” the combined non-price factors. 

Then we list each factor/sub factor in turn, describing its submission requirements and its evaluation criteria.

It’s much easier to follow and to directly correlate the factors, their submission requirements, their evaluation criteria, the rating scheme, the basic process and the basis of award when read together than in two separate sections. It also makes the solicitation writers more accountable for describing meaningful evaluation criteria. Think of it as merging L and M.

I personally think that organization of the information in separate “L” and “M” sections is archaic. It should all be in one, fully integrated and coordinated section.

However, the samples are  adaptable for solicitations using the UCF and for service solicitations.

*The USACE uses the Construction Specifications Institute (CSI) Masterformat for construction and design-build construction solicitations and contracts. It is a standard industry format. 

Link to comment
Share on other sites

In my limited experience in aviation and shipbuilding, recency is always defined and it is generally 5-10 years depending on the weapon system. I have seen the degree of relevance defined in a shipbuilding contract that I liked but decided against calling out something similar in my solicitation to make sure the past performance evaluation team (PPET) has more flexibility in their determination of relevance. This likely worked because the Navy was buying something unique and specific but as others have mentioned a similar language could be used under the experience factor and you will have the benefit of maybe assigning a strength or weakness for this type of specific experience. If you have both past performance and experience as a factor, be careful that you are not evaluating the same aspect of the offer twice, clearly distinguish between both and talk through what exactly you are looking for within both factors with your team.

In determining relevancy of an Offeror’s past performance, the order of relevancy is design and construction of heavy icebreakers, followed by design and construction of other marine vessels of similar size and complexity.

We also sometimes send the PP examples to the technical team to determine relevance to our requirement before the PPET begins their assessment/determination. With some of these systems most people on the PPET do not have the same level of technical expertise as the technical team. If we can, we try to have an engineer on the price and PP team but human resources are sometimes limited.

Link to comment
Share on other sites

Under an experience factor or sub factor, the government should be evaluating the extent or depth of recent experience that is relevant to the instant project.

Under a past performance factor or sub factor, the government is evaluating the QUALITY of the project experience. Naturally a project similar in nature will be more relevant than other experience for past performance.

 

 

Link to comment
Share on other sites

17 hours ago, joel hoffman said:

Under an experience factor or sub factor, the government should be evaluating the extent or depth of recent experience that is relevant to the instant project.

Under a past performance factor or sub factor, the government is evaluating the QUALITY of the project experience. Naturally a project similar in nature will be more relevant than other experience for past performance.

 

 

I think Joel explanation lines up with the way I have heard a few people differentiate between experience and past performance.

Experience is, have you done this before? while past performance is, how well did you do it?

Link to comment
Share on other sites

On 4/3/2021 at 4:40 PM, MAY-D-FAR-B-WIT-U said:

Experience is, have you done this before? while past performance is, how well did you do it?

Exactly.  Now the challenge with past performance is doing meaningful evaluations that benefit selection decisions.  Asking offerors in your solicitation to furnish X number of references doesn’t really cut it.  What offeror will give you a poor reference? Using CPARS isn’t much better because a company really has to screw up to get poor ratings.  So many government employees who tried giving poor ratings to a company swear they won’t do it again.  They are like the kiss of death and companies resport to complaining to senior management, IGs, Congressional reps, etc. to get things changed.  Sending out questionnaires doesn’t help much either unless you follow up with phone calls or interviews.  Plus questionnaire return rates aren’t that high.  

Link to comment
Share on other sites

3 hours ago, formerfed said:

Exactly.  Now the challenge with past performance is doing meaningful evaluations that benefit selection decisions.  Asking offerors in your solicitation to furnish X number of references doesn’t really cut it.  What offeror will give you a poor reference? Using CPARS isn’t much better because a company really has to screw up to get poor ratings.  So many government employees who tried giving poor ratings to a company swear they won’t do it again.  They are like the kiss of death and companies resport to complaining to senior management, IGs, Congressional reps, etc. to get things changed.  Sending out questionnaires doesn’t help much either unless you follow up with phone calls or interviews.  Plus questionnaire return rates aren’t that high.  

@formerfed So how do you meet the challenge? 

Link to comment
Share on other sites

Vern,

Good question.  I haven’t seen one approach that works consistently well across all actions.  Some that seem successful in appropriate circumstances are requiring offerors to submit all relevant experiences, conducting interview type meetings with customers,  witness products or services in use, asking end users how satisfied they are, and inquiring with customers how well problems or issues were resolved.  

Link to comment
Share on other sites

Interviewing past customer references worked the best for me, if we didn’t already have an interview on file or weren’t personally familiar with that project PP. We reserved the right to call the proposer provided references to verify ratings.

We provided the proposers a specific form/format for them to describe each project experience, which included a line for a customer rating, if known, an owner reference and their contact information. 

As for the return rate on written questionnaires, I personally agree. I hated getting one to fill out - especially if it was repeated for the same project . It’s a lazy SS team that uses customer returned questionnaires.

You can learn a lot from calling a reference to interview.

And believe it or not, they can be quite honest and frank. It’s not always “rosy” and perfectly wonderful. They were usually willing and able to orally explain in detail, whether it was a great experience, less than or not good.

I followed an interview format for consistency.  But I let them elaborate or add comments, if I didn’t ask something that was specifically of interest to them.

Edit, add: When proposed Key Personnel were also being evaluated, I would sometimes ask if one or more of them were on their job, especially if that project was included in a resume(s). If so, I would ask more about them. 

Edited by joel hoffman
Link to comment
Share on other sites

20 minutes ago, formerfed said:

One more,  when conducting interviews, especially with offeror provided sources, ask who else the offeror did work for.  Often they learn about other projects the same company is working on.  

Interesting...

Link to comment
Share on other sites

Interviewing sounds labor intensive and time-consuming. Is it efficient? How many interviews are you going to conduct? How often is the payoff worth the effort? What kind of information does it produce: verifiable facts, opinions, rumors, all of the preceding? Why rely on "information" provided by someone of unknown reliability?

Does anyone ever search court (Federal and state) and board decisions?

How about news reports and business journals? State and local agencies? Information sources like Bloomberg Intelligence? Does the government ever use business info subscription services?

We live in the age of information, but some of what I have been reading in this thread strikes me as amateurish. I think the government should research offerors the way that businesses research companies when they are considering mergers and acquisitions. It seems to me that some government agencies award important contracts to companies they know very little about, and they do it without even conducting "discussions".

A lot of contract specialists have MBAs. Do they bring what they have learned about business research to source selection teams?

 

Link to comment
Share on other sites

4 hours ago, Vern Edwards said:

Interviewing sounds labor intensive and time-consuming. Is it efficient? How many interviews are you going to conduct? How often is the payoff worth the effort? What kind of information does it produce: verifiable facts, opinions, rumors, all of the preceding? Why rely on "information" provided by someone of unknown reliability?

Does anyone ever search court (Federal and state) and board decisions?

How about news reports and business journals? State and local agencies? Information sources like Bloomberg Intelligence? Does the government ever use business info subscription services?

We live in the age of information, but some of what I have been reading in this thread strikes me as amateurish. I think the government should research offerors the way that businesses research companies when they are considering mergers and acquisitions. It seems to me that some government agencies award important contracts to companies they know very little about, and they do it without even conducting "discussions".

A lot of contract specialists have MBAs. Do they bring what they have learned about business research to source selection teams?

 

If you do most of your awards from your organizations MATOCs, one would seldom have to interview references. if you are dealing with a regular set of contractors whom you do business with often, one might  only have to conduct an interview for a project that they are not personally familiar with or when they they want to verify the performance evaluation.

I found oral interviews to be very productive.  of course, it takes some effort. When there is a standard interview format, several people can make the calls.

What makes an oral interview any less reliable than a written returned form from the same people or from “whom” ???

As for protests, the only disappointed proposer was the one I wrote about a few days ago. The reference that the firm provided was no longer employed by the organization . Another employee with the same first hand knowledge was interviewed. The GAO did not sustain that (or any other) aspect of the protest. 

Link to comment
Share on other sites

If there are CPARS Ratings, you’d probably not do any interviews. We reserved the right to contact references to verify  ratings, e.g., for non-government job ratings. 

Link to comment
Share on other sites

Interviews might consume some time but often worth it.  Awarding a $10 million contract on just emailed questionnaire responses on some canned questions or looking at CPARS often doesn’t give a good indication.  You might want a company that holds themselves accountable, practices continuous improvement, deals with potential issues on a proactive basis, and makes customer communications a key part of their business practices.  Sure, you can ask for that on a questionnaire but interviews where examples come up and real experiences get mentioned is eye opening sometimes. 
 

Link to comment
Share on other sites

4 hours ago, formerfed said:

Interviews might consume some time but often worth it.

No doubt that interviews can score on occasion, but I'm not convinced that occasional scores yield greater riches than systematic digging.

Again, I wonder what the MBAs have learned about solid business research.

Link to comment
Share on other sites

 

1 hour ago, Vern Edwards said:

Again, I wonder what the MBAs have learned about solid business research.

I’m slow sometimes.  I see now what you’re asking.  Maybe this line of research can lead to something positive.

Link to comment
Share on other sites

  • 1 month later...

If two offerors have exactly the same proposals but the only difference is that one is the incumbent, how much lower/higher priced does the incumbent need to be to be awarded the contract? How much lower/higher priced does the new contractor need to be to be awarded the contract?

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.

×
×
  • Create New...