Jump to content

Evaluation criteria very vague


lotus

Recommended Posts

The evaluation criteria that I typically see are very vague and / or subjective.  They are so vague or subjective that any offer can be chosen, so likely the pre-determined awardee is chosen.

Since I'm not the pre-determined awardee, I'd like to break that.  Are there any published rules on how clear, specific, and objective evaluation criteria need be?  I'm itching to protest those that I think that I will win and try to turn the criteria to my advantage.

Along those lines, contracting officers and others do harbor grudges.   Is there any precedent to have procurement taken away from one contracts shop and given to another to avoid potentially biased evaluations?

 

Link to comment
Share on other sites

Guest Vern Edwards
On 4/7/2018 at 9:03 AM, lotus said:

The evaluation criteria that I typically see are very vague and / or subjective.  They are so vague or subjective that any offer can be chosen, so likely the pre-determined awardee is chosen.

Emphasis added. Please quote two or three examples of those that you have seen in real RFPs. Please provide the RFP numbers.

Link to comment
Share on other sites

20 hours ago, Vern Edwards said:

Emphasis added. Please quote two or three examples of those that you have seen in real RFPs. Please provide the RFP numbers.

     Let's try this one for starters.  It is from an RFP on the street now.

     "Technical Approach - The evaluation will assess the Offeror’s technical approach and assumptions to evaluate the offeror’s understanding of the work. Proposals that unreasonably assume away contractor performance risk will be rated less favorably than those that do not and may be rated unacceptable."

      The relevant definition of approach is "a way of dealing with something."  So, they are going to evaluate a way of dealing with something.

      But they don't say what they are looking for in that way of dealing with something.  What are the benchmarks, or touch points, that they are looking for?  

      If the favored vendor says "we will stand on our heads for 30 minutes each day", then I suspect that suddenly becomes the "way" that they want.

 

  

 

 

Link to comment
Share on other sites

Guest Vern Edwards
23 hours ago, lotus said:

But they don't say what they are looking for in that way of dealing with something.  What are the benchmarks, or touch points, that they are looking for?  

Did you look at the proposal preparation instructions? Did those instructions describe what the government wants to know about proposed technical approach?

Link to comment
Share on other sites

2 hours ago, Vern Edwards said:

Did you look at the proposal preparation instructions? Did those instructions describe what the government wants to know about proposed technical approach?

It does not do so.  It says things like 12 point font, no more than 20 pages, 1 inch margins, submit by and to whom, but does not do as you suggest.

Link to comment
Share on other sites

You'll lose that protest.

On 4/7/2018 at 5:03 PM, lotus said:

They are so vague or subjective that any offer can be chosen, so likely the pre-determined awardee is chosen.

Since I'm not the pre-determined awardee, I'd like to break that.  Are there any published rules on how clear, specific, and objective evaluation criteria need be?  I'm itching to protest those that I think that I will win and try to turn the criteria to my advantage.

Along those lines, contracting officers and others do harbor grudges.   Is there any precedent to have procurement taken away from one contracts shop and given to another to avoid potentially biased evaluations?

 

Is it commercial? If so, some contracting officers like the flexibility of choosing whatever offer they determine is the most advantageous to the Government. See FAR 12.602(c). It's permitted, and if simplified acquisition procedures are being used as well, the contracting officer is likely to be trying to keep it simple. GAO has stated many times that the CO has wide discretion in making use of what the contractor proposes. So the onus is on the contractor to put forth their best terms and conditions at their best price right away.

In regards to protests, you're right, they do harbor grudges. Instead of moving a procurement from one shop to another (you can't), work on developing a relationship with the Contracting Officer. 

Link to comment
Share on other sites

Guest PepeTheFrog
On 4/7/2018 at 12:03 PM, lotus said:

The evaluation criteria that I typically see are very vague and / or subjective (in your subjective opinion).  They are so vague or subjective that any offer can be chosen (flexibility in outcome is a feature, not a bug), so likely the pre-determined awardee is chosen (maybe, maybe not).

PepeTheFrog's comments are in bold.

If the evaluation criteria were not subjective, why would the government need humans to make choices?

If the evaluation criteria were "entirely objective," wouldn't the government mark a line in the sand for "all of these are good enough" and then pick the offer which is "good enough" and has the lowest price? What does that sound like?

Link to comment
Share on other sites

1 hour ago, PepeTheFrog said:

PepeTheFrog's comments are in bold.

If the evaluation criteria were not subjective, why would the government need humans to make choices?

If the evaluation criteria were "entirely objective," wouldn't the government mark a line in the sand for "all of these are good enough" and then pick the offer which is "good enough" and has the lowest price? What does that sound like?

While you are leading the LPTA conclusion, even LPTA criteria are typically subjective.

A points scale would help, if the assignment of points were objective (for example, finish in 1 week, 5 points; finish in 4 days, 6 points.) 

Link to comment
Share on other sites

I don't think it's the responsibility of the Government to lay out that criteria for points, especially in the solicitation. Because you're going to get a bunch of offers providing exactly what you lay out in your points scheme.

Offerors need to put forth their best terms and conditions from a technical and price standpoint. And I don't think assigning a point scale changes anything. What's the difference between that and "Finishing in 4 days is better than 1 week, and I think the price is worth the tradeoff because A, B, and C, so I'm awarding to the company who provided a faster completion date?"

Instead of relying on the Government to make some wonky point system for selecting a vendor, put forth better terms and conditions. This is where debriefings help. If the Government doesn't tell you why you weren't selected, THEN protest. But offerors are not selected because their price is too high, their technical approach was lacking or weaker than another, or past performance. The system NEEDS to be subjected because the Government would be forced to award contracts based off some technicality.

Link to comment
Share on other sites

Guest Vern Edwards
21 minutes ago, ContractingCowboi said:

I don't think it's the responsibility of the Government to lay out that criteria for points, especially in the solicitation. Because you're going to get a bunch of offers providing exactly what you lay out in your points scheme.

But... if an offeror proposes what will earn it the maximum number of points in the government's point scheme, won't that be what the government wants to buy and the best value?

Link to comment
Share on other sites

Wouldn't your requirement then be 4 days vs. 1 week and then state LPTA will be your selection basis?

GAO has even stated that a tradeoff based on points alone (and adjectival ratings) isn't sufficient enough to support an award decision. So you'll have to document why there's value in the tradeoff anyways in some sort of narrative.

I'd hate to be a contractor who didn't get an award because I got a 1 on some minor detail, like a delivery schedule but I was off-the-charts better on another factor that seriously could impact the performance of the contract, but since another vendor hit the same threshold, technically, we were rated the same. I'd hate to be the Government too, in which they'd have to select the technically worse offeror based off the criteria they put in the solicitation when they could have left it subjective. It's limiting.

While I understand it's more objective to assign point values and give criteria to meet those values, it's limiting and could result in the Government not getting the best deal.

FAR 15.304(d) states that the rating method need not be disclosed in the solicitation (except for past performance), so I just don't see the point in doing so. But that's just my opinion, I can see both ways.

Link to comment
Share on other sites

Guest Vern Edwards

Well, I was really focused on the notion that if you give them a precise evaluation scheme they'll give you what you want.

1 hour ago, ContractingCowboi said:

Because you're going to get a bunch of offers providing exactly what you lay out in your points scheme.

I have heard that many times over the years and have always wondered why, if you know what you want and can lay it out in a points scheme, you wouldn't want to get it.

As for the GAO, they say many things. There are ways to use a precise point scheme that can avoid their complaint. I don't advocate it, but there are ways.

Link to comment
Share on other sites

Contracting Cowboy,

Delivery schedule wouldn't be a "minor detail", if the government determines delivery is a critical element for award (e.g., "time is of the essence').

Regarding FAR 15.304(d), I never understood why the government wouldn't want to disclose the rating methods in the solicitation. 

We have nothing to hide, so why not disclose the rating methods in the solicitation.  For transparency sake- if anything at all.

 When I started working at one agency,  the contracting office didn't disclose the rating methods in the solicitations. I asked the contracting officers why, and they said "The program office told us not to."  The program office's excuse was, "We'll provide the methods when we start evaluating the technical proposals."  Whenever the contracting office asked the program office to provide rating methods, the program office would elevate the issue to the procurement executive level.

 

 

 

Link to comment
Share on other sites

I'm on the contracting side and I see benefits to there being some wiggle room in evaluation criteria.   It invites a better mousetrap.   An Agency may know exactly what it wants when it procures pencils, as that's a 100+ year old technology.  Such an acquisition lends itself to an LPTA approach, or a tightly defined points scheme.   However, beyond basic commodities, the risk in evaluating ONLY particular specs or skills is that "better" specs or skills cannot be evaluated.    So what's "better"? - well, a contractor proposes what THEY think is better, and sees if the Agency agrees.   

My former company won a contract one time with a price that was about 20% higher than our next closest competitor.   The Agency knew what they wanted, but didn't know how to get it.   We proposed a way to deliver that matched their needs and won.    Everyone was bidding the same underlying THING, but we solved their problem in a way that overrode other considerations like price.  

Now, a losing contractor might say "Hey - but now that the Agency knows HOW they can get that thing, they should re-solicit and ask everyone to bid using that same approach" to which I say...no.   Our price was higher because the approach added some cost.   The extra cost would have been roughly true for all bidders, so the Government wouldn't have saved much in round 2, yet would spend a bunch of money and time trying to make it a level playing field.   With the first solicitation, someone (my company, in this case) cracked the code and an award was made.  

The answer, for Lotus, is to ask for debriefs and learn, learn, learn...  Go into a debrief with questions the Agency can answer, that help you learn why you didn't finish first.   Learn about what the other contractors are doing.    Don't focus on deals where someone beat you by five cents on the same exact thing, rather, focus on deals that you thought you should win, but that some 'better mousetrap' won.   

Maybe Lotus is in a commodities business and leads with lowest price only?  I don't know.  I'd suggest that every successful contractor I know wins by offering better mousetraps, not by hoping the solicitation matches their exact mousetrap for which they have the lowest price.    The better mousetraps generally win - and that's a good thing.   

Link to comment
Share on other sites

On 4/13/2018 at 4:37 PM, Vern Edwards said:

jayandstacey:

What business was your former company in? What did it sell? Supplies? Services? Something else?

Hi Vern: Information Technology products and services.  

In my particular example; the product was an enterprise software agreement.   Our proposal matched the Government's payment structure to when/if they had the budget, which allowed us to win.   The Government was precise in the definition of the software, what was left for the contractors to solve was HOW to deliver the licenses so the Government could be successful.   We solved it with a financed structure.   The item was simple; the structure was a bit complex but necessary.  

More broadly (not to Vern per se):  While a "better mousetrap" isn't always in play; I'd say it often is, even if not asked for.    These can be small (like showing how commercial past performance might be more relevant) to large (proposing a much more efficient backhoe when the Government envisioned a team of laborers with shovels).   Most every company Ive ever known was formed with some unique value proposition in mind, and RFPs are sometimes a battle of who's value proposition is either listed as more important in the RFP, or not listed but evaluated from the proposals.   That's the rub for lotus - don't show up late to the party, then try to protest into the win.  That's just not a sustainable business practice.   Convince buyers ahead of time of the value prop your company brings, and be the thing they're looking for.  

My old company had a general policy that we wouldn't bother to bid on RFPs that were "news" to us upon their release.   Because it meant we'd already probably lost that battle and our resources were better spent somewhere else.   

Link to comment
Share on other sites

Guest Vern Edwards

The problem is this:

Source selection is not always a matter of specifying and buying exactly what you want. It is often a method of market research. Think of it as a form of shopping.

Sometimes the government knows it wants something or someone to perform a certain function, but it does not know of all that is available, or potentially available, to perform that function and what specific variations are possible that would be beneficial or detrimental. In such cases the government uses broad selection criteria in order to be able to take advantage of the unanticipated and the unexpected.

Link to comment
Share on other sites

26 minutes ago, Vern Edwards said:

The problem is this:

Source selection is not always a matter of specifying and buying exactly what you want. It is often a method of market research. Think of it as a form of shopping.

Sometimes the government knows it wants something or someone to perform a certain function, but it does not know of all that is available, or potentially available, to perform that function and what specific variations are possible that would be beneficial or detrimental. In such cases the government uses broad selection criteria in order to be able to take advantage of the unanticipated and the unexpected.

Agreed.  The vagueness is a function of the fact that the government is not always an expert at these things/services....which partly explains why they are reaching outside the government to procure them.  The vagueness is also a strength; it allows the government to maximize competition, stay current and to change gears when they want or need.

Is the incumbent favored sometimes?   Sure.  But if that were ALWAYS true, there'd be no competition at all.    

Link to comment
Share on other sites

Guest Vern Edwards
On 4/7/2018 at 9:03 AM, lotus said:

The evaluation criteria that I typically see are very vague and / or subjective.  They are so vague or subjective that any offer can be chosen, so likely the pre-determined awardee is chosen.

The second sentence is a classic example of a non sequitur.

Link to comment
Share on other sites

On 4/9/2018 at 9:35 AM, lotus said:

But they don't say what they are looking for in that way of dealing with something.  What are the benchmarks, or touch points, that they are looking for?  

That's what they want you to tell them: "what is the best way to deal with this problem we have presented"? 

Avoid boilerplate nonsense provided by the Marketing department.  Fill in knowledge gaps with your  previous experience.  Provide solutions they clearly hadn't considered.  Explain why accepting your offer is in their best interests. Above all, show them that you understand what they are trying to achieve. 

The nice thing about "vagueness" is that it allows the Government to pick the guy who clearly knows more about how to define and address the problem than they do.  

Link to comment
Share on other sites

On 4/13/2018 at 7:44 PM, Weno2 said:

Contracting Cowboy,

Delivery schedule wouldn't be a "minor detail", if the government determines delivery is a critical element for award (e.g., "time is of the essence').

Regarding FAR 15.304(d), I never understood why the government wouldn't want to disclose the rating methods in the solicitation. 

We have nothing to hide, so why not disclose the rating methods in the solicitation.  For transparency sake- if anything at all.

 When I started working at one agency,  the contracting office didn't disclose the rating methods in the solicitations. I asked the contracting officers why, and they said "The program office told us not to."  The program office's excuse was, "We'll provide the methods when we start evaluating the technical proposals."  Whenever the contracting office asked the program office to provide rating methods, the program office would elevate the issue to the procurement executive level.

 

 

 

Weno, because you're not going to always know what the contractor is going to propose. Say you disclose that you'll be doing a point method, and all of a sudden proposals come back and it's discovered that the point method isn't the most efficient way of evaluating offers. You'd want the flexibility to switch to maybe a adjectival rating, or a narrative, or a simple pass/fail since all technical approaches are extremely similar. If you disclose the rating method, you're stuck with that rating method unless you amend the solicitation.

It's not about hiding anything-- you should be able to tell everything in the debriefing without trying to hide something.

Does disclosing the rating method make everything a bit more objective? Sure. Does it make good business sense? No. Unless you're doing LPTA and your requirement is so well defined that you know exactly what offerors are going to be providing, you want as much flexibility to make good business decisions as possible.

 

Link to comment
Share on other sites

On 4/13/2018 at 6:29 PM, Vern Edwards said:

Well, I was really focused on the notion that if you give them a precise evaluation scheme they'll give you what you want.

I have heard that many times over the years and have always wondered why, if you know what you want and can lay it out in a points scheme, you wouldn't want to get it.

As for the GAO, they say many things. There are ways to use a precise point scheme that can avoid their complaint. I don't advocate it, but there are ways.

My opinion is that if you know exactly what you want, you might as well use LPTA with a pass/fail or acceptable/unacceptable rating rather than a point scheme. I think you had a post with comparative evaluations on FAR Part 13 awhile back that spoke of a pretty good point scheme, but in reality I just think that if you're doing tradeoffs, look at the advantages vs disadvantages and make a good business call. Which I know is harder than it sounds, but doable with some word-smithing. 

Link to comment
Share on other sites

Guest Vern Edwards
1 minute ago, ContractingCowboi said:

My opinion is that if you know exactly what you want, you might as well use LPTA with a pass/fail or acceptable/unacceptable rating rather than a point scheme.

I agree.

 

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...