Jump to content

Blind competitions?


Recommended Posts

My supervisory CO has asked me to research how to do a "blind competition," which I understand is a process whereby companies interested in submitting offers do not use their own identity in their technical (and cost?) proposals. The purpose of the blind competition is to establish a true(r) objectivity on behalf of the technical evaluation committee. I have searched the wifcon pages and done a general online search, but have not found any guidance on how to set up a blind competition. I am looking for best ways to set-up/structure a blind competition, e.g. does it work better if offerors are instructed to write their technical proposals in such a way so as to not identify themselves or we ask them to register and then assign a number or other name to them? What are other lessons learned, best practices, guidance, pitfalls, etc.?

Thanks in advance for any inputs!

Link to comment
Share on other sites

My agency procures technical assistance services (development assistance) usually through CPFF contracts (sometimes award fee or firm fixed price). It's largely personnel based assistance (if that makes sense), e.g. helping a host country gain WTO membership, developing Customs systems/offices, agriculture development, banking sector, etc. It could include some hard- and software, but the majority of what we pay for is labor. We have a couple of procurements coming up, but the program descriptions are still being developed.

Link to comment
Share on other sites

I should have also asked what the intended evaluation factors are? Certain factors may make your goal impractical (e.g. technical approach narratives, key personnel, facilities, past performance/experience).

Based on what you buy, I presume your office and customer have become familiar with the offeror pool. Is the make-up of the competition pool varied or constant? If evaluators are familiar with offerors approaches, key personnel, past contracts, etc. it may be difficult to mask their identity.

You can ask for santized proposals void of identifying information (logo, letterhead, name, etc.), but this may only be a veneer of 'blind competition' for the reasons stated above.

What happens if an offeror's proposal includes identifying information? Is it removed from competition?

I recommend asking the CO to clarify what it is they want you researching. Is it simply redacting or not submitting offers identifying authorship or something more complex?

Link to comment
Share on other sites

1 hour ago, Jamaal Valentine said:

Based on what you buy, I presume your office and customer have become familiar with the offeror pool. Is the make-up of the competition pool varied or constant? If evaluators are familiar with offerors approaches, key personnel, past contracts, etc. it may be difficult to mask their identity.

Piling on:

It is sometimes possible to identify the offeror by the color palate of the proposal, the nomenclature of the figures, the writing style, the paper chosen, even the page formatting.  This is purposeful, and it is possible that having "blind" competitions may inadvertently disadvantage a lesser known offeror.    

Link to comment
Share on other sites

How does one check the references of an anonymous proposal?  I'm skeptical of this approach- seems like it is not worth the heartache.

Sascha, what problem is the CO trying to solve?  Trying to defuse accusations of favoritism towards a particular firm?  Some problems may not be solvable . . .

Link to comment
Share on other sites

Guest Vern Edwards
8 hours ago, Sascha Kemper said:

My supervisory CO has asked me to research how to do a "blind competition," which I understand is a process whereby companies interested in submitting offers do not use their own identity in their technical (and cost?) proposals.

People have been doing what you call "blind" competitions, i.e., blind proposal submissions, for as long as I can remember, back to the 1960s. It's done by instructing offerors to leave all identifying markings out of proposals. It's not foolproof, which can be a problem for the agency, and it can be very difficult to accomplish in some kinds of high tech competitions in which offerors' technology choices are a giveaway to knowledgable evaluators. I had to do a few back in the 1970s.

It is a complete waste of time and attention.

Why can't government people just conduct simple, professional competitions without gimmicks? I guess they just lack the simplicity gene. The government's institutional memory is as long and deep as the lifespan of an adult mayfly.

Link to comment
Share on other sites

20 hours ago, Jamaal Valentine said:

I should have also asked what the intended evaluation factors are? Certain factors may make your goal impractical (e.g. technical approach narratives, key personnel, facilities, past performance/experience).

Based on what you buy, I presume your office and customer have become familiar with the offeror pool. Is the make-up of the competition pool varied or constant? If evaluators are familiar with offerors approaches, key personnel, past contracts, etc. it may be difficult to mask their identity.

You can ask for santized proposals void of identifying information (logo, letterhead, name, etc.), but this may only be a veneer of 'blind competition' for the reasons stated above.

What happens if an offeror's proposal includes identifying information? Is it removed from competition?

I recommend asking the CO to clarify what it is they want you researching. Is it simply redacting or not submitting offers identifying authorship or something more complex?

Evaluation factors would be technical excellence, management capability (might need to leave that one out), and personnel qualifications. Past performance is being reviewed by the contracting office on a pass/fail basis.

Your second point is our concern - we have a small number of established partners that make up the offeror pool, but technical approaches (due to what we buy) are not necessarily giveaways for the the evaluators.

My CO was looking for best practices' tips for the entire process, e.g. does it work better if we ask offerors to withhold all identifying info, redact certain parts, if we ask offerors to register and then we assign names to use in the proposal. If we use this approach, what's the cleanest way to do it (also for offerors who haven't been asked to do this before).

19 hours ago, apsofacto said:

How does one check the references of an anonymous proposal?  I'm skeptical of this approach- seems like it is not worth the heartache.

Sascha, what problem is the CO trying to solve?  Trying to defuse accusations of favoritism towards a particular firm?  Some problems may not be solvable . . .

References would be checked by the CO's office, not by the evaluation committee. The problem is we have a small pool of offerors that our CORs know well, so we're trying to remove favoritism or preferences for certain offerors.

Link to comment
Share on other sites

If past performance is separate from the technical evaluation and technical approaches wouldn't necessarily be specifically distinguishable or self evident , how would bias or favoritism be evident or possible in the evaluation criteria and ratings, even with firms identified?  

Can the evaluation board not  detect favoritism in the technical evaluation comments that aren't Germaine to the technical  evaluation criteria?  Just curious. 

From your description of the services being procured, it would seem that the level of performance quality provided is very important, though. 

Link to comment
Share on other sites

Guest Vern Edwards

One concern will be what to do with the offeror who, despite instructions, includes identifying information in its proposal? (It might not be obvious to the CO. It might be buried deep.) Do you have to kick them out of the competition? If you don't, are you laying the foundation for a sustained protest? If you decide to keep them in, how do you "equalize" the competition? If you decide to solve the problem by revealing all identities to the evaluators, how do you answer the charge that you didn't prevent prejudice from tainting the evaluation, which was why you tried the blind competition in the first place? You were obviously afraid that your evaluators were prejudiced.

The more gimmicks you try, the more traps you set for yourself. Blind competition is not innovative thinking. It is needless excess.

Link to comment
Share on other sites

On ‎10‎/‎4‎/‎2016 at 1:20 AM, Sascha Kemper said:

My supervisory CO has asked me to research how to do a "blind competition," which I understand is a process whereby companies interested in submitting offers do not use their own identity in their technical (and cost?) proposals.

I'm going to be unusually optimistic and assert, without any evidence whatsoever in support of my assertion, that the supervisory CO has assigned Sascha this project as an employee development exercise to help Sascha learn more about competitions and source evaluations. The supervisory CO knows what Vern and others know about such things, but wanted Sascha to delve into this topic, including researching relevant bid protests decisions, in order to grow professionally.

H2H

Link to comment
Share on other sites

Guest PepeTheFrog

PepeTheFrog seconds Vern Edwards.

PepeTheFrog has seen many source selections where some frog was dying to use "blind" evaluations. In nearly all of them, afterwards, the evaluation frogs told PepeTheFrog they deduced the identities of the well-known offerors anyway, strictly from content.

The expected return on your investment of time and resources is minuscule or negative. It's also unnecessary work and trouble for the offerors, who like to brand their proposals. 

Link to comment
Share on other sites

7 hours ago, Sascha Kemper said:

The problem is we have a small pool of offerors that our CORs know well, so we're trying to remove favoritism or preferences for certain offerors.

If the problem is internal, I think the effort should be spent internally, not on extra solicitation provisions which place a burden on you, not the evaluators.

Can you contest the technical evaluation during source selection if your CO suspects there is bad faith on that side of the fence? 

Link to comment
Share on other sites

Not to pile on, but I agree with the position that utilizing or attempting to utilize a "blind" source selection process is not a wise approach.  Your time and energy would be better spent researching other, more impactful techniques, such as avoiding an "essay writing contest."  If you search these forums I'm sure you'll find plenty of helpful information.  Look here for starters:

For example, try oral presentations (FAR 15.102).  Maybe you could even have your evaluators wear blindfolds if you're still concerned with the identity of offerors...

Link to comment
Share on other sites

Sascha:

Your instructions to offerors can include a structured proposal submission section that clearly identifies how to submit technical, past performance, and price/cost volumes. In those instruction you can instruct offerors to leave all identifying markings out of their written offers. Internally, one proposals are received you can identify them however you want for tracking purposes (e.g. Offeror A -- supposedly numbering is not recommended i.e. Offer 1, etc.).

I offer that simply in an attempt to respond to your question. I do not recommend that approach. Rather, I would suggest you demand that your evaluation teams provide adequate written analysis of the offers. You can mitigate bias by demanding that evaluators stick to the evaluation criteria and articulate the underlying qualitative distinctions between offerors and their offers. I do not think you can hope to remove all bias ... as long as the evaluation is reasonable and consistent with the terms of the solicitation you may have to settle. 

Find a few discriminating evaluation factors and give it a shot. Sounds like past experience, performance, and price/cost is adequate enough since you are familiar with the regular pool of offerors. What do you hope to gain by evaluating "technical excellence, management capability (might need to leave that one out), and personnel qualifications"? Are you going to incorporate elements of the successful offeror's proposal into the contract?

Link to comment
Share on other sites

On 10/6/2016 at 3:53 AM, Jamaal Valentine said:

I offer that simply in an attempt to respond to your question. I do not recommend that approach.

Find a few discriminating evaluation factors and give it a shot. Sounds like past experience, performance, and price is adequate enough since you are familiar with the regular pool of offerors. What do you hope to gain by evaluating "technical excellence, management capability (might need to leave that one out), and personnel qualifications"? Are you going to incorporate elements of the successful offeror's proposal into the contract?

Got it. Thanks, Jamaal!

No, we don't incorporate elements of the successful offeror's proposal into the contract. The other factors are suggested by FAR part 15 and are what lends itself to evaluating the proposals we receive to discern between the offerors. 

Link to comment
Share on other sites

Since the proposal will not be incorporated into the contract you may want to consider what you expect to gain from an essay contest. Since it is a cost-type contract you are likely looking for the offeror's methods of performance  and understanding of the work. Problem is that the offerors will not be bound to anything other than their offered pricing/costs. The contract may not include any inferred proposal promises or elements regarding technical features/approaches or personnel qualifications.

Seems you can reasonably determine management capability and technical excellence through past experience and performance. These would possibly be my primary non-price/cost related factors. (FAR 15.304[c][2])

Good luck! Let us know how it turns out.

Link to comment
Share on other sites

Guest Vern Edwards

S

2 hours ago, Sascha Kemper said:

No, we don't incorporate elements of the successful offeror's proposal into the contract. The other factors are suggested by FAR part 15 and are what lends itself to evaluating the proposals we receive to discern between the offerors. 

Depending on what you're buying and the market from which you'll be buying it, there may be a set of good offerors among which there are no important differences. Sometimes, everybody is about the same. Think about that. Don't assume that there will be significant differences.

If you're buying services, start with the three basics: (1) experience, (2) price, and (3) past performance. Treat them as equally important.

Experience is the number of job opportunities an offeror has had in which it could learn about the kinds of problems it will face on your job and to learn how to solve them. The broader an offeror's experience (the more your tasks it has performed), and the deeper (the more times it has performed a particular task) , the more qualified the offeror is likely (ought) to be.

Price is price. Within limits, the lower the better.

Past performance is nothing but what other people say about an offeror, which will be, at best, half-truths. The only thing really useful about past performance evaluations is bad reports, which are warnings. (When I go to Amazon to buy a book, the only ratings I want to read are the really bad ones.) Also, consider financials as a basis for evaluating past performance, if financials are available and reliable. The longer a company has been making money and the lmore money it has made, the better its performance has likely been.

Now, with that settled, what other information do you need in light of the market from which you'll be buying? Consider only those attributes  that are likely to distinguish offerors from one another. Avoid the superficial and the unverifiable. You shouldn't evaluate something just because you can.

Things can be a lot more complex and difficult if you're buying commercial supplies, but given that you're in Kosovo, I presume that you're buying services.

Quote

Good luck! Let us know how it turns out.

Luck has nothing to do with it. Professional competence is what matters.

Link to comment
Share on other sites

  • 4 months later...
On 10/5/2016 at 10:06 AM, Vern Edwards said:

One concern will be what to do with the offeror who, despite instructions, includes identifying information in its proposal? (It might not be obvious to the CO. It might be buried deep.) Do you have to kick them out of the competition? If you don't, are you laying the foundation for a sustained protest? If you decide to keep them in, how do you "equalize" the competition? If you decide to solve the problem by revealing all identities to the evaluators, how do you answer the charge that you didn't prevent prejudice from tainting the evaluation, which was why you tried the blind competition in the first place? You were obviously afraid that your evaluators were prejudiced.

The more gimmicks you try, the more traps you set for yourself. Blind competition is not innovative thinking. It is needless excess.

Reject the proposal!!

 

http://www.gao.gov/products/B-414060?utm_medium=email&utm_source=govdelivery#mt=e-report

Link to comment
Share on other sites

Hmmm.  So you can't take credit for your experience on the tasks being recompeted?   What is this world coming to?  I wonder how they handled evaluating past performance.

Link to comment
Share on other sites

The GAO decision didn't indicate which Best Value method was used. Whether LPTA or tradeoff, the CO  need not consider past performance if he or she documents the file: 

 

Quote

 

15.101-2 Lowest price technically acceptable source selection process.

(a) The lowest price technically acceptable source selection process is appropriate when best value is expected to result from selection of the technically acceptable proposal with the lowest evaluated price.

(b) When using the lowest price technically acceptable process, the following apply:

(1) The evaluation factors and significant subfactors that establish the requirements of acceptability shall be set forth in the solicitation. Solicitations shall specify that award will be made on the basis of the lowest evaluated price of proposals meeting or exceeding the acceptability standards for non-cost factors. If the contracting officer documents the file pursuant to 15.304(c)(3)(iii), past performance need not be an evaluation factor in lowest price technically acceptable source selections. If the contracting officer elects to consider past performance as an evaluation factor, it shall be evaluated in accordance with 15.305. However, the comparative assessment in 15.305(a)(2)(i) does not apply. If the contracting officer determines that a small business’ past performance is not acceptable, the matter shall be referred to the Small Business Administration for a Certificate of Competency determination, in accordance with the procedures contained in Subpart 19.6 and 15 U.S.C. 637(b)(7)).

(2) Tradeoffs are not permitted.

(3) Proposals are evaluated for acceptability but not ranked using the non-cost/price factors.

(4) Exchanges may occur (see 15.306).

 

 

 

See also 15.304(c ) (3) (iii):

Quote

 

15.304 Evaluation factors and significant subfactors.

(a) The award decision is based on evaluation factors and significant subfactors that are tailored to the acquisition.

(b) Evaluation factors and significant subfactors must—

(1) Represent the key areas of importance and emphasis to be considered in the source selection decision; and

(2) Support meaningful comparison and discrimination between and among competing proposals.

(c) The evaluation factors and significant subfactors that apply to an acquisition and their relative importance, are within the broad discretion of agency acquisition officials, subject to the following requirements:

(1) Price or cost to the Government shall be evaluated in every source selection (10 U.S.C. 2305(a)(3)(A)(ii) and 41 U.S.C. 3306(c)(1)(B)) (also see part 36 for architect-engineer contracts);

(2) The quality of the product or service shall be addressed in every source selection through consideration of one or more non-cost evaluation factors such as past performance, compliance with solicitation requirements, technical excellence, management capability, personnel qualifications, and prior experience (10 U.S.C. 2305(a)(3)(A)(i) and 41 U.S.C. 3306(c)(1)(A)); and

(3)(i) Past performance, except as set forth in paragraph (c)(3)(iii) of this section, shall be evaluated in all source selections for negotiated competitive acquisitions expected to exceed the simplified acquisition threshold.

(ii) For solicitations that are not set aside for small business concerns, involving consolidation or bundling, that offer a significant opportunity for subcontracting, the contracting officer shall include a factor to evaluate past performance indicating the extent to which the offeror attained applicable goals for small business participation under contracts that required subcontracting plans (15 U.S.C. 637(d)(4)(G)(ii)).

(iii) Past performance need not be evaluated if the contracting officer documents the reason past performance is not an appropriate evaluation factor for the acquisition.

 

 

Since this was an 8(a) set aside, I can understand why past performance was not used as a factor.

Link to comment
Share on other sites

Guest Vern Edwards
18 hours ago, napolik said:

Reject the proposal!!

I'd be careful about that advice. If the defect is minor, say, a single improper reference in a short proposal, and easily corrected by blacking out the reference, then GAO might consider rejection to be an overreaction. Moreover, an RFP should put offerors on notice that failure to comply with an instruction might result in wholesale rejection.

See Mcfadden & Associates, Inc., Comp. Gen. Dec. B-275502, 97-1 CPD ¶ 88:

Quote

In short, what we have here is the rejection of a proposal based on an automatic, mechanical application of the RFP's page limit even though the RFP provided only that the agency “may” reject the proposal for that reason. As indicated above, we believe when such language appears in an RFP the agency must have a reasonable basis, above and beyond the page limit itself, for the rejection. The Customs Service has offered no reasonable basis for the rejection--it has not determined that the proposal is unacceptable or should not be in the competitive range, and it has not suggested any other reason, apart from the RFP page limit, for the rejection. Accordingly, the protest is sustained.

In the protest to which napolik provided a link, it appears that the improperly included information was extensive and detailed.

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...