Jump to content

Do I Need To Remove Offerors' Identifying Information from Proposals Before Tech Eval?


Recommended Posts

A contracting officer and lawyer have both told me that I must remove/redact all of a contractor's identifying information (e.g. Company name/Address/Letterhead/contact info) from their technical proposal prior to the technical evaluation panel's review.

The fear is that not redacting the identifying information could be used against us in a GAO protest in as much as it could be argued that knowing who the offerors are somehow made the evaluation biased.

This sounds crazy to me and a lot of additional work/hassle (some companies have their logo on every single page) that could also easily lead to confusion on the Government's part if I/someone else forgets that HugeCorp should also be known as Offeror "A"...or was it Offeror "B"?.

I asked "where in the FAR or GAO case law does it say that proposals must be redacted before technical evaluation " and received no answer.

I also googled "Bias" and "GAO" and wasn't able to find any cases that successfully argued "they knew the names of the companies they were evaluating and favored one over the other".

Is this a case of training by rumour and innuendo ("we've always done it this way...it avoids bias, yeah, that's what it does"), or is this approach based on some kind of common sense or legal savvy that I'm missing?

Link to comment
Share on other sites

Guest Vern Edwards

There is no requirement in statute or in the FAR to remove identifying information before sending proposals to evaluators. There may be such a requirement in some agency FAR supplements or in policy documents. If the people who told you to do that claim it is a policy, then make them show it to you.

The idea that you must do it to protect the agency from charges of bias in a GAO protest is absurd. The people who told you that are professionally ignorant. You can quote me.

It has been a traditional practice in some offices. It is a waste of time in my experience.

Link to comment
Share on other sites

There is no requirement in statute or in the FAR to remove identifying information before sending proposals to evaluators. There may be such a requirement in some agency FAR supplements or in policy documents. If the people who told you to do that claim it is a policy, then make them show it to you.

The idea that you must do it to protect the agency from charges of bias in a GAO protest is absurd. The people who told you that are professionally ignorant. You can quote me.

It has been a traditional practice in some offices. It is a waste of time in my experience.

Bologna!

While there is no legal or regulatory requirement to remove the identity of contractors from proposals, I can understand why it would be done.

In my career, I have not encountered any circumstances representing “bias or bad faith” leading to a GAO protest. However, I have encountered cases where members of technical panels evaluated proposals using not only the proposals’ narratives, but also their personal experience with the contractors, particularly incumbents. While use of evaluators’ personal experiences may be acceptable when assessing contractors’ experiences or past performances, it is not acceptable when evaluating other factors (e.g. résumés, staffing plans, technical approach, etc.).

Given the possibility of evaluators inserting their personal experiences with contractors, it might not be a bad idea to remove the contractors' names from the proposals.

Link to comment
Share on other sites

I know of no such requirement.

However, I have seen in my experience where a technical reviewer will be biased because they are familiar with a contractor (especially if they are the incumbent) and I have actually encouraged the SSA to remove someone from an SSEB for that very reason.

In the future, if you decide to redact, a step I found to make that easier is to require the offerors to submit their technical proposal absent of any company-identifying information. Better for them to redact than for me to have to do it.

Link to comment
Share on other sites

Guest Vern Edwards

Salami!

In many cases, especially in the biggest procurements, it's well-nigh impossible to render proposals anonymous. Evaluators might know an offeror from features of a proposed approach, or from the technology proposed, or from the names of proposed key personnel, or from references to offeror in-house technical reports, or from statements in a proposal about approaches used in the past. When agency evaluators are familiar with one or more offerors, they can often easily identify the source of a particular proposal, even without identifying remarks. An evaluator who has worked with an incumbent and who cannot recognize the incumbent's proposal is brain dead.

If proposals are large, think of the work involved in reviewing them to ensure that they contained no references to the companies that submitted them. What if you told offerors not to include any indicia of their identities, and what if one offeror missed something? Would you throw them out of the competition. This is the kind of thing that makes government personnel look stupid.

The head of the evaluation board is responsible for its product. If you cannot find someone who'll ensure that evaluators produce unbiased results, you've got big problems. In any case, as Professor Nash has written, it's very difficult for a protester to win a protest based on accusations of bias. See Formation of Government Contracts 4th, 947-948.

Link to comment
Share on other sites

Peperoni!

The first solicitation I ever posted was for holter/event monitoring, and I was told to remove company markings from proposals prior to the evaluations. In the rationale for the rating of a particular contractor, it mentioned the actual contractor’s corporate name. When I asked the technical people on the SSEP, they mentioned that they had googled a couple of the salient characteristics of these monitors, and when they matched a sample event report submitted with a proposal, to one on the contractor’s website, well… pretty easy to connect the dots (whether or not the panel should be googling additional information about an offer is another discussion).

I questioned why we were doing this to my immediate manager, at one time a QA individual had introduced the idea, but in the end, he stopped making make me redact technical proposals.

I see little benefit to doing this.

Link to comment
Share on other sites

Guest Vern Edwards

Vern,

Are you plaing it both ways? Bologna -- it might not be a bad idea to remove the contractors' names from the proposals. Salami -- it's well-nigh impossible to render proposals anonymous.

Uh, I'm not sure what you mean. I think that both Napoli and I would rather have beer and Prosciutto at Aci Trezza and salute Odysseus on his escape.

Link to comment
Share on other sites

Beer and Prosciutto, count me in.

I don't know of any such requirement, but I would caution to be consistent. If your agency redacted the information for five similar consecutive procurements, but then for some reason did not on the sixth, I can see it come to question as to "why?". Was there a contractor in that sixth acquisition that 'someone" wanted to make sure their identity was known?

Good luck.

Link to comment
Share on other sites

Really? Redact markings from hundreds if not thousands of pages of proposals. Are we talking sharpie, adobe PDF, or .DOC. Literally sounds insane. No way would I waste time doing that. I'd tell your lawyer to grab and marker and start redacting.

Link to comment
Share on other sites

If the KO and lawyer wanted proposals to be anonymous then they should have issued the instructions during RFP preparation and should have verified that it was in the RFP. To require editing out such info after proposal receipt is ridiculous and you can quote me. Beyond that, the acquisition leadership team ought to be able to detect bias by technical evaluators or rely on someone on the team to control such actions. In my experience, we were able to control it. I think that trying to eliminate knowledge of firms' identities is a waste of time, expense and resources. Manage and control the team! Require professionalism! Period.

Link to comment
Share on other sites

Guest Vern Edwards

If, despite the overwhelming response here against trying to make offerors anonymous, you decide that you will do it, be wary.

If you tell offerors to remove identifying marks, you'd better triple check to make sure that they have done so, otherwise you'll face the prospect of supplementary protests that one or more offerors has failed to do so. I doubt that you'd lose the protest, but why give anyone another reason to file?

If you say in your source selection plan that you will render offerors anonymous to evaluators, you'd better do it, or you could face a supplementary protest that you didn't comply with your plan. Again, I doubt that you'd lose, but why take a chance?

Link to comment
Share on other sites

napolik - you stated:

"While use of evaluators’ personal experiences may be acceptable when assessing contractors’ experiences or past performances, it is not acceptable when evaluating other factors (e.g. résumés, staffing plans, technical approach, etc.)."

Why not? Why can personal experience, or for that matter information gleened during market research but not called out in a vendor's response, be used? I would think an evaluation panel member would be entirely remiss (negligent) if they didn't account for that which they know, or better yet accounted for something they know is not technically possible from a proposed solution despite the claim in the response otherwise. Not in a biased way (heading Desporado off at the pass) but in an informational way that adds to but not detracts from the quality of the evaluation.

Link to comment
Share on other sites

napolik - you stated:

"While use of evaluators’ personal experiences may be acceptable when assessing contractors’ experiences or past performances, it is not acceptable when evaluating other factors (e.g. résumés, staffing plans, technical approach, etc.)."

Why not? Why can personal experience, or for that matter information gleened during market research but not called out in a vendor's response, be used? I would think an evaluation panel member would be entirely remiss (negligent) if they didn't account for that which they know, or better yet accounted for something they know is not technically possible from a proposed solution despite the claim in the response otherwise. Not in a biased way (heading Desporado off at the pass) but in an informational way that adds to but not detracts from the quality of the evaluation.

jmj,

Personal experience or knowledge can be used in the evaluation of proposals when evaluating factors other than contractor experience or past performance. However, my personal experience with technical panels – primarily in field contracting vice major systems – has taught me to focus the panel’s attention on the facts contained in proposals rather than on the panel member’s personal experience with the contractors.

The GAO has issued decisions approving the use of “extrinsic information” or “extrinsic evidence”, or of information that is not within the “four corners of the proposal”, when consideration of the information or evidence is “consistent with established procurement practice”. At least one COFC judge has issued a decision accepting the use of extrinsic information on a case decided by the GAO allowing the use of the evaluators’ personal knowledge, but she reached a different conclusion than GAO as to the rationality of the application of the knowledge.

In my career, I have frequently dealt with panel members who have little experience evaluating proposals. Frequently, they wished to introduce considerations that are not covered by the evaluation criteria. This has been particularly true when the evaluators had experience with one or more of the competitors.

I prefer to provide the panel with concise, specific, explicit guidelines on what information to consider and on how judge the information against the evaluation criteria. In those cases when the evaluation includes an assessment of contractor experience and past performance, I expect to see specific facts underlying their knowledge of or experience with the contractor, not merely expressions of opinion.

When doing major systems competitions, I can understand the need to consider extrinsic information. In the few competitions I handled involving major systems, the circumstances were similar to those described in GAO’s Northrop decision - Northrop Grumman Systems Corporation, B-406411, B-406411.2, May 25, 2012. I first awarded multiple “concept definition” (CD) contracts for the development of studies and reports. The final source selections took into consideration the contents of the studies and reports developed by the competitors. Section M of the solicitations stated that they would be considered, and the directions to the tech panels referenced them.

If I were to handle another major system’s competition, I would take your advice, but I would be sure to draft the solicitation in a manner consistent with the GAO decisions and to give explicit guidance to the panel on what to consider.

Link to comment
Share on other sites

If I were to handle another major system’s competition, I would take your advice, but I would be sure to draft the solicitation in a manner consistent with the GAO decisions and to give explicit guidance to the panel on what to consider

Thank you napolik. Your response was educational, and I can't say that I disagree. The ability to do so depends on the quality and experience of evaluators.

Link to comment
Share on other sites

Thank you napolik. Your response was educational, and I can't say that I disagree. The ability to do so depends on the quality and experience of evaluators.

Beyond that it, it depends upon the quality and experience of those leading, moderating and managing the evaluation process.

Link to comment
Share on other sites

Guest Vern Edwards

Suppose that a CO had successful removed all identifying information from a set of proposals.

Suppose also that the solicitation told offerors to describe their proposed method of performing the work and that one of the evaluation factors was to be "soundness of proposed method." (There are several "standard" methods of doing the work, any of which would work. But there are several nonstandard methods, some of which are acceptable and some of which are dubious.)

Now suppose that the agency received three proposals -- A, B, and C. Offerors A and B each proposed a standard method, but Offeror C proposed a patented nonstandard method that it asserted was unique to itself and uniquely effective and valuable.

There are five technical evaluators. They evaluated the methods proposed by Offerors A and B as "Acceptable" based on their personal familiarity with those methods, even thought they did not know the identities of Offerors A and B.

But when it came to Offeror C, four of the evaluators knew nothing more about Offeror C's approach than what it described in the proposal. Based on that description, they thought Offeror C's approach was Excellent. However, the fifth evaluator immediately recognized the method as belonging to Offeror C, based on her personal experience with it in another program. She knew things about Offeror C's patented method that the offeror had not described in its proposal, facts that she had documented in an official project manager's diary that she had kept while working in the other program. Those facts were unfavorable. She shared her knowledge with the other evaluators during the consensus meeting, showing them her diary. She was able to validate her unfavorable assertions. As a result of the fifth evaluator's documented assertions, all five evaluators agreed to rate Offeror C's method as "Marginal."

Consequently, the source selection authority chose one of the other two offerors, even though Offeror C had proposed a slightly lower price, and the CO awarded the contract without discussions.

Do you think the evaluation was improper? Do you think the GAO would sustain a protest that is based on what the evaluators did? Can you support your opinion with citations from the FAR or GAO decisions?

Link to comment
Share on other sites

Guest Vern Edwards

I am concerned that some statements made in this thread will mislead some readers. Specifically, I am concerned about assertions that evaluators cannot take into account their personal knowledge and experience except to evaluate offeror experience and past performance. Such assertions are not true.

The thread began with a question about whether proposals must be scrubbed to render offerors anonymous to evaluators. Some persons think that is necessary or appropriate in order to prevent prejudicial evaluations arising out of evaluator biases. But there is no statutory or FAR requirement to do so. It is permissible, and it has been a practice in some offices for many years, but most commenters in this thread, including me, have felt that it is unnecessary and impractical.

At some point in the thread the discussion shifted to evaluator use of personal knowledge and experience in the evaluation of proposals. It was asserted that evaluators could apply their personal knowledge and experience only with respect to offeror past performance and experience, and not with respect to things like resumes, staffing plans, and technical approach. Then came comments about “four corners” and “extrinsic evidence.” Let’s sort this out.

There is absolutely no prohibition against evaluators applying their personal knowledge and experience and other extrinsic evidence when evaluating the content of proposals: the merits of promises and statements of intention, the veracity of assertions of fact, and the validity of inferences. Evaluators are not limited by the “four corners” of a proposal for when performing those tasks. The GAO case law to that effect is well-established and long-standing. See e.g., Northrup Grumman Systems Corp., GAO B-406411, 2012 CPD ¶ 164:

Northrop Grumman argues that it was improper for the Air Force to consider extrinsic information in evaluating Raytheon's proposal. The protester cites to the general instructions in section L of the RFP, which stated that “offerors shall assume that the government has no prior knowledge of their facilities and experience and will base its evaluation on the information presented in the offeror's proposal.” RFP §L.2.0.

An agency's evaluation is not generally limited to the four corners of an offeror's proposal; an agency may rely upon other extrinsic information of which it is aware. PWC Logistics Servs. Co., B –400660, Jan. 6, 2009, 2009 CPD ¶67 at 7–8; Bath Iron Works Corp. B–290470, B–290470.2, Aug. 19, 2002, 2002 CPD ¶133 at 23 (in evaluating proposals, contracting agencies may consider any evidence, even if that evidence is entirely outside the proposal).

So much for that.

What evaluators may not do is credit a proposal with a promise or statement of intention that an offeror did not make. For example, it sometimes happens that an incumbent contractor who has lost a recompete protests that the evaluators did not credit it with an intention that it did not state in its proposal, but that they knew about through their familiarity with the incumbent’s work. See, e.g., Leader Communications, Inc., GAO B-298734, 2006 CPD ¶ 192. In such cases the GAO has held such that “An agency's evaluation is dependent upon information furnished in a proposal, and it is the offeror's burden to submit an adequately written proposal for the agency to evaluate.”

Evaluators may not read promises and intentions into a proposal, unless, perhaps, they are logical and inescapable consequences of promises and statements that an offeror did make. They must evaluate the promises and statements of intention that are in the proposals. Thus, if evaluators know that a contractor did X under a prior contract, but the contractor does not say in its proposal that it will do X under a new contract, then when evaluating the proposal they cannot presume that it intends to do X under the new contract. If they believe that the offeror will do X, then the CO would have to conduct discussions, not clarifications, in order to confirm that belief and request an explicit promise or statement to that effect.

The evaluation of experience and past performance are matters of history, not promises or statements of intention. Thus, when assessing an offeror’s past performance and experience, evaluators must consider any relevant information that they have that is “too close at hand to ignore,” no matter where it came from, even if the offeror did not mention it.

To recap: There is no prohibition against evaluators considering personal knowledge and experience or other extrinsic evidence when assessing promises and other statements in proposals, but they may not use personal knowledge and experience or other extrinsic evidence to credit offerors with promises and statements that they did not make.

Admittedly, the GAO’s statements of the rules in this regard are confusing and require a lot of close reading of a number of decisions issued over the course of many years. See the discussion in “Evaluation of Offers: Using Information Not In Proposals,” in The Nash & Cibinic Report (December 2003), 17 N&CR ¶ 61. But I hope this post prevents any confusion.

Link to comment
Share on other sites

I am concerned that some statements made in this thread will mislead some readers. Specifically, I am concerned about assertions that evaluators cannot take into account their personal knowledge and experience except to evaluate offeror experience and past performance. Such assertions are not true.

The primary source of confusion is probably my statement in post #3.

“While use of evaluators’ personal experiences may be acceptable when assessing contractors’ experiences or past performances, it is not acceptable when evaluating other factors (e.g. résumés, staffing plans, technical approach, etc.).”

I wrote this very poorly. I was describing my views on what tech panels should be directed to evaluate given my experience with them. I was not stating GAO views.

As you point out, the GAO has issued a number of decisions reaching back to the 80s on the use of “extrinsic evidence” that is outside the proposal so long as the use of extrinsic evidence is consistent with established procurement practice.

I would not direct the evaluators to look for extrinsic evidence except in cases involving past performance / experience evaluation factors, but if it was discovered, its source and relevance to the evaluation must be examined carefully in light of the solicitation’s evaluation criteria and the GAO decisions.

BTW, I could only locate a single COFC decision addressing extrinsic evidence and info outside the four corners of the proposal: Contracting, Consulting, Engineering, LLC v. U. S. and DYNCORP International, LLC , No. 12-97C, April 16, 2012.

The case dealt with the use of evaluators’ “personal experience” to evaluate résumés. The judge allowed consideration of the evaluators' personal experience, but she concluded that it was applied irrationally.

Link to comment
Share on other sites

Guest Vern Edwards

The rules about "extrinsic evidence" (also called extrinsic information, extraneous information, and extraproposal information) are confusing. Let me restate them:

1. Evaluators may not use personal knowledge or other extrinsic information to attribute to an offeror a promise or statement of intention that the offeror did not make in its proposal. See Anderson Consulting, Inc. v. U.S., 959 F.2d 929, 934 (Fed. Cir. 1992), in which the Federal Circuit affirmed a GSBCA protest decision, saying that evaluators may not attribute to a proposal the "subjective, unexpressed intent" of an offeror.

2. Evaluators may use personal knowledge and other extrinsic information in order to assess promises and statements of intention that are in a proposal. See Northrup Grumman Systems Corp., GAO B-406411, 2012 CPD ¶ 164. It appears that evaluators might be able to make inferences based on information that is in a proposal in order to attribute an offeror something that it did not expressly mention in its proposal.

3. Evaluators must use personal knowledge or extrinsic information concerning relevant past performance when evaluating past performance. See Nuclear Production Partners LLC; Integrated Nuclear Production Solutions LLC, GAO B-407948, 2013 CPD ¶ 112.

The GAO has expressed itself very confusingly in this matter by relying on stock phrases in its decisions and not sorting things out properly. The only way to sort it all out is to read numerous decisions very carefully. Moreover, the GAO case law appears to have evolved over time. When parsing the GAO's position, it's better to rely on relatively recent decisions, those from 2000 onward, rather than going back to decisions of the 1980s.

Link to comment
Share on other sites

I always caution my evaluators that they cannot 'un-know' something that they know, and that they should pipe up and tell the group if they have that information close at hand. I'm a little suspicious of any policy (such as redacting the company name) which makes our evaluators know less about what they are reading, rather than more.

Upon reading this thread, I will also emphasize point 1. above, which I have not been doing as explicitly as I should . . .

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...