Every now and then (actually, more often than that) an agency conducts a source selection in a way that stupefies me. Such is the case with NASA's $1.62 billion source selection for protective services at 14 NASA sites, such as the Kennedy Space Center. The contract was to be a firm-fixed-price, indefinite-delivery indefinite-quantity (IDIQ), for five years with five one-year options. It appears that an FFP task order was to be issued for each NASA site for which such services were to be acquired.
NASA published a sources sought synopsis on March 7, 2007. It published its presolitication synopsis on May 24. It issued its 2,600 page RFP, including a 72-page "performance work statement," on September 14, and amended it nine times. The RFP required offerors to submit proposals by November 30. The proposals were to include technical and management volumes, each of which was limited to 150 pages. In addition, offerors had to submit past performance, safety program, and pricing information.
The agency received five proposals, conducted discussions, and awarded the contract on May 20, 2008. One of the losers submitted a protest to the GAO on June 10, which the GAO denied on September 10. See Wackenhut Security Services, Inc., Comp. Gen. Dec. B-400240, 2008 CPD ? 184. The same protester then filed a complaint at the U.S. Court of Federal Claims on September 17. On December 15, Judge Susan G. Braden found for the plaintiff on the grounds that NASA had failed to explain and justify its proposal ratings, thereby violating the FAR and the Administrative Procedures Act. See Wackenhut Security Services, Inc. v. U.S., No. 08-660C, -- Fed. Cl. ---. The court granted injunctive relief by setting aside the contract award and ordering NASA to appoint a new source selection board and a new source selection authority. Those fresh faces must reevaluate portions of the offerors' final proposal revisions, make a new award decision, and send a copy of the decision memo to the court.
Why did NASA lose? The main reasons, in my judgment, are that (1) it developed a source selection plan that was needlessly complex and (2) it didn't maintain a good paper trail of its evaluation process. I haven't seen the source selection plan, but I cobbled it together by reading the NASA regulations, the RFP, and the two protest decisions. (I downloaded the RFP from Federal Business Opportunities: RFP No. NNX077040R, issued by the John F. Kennedy Space Center on September 14, 2007). The agency ran an essay writing contest, had too many evaluation factors and subfactors, and imposed absurd proposal requirements.
The offerors had to prepare technical and management volumes, limited to 150 pages each (8 1/2 by 11, one inch margins, 12 point font--no line spacing limitations). To give you a sense of the proposal requirements, as part of the technical proposal the offerors had to prepare two-page responses to seven scenarios like the following:
A 9-1-1 call reported a disgruntled employee has barricaded himself in an office on the 4th floor of a building. It is reported that the Subject was armed with a handgun and yelled out that he had a bomb. He has 4 hostages; no shots have been fired. Explain what action you would take as an emergency responder to defuse this situation and provide a safe rescue of the hostages and Subject.
Assumptions: You are the incident commander at the location. You have jurisdiction to conduct the full range of protective services operations. Federal agencies are available to you for assistance, however none can respond within 45 minutes.
Who would you involve in the response? What would be your tactics and strategies? Explain how your actions may have been different if shots were fired before, after, or during your response. Explain your after-action procedures.
You would have to be an idiot to have a law enforcement background and yet be unable to write a sensible two-page response to that scenario. If you are an idiot, you could hire a consultant to write it.
After the 14 pages of such scenario responses, offerors had to use the remaining 136 pages for such as the following:
The Offeror shall describe their overall technical approach for accomplishing the contract, PWS, and each of the task order requirements. Offerors shall demonstrate their understanding through a separate discussion for each task order by including, but not limiting the discussion to, the following:
a. Identify critical work functions, in relation to your identified critical skills provided as part of your staffing plan below, to be accomplished within this contract, PWS, and task orders and describe your approach for ensuring these critical functions are accomplished.
b. Demonstrate your understanding of the key performance characteristics of the contract and describe the key metrics you would use to measure these performance characteristics. In addition, describe the benefits to management insight that the metrics will provide.
c. Quality Management.
(i) Submit a Quality Control and Assurance Plan DRD.
(ii) Describe an overall approach for ensuring that personnel are appropriately
qualified and/or certified for their specific duties.
d. Specific Technical Understanding and Resources. Offerors shall demonstrate their understanding of the requirements and the specific labor resources needed to successfully perform the requirements of this contract and the task orders, utilizing their proposed technical approach.
e. Describe how Work Management and Control will be structured and accomplished, including all delegations of authority, to meet the contract requirements specified. Describe your process for receiving, scheduling, tracking, completing, and closing out work. Describe the mechanism you will use to match the correctly trained and skilled manpower to the work requirements.
After that the offerors had to use whatever pages were left for a staffing plan and for the following:
Describe the innovative techniques you plan to employ to maximize operational efficiencies and achieve contract stated objectives. Describe in detail specific plans, strategies, and practices for enhancing standardization and uniformity across NASA. Describe any innovations and operational efficiencies you plan to implement and gain. Provide rationale for proposed innovations.
(What is it with government acquisition people and the word "innovative"? They love that word. They use it all the time. If they treasure innovation so very much, why do they keep conducting source selections in the same old, dumb, labor-intensive, and time-consuming ways? Come on people! Where's your sense of irony!)
Remember that the technical volume was limited to 150 pages and that the PWS was 72 pages long. Now here is a paragraph from the PWS, to give you the flavor of the thing:
188.8.131.52 Emergency Response Team
A. The contractor shall provide a team of security police officers who meet the enhanced physical standards for specialized teams found in NASA NPR 1620.2.
B. The team shall be trained in hostage and victim rescue, special tactics and planning, and the use of specialized equipment, firearms and other weapons.
C. The team shall be capable of an immediate on-site response to effectively and safely resolve a variety of critical security incidents including, workplace violence, active shooter, hostage situation, and terrorist aggression.
D. The team shall develop detailed plans for operations in critical/hazardous and public facilities.
E. When not training, these officers shall assume normal security police officer patrol duties as described in the Task Order; coming together as a team only as needed.
Think of how many of the 150 pages you could use to describe your approach to keeping a team ready for hostage rescue. And that's only one of at least 100 tasks. (Interesting work for a contractor, eh? Hostage rescue? With the rescuer chosen on the basis of an essay-writing contest and price?)
The evaluation findings were to be assembled into a goofy apparatus of "regular" strengths, significant strengths, "regular" weaknesses, significant weaknesses, adjectives, percentages, and numbers. ("Regular" strengths and "regular" weaknesses, I love it. You can't make this stuff up.) The protester was upset because although it got more points than the winner (1,788 to 1,760) and the same past performance rating (very good), NASA gave the contract to the other company because it's price was lower. Among other things, the protester complained that NASA was too generous in assigning points to the winner and that the point spread did not reveal the true superiority of its proposal over the winner's. Ordinarily, that would be a pretty weak protest, but NASA's problem turned out to be that it couldn't explain the winner's score to the judge.
(According to a comment in the court's decision, the protester's price was five percent higher than the winner's. The difference was significant in absolute terms. According to the GAO decision, the winner's final evaluated price was $1.186 billion. A five percent increase would make the protester's price $1.245 billion, a difference of $59.3 million.)
If an agency is going to conduct a source selection as an essay-writing contest, and no agency should, it better develop an auditable paper trail from the contents of each proposal through each evaluator through to any consensus findings through to the decision memorandum. I know that the GAO does not require agencies to maintain individual evaluator working papers or that individual evaluator findings track to consensus document(s) or to the final decision (see, e.g., Smart Innovative Solutions, Comp. Gen. Dec. B-400323.3, November. 19, 2008). But agencies had better maintain an audit trail anyway, because the Court of Federal Claims does not always go along with the GAO. In this case, the judge was troubled by her inability to track changes in scoring from evaluators' "preliminary worksheets" to their "final worksheets." During oral argument, the judge told the government's lawyers:
When [the Source Evaluation Board] began the technical evaluation [the winner's] score was [deleted], and it popped up to [deleted], moving it up to an [deleted] rating. There is not a word in any of those worksheets that justifies why that increase happened. Not a word...
So I have no basis in the record, from what I can see, to decide whether that jump in score, which was significant in terms of who received the contract, whether what they did was rational or not because there's no explanation. All we have is the scores...
So it makes me wonder what in the world happened? How did they get all those additional points and increase their rating, and why was none of that discussed?
Okay. Now we have the same problem, but to a lesser extent, in the small business factor score...
They get a [deleted] [point] increase in their rating, and yet when you look at the evaluation sheet--there's only one evaluation sheet, by the way, on that--not one word about why the scores change and based on what.
So I don't have an ability to say based on this record that what they did was appropriate on those two areas... [T]o me, those two flaws alone, aside from whatever else [the plaintiff has been] arguing, are really fatal.
Those SEB problems are fatal to the bid protest because of their significance in how the overall rating was presented to the source selection authority. Do you see?
They do now, Your Honor. Now they see that if they're going to run an essay writing contest they'd better (1) make sure that they know where and how each sheet of paper fits in the audit trail and (2) explain the logical progression from offeror statement to findings, ratings, or scores, and ultimately to source selection decision. And their premises had better be true and their logic valid.
So let's sum this up: NASA started this procurement sometime before May 27, 2007. It took almost six months to evaluate five proposals and to award a contract on May 20, 2008. It received two protests, and had the award thrown out by the Court of Federal Claims on December 15, 2008. It now must reevaluate parts of the proposals and make a new decision. A lot of time and effort down the drain and more to come.
That is mission failure, plain and simple.
An agency is not supposed to lose a protest of a $1.62 billion source selection because it did not adequately document the basis for its decision. That's just not supposed to happen. I do not doubt for even a moment that the NASA participants will argue that they did everything by the book and that the judge was out to lunch. Maybe she was. But out to lunch or not, if she's judging your protest you have to make a believer out of her.
Here's my take: Every source selection decision is a rhetorical challenge. When writing the decision memo, the SSA must argue that the decision is sound, i.e., consistent with law and regulation, the terms of the solicitation, and the dictates of reason. The SSA must thus persuade a fair but skeptical judge--whether the judge be the GAO, the Court of Federal Claims, Congress, the press, or the public. (Congress, the press, and the public won't be fair.) If the SSA fails to do that, then he or she has failed to do the job properly. Period. No excuse is acceptable. In any event, all excuses are pointless.
Who is to blame for this failure? Who is accountable? Or is it like in the Howard Jones song: "No one, no one, no one ever is to blame"?