Jump to content

Experiments in Service Contracting?

Posted by Vern Edwards, 01 March 2011 · 5,598 views

Show of hands: How many of you think that the government can reduce costs under cost-reimbursement service contracts by tripling the number of government personnel involved in contract administration?

Steven Kelman, Harvard professor and former OFPP administrator (and the best one we?ve ever had), has proposed an ?experiment? to do just that in the February 14, 2011 print edition of Federal Computer Week, page 15. Read it here: http://fcw.com/artic...management.aspx. (If the link doesn't work, copy it and paste it into your browser.)

Prof. Kelman begins his commentary by pointing out that the government spends a lot on services every year and that service contract cost savings would contribute a great deal to deficit reduction. Agreed. He then says: ?We all know anecdotally about unnecessary work performed under service contracts and insufficient cost controls? .? Well, yes, I have heard that, and I believe it?s true to some extent, although I cannot say to what extent. He then makes an analogy between cutting service costs and house-to-house fighting in war. (An unfortunate analogy, but I?ll let that go.) He then says:

To help us make progress on that house-to-house fighting, I propose that we provide more resources to the people doing the fighting ⎯ the contracting officers and program officials who manage the contracts.

So he wants a surge. He then says:

Many contracting professionals believe that providing more contract management employees can pay for itself ⎯ perhaps many times over ⎯ in cost savings and, ideally, performance improvement. I am inclined to agree, but nobody knows whether than assumption is correct.

He then proposes an experiment: (a) choose 20 cost-reimbursement contracts for ongoing services worth $5 to $10 million each, (b) double or triple (or maybe quadruple) the number of contract management personnel current assigned to each of them, ( c) track them for two years and then (d) compare the costs with 20 similar contracts that do not receive additional personnel.

If costs go down and/or performance goes up on the contracts receiving additional resources, we should use that approach more broadly in the future.

Steve acknowledges that the details of the experiment would have to be worked out.

I have strong ideas about services and service contracting. I have explained them in a number of articles published since the late 1990s, many of them are available only to subscribers to Westlaw. But Ralph Nash and I wrote an article for the September 2007 issue of Defense Acquisition Review Journal in which my ideas are briefly described. That article is accessible here: http://www.dau.mil/p...J45_Edwards.pdf. Three of my thoughts are summarized in the following three passages from that article:

This article proposes that there are two categories of services, and that PBSA as it is known at the beginning of 2007 works for one, but not the other. The first category includes many common, routine, and relatively simple services that can be acquired through PBSA as it is currently defined, including many housekeeping services, simple equipment maintenance and repair services, and the like. The second category includes services that are too long-term and complex to permit complete specification of results and competitive pricing at the outset of contracting. These include many long-term information technology services, services to operate government-owned facilities, and long-term and multifunction or multitask professional, administrative, and management support services. These are the services for which the government spends the most money.
* * *
In real life, parties to long-term and complex service contracts do not specify all requirements at the time of contract award in clear, specific, objective, and measurable terms. Instead, they engage in ad hoc decision making in response to emerging and changing requirements, shifting priorities, and unexpected circumstances. They make it up as they go along, developing and adjusting expectations and agreements accordingly. Reality is never the same as expectations and projections, and plans and agreements go awry. No matter how long and hard future needs are considered, contracts will include things that will not be needed and leave out things that will be. Specifications and expectations must be adjusted over the course of time.
* * *
Unlike most supply purchases, long-term service contracts entail close human relationships that enable the parties to deal with dynamic complexity and respond to emerging and changing needs and circumstances. Relationships are crucial, and it is well established in service marketing literature that subjective customer satisfaction is as important, and sometimes more important, than technical success. [Footnotes omitted.]

Now, with those passages in mind, here is what I think about Steve?s idea: While I love experiments, I would distrust results produced by any such as he describes. Why?

First, because it would be virtually impossible to establish a valid control group of 20 ?mirror? services, due to (a) differences in the conduct of competitions for contract award, (b) differences in the events and the circumstances of performance, and ( c) differences in the management styles of the government personnel involved. Every contract for long-term complex services is somewhat unique and not entirely comparable to others. Even a contract for a given service at a given location might not mirror its predecessor contract.

Second, without a valid control group to serve as bases for comparisons, it would be impossible to reliably measure savings and quality differences. Moreover, applying the experiment to cost-reimbursement contracts is problematical, because the use of a cost-reimbursement contract suggests that there is significant uncertainty with respect to cost outcomes. A cost underrun (which is unlikely in any event) could be due to savings or to estimating error. On the other hand, an overrun might reflect savings from what the costs more accurately otherwise would have been. Failure to underrun, or an overrun, might reflect a decision by the agency to use all of the funds available to it for the contract in the fiscal year.

Third, such experiments in acquisition are almost always political events, more akin to gimmicks than exercises in the scientific method. In my experience, the people who conduct them often are not trying to test a hypothesis, but trying to prove that their hypothesis is right. Thus, the tendency is to conduct the ?experiment? and then declare it a ?success? in the sense that it confirms an idea that supports a preferred course of action. Then all sorts of gurus and consultants jump on the bandwagon touting the rightness of the idea.

Performance-based contracting is a classic illustration. Something like Steve?s new idea was tried in the mid-1990s with performance-based contracting. See 59 FR 26679-01, May 23, 1994:

To stimulate the government's conversion to performance-based service contracting, OFPP has developed a government-wide pilot project which relies on voluntary pledges by individual agencies to convert specified contracts for services to performance-based contracting methods. Agencies will be encouraged to limit their pledged contracts to recurring requirements to facilitate before and after measurement and assessment of results. Agencies will also be asked to consider breaking up large level-of-effort umbrella contracts in order to increase competition and convert to completion-type contracts where possible, thereby generating more innovation and cost effective proposals.

Agency pledges are also expected to include:

Endorsement of the project by the agency head; establishment of a high-level agency task force to oversee the effort; agreement to share lessons learned and best practices with other agencies; and measurement of the results of the project using predetermined project measurement criteria. The proposed project measurement criteria are: contract price; level of competition; number of contract audits; customer (project officer) satisfaction; length of procurement cycle; and small business participation.

In May 1998, OFPP published a report of the results of the project, available here: https://www.acquisit...on-the-perf.pdf. The results are summarized nicely in the October 1998, ?final edition? of OFPP?s Guide to Best Practices for Performance-Based Services Contracting:

As a result of the pledge, 15 agencies converted 26 contracts with an estimated value of $585 million to performance-based methods. The agencies reported an average 15 percent reduction in contract price in nominal dollars, and an 18 percent improvement in satisfaction with the contractors? work. Moreover, reduced prices and increased customer satisfaction occurred at all price ranges, for both nontechnical and professional and technical services, and whether the contract remained fixed-price or was converted from cost reimbursement to fixed-price.

The Guide is available at http://georgewbush-w...c.html#chapter1.

The report itself concluded:

The results of the project strongly support the hypothesis that PBSC, when fully and properly applied, enables agencies simultaneously to obtain improved performance and reduced prices. PBSC is appropriate for professional and technical services as well as non-technical services, and for large, complex contracts as well as small contracts. Moreover, PBSC?s benefits are amplified when awarded using fixed price contracts.

It would be an understatement to say that the "report" did not back up its claims with hard data about specific contracts or offer to provide such hard data upon request. Thus, it was impossible to independently verify the claims. As far as I?m concerned, unverifiable claims for the results of pilot projects, or experiments, are just that. To put it nicely, they are hooey. The confirmation of OFPP?s ?hypothesis? was a foregone conclusion and about as scientifically valid as professional wrestling and roller derby, which are at least fun to watch.

?Pilot project,? ?experiment,? it?s all the same.

That is not to say that increasing contract management staffing would not produce good results. Let?s consider: What would be the likely effect on cost and quality of putting more government personnel to work managing cost-reimbursement service contracts.

On the one hand, greater monitoring might detect more instances of subpar performance, which might lead to prompt corrective action and better outcomes. It might facilitate better communication between the government and its contractor. With more eyes on the contractor, the government may be better able to spot inefficient practices and performance shortfalls in progress. If we assume that to be true, then adding people would be beneficial.

But, on the other hand, adding government personnel might increase costs if the they try to squeeze more work out of the contractor or slip in unplanned requirements. It might increase costs without much to show for it if they are persnickety and demand reperformance or other corrective action in each and every instance of customer dissatisfaction, however minor or inconsequential the quality shortfall. Such a management approach will likely reduce customer satisfaction in the long run and make the government-contractor relationship more tense and disfunctional. No compromise, no savings, and a bigger bill without much real improvement.

Yet, on the other hand, if the management approach is to be helpful and cooperative, rather than merely corrective and demanding, good things could happen: cost inefficiencies might be detected and efficiencies might be found, and customer satisfaction might be improved.

Here?s a hypothesis of my own: While the government needs enough people to properly monitor performance, the quality of the government personnel, not the number, is the most important factor in the government?s contribution to performance. In other words, one wise and effective contracting officer's technical representative is better than two or three nitpicking nuisances. How can we test that hypothesis? I don?t know. Like I said, experiments in government contracting are dubious affairs.

But why test it? Why not just accept several decades of management thinking and assume that (a) the government needs the right number of people for the job, which depends on consideration of several factors ⎯ number of functions and tasks, number of performance locations, number of contractor personnel, etc. ⎯ and should not be determined by gross doubling or tripling of the current number; (b) they have to be the right kind of people ⎯ wise, competent, good communicators and capable negotiators; and ( c) they have to have the right kind of contractor to work with ⎯ honest. cooperative, capable, and seeking to please.

But some government managers like demonstrations of ?success,? and so somebody is likely going to want to conduct Steve?s ?experiment.? And why not, if it can lead to getting more personnel? After all, the time honored government management approach, when you don?t know what else to do, is to throw people at the problem and hang on to them until the next round of budget cuts. And whatever else you do, declare success.

I frankly cannot see the point in this kind of thing, which strikes me as gimmicky, as is so much of acquisition management, with its experiments, pilots, and initiatives. We have done those kinds of things many times and I don't see that they have gotten us to where we need to be. What we need to do instead is the hard work of determining, on a case by case basis, the number and kind of people that we need for each contract, how we're going to get them, and from where. It is likely that we will never have enough people, which makes it all the more important that we have the right kind of people. Where do we find them? How do we prepare them for the task?

Why can't the government get past the "Any Butt in a Seat" philosophy of personnel management?
  • Report
?Tense and dysfunctional? would be a good description of the government-contractor relationship on the 10-year, $230+M FFP/IDIQ multi-functional BOS contract at my installation. Unfortunately the government approach is to be corrective and demanding, and this is whether or not the work is actually in the contract. Also, the PWS was written by people who don?t understand contracts, and wanted to be sure they threw everything they could into the contract to ?correct? perceived problems from the previous BOS contract so they could have a way to ?get? the contractor. With this type of mind-set it?s hard to have a decent working relationship. And although I?m talking about a FFP contract at my installation vice cost-reimbursement, I don?t think such an experiment would work. You?d have to start with a fairly well written contract to begin with, and then have the right people with the right training and attitude to oversee it, then this could be a worthwhile experiment. Good luck finding such people, and such a contract. I do like the idea of relational contracting, but again you have to have the right people with the right training and right attitude to make it work. Good luck!
  • Report
Vern - your argument no-doubt has merit drawn from the biases associated with any such test. However, even well methodically performed and statistically sound testing techniques contain judgment (e.g. precision, error acceptance, etc.), resulting from the inherent risks. Could some of these factors/elements of risk be mitigated to an acceptable level? For example, use a qualified third party to conduct the test and develop the control (i.e. independent coefficients) and provide proven (howbeit difficult as you pointed out) results. Essentially, what I?m saying is that there is enough similarity to provide an acceptable analysis. Such dynamic features (containing many variables) are faced by many disciplines, other than contracting, who have overcome them, not eliminating the variability entirely. Maybe a system that determines the risk of cost-reimbursable that generates a general level of required resources at an optimal level may provide a solution. Identifying independent variables leveraged from actual data (assuming the data is good) with highly correlated data to draw conclusions. It?s hard not to believe that no econometric model could be developed to depict this condition with all the required documentation that the Government and Contractors already posses. At the end, however, it may just prove your point -- it's not possible. Yes -- Steve overly simplified the testing.

Your solution of qualified (in all its forms) personnel offers a utopian picture, so worthy of more time to discuss, but so would honesty and forthcoming on both contractor's and government's part. Until such a world comes about, we must traverse the valleys and hills of the multispeed treadmill of contracting.
  • Report