Jump to content

More effective use of past performance evaluations


formerfed

Recommended Posts

I've noticed several new attempts to speed up awards and reduce time and effort required by both government and industry. Here's one example. I highlighted this one because of the attention Anne Rung gave.

http://fedscoop.com/hhs-buyers-club-an-innovative-acquisition-case-study

Essentially HHS used a streamlined process to select a contractor for IT development work. The contracts office initially faced a 26 page SOW and looked for something quicker because "they were clearly headed down the path of a traditional six-month procurement likely resulting in the selection of a contractor with the best written proposal." What they came up with is a process fairly typical for Agile where a limited number of sources get selected for prototype development. Then a selection for long term development is made based on the prototype results.

Since there are multiple ways to quickly reduce the number of potential sources, it looks like the real problem needing solved is how do you know which source is really the most capable? Seeing actual prototype results rather than just reading a proposal is an easy answer but is that conclusive?

I'm wondering if more effective use of past performance is a better solution. I don't mean the more typical means of sending out email questionnaires or canned phone call questions to references and checking PPIRS. I'm thinking doing comprehensive research and finding the most relevant customers. Once located, go through a fact finding process which may include site visits and seeing how well the offerors delivered on their promises. In the case of IT, the questions are do the products do everything intended, are the users satisfied, was the work performed within budget, were user long term needs truly accommodated (this is a very important consideration with Agile), etc.

What do you think?

Link to comment
Share on other sites

formerfed,

You frame the question as one approach vs. another. I think that both the past performance of an offeror (evaluated in the manner you described) and the performance of a prototype would be valuable information to a decisionmaker. I see no reason why they couldn't both be considered.

Link to comment
Share on other sites

Guest Vern Edwards

I'm wondering if more effective use of past performance is a better solution...

What do you think?

I don't think so. I was an early supporter of past performance (1987). The use of past performance has a long history. It is a useful factor, but it probably should not be determinative in the acquisition of complex services, for more reasons than I have time to explain. Source selection is important, but the fact is that you cannot ensure success in the acquisition of complex services through the source selection process. A good source selection decision is a must, but successful project execution requires much more.

What I find interesting in the series of stories to which your link led me is that the "innovation" probably is not new, but the stories do not provide enough detail to make it clear just what rules applied to the acquisition in question and what procedures were used. How big was that procurement? Was it commercial item? Was it done under FAR Subpart 13.5?

The "prototype" "staged" approach described in the fedscoop blog entry < http://fedscoop.com/hhs-buyers-club-an-innovative-acquisition-case-study > seems like the "competitive fly-off" approach used by DOD since the early 1960s, but I can't be sure. I used something like it for the classified "Talon Gold" project in 1982, and it wasn't new then. So HHS's IDEA Lab just thought it up? HHS? I don't think so, and Ms Rung may not have enough background in acquisition to know a new technique when she sees one. Naggar's flow chart in one of the articles is the typical work product of an ambitious and well-meaning "innovation" salesman.

As for speeding up the source selection process, many suggestions have been made over the years. There was a big "streamlining" movement (the current term for this is "agility") in the 1990s, but few of the suggestions were fully and widely adopted and perfected. Instead, people kept coming back to the old "technical" proposal/SSEB procedure, despite its obvious and well-documented weaknesses and deficiencies. Like Chuckie, it just keeps coming back.

What Ms Rung ought to be doing instead of podcasts is working on Congress to revise the Competition in Contracting Act (CICA) to allow more procedural flexibility by dropping the requirement that price be an evaluation factor in every source selection. That would open the door to speeding up the process and improving project design and contractual implementation by separating contractor selection from contract formation.

Link to comment
Share on other sites

I don't think so. I was an early supporter of past performance (1987). The use of past performance has a long history. It is a useful factor, but it probably should not be determinative in the acquisition of complex services, for more reasons than I have time to explain. Source selection is important, but the fact is that you cannot ensure success in the acquisition of complex services through the source selection process. A good source selection decision is a must, but successful project execution requires much more.

Vern,

Assuming that we must operate within the current rules, what evaluation factors do you think are better than an offeror's past performance and experience in acquisitions of complex services?

Link to comment
Share on other sites

Guest Vern Edwards

It depends.

I consider experience (E) and past performance (PP) to be bedrock factors. Always evaluate E and PP. The real question is: What else, if anything, should you consider, and when, if ever, should you consider anything else to be more important?

Think about the Air Force and the new strategic bomber. I understand that the competitors will be a Boeing-Lockheed team and Northrup Grumman. What's the point of evaluating E and PP? They all have lots of E, and they all have issues with PP. I think those factors will be a wash. So what else should you consider? What attributes of a contractor will make for greater or lesser chances of success, however that is defined?

If you are hiring someone to manufacture a product according to government specifications, production capability and capacity might be more important that E and PP in a given case, depending upon the competitor pool.

If you are buying a commercial product, then product performance (broadly defined and distinguished from offeror past performance) might be more important that E and PP.

If you are buying custom commercial services, reputation might outweigh everything else. Right now my wife and I are undergoing a major house remodel. We have done several, and we learned early on that reputation outweighs everything else. For years now we always go with the same company, the best in Portland, also the most expensive, but also the absolutely, positively most responsive and reliable. We no longer even think about anyone else and price is a factor only in saying what we want, not whom we choose.

So, it depends -- on what you're buying, the state of the technology, the industry, the market, and your budget. Start with E and PP, then decide if you need to consider anything else and, if so, if anything else is more important than E and PP.

Link to comment
Share on other sites

You frame the question as one approach vs. another. I think that both the past performance of an offeror (evaluated in the manner you described) and the performance of a prototype would be valuable information to a decisionmaker. I see no reason why they couldn't both be considered.

Don,

They certainly could be used together. But I'm think if past performance was done in the manner described, agencies could get to award quicker and less effort for both industry and the government. From what I gather this process involves multiple awards initially. Each contractor performs for a certain period of time, delivers their products, the government evaluates and makes a selection decision. That involves a lot of time and expense for everyone.

My thinking is could the government through the past performance process gain enough information and insight about the offerors to make a selection without the prototypes? I believe they could assuming the work is the typical IT development.

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...