Jump to content

Performance Price Tradeoff (PPT) Method used by Air Force - How does this work?


govt2310

Recommended Posts

I am researching the Performance Price Trade-off (PPT) method used by the Air Force. I have never seen one done before at my agency. We are not the Air Force.

FACT PATTERN:

Say an agency does PPT. It announces in the solicitation that the the evaluation criteria and methodology will be as follows:

Price and Past Performance are weighted equal.

The agency will first RANK proposals by PRICE. So the agency will do a Price Reasonableness analysis, and then rank proposals from Lowest Price to Highest. If a proposal's price is found unreasonable (meaning TOO HIGH), it will be thrown out. The agency also says in the solicitation that it will make a cut-off at the first top ten proposals on this ranking.

So if the agency rank proposals #1 to #25, lowest-priced to highest-priced, even if all 25 proposals had prices that were evaluated as "reasonable," the agency would only take proposals #1 to #10.

The Solicitation further states the following:

The Government will initially evaluate for technical acceptability the ten (10) lowest priced proposals. If the government receives less than 10 proposals, all proposal will be evaluate for Technical Acceptability. If the Government receives 10 or more proposal, the Government will rank the proposals by initially proposed total price, and then select a minimum of the 10 lowest priced proposals, and then evaluate only those proposals for Technical Acceptability. If any of the proposal are found to be Technically Unacceptable, the unacceptable proposal will be replaced with the next lowest priced proposal on a one-for-one basis, until there are a minimum of 10 technically acceptable proposals, or until all proposal have been evaluated for technical acceptability.

So proposals #1 through #10 move on to the next phase: the agency will evaluate technical acceptability on a GO/NO-GO or PASS/FAIL basis. Only the proposals found Technically Acceptable will move on to the next phase: evaluation for Past Performance.

Any proposals found to have a rating of "LIMITED CONFIDENCE" or "NO CONFIDENCE" are thrown out.

Now the agency sets a COMPETITIVE RANGE and holds DISCUSSIONS. It allows for revised proposals and then evaluates on the criteria and methodology in the solicitation again.

Now, the remaining proposals are included in the SOURCE SELECTION DECISION. The agency will go down the list until it reaches the first offeror's proposal with a rating of "HIGH CONFIDENCE" on Past Performance. That offeror gets the award. If that offeror was not the first guy on the list, then the agency must write a TRADE-OFF decision.

MY QUESTION:

During the first round of evaluations, before the competitive range cut, if I was offeror #11 on the ranking from lowest price to highest price, shouldn't I be able to protest at GAO or COFC on the grounds that this artificial, mechanical "top ten cut-off" was a de facto competitive range decision WITHOUT evaluating my proposal on ALL evaluation factors, as is required by FAR 15.306©?

Link to comment
Share on other sites

Additional Info:

According to the Air Force "Informational Guidance" on PPT, it states in Section 5 that the Contracting Officer can limit the number of proposals evaluated to a "specified number." For example, it could state that Price would be evaluated first, so a Ranking by Price would be established, and then only the top 10 lowest priced proposals would move on to be evaluated for Technical Acceptability.

I really find this strange. I can't understand why this is NOT considered a violation of FAR 15.306© . . . What am I missing?

Link to comment
Share on other sites

Hasn't this issue been discussed more than once in this forum? Have you searched the forum archives at all? The most recent discussion I can readily recall is at http://www.wifcon.com/discussion/index.php?/topic/2269-can-a-proposal-be-excluded-based-solely-on-price/?hl=lpta. Give it (and the Gold Cross decision cited in Post #4) a look.

If you search, I expect you'll find a couple of other discussions, too.

Link to comment
Share on other sites

Guest Vern Edwards

In a tradeoff process source selection in which the only criteria will be price and past performance, which will be of equal importance, I think that a process in which you evaluate the past performance of only the first ten proposals, ranked on price, from lowest to tenth highest, will potentially be found to be arbitrary. What if #1 is priced at $990,000, #10 is priced at $1,000,000, and #11 is priced at $1,000,001? And what if #11 turns out to have the best past performance of all?

I would not say such a thing in an RFP. On the other hand, if #10 is $1,000,000 and #11 is $5,000,000, you might not have a problem.

Link to comment
Share on other sites

Navy_Contracting_4:

Thank you. Sorry, I did not think to search the discussion board archives. Well, I looked at that thread and the Gold Cross decision. Gold Cross held that if the agency found the offeror's price "unreasonable," it could leave it out of the Competitive Range. That is not the scenario in my fact pattern above. In my fact pattern, offeror #11's price was found reasonable, it's just that he's #11, and the agency is only taking the top 10. So Gold Cross doesn't answer my question.

Vern Edwards:

Yes, this is exactly the scenario I am worried about. If #10 is $1,000,000 and #11 is 1,000,001, and both are deemed reasonable prices, if I was #11, I would protest. However, this is exactly the evaluation methodology spelled out in the Air Force Guide. I don't understand why the Guide says this.

Link to comment
Share on other sites

Navy_Contracting_4:

My apologies. I just reread the Fact Pattern I wrote that began this thread. I did indeed say that proposals #1 to #10 were deemed "reasonable," implying that proposals #11 and onwards were not reasonable. I just now went back and EDITED THE FACT PATTERN above to make it say what I originally meant to say. Sorry again.

Link to comment
Share on other sites

The agency will first RANK proposals by PRICE. So the agency will do a Price Reasonableness analysis, and then rank proposals from Lowest Price to Highest. If a proposal's price is found unreasonable (meaning TOO HIGH), it will be thrown out. The agency also says in the solicitation that it will make a cut-off at the first top ten proposals on this ranking.

So if the agency rank proposals #1 to #25, lowest-priced to highest-priced, even if all 25 proposals had prices that were evaluated as "reasonable," the agency would only take proposals #1 to #10.

The remaining proposals #1 through #10 move on to the next phase: the agency will evaluate technical acceptability on a GO/NO-GO or PASS/FAIL basis. Only the proposals found Technically Acceptable will move on to the next phase: evaluation for Past Performance.

How could a price be determined reasonable prior to determining technical acceptability?

Link to comment
Share on other sites

jwomack:

Are you saying that Technical Acceptability MUST be determined first? I used to think that, but after seeing the Air Force Guide and some Performance Price Trade-off (PPT) GAO decisions, I now see that is not true. There are numerous GAO decisions where protesters have protested Air Force procurements using the PPT method. Interestingly, all the GAO decisions I have seen so far on this only involve a protest on the grounds of misevaluation. None of them concern a protester saying the PPT "evaluation methodology," where often Price is evaluated and ranked first, was somehow unfair. None of them concern a protester arguing that the Air Force failed to comply with FAR 15.306©, which requires the evaluation of ALL proposals on ALL factors prior to establishing a competitive range. So the Air Force is apparently getting away with evaluating Price first.

In any case, in the second phase of the Fact Pattern, it says the first top 10 proposals deemed "reasonable" would move on to be evaluated for technical acceptability.

Link to comment
Share on other sites

Vern Edwards:

Sorry, I don't have a link to the USAF guide. The Guide document was emailed to me by colleague as a MS Word attachment. However, I was able to find the USAF guide online by putting the search term phrase "Air Force Performance Price Trade-Off Guide" into Google. In the search results, it is the second item listed that says "Performance Price Tradeoff Guide" on Farsite. The "link" does not open up into an internet browser, it only opens up into MS Word. So there is no real "link."

I thought it was strange that the attachment sent to me does not even show the Air Force Logo on the first page, it does not even say "Air Force" at the top. However, when I examined the document further, I found citations to "AFFARS," The colleague who sent this to me contends that this is a USAF guide document that he received as part of his "training materials" when he went to an AF Base in Georgia recently for "PPT" Training.

Here is a cut and paste of what the USAF guide on PPT says starting on p. 8:

5.1.1 Approach #1 - Evaluate Technical, Rank by Price, Assess Performance

Step 1 - Evaluate all proposals for technical acceptability.

Step 2 - Evaluate price reasonableness of all technically “acceptable” or “reasonably susceptible of being made acceptable” proposals, then rank by total evaluated price.

Step 3 - Assess performance confidence for each offeror, or specified number of lowest priced technically acceptable offerors. SSA does an integrated best value assessment of the evaluated offerors.

Note: If a large number of proposals is anticipated, the contracting officer may plan to assess performance confidence on only a specified number of the lowest priced technically acceptable offerors (need at least two responsible offerors for adequate price competition). Make sure you address this in your description of the evaluation process in the solicitation.

5.1.2 Approach #2 - Rank by Price, Evaluate Specified Number for Technical, Assess Performance

Step 1 - Rank all proposals by price.

Step 2 - Evaluate a specified number of the lowest priced proposals for technical acceptability, if applicable, and price reasonableness.

Note: The number of proposals selected and surveyed will depend on the total number and quality of proposals received. For purposes of efficiency consider reducing the number of offers to the lowest priced proposals (usually the first five to seven). The number selected is the choice of the contracting officer.

Step 3 - Assess performance confidence for a specified number of the lowest priced technically acceptable offerors. SSA does an integrated best value assessment of the evaluated offerors.

5.1.3 Approach #3 – Evaluate Technical, Rank by Price, Assess Performance until Proposal rated “Substantial Confidence

Step 1 - Evaluate all proposals for technical acceptability.

Step 2 - Evaluate price reasonableness of all technically “acceptable” or “reasonably susceptible of being made acceptable” proposals, then rank by total evaluated price.

Step 3 - Evaluate lowest priced offeror’s past performance. If past performance is “Substantial Confidence”, evaluation is complete. If the lowest priced offeror’s past performance is not “Substantial Confidence” evaluate the next lowest priced offeror’s past performance. Continue evaluating next lowest priced offeror’s past performance until an offeror is rated “Substantial Confidence” or until all offerors are evaluated. At this point, the evaluation is complete and the SSA proceeds to the integrated best value assessment of the evaluated offerors.

Link to comment
Share on other sites

I’m saying I don’t understand how a price could be determined F&R for a proposal that doesn’t even meet the government’s requirements. What would that price be fair and reasonable for, something you don't even want? Also –

Problem with the protocol in your chain-of-events

You receive 50 proposals. First phase selection identifies 10 offerors with prices that are determined F&R. The other 40 proposals are discarded. During the second phase you discover that none of the 10 offerors actually understood the requirement and, thus, you discard those 10 from consideration. Now what? You can’t review any of the 40 other offerors without violating your own protocols.

Link to comment
Share on other sites

The protocols you've posted in Post #10 are not the same protocols you presented in your original question. Each of these protocols require technical review prior to discarding contractors.

5.1.1, step 1, “Evaluate all proposals for technical acceptability.” And the Note, “the contracting officer may plan to assess performance confidence on only a specified number of the lowest priced technically acceptable offerors

5.1.2, Note, “The number of proposals selected and surveyed will depend on the total number and quality of proposals received.”

5.1.3, step 1, “Evaluate all proposals for technical acceptability.”

Link to comment
Share on other sites

Guest Vern Edwards

This is from footnote 3 of the USAF guide:

If a large number of proposals is anticipated, the contracting officer may plan to assess performance confidence on only on a specified number of the lowest priced technically acceptable offerors (need at least two responsible offerors for adequate price competition). Make sure you address this in your description of the evaluation process in the solicitation.

What's worrying you is a scenario in which the agency receives a large number of proposals and (1) ranks them on price, from low to high, (2) eliminates all in excess of, say, 10 offerors, then (3) evaluates technical acceptability and past performance of the lowest 10. Right?

That is a form of phased evaluation, in which an agency applies evaluation factors in phases, eliminating some at the end of each phase. Phased evaluation is okay if done properly. However, even though the USAF guide has described that procedure since at least 2005, I would not state in an RFP that I will evaluate only the lowest-priced 5 or the lowest-priced 10. In my opinion, it would risk a protest of arbitrariness that might be hard to defend. I could not find a GAO protest decision in which an agency announced that it would use the Air Force procedure and evaluate only the lowest-priced 5, 10 or some other number of offerors.

What I would say instead is that if the agency receives a large number of proposals I will (1) rank the proposals on price and determine a cutoff point at which the price would be considered "too high" to be reasonable, in light of the competition, no matter what the outcome of the technical and past performance evaluation. By not stating a specific number I get more flexibility and avoid being accused of arbitrariness.

Link to comment
Share on other sites

jwomack:

Question on how can Price be determined Reasonable if we don't know if the proposal is Technically Acceptable:

I see what you're saying. Well, if F&R is to be established by Adequate Price Competition, then you are right, the Fact Pattern I posted won't work. If a proposal is Technically Unacceptable, it cannot be used to establish F&R for other proposals. However, if the method used is simply a comparison to the IGCE, then it should work.

In any case, your posts made me go back and reread the USAF guide on PPT at Section 5.0, Proposal Evaluation. For PPT Approach #2, described at 5.2 on p. 8 of the USAF Guide, you are right, it does state, "Step 1 - Rank all proposals by price." Hmm, it says "Rank" not "Evaluate." Perhaps the Air Force is simply creating a ranking based on the initially proposed total price of the proposals, and not actually evaluating them for Price Reasonableness and/or Realism? I just now noticed that it says in Step 2:

Step 2 - Evaluate a specified number of the lowest priced proposals for technical acceptability, if applicable, and price reasonableness.

Your question on what happens if all 10 proposals are technically unacceptable, then what:

Good point. I have gone back and revised the Fact Pattern to now say the following language:

The Solicitation states:

The Government will initially evaluate for technical acceptability the ten (10) lowest priced proposals. If the government receives less than 10 proposals, all proposal will be evaluate for Technical Acceptability. If the Government receives 10 or more proposal, the Government will rank the proposals by initially proposed total price, and then select a minimum of the 10 lowest priced proposals, and then evaluate only those proposals for Technical Acceptability. If any of the proposal are found to be Technically Unacceptable, the unacceptable proposal will be replaced with the next lowest priced proposal on a one-for-one basis, until there are a minimum of 10 technically acceptable proposals, or until all proposal have been evaluated for technical acceptability.

Link to comment
Share on other sites

jwomack:

You pointed out that 5.1.2 has a Note stating that the "number of proposals" selected would depend on "quality" of the proposals and other things. That's a good point. I don't see how an agency can know the "quality" of a proposal without first evaluating its technical acceptability. However, 5.2.1 expressly states to "Evaluate a specified number," not all, of the "lowest-priced proposals for technical acceptabilty." It is saying NOT to evaluate ALL proposals for Technical Acceptability.

Link to comment
Share on other sites

Vern Edwards:

I tried to do a mindwalk to see how that would play out. This is about your solution of setting a cut-off that is NOT an exact number of proposals. Well, what if got the agency received 25 proposals, the IGCE is a million dollars, and all the proposals were greater than the IGCE, but each one is only $1 more than the next lowest-priced one? Like this:

#1-$1.000,001

#2-$1,000,002

#3-$1,000,003

and so forth until

#25-$1,000,025

So the price premium for the highest-priced proposal, #25, versus that of the lowest priced proposal, #1, is not even a difference of 1 percent. And even price #25 is very close to the IGCE (how could the agency argue that it is not?).

So if the agency tried to say that, for example, that proposals #11 through #25 are “unreasonable,” well, if I were one of those vendors, I would protest on the grounds of misevaluation of my price proposal.

Link to comment
Share on other sites



Another colleague brought to my attention the Champion Service Corp. GAO decision, B-284116, Feb. 22, 2000. In this decision, GAO simply mentions in the facts that the Air Force did a ranking by price and then evaluated only the top 5 lowest-priced proposals for technical acceptability:



http://www.gao.gov/products/403297



"The Air Force received nine offers by the July 20, 1999 closing date. Proposals were ranked by price and, as called for by the solicitation, the technical proposals of the five low-priced offerors, including Champion's and SelRico's, were evaluated."



So, although the Champion case is not about a protest of the PPT evaluation methodology involving a cut-off at a "specified number," it does mention in passing that something like that appears to have happened in this case. The case does not give enough detail to truly conclude that. I don't have the solicitation to actually see what the Air Force did. But it sounds like that's what the Air Force did . . .

Link to comment
Share on other sites

I must say, I find it really strange that the USAF would have a guide on a source selection method and not have it posted for the public on an official USAF website. If it is posted, I could not find it. I also find it strange that the guide does not have the Air Force name and logo on the front. It's bizarre. Is this even a real USAF guide? If it is, is it still a current USAF guide? I contacted the only USAF civilian procurement professional I know, and he said, he has never done PPT, and he has never heard of this USAF guide.

Link to comment
Share on other sites

deleted - there are 3 alternate approaches in the "Guidance."

Link to comment
Share on other sites

Guest Vern Edwards

govt2310:

I don't know what to think of you and your fact patterns. Your Post #18 is ridiculous. What do you think the chances are of that occurring? Why post something like that? Why even think about something like that? If it did happen, then you would have to adjust. Are you capable of adjusting?

As for your last post, do some research. What if there were no current guide? What then?

Link to comment
Share on other sites

Vern Edwards:

Adjustment Technique:

Well, I am not the inventor of the Adjustment Technique. The Air Force used it in a LPTA procurement for "various advisory and assistance support services" for its Air Combat Command bases back in 2011.

I know, LPTA and PPT are not the same. LPTA is LPTA. PPT is obviously Tradeoff. That is what the "T" stands for. So the Adjustment Technique works in a LPTA procurement, but it does not necessarily work in a PPT.

But because your point seems to be how onerous it would be continuously "adjust," I just wanted to point out, it has been done in LPTA, and it was likely significantly onerous as well, but it was done. By the Air Force.

Here is the Luke & Associates, Inc. GAO decision, B-405348, Oct. 13, 2011, which concerned a LPTA Solicitation involving the Adjustment Technique:

http://www.gao.gov/assets/590/586044.pdf

And here is the Solicitation itself on the FBO Archive. Please see Section M on p. 74 of 81 of the solicitation, at M-3, Evaluation Factors and Methodology, where it describes how the top ten or top 16 lowest priced proposals would be evaluated, and the Adjustment Technique:

https://www.fbo.gov/?s=opportunity&mode=form&id=61a38efb6986fa0a5ee6817f95f0b355&tab=core&_cview=1

Link to comment
Share on other sites

Note, although the Solicitation for the Luke & Associates GAO decision says at M.1.1 on p. 74 that it was a "PPT," in fact, it was a LPTA. If read all of Section M, you will see, there was no Tradeoff at all. And in the GAO decision, it expressly states that it was a LPTA.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...