Jump to content

Probability Problem #2


Recommended Posts

Assume you are evaluating cost realism for the award of a competitive cost-reimbursement contract using the tradeoff process. You receive two offers--one from Edwards and one from Hoffman. You've determined that the most probable cost for both Edwards and Hoffman is $100 million. However, the probability of the cost being $100 million dollars differs between the two offerors.

The probability of Edwards's cost being $100 million is 80%. There's a 20% chance that the cost will be $120 million.

The probability of Hoffman's cost being $100 million is 60%. There's a 40% chance that the cost will be $120 million.

For purposes of conducting tradeoffs, what values do you use for Edwards's proposal and Hoffman's proposal?

Link to comment
Share on other sites

The expected cost of Edwards’ offer is $104M (($100M * 0.8) + ($120M * 0.2)).

The expected cost of Hoffman’s offer is $108M (($100M * 0.6) + ($120M * 0.4)).

However, whether one uses expected cost calculations like the two above or the most probable cost (which for both offers is $100M) depends on the language in one’s RFP.

Link to comment
Share on other sites

I’d hire Edwards and select the method that would justify it.  

Link to comment
Share on other sites

2 hours ago, bob7947 said:

Joel:

It sounds like a biased evaluation to me but I do not have concrete evidence to win a protest at GAO.

I am sure that Hoffman won’t protest 😄

Link to comment
Share on other sites

12 hours ago, Matthew Fleharty said:

I don’t think that’s helpful in deciding between the two - if the RFP merely stated that I think a Contracting Officer could make an argument for either approach estimating the “probable cost.” Neither seems entirely unreasonable to me.

So you think using the expected cost would be compliant with FAR 15.404-1(d)?

Link to comment
Share on other sites

I'd use $100m for each.  

FAR 15.404-1(d) includes:

(2) Cost realism analyses shall be performed on cost-reimbursement contracts to determine the probable cost of performance for each offeror.

(i) The probable cost may differ from the proposed cost and should reflect the Government’s best estimate of the cost of any contract that is most likely to result from the offeror’s proposal. The probable cost shall be used for purposes of evaluation to determine the best value.

In each offer, the probable cost is $100m, since $100m is the most likely outcome from either offer.    

Link to comment
Share on other sites

The UK uses probability distributions and Monte Carlo simulations to quantify "cost risk"--which can then be distributed between government and contractor through negotiations. I don't believe the US government is set up to do that. I agree with jayandstacey.

Link to comment
Share on other sites

I may be wrong, but I don’t think FAR 15.404-1(d) is so clear that the cost with the highest probability is necessarily the probable cost.  @jayandstacey honed in on the words most likely, which is an argument in favor of that point; however, that argument ignores the words best estimate.  An estimate that outright ignores, a 20%, 40%, etc. of cases is, arguably, an incomplete estimate regardless of how much probability the one most likely event captures.

Let’s imagine we had a third contractor, we’ll call him Bob.  The probability of Bob’s cost being $100M is 51%.  There’s a 49% chance that the cost will be $200M.

While $100M is technically more likely than $200M, does $100M represent the best estimate of the potential cost of a contract with Bob?

I’m traveling so I’m without my usual resources - if someone could point to particular case law I’d be curious to know how this matter has been adjudicated.

Link to comment
Share on other sites

Guest Vern Edwards

Here is the problem as posted:

On 7/5/2018 at 2:12 PM, Don Mansfield said:

Assume you are evaluating cost realism for the award of a competitive cost-reimbursement contract using the tradeoff process. You receive two offers--one from Edwards and one from Hoffman. You've determined that the most probable cost for both Edwards and Hoffman is $100 million. However, the probability of the cost being $100 million dollars differs between the two offerors.

The probability of Edwards's cost being $100 million is 80%. There's a 20% chance that the cost will be $120 million.

The probability of Hoffman's cost being $100 million is 60%. There's a 40% chance that the cost will be $120 million.

For purposes of conducting tradeoffs, what values do you use for Edwards's proposal and Hoffman's proposal?

Don asks for decisions based on the values provided. Matthew provided both his decisions and his calculations. Don then asked:

22 hours ago, Don Mansfield said:

So you think using the expected cost would be compliant with FAR 15.404-1(d)?

We cannot judge the legal merits of anyone's answer, whichever values or combinations of values are chosen, because we have not been told how the given values were determined.

Here are three quotes from a GAO decision concerning a cost realism protest. The quotes are based on more or less standard boilerplate decisional verbiage GAO uses when deciding cost realism protests. The quotes are from Science Applications, International Corporation, B-290971, October 16, 2002:

Quote

Where an agency evaluates proposals for the award of a cost-reimbursement contract, an offeror's proposed estimated cost of contract performance should not be considered controlling, since the offeror's estimated costs may not provide valid indications of the final actual costs that the government is required, within certain limits, to pay. Advanced Communication Sys., Inc., B-283650 et al., Dec. 16, 1999, 2000 CPD ¶ 3 at 5. Accordingly, a cost realism analysis must be performed when a cost-reimbursement contract is contemplated in order to determine the probable cost of performance for each offeror. Federal Acquisition Regulation (FAR) § 15.404-1(d)(2). A cost realism analysis is the process of independently reviewing and evaluating elements of each offeror's proposed cost estimate to determine whether the proposed cost elements are realistic for the work to be performed, reflect a clear understanding of the requirements, and are consistent with the methods of performance and materials described in the offeror's technical proposal. FAR § 15.404-1(d)(1). Because the agency is in the best position to make this cost realism determination, our review is limited to determining whether its cost evaluation was reasonably based and not arbitrary. NV Servs., B-284119.2, Feb. 25, 2000, 2000 CPD ¶ 64 at 7.

 
 
Quote

An agency need not achieve scientific certainty in analyzing costs proposed by offerors, or conduct an in-depth cost analysis. Instead, any methodology used by an agency must only be reasonably adequate and provide some measure of confidence that the rates proposed are reasonable and realistic in view of other cost information reasonably available to the agency from its own and outside sourcesRadian, Inc., B-256313.2, B-256313.4, June 27, 1994, 94-2 CPD ¶ 104 at 7; see also E.L. Hamm & Assocs., Inc., B-280766.5, Dec. 29, 1999, 2000 CPD ¶ 13 at 7. A reasonably derived estimate of direct, unburdened labor rates for comparable labor categories can provide an objective standard against which the realism of proposals can be measured. United Int'l Eng'g, et al., B-245448.3 et al., Jan. 29, 1992, 92-1 CPD ¶ 122 at 11. However, an agency may not mechanically apply that estimate to determine evaluated costs. In instances where an estimate has limited applicability to a particular company, an absolute reliance on estimates could have the effect of arbitrarily and unfairly penalizing the firm and depriving the government of the benefit available from such a firm. Accordingly, in order to undertake a proper cost realism evaluation, the agency must independently analyze the realism of an offeror's proposed costs based upon its particular approach, personnel, and other circumstances. Id. Our review of the record shows that, under the circumstances of this procurement, the agency's use of attachment 3 as an objective standard against which to measure the realism of offerors' direct labor rates was unobjectionable.

 
Emphasis added.
 
Quote

SAIC argues that the letters cited by the Navy do not reflect actual historical rates, but refer to then-future periods of performance and, in one case, “maximum” labor rate ranges, and that the sampling data used by the Navy is misleading. However, the RFP put the burden on the offeror to present supporting explanation for its proposed escalation factor, and SAIC's cost proposal provided none of the explanation it now provides. The fact that the rate was included in its recommended forward pricing rates, and that the DCAA took no exception, is not dispositive. Contracting agencies are not bound by such recommendations since they are only advisory. Purvis Sys. Inc., B-245761, B-245761.2, Jan. 31, 1992, 92-1 CPD ¶ 132 at 10. Although SAIC could well be correct in its prediction about future cost escalation, it is the Navy, not SAIC, that must bear the risk if actual rates increase during performance. AmerInd, Inc., supra at 8; see also Infotec Dev., Inc., B-258198 et al., Dec. 27, 1994, 95-1 CPD ¶ 52 at 5. With this concern in mind, and given the information in its possession, the RFP's explicit instructions, and SAIC's incomplete supporting explanation, we find reasonable the Navy's decision to apply the 3.5 percent escalation factor to SAIC's proposed labor costs.

 
Matthew's answers are based on straightforward mathematical calculations, which is what I assumed that Don was asking for. Don is fascinated by probability, and I assumed he was testing our knowledge of probability calculation procedures. jayandstacey's answer appears to be based on regulatory interpretation, as does here_2_help's concurrence.
 
But if the issues are legal, i.e., (1) whether using expected cost, or jayandstacey's $100M, as the most probable cost and as the basis for adjusting offerors' estimated costs complies with FAR, and (2) whether the analysis would survive a protest, then it is impossible to offer an informed opinion. The answers to those questions would depend on the reasonableness of the method(s) and data used, which we do not know. Cost realism analysis is an art, not a science.
 
I do not agree with jayandstacey's narrow interpretation of FAR 15.404-1(d). The regulation does not use the phrase "most likely" or most probable. It says only "probable." I have seen nothing in any cost realism protest decision indicating that the probable cost cannot be determined on the basis of combined probabilities.
 
Cost realism analysis is a legal requirement. As a practical matter, determinations of probable cost (or most probable or most likely cost) have largely been exercises in futility. We should not be evaluating cost in competitions for cost-reimbursement contracts, as required by CICA and FAR. It's an unsound business practice and a reflection of the ignorance of politicians. See "Estimated Cost Competition: An Idea Whose Time Has Come... And Gone," The Nash & Cibinic Report, May 2018.
Link to comment
Share on other sites

To add to Vern's point, Don's problem has omitted significant information regarding the probability distribution of the contractors' costs. Don has provided a probability estimate that the actual value may be $20 million higher than the most probable cost (which I assume is the mean of the distribution), but he has omitted a probability estimate associated with an actual value that is $20 million lower than the most probable cost. We need the distribution around the mean, and the standard deviation, if we are going to play probability games.

Link to comment
Share on other sites

6 hours ago, Vern Edwards said:

The answers to those questions would depend on the reasonableness of the method(s) and data used, which we do not know.

Assume the method was reasonable. 

10 minutes ago, here_2_help said:

To add to Vern's point, Don's problem has omitted significant information regarding the probability distribution of the contractors' costs. Don has provided a probability estimate that the actual value may be $20 million higher than the most probable cost (which I assume is the mean of the distribution), but he has omitted a probability estimate associated with an actual value that is $20 million lower than the most probable cost. We need the distribution around the mean, and the standard deviation, if we are going to play probability games.

Not in my example. There's only two possible cost outcomes for each offeror.

Please don't fight the hypothetical. 

Link to comment
Share on other sites

3 hours ago, Don Mansfield said:

Assume the method was reasonable. 

Well the I think that answers it then - I didn’t have resources handy to look up the reference, but Vern provides the reasonable standard used by the GAO to evaluate cost realism methodologies.  So the question is whether a method is unreasonable given what the solicitation states - in this case, I don’t think the expected cost estimate methodology I provided in my first post is unreasonable.

Are there any other methods you or others would use to estimate the probable cost in the given hypothetical?

Link to comment
Share on other sites

16 hours ago, Vern Edwards said:
 
I do not agree with jayandstacey's narrow interpretation of FAR 15.404-1(d). The regulation does not use the phrase "most likely" or most probable. It says only "probable." I have seen nothing in any cost realism protest decision indicating that the probable cost cannot be determined on the basis of combined probabilities.

I’m confused by this: the FAR citation I provided does say “most likely”, doesn’t it?  Am I quoting an older FAR version?

I agree with everything else you’ve said, including the slippery slope argument.  I almost posted a longer version of my earlier post, which would have continued on...I would have taken the risk of the higher cost basis and shifted this into my risk evaluation.

This is not to say that the Hoffman cost isn’t reasonable, or that any more math has to be done.  Just that the Hoffman offer has a higher chance for a higher cost...and thus gets a “ding” (however that was defined in the RFP) in the risk evaluation.  But in the cost evaluation section, they are both evaluated at $100m.

I think FAR 15.404-1(d)(2)(i) instructs this.

I also happen to believe this is a reasonable approach.  While I understand probabilities, they also present us here with impossible numbers (for evaluation purposes) that is better addressed in a separate risk evaluation.

To illustrate: let’s say we are evaluating Edwards’ and Hoffman’s ability to drive from their respective houses to mine.  Both can do so in 100 minutes and both normally would do so.  But both have a drawbridge that opens occasionally for 20 minutes.  Thing is, Hoffman’s drawbridge opens twice as often as Edwards’.

As an exercise I could get to 104 and 108 minutes as their time to get to me.  But of course neither of these times would EVER be the actual times.  I think it is more appropriate to say “both are likely to get here in 100 minutes.  Both have a risk of a 20 minute delay.  Hoffman’s risk of delay happens to be twice Edwards’.”  Doesn’t that accurately describe the cost situation?

I didn’t mention this as Don didn’t mention anything about how risk is considered in the solicitation.  Nor am I a contracting officer...so I don’t have working familiarity with this.  But if I were, I’d be reluctant to introduce manufactured numbers like $104m when I could factor the increased risk elsewhere.  You mentioned that you’ve not seen a protest addressing combined probabilities used in cost evaluations...but why even go there (assuming there’s room in the evaluation factors to plug in general risk factors?)  If this is a dufus question I won’t expect an answer. :)

As a side note, years ago I almost did protest on a fairly similar instance; a talk with the CO shifted the thinking.

 

 

 

Link to comment
Share on other sites

4 hours ago, Matthew Fleharty said:

One more thought regarding the concept of probably cost and the words most likely:

What if the probabilities for a proposal were 50% and 50%? Since neither is technically most likely what then under @jayandstacey‘s standard for determining probable cost?

I think we are assuming that these are %s that the CO has independently calculated.  To me, it points to the idea that more offeror info/discussions/research should be conducted to break the tie and determine the most likely cost.  

I could be wrong about this, but I feel like these are simple %s applied to complex scenarios.  To look to the future and say “there is exactly a 50/50 chance” between two outcomes in such a scenario seems either virtually impossible OR an admission that very little is known about the scenario.  

Are there real world contracting scenarios where a proposed cost may be A or B, and the chance of A or B is known to be an even 50%?  I sense the writers of the FAR clause didn’t think such things existed. 

(Edit- using my own drawbridge example from above, then let’s assume a third offeror “Smith” has a drawbridge that’s open 50% of the time, thus creating the scenario you ask about here...in that case, if I’m the CO, I think I realize that the FAR doesn’t cover everything, it can’t.  And I think I make the call that says “the tie goes to the runner” and I allow the Smith offer to be evaluated at the lower cost (with a higher ‘ding’ in the risk evaluation).  I document the file as such, and if any other offeror also has a true 50/50 cost, I treat them the same.  Having said that; I’d at least try to see if in fact the drawbridge is open EXACTLY 50% of the time, given the aging infrastructure and all that...)

(a little more editing:  I believe one of basic tenants of the FAR is that the Government is choosing.  Choosing awardees, requirements, evaluation factors, etc.  It is the job of the CO to choose in many situations.   Often there just isn’t room for a “tie”; the Government can’t buy the same tank from two different companies.  It has to choose one.  I believe this clause is an instance of that, suggesting that the CO reasonably determine the most likely cost and use that as the basis for evaluation.  It doesn’t leave room for a 50/50 because...it’s in the business of choosing.  It instructs the CO to find the most likely cost; 50/50 is not following the instruction.

Have I gone off the rails?  It’s late.)

Link to comment
Share on other sites

25 minutes ago, jayandstacey said:

I think we are assuming that these are %s that the CO has independently calculated.  To me, it points to the idea that more offeror info/discussions/research should be conducted to break the tie and determine the most likely cost.  

I could be wrong about this, but I feel like these are simple %s applied to complex scenarios.  To look to the future and say “there is exactly a 50/50 chance” between two outcomes in such a scenario seems either virtually impossible OR an admission that very little is known about the scenario.  

Are there real world contracting scenarios where a proposed cost may be A or B, and the chance of A or B is known to be an even 50%?  I sense the writers of the FAR clause didn’t think such things existed. 

(Edit- using my own drawbridge example from above, then let’s assume a third offeror “Smith” has a drawbridge that’s open 50% of the time, thus creating the scenario you ask about here...in that case, if I’m the CO, I think I realize that the FAR doesn’t cover everything, it can’t.  And I think I make the call that says “the tie goes to the runner” and I allow the Smith offer to be evaluated at the lower cost (with a higher ‘ding’ in the risk evaluation).  I document the file as such, and if any other offeror also has a true 50/50 cost, I treat them the same.  Having said that; I’d at least try to see if in fact the drawbridge is open EXACTLY 50% of the time, given the aging infrastructure and all that...)

(a little more editing:  I believe one of basic tenants of the FAR is that the Government is choosing.  Choosing awardees, requirements, evaluation factors, etc.  It is the job of the CO to choose in many situations.   Often there just isn’t room for a “tie”; the Government can’t buy the same tank from two different companies.  It has to choose one.  I believe this clause is an instance of that, suggesting that the CO reasonably determine the most likely cost and use that as the basis for evaluation.  It doesn’t leave room for a 50/50 because...it’s in the business of choosing.  It instructs the CO to find the most likely cost; 50/50 is not following the instruction.

Have I gone off the rails?  It’s late.)

Don’t fight the hypo - if based on extensive analysis a CO thinks two outcomes are equally likely, what then?

Link to comment
Share on other sites

Guest Vern Edwards
20 hours ago, Don Mansfield said:

Assume the method was reasonable. 

@Don Mansfield Okay. Let's look again at the scenario:

On 7/5/2018 at 2:12 PM, Don Mansfield said:

You've determined that the most probable cost for both Edwards and Hoffman is $100 million. However, the probability of the cost being $100 million dollars differs between the two offerors.

The probability of Edwards's cost being $100 million is 80%. There's a 20% chance that the cost will be $120 million.

The probability of Hoffman's cost being $100 million is 60%. There's a 40% chance that the cost will be $120 million.

Emphasis added.

If you document the file to say that you've determined that the MPC is $100M, as you said in the quote above, then that's the value you should use. But if, instead, you document the file to say:

A. Our best estimate of the cost of any contract that is most likely to result from [Edwards'] proposal falls within a range of $100M (80% chance) to $120M (40% chance), and has an expected value of $104M.

and

B. Our best estimate of the cost of any contract that is most likely to result from [Hoffman's] proposal falls within a range of $100M (60% chance) to $120M (40% chance), and has an expected value of $108M.

then I say you could use the values $104M and $108M when making tradeoffs. I believe those would be better numbers than $100M. Of course, as you have set up the problem, I'm not disinterested.

@jayandstacey

The italicized phrases in the statements A and B above are based on the sentence in FAR 15.404-1(d)(2)(i) that says:

Quote

The probable cost may differ from the proposed cost and should reflect the Government’s best estimate of the cost of any contract that is most likely to result from the offeror’s proposal.

Emphases added.

FAR 15.404-1(d) does not use the phrase "most likely cost." As I'm interpreting the above sentence, the antecedent of the pronoun "that" is the phrase "any contract", not the phrase "the cost".

If I'm interpreting the quoted sentence properly, as taught by Sister Mary Elizabeth, the adjectival phrase "that is most likely to result" does not apply to the phrase the cost, but to the phrase any contract, which might not be what the FAR councils meant, but which is what they said. In other words, I interpret the sentence to say any contract that is most likely to result, not any cost that is most likely to result.

😈

Besides, it is a contract that results from a proposal, not a cost. A cost results from a contract. If the FAR councils meant:

The probable cost may differ from the proposed cost and should reflect the Government's best estimate of the cost that is most likely to result from any contract that is awarded to the offeror

then that's what they should have written. 😎

@All

I think what Don was trying to get at was whether FAR 15.404-1(d) requires a point estimate of probable cost or permits a range estimate. Do you have to state a particular number or can you say that the probable cost is between X1 and X2 with a specified probability distribution? If it permits such a range estimate, then how do you apply it during tradeoff analysis? 

What Don's scenario and our responses should teach us is how treacherous language can be when describing probabilities. What, exactly, do you mean when you say that something "is likely" to happen or "will probably" happen? What do you mean by "most likely" or "most probable"? In other words, do we know what we're talking about when we're talking about cost realism?

Finally, what worries us most? Meeting the GAO's legal standards of evaluation or meeting sound business standards? I say that GAO's legal standards are not particularly high.

Link to comment
Share on other sites

On 7/8/2018 at 8:46 AM, Vern Edwards said:

FAR 15.404-1(d) does not use the phrase "most likely cost." As I'm interpreting the above sentence, the antecedent of the pronoun "that" is the phrase "any contract", not the phrase "the cost".

If I'm interpreting the quoted sentence properly, as taught by Sister Mary Elizabeth, the adjectival phrase "that is most likely to result" does not apply to the phrase the cost, but to the phrase any contract, which might not be what the FAR councils meant, but which is what they said. In other words, I interpret the sentence to say any contract that is most likely to result, not any cost that is most likely to result.

Got it, agree. 

I recall coming out of the FAR Bootcamp thinking "Who knew that Sister Boniface's sentence diagraming would be the most useful skill in this course?"  (BTW, great course, had fun and learned a lot.  Get a nun to teach it and you'd really have something! ;)  )

The result, as you lay it out in the top of the same posting, also makes business sense.  The gotcha is to get away from the notion that a cost evaluation must result in a single, fixed number.   There's a strong temptation to distill a complex offer into a single evaluation data point, and that isn't always the right approach.   

Link to comment
Share on other sites

Guest Vern Edwards
3 hours ago, jayandstacey said:

Get a nun to teach it and you'd really have something! ;) 

Most of the students would have nervous breakdowns.

Link to comment
Share on other sites

On 7/8/2018 at 5:46 AM, Vern Edwards said:

I think what Don was trying to get at was whether FAR 15.404-1(d) requires a point estimate of probable cost or permits a range estimate. Do you have to state a particular number or can you say that the probable cost is between X1 and X2 with a specified probability distribution? If it permits such a range estimate, then how do you apply it during tradeoff analysis?

Not quite. I wanted to see how people would react to information about possible, but less probable, costs. jayandstacey reacted the way I think most contracting people would--they ignored it (I'm not picking on you, jayandstacey). I think standard practice when it comes to cost realism is to determine a most probable cost point and use that when making tradeoff decisions. There's no consideration given to how probable the most probable cost is, and what are the probabilities of other costs. 

An alternative to using the most probable cost is to use the expected value. This is what Matthew calculated. This takes into account every probable cost and its probability of occurring. Given a probability distribution function, we can calculate the expected value. In my opinion, that's a better number to use when making decisions than most probable cost. 

There's a good illustration of two different probability distributions in Figure 2 of this article: https://www.dau.mil/library/arj/_layouts/15/WopiFrame.aspx?sourcedoc=/library/arj/ARJ/arj62/Dorey_ARJ62.pdf&action=default

Using most probable cost, we would favor the red cost estimate. However, if we factor in the probability of the all potential costs, both the blue and red cost estimates are equal. 

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...