joel hoffman

In simple terms, adding a 10% contingency to each high risk is saying that there is a 100% probability that every risk will occur.

With risk management and using a Monte Carlo analysis, the analysis can be used to predict probabilities of delays and costs due to one or multiple risks. It’s not a linear cumulative value. 

Since my experience was almost 20 years ago, I don’t remember all the details but one can look it up and there are knowledgeable practitioners to help you.

Here is some sample info on it.

https://www.proprofsproject.com/blog/monte-carlo-analysis-in-project-management/#:~:text=The Monte Carlo Analysis is,project cost and project timeline.

Disclaimer: I’m certainly not an expert in risk management and RM estimating, if that is what you are asking about. 

We don’t know what type of contract or delivery order pricing is involved, e.g., cost reimbursement, FFP, etc.

However, your flat rate 10% add on for “high risk” suppliers appears to be a simple contingency.

I was on the government side of the early phases of a highly technical cost reimbursement systems IDIQ contract with separate phases for design, construction, process systemization, pilot operations, full scale operations, closure, dismantling…  It was a Chemical Weapons Demilitarization plant for assembled chemical weapons as well as various bulk chemical agents using a process that had only been bench tested.

 The lead firm of the joint venture Prime was one of the two largest US engineering and construction contractors. During the negotiations for design-construct phases, the JV and us identified numerous, various cost and schedule risks, then the SC used a Monte Carlo risk analysis to determine and estimate risk to the schedule and cost.

I left the project for other duties before seeing the results, though. DoD purposely slowed down the last two plants in the Chemical Weapons Demilitarization Program by at least 15 years! 

17 hours ago, formerfed said:

This is also getting way off topic from the original question

I think that this discussion directly pertains to @Atlas STS stated concern with having to provide “certified cost or pricing data” that is accurate, current and complete (in particular, “terror” in relation to “complete”) in of support its proposed Price.

And it shows how judgmental data  (which isn’t necessarily certified cost or pricing data) can be used when historical (I.e., actual/factual) cost or pricing data is used but needs to be adjusted for the current acquisition to fit the current conditions or circumstances.

I realize that @Atlas STS is upset that the KO won’t accept a commercial item exception to single offeror cost or pricing data since Atlas’ firm is a “Nontraditional Defense Contractor” (as are most small business DoD contractors).

21 hours ago, Retreadfed said:

There are two thoughts here. First, you seem to be saying that the cost or pricing data has to support the price proposed.  I disagree with that premise.  The price proposed does not have to be based on the cost or pricing data disclosed to the government, although contractors frequently do so.  All the contractor has to do is disclose the data to the government in a manner that reveals the significance of the data to the particular procurement.  That leads to the second thought concerning the format in which cost or pricing data is submitted.  The solicitation provisions in the FAR and DFARS both refer to FAR Table 15-2 as the default format for submission of cost or pricing data.  As an alternative, the contracting officer can permit another format to be used.  As long as the contractor submits cost or pricing data in the format required by the solicitation, I think the contractor is good to go and the contracting officer should have no complaints because the contractor has done precisely what the contracting officer asked it to do.

Often a price for a non-commercial product or service may be based upon both judgmental cost data (e.g., number of mhrs when not known or firmly established, risk factors, etc.) and factual cost or pricing data (e.g., material quotes, manufacturing means and methods, labor rates, subcontracts, etc.).

EDIT: [In other words, cost or pricing data aren’t necessarily historical data. It can also be the non-judgmental cost components of prices developed for the current solicitation.]

Sometimes there are historical cost or pricing data that isn’t current and costs will be adjusted (evolve) for the instant acquisition. The basis for adjustment may be judgmental and/or factual. But it is a requirement to explain it in the proposal per the instructions for the default Table 15-2. ** in FAR 15.408.

Edit: [And yes, @Fara Fasat. The FFP contract and any line item price obviously includes both costs and profit. I’m only addressing the basis for proposed cost within the proposed price(s) here. Profit is separately identified. If the supporting cost breakdown is only 50% of the proposed price, then that will be or should be apparent -  not have to be deduced by the buyer.

For single source, negotiated DoD acquisitions of this magnitude, the cost principles in Part 31 apply when cost analysis is used in addition to price analysis.] 

20 hours ago, formerfed said:

Edit:  I think the difference here is partially based on my not explaining my thought adequately.  Suppose a proposal is based on 30 hours of labor but the cost or pricing data using historical experience uses 20 hours.  As a contracting officer I expect the offeror to explain the assumptions used to arrive at 30 hours.

This is also getting way off topic from the original question.

It’s more than an expectation - it is a specific requirement in the Instructions for default Table 15-2. **

**From the Instructions for Table 15-2 

“C. As part of the specific information required, you must submit, with your proposal-

(1) Certified cost or pricing data (as defined at FAR 2.101). You must clearly identify on your cover sheet that certified cost or pricing data are included as part of the proposal.

(2) Information reasonably required to explain your estimating process, including-

(i) The judgmental factors applied and the mathematical or other methods used in the estimate, including those used in projecting from known data; and

(ii) The nature and amount of any contingencies included in the proposed price.

D. You must show the relationship between line item prices and the total contract price.You must attach cost-element breakdowns for each proposed line item, using the appropriate format prescribed in the "Formats for Submission of Line Item Summaries" section of this table. You must furnish supporting breakdowns for each cost element, consistent with your cost accounting system.”

We would also instruct the offeror or contractor (mods and claims) to designate judgmental data as (J) and actual or otherwise factual data as (F), for instance.

@USGCO, I like the question concerning the customer’s reasoning for requesting a “spend plan” for (an assumed) fully funded FFP (supply?) contract.

1. Can you explain the basis/reasons for customer’s concerns or needs?

2. Also, why was this type of financing chosen for the acquisition?

3. Can we assume that regular progress reporting, independent of progress payments, is a contract requirement?

4. One more question. Was this a competitively awarded contract or was it a sole source negotiated contract?

Thanks, in advance for clarifying…

P.S., if your question has been answered and you don’t want to publicize the details, that’s ok but please confirm if it has been answered. I noticed that you last signed in last Friday morning when you posed the question.

On 9/22/2023 at 8:08 AM, USGCO said:

I am a CO for the USG. I have a big contractor that does not like to submit invoices on our FFP contracts until delivery. My customer would like to have regular invoicing, and has requested we hold the contractor to a "spend plan". My contract includes the Progress Payments clause, but my understanding is that we can't force the contractor to submit progress payments if they don't want/need to. Someone please tell me I'm wrong. Can I add a payment schedule to the contract and enforce regular billing on my contractor?

No. Not under the Progress Payments clause at 52.232-16, which begins with: “The Government will make progress payments to the Contractor when requested as work progresses, but not more frequently than monthly, in amounts of $2,500 or more approved by the Contracting Officer, under the following conditions…”

I don’t think that it would be within the scope of the Changes clause to unilaterally change the payment terms of the contract from voluntarily requested progress payments, to mandatory progress payments. It will likely increase the contractor’s costs to administer and prepare regular invoices. 

The base fee is generally fixed, unless the there is a change to the scope of work or other circumstance providing for an adjustment (not due to cost growth/overruns/etc.).

Let me finally (hopefully) add that the instructions for providing cost or pricing data require contractor to summarize all costs and separately show profit.

It’s not the government’s responsibility to interpret the differences between raw cost or pricing data and/or data other than cost or pricing data and the proposed contract or line item costs which may evolve from the raw data, to determine the amount of profit proposed. 

On 9/15/2023 at 11:51 AM, Retreadfed said:

Out of curiosity, if the contractor merely disclosed the lower quote and did not provide any explanation as to why that subcontractor was not going to be used, would defective pricing exist?

Are you referring to post award review or discovery or to pre-award proposal review and negotiation (proposal adequacy)?

I will address proposal adequacy here. 

The proposed DoD contractor in the situation of only one offer received must provide breakdowns of all costs (EDIT: not addressing commercial products or services or other exceptions to certified cost or pricing here) in its proposal in enough detail for the KO to determine price reasonableness, per the numerous FAR and DFARS provisions and 15.4/215.4 references. This is not restricted to cost or pricing data. It also includes appropriate data other than cost or pricing data, if necessary.

FAR 15.408, Table 15-2, Section I Paragraph C(2)(i) requires that the proposal disclose the judgmental factors applied and the mathematical or other methods used in the estimate.

EDIT: BTW, this paragraph includes disclosing those methods used in projecting from known data - such as disclosing differences between known data and the current situation. Don’t merely disclose historical cost or pricing data, then change the pricing without disclosing how the current pricing varies or evolved from the historical data. This may include judgmental and/or factual differences. The “contractor’s  duty in regard to cost or pricing data [does NOT end] when it disclosed all the data it had”, then left it up to the government how it interprets the disclosed cost or pricing data. 

EDIT: The prime must perform and document price analyses for all proposed subcontracts per the Table 15-2 instructions at Ii A., in accordance with FAR 15.408 and the applicable Provision(s) for submission of current cost or pricing data. 

For those subcontracts based upon adequate price competition, this would include information concerning the competition, comparison of prices and the judgmental basis and rationale for selection of the source and price. And yes, you include the cost of the sub you intend to use in the proposal breakdown.

Thus, the price analysis obviously should address comparisons with the other prices, including the lower priced proposal.

Also - the cost principles in FAR 31 are applicable to negotiated FFP contracts when cost analysis is applicable, per 31.103. Subcontracts are “costs” to the prime.

Per FAR 31.201-3 (Determining Reasonableness): “A [prime contractor] Cost is reasonable, if in its nature and amount, it does not exceed that which would be incurred by a prudent person in the conduct of competitive business…No presumption of of reasonableness shall be attached to the incurrence of costs by a contractor.  If an initial review of the facts results in a challenge of a specific cost by the [KO or COR], the burden of proof shall be upon the contractor to establish that such cost is reasonable.”

In my opinion, reasonableness  of subcontractor competitive selection should be based upon the standard as though there is competition at the prime contract level.

To assert adequate competition as the basis for subcontract pricing, the contractor should justify why a higher priced subcontractor proposal is necessary.

Assuming that the prime actually used the higher priced subcontract without explaining it in its price analysis, I don’t know if there would be “defective pricing”. But it would appear to be an inadequate proposal (withholding required information) for the government to determine the reasonableness of the proposed price.

EDIT:

If the KO becomes aware of the lower offer without price analysis or justification before award, the KO should require the prime proposer to justify why it selected the higher price subcontract proposal, before awarding the contract.

If the prime didn’t reveal that there was/were lower priced offer or offers before award, I think that there could be a truthfulness issue. The government might have been able to negotiate a lower price…

If the prime awards to the lower priced subcontractor after negotiations concluded and effective date of the certification of current cost or pricing, but still included the higher price in its final proposal, there may be defective pricing.

I saw that situation occur in one of my Districts.

EDIT: I didn’t discuss the requirement for the prime to also conduct cost analysis of subcontract proposals, when applicicable. Here, I am responding to a specific scenario where a subcontract price is based upon adequate competition.

4 hours ago, formerfed said:

Right.  Besides any experienced contracting officer and PM should ask for a comprehensive technical proposal for items like networks, components and services.  The government can also check for 889 compliance.  Of course that’s subject to the limitations Fara noted.

Formerfed, your initial statement is so obvious, I overlooked it. The government can certainly ask for relevant information in the technical proposal.

1 minute ago, formerfed said:

One thing to keep in mind the equipment is highly sophisticated stuff.  I imagine any offeror selling it to the government is very knowledgeable of the sources.  

Probably so.

So it shouldn’t be a problem for them to identify the sources of the equipment and services they are certifying about. .

20 minutes ago, Fara Fasat said:

Answer: they know of no entities identified by SecDef, nor any published list.

Yet all prospective providers of covered telecommunications equipment or services must certify whether or not any sources are affiliates of one or more of the dirty five excluded sources?

Apparently neither the government nor the industry has access to a list of excluded affiliates of the dirty five!

This is even more reason why a contracting officer may not (should not) simply rely upon a simple “no” certification.

ji, FAR 4.2103 says that the contracting officer “may rely” on the representations. It doesn’t say “shall rely”.

“If the contracting officer has a reason to question the representation, the contracting officer shall follow agency procedures.”

I don’t know what the agency procedures are. But I wouldn’t rely on a simple all encompassing certification unless  there were some substantiation concerning the source company.

I described my possible procedure. It doesn’t involve any more research by the industry than what the certification requires. If a KO wants to spot check or totally check behind the offeror, they can search SAM for the identified source themselves (or have it searched by someone).

There is no need to require an offeror to provide all the information in the paragraph at 52.204-24 if they reviewed the SAM list and the entity wasn't listed or they otherwise determined that the source of the equipment or services is not an affiliate of the dirty five..

EDIT: Well it seems that Fara Fasat has discovered that SAM doesn’t list affiliates of the excluded dirty five after all. In fact there is apparently no 889 list of affiliates.

What a joke this whole thing is if a proposing firm can’t even find affiliates after being required to search SAM for affiliates of the “dirty five”!!

Bureaucracy!! 

5 hours ago, C Culham said:

I am confused.  Does not 52.204-26 require this?  Onerous or not I think the message of the requirement to check is clear.

Yes, you are confused.

Im proposing that the bidder/offeror/proposer provide substantiation beyond simply a certification X’d/checked box. That’s not verifiable.

They are already required to review SAM for each proposed source or supplier. To me then it should be no more onerous to identity/list them in their proposal and confirm that they specifically reviewed SAM.

I wrote performance specs during my career. Performance specifying should always include means of  “substantiation” (verification).

The same concept would seem prudent here - particularly since this is an important NATIONAL Cybersecurity issue. Fara Fasat has raised real concerns…

However, I don’t know any reasonable or practical way for either the company or the government to verify that the firm doesn’t have any prohibited sourced items or services in their own company operations.

To attempt to effectively comply with the Section 889 ban, It would seem that solicitations for telecommunications equipment or services  should also require the proposers to identify all sources and specifically check SAM and specifically affirm that each source is not a disqualified firm in SAM.

The ban includes using any banned sources of equipment, services or systems/programs in a firm’s own business operations. I think it’s an unrealistic (unstated) expectation for a (all) prospective federal bidder, offeror, contractor to inventory every service or telecommunications device, then search SAM to determine compliance, then certify compliance with the ban.

If the Section 889 ban is truly an  Nationally important imperative, then similar to a performance specification, two simple certification check boxes don’t really make any sense without some substantiation requirement to verify that the proposer has actually cross-checked SAM for each equipment or service source .

As President Ronald Reagan learned, then proclaimed: “Trust but Verify”. *
 

*The Russian proverb “doveryai, no proveryai (доверяй, но проверяй)” means 'trust, but verify'.

https://en.m.wikipedia.org/wiki/Trust,_but_verify#:~:text=Soviet–American relations,-1985 Reagan–Gorbachev&text=Suzanne Massie%2C an American scholar,'trust%2C but verify'.

1 hour ago, Don Mansfield said:

@joel hoffman,

What do you expect a contracting officer to do? It seems like Fara Fasat is trying hard to do more than just rely on a self-certification, but is getting nowhere. 

@Don Mansfield, I’m on @Fara Fasat’s side!  I commend him/her for his/her concerns and efforts. I probably would have done something similar if I was still working - especially concerning cybersecurity in today’s world situation.

I learned from my boss 45 years ago not to rely on or trust certifications. It’s the system here that sucks.

I don’t agree with a statement that relying on a certification** concerning cybersecurity is “entirely reasonable”.

You go, Fara!!!
 
**especially from the business (sales) side of a firm. My consulting firm owner would NEVER accept a certification from the business side of a firm.

13 hours ago, ji20874 said:

Relying on a self-certification in these circumstances is entirely reasonable.

Relying on self-certification is hollow, bureaucratic B.S. in these circumstances (Cybersecurity). It’s a gaping security hole.

I wonder what the new US Cyber Command would think of relying on contractor self-certification concerning their sources - especially when everyday government offices can’t even find a database of affiliates of the “big five”. 

19 hours ago, joel hoffman said:

Relying upon an offeror’s self certification is like burying your head in the sand. 

My opinion.

Maybe we need to rely more on ARTIFICIAL intelligence (AI) to seek out and maintain a current list of affiliates of banned companies if it is too much trouble for a government employee to do it.

3 hours ago, ji20874 said:

Yes.  "[W]here an agency has no information prior to award that would lead to the conclusion that the vendor, or the product or service to be provided, fails to comply with the solicitation’s eligibility requirements, the agency can reasonably rely upon a vendor’s representation/certification of compliance."

From the GAO case provided by Carl--

Relying on a self-certification in these circumstances is entirely reasonable.

And entirely stupid.

ji, I was fully aware of the GAO case. I quoted from it.

If the US is truly, deeply concerned about “Cybersecurity”, they wouldn’t rely upon self-certification by anybody, which apparently means that they don’t know who the constantly changing subsidiaries of the five listed firms are.

Fara Fasat rightfully wonders and is reasonably concerned where or if the US is tracking and/or listing subsidiaries.

Cybersecurity is critical, not to be trifled with.

Bureaucracy at its lowest.

My opinion. 

21 hours ago, Fara Fasat said:

provided link is a list of Chinese Military companies operating in the US, as required by the 2021 NDAA. There may be some overlap, but it is not a list of the subsidiaries covered by the 2019 NDAA telecommunications prohibition.

When the 889 prohibition first came out, the University of Minnesota and a couple law firms published lists. I'm wondering if they or anyone else maintain updated lists. I suppose it's too much to ask of the government itself to provide a list of the companies it has made subject to the prohibition. 

Relying upon an offeror’s self certification is like burying your head in the sand. 

”Our Office has repeatedly explained that where an agency has no information prior to award that would lead to the conclusion that the vendor, or the product or service to be provided, fails to comply with the solicitation’s eligibility requirements, the agency can reasonably rely upon a vendor’s representation/certification of compliance. See, e.g., Kipper Tool Co., B-409585.2, B‑409585.3, June 19, 2014, 2014 CPD ¶ 184 at 5 (denying protest that agency could not reasonably rely on representations regarding compliance with the Trade Agreements Act); KNAPP Logistics Automation, Inc.,”

How is an offeror supposed to know if a company is or isn’t a subsidiary of a prohibited source? By relying upon a certification by that company?

When I was a consulting engineer in the late 1970’s, my boss repeatedly derided the entire idea of self “certications” by vendors and contractors. We avoided reliance upon certifications to the maximum possible extent. 

If you are wondering about forms for project owner references, in our organization, we developed standardized questionnaires for project owner references to respond to.

You could contact some organizations to ask for a copy of what they use.

18 hours ago, Retreadfed said:

Joel, in this hypo, all the data the contractor has (e.g. time cards, production data, etc.) was disclosed to the government.  The issue is that there are various ways that data can be analyzed.  The contractor chose one way that leads to a specific result.  However, the government has the same data as the contractor and can analyze it the way the contractor did or use another analytical method.  In any event, the contractor's duty in regard to cost or pricing data ended when it disclosed all the data it had.  The contractor is not responsible for what, if any, use the government makes of that data.  The proposed cost is not cost or pricing data and a contractor does not certify its proposed price.  Instead, the contractor certifies the cost or pricing data it disclosed to the government.  For there to be defective pricing, the contractor must have failed to disclose current, complete and accurate cost or pricing data.  Here the data disclosed met that test.

Out of curiosity, if the contractor merely disclosed the lower quote and did not provide any explanation as to why that subcontractor was not going to be used, would defective pricing exist?

Retreadfed, we’re out of state this weekend on family business. I will respond next week. Have a good weekend!

5 hours ago, Retreadfed said:

Fara, I think the issue goes deeper than profit.  Two examples, in the first case, contractor X looks at the actual labor costs incurred to perform a previous contract for the same item being procured.  X uses the actual cost data in developing its proposed price and discloses that data to the government.  However, if a learning curve analysis is applied to the data, it would become apparent that the per unit cost for later procured items is less than the average unit cost computed using the actual labor costs.  Using Joel's analysis, the contractor would be required to use the learning curve data to develop its proposed price instead of the actual labor cost data.

Example 2, Y has used a particular subcontractor for a part for several years.  When preparing a proposal for a government contract, Y discovers a potential new supplier for that part who can provide the part at a cheaper price.  Y is not certain that the new supplier can provide and acceptable part within the time required by the potential contract.  Y discloses the lower quote from the new vendor, but relies upon prices quoted by its usual supplier in developing Y's proposal.  The way I understand Joel's position, Y would be required to base its proposal on the new and lower quote although Y does not intend on using that vendor.

Example 1: if it is “apparent that the per unit cost for later procured items is less than the disclosed, average unit cost computed using the actual [historical] labor costs” then the disclosed cost or pricing data used to price the contract isn’t “accurate, current or complete.” - to me it would be considered defective cost or pricing data. You are aware that the current cost will be lower than the disclosed cost. It isn’t Truthful.

Example 2: “The way I understand Joel's position, Y would be required to base its proposal on the new and lower quote although Y does not intend on using that vendor.”

That is not my position. As long as you explain the circumstances and disclose that you aren’t going to use the lower priced vendor in your second example , that is not defective pricing. You disclosed the vendor’s pricing that you intend to purchase and why you didn’t use the lower price.

1 hour ago, Atlas STS said:

I did a relatively small order three years ago for the same product so I have to provide that data.  But it would be inaccurate to use that data in my cost and pricing calculations for the current order being proposed.

Why do you have to provide data that is inapplicable to your price? You could mention it then say that it isn’t applicable because…

Who was the source of the briefing you referred to?

Yes, DoD organizations or specific offices often use a team to perform the Performance confidence evaluations.

1 hour ago, Steward said:

Out of curiosity, has any solicitation successfully included evaluation criteria that takes a look at the volume of protests an offeror files?  I ask due to the sheer disruption and often times few bad apples filing protests nearly every single time they lose, has there been any eval criteria that allows an agency to factor that in? 

This is more focused on small business procurements, rather than large $1B+ where a lot more is at stake financially, etc.  The $100M or less range

Thank you in advance

@StewardIf you evaluate that as a consideration (add evaluation criteria considering frequent protests) under an  factor and you don’t select the firm, what good is it? You said they will protest “almost every time” they aren’t selected…

I don’t think that you will accomplish anything other than provide another grounds for a protest, right?