Matthew Fleharty

Bingo - there is a difference between wanting to understand various approaches during market research and evaluating approaches (which are usually not promises) during source selection. @Scrutor Had your story occurred during a source selection (not some measly purchase order) and the one offeror wrote that as their approach, received the contract because the evaluators liked what they read, and then started to perform without using the ground protection mats (let's say because using the mats would have cost more money and a dollar saved is a dollar earned for the company), what then?

❤️ 😂

I don’t contest that this is the norm - that said, are our R&D projects generally successful (choose your measure(s))? That’s worth thinking about because, if they’re not, well we all know what they say the definition of insanity is…

Agreed

Thanks everyone who has contributed to the discussion. I’m working on an article that critiques the common practice of requesting approaches so I’ll have a more fulsome response for your thoughts/feedback soon. While I rarely say always or never, I have not seen/encountered a situation in my professional career that would have benefited from evaluating approaches in competitive acquisitions - quite the contrary, I’ve only seen the practice make situations worse. I strongly disagree with this position - if most acquisition “professionals” lack competency, most customers are even more ill informed about how to properly structure a competitive acquisition’s decision making process. In almost all cases, they’re insistent on doing something because that’s what they did last time which certainly doesn’t make it right. Vern once told me (or wrote) contracting officers don’t have “customers” we have “clients” and unlike the adage “the customer is always right,” part of doing professional work is being able to tell a client “no.”

Without details of what? We can evaluate details of the offer and the offeror. Is it necessary or even beneficial to evaluate details of their approach? If so, what's the strongest argument(s) for this assertion?

@Don Mansfield Towards the end of the podcast you state you're looking for a word to describe the FAR Part 15 "competitive" negotiations process - I think the appropriate word is "begging" (e.g. we send evaluation notices asking offerors to improve/revise an aspect of their proposal/offer) which contrasts nicely to what we should be doing, and as Vern notes we do under the A&E process, which is "bargaining."

Thanks for the prompt Vern - it does make sense to get on the same page first. In this context, "approach" means "information in a proposal describing how a contractor may do something." I'll not that in most cases this information is not an offer/promise. In this context, for "understanding" I like the framework of Bloom's Taxonomy so understanding means "ability to demonstrate comprehension of something by explaining ideas or concepts."

@formerfed @FrankJon & any others: I’m genuinely curious and want to continue the discussion, but not further derail Vern’s thread so: What’s your best argument(s) for why the government should evaluate “approaches” or “understanding”?

In almost all cases an offeror’s approach is not a promise/offer - assuming the approach is not a promise, do you still believe the government should evaluate approaches?

I would not consider an offeror's approach as an evaluation factor. Why is there such an obsession with evaluating offerors' approaches? As for "risk" in a source selection, how do you measure it? And, if you try to, how is it distinct from what has already been evaluated (the offer and the offeror's capabilities) as to not create double counting? Moreover, the "risk" might not be attributable to the offer/offeror at all - it may simply be the product of the government's own requirements (consider the risk in recapitalizing the country's intercontinental ballistic missiles...) at which point why waste the time.

I should have been more clear - I don't take exception to evaluating past performance or experience (both fit within the definition I provided as attributes of an offeror). I do however take exception with evaluating things like "understanding" and "risk" as I think they produce noise. I think both current and future part 15s are wrong for stating that. I'm curious what your difference response would be.

Here's my concern with applying AI to government acquisitions - it's drawing from what we've already done, it is not creating anything new/novel. And what we've done/been doing over the past two decades isn't exactly great. The result is that AI is going to spit out the same or similar products and we're going to get the same or similar results. Is that what we need right now? To get to mediocrity (or worse) faster? I don't think so, but I'm just speaking for myself.

I disagree as to whether that's proper. When you say "we" if you're talking about the acquisition workforce you're correct in that for some absurd reason many people usually consider aspects that go beyond the offer through insane essay writing contests. This is NOT a good thing. What does it mean to evaluate their "understanding" particularly when proposals are written by capture teams (or today it could even be AI) and then, once awarded, tossed over the fence for a separate team to perform? What "value" does that provide? Vern has written extensively critiquing the all too common practice of evaluating things/proposals that do not provide value to the government. See the following: 1. “A Primer on Source Selection Planning: Evaluation Factors and Rating Methods” by Vern Edwards 2. “Contracting Process Inertia: The Enduring Appeal of The Essay-Writing Contest” by Vern Edwards, Addendum by Ralph C. Nash 3. “Streamlining Source Selection by Improving the Quality of Evaluation Factors” by Vern Edwards, Addendum by Ralph C. Nash Nate Silver wrote in his book The Signal and the Noise "Information is no longer a scare commodity; we have more of it than we know what to do with. But relatively little of it is useful. We perceive it selectively, subjectively, and without much self-regard for the distortions that this causes. We think we want information when what we really want is knowledge." (emphasis added) We can do better - let's not regurgitate the poor source selection practices that have built up over time having been copy and pasted ad infinitum to deliver acquisition outcomes that no sane person would consider respectable.

I would define value as: worth assigned to an attribute of an offer or offeror

I don't intend to be mean, but I will be critical: this is an example of one of the biggest problems in our profession today - people don't take the time to research or think things through, they just go along saying or doing things unthoughtfully. For example, why did you choose the word "proposal"? There is a good discussion going on right now in these forums about the difference between "proposals" and "offers" - might be worth a read and it will likely (hopefully) prompt you to rethink your choice of the word "proposal." I'm going to heed my own advice and think on Vern's question for a bit before providing my input. Looking forward to the discussion.

This is faulty logic - transaction count does not automatically mean most of the work hours are spent on those activities - it just means there are more of those transactions. For example, if we have 100 sprinters run 400 meters and it takes each of them 60 seconds to run that distance, the total aggregated time spent running is 100 minutes. If we have 1 person run a marathon and it takes that runner 3 hours (180 minutes) to run that distance, despite considerably fewer runners (only one!), the total aggregated time spent running is longer. You cannot draw any sound conclusions on where most work hours are spent (FAR Part 15 vs. Commercial...to say nothing of contract administration) based solely on the FPDS data you allude to in your post.

This is one of the problems with the "statutory" filter they are using for the RFO - even if some language in the FAR is not required by statute, that language could be useful in producing better acquisition outcomes for both parties. As to whether contracting officers free to use the clause or any variation will do so, I doubt it with the current state of education and training in our profession and the mandate of contract writing systems/clause logic generators that constrain contract writing. Many people will not know to use this clause or others like it (or even that they exist or how to find them post-RFO transition), while others simply will not want to go through the trouble of fighting the system(s) to do so. Meanwhile, other language not required by statute remains. See RFO FAR 12.304(b)(1) "Before terminating a contract for cause, send a cure notice to the contractor, unless the reason for the termination is late delivery." I had my eye on whether that language (which is in the current FAR at 12.403(c)(1) "The contracting officer shall send a cure notice prior to terminating a contract for a reason other than late delivery.") would stay, go, or be rewritten after a frustrating experience where it constrained our options on an important service contract. I have found no statutory basis for that language/requirement for a cure notice. At least they're consistent in their inconsistency...

@Self Employed I think you came here with a point of view that you're stuck on and you're not reading what I wrote. I wrote that I agreed with both those sentiments, but those are separate issues worthy of a separate more fulsome discussion. You also contradict yourself when you state: So why do we need a different pay, development, incentive system if the people who are going to develop are going to do so anyhow, development program or not? (I don't agree with your sentiment btw - some people need to be shown the way and once inspired will thrive) I think there is a fundamental misunderstanding in the contracting and acquisition community of the power of expectations. That's why I think the 80 CLP standard and the Powerpoint deep CBT nonsense we're constantly exposed to and multiple guess exams to demonstrate proficiency is a serious problem. It sets an expectation that merely going through the motions is enough and in a world full of competing priorities many people are simply going to do what is expected of them. That is why we need to raise the standards - many will work to achieve them, some won't be able to, but we'll recruit and retain those who can. And yes, with raised standards I think raised compensation is appropriate/necessary too.

I partially agree that many good CO's leave or promote out of doing CO work, but that's not the only reason we are where we are (as you assert) - we still have a development problem and I think it's been getting consistently worse since I entered the career field over 15 years ago. Do I think we need to re-examine compensation and some of the career ladder concepts? Absolutely - that's an article for another time. But if we only did that, we would still have a problem developing the next generation of contracting professionals and have no one to offer those things to...

Why do you think it is a best practice to not have people on multiple teams? What if you only have a small shop with minimal people? Or only one person who has any experience leading a source selection? Since you're referring to the DoD Source Selection Procedures, has the DoD been successful in implementing and utilizing those procedures and following the practice you detail in your post? If not, why would you want to continue to do it, especially when you claim it isn't "explicitly stated in the procedures that they can't overlap..."? In short, what do you think is best and why?

(Emphasis added above) Thanks for highlighting this distinction Vern - the takeaway is spot on. If readers want to know why training without education is dangerous, just look at our source selection process - all training, zero education. The result is we often end up with copy/paste solicitations without thoughtful evaluation factors or methodologies that cause source selections to take way too long evaluating "essays" that are not important to predicting which offeror and their proposal might be the best value.

Both To note a couple items that stood out from the DoDI, Section 5 establishes the "low expectations" I criticized in my piece and Section 3 makes the President of DAU as the Chief Learning Officer... in my 16 years I haven't seen any real leadership from that role, if anything the standards and courses have eroded more and more...whose responsibility is it to say "this is not or cannot produce the acquisition workforce we need to generate the acquisition outcomes we need" if not a Chief Learning Officer? Execution is a huge problem too. It's apparent from some of the pockets of excellence that people can still excel in this profession and pursue mastery (as Vern likes to say), but we definitely make that pursuit way more difficult than it ought to be. For one, we don't leverage peoples' most formative moments when they enter contracting to properly educate and inspire them. I mention this in the piece so I won't further belabor the point here. I listened to the two podcast episodes you recorded (thank you for those btw) and I think in both comments were made that contracting people don't read. I think that's fairly accurate, BUT we ought to consider why they don't read: one of my hypotheses is because the "professional" products the acquisition system currently provides them are mostly garbage. I remember in high school when I went through a stretch in my English class of reading a few assigned books I did not enjoy - it made me not want to read for a bit. That's our contracting people's daily existence - suffering through reading bad policies, Powerpoints, and guides that they don't benefit from so at some point many of them just give up (i.e. people won't go back to try and drink from a well that's dry).

It's the kind of thing that just instills confidence in what might come out the other end of this process...

Admiral Rickover has a quote that I am fond of (emphasis added): "Organizations don't get things done. Plans and programs don't get things done. Only people get things done. Organizations, plans, and programs either help or hinder people." FAR reform will not fix our acquisition system because our system relies on people to make decisions. People will still need to read, interpret, and apply the FAR - whether they make sound decisions or bad decisions is based on their knowledge and abilities, little of which has to do with what is or is not in the FAR. A better written FAR could help them, while a poorly written FAR could hinder them, but it's the people that policymakers and leadership have lost sight of. We are not setting our people up for success. None of the people in the government acquisition workforce receive quality education on contracting as a standard practice. None of the people in the government acquisition workforce have access to best in class materials like the Cibinic and Nash Report or Briefing Papers. What our people get instead are Defense Acquisition University (DAU), a littany of computer based trainings (CBT), and on the job training (OJT) that, in most cases, is little more than showing them what was done last time and expecting them to do the same....oh, and calls for AI, "innovation," and buying commercial as if they are panaceas... If we have a well educated acquisition workforce, I believe they could overcome even the most convoluted FAR (and any other organizations, plans, or programs that might serve as hinderances). Conversely, if we have an uneducated acquisition workforce, I believe we could have the best written FAR and that workforce would still find ways to break it.