Jump to content

DoD Source Selection Procedures (March 31, 2016)


Guest PepeTheFrog

Recommended Posts

  • Replies 50
  • Created
  • Last Reply

Top Posters In This Topic

Guest Vern Edwards
1 hour ago, Matthew Fleharty said:

If you're not willing to discuss numerical scoring (or even if you are), would you be willing to discuss your vision for Source Selection Procedures that provide helpful guidance for the DoD contracting workforce?

At the very least, off the top of my head, good source selection guidance would address:

  • source selection as a project and source selection project management;
  • the two objectives of the source selection process: (1) contractor selection and (2) contract formation;
  • the three main functions of the solicitation;
  • the nature, content, and two-part organization of proposals:
  • solicitation quality management: control and assurance;
  • the nature and selection of evaluation factors and the meaning and application of "relative importance";
  • the two steps of the evaluation process;
  • evaluation team organization, training, discipline, and management;
  • the distinction between evaluation and rating (scoring);
  • assessing proposals against factors;
  • the purpose and types of proposal rating (scoring) systems;
  • content, preparation, and presentation of evaluation documentation;
  • the distinction between and uses of clarification and discussion;
  • deciding whether to conduct discussions;
  • establishing a "competitive range";
  • pairwise proposal comparisons and proposal ranking;
  • tradeoff analysis methods and decision logic;
  • decision validation;
  • content and presentation of decision documentation;
  • the all-important distinction between decision explanation and decision justification;
  • notice and debriefing management and conduct;
  • protest venues, remedies, and procedures.

 

 

 

 

 

Link to comment
Share on other sites

ji20874:

How could the guide not affect offerors, they are the other party to solicitations? I'm not insisting just throwing it around.

If all evaluations with technical shall evaluate risk isn't that a consideration offerors will have to account for in their offer? Proposal preparation is critically important to business, time consuming, and expensive.

Could way say offerors are impacted in regards to how they evaluate Government requirements/solicitations and prepare offers? Again, just throwing it around without much slow thinking.

The procedures are in DFARS PGI 215.3.

Link to comment
Share on other sites

Guest PepeTheFrog
16 hours ago, Vern Edwards said:

Hi Pepe:

Love the avatar and moniker.

The long-standing knock on numerical scoring is that it falsely indicates objectivity. Numerical scoring doesn't do that. Rather, some people who interpret numerical scores interpret them that way. When used by people in the know, numerical scoring is a powerful analytical tool that is far superior to adjectives and colors. I will quote from a very great book, written under contract to the U.S. Navy by two very great decision scientists, Detlof von Winderfeldt and the late Ward Edwards. The book is Decision Analysis and Behavioral Research (Cambridge University Press, 1986). The quote that follows is from page 20:

Quote

Numerical subjectivity

The SEU [subjectively expected utility] model [of decision-making] embodies a fundamental principle. Both the utilities and the probabilities that enter into it are numbers, but they are inherently subjective in the sense that you and your colleagues might disagree about them and would have no way of resolving any such disagreement. The fundamental principle might be called numerical subjectivity, the idea that subjective judgments are often most useful if expressed as numbers. For reasons we do not fully understand, numerical subjectivity can produce considerable discomfort and resistance among those not used to it. We suspect this is because people are taught in school that numbers are precise, know from experience that judgments are rarely precise, and so hesitate to express judgments in a way that carries an aura of spurious precision. Judgments indeed are seldom precise--but the precision of numbers is illusory. Almost all numbers that describe the physical world, as well as those that describe judgments, are imprecise to some degree. When it is important to do so, one can describe the extent of that imprecision by using more numbers. Very offend, quite imprecise numbers can lead to firm and unequivocal conclusions. The advantage of numerical subjectivity is simply that expressing judgments in numerical form makes it easy to use arithmetic tools to aggregate them. The aggregation of various kinds of judgments is the essential step in every meaningful decision.

Emphasis added.

Back in the 1970s and early 1980s, the GAO encountered faulty use of numerical scoring by source selection personnel in a number of protest cases and complained that the method lent a false air of objectivity and precision to what is a matter of subjective judgment. But numbers didn't do that; poorly educated and trained human beings did that. The GAO has not said that numerical scoring cannot or should not be used. See IAP World Services, Inc.; Jones Lang LaSalle Americas, Inc., B-411659.2, 2015 CPD ¶ 302:

Quote

IAP argues that the agency used the artificial precision of a weighted numerical score to gloss over what would otherwise be a complex weighing of proposals' strengths and weaknesses. IAP Protest at 17–19; IAP Comments (July 27, 2015) at 31. The record shows that although the SSA's best value tradeoff discussion makes multiple references to the offerors' numerical scores, the SSA was well aware of the strengths and weaknesses underlying the evaluations. See AR, Tab 33, Source Selection Statement, at 20 (“I conducted my deliberations by first looking at the overall evaluated scores for Mission Suitability and the findings that led to those scores....”) While numerical scoring cannot substitute for a nuanced evaluation of proposals, here, the selection official had a basis to conclude that the scores were an accurate representation of the relative strengths and weaknesses of the proposal.

There we have a case of an SSA who knew what he or she was doing. However, some agencies (e.g., DOD) have prohibited the use of numerical scoring rather than properly train their people in the use of that extremely effective tool. Ironically, DOD's contractors use numbers when conducting systems engineering tradeoff analyses, and no one thinks anything of it.

Sigh.

There was a time when I used to try to explain all this to contracting people, but ignorance is so deeply rooted among us that I no longer waste my time. I have learned that many people in our business simply will not read books and will not study and prepare themselves for the tasks ahead unless their boss sends them to some government-sponsored class or they can hop on Wifcon for a quick and easy answer. They have allowed their ignorance to deny them the use of a powerful tool for the conduct of complex source selections. In explaining now I am making an exception, because I like your style and because I sense a keen intelligence behind the humor.

PepeTheFrog really appreciates this, Vern, and is thankful for your contributions to the intellectual and practical body of knowledge in contracting. Years ago, a Government contracts tadpole discovered your insights, analysis, and commentary and realized there was more to the profession than meets the eye. That young tadpole learned the Government contracts pond was quite deep, even intellectually stimulating, for those willing to take the plunge.

 

PepeTheFrog should have stated that the "veneer of objectivity" criticism applies only to those in contracting who misuse numerical scoring, not the concept or practice itself. PepeTheFrog was thinking cynically about what is rather than what should be. Unfortunately, there are not enough "people in the know" in federal contracting, which is a shame. Bring back the civil service exam, set "LPTA" to one standard deviation above the mean, and for cover, use the same attorneys who justify corporate screening exams, e.g. Procter & Gamble's "Reasoning Test."

 

 

PepeTheFrog suspects one reason DoD went to kindergarten colors was to emphasize ordinal instead of cardinal numbers, and to prevent the ability to aggregate the numbers, or manipulate them mathematically. Five colors, five ratings could just as well be five ordinal numbers: one, two, three, four, and five. Luckily, the Better Buying Power Mantras haven't inspired anyone to "do more (mathematics) with less (numbers)" by adding purple, blue, and blue, and comparing that "sum" to purple, purple, and blue...or have they?

 

PepeTheFrog will sit on a log and think for a long, long time about whether it is true that "subjective judgments are often most useful if expressed as numbers." There's a lot to contemplate. If anything useful and coherent bubbles up, it will be posted to the forum.

Link to comment
Share on other sites

Guest PepeTheFrog
On 4/9/2016 at 4:15 AM, Jamaal Valentine said:

Pep, pep, Pepe! My imaginary contracting office just grew by one for a total of 8 (not including myself).

Thanks, Jamaal Valentine. PepeTheFrog is happy to hop on the Dream Team.

Link to comment
Share on other sites

On 4/8/2016 at 4:26 PM, Jamaal Valentine said:

I was thinking the same, but based on some of Don's latest posts I'm unsure. Curious to know how they reconcile FAR 1.401(e),1.401(f), and 1.404 requirements.

I think a reasonable argument can be made that the DoD SSP has a significant effect beyond the internal operating procedures of the agency or has a significant cost or administrative impact on contractors or offerors. 

I can only assume that someone in DOD consciously considered these sections before incorporating the Source Selection Procedures into DFARS PGI (see DFARS PGI 215.300.)  They must have concluded as Vern suggested, that the procedures are internal, and do not constitute a deviation.

Link to comment
Share on other sites

Jamaal,

The guide is addressed to internal agency personnel, and nothing in it is inconsistent with the FAR -- it is guidance on what internal agency personnel might do when working under FAR Subpart 15.3 -- it does not impose any burden on any contractor.

Link to comment
Share on other sites

Does the DFARS PGI 215.300 (Source Selection Procedures) infringe on the wide latitude contracting officers are afforded by FAR 1.602-2? If yes, have the requirements in FAR 1.401, 1.301, and 1.404 been considered and/or complied with?

I've already offered an example, above, of how prospective offers can be affected by the procedures in the guide.

Just wondering how these things come to pass and if any of the above is applicable to publishing a PGI or similar internal policy.

Link to comment
Share on other sites

Jamaal,

Are you insisting yet?  Or still just throwing it around?

By the logic of your questions, a contractor "can be affected" by an internal policy that established a lunch hour for agency employees between 11:30 and 12:30.  A contractor "can be affected" by an individual contracting officer decision to stop negotiations for today and resume tomorrow.  Would you want to apply the procedures for deviations and public comment to these?  I think you're pushing too hard.  The guide does not impose any burden on any contractor or do anything else described in FAR 1.301.  The standard for deviation or public comment is not whether a contractor "can be affected."

I don't like the guide -- for example, I don't like the mandating of 5 ratings (outstanding, good, satisfactory, marginal, and unacceptable) -- but I support the management prerogative of the agency head to issue guidance to agency employees.

Link to comment
Share on other sites

2 hours ago, Jamaal Valentine said:

Does the DFARS PGI 215.300 (Source Selection Procedures) infringe on the wide latitude contracting officers are afforded by FAR 1.602-2? If yes, have the requirements in FAR 1.401, 1.301, and 1.404 been considered and/or complied with?

I've already offered an example, above, of how prospective offers can be affected by the procedures in the guide.

Just wondering how these things come to pass and if any of the above is applicable to publishing a PGI or similar internal policy.

Infringement "on the wide latitude contracting officers are afforded by FAR 1.602-2" is not the standard for whether or not the requirements of FAR 1.401, 1.301, and 1.404 need to be considered and/or complied with.  For deviations we have a definition (which you reference at 1.401) and nowhere within the six standards for "deviation" is 1.602-2 mentioned.  1.602-2 is also absent from FAR 1.301, where the consideration is instead whether or not the SSPs are an agency acquisition regulation (1.301(a)(1)) that meets the standard for publication for public comment.

If you want to make an argument that DFARS PGI 215.300 meets one of those standards, let's discuss that instead of a standard that isn't present in any of the FAR references you state should have been considered and/or complied with.

Link to comment
Share on other sites

3 hours ago, Jamaal Valentine said:

Does the DFARS PGI 215.300 (Source Selection Procedures) infringe on the wide latitude contracting officers are afforded by FAR 1.602-2? If yes, have the requirements in FAR 1.401, 1.301, and 1.404 been considered and/or complied with?

Jamaal, do you really think that Contracting officers have such a "wide latitude" under FAR 1.602-2 that the organization that issued their warrants has no right to issue internal regulations governing how a contracting officer's authority is exercised?  Perhaps you skipped over 1.601 ( a )? A KO is far from being the final authority in an agency on how to contract for supplies and services.

So, if there is no applicability concerning a deviation to FAR 1.602-2 in the Source Selection Procedures, where do they otherwise deviate from the FAR or DFARS?

Link to comment
Share on other sites

1 hour ago, Matthew Fleharty said:

Infringement "on the wide latitude contracting officers are afforded by FAR 1.602-2" is not the standard for whether or not the requirements of FAR 1.401, 1.301, and 1.404 need to be considered and/or complied with.  For deviations we have a definition (which you reference at 1.401) and nowhere within the six standards for "deviation" is 1.602-2 mentioned.  1.602-2 is also absent from FAR 1.301, where the consideration is instead whether or not the SSPs are an agency acquisition regulation (1.301(a)(1)) that meets the standard for publication for public comment.

If you want to make an argument that DFARS PGI 215.300 meets one of those standards, let's discuss that instead of a standard that isn't present in any of the FAR references you state should have been considered and/or complied with.

There were two questions. You appear to have provided a response to one so I'll discuss that further if you outline - what are the standards in FAR 1.401(a),(e), and (f) referring to if not FAR, etc.?

Link to comment
Share on other sites

Where else do the Source Selection Procedures deviate from the FAR or DFARS, since they don't deviate from 1.602-2?

Link to comment
Share on other sites

4 minutes ago, joel hoffman said:

Jamaal, do you really think that Contracting officers have such a "wide latitude" under FAR 1.602-2 that the organization that issued their warrants has no right to issue internal regulations governing how a contracting officer's authority is exercised?  Perhaps you skipped over 1.601 ( a )? A KO is far from being the final authority in an agency on how to contract for supplies and services.

So, if there is no applicability concerning a deviation to FAR 1.602-2 in the Source Selection Procedures, where do they otherwise deviate from the FAR or DFARS?

Not at all what I think. For starters, I'm just kicking this around to see what other's think regardless of my true personal opinion. It's interesting to hear others opinions, interpretations, defense strategies, etc.

I think agencies issuing warrants have the authority to manage their agencies as they see fit - the question is what is the process.

You mention internal regulations that govern contracting officers, but FAR 1.3 provides the process for promulgating agency regulations that implement or supplement the FAR. Can this be done haphazardly on an ad-hoc basis?

Link to comment
Share on other sites

The source selection procedures do not supplement the FAR in my opinion. 

Link to comment
Share on other sites

Jamaal,

The criteria for required publication, stated at 41 U.S.C. 1707, are whether the policy, regulation, procedure, or form:

  1. has a significant effect beyond the internal operating procedures of the agency issuing the policy, regulation, procedure, or form; or
  2. has a significant cost or administrative impact on contractors or offerors

What in the Source Selection Procedures do you think meets either (or both) of these criteria?

Link to comment
Share on other sites

3 hours ago, Jamaal Valentine said:

There were two questions. You appear to have provided a response to one so I'll discuss that further if you outline - what are the standards in FAR 1.401(a),(e), and (f) referring to if not FAR, etc.?

Asking another question does not make an argument.  I agree that the standards in FAR 1.401(a), (e), and (f) are referring to the FAR (after all they explicitly state as much), but that does not mean that FAR 1.602-2 is some sort of trump card that results in regulations or procedures (in this case the DoD SSPs) requiring deviation authorization or publication.  In regards to whether or not the DoD SSPs require(d) as much...:

  • For (a) you need to provide an argument showing that the DoD SSPs are "inconsistent with the FAR."
  • For (e) you need to provide an argument showing that the DoD SSPs authorize "lesser or greater limitations on the use of any solicitation provision, contract clause, policy, or procedure prescribed by the FAR."
  • For (f) you need to provide an argument showing that issuance of the DoD SSPs are not "in accordance with 1.301(a)."  Here, Don Mansfield has provided the criteria in the above post for consideration.

Such arguments would warrant a meaningful discussion regarding the merits of how the DoD SSPs came to be (after all, whether or not the DoD SSPs even impede on the amount of latitude contracting officers have in exercising business judgment has yet to be shown, it has only been inferred by referencing 1.602-2).  Also, it may be worth nothing that 1.602-2 merely states "contracting officers should (emphasis added) be allowed wide latitude to exercise business judgment" versus shall.

Link to comment
Share on other sites

Matthew: 

Neither does avoiding pointed questions or twisting words. I'm not saying you are wrong so don't take it personal. Just read my post, the FAR, and the SSP carefully and participate at your leisure. This is just a discussion. I'm not married to any thoughts I've posted on this subject.

I asked if the SSP infringed on the wide latitude contracting officers are afforded by FAR. A question, not an inference. If you had answered that question it would have changed the discussion.

For arguments sake, in your opinion, would the SSP be inconsistent with FAR (a deviation) if FAR permitted evaluating a technical factor without evaluating risk, but now contracting officers shall evaluate risk when evaluating technical factors as outlined in the SSP? If you believe that's a deviation what is the process that should be followed? I have not and am not saying the process was not followed.

Note:

Should means an expected course of action or policy that is to be followed unless inappropriate for a particular circumstance.

The SSP states that its policies supplement existing statutes and regulations.

 

Don:

Frankly, not at this time, but what's the fun in that? I perused the SSP and reserve the right to change my mind.  It does seem that the SSP governs the process between contracting agencies and prospective offers (FAR 1.301) and will have some affect on offerors. I do not claim to know the degree of impact. From my experience on the industry side - responding to solicitations is time consuming and expensive. The more requirements placed in a solicitation, the more time and money it takes.

I really just wanted to see what people think about the process of issuing agency regulations, and why they think it. I honestly would like to know more about the process and requirements for issuing a PGI or mandatory procedure.

Link to comment
Share on other sites

Jamaal,

My mistake for assuming you were taking a position on the issue based on your line of questioning.  I'm certainly not taking anything personally or avoiding pointed questions (the question was quite open/broad) or twisting words (I quoted fairly heavily from the FAR and your posts); still I think this discussion would have been better served had you taken or stated a position one way or the other from the beginning because, as someone that does not think the DoD SSPs take away contracting officers' "wide latitude to exercise business judgment," I (or others) who felt similarly were in a position to try and prove a negative (which is a fallacy in reasoning).  Until someone makes a credible argument that they do, I don't believe there is much discussion to be had regarding that line of questioning.

27 minutes ago, Jamaal Valentine said:

For arguments sake, in your opinion, would the SSP be inconsistent with FAR (a deviation) if FAR permitted evaluating a technical factor without evaluating risk, but now contracting officers shall evaluate risk when evaluating technical factors as outlined in the SSP? If you believe that's a deviation what is the process that should be followed? I have not and am not saying the process was not followed.

Note that the SSP states that its policies supplement existing statutes and regulations.

Of the six criteria for "deviation," I think two are pertinent to your question: 1.401(a) (is it inconsistent with the FAR) and 1.401(e) (does it impose lesser or greater limitations on the use of any policy or procedure prescribed by the FAR).

For 1.401(a), inconsistency is the operative word and one definition is "not compatible with."  Your scenario lacks specificity, so it would depend on the hypothetical FAR language that permits evaluation of a technical factor without evaluating risk as to whether or not the DoD SSPs are inconsistent.  The likely case is that the FAR is silent and broad on what constitutes an appropriate evaluation factor (as it is today) and therefore, the additional specificity in the DoD SSPs wouldn't necessarily represent an inconsistency, but rather more specificity.

In which case, 1.401(e) becomes an interesting discussion.  Requiring the evaluation of risk when evaluating technical factors would likely be a requirement above merely evaluating the technical factor; however, it's impossible to answer this hypo without specific language because the FAR currently states under 15.304( c ) "The evaluation factors and significant subfactors that apply to an acquisition and their relative importance are within the broad discretion of agency acquisition officials, subject to the following requirements..."  In that case, I don't believe it imposes greater limitations because the FAR has already ceded that discretion to the agency officials.

Therefore, because the DoD does not meet either of the relevant two criteria for a "deviation" and because of 15.304( c ) I'd argue that it would not be a deviation needing to follow the process.  As for the process that should be followed, I know from reading your posts on Wifcon that you're smart enough to have already drilled down to the DFARS, AFFARS or whatever agency supplement applies to your organization and that you're likely asking as a matter of practice.  Unfortunately, I cannot help you there as I have not had to process a deviation before.  Happy hunting for someone who has!

Link to comment
Share on other sites

  • 2 weeks later...

Is the Value Adjusted Total Evaluated Price (VATEP) process essentially a weighted numerical rating?

VATEP is aimed at reducing subjectivity in the tradeoff process, but SSAs are still required to evaluate the benefits/advantages, if any, of trading one or more factors for more or less of another. Specifically, SSAs have to determine if the tradeoff is worth it ... isn't VATEP supposed to determine "worth" and "value" in advance so offerors can make informed proposal decisions?

If VATEP doesn't facilitate clear, objective, fixed "mechanical" evaluations and award decisions what real value is it to the offerors it's trying to provide proposal instructions and basis for award information to?

Just trying to think through benefits and drawbacks of VATEP in advance of being prompted to try it.

Link to comment
Share on other sites

VATEP is an approach to quantify or monetize the tradeoffs in a selection decision.  FAR 15.308 clearly says it is not necessary to quantify the tradeoffs, and the GAO has consistently upheld this principle, but some people insist on doing it anyway.  VATEP is a structured way to do it.  VATEP is a way to remove subjectivity from the selection process and make it more objective -- instead of an SSA's subjective and independent judgment, a VATEP selection decision can be made by a clerk with a scoring worksheet.  Well, maybe I exaggerate a little.  

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.

×
×
  • Create New...