Jump to content
The Wifcon Forums and Blogs
Guardian

Immediate Comparative Analysis for Fair Opportunity Selection

Recommended Posts

I am working with our customer to write evaluation criteria for a Fair Opportunity Proposals Request (FOPR) against an MA IDIQ/GWAC.  I found a document online entitled "Air Force Materiel Command (AFMC) Guiding Principles For Fair Opportunity Selection Under Federal Acquisition Regulation (FAR) 16.505(b)(1)."  To quote from within, "[concerning FOPRs and comparative analyses] No process is dictated by FAR 16.5.  Immediate comparison of responses received is allowed without having to "independently score" proposals and then separately do comparative analysis."  This is stated in comparison to the procedures required by FAR subpart 15.3.

The acquisition team has agreed to use a multiphase approach, whereby phase one would incorporate a downselect.  It is within this first phase that we would like to move immediately to a comparative analysis on a single non-price factor, considering also price, as opposed to first conducting an in-depth review of each proposal against the factors in the solicitation. Our attorney accepts this approach. The estimated award amount is well below the threshold for protestability.  However, I am receiving push-back from above.  That person says they do not understand how we can do a comparative analysis without first conducting an independent evaluation of each contractor (and document accordingly), be it through a table or however.  I explained that any comparative analysis is always going to be based on having first looked at each individual proposal and assessing the factors that we say we will.  I went on to explain that the time saved is realized in the documentation of phase one, meaning that we include all relevant information to justify our downselect in a single document labeled "Phase One - Comparative Analysis."

I was able to find another article online entitled "How to Write a Comparative Analysis" from the Harvard College Writing Center."  One suggested approach is as follows--

Organizational SchemeYour introduction will include your frame of reference, grounds for comparison, and thesis. There are two basic ways to organize the body of your paper.

  • In text-by-text, you discuss all of A, then all of B.
  • In point-by-point, you alternate points about A with comparable points about B.

If you think that B extends A, you'll probably use a text-by-text scheme; if you see A and B engaged in debate, a point-by-point scheme will draw attention to the conflict. Be aware, however, that the point-by- point scheme can come off as a ping-pong game. You can avoid this effect by grouping more than one point together, thereby cutting down on the number of times you alternate from A to B. But no matter which organizational scheme you choose, you need not give equal time to similarities and differences. In fact, your paper will be more interesting if you get to the heart of your argument as quickly as possible.

[end citation]

My question is, do any of the contributors have another suggested approach for how to write the comparative analysis barring an initial independent evaluation of each contractor in the downselect phase? [or] Does anyone have any suggestions as how I can make a more compelling case to my management?  I have addressed with them and general counsel my concerns that our required templates mandated for fair opportunity selection tend to use part 15 language.  The attorneys and others are consistently resorting to part 15 terms during conferences.  Personally, I am a stickler for language and use of the proper terminology.  I for one happen to think that is the starting point, and even though I am working with people who should know otherwise, I tend to question if they know the differences when I hear them apply the wrong terms.  We also tend to overly complicate our fair opportunity selection procedures by aligning them more so with those found in part 15.  Lastly, I am curious, does anyone have an opinion as to which agency seems to be the most innovative in its approach to simplified procedures and fair opportunity awards using category management?  I am reading more and more that this reluctance to move away from FAR part 15 procedures, when we are not under the authority of part 15, is a widespread enough problem in Government.

Share this post


Link to post
Share on other sites

https://www.dhs.gov/sites/default/files/publications/pil_boot_camp_workbook.pdf

Technique 5 in the DHS Procurement Innovation Lab Boot Camp Workbook (link above) is on comparative evaluation -- and Technique 4 is on down-select.  You can do a down-select on a single non-price factor (or a couple of non-price factors), and you can make your down-select decision on a comparative evaluation basis (no adjectival ratings).

One of the key messages from the DHS procurement innovation community is to "stay in your lane" and to keep FAR subpart 15.3 procedures out of your fair opportunity considerations.  DHS is also encouraging streamlined evaluation and selection documentation, and has won GAO bid protest decisions where unsuccessful offerors say the evaluation and selection documentation are too short.

Share this post


Link to post
Share on other sites

From p. 14 T/F table titled "Fair Opportunity/Orders Under Multiple-Award Contracts"

Quote

A notice/solicitation for a competitive order in a fair opportunity consideration less than $5.5 Million must list the relative order of importance of the evaluation factors

Some people would call that sloppy.

Share this post


Link to post
Share on other sites

I actually like the construction — it combines both “notice” which is what it really is/what the FAR calls it and “solicitation” which is what some people think of it as when they’re in a hurry, so everyone is happy and no one is offended.  It helps reinforce correct principles, and is a good compromise for a document where the purpose is meaningful learning rather than pedantry.

Share this post


Link to post
Share on other sites
54 minutes ago, ji20874 said:

I actually like the construction — it combines both “notice” which is what it really is/what the FAR calls it and “solicitation” which is what some people think of it as when they’re in a hurry, so everyone is happy and no one is offended.  It helps reinforce correct principles, and is a good compromise for a document where the purpose is meaningful learning rather than pedantry.

Not surprised you like it--aren't you one of the PIL team members?

In any case, I didn't say I didn't like it. It's just someone who wrote this in another thread:

Quote

A fair opportunity notice is not a solicitation within the construct of the FAR, even though it may be informally (sloppily?) referred to as such within our community.

A pedant like that might take issue with your document.

Share this post


Link to post
Share on other sites
23 hours ago, Don Mansfield said:

There's no need to evaluate each factor on a scale to use these methods.

 I understand, Don.  It seems the "even swaps" method would work better using rankings as the HBR article states.  I like the ranking or scaling method for this approach.  However, I know a lot of contracts managers are averse to the idea of ranking.  Why do you think this is?  Perhaps it is equated with numeric scoring, which has become a bit of an anathema.  For example, no one wants to explain to the GAO why an offeror lost to an awardee by a single point.  It seems to me we rank offerors regardless as evaluators and selection authorities, if in no other way, in our minds as we are performing the evaluation.  The question is whether we document it as part of the file, making it something subject to discovery.  If we don't rank, then we have to document our evaluation narratively, which some might consider a more difficult exercise, leading to something less definitive and more abstract.

 

16 hours ago, Ibn Battuta said:

Do you think you can't do "immediate comparisons" when conducting a source selection under FAR Subpart 15.3?

No.

 

17 hours ago, ji20874 said:

Technique 5 in the DHS Procurement Innovation Lab Boot Camp Workbook (link above) is on comparative evaluation -- and Technique 4 is on down-select.  You can do a down-select on a single non-price factor (or a couple of non-price factors), and you can make your down-select decision on a comparative evaluation basis (no adjectival ratings).

Some contracts managers believe that we have to document risks and benefits as part of every evaluation, something pushed by our attorneys, as well. Are the attorneys at your agency encouraged to attend these seminars? I appreciate the innovation lab and the work it does.  Unfortunately, I think the continued aversion to risk and unwillingness to adopt the techniques the lab suggests still emanate from the torso down.  Some view the concept of innovation as the innovation lab having provided them with a finite list of techniques. FAR 16.5's use of the term, "broad discretion," allows for much more, i.e., the lab is but a starting point and does not preclude the application of further innovation and other techniques not therein described.  A division will adopt an innovative approach, but then becomes reliant on only that one approach.  Contracts managers become unwilling to flex and further adapt that approach.  Program offices become lazy, recycling the same evaluation method time and again as if it is a one-size fits all approach to selection.  It can be rather stressful for the thinkers and creative types in the 1102 community who do not want to be relegated to factory work.  Those for whom employee retention is a concern somehow remain baffled over the attrition rates.  The question for the practitioner is whether it is better around the bend. 

Share this post


Link to post
Share on other sites

The workbook shows a way to do a comparative evaluation. It neither pretends nor presumes to show the only way.  The workbook is not a policy document; rather, it is an aid to practice (but everything in it conforms to DHS policy requirements).

PIL Boot Camps are for contracting officers, program managers, and procurement attorneys.  Everything is legal and honorable, and is offered for those who are looking to innovate.

If you like the technique, feel free to use it or let it inform your own approach.  Afterwards, share your story with your colleagues so others can benefit from your learning.

Share this post


Link to post
Share on other sites

FAR 15.305 speaks of evaluating offers and FAR 15.308 speaks of a comparative assessment, both as part of the formal 15.3 source selection process.  Generally, in formal 15.3 source selections, adjectival ratings are assigned in the evaluation process before the comparative assessment occurs.  

In contrast, comparative evaluation is a practice to skip the adjectival ratings step completely.

Share this post


Link to post
Share on other sites

I love the fact that a comparative evaluation used in phase one of a multi-phase source selection has serious mathematics hiding just below the surface. 

Bottom Line: Don't try comparative evaluations if you expect to get a lot of proposals.  Its a much more challenging math problem that it may first appear.  

Its a special case of what I think its called a comparison sorting algorithm (note: I like math, but am no mathematician) that is known in math and computer science.   When the numbers are low, just basic math required to figure out the number of comparisons necessary.  But as the number increases, this problem gets exponentially more difficult.  Quickly goes from intuitive to computationally hard to God-level AI needed.

Number To Be Compared & Sorted

Number of Highest Rated You Need To Know

Who Can Do The Math To Figure Out How Many Comparisons Are Needed

3

1

Child

5

2

Educated Adult

8

2

Well-Educated Adult

15

5

Computer Scientist

30

5

PhD in Combinatorics

100

10

Von Neumann + Quantum Supercomputer

1000

50

Deep Thought

Share this post


Link to post
Share on other sites
15 minutes ago, Ibn Battuta said:

So, for example, if evaluating past performance you would make findings about each offeror's past and then compare their pasts directly without first converting your findings to ratings and then comparing offerors on the basis of the ratings. Is that the idea?

Yes.  My only caution would be not to put too much formality on the word "findings."

For example, for the past performance factor in a selection process, we might say that "A" is better than "B" because the CPARS history for "A" shows 100 satisfactory or better ratings and zero marginal or unacceptable ratings, of which 80 for were similar services, while the history for "B" shows zero satisfactory ratings and 14 marginal or unsatisfactory ratings, of which all 14 were for similar services.

I believe comparative evaluations can be done subjectively, without point scores, mathematics, or algorithms.

Edited by ji20874
to add ending sentence

Share this post


Link to post
Share on other sites

I would aver that I decided subjectively that "A" is better than "B" based on the information available to me. 

Then, assuming the other factors were performance, delivery, and price, and "A" was better for past performance and price and "B" was better for performance and delivery, then I would do a subjective tradeoff to select the successful firm.

Edited by ji20874
to add "based on the information available to me" and the second paragraph

Share this post


Link to post
Share on other sites

I know we all know where to find this but as this thread wraps itself around the axle I figured it was good to post it as a reminder.  No comparison just "Fair Opportunity" to be considered!  Or in other words do not give someone an unfair advantage to be considered but once you have all to be considered then just figure out who you are going to award to (consider them), document why you are awarding to one over the others and do it.  In my view the discussioin complicates it just like the OP's higher ups do.   

(Emphasis added)

FAR 16.505  "...(b) Orders under multiple-award contracts—

(1) Fair opportunity.

(i) The contracting officer must provide each awardee a fair opportunity to be considered for each order exceeding $3,500 issued under multiple delivery-order contracts or multiple task-order contracts, except as provided for in paragraph (b)(2) of this section.

                (ii) The contracting officer may exercise broad discretion in developing appropriate order placement procedures. The contracting officer should keep submission requirements to a minimum. Contracting officers may use streamlined procedures, including oral presentations. If the order does not exceed the simplified acquisition threshold, the contracting officer need not contact each of the multiple awardees under the contract before selecting an order awardee if the contracting officer has information available to ensure that each awardee is provided a fair opportunity to be considered for each order. The competition requirements in part  6 and the policies in subpart  15.3 do not apply to the ordering process. However, the contracting officer must—

                     (A) Develop placement procedures that will provide each awardee a fair opportunity to be considered for each order and that reflect the requirement and other aspects of the contracting environment;

                     (B) Not use any method (such as allocation or designation of any preferred awardee) that would not result in fair consideration being given to all awardees prior to placing each order;

                     (C) Tailor the procedures to each acquisition;

                     (D) Include the procedures in the solicitation and the contract; and

                     (E) Consider price or cost under each order as one of the factors in the selection decision.

 

                (iii) Orders exceeding the simplified acquisition threshold.

(A) Each order exceeding the simplified acquisition threshold shall be placed on a competitive basis in accordance with paragraph (b)(1)(iii)(B) of this section, unless supported by a written determination that one of the circumstances described at 16.505(b)(2)(i) applies to the order and the requirement is waived on the basis of a justification that is prepared in accordance with 16.505(b)(2)(ii)(B);

                     (B) The contracting officer shall—

                          (1) Provide a fair notice of the intent to make a purchase, including a clear description of the supplies to be delivered or the services to be performed and the basis upon which the selection will be made to all contractors offering the required supplies or services under the multiple-award contract; and

                          (2) Afford all contractors responding to the notice a fair opportunity to submit an offer and have that offer fairly considered.

..."

A read here is probably in order as well http://www.wifcon.com/pd16_505b.htm

Share this post


Link to post
Share on other sites

Carl,

FAR 16.505(b)(1)(iv) speaks of offers and factors and evaluation and relative order and best value — how does one simply consider and then choose?  Can you sketch out a strawman of the process you envision?  If it is simpler and better than my practice, I’ll adopt it.

Or, are you stopping short of 16.505(b)(1)(iv) and only covering up to (iii)?  That’s okay if that is your thought process, as (iii) is different ftom (iv) (for a reason, right?) and (ii) is even different from (iii).  

Share this post


Link to post
Share on other sites

ibn battuta (1),

  • I didn't want to put too much emphasis on the word "facts," because the process works just as well for facts as it does with other information.  You spoke of a finding based on facts, and it could be an observation based on other information.  The comparative evaluation process doesn't require facts -- it requires information, and that information sometimes will be factual and sometimes might not be, and it may be judged objectively or subjectively as appropriate.  I wouldn't want a simple-minded reader to go through this thread and think that "facts" are required for a comparative evaluation and a selection (the formal 15.3 source selection process also doesn't require facts).  So rather than saying that one vendor is better than another based on facts, I prefer to say that I subjectively decided one vendor was better than another based on the information available to me.

don,

  • You are not contributing to this thread's discussion.

Ibn battuta (2),

  • The "notice" versus "solicitation" discussion was another thread, not this one.  Many do use "solicitation," but some use "notice" consistent with the FAR usage.  But I agree that the fair opportunity process was initially intended to be a matter of post-award administration, and we* have turned it into a pre-award process.
    *not you and me personally, but our community generally

guardian (original poster),

  • You can be as innovative as you and your organizational culture will allow.  Many organizations are getting better at staying true to the FAR and leaving 15.3 methodology out of their ordering situations.  Professional dialogue is important.  Maybe you can convince your superiors and reviewers to let you do a comparative evaluation on a fair opportunity below the protest threshold?  Your success there might help start a thaw in your organization.

Share this post


Link to post
Share on other sites

Facts are nice, and they should be used wherever they are available.  But often, we have to make acquisition decisions based on information that may fall short of being factual.  I'm okay with that.

When one person quotes someone else, the quoter should take care not to faithfully convey the essential thought and to avoid skewing the quoted person’s thought.  I wrote, “The comparative evaluation process doesn't require facts -- it requires information, and that information sometimes will be factual and sometimes might not be...” (emphasis added on my text that you dropped in your quote, but that essential to faithfully convey my thought).

Share this post


Link to post
Share on other sites

Apparently, there may be more than one definition of "fact."

Do you have anything to contribute to help the original poster with comparative evaluation?

Share this post


Link to post
Share on other sites
3 hours ago, Ibn Battuta said:

 

That was written almost 14 years ago, and it apparently hasn't much good given some of the threads at Wifcon.

A group of the people at the company I work for was tasked to facilitate and oversee proposal evaluations.  We did training and then watched evaluators.  We kept telling them to stop scoring first and then doing narrative supporting the score.  We could see evaluators frustrations because we kept stopping them - “focus on identifying attributes first.”  They complained that’s not the way they do it.  One morning we showed up and the door was locked.   We called the CO and he promised to straighten it out.  The next morning the building was dark and the evaluators moved.  Our engagement ended. 

Share this post


Link to post
Share on other sites
1 hour ago, ji20874 said:

Apparently, there may be more than one definition of "fact."

Yes, I've heard of alternative facts. Sometimes people contribute them in this forum.

Share this post


Link to post
Share on other sites

@ji20874,

Might an evaluators confidence rating be an example of what you’re talking about?  I can see that subjectivity plays into assessing how well an offeror understands the requirement, the degree of soundness in a technical approach, and the likelihood of success.  
 

Share this post


Link to post
Share on other sites
10 hours ago, ji20874 said:

Can you sketch out a strawman of the process you envision?

– My sole attempt, take it or leave it. 

1)Multiple award IDIQ – Value exceeds the SAT.  For parent contracts you pick any evaluation factors to place contractors with a parent contract that you want but the factors are to embrace the imperatives of FAR 15.304.

2)Fair Opportunity – The FAR guiding principles of 16.505 are applied but nothing from the FAR rules the contract imperatives rule.

3)Proposed Fair Opportunity Placement Procedures Clause –

Each contractor will be given a fair notice of the need for a task order under the contract.  The notice shall contain the description of work, a response period of not less than 10 days with the length of response period beyond 10 days determined per task order. 

Factors for selection will be stated in each notice.  Factors included in every notice will be - price, past performance on previous task orders including the sub factors of quality of work, timeliness of work and cost control (if applicable), and other orders currently awarded to the contractor.  Additional evaluation factors and sub factors may be included by the government as they relate to the specific work anticipated.  All factors and associated sub factors will be stated in their relative order of importance in the notice.  Oral presentations may be requested as appropriate at the discretion of the Government.

Contractors submitting a response to a notice of fair opportunity shall provide pricing or cost information along with information that addresses the other evaluation factors.  The information provided for price/cost and other factors shall be that minimally necessary for evaluation.  The Government reserves the right to consider any information available to the Contracting Officer in addition to that provided by the contractor.

Selection for award will be on the basis of a basic comparison of each response received to the other responses received with regard to the factors stated in the notice.  A debriefing will be afforded on all award decisions that are more than $5,000,000.

4) The internal process?  Do a comparison any old way that you want but document it in accordance with

“16.505 (b)(7) Decision documentation for orders.

(i) The contracting officer shall document in the contract file the rationale for placement and price of each order, including the basis for award and the rationale for any tradeoffs among cost or price and non-cost considerations in making the award decision. This documentation need not quantify the tradeoffs that led to the decision.”

5) Any old way you want?– First the caveats are - exercise broad discretion, streamlined procedures, policies in subpart  15.3 do not apply and formal evaluation plans or scoring is not required.  While the following may border on a “formal” effort  consider this approach from a March 16, 2017 post by Vern Edwards regarding evaluation of quotes understanding that we are talking here responses to fair opportunity notices so appropriately edited to fit the  fair opportunity construct..….

“Assume that we get four quotes, from companies A, B, C. and D. We'll evaluate using a method that is sometimes called "pairwise comparisons" and we'll assume transitivity.

First, we'll evaluate for experience. We'll compare A's description of its experience to B's and decide, subjectively, which has the better experience by taking note of asserted facts, identifying differences, determining their significance to us, and documenting our conclusions. Let's say we decide that A is better than B. We'll then compare A to C. This time we think C is better than A. Since C is better than A and A is better than B, we assume that C is also better than B. We'll then compare C to D. We decide that D is better than C. Since D is better than C and C is better than A and B, D is also beter than A and B. So D is best on experience. Since there were four offerors, and since D is best, we'll give D four points. Since C is better than A and B we'll give C three points. Since A is better than B we'll give A two points. Finally, we'll give B one point.

Experience: D = 4, C = 3, A = 2. and B = 1.

Second, we'll evaluate for past performance, using the same procedure as we did for experience. This time the result is as follows:

Past performance: D = 4, A = 3, B = 2, and C = 1.

Third, we now compare the four quoters' prices. This is easy. The lowest price gets four points and the highest gets 1 point. The result is:

Price: B = 4, A = 3, C = 2, and D = 1.

Fourth, we total the points.

A = 8, B = 7, C = 6, D - 9.

Fifth, D is best overall, so we award to D.

We evaluated on the basis of direct comparisons, based on subjective assessments and without standards. We made no tradeoffs, so we have none to document.

That's one way to do it. There are other ways. Is it a good way to do it? That depends on what you're buying and on your notions of value.

The hardest part is writing up the rationale for your subjective assessments. It takes some thinking and word-smithing. If you don't know how to do that, then don't try this method.

Do you want to weight the factors differently? If so, assign each a decimal weight, such as 0.5 for experience, 0.4 for past performance, and 0.1 for price, so that they add up to 1.0. Then multiply the points by the weights before totaling the points.

The fact that you and your colleagues have not heard of this approach should not be surprising. Government agencies make millions of simplified acquisitions every single fiscal year. Millions. No one knows, much less keeps track of, how they do them. They're rarely protested or the subject of explanatory articles. You've worked for the government for five and one-half years and in two agencies. In the words of BoB Dylan, your experience "is limited and underfed." Your lack of experience, knowledge, or passing familiarity with the method does not make the method nontraditional.”

Are there any real efficiencies? Compared with what other methods? What I described might take a skilled buyer no more than a couple of days so to complete, depending on what you're buying and assuming that the buyer can write and won't have to go to the boss to ask questions every hour or so. That's longer than a price-only approach but shorter than a tradeoff process type approach. Forget efficiency if you're going to create an "evaluation board."

Risk? There is no appreciable risk for any but the brain dead. Less risk than a tradeoff approach."

 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...