Jump to content

Latest & Greatest Evaluation Factors


Recommended Posts

All, 

Was wondering what you'll have been seeing as the currently trending evaluation factors in solicitations? I know sometimes people get really creative. 

I know FAR 15 gives us some very general ones: 

"FAR 15.304 (c):

(1) Price or cost to the Government shall be evaluated in every source selection (10 U.S.C.2305(a)(3)(A)(ii) and 41 U.S.C.3306(c)(1)(B)) (also see part 36 for architect-engineer contracts);

(2) The quality of the product or service shall be addressed in every source selection through consideration of one or more non-cost evaluation factors such as past performance, compliance with solicitation requirements, technical excellence, management capability, personnel qualifications, and prior experience (10 U.S.C.2305(a)(3)(A)(i) and 41 U.S.C.3306(c)(1)(A)); and

(3) (i) Past performance, except as set forth in paragraph (c)(3)(iii) of this section, shall be evaluated in all source selections for negotiated competitive acquisitions expected to exceed the simplified acquisition threshold." (EA)

 Any and all examples will be greatly appreciated. Trying to put together a list of what our fellow professionals are using. 

Thanks in advance/CO

Link to comment
Share on other sites

Guest PepeTheFrog

Highest Technically Rated Fair and Reasonable Price (HTRFRP) is both innovative and useful

if somebody can smash CICA's nonsense they deserve several medals

Link to comment
Share on other sites

  • For a agile software development RFQ my instructions are for offerors to answer our questions, rather than have them write a so-called technical/management approach.  Questions (slightly modified here) are:

1.       What is your management process for this order, including working with the Product Manager, COR, and End Users to capture and size user stories, prioritize, and work-off the product backlog?

3.       What is your definition of done?

5.       What is your anticipated velocity?

6.       What is your quality management plan for [something]

8.       How do you plan on allocating work between the teams, particularly between [X] & [Y]?

Page Limit: 5 Pages.

Link to comment
Share on other sites

General gives an excellent example of structuring the evaluation around key attributes of the specific acquisition.  In this case, he cites questions offerors must answer.  However the evaluation factors likely must need to reflect consideration of that information.

Market research at the strategic level is important for nearly all acquisitions.  One needs to learn what makes the best performers that way and what are important to the governments specific needs.  That applies to whether one is buying equipment, services, or major systems.  Then you devise factor that differentiate among offerors on basis of best fulfilling your program need.  Tailoring instructions to offerors like General suggested helps in some cases too.

Link to comment
Share on other sites

  • 2 weeks later...
On 8/2/2019 at 10:11 AM, General.Zhukov said:
  • For a agile software development RFQ my instructions are for offerors to answer our questions, rather than have them write a so-called technical/management approach.  Questions (slightly modified here) are:

1.       What is your management process for this order, including working with the Product Manager, COR, and End Users to capture and size user stories, prioritize, and work-off the product backlog?

3.       What is your definition of done?

5.       What is your anticipated velocity?

6.       What is your quality management plan for [something]

8.       How do you plan on allocating work between the teams, particularly between [X] & [Y]?

Page Limit: 5 Pages.

Better, but you are still requesting information with little predictive value. Unless you are asking for promises, an offeror's statements about the future will be written to show them in the best possible light--as opposed to providing the most accurate depiction of the future. This is improved crawling. 

Link to comment
Share on other sites

2 hours ago, Don Mansfield said:

Better, but you are still requesting information with little predictive value. Unless you are asking for promises, an offeror's statements about the future will be written to show them in the best possible light--as opposed to providing the most accurate depiction of the future. This is improved crawling. 

Don,

These questions are dated but still are good as examples.  They don’t ask about the future; they asked about methodology in performance.  The government awarded many Agile development contracts to companies that had no Agile experience or expertise but said they could do the work.  They generally failed.  Then the government got smart and questions like this challenged companies.  I saw oral presentations when an offerors so called Agile experts started stuttering and didn’t know how to respond to a question about what is done? 

These challenge capabilities and probably better asked when companies don’t have chances to look up answers.  But the gist is good - see what they really know and what they have done. Easy to verify with past performance checks.

Link to comment
Share on other sites

On 8/2/2019 at 7:10 AM, PepeTheFrog said:

Highest Technically Rated Fair and Reasonable Price (HTRFRP) is both innovative and useful

if somebody can smash CICA's nonsense they deserve several medals

Other than GSA's success story highlighted in the  May 2016 Nash & Cibinic Report, has anyone had experience, success, or lessons learned using this, or similar methodology?  The strategy being considered for this multiple award IDIQ engineering services requirement is to conduct an evaluation with the exclusion of price as an evaluation factor.  Establishing factors and standards for technical acceptability has been interesting to say the least.  

Link to comment
Share on other sites

On 8/15/2019 at 8:07 PM, formerfed said:

Better, but you are still requesting information with little predictive value.

I know.  Its incremental improvement in the face of constraints.  There is a whole team of folks who need to approve of evaluation methodology.  Evaluators who have to understand what they are doing.  I am no expert in the subject, nor is the program office particularly knowledgeable about contracting.  Institutional risk-aversion (not necessarily a bad thing).  Etc.

I would actually really be interested if there is any research about the connection between source selection methodology and contract performance. Such as - what information can we get pre-award that predicts good performance.   I only know of folksy wisdom about past performance and competition.  I am joking, but not really - is there is any actual high-quality research on the subject?

 

Link to comment
Share on other sites

I was part of a group trying to answer that question.  We found nothing conclusive on source selection methodology and performance.    We did receive lots of anecdotal and subjective information though.  The biggest hurdle about demonstrating a connection is time passing from award to concluding performance was successful or not.  That often covers years.  Many people involved with contract performance weren’t involved in source selection or weren’t aware of the process.

One interesting point expressed by many involved in successful contracts was the need for continuous collaboration and open communication with the contractor.   Their suggestion was use that as a factor in assessing past performance.  They agreed it’s necessary to go past CPARS for this and stressed frank dialogue with major customers, both commercial and government, to get at the issue.

Link to comment
Share on other sites

On ‎8‎/‎19‎/‎2019 at 1:29 PM, General.Zhukov said:

I know.  Its incremental improvement in the face of constraints.  There is a whole team of folks who need to approve of evaluation methodology.  Evaluators who have to understand what they are doing.  I am no expert in the subject, nor is the program office particularly knowledgeable about contracting.  Institutional risk-aversion (not necessarily a bad thing).  Etc.

I would actually really be interested if there is any research about the connection between source selection methodology and contract performance. Such as - what information can we get pre-award that predicts good performance.   I only know of folksy wisdom about past performance and competition.  I am joking, but not really - is there is any actual high-quality research on the subject?

 

A prediction of good performance is generally based on records of performance data that is provided or obtained in the past performance evaluation.  These records don't always paint an accurate picture of how the contractor actually performed but we use the information as best we can.  I am trying to engage with requirement owners and engineering community to think in terms of technical capability and experience, and then, once we have done that, determine what we can ask for that will demonstrate whether a proposal meets those standards or not.   It sounds so simple in theory but not so much in practice. 

Link to comment
Share on other sites

• Recent relevant experience, with a past performance rating including names and contact info for customer references for those specific govt or no-govt projects . EDIT: Use a standardized format for the experience submission.

•. Proposed  key personnel (define the positions) with a contract clause that  makes the qualifications of those proposed and evaluated for award the new minimum for proposed substitutions if they exceed any RFP minimums and substitutions. Substitutions are subject to government approval . If you propose the A team, you must provide an A-team,  not a B or C-team as substitutes. EDIT: use a standardized format for each key personnel submission. 

• same submission and evaluation criteria for your key subcontractors (define what is the key work, if subcontracted). If much or most of the key aspects of  work is subcontracted, you don’t want to simply rely upon prime qualifications and experience. EDIT: Use a standardized format for each proposed key sub submission.

Edited by joel hoffman
Link to comment
Share on other sites

Past performance is an excellent factor but only a handful of offices really leverage it the way they could.  Most in the government are afraid.  They only want to use CPARS and email questionnaires sent to just offerer  provided references.

To really benefit from past performance, one needs to talk with relevant customers of offerers.  By talk, I mean using a phone or visiting customers.  By customers, I mean anyone that did business with the offer whose work is deemed significant in terms of size, scope and complexity, including commercial sources.  It also means going way beyond just using references provided by the offeror.  

There’s also a notion that you have to ask the same questions of every source, which is crazy.  

Doing it this way is closer to the way you might seek a contractor to remodel your house. 

 

Link to comment
Share on other sites

1 hour ago, formerfed said:

Past performance is an excellent factor but only a handful of offices really leverage it the way they could.  Most in the government are afraid.  They only want to use CPARS and email questionnaires sent to just offerer  provided references.

To really benefit from past performance, one needs to talk with relevant customers of offerers.  By talk, I mean using a phone or visiting customers.  By customers, I mean anyone that did business with the offer whose work is deemed significant in terms of size, scope and complexity, including commercial sources.  It also means going way beyond just using references provided by the offeror.  

There’s also a notion that you have to ask the same questions of every source, which is crazy.  

Doing it this way is closer to the way you might seek a contractor to remodel your house. 

 

Amen and Amen!!!! I HATE those written reference questionnaires. Bull crap. Call and speak with the references!  I got much better info by speaking directly to the references.  I will clarify my above post...

Unfortunately, it means that government employees actually have to TALK to somebody- oh NO!!! And take notes - Double OH NO NO!!! 

Look at it from the references’ perspective. I hated getting a questionnaire to fill out and return - especially when contractors repeatedly  cited me as a reference on multiple projects and/or for multiple source selections. 

I also stated that we reserved the right to call references to verify a performance rating. If we were already familiar with a project, there was no need to keep bothering the references to verify or repeat what they said previously. 

Link to comment
Share on other sites

16 hours ago, joel hoffman said:

Proposed  key personnel (define the positions) with a contract clause that  makes the qualifications of those proposed and evaluated for award the new minimum for proposed substitutions if they exceed any RFP minimums and substitutions. Substitutions are subject to government approval . If you propose the A team, you must provide an A-team,  not a B or C-team as substitutes. EDIT: use a standardized format for each key personnel submission. 

I am definitely stealing this idea.

Link to comment
Share on other sites

I saw a state solicitation for construction recently where price was evaluated using a cost and time method. The contractor would propose their cost for the project and then the number of days from NTP to completion. The number of days was multiplied by $X and added to the cost for the total evaluated price. The awarded price was the cost portion but the evaluated price included time also as a price related factor. 

I suppose the idea is you could pay more for the project but get it completed faster. I don’t know if it’s new, but it was the first time I had seen it.  

Link to comment
Share on other sites

2 hours ago, Deaner said:

I saw a state solicitation for construction recently where price was evaluated using a cost and time method. The contractor would propose their cost for the project and then the number of days from NTP to completion. The number of days was multiplied by $X and added to the cost for the total evaluated price. The awarded price was the cost portion but the evaluated price included time also as a price related factor. 

That's interesting!  Do you remember if there was a liquidated damages clause?  If so, was it the same amount as the multiplier?  Seems like that would be internally consistent . . .

Also, which state?  I'm working under state code now and it doesn't seem very flexible about that kind of thing, but maybe it's just Texas . . .

Link to comment
Share on other sites

3 hours ago, apsofacto said:

Do you remember if there was a liquidated damages clause?

I don’t remember what solicitation it was. I tried to google it and find it again on FBO but could not find it.

I did find what appears to be commonly called A+B Bidding, which describes how the solicitation I saw evaluated price. Apparently federal and state DOTs use it.

Top of PDF sheet 2 describes what it is. This document is dated in 2013, so I guess it’s not that new.

http://onlinemanuals.txdot.gov/txdotmanuals/glo/glo.pdf

From this document the multiplier (part b) is the Road user cost per day. It appears to be a value assigned for motorist delays based on wages, excess fuel, etc. as a result of road construction projects. I didn’t look at the liquidated damages clause in the solicitation I had seen previously. My guess would be if you assigned a cost per day during the solicitation, that would be the cost per day for liquidated damages. I don’t know enough about road user costs, but it appears to be a cost to the public and not a damage to the government, so I don’t know that the part B multiple would be appropriate for liquidated damages. But again I don’t know enough about the method or road user cost calculations. I just saw a solicitation that apparently was using it. 

Link to comment
Share on other sites

9 hours ago, General.Zhukov said:

I am definitely stealing this idea.

Here is a clause that USACE has been using for design-build solicitations and contracts:

“Key Personnel, Subcontractors and Outside Associates or Consultants (MAY 2006)

In connection with this contract, any in-house key personnel, key subcontractors, and outside associates or consultants will be limited to individuals or firms that were specifically identified in the Contractor's accepted proposal. The Contractor shall obtain the Contracting Officer's written consent before making any substitution for these designated in-house personnel, subcontractors, associates, or consultants. If the Contractor proposes a substitution, it shall submit the same type of information that was submitted in the accepted proposal to the Contracting Officer for evaluation and approval. The level of qualifications and experience submitted in the accepted proposal or that required by the Solicitation, whichever is greater, is the minimum standard for any substitution.
(End of clause)”

The clause only applies to those specific key personnel positions or key trades,  if to be subcontracted, that the government identified to be submitted in the proposals for evaluation. 

Somewhere I have the proposal submission  requirements and evaluation criteria with the forms, etc

Link to comment
Share on other sites

20 hours ago, Deaner said:

My guess would be if you assigned a cost per day during the solicitation, that would be the cost per day for liquidated damages. I don’t know enough about road user costs, but it appears to be a cost to the public and not a damage to the government, so I don’t know that the part B multiple would be appropriate for liquidated damages.

That's what I thought too.  That passage states also:

Quote

A disincentive provision, that assesses road user costs, is incorporated into the contract to discourage the contractor from overrunning the time "bid" for the project. In addition, an incentive provision should be included to reward the contractor if the work is completed earlier than the time "bid."

I'm actually in Texas.  So deep in Texas the BBQ place near my house gives you a discount if you open carry.  I didn't think we were allowed to apply a penalty in that fashion.  That's why I was assuming the multiplier and the LD amount would be the same.

Thanks for sharing all that- it was interesting!

Link to comment
Share on other sites

2 hours ago, apsofacto said:

That's what I thought too.  That passage states also:

I'm actually in Texas.  So deep in Texas the BBQ place near my house gives you a discount if you open carry.  I didn't think we were allowed to apply a penalty in that fashion.  That's why I was assuming the multiplier and the LD amount would be the same.

Thanks for sharing all that- it was interesting!

I did some research and it seems to be really simple and innovative at the same time. Contractor comes through they get the daily rate as a bonus, on-time completion means the government wins/contractor is whole and late completion means they pay what it cost the government to wait. 

I really like it as a tool, but It would be hard to use in my current line of work for most instances (services). Hard to measure by a day or event. It has been a long day so maybe I am not thinking "Big Picture."

Any ideas group?

Link to comment
Share on other sites

High priority Construction contracts where time is of the essence.

 For department of defense construction it is not allowed by law to pay more for an earlier completion date without certain approvals (it is called “expediting”).

Link to comment
Share on other sites

Jumping back to contractually locking in certain positions (what is proposed must be delivered unless an equivalent substitute is approved), the Government must be reasonable in applying use.  It’s a good idea, especially when quality of personnel is a major part of source selection.  But in many instances, the Government acts unreasonably by demanding the proposed people get delivered immediately and without exception.

Companies propose people and assume a contract award occurs in 4-6 weeks.  Often that turns into 4-6 months.  Reasons are many - multiple solicitation amendments, lots of Q&A’s, weather, evaluation team not ready, evaluation delayed by lack of time commitments by members, unplanned need for clarification or discussions, availability of funding, and so on.  

Companies don’t keep proposed staff just sitting around waiting for the government to make an award. Their employees must be billable as much as possible.  Then there’s smaller companies that do an extremely good job and in demand.  The problem is as they grow, it is progressively more difficult to recruit quality employees - the pool relative to size gets smaller.  So people proposed often get assigned the award in hand.  I’ve heard stories of small companies proposing the same employees a half dozen times.  

If this type of clause gets used, the CO needs to be fair and allow time for replacements if award gets delayed.  A good practice is communicating with offerors to let them know status and get companies to verify employees are still available.

Link to comment
Share on other sites

Formerfed, Agreed and that is how we taught the clause.  

Evaluation Factors are generally meant to be discriminators in competitive source selections.  

Therefore, while recognizing that specific individuals  might not be available for the length of the contract or even at the start, especially after an extended period before award, what was proposed becomes the new minimum qualifications if and when it exceeds the minimum solicitation requirement.  Like I said above, don’t propose the A-Team to win the job, then furnish the B or C team. Commit to provide the quality of personnel that you offered during the competition.  

_______________________

With key subcontractors, a proposed substitution requires more scrutiny in order to discourage  bait and switch and/or bid shopping practices.  Anti-bid shopping concerns and, thus measures for discouraging bid shopping are not uncommon in public contracting. 

If a proposed sub is being replaced, we want to have some control over bid shopping.  It may be prudent and appropriate to require consideration or a credit . One could assume that the prime’s  price was based upon the proposed subcontractors’  prices. Some prime contractors have a reputation for bid shopping. 

It also is intended to look out for subcontractor’s interests. In construction and design-build , key subs carry their weight and are critical members of the D-B team. We don’t want them to have to cut corners to stay on the D-B team that won the contract. 

So we don’t like it when a contractor squeezes the A-team partners into becoming B-team performers or replaces the proposed A-team partners with B-Team firms to make an extra buck. 

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...