Jump to content

Adjectival Rating and Awards Question


Salus

Recommended Posts

Good Day All,

A rather grueling and rather long award process just concluded after three separate 3-month extensions to the award process. We were one of three teams competing for the contract. We did not win. The contract was for technical support under a labor hour contract. The solicitation stated that the technical and management components of the project, followed by transition plan, and past performance were significantly more important than price. No negotiations were held.

During our debrief, we found out that our proposal had 7 strengths for the tech/mgt approach, and 6 of those were specific to the 4 elements described in the evaluation factors. We had one strength for the transition plan. We had no weaknesses or deficiencies. The technical reviewers specifically included that the companies and key personnel for the proposal were well qualified for each of their roles.

Our team and the winning team both received acceptable on all rating categories. We assume the third team received acceptable as well, because if they had received good or better, they should have won. We were told in the debrief, because all teams received the same rating, it was awarded based on cost alone (we were more expensive). 

Our question comes from this: the adjectival evaluation guidance ranged from Outstanding to Unsatisfactory. However, to achieve outstanding or good, a proposal had to receive a 'significant strength' which feels a little amorphous to start (stated as significant cost savings or efficiency for the government, and many regular strengths do not add up to a significant strength). However, based on the adjectival criteria, our 8 strengths would be rated exactly the same as any team who had few or no strengths so long as strengths and weaknesses offset. But for the specific requirement of 'significant strengths', our technical factor should have been outstanding (strengths significantly outweigh weaknesses) or at the very least good (strengths outweigh weaknesses).

There are not many ways to achieve significant cost savings or efficiency on a labor hour contract where the function of that contract is to put people at their facility for a specific number of hours. And specifically, one of the major concerns included as an evaluation factor was recruitment and retention, which for the most part necessitates paying market wages. Thus cutting costs to offer very low prices is antithetical towards this goal. So for this specific proposal, it feels very much like there wasn't a way for any vendor to achieve a good or outstanding rating based solely on the requirement for significant strength (you can only cut costs by subverting one of the technical factors). This effectively means that every company ends up with an acceptable rating, and the proposals are then just evaluated on a cost basis. 

Is there a way to avoid this situation? This was for a fairly large award, and to spend quite a bit of effort to develop a proposal that had very little room for improvement (there was no feedback on how we could have improved our proposal, and we received detailed positive feedback on several elements) only to have it awarded as a lowest cost proposal is very frustrating. A lowest cost award is a much different animal and much less expensive to develop, so if that is the ultimate goal, then we would much rather that be clear up front.

All of this comes with the caveat that obviously we don't know how the winning teams strengths and weaknesses broke out, we only know that we were told it was awarded based on cost.

I appreciate any input or advice!

Link to comment
Share on other sites

On 6/22/2021 at 7:05 PM, Salus said:

We were told in the debrief, because all teams received the same rating, it was awarded based on cost alone (we were more expensive).

If the agency promised to do a tradeoff using adjectival ratings, but mechanically made the selection decision on the ratings, they may have erred and you might have a valid basis for a protest.  You. might be interested in the article at https://smallgovcon.com/gaobidprotests/gao-agencies-cant-blindly-rely-on-adjectival-ratings-to-make-award-decisions/.

Link to comment
Share on other sites

Wow, nine months of extensions alone for a competition with only three offers and NO discussions (!).  Hmmmm…

Link to comment
Share on other sites

Thanks JI.

We are trying to set up a follow up meeting with the CO to make sure there were no miscommunications during the debrief on how the process was implemented. We are obviously not happy with receiving an acceptable rating, but in theory that could still be okay if they considered the relative merits of the proposals. Unfortunately, we can't get any additional information about the relative strengths and weakness of the winning team, so there is no way for us to tell if the winning team also had a bunch of strengths or if the agency did just weight the ratings the same and then make a lowest cost decision.

We are also pretty sure that even after the extensions, they were still not quite ready to make the award as they announced the week before that they intended to extend the existing contract by a month, but that didn't work out and they made the award 2 business days before the start of the contract.

Sometimes things just don't go smoothly at all, and this definitely seems like one of those times!

Link to comment
Share on other sites

Hi Vern,

We did not use a professional proposal management consultant for this project. Our team is just on the verge of being able to regularly incorporate that kind of extra support (which this contract would have gone a long way to supporting). However, we have aggressively pursued as much education on federal contracting as we can through PTACs, and the Virginia PTAC specifically (they have provided a lot of high quality trainings), and have worked hard to incorporate any feedback that we do receive during our debriefs.

I am sure that would be helpful for all of our proposals in improving the quality. However, in this case, I don't know that it would have helped. We didn't have the adjectival rating definitions until after our debrief (something we now know to ask for during the question phase of the process), and they used a different set of definitions than the other half of their acquisitions division. So we weren't able to address the language in their definitions, and we wrote towards the factors included in the proposal and our understanding of the other set of definitions that we did have access to from that agency. Even still, we got strengths that were described as innovative approaches directly applicable to the criteria contained in the technical/management factor, and they were still not considered significant strengths, even though by the definitions used for this solicitation, they could have been considered as such.

We also specifically requested any feedback on how the proposal could have been improved and they were unable to provide any. They were also unable to provide any example of anything that would qualify as a significant strength. I know it can be hard to come up with answers on the fly during a live conversation, so some of the information may just come from that live format and it being really hard to communicate everything effectively during a short period of time.

Since we don't know what instructions are provided to the technical reviewers, or how we actually compare in relative strengths and weaknesses, we are totally in the dark as to whether they did consider relative strengths or not. All we have is the statement that they made a lowest cost award during our debrief, which might have just been an attempt to provide a concise answer and they actually did do a full consideration of the relative merits within the rating category. It doesn't help that the CO hasn't responded to any questions since the debrief!

So at the moment, we are left without any clarifying information to try to make a decision as to seriously considering a protest or just walking away from a large contract that we are pretty confident that we were going to provide the best service on (but not the lowest price...). 

Link to comment
Share on other sites

@Salus

Some thoughts:

I don't know that a proposal consultant would have made a difference, but it might have helped. You might have gotten that sought-after "significant strength" simply through better presentation of the same material. That's mainly what the consultants are about.

The government's various definitions of things like "strength" and "significant strength" are exceedingly vague, and likely deliberately so. I wouldn't rely on them as the basis for any kind of challenge. The best thing you could hope for in a protest is that your attorneys find evidence of disparate treatment in that regard.

As for feedback on how the proposal could have been improved, if by that you mean what you could have proposed that would have resulted in a significant strength, the agency could reasonably say that they are not in the business of proposing solutions but of seeking them, and that if they had had a better solution in mind they would have specified it for everyone to respond to. The same with a "significant strength." They could say that it's like art—they might know it when they see it, but can't describe it in advance. The agency can't tell you much about what the other offerors proposed, since that might violate laws against disclosure of proprietary information.

You could, of course, file a protest in order to obtain more information. But that is a costly way to research the market. Your chances of winning, all things considered, are about 1 in 5.

Better luck next time, but better not to trust to luck, fairness, genius, or generosity. Take a close look at the presentation you made in your proposal.

Link to comment
Share on other sites

Hard to tell in this instance what the issues are from the government side.  Somethings evaluators get drafted to do that job.  They might be inexperienced and just follow the process laid out to them.  Often without some help and people looking over their shoulder, all offers look similar.  They don’t see differentiators because they don’t know the program.  So all offerors get similar ratings.  

Two things I learned about proposals.  One is if you don’t have some information and background about the requirement and what the government is looking for before the RFP is released, your chances of winning are slim.  What program offices really want versus the RFP wording literally says can be different.  Second, and this is what Vern mentioned, a professional proposal writer can turn responses into hard hitting and impressive messages that catch evaluators attention.

Link to comment
Share on other sites

What we have here is a disappointed offeror who thinks his company maybe should have gotten a higher rating than it did.

On 6/22/2021 at 4:05 PM, Salus said:

[B]ased on the adjectival criteria, our 8 strengths would be rated exactly the same as any team who had few or no strengths so long as strengths and weaknesses offset. But for the specific requirement of 'significant strengths', our technical factor should have been outstanding (strengths significantly outweigh weaknesses) or at the very least good (strengths outweigh weaknesses).

Uhhh, okay.

Absent an equation of some kind, the determination of whether something is merely acceptable, a strength, or a significant strength is purely judgmental. The only way I can think of to successfully contest such a judgment is to show that the agency was inconsistent in its application of those ratings between or among various offerors.

 

Link to comment
Share on other sites

As differences between non-price factor ratings and underlying distinctions become non-discriminators, the difference between pricing becomes more important. The government must ultimately decide whether small technical differences would warrant paying the difference in pricing.

Of course, this assumes that there were no meaningful errors or inconsistencies in the technical evaluation(s). 

Can you reveal roughly the price differences between the offerors, at least in terms of percentage?

 

Link to comment
Share on other sites

I want to reiterate that I don't necessarily think we should have won. We don't know how the winning team was evaluated for comparison, and it is totally possible they got many strengths and were cheaper. We just found it very strange that we could have several strengths that the reviewers themselves linked as innovative approaches addressing the components of the most important rating factor and we still received an acceptable. Then we were told that it was awarded on a lowest cost basis because all ratings were equal. This combined with all of the other issues that occurred during the award process greatly reduced our confidence in the process.

We did have a pretty good idea of what they are looking for based on working relationships with some people associated with the project, so we weren't going in blind. We addressed issues brought up by project personnel as well as the components of the evaluation factors. Next time we come up on another large proposal contract like this, I will suggest we bring in a professional proposal consultant.

Price I am sure ended up being a large consideration. The original solicitation was for a few dozen positions. After the agency performed their reviews and evaluated the price (and extended the proposal process twice), they sent an amendment where they removed 9 positions and removed the requirement for a compensation plan. They did not request any price changes, discounts, or final best offer. So at that point, it seems like they should have known that if our price was higher enough that we weren't in the competitive range, they should not have incurred additional work to us by requesting a revised proposal from us.

The difference in price was fairly large, around 25% (but still a 35% or so discount on the approved labor rates for the master contract), although a good portion of that extra cost is because we included overtime rates. For some reason, the previous iteration of this contract has not been paying overtime, and there was a lot of information indicating that there was a strong preference that these positions not be paid overtime. There was an absolute refusal by the CO to request a wage determination for the positions even though in every other region for the same exact positions, the agency is clear that the positions are OT eligible. The documentation included the following:

NOTE: If the Contractor determines any labor categories to be SCA applicable, then a Wage Determination shall be provided (in a clarification email, we were told that WE were responsible for providing a wage determination that supported our assertion that these positions were OT eligible). 

We pushed back on that note because we were very concerned that we would be bidding on the proposal using OT rates and others would not based on the positive requirement that the contractor justify considering the positions OT eligible.

We really wanted to be able enter negotiations. We had room to come down because non-labor hour approaches were not allowed to be proposed in our quote, and we wanted to consider a cost plus approach or FFP with Economic Adjustment. Since we were not allowed to do so we had to price out risk by covering the range of market based wages for every position. Being a team of actually small businesses doesn't really allow us to take much risk in our pricing (nor do we want to, we want to pay reasonable wages and limit staff turnover), and we just have to deal with that structural disadvantage and live with competing with large businesses under small business set asides because this is an IDIQ. Hopefully a professional proposal consultant can provide advice on how to address those issues as well.

Unless some sort of information comes out that makes it clear that they did not follow the process correctly, we won't be protesting. I am curious if there is some alternative to protest that initiates a review of the process by a relatively independent body, and which might then trigger a re-compete at the option year or something similar if the process did not proceed properly.

 
Link to comment
Share on other sites

If you believe the work is covered by the Service Contract Act and that a wage determination should have been provided, you may have raised the matter in protest before proposals were due.

2 hours ago, Salus said:

I am curious if there is some alternative to protest that initiates a review of the process...

That is exactly why the protest process exists.  But you did mention parent contract somewhere along the line -- if this procurement was a fair opportunity consideration under multiple-award IDIQ contracts, then there is a task order ombudsman you may talk to.

Link to comment
Share on other sites

Three observations:

1. You stated that the government amended the solicitations after proposal submissions to reduce the number of labor hour positions and delete any compensation plan. However there was never any request for revised proposals? It was unclear whether they asked for revised technical proposals but you said no revised pricing or “final best offer”. It is very strange that the requirements changed without any changes in the proposed pricing.

2. You said there were no “discussions” or “negotiations” but there was “absolute refusal” by the government… and you “pushed back” and in a “clarification email” you “were told” in response to a note in the solicitation  “that WE were responsible for providing a wage determination” to support your position that these positions were overtime eligible. Looks like there were some discussions.

3.  “The difference in price was fairly large, around 25%”. Voila! As I said, as the technical proposals become closer, then the price differences become more important. The government apparently didn’t think the differences, if any, between the non-price proposals warranted paying a 25% premium. That difference in cost may have reflected the post proposal due date amendment to reduce labor hours requirement. 

Link to comment
Share on other sites

Thanks JI. 

I wish we had thought of that as protestable. We did explain that were concerned about our inclusion of overtime rates potentially pricing us out of competition, and with the language included requiring that the contractor justify paying overtime, that a larger company (with enough lawyers) would be able to bid without using OT rates, and then if forced to pay OT because the employees went to the Department of Labor, they would be in a good position to request an equitable adjustment based on the inclusion of that language. It looks like that was a factor, and getting that affirmatively resolved could have been helpful in significantly narrowing the price gap.

Joel

I was keeping price vs. cost separate. Because this was a labor hour contract, price is the fully loaded labor rate per hour (unit) by each labor category. They amended the solicitation reducing the number of hours requested, thus reducing cost, but price per unit was unchanged (and no change in price was requested). Thus the resultant change in cost was entirely predictable. They also used the opportunity to remove the requirement for the compensation plan and add the statement requiring the contractor to provide a wage determination to justify OT. That makes it seem as though total cost was a major issue, and maybe we should have been more proactive in trying to trigger discussions about how we could lower prices. 

My understanding is that entering discussions is dictated by the government, not the contractor. We raised concerns on the requirement for the contractor to justify paying OT rates based on the reasons stated above. I am not sure how this would be considered discussions, but if it was, then there is at least one company that would probably have an issue with that.

And I understand that they should absolutely be considering the tradeoffs between price and performance. We have had the situation before where we were rated higher than a competitor, but our cost was somewhat higher and it was awarded to the lower rated, lower cost competitor. That makes sense and is understandable. Our issue is with the blanket and routine issuance of 'acceptable' ratings and subsequent award to the lowest cost bidder. Nowhere else am I aware of any sort of rating system where you would consider a person or company very well qualified with strengths in every component of the most important evaluation factor equal to someone who was minimally qualified. At that point, you aren't actually using a rating system, you are just determining whether a company is technically acceptable. When we received the feedback that was very positive, with no negatives but then just acceptable in all rating factors, it calls into question what kind approach they are using to implement the rating system. It raises further questions when we consider the weirdness about the agency even arguing against SCA/OT for technician level positions and the consistent implementation of the rating system that nearly always results in acceptable ratings that only apply to this particular region (these issues don't exist elsewhere that we have encountered). This makes it all seem as though price was much more important than the technical approach. Which is totally fine if you state so upfront!

Finally, we were just also unhappy with the unwillingness to consider alternative approaches that would allow us present better price options without incurring additional risk to a small business. That just seems like bad business planning, but I understand opening the contract to other strategies makes it more complicated, and this contract obviously had enough issues getting off the ground already.

If I had to guess, it would be that we ran up (and over) the actual budget for the work, and they didn't feel they had time to enter into discussions, or they felt that we would not be able to come down enough to meet their cost cap. But that is just wild speculation though.

 

Link to comment
Share on other sites

Oh, and I wasn't clear earlier. The reason for the amendment requesting updated proposal was because the process had gone on long enough that all of the original quotes had expired. They did request updated cost information and make the tweaks described above, but the stated reason was to extend the price quote duration another 180 days.

Link to comment
Share on other sites

Thanks for the clarifications, Salus.

I forgot to include another observation that I don’t think Salus mentioned originally. This is apparently a task order competition under an ID/IQ contract. Therefore, the Part 15 source selection process  isn’t necessarily mandatory for a fair opportunity TO competition, unless otherwise specified in the ID/IQ master contract.

The process can be much less formal, although we don’t know what process was spelled out. 

Link to comment
Share on other sites

On 6/25/2021 at 4:23 PM, Salus said:

…and we just have to deal with that structural disadvantage and live with competing with large businesses under small business set asides because this is an IDIQ.

Salus, how can a large business compete under a small business set aside? 

Link to comment
Share on other sites

They were (just) small when the proposals were submitted for the master contract. One of the teams was a joint venture, mentor protege that are now affiliated (this was one of 3 contracts they picked up in 2 years), and the other was a team of a 'small' under the IDIQ ($50 million in revenue now) teaming with a large business that was the incumbent. Both of those teams are more than 70 million in revenue against the size standard of $16.5. But they both qualify for set asides for the life of the IDIQ. So fairly large 'small' businesses then.

As far as source selection procedures, you are right that the master contract sets different standards than normal. Now I have to go try to find this specifically, but my understanding is that for contracts over a few million dollars follow standards procedures, while those less than that threshold follow simpler rules for the IDIQ.

Link to comment
Share on other sites

Ah, good luck, Salus! I remember an earlier thread where one of the pool members is now a large business. Here there are 2 that you must compete with. Wow! 

Link to comment
Share on other sites

On 6/25/2021 at 4:54 PM, ji20874 said:

If you believe the work is covered by the Service Contract Act and that a wage determination should have been provided, you may have raised the matter in protest before proposals were due.

 

 

Or after award to the DOL......FAR 22.1015.

Further I could be off base here but SCA is not the determining statute for OT.  The Contract Work Hours and Safety Standards Act and FLSA are.  

Minor after the fact clarification that may help in the future noting the discussion is with regard to an IDIQ and its processes.

https://www.dol.gov/agencies/whd/government-contracts/cwhssa

https://www.dol.gov/agencies/whd/overtime

 

Link to comment
Share on other sites

Thanks Joel, we'll take all the luck we can get to even the playing field!

C Culham, you are correct. That made it all the more difficult to provide an SCA wage determination to justify paying overtime!

Link to comment
Share on other sites

Quote

 

Our issue is with the blanket and routine issuance of 'acceptable' ratings and subsequent award to the lowest cost bidder. Nowhere else am I aware of any sort of rating system where you would consider a person or company very well qualified with strengths in every component of the most important evaluation factor equal to someone who was minimally qualified

I don’t think you’ll ever find out what’s behind this evaluation.  But one possibility alluded to earlier in the thread is none of the offers really stood out.  That may be why all the offerors got similar ratings of Acceptable. 

Often what differentiates one offer from another is simply what is said and how facts are presented.  For example two offerors may have similar past performance.  One states what their experience was in an engagement.  Another does the same and then highlights accomplishments and positive outcomes for the agency because of their work.  The implication is selection of them will provide benefits that competitors won’t. 

Or with a factor involving technical approach, an offeror distinguishes themself by stating uniqueness of what they do.  They come up with a combination of tools and techniques that will help in performance instead of just simple compliance with the tasks. Then for icing on the cake, state how that helps the agency.  An example is proactively monitoring and forecasting risks and mitigating early on.  This allows your company to perform and consistently meet or exceed customer requirements all the time.   Another example is you have an internal quality control system that utilizes subject matter experts and reviews all reports and deliverables before submission to clients. 

For contracts that essentially buy contract labor, there are possibilities to demonstrate advantages over competition.  You could say you have a deep bench that you will train on the clients needs at no cost to the agency.  That means if an employee is gone for any reason, a replacement will be virtually seamless.  Or you cross train employees so they know the work of others.  In event of a surge, others can help out.  This may eliminate the need for overtime.  From what you said it seems like the agency may be looking to reduce costs and minimizing overtime.  Focusing on creative approaches on this might go a long way on winning.

Just some thoughts for the future.


 

Link to comment
Share on other sites

8 hours ago, formerfed said:

I don’t think you’ll ever find out what’s behind this evaluation.  But one possibility alluded to earlier in the thread is none of the offers really stood out.  That may be why all the offerors got similar ratings of Acceptable. 

Often what differentiates one offer from another is simply what is said and how facts are presented.  For example two offerors may have similar past performance.  One states what their experience was in an engagement.  Another does the same and then highlights accomplishments and positive outcomes for the agency because of their work.  The implication is selection of them will provide benefits that competitors won’t. 

Or with a factor involving technical approach, an offeror distinguishes themself by stating uniqueness of what they do.  They come up with a combination of tools and techniques that will help in performance instead of just simple compliance with the tasks. Then for icing on the cake, state how that helps the agency.  An example is proactively monitoring and forecasting risks and mitigating early on.  This allows your company to perform and consistently meet or exceed customer requirements all the time.   Another example is you have an internal quality control system that utilizes subject matter experts and reviews all reports and deliverables before submission to clients. 

For contracts that essentially buy contract labor, there are possibilities to demonstrate advantages over competition.  You could say you have a deep bench that you will train on the clients needs at no cost to the agency.  That means if an employee is gone for any reason, a replacement will be virtually seamless.  Or you cross train employees so they know the work of others.  In event of a surge, others can help out.  This may eliminate the need for overtime.  From what you said it seems like the agency may be looking to reduce costs and minimizing overtime.  Focusing on creative approaches on this might go a long way on winning.

Just some thoughts for the future.


 

I wouldn’t have paid a 25% premium for a labor hour contract based upon what I read here.

Link to comment
Share on other sites

13 minutes ago, joel hoffman said:

I would t have paid a 25% premium for a labor hour contract based upon what I read here.

We don’t know what any offerer said or didn’t say nor any real history.   So any opinion is purely speculative.  LH contracts can be grossly inefficient or very efficient depending upon individual circumstances.  Offerors can come up with innovative ways to improve efficiencies.  What if an offeror proposed a way to reduce prices by 30%?  We just don’t know here.

Link to comment
Share on other sites

formerfed, one can glorify or extol their past performance but if it is “similar” to another firm’s past performance ratings, I wouldn’t pay a 25% premium. Past performance generally is rated by clients, not flowery self aggrandizement.

As for proposing methods or improvements beyond the stated requirements, if not sought after or stated as desirable in the task order competition, they aren’t enforceable, unless the government states that it will and does incorporate those features of the proposal into the contract. I saw several large (> $2 Billion) major Defense Program Systems contracts with extensive service phases where I was able to read the proposal after award. I found numerous differences between flowery management promises and actual execution. I’ve also seen the revolving doors of key personnel substitutions. Many of the substitutes were replacements for the proposed “A team” players with newbies or what turned out to be the “B team”. In all those cases, the government didn’t incorporate any of those aspects of the proposal nor include any requirements for personnel substitution. I wasn’t on the source selection team for those Systems contracts. 

Like I said, from what was described in this thread, I wouldn’t pay a 25% premium here for ?? additional value.  No offense intended to the OP here. 

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...