Preferences are not always transitive

[Vern Edwards]

See the Court of Federal Claims bid protest decision in Stratera Fulcrum Technologies v. U.S., May 2, 2022.

https://ecf.cofc.uscourts.gov/cgi-bin/show_public_doc?2021cv1770-79-0

The agency (Patent and Trademarks Office, to which the FAR does not apply) used the Highest Technically Rated with a Fair and Reasonable Price (HTR w FRP) method of source selection in combination with what is called the "transitive analysis" technique. That technique involves ranking proposals through paired comparisons in reliance on the transitivity axiom, what holds that if Offeror A is better than Offeror B, and Offeror if B is better than Offeror C, then Offeror A must be better than Offeror C, so there is no need to compare Offeror A to Offeror C directly. The court referred to that axiom as "the transitive property of inequality."

See decision footnote 5:

Quote

The transitive property of inequality can essentially be boiled down to the following: if a > b, and b > c, then a > c. For the purposes of this procurement, it would be used to say that if Offer 1 is better than Offer 2, and Offer 2 is better than Offer 3, then Offer 1 is better than Offer 3, rendering a direct comparison between Offers 1 and 3 unnecessary.

The court denied the protest.

Reliance on the transitive property when making source selection decisions might be considered a logical and efficient procedure. But it has been an issue in two protest decisions:

• Metric 8 LLC; M6-VETS, LLC; RCH Partners, LLC; Stratera Fulcrum Technologies, LLC; MERPTech, LLC., GAO B-419759.2 (2021)
• Salient CRTG, Inc, GAO B-419759 (2021)

The GAO has not objected to its use, so far.

There is a problem, however: Although logically sound, Amos Tversky showed in 1969 that transitivity does not always hold in matters of human decision making. See Tversky, "Intransitivity of Preferences" in Psychological Review, Vol. 76, No. 1, 31-48.

https://pages.ucsd.edu/~mckenzie/Tversky1969PsychReview.pdf
 

Quote

Individuals, however, are not perfectly consistent in their choices. When faced with repeated choices between x and y, people often choose * in some instances and y in others. Furthermore, such inconsistencies are observed even in the absence of systematic changes in the decision maker's taste which might be due to learning or sequential effects. It seems, therefore, that the observed inconsistencies reflect inherent variability or momentary fluctuation in the evaluative process. This consideration suggests that preference should be defined in a probabilistic fashion.

*     *     *

When the difficulty (or the cost) of the evaluations and the consistency (or the error) of the judgments are taken into account, a model based on component-wise evaluation, for example, may prove superior to a model based on independent evaluation despite the fact that the former is not necessarily transitive while the latter is.

The late Tversky is a legend. See Lewis, The Undoing Project: A Friendship That Changed Our Minds (2016). His paper is highly regarded and has been cited more than 3,000 times. And since Tversky, many, many papers have raised issues about the validity of the transitivity axion in decision making. Apparently, some lawyers don't know about them. I didn't, until I read The Undoing Project.

I doubt that the protester's lawyers knew of Tversky's paper or that transitivity of preference does not always hold when people make decisions. If they knew they might have asked whether the USPTO undertook measures to validate its procedure. (I doubt that it did.) But, not knowing, they probably did not bring that matter to the court's attention. If they had, and if the agency had not been able to show that it had taken steps to confirm the transitivity of inequality in its procedure, then it might have cast a shadow on their source selection decision in the mind the judge.

But while most writers think Tversky got it right, some differ. In the opinion of one writer, "Any claim of empirical violations of transitivity by individual decision makers requires evidence beyond a reasonable doubt." See Regenwetter, et al., "Transitivity of Preferences," Psychological Review, Vol. 118, No. 1, 42-56.

https://www.chapman.edu/research/institutes-and-centers/economic-science-institute/_files/ifree-papers-and-photos/michel-regenwetter1.pdf

Sooner or later, protest lawyers are going to get wise to this and raise the possibility of intransitivity in their arguments. The GAO is often bull-headed about sticking to its "case law," so you might be safe there, but judges at the COFC can be more thoughtful, probing, and skeptical.

I'm not saying that you shouldn't rely on transitivity when determining best value. I'm saying validate your procedure if you do and document how you did it. If you cannot figure out how to do that, then be very careful. Don't jump on the bandwagon and rely on something (cut and pasted from another RFP) that you may not fully understand. Transitive analysis can save some time and effort, but only if can withstand challenges.

Forewarned is forearmed. Be ready.

[formerfed]

I’ve seen the transitivity method used twice (once 30 years ago) and wondered, if protested, what the outcome might be.  I wasn’t aware of the GAO decisions until now.  The one aspect I’m not completely comfortable with is this:

Quote

Based on the proposals’ adjectival ratings, the agency identified Halvik, Steampunk, and [*****] as possible controls, 

It seems like just relying on adjectival ratings rather than the reasons such as respective merits, strengths and weaknesses behind the summary ratings, is counter to many other protest decisions.  I know the objective was selecting Highest Technically Rated with a Fair and Reasonable Price,  maybe highest rated is defined by adjectival terms.