Jump to content

Source selection with minimal factors


formerfed

Recommended Posts

Just read this article by Steve Kelman on Agile.  

https://fcw.com/blogs/lectern/2019/12/kelman-agile-lessons.aspx?m=2

The government employee featured in it has done a dozen Agile projects.  To me, the interesting part is emphasis on vendor selection and “oral interactions.”  He likes asking offerors “tough detail-oriented questions” and use of statements of outcomes.

That makes sense to me.  Do an RFP with a SOO and plan on oral interactions such as a pop quiz.  I would maybe skip written proposals and have offerors do oral proposals followed by detailed questions challenging their approach.  Past performance would be critical but not the lazy way many do by just looking at CPARS and and sending out email surveys.  Offerors would submit references in advance and the government would visit each to find out exactly how offerors did, talking with end users, customers, as well as CORs and CO/KOs. Since this likely would be a LH contract, price would be relatively simple.  Offerors justify any deviations from the government LOE.

This goes a long way in source selection based on relevant factors - soundness of approach, meaningful and relevant past performance based on actual results, and price.  Past performance also would include ability to work in a collaborative manner with the government since that’s a large part of what Agile is about.

What do you think?

Link to comment
Share on other sites

  • 2 weeks later...

I agree that there are situations where use of a SOO, oral presentations, and real past performance evaluation makes sense to select the best offeror. But once you've selected the best offeror what would the resultant contract look like? You say it would likely be labor hour, so at that point you've got a set of pre-priced labor categories working to achieve what contractually-defined objective/outcome? Will the SOO have to be turned into a PWS with measurable objectives (by the Government or the offeror? Before or after award?), or will the contractor just be providing the labor hours to work on whatever the Government asks them to do?

I don't have any experience contracting for agile software development, so maybe it's a gap in my knowledge of how an agile software development contract would be structured in the first place.

Link to comment
Share on other sites

2 hours ago, Witty_Username said:

I don't have any experience contracting for agile software development, so maybe it's a gap in my knowledge of how an agile software development contract would be structured in the first place.

Take a look at the Agile Contracts Primer. There are pricing arrangements starting on p. 25. Chapter 3 covers contract models. The primer is not specifically written for Government contracting, which is one reason I like it. The Government publications that I've seen on contracting for agile software development don't seem to fully let go of the traditional Government contracting mindset. "Soundness of approach"? Please. (No offense formerfed).

Here's a must-read account of the realities of IT acquisition: https://medium.com/@EricHysen/lessons-learned-from-the-governments-biggest-attempt-to-fix-tech-procurement-bd2265421211

Link to comment
Share on other sites

On 12/27/2019 at 12:15 PM, Don Mansfield said:

The Government publications that I've seen on contracting for agile software development don't seem to fully let go of the traditional Government contracting mindset. "Soundness of approach"? Please. (No offense formerfed).

You may not like that term, Don, but evaluating how a company plans on tackling the effort is important.  Chose another term then.  A company’s development philosophy and methodology, how their process adheres to the government’s development processes, how the contractor plans to partner with the customer to manage the product backlog, their plans to demo software to the program office user; and what artifacts get delivered at the end of each sprint as well as complying with the governments definition of done all needs to be part of the evaluation. 

Link to comment
Share on other sites

To the extent that you will be evaluating verifiable information, I agree. However, by "approach" I assume that you will be evaluating an offeror's statements about the future, which are typically nonbinding and inherently unverifiable. It's marketing material that the Government traditionally believes correlates to the capability of the offeror. 

Link to comment
Share on other sites

17 minutes ago, Matthew Fleharty said:

I’m with Don on this one - evaluating an approach or plan (particularly since most of them are not promises) is not only wasted time and effort, it actually introduces noise to the decision making process.

Every company has their own means and philosophy of producing working software.  How stories are gathered, epics produced, and backlogs worked are important matters.  That needs to sync with the governments.   Those aren’t promises or formal plans events.  Those issues are topics for oral presentations and verified as part of past performance.   

Link to comment
Share on other sites

And, as we all know, the government has a stellar track record of software development contracts staying within budget and within schedule.

Link to comment
Share on other sites

4 hours ago, formerfed said:

Every company has their own means and philosophy of producing working software.  How stories are gathered, epics produced, and backlogs worked are important matters.  That needs to sync with the governments.   Those aren’t promises or formal plans events.  Those issues are topics for oral presentations and verified as part of past performance.   

Why even evaluate approach as a separate factor when you can get the same information more accurately through a past performance and experience evaluation?

Link to comment
Share on other sites

1 hour ago, Matthew Fleharty said:

Why even evaluate approach as a separate factor when you can get the same information more accurately through a past performance and experience evaluation?

There’s a lot of things to consider.  For example you want them to describe how the product vision was created.  Then how epics we’re formed.  Then user stories and so forth down to tasks.  There’s questions about how releases get bundled and when they are released.  More questions about how and when users see and provide feedback.   More questions on scheduling,  to do lists, use of dashboards, definition of done, etc.  These aren’t things you want them to give you in written or oral proposals in detail.  You want answers from questions and pop quizzes.  Then you get verification from past performance.   You want them to convey how they will meet your needs 

Link to comment
Share on other sites

Even if they do what you want, none of that information is a binding promise so it basically amounts to a sales pitch.  I don’t think either of us is going to convince the other of our respective positions - basically, you just want the contractor to tell you things that will make you feel good about your decision which I think is distinctly different than getting information that will help you make a better decision.

Link to comment
Share on other sites

18 minutes ago, Matthew Fleharty said:

Even if they do what you want, none of that information is a binding promise so it basically amounts to a sales pitch.  I don’t think either of us is going to convince the other of our respective positions - basically, you just want the contractor to tell you things that will make you feel good about your decision which I think is distinctly different than getting information that will help you make a better decision.

Agree with you. 

Link to comment
Share on other sites

This is getting away from the original thought and that’s use of a SOO and oral interactions.  The idea of a SOO is getting a variety of innovative and competitive approaches.  If only experience and past performance is evaluated, how do you differentiate among approaches?  What if one offeror uses a scrum methodology, another uses lean, and a third proposes.  Let’s say the offeror using scrum stumbles through orals with poor answers.  The one with lean can’t produce employees for the orals with extensive knowledge but their experience is good.  The third using kanban did a great job in orals but your CIO doesn’t know it well and isn’t comfortable.  How do you evaluate the three?

Link to comment
Share on other sites

10 hours ago, formerfed said:

This is getting away from the original thought and that’s use of a SOO and oral interactions.  The idea of a SOO is getting a variety of innovative and competitive approaches.  If only experience and past performance is evaluated, how do you differentiate among approaches?  What if one offeror uses a scrum methodology, another uses lean, and a third proposes.  Let’s say the offeror using scrum stumbles through orals with poor answers.  The one with lean can’t produce employees for the orals with extensive knowledge but their experience is good.  The third using kanban did a great job in orals but your CIO doesn’t know it well and isn’t comfortable.  How do you evaluate the three?

Right there with you former especially in the SOO approach.  It would seem that resource/guidance documents such as the "Guide" and "7 Steps" acknowledge looking at approaches is a good thing.

 http://acqnotes.com/acqnote/acquisitions/performance-based-acquisitions

Link to comment
Share on other sites

13 hours ago, formerfed said:

This is getting away from the original thought and that’s use of a SOO and oral interactions.  The idea of a SOO is getting a variety of innovative and competitive approaches.  If only experience and past performance is evaluated, how do you differentiate among approaches?  What if one offeror uses a scrum methodology, another uses lean, and a third proposes.  Let’s say the offeror using scrum stumbles through orals with poor answers.  The one with lean can’t produce employees for the orals with extensive knowledge but their experience is good.  The third using kanban did a great job in orals but your CIO doesn’t know it well and isn’t comfortable.  How do you evaluate the three?

1. What assurance would you have that the "approach" presented in response to the SOO would be the "approach" that the offeror would use in performing the contract?

2. What evidence do you have that a person stumbling through an oral presentation with poor answers correlates to poor software development? Developing software and providing oral responses to technical questions are not the same skills. I would not expect software developers to impress at an oral presentation. I would expect them to be more like Elliot Alderson.

3. What assurance would you have that the people answering the questions in an oral presentation would be the people developing the software?

Link to comment
Share on other sites

15 minutes ago, Don Mansfield said:

1. What assurance...?

2. What evidence...?

3. What assurance...?

Don't these questions apply to EVERY evaluation of a technical proposal where the technical proposal is not incorporated into the contract?

I am generally okay with subjective evaluations for many procurements.  After all, as the GAO reminds us, the responsibility for persuading the Government evaluators falls squarely on the offeror.

Link to comment
Share on other sites

4 hours ago, Don Mansfield said:

1. What assurance would you have that the "approach" presented in response to the SOO would be the "approach" that the offeror would use in performing the contract?

2. What evidence do you have that a person stumbling through an oral presentation with poor answers correlates to poor software development? Developing software and providing oral responses to technical questions are not the same skills. I would not expect software developers to impress at an oral presentation. I would expect them to be more like Elliot Alderson.

3. What assurance would you have that the people answering the questions in an oral presentation would be the people developing the software?

I’m not advocating this but a well known CIO had a way of dealing with these issues.  His agency made multiple awards and each awardee received some task order work.  After that new task orders were awarded based primarily on past performance.  He did the assessment.  If a company didn’t perform well or tried something like your questions suggested, they didn’t receive any new work for quite some time.

Link to comment
Share on other sites

46 minutes ago, formerfed said:

I’m not advocating this but a well known CIO had a way of dealing with these issues.  His agency made multiple awards and each awardee received some task order work.  After that new task orders were awarded based primarily on past performance.  He did the assessment.  If a company didn’t perform well or tried something like your questions suggested, they didn’t receive any new work for quite some time.

I would advocate how the CIO handles tasks for an ID/IQ after award. 

Link to comment
Share on other sites

6 hours ago, Don Mansfield said:

2. What evidence do you have that a person stumbling through an oral presentation with poor answers correlates to poor software development? Developing software and providing oral responses to technical questions are not the same skills. I would not expect software developers to impress at an oral presentation. I would expect them to be more like Elliot Alderson.

Questions for orals with Agile are straight forward.  How many points do you typically include in a sprint?  How many sprints are part of a release?  What’s your experience using multiple teams in parallel to speed up development?  What issues have you encountered combining parallel efforts?  How long is your daily standup?  What is your average velocity?  What’s the ideal sprint duration for your teams? 

I could go on and on.  Experienced developers will bore you with answers.  It doesn’t take skill or a good image.  Just having done it a few times.  These aren’t technical questions either.

Link to comment
Share on other sites

10 hours ago, ji20874 said:

Don't these questions apply to EVERY evaluation of a technical proposal where the technical proposal is not incorporated into the contract?

 

7 hours ago, Don Mansfield said:

@ji20874, I don't know what you mean by technical proposal. Do you mean a description of what an offeror plans to do if they awarded a contract? No promises, just plans and other statements of intention?

Does not the PWS that the selected offeror in a SOO approach is to provide and is negotiated as part of their offer in essence include their technical approach and in fact provide more than a plan but promises of the offeror?  I think the PWS as created by the offeror does.

Link to comment
Share on other sites

19 hours ago, formerfed said:

I’m not advocating this but a well known CIO had a way of dealing with these issues.  His agency made multiple awards and each awardee received some task order work.  After that new task orders were awarded based primarily on past performance.  He did the assessment.  If a company didn’t perform well or tried something like your questions suggested, they didn’t receive any new work for quite some time.

What if the company didn't do what they said they were going to do, but performed well?

17 hours ago, formerfed said:

Questions for orals with Agile are straight forward.  How many points do you typically include in a sprint?  How many sprints are part of a release?  What’s your experience using multiple teams in parallel to speed up development?  What issues have you encountered combining parallel efforts?  How long is your daily standup?  What is your average velocity?  What’s the ideal sprint duration for your teams? 

I could go on and on.  Experienced developers will bore you with answers.  It doesn’t take skill or a good image.  Just having done it a few times.  These aren’t technical questions either.

I think the responses to those questions would be verifiable. The offeror would report facts. That could yield useful information in predicting an offeror's performance.

However, that's different than:

On ‎12‎/‎28‎/‎2019 at 2:58 PM, formerfed said:

how a company plans on tackling the effort

A company’s development philosophy and methodology

how their process adheres to the government’s development processes

how the contractor plans to partner with the customer to manage the product backlog

their plans to demo software to the program office user

If you request this information, you will get unverifiable nonbinding statements about the future. The offeror will have to invent responses instead of report facts. This information is not as useful in predicting how an offeror would actually perform. As Matthew wrote, this introduces noise into the decision-making process.

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...