TLR Posted January 21, 2009 Report Share Posted January 21, 2009 I have a question revolving around past performance and technical evaluations. We are attempting to acquire COTS software that will need to be tailored to meet our specific needs, needs which are not uncommon and we expect other agencies have had similar needs. We are wanting to request that offerors submit three references as usual, but also request that they allow us, the contracting office, to request an on-site demonstration from the past performance references. In other words, we want to go above and beyond normal past performance evaluations and we want to see similar tailored COTS software in action at other agencies, not just a sample provided by the offeror. I cannot find where this has been done, but I am sure others have tried to do this. If someone knows a solicitation where this has been done, I would like to review the language. Also, any guidance on how you went about approaching the other agencies, and whether or not the process was a success or a failure would be great. We want to only do this for those offerors that remain in the competitive range after initial reviews by the technical evaluation team. Thanks in advance for any assistance. Link to comment Share on other sites More sharing options...
Guest Vern Edwards Posted January 21, 2009 Report Share Posted January 21, 2009 Interesting idea. What would you do, go to other agencies and ask to look over their shoulders while they use the software? Would you ask them to run specific tests? Would you ask to sit at a workstation and run your own tests? Would the test outcomes be affected by different database structures? How much of their time would you expect to take? Link to comment Share on other sites More sharing options...
TLR Posted January 21, 2009 Author Report Share Posted January 21, 2009 Interesting idea. What would you do, go to other agencies and ask to look over their shoulders while they use the software? We would more than likely ask that the vendors pre-confirm which of their references would be willing to let us see the software, and yes, we would be looking over their shoulder but it could be with sample information so there are no confidentiuallity issues. We would have to spell that out clearly in the RFQ. This is an idea that generated from the technical team as they have concerns due to the high profile nature of the system and the strict timeline to implement. Would you ask them to run specific tests? That has yet to be determined, but I anticipate that a general run through the process of utilizing the software under their normal circumstances. Would you ask to sit at a workstation and run your own tests? No, I believe we would not ask them to do anything above and beyond their normal operations so as to not cause any undue burden. IT folks get particular when it comes to things like this, and for good reason. Would the test outcomes be affected by different database structures? Possibly, that would be something that we would ask them to address given the current structure at our location. How much of their time would you expect to take? I would expect 1-2 hours at the most since we are not talking about actual testing, merely observance under normal usage. Thanks for the thought provoking questions, this helps me frame the issue better. Link to comment Share on other sites More sharing options...
Guest Vern Edwards Posted January 21, 2009 Report Share Posted January 21, 2009 Interesting. I cannot think of any legal bar against doing it. However I see it as fraught with peril. Essentially, you plan to observe and then make some kind of inference(s) based on your observations. You had better have a good observational checklist and idea about what attributes you are looking for and what kind of inferences you plan to make based on your observations, and some explanation of why you think that you can reasonably make such inferences based on what you see. You had also better make sure that you treat all offerors fairly and apply the same criteria to each. Might be a lot of trouble just to evaluate past performance. Link to comment Share on other sites More sharing options...
jason_a Posted January 22, 2009 Report Share Posted January 22, 2009 This lends itself to be very subjective. For example what if Company A only had to modify it slightly. During your evaluation you see it's easy to use and functions smoothy. Company B on the other hand did a major modification. During thier evaluation it doesn't seem as user friendly and requires more input. How do you evaluate that? Will you have a software engineer looking at the modified code? You only mentioned the contracting office. I like the way of thinking but the application seems a little tough. Link to comment Share on other sites More sharing options...
formerfed Posted January 26, 2009 Report Share Posted January 26, 2009 I know of a couple agencies that did something similar. However they didn't use the actual demonstrated performance as part of their evaluation. In addition to what jason_a stated, too much is dependent on other events such as whether the software runs on a shared or dedicated server, the size of the server, communications, how the software and other things are configured. What the agencies I know of did was verify that the COTS modifications performed as indicated by the contractor, how responsive the contractor was in correcting discovered problems and addressing operational issues, etc. While this could be done over the phone, the agencies felt that getting a number of users and support staff together in a face-to-face setting results in much more meaningful input. Link to comment Share on other sites More sharing options...
brian Posted January 28, 2009 Report Share Posted January 28, 2009 . this post may demonstrate that I shouldn't post this early in the AM, but here goes. The sort of on-site monitoring of how a tailored system works for a particular customer, I don't think that's looking at Past Performance. I think that's a Technical Evaluation. Implications ? I don't know. . Link to comment Share on other sites More sharing options...
Velhammer Posted January 28, 2009 Report Share Posted January 28, 2009 Brian: Early or not, I agree. Not withstanding the fact that some of the qualifiers could be used to establish relevance to the current solicitation, this seems to have gone down a path of more than just "how well did they perform". Link to comment Share on other sites More sharing options...
.gif)
Recommended Posts