Jump to content
The Wifcon Forums and Blogs

Sign in to follow this  
j_dude77

Past Performance

Recommended Posts

We are looking to award a contract utilizing a past performance trade-off method. So just price and past performance. We are asking contractors to submit samples of work (audit reports) that they have previously performed to be evaluated under past performance. The attorneys who have reviewed the evaluation criteria keep saying that we cannot review work samples under past performance because this should be technical. My answer to them is yes we can evaluate it under past performance since it demonstrates the quality of work done by the contractor. It was my understanding that the FAR gives contracting officers wide latitude when it comes to what and how to evaluate, as long as we state it.

Am I missing something here? Can someone point me to documentation/decisions that would support this? 

I have read numerous GAO decision and white papers, but no specifically address what can be considered past performance. Unfortunately I can't tell the attorneys to piss off.

Share this post


Link to post
Share on other sites

Are you  planning to request samples of previous audit reports? Do you intend to evaluate the quality and or relevance of those previous audit reports?  Or will you evaluate performance ratings or some other quality satisfaction criteria from those owners' perspectives for the identified previous Audits? 

You can use relevance as a consideration to distinguish between similar types and/ or magnitude of audits and audits that have little relevance to audits you will contract for.

In fact, DOD's  method of assessing a level of "confidence" does distinguish the relevance of previously performed contracts or tasks   

It's not clear what you are planning to evaluate ???

 

 

Share this post


Link to post
Share on other sites
2 hours ago, j_dude77 said:

The attorneys who have reviewed the evaluation criteria keep saying that we cannot review work samples under past performance because this should be technical.

What is the basis of their assertion?

Share this post


Link to post
Share on other sites

Well, I'm confused.

Seems to me like j_dude77 is not evaluating past performance at all. Seems to me what is being evaluated is evidence of past performance without any regard to relevance, complexity, etc. For example, I can create the world's greatest audit report but it's going to cost you quite a bit of money and take a really long time.

Other concerns: The audit report is the result of the work but it is not evidence of the work. The work is evidenced by the working papers that support the conclusions reached, which are documented in the audit report. If you aren't evaluating the working papers then you're just evaluating a pretty piece of writing. In addition, what differentiates audit reports from each other? What constitutes a "great" audit report from a merely "adequate" audit report? Both excellent and poor audit reports exhibit the same characteristics: objective, scope of work, methodology, conclusion, signature. How do you tell the difference?

I always thought past performance was a documented rating/discussion in a government system, and evaluators also looked at relevance of the past performance rating(s) to the work being proposed. I'm not seeing that here. [ETA: What Joel posted.]

In short: I'm with the attorneys.

If I were going to be evaluating audit firms I would care very much about the personnel staffing the work. I would want to see resumes and CPA designations and minimum levels of relevant experience and minimum levels of training. The better the people, the better the result, or so my experience informs me. I would try to get the best audit team for my budget, and let the audit report take care of itself.

Hope this helps.

Share this post


Link to post
Share on other sites

FAR 15.305(a)(2)(i) describes a Past Performance Evaluation as a “…comparative assessment of Past Performance Information…”

FAR 42.1501 then describes Past Performance Information as follows:

(a) Past performance information (including the ratings and supporting narratives) is relevant information, for future source selection purposes, regarding a contractor’s actions under previously awarded contracts or orders. It includes, for example, the contractor’s record of—

  (1) Conforming to requirements and to standards of good workmanship;

  (2) Forecasting and controlling costs;

  (3) Adherence to schedules, including the administrative aspects of performance;

  (4) Reasonable and cooperative behavior and commitment to customer satisfaction;

  (5) Reporting into databases (see subpart 4.14, and reporting requirements in the solicitation provisions and clauses referenced in 9.104-7);

  (6) Integrity and business ethics; and

  (7) Business-like concern for the interest of the customer.

You said, “We are asking contractors to submit samples of work (audit reports) that they have previously performed to be evaluated under past performance.”  It is unclear whether you mean the PP rating will be based solely on an evaluation of the sample work, or whether that sample evaluation would be one of many factors to be considered in establishing the rating.

An example of a work product may give you some insight into the offerors level of “workmanship” (i.e., (a)(1)).  It may also help a contracting officer determine the relevancy of that effort to the proposed work.  But, it provides little to no information with regard to the many other factors that should go into an overall Past Performance Evaluation.

Therefore, I think a PP evaluation that only considers a snapshot example of “workmanship” would be insufficient.  However, I am not aware of any prohibition on requesting and considering such information as part of a larger overall PP evaluation.

Share this post


Link to post
Share on other sites

I am going to throw my support to H2H's opinion. 

Share this post


Link to post
Share on other sites

Are we talking about a past performance evaluation under FAR 15.305( a )( 2 )?  Or some other setting?

Share this post


Link to post
Share on other sites

I believe the best way to answer this is by considering which type of factor best describes what you wish to accomplish.  If I understand correctly, you want contractors to submit work samples from past projects to be evaluated, presumably the work samples that were complete, accurate, and of good quality would be rated higher than those which are not.

A technical evaluation is an assessment of each offeror’s ability to accomplish the technical requirements (FAR 15.305(a)(3)).  This can be accomplished through the application many different factors, including but not limited to, experience, technical approach, product samples, capacity, capabilities, key personnel, and personnel qualifications.

Past performance is an indicator of a firm’s ability to perform successfully by evaluation of how well a firm has performed in the past.  The currency and relevance of the information, source of the information, context of the data, and trends in performance must be considered. (FAR 15.305(a)(2)). 

It seems to me you are assessing the contractors ability to accomplish technical requirements (work) required by the solicitation, which is a technical evaluation.  While the work samples are from past projects, I don’t think this best fits the description of what past performance evaluation is.  Normally past performance is not what the contractor thinks of their performance or what the government thinks of a sample work product from the past, but rather the opinion of those evaluating their performance on past contracts.  Sources of such information include PPIRS, personal knowledge of past performance, and questionnaires completed by their previous customers, etc.  Also, FAR 15.305(a)(2) states that the evaluation “should take into account past performance information regarding predecessor companies, key personnel who have relevant experience, or subcontractors that will perform major or critical aspects of the requirement when such information is relevant to the instant acquisition.”  If applicable, these elements probably would not be able to be evaluated by only looking at a work product.

Also, I think looking at one work product from the past can be misleading and not be indicative of how they have performed in the past generally.  For example, a firm could have failed on 9 out of 10 contracts, but sent you the sample for the one they did outstanding on.  Also, there is the potential for the work product to be altered, to improve its appearance.

Share this post


Link to post
Share on other sites

As is frequently the case here, someone initiates a thread with ambiguous information, then seems to disregard requests for clarification, perhaps waiting for someone to stumble upon an answer that suits the OP. 

Ok  evaluating the quality of samples of previous audits is not evaluating " past performance" as define by FAR (see above references). 

Determining  the extent of experience of a firm in preparing audits by requesting "samples" is not evaluating "past performance". There have been numerous threads here that distinguish between "extent of experience"and "past performance".  

Iin DOD's confidence assessment system the tie between experience and past performance is primarily to determine the relevance of the past performance ratings to the instant project. Thus excellent past performAnce in baking bread doesn't equate to excellent past performance in construction of a barracks for a new barracks project. Thus, there is a basis to assign a lower level of confidence that the baker will be successful in building a barracks. 

Share this post


Link to post
Share on other sites

I agree with H2H.  I suspect that the audit reports that the GAO faulted DCAA on for not adequately documenting audit findings in the workpapers were skillfully written pieces of prose.  However, the GAO found that the audit reports were unacceptable because they did not comply with Yellow Book standards (GAGAS).  In short, as H2H intimated, you cannot fully judge the quality of an audit report simply by reading it.  It may be good prose, but bad auditing.

Hopefully, there is more guidance on what is considered relevant in regard to the audit reports.  There are many kinds of audits that are performed using different standards.  For example, financial statement audits are different from audits of government contracts using GAGAS.  Even government audits are performed using different criteria and standards.  For example, if someone has audited non-profit organizations but not commercial entities, the auditor would have experience with one set of cost principles but not the relevant cost principles in 31.2.

Share this post


Link to post
Share on other sites
7 hours ago, Lionel Hutz said:

FAR 15.305(a)(2)(i) describes a Past Performance Evaluation as a “…comparative assessment of Past Performance Information…”

 

 

 

 

FAR 42.1501 then describes Past Performance Information as follows:

 

(a) Past performance information (including the ratings and supporting narratives) is relevant information, for future source selection purposes, regarding a contractor’s actions under previously awarded contracts or orders. It includes, for example, the contractor’s record of—

 

  (1) Conforming to requirements and to standards of good workmanship;

 

  (2) Forecasting and controlling costs;

 

  (3) Adherence to schedules, including the administrative aspects of performance;

 

  (4) Reasonable and cooperative behavior and commitment to customer satisfaction;

 

  (5) Reporting into databases (see subpart 4.14, and reporting requirements in the solicitation provisions and clauses referenced in 9.104-7);

 

  (6) Integrity and business ethics; and

 

  (7) Business-like concern for the interest of the customer.

 

You said, “We are asking contractors to submit samples of work (audit reports) that they have previously performed to be evaluated under past performance.”  It is unclear whether you mean the PP rating will be based solely on an evaluation of the sample work, or whether that sample evaluation would be one of many factors to be considered in establishing the rating.

 

An example of a work product may give you some insight into the offerors level of “workmanship” (i.e., (a)(1)).  It may also help a contracting officer determine the relevancy of that effort to the proposed work.  But, it provides little to no information with regard to the many other factors that should go into an overall Past Performance Evaluation.

 

Therefore, I think a PP evaluation that only considers a snapshot example of “workmanship” would be insufficient.  However, I am not aware of any prohibition on requesting and considering such information as part of a larger overall PP evaluation.

 

Evaluating previous audits (e.g., the quality thereof) is not evaluation of past performance.  Evaluating the relevance of previous audits can be valid criteria for distinguishing between similar track records of audit firms. 

Share this post


Link to post
Share on other sites
11 hours ago, here_2_help said:

Well, I'm confused.

Seems to me like j_dude77 is not evaluating past performance at all. Seems to me what is being evaluated is evidence of past performance without any regard to relevance, complexity, etc. For example, I can create the world's greatest audit report but it's going to cost you quite a bit of money and take a really long time.

Other concerns: The audit report is the result of the work but it is not evidence of the work. The work is evidenced by the working papers that support the conclusions reached, which are documented in the audit report. If you aren't evaluating the working papers then you're just evaluating a pretty piece of writing. In addition, what differentiates audit reports from each other? What constitutes a "great" audit report from a merely "adequate" audit report? Both excellent and poor audit reports exhibit the same characteristics: objective, scope of work, methodology, conclusion, signature. How do you tell the difference?

I always thought past performance was a documented rating/discussion in a government system, and evaluators also looked at relevance of the past performance rating(s) to the work being proposed. I'm not seeing that here. [ETA: What Joel posted.]

In short: I'm with the attorneys.

If I were going to be evaluating audit firms I would care very much about the personnel staffing the work. I would want to see resumes and CPA designations and minimum levels of relevant experience and minimum levels of training. The better the people, the better the result, or so my experience informs me. I would try to get the best audit team for my budget, and let the audit report take care of itself.

Hope this helps.

I meant to quote H2H above in my last post. Sorry. 

Evaluating previous audits (e.g., the quality thereof) is not evaluation of past performance.  Evaluating the relevance of previous audits can be valid criteria for distinguishing between similar track records of audit firms. 

Share this post


Link to post
Share on other sites

Sorry it took so long to reply. To clarify, yes we are asking for the associated working papers. The audit and working papers is only one part of the evaluation. We are also asking for narratives and questionnaires. The technical folks biggest concern is that many firms are confusing contract audits with financial audits. This became evident during market research. Either way the issue has been settled at a level much higher than me. As an aside, other agencies have released RFPs evaluating the audits and working papers under past performance. Just have to do what I can with what I am given.

Again appreciate all the input.

Share this post


Link to post
Share on other sites
1 hour ago, j_dude77 said:

The technical folks biggest concern is that many firms are confusing contract audits with financial audits.

I'm not claiming to know better than j_dude77's team, because I'm just a random voice on the internet. That being said, I would have to think that there would be easier ways of determining a qualified firm--and of determining whether that firm knew the difference between the two types of audits -- than evaluating audit reports.

Also: resumes, resumes, resumes. Perhaps unlike building planes or tanks, it's the people, not the firm.

H2H

Share this post


Link to post
Share on other sites

H2H - Although I agree it's the people, not the firm, accepting/evaluating resumes always gives me pause because I have had too many occasions where the person proposed in the proposal isn't the one that shows up to do the work. I hear, "well, we were going to hire that person but he took another job" or "he's needed on another project". The ol' bait-and-switch. I'd rather supplement the resume with the offeror's proposed minimum requirements that any employee used to fill that position will possess.

Share this post


Link to post
Share on other sites

Desparado,

Understood. You are correct. However, I would suggest that firms that don't already have qualified staff in-house may not be the right firms. I get this shuts out SBs but if you want quality audits you need quality people. The firm's credentials are almost irrelevant. This is based on my 10 years in the ranks of those firms.

Share this post


Link to post
Share on other sites
Guest
This topic is now closed to further replies.
Sign in to follow this  

×