Jump to content

Metics of performance question


byrsch

Recommended Posts

We have a contract for language translation that we want to put performance metrics in, however; we have not been able to find any industry standards for these. I have suggested putting what are considered the ?best practices? in the industry into the SOW or PWS as the standard but am getting push back that we can?t impose them. I think if we write them into the SOW or PWS as the standard we can enforce them but the legal expert here says no. If we agree and the Contractor agrees and the contract is signed isn?t that legally binding them to the standard imposed in the SOW or PWS?

Link to comment
Share on other sites

When I'm in a similar situation, I do market research. I search and find who the industry "leaders" are and invite them in for one-on-ones. Preapre an agenda in advance so they come prepared.

In your case the questions I might ask are:

- what are industry standards?

- if we make this contract performance-based, are there appropriate incentives?

- if we switched roles and you are now the government, how would yopu do the contracting for interpretors?

- what have other organizations done to contract for these services?

Link to comment
Share on other sites

You may want to contact a Federal Government contracting office that currently has a translation contract/purchase/delivery order in place. They may be able to assist you. Look for places like Fort Benning because the Western Hemisphere Institute for Security Cooperation is there and documents are translated from Spanish to English or visa versa regularly. You may also want to try Southcom units who deal a lot with South American governments and agencies. Perhaps even the Defense Language Institute will be able to assist.

Link to comment
Share on other sites

At fisrt I thought your question pertains to translating written documents but now I wonder if you're dealing with verbal/oral interpretation services. Can you clarify?

If it's the former, standards depend alot of the quality of the translation needed. For example, sevral agencies contract out for "quick"translation because they just want foreign jouranls and technical articles translated so the reader can decide if there's further interest. In other words, the reader just wants to get a good sense on what the article is about and not every word literally needs translated correctly. Other requirements need the translation to be virtually error free even if the translator needs to add additional context to fully convey the meaning.

Can you elaborate?

Link to comment
Share on other sites

At fisrt I thought your question pertains to translating written documents but now I wonder if you're dealing with verbal/oral interpretation services. Can you clarify?

If it's the former, standards depend alot of the quality of the translation needed. For example, sevral agencies contract out for "quick"translation because they just want foreign jouranls and technical articles translated so the reader can decide if there's further interest. In other words, the reader just wants to get a good sense on what the article is about and not every word literally needs translated correctly. Other requirements need the translation to be virtually error free even if the translator needs to add additional context to fully convey the meaning.

Can you elaborate?

I am dealing with written translations from different languages and into different languages; it can be English to Arabic, English to Spanish, etc.... I have tried the UN, State, Defense Language Institute, plus numerous others, while I find standard with very vague requirements I can not find anything concrete and am beginning to doubt any exist. I was hoping to get lucky here, will try to putting language into the SOW defining the requirements if I can get agreement from the various sections involved. Thanks to all for trying to help.

Link to comment
Share on other sites

Rather than reinvent the wheel, you may want to contact some folks that have already been through this. Have you checked FedBizOpps? I did a quick check on ?translate? and received multiple hits, including FA4885-09-R-0005 . Maybe phone or e-mail the POC and see how they overcame similar issues; maybe they did or didn?t consider them but it?s worth a shot.

Randy

Link to comment
Share on other sites

Guest Vern Edwards

If you have contacted the UN, the Department of State, the Defense Language Institute, and some of the many other organizations that provide translation services and have not found any "metrics," then it may be time to take the hint. If there are no established "metrics," then it would be foolhardy to make up your own, and the results could be catastrophic.

What I hate about the "metrics" movement is how many "metrics" advocates make presentations about it but cannot explain exactly what "metrics" is/are or how to develop one/some. Instead of providing solid conceptual descriptions and describing a development procedure, they yak about market research and say that they cannot prescribe a development procedure because there is no one-size-fits-all solution.

The obvious "metrics" for translation are accuracy and speed. But how are you going to apply an accuracy metric if you need a translator? And the speed metric should depend on the accuracy metric, otherwise you might reward speedy mistranslation. Are you going to have a translator observe the translator and document errors? And what is the right standard for accuracy and speed? And what counts as an error when you're translating foreign terms into English? What if there is no precise English equivalent of a foreign word or expression?

Link to comment
Share on other sites

Guest carl r culham

In determining how to measure performance, and whether inclusion of performance metrics is the way to go, or not, you may want to propose to your legal advisor one of two approaches. Either one gets to the point of your question in providing a performance measure, including metrics or not, that become binding and enforceable in the contract.

First approach would be to make it a part of the PWS approach of the solicitation to have the offerors propose the QASP. Having them do so would provide opportunity for the industry to help you on how success in delivering the services is measured.

The other approach would be to use the Statement of Objective approach for your procurement rather than a SOW/PWS approach. Here not only would the offerors develop your PWS but provide the QASP as well.

Approaching in either way gets to the point of your original question as the contractor that you end up selecting has in essence developed the performance measurement standards that become part of the contract and may meet the level of enforceability that your legal advisor is having trouble with.

Link to comment
Share on other sites

Guest Vern Edwards

I don't know how a legal advisor got involved, since there has been no mention of one before now.

byrsch: I recommend that you take neither a PWS or SOO approach. The best way to ensure quality translation services is to hire qualified translators, qualifications being based on demonstrated language skills, training as a translator, and on references from persons or organizations for whom he or she has translated. There may also be a professional certification. It may be wise to establish a testing program for contractor personnel to verify that they have the necessary skills.

Both the military and the State Department have hired many translators over the years (especially persons to translate Baltic languages and Arabic). How do they verify the qualifications of the persons that they hire? How do they evaluate translator performance?

The statement of objectives (aka, SOO) technique was developed for use in aerospace/defense programs, where the contractors have extensive knowledge and experience in government-style statement of work preparation. Then some people, encouraged by advocates of performance-based contracting, decided to apply it to service contracts for which the providers have never heard of such things. Why? Because they didn't know how to write PWSs, so they decided to get contractors, who also did not know, to do it for them. "Industry" must know. Right? That provided a lot of performance-based service contracting revenue for training firms for a decade. I know, because for about five years it seemed that that was what everyone wanted to hire me to teach. I doubt that many firms that provide translation services have ever heard of a QASP (quality assurance surveillance plan). If your prospective contract is big enough, and if you tell them they have to write one in order to get the business, then they'll immediately look for consultants to show them how to do it or, better yet, to do it for them.

Maybe I'm wrong. There's a way to find out.

Link to comment
Share on other sites

Guest carl r culham

Vern - No mention of legal expert or advisor? By my read the initial post by byrsch did raise the subject in the original post in this discussion thread by stating in part - "legal expert here says no".

byrsch - With regard to the idea or not of using a SOO approach I see no prohibition in FAR Part 37.6 regarding application of a SOO to this particular procurement. Further , FAR 37.102, provides that performance based acquisition is the preferred method of contracting for services. On these facts, and noting that you did raise the subject of ?PWS? as well I would offer that both avenues (PWS w/QASP and SOO) should be considered and as I noted proposed as an alternative to the ?legal expert? to see if such an approach helps in addressing the concerns he/she may have about contract enforceability.

This is not to say that you could not propose to the program area to abandon the procurement route and do a hire of individual(s) for the need, as suggested in the thread, as like PWS/SOO it is an alternative as well. Using the tool that best fits is always advisable.

Speaking of tools and trying to find the best solution that helps you in the procurement approach, if that is the course that is chosen, here are a few other suggestions which you may want to explore if you have not already.

Google ? Doing a search on the translation services does produce examples of SOWs, some in PWA format.

FEDBIZOPPS ? You will find further examples again some using the PWA approach.

GSA Schedule ? Schedule 738II Language Services. This may help in narrowing your search of a contractor that is both expert in language translation services and that has experience in responding to a solicitation under the Schedule where PBA has been utilized.

Overall, I would not abandon PBA as an approach that could be used successfully and address the legal concerns, as it appears from my research that it has been done in the past.

Link to comment
Share on other sites

Guest Vern Edwards

byrsch:

Forget the SOO and the PWS. Sure, FAR permits the use of the SOO technique. So what? The fact that FAR permits it is not a reason for using it. I'm telling you, that approach was designed for and is most suitable for aerospace/defense system development work, which is done by an industry which understands such things. If you do not find that the performance work statement is an established approach among translation service providers, then stay away from it, as far away as possible. And don't use a PWS because you find one on FedBizOpps. Agencies post a lot of crappy stuff on FedBizOpps.

Instead of messing with a PWS, simply require that the contractor provide translators with one of the following qualifications:

Translator certification/degree from an accredited university. See List of Translation Degree Programs.

American Translators Association (ATA) Certification, (see American Translators Association) or similar certification from another association. Here's a list of translators associations: List of Translators Associations.

If you haven't done it already, contact the U.S. Department of State, Office of Language Services, Translating Division. Their email address is: translation@state.gov. Ask them about "metrics" and performance work statements. If they don't use them, then you shouldn't either.

Link to comment
Share on other sites

Guest carl r culham

byrsch - If you are still looking and your legal expert is still confused as to what constitutes a legal and binding contract here is the solicitation link from FBO for the most recent award by the Department of State for translation services. Maybe it will help. Note it is stated as being the the PBA format and does require a QASP as part of the submissions.

https://www.fbo.gov/?s=opportunity&mode...re&_cview=1 (If the link will not work just "Copy" and "Paste" in your web browser and you should get to the document links).

Link to comment
Share on other sites

Guest Vern Edwards

Carl,

I followed the link to the synopsos and solicitation in question. I could not find the terms "performance based," PBA, or PWS. I found nothing labeled "performance work statement" and the statement of work included no metrics. The only mention of a quality assurance surveillance plan was in the proposal preparation instructions where it told offerors to submit one but did not prescribe specific content. There was no mention of measurable performance standards or acceptable quality levels (AQLs).

The synopsis and solicitation are complex, and I probably missed those things. Would you please point out where they are? Absent those things I cannot see why we should consider that acquisition performance based.

Link to comment
Share on other sites

Guest Vern Edwards

It does indeed. Thankyou. But I have looked throught the solicitation and could find no evidence of performance-based acquisition. The solicitation itself makes no mention of PBA that I could find. The SOW is not referred to as a PWS and does not contain any measurable performance standards. See FAR 37.601 (B), 37.602(B)(2) and (3), and 37.603. Instead, it describes translator position "assessment standards," such as:

• Accuracy – Translation must be free from major errors with respect to content (omissions or mistranslations), grammar, spelling, and punctuation.

• Consistency – Translations must be uniform with respect to the use of terminology to avoid confusion, ambiguity or obscurity

• Congruity – Translation must demonstrate correct and idiomatic usage of the target language including register. Candidates are expected to use prepositions correctly and idiomatically, to reproduce the register of the source text, to use appropriate language and to capture the tone of the source text and to avoid anachronistic or culturally inappropriate expressions. Candidates are expected to adhere to format instructions.

See the SOW at C.4.3. I'm not sure whether that is an employee qualification standard or a performance standard for individual translator employees, but it does not appear to be a contractor performance standard. I do not consider them to be measurable performance standards for the contractor.

The solicitation contains a "technical direction" clause, H.20, which makes the work subject to the the COR's instructions and direction, which is inconsistent with the PBA philosophy of specifying results instead of how-to and letting the contractor decide how best to do the work. See FAR 37.602(B)(1). The contract type is time-and-materials/labor-hour, which is hard to reconcile with PBA. The QASP mentioned in the proposal preparation instructions appears to be for the contractor's use, not the government's, although it is hard to say for certain.

I think the State Department did the old thing of calling it performance-based without making it performance-based.

Link to comment
Share on other sites

It does indeed. Thankyou. But I have looked throught the solicitation and could find no evidence of performance-based acquisition. The solicitation itself makes no mention of PBA that I could find. The SOW is not referred to as a PWS and does not contain any measurable performance standards. See FAR 37.601 (B), 37.602(B)(2) and (3), and 37.603. Instead, it describes translator position "assessment standards," such as:

See the SOW at C.4.3. I'm not sure whether that is an employee qualification standard or a performance standard for individual translator employees, but it does not appear to be a contractor performance standard. I do not consider them to be measurable performance standards for the contractor.

The solicitation contains a "technical direction" clause, H.20, which makes the work subject to the the COR's instructions and direction, which is inconsistent with the PBA philosophy of specifying results instead of how-to and letting the contractor decide how best to do the work. See FAR 37.602(B)(1). The contract type is time-and-materials/labor-hour, which is hard to reconcile with PBA. The QASP mentioned in the proposal preparation instructions appears to be for the contractor's use, not the government's, although it is hard to say for certain.

I think the State Department did the old thing of calling it performance-based without making it performance-based.

That is exactly what I am finding in my research statement like these that really do not describe anyway to measure performance to make certain we are receiving what we contracted for. Our current situation is a SOW that contains no way to measure the work performance of the contractor, the COR and PM are complaining of lack of performance but have nothing to hold the contractor to since no performance standards are in the SOW. They want to recompete using a PWS, which I agree is the way to go, my problem is I cannot find anything concrete to use as a performance measure to assure we are getting what we contract for. We may as well stay with what we have if we can't come up with objectives that can be measured.

I can't believe that no one has written any standards for translation in all the centuries we have been translating documents from one language to another. However; I am beginning to believe that I will have to write something myself and have legal review it. Should make for an interesting procurement and contract management exercise.

Link to comment
Share on other sites

Guest carl r culham

byrsch - I found these in about five minutes worth of Google searches. While you are right PBA does not always fit and is not always the way to go there is FAR preference so I have concnetrated on the PBA approach in my Google search. Hope all this helps and good luck on the effort.

http://www.dsca.mil/programs/biz-ops/dbc/2...olicitation.pdf

From FBO Solicitation No. W91WAW08R0153

TECHNICAL EXHIBIT 1

Performance Requirements Summary

The contractor service requirements are summarized into performance objectives that relate directly to mission essential items. The performance threshold briefly describes the minimum acceptable levels of service required for each requirement. These thresholds are critical to mission success.

Performance Objective

(The Service requiredusually a shall statement) Standard Performance Threshold (This is the maximum error rate. It could possibly be Zero deviation from standard)

Method of Surveillance

PRS # 1.

Contractor shall translate the QR from English to Arabic.

PWS 5.1 Translations are 100% accurate. No deviation 100% inspection by the U.S. State Department

PRS # 2 Contractor shall have translation and review process completed within 55 days.

PWS 5.2 As mandated by Congress No deviation 100% inspection COR

PRS #3

Contractor shall format Arabic versions of documents to be visually consistent with published English version PWS 5.3 Published English versions of documents Reasonable facsimile of English version, allowing for changes in pagination 100% inspection by COR

And food for thought from GAO report number GAO-08-1087

entitled 'Military Operations: DOD Needs to Address Contract Oversight

and Quality Assurance Issues for Contracts Used to Support Contingency

Operations' released on September 29, 2008.

"For example, quality assurance officials for the linguist contract were

unable to speak the language so they could not judge the quality of the

contractor?s work. Without adequate levels of qualified oversight

personnel, proper maintenance of contract files, and consistent

implementation of quality assurance principles, DOD may not be able to

determine whether contractors are meeting their contract requirements,

which raises the potential for waste. "

Link to comment
Share on other sites

byrsch,

After looking at this, forget about PBA. Trying to make something simple like translations (give contractor documents and they produce them in a different language) isn't really PBA. It's a joke describing that with a PWS. Performance measures would be what - 100% or something close. Do you think people will count works to see that the standard is met? If you, the deduction would likely very small and the administrative time and expense wouldn't be worth it.

I did a quick search just about everything I found is very simple - the contractor translates document and is paid by the word.

It's good you are trying to find better ways to contract but I would just leave it alone.

Link to comment
Share on other sites

Guest Vern Edwards

byrsch:

The GAO report mentioned by Carl is illuminating. How do you judge the quality of translation if you do not know the language being translated.

Forget performance based contracting and metrics.

Link to comment
Share on other sites

byrsch:

The GAO report mentioned by Carl is illuminating. How do you judge the qualify of translation if you do not know the language being translated.

Forget performance based contracting and metrics.

But what about using the 7 Steps to Performance Based Acquisition???

That was a joke. This thread is an example, in my opinion, of the overemphasis on PBA where COs try to apply it even where it appears truly good measures are impossible or impractical to implement.

Mike

Link to comment
Share on other sites

  • 2 weeks later...

There are industry standards for translation services.

http://www.thelanguagetranslation.com/tran...on-quality.html lists standards for translation services from different countries.

Here's the link to the ASTM Standard Guide to Quality Assurance in translations: http://www.astm.org/Standards/F2575.htm. It costs about $40.00.

If the COR and PM are complaining, something is wrong... Are the complaints about grammar or word choice? Did you provide a glossary of terms that they are not following? Is it a problem with the same translator or the whole translation service company? Are their reviewers and proof readers qualified? Is that specified in the contract - certification or qualifications of the personnel performing Q/A for the translation service?

Also, I just Googled GSA translation services and got a list of over 85 GSA contracts for translation. Why not use these?

Late to the party but hope it might help.

-B

Link to comment
Share on other sites

Glad you mentioned the ASTM standard. ASTM is always a good place to start with standards research. You have to pay for the resources, but they have an excellent reputation for high quality, peer reviewed & unbiased standards.

There are industry standards for translation services.

http://www.thelanguagetranslation.com/tran...on-quality.html lists standards for translation services from different countries.

Here's the link to the ASTM Standard Guide to Quality Assurance in translations: http://www.astm.org/Standards/F2575.htm. It costs about $40.00.

If the COR and PM are complaining, something is wrong... Are the complaints about grammar or word choice? Did you provide a glossary of terms that they are not following? Is it a problem with the same translator or the whole translation service company? Are their reviewers and proof readers qualified? Is that specified in the contract - certification or qualifications of the personnel performing Q/A for the translation service?

Also, I just Googled GSA translation services and got a list of over 85 GSA contracts for translation. Why not use these?

Late to the party but hope it might help.

-B

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...