Jump to content

Ways to Improve Acquisition Training in FY16


metteec

Recommended Posts

I would like your opinions to brainstorm ways that acquisition-related training can be improved. A few months ago, Jamaal had created a post concerning improvements to the acquisition systems; one of those items was training. I believe that improving training is an attainable opportunity this year. It is unlikely that we will see an overhaul of acquisition training system, but there are smaller changes that I think could enhance the major goals of training: 1) improving job performance; 2) measuring effectiveness; 3) determining future training needs.

I recently took a training course from an FAI-approved and DAU-certified Vendor for continuous learning points (CLPs) towards renewal of my FAC-C certification. The material was of decent quality; the instructor was knowledgeable; on paper, the students did great, since everyone received 100% on the test. The hidden academic integrity issues, however, hindered students’ ability to truly learn the material and reduced the importance of classes to merely checking off a box. To get the discussion started, here are a few solutions that I think could improve federal acquisition training (note that the problems apply towards both private and Government training providers):

1) Solution: Test answers closely guarded and pre-release harshly punished.
Issue: Everyone must pass mentality.

In nearly every FAI-approved training class, the instructor either specifically identified which questions would be on the test, or let us know to “highlight” a specific area in the text. In any other setting, this would be an academic integrity violation, with the professor being fired and students having to retake the test. In the Government academia, however, both the agencies and training provider have a mutual interest in ensuring everyone passes.

To resolve this issue, FAI should suspend or debar training providers that release test questions or identify answers with an intent to violate the integrity of the test. Students that received training certification from a class in such an instance should be required to retake the class.

2) Solution: Closed book tests
Issue: Open books as a crutch during testing

Acquisition-related courses routinely allow students to use their notes and book during the test. Open book exams, especially combined with #1 above, teach students how to effectively find the correct page, and not employ deep knowledge. The open book presents a crutch for the student so that they do not need to memorize content. By not memorizing content, it reduces that student’s chance to retain the information after class, and does not reflect knowledge of material during test time.

All classes should have closed book tests to measure knowledge. By having all tests as closed book, you force students to study and learn the training material and better apply that material on the job.

3) Solution: Training lasting the whole 8-hour day.
Issue: Class ending early, but still providing the full CLPs

All acquisition classes in the FAI/DAU framework include a number of CLPs. Those CLPs represent one hour of training. Training providers routinely let students out early on the last day, and sometimes on normal training days. One time, I was released from training at 11 a.m. while in a class that was supposed to end at 4 p.m. Students being released from training early is paramount to theft of taxpayer dollars. The agency paid for 40 hours of training but received only 34-36 hours. For any normal contract, you would send a show cause letter for why the Contractor failed to deliver as obligated. Further, it robs students of the opportunity to learn; during that four hours or more, the instructor could have provided additional instruction or a group activity to further hone the student’s knowledge.

All classes should be the duration of their advertised CLPs or the training provider should provide a partial refund. Classes that were not of the duration of DAU/FAI-approved classes should not count towards certification.

4) Solution: Expanded test and individual scorecards after class.
Issue: Training providers do not provide students and agencies with feedback for individual improvement.

Currently, students receive one or two tests, usually ranging from 25 to 50 questions each, respectively. Standard DAU/FAI policy is to receive an 80-percent score on those tests, or the student is required to retake the incorrect portion of the test. Hardly anyone fails these tests (namely because of the issues in #1 and 2). Afterwards, the student received his or her percentage score. This score is meaningless to the student and the agency because it does not identify areas for improvement.

Instead, tests questions should relate to specific categories, and results for those categories should be relayed back to those students with recommended additional training that could help enhance knowledge and improve effectiveness. Supervisors could use this information to target on-the-job training for employees.

Please let me know what ideas you have to improve training.

Link to comment
Share on other sites

Guest Vern Edwards

All of your ideas except number 3 are about testing. Testing is useful, but it isn't the main problem or the main road to improvement. The main problem is learning. Students must do that on their own.

No class should go for eight hours a day. That's much too long. People can't mentally attend that long. Six hours max, and even that's too long. The problem is that longer courses with shorter hours are not economically feasible.

Supervisors buy training for personnel en masse. They don't seem to send people to training based on individual needs. They send people to an introductory class who have twenty years experience. Supervisors need to do a better job of getting training for their people.

Supervisors don't explain to prospective students that they're being paid their salary while in class, that training is work, not bleep-off time, and that they are expected to consider it work and to work diligently.

Students don't take training seriously, even when they want training. They don't come prepared to learn. They show up for class without pencil and paper. They don't know how to take notes or how to organize them systematically for study. (They think they do, but most of them don't. Anybody hear of the Cornell method?) If you assign pre-course reading, half of the students don't read it and the other don't read it properly. They complain about having to do homework. They don't do the homework. A certain number of them arrive late for class every single day or come back late from lunch and breaks. They can't take their eyes off their bleeping phones. Good students are a rarity. Oh, for a class full of tiger learners, like I get at the Air Force Space and Missile Systems Center, where they have real leadership.

Supervised self-study groups would be the best way to learn, but for that you need good textbooks and disciplined students. There are not enough of either of those in government contracting.

Classroom training is not sufficiently well-coordinated with OJT. The greatest problem in contracting is that people are hired and paid before they know how to do the work they're being paid to do. They learn most of what they will ever know while on the job. What they learn is what the old-timers teach them. If the old-timers are not first rate, then the products of their teaching will not be first rate, unless the students are the kind of people who are motivated to learn on their own. People who want to learn, and who know how to learn, will learn, training or no training. You need special 1102s to conduct OJT. They should receive extra pay. It should be a hard job to get.

I have a hundred suggestions, but not enough time.

Link to comment
Share on other sites

My suggestion is to afford personnel the opportunity to use what they learn in self-study, class, or OJT. I personally don't believe we get enough reps doing things the right way simply because it usually means expanding lead times when the current goal is to reduce them. Secondly, make more resources available.

I find it peculiar and frustrating that there is a DoD Acquisition Workforce Development Fund but, nobody I've contacted seems to know how to tap into it.

I've requested books, classes (Far Bootcamp, Critical Thinking, Level III DAU courses, etc.), and subscriptions only to hear those dreadful words - we don't have the money.

I mention the fund and get blank stares or more questions than answers.

Link to comment
Share on other sites

Here are a few ideas:

1. No required training classes. Required competencies, developed and published by DAU/FAI, that employees must obtain using whatever method(s) they choose (self-study, online classes, DAU/FAI classes, commercial training organizations, etc.).

2. A series of exams on specific contracting topics that must be passed to advance in the career field, similar to what exists for actuaries: https://www.soa.org/Education/Exam-Req/edu-asa-req.aspx Exams developed by DAU/FAI. Exams given at commercial test centers. When the candidate is ready for the test they apply to DAU/FAI, who then mails the test to the candidate's local test center.

The Government's approach to training is to use a design specification (everyone must take these classes). We need to move to a performance specification (don't care how you get trained, as long as the results meet the standard).

From http://www.beanactuary.org/exams/:

Like other top-ranked professions (such as law and medicine), one must pass a set of examinations to achieve professional status as an actuary. Unlike other professions, in actuarial science you’ll have the opportunity to work as an actuary while completing the examination process—employers often allow study time during working hours, pay exam fees, provide internships, and even award raises for each exam passed. Though, to get the best start on a rewarding career, many soon-to-be actuaries begin taking exams while still in college. Of those that do, most achieve associateship in three to five years. All candidates acquire a core set of knowledge from required preliminary exams. The preliminary exams and Validation by Educational Experience requirements are the starting points for an actuarial career. The Casualty Actuarial Society (CAS) and the Society of Actuaries (SOA) websites contain complete descriptions of their education and examination systems, including syllabus and study materials, registration information, rules and regulations and resources of each exam.

Picture this:

Like other top-ranked professions (such as law and medicine), one must pass a set of examinations to achieve professional status as a contract manager. Unlike other professions, in contract management you’ll have the opportunity to work as a contract manager while completing the examination process—employers often allow study time during working hours, pay exam fees, provide internships, and even award raises for each exam passed. Though, to get the best start on a rewarding career, many soon-to-be contract managers begin taking exams while still in college. Of those that do, most achieve associateship in three to five years. All candidates acquire a core set of knowledge from required preliminary exams. The preliminary exams and Validation by Educational Experience requirements are the starting points for a contract management career. The Federal Acquisition Institute (FAI) and the Defense Acquisition University (DAU) websites contain complete descriptions of their education and examination systems, including syllabus and study materials, registration information, rules and regulations and resources of each exam.

Link to comment
Share on other sites

Guest Vern Edwards

People didn't seem enthusiastic about it to me, so I let it go. But I note that the number of viewers has grown. Or maybe it's the same 20 people checking over and over.

Link to comment
Share on other sites

Don't forget that contract specialists must master their automated procurement systems!

Capturing the right bits and bytes takes precedence over following the correct policies and procedures, using the correct provisions and clauses, and getting the best deal for the customer.

Link to comment
Share on other sites

How about rebalancing and redistributing the workforce in an effort to incentivize training and have unqualified people self-eliminate from consideration as contracting officers.

Maybe it's easier to implement stringent competency, education, testing, and certification requirements if we limit it to key positions - say contract specialist and contracting officers.

Bring in other 1100 series, outside of 1102s, to do the more routine or simplified actions as well as support the 1102s.

Then implement, as Don stated, required competencies, developed and published by DAU/FAI, that employees must obtain using whatever method(s) they choose (self-study, online classes, DAU/FAI classes, commercial training organizations, etc.). This pipeline could lead to 1102 positions, which should be significantly and progressively higher on the pay scale than the positions below and supporting it.

Money incentivizes many things. People could take the training mentioned above seriously if it paid to do do.

Special Pay Positions:

OJT Trainer

Special Qualifications

Reviewer/Policy

Just thinking out loud...

Link to comment
Share on other sites

Guest Vern Edwards

Don:

In my opinion, multiple choice questions are not appropriate for professional education. They're a sop to the weak student. Any exam question that both facilitates guessing and suggests the answer to guess at is an insult to the serious student. It just gives them a chance to pass whether they deserve to or not.

But you and I have argued about this in the past, and I don't expect to change your mind.

Vern

Link to comment
Share on other sites

Vern,

I haven't changed my mind. I would not agree that using multiple-choice questions is inappropriate for professional education as a general proposition. I think it depends on what you are attempting to assess. In terms of Bloom's cognitive levels, I think multiple-choice questions can be used to assess knowledge, comprehension, application, and analysis. However, they would not be appropriate for assessing higher cognitive levels (synthesis, evaluation). I acknowledge the risk of a student "guessing" successfully, but this can be somewhat mitigated by increasing the number of choices and having good distractors (wrong answers).

Link to comment
Share on other sites

Guest Vern Edwards

Don:

If I want to test knowledge, etc., instead of asking a question and then asking, "Which of the following is the correct answer?" why not just ask, "What's the answer?"

Instead of asking:

For which of the following contract types must the CO prepare a D&F... [a]... ... [c]... or [d]...?

why not just ask:

For which contract type(s) must the CO prepare a D&F?

Link to comment
Share on other sites

For contracting, I believe a written test similar to the Maryland Bar Exam, written test, would suffice. 10 questions - three hours:

http://mdcourts.gov/ble/examquestionsanswers.html

No multiple choice. Follow, the written test with an oral exam of some sort and call it good.

I'm not commenting on the extent of Maryland's Bar Exam because if its anything like California it has additional testing, the California Bar Examination consists of the General Bar Examination and the Attorneys’ Examination. The General Bar Examination has three parts: six essay questions, the Multistate Bar Examination (MBE, which is multiple choice), and two performance tests (PT, which is another narrative).

Link to comment
Share on other sites

I don't see how you improve the acquisition environment without better training the PMs and CORs. How much of the CO's workload would be reduced if PMs and CORs made better decisions?

Of course, I'm an outsider in this discussion (not a Government employee) so take my POV with a grain or two of salt.

H2H

Link to comment
Share on other sites

An organization can't accept failure in an individual employee when his position is so difficult to fill. Perhaps rationally the long-term benefit should outweigh the short-term pain, but in reality we don't see it that way. Every hire simply has to succeed when you employ the absolute minimum number of people in a job category.

In order to have high standards, we need to institute a farm team of potential talent. Then let the cream rise to the top. That also means there has to be a place for the non-star. I like Jamaal's comment #13 to get there. I think Vern has also spoken along similar lines.

Link to comment
Share on other sites

Just adding a question to the roiling multiple-choice debate.

I had a policitcal science professor who favored "multiple multiple choice" questions, and they were challenging. They looked something like this:

i.) Cost-plus-no-fee

ii.) Cost-plus-fixed-fee

iii.) Time-and-materials

iv.) labor hour

v.) fixed price with economic price adjustment

vi.) fixed price

For which of the preceding contract types must the CO prepare a D&F?

A. i & ii

B. i, ii, & iii

C. i

D. ii

E. vi, v & vi

F. i, ii, iii, iv, & v

He was a real character, if you were able to achieve a perfect 0 on one of his tests you received an "A" in the class and never had to return. If you inadvertantly got one wrong, well, you wound up with a 2. Never had sufficient command of that material to go for the perfect 0.

Link to comment
Share on other sites

Don:

If I want to test knowledge, etc., instead of asking a question and then asking, "Which of the following is the correct answer?" why not just ask, "What's the answer?"

Instead of asking:

For which of the following contract types must the CO prepare a D&F... [a]... ... [c]... or [d]...?

why not just ask:

For which contract type(s) must the CO prepare a D&F?

If I wanted to assess a student's ability to recall that information from memory, then a short answer test question would probably be more reliable than a multiple choice question. However, short answer questions become impractical if there are many right answers. For example, I could ask:

"Which numbers are odd?"

or, I could ask:

"Which numbers are odd?

a. 1

b. 2

c. 3

d. 4"

By using multiple-choice, I could use a slight variation of your question to assess the student at higher Bloom's Levels. For example:

"For which type of the following contracts must the CO prepare a D&F?

a. Requirements contract for mortuary services

b. Commercial time-and-materials contract with options

c. Cost-plus-award-fee contract for research that resulted from a Broad Agency Announcement

d. Service contract for an inherently governmental function"

This question requires the student to apply knowledge, determine what information is relevant, and use deductive reasoning.

Choosing a type of assessment item is like choosing a type of contract. No one type is inherently good or bad. Each type has its place depending on what you are trying to assess.

Link to comment
Share on other sites

Guest Vern Edwards

Q: Which numbers are odd?

A: All integers (whole numbers) that are not multiples of 2.

Your multiple choice approach would show only that the student knows (or does not know) that certain numbers are odd, but would not show that he knows why they are odd or what "odd" means.

Look, you're okay with multiple choice, and I don't like them. But I understand your point that if used properly they can be respectable.

Does DAU use them properly, as a general rule?

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...