HOME  |  CONTENTS  |  DISCUSSIONS  BLOG  |  QUICK-KITs|  STATES

Google

       Search WWW Search wifcon.com

Chapter 4

The Surveillance Plan

4-1. -- Basic Approach.

This chapter describes the major contents of a surveillance plan.  There are three key ideas that are the basis for a surveillance plan.

a.       Management By Exception.  Quality assurance relates to the output service provided by the contractor.  As pointed out earlier, the output service can result either from a contractor-developed procedure or from a government specified procedure.  When the procedure is specified by the government, compliance with the procedure is the desired output service.

(1)     When the output is based on a contractor developed procedure, the procedures are only looked at on a by-exception basis; that is, satisfactory performance of the output service as specified in the contract normally indicates that the contractor is using satisfactory procedures.  The government should be concerned only when services are not adequately performed.

(2)     In this case, the inspector looks beyond the level of services provided only to determine if the problem is caused by the government or the contractor.  If government provided items to the contractor’s operation (such as, parts, equipment, or facilities) are at fault, action must be taken through government channels to correct the problem.  No action will be required of the contractor.  When the problem is the contractor’s fault, the contractor is told to take corrective action.

b.       Performance Indicator.  The level of contractor provided services is monitored by checking the performance values in the statement of work (SOW).  As described in chapter 2, a performance value is a feature of the service that can be measured by a number.  For example, two important performance values in vehicle maintenance and vehicle operations are vehicle out-of-commission (VOC) rate and taxi response time.

c.       Problem Location.  When performance values show that the service is not adequately performed, the QAE uses decision tables to locate the problem.  The tables provide a logical sequence to find the problem cause.  Basically, they are a set of pointers which should find the problem’s source in a step-by-step fashion.  The construction and use of decision tables re described in paragraph 4-4b.

4-2. -- Surveillance Information Sources.

There are four principal sources of information for surveillance:  management information systems, random sampling, checklists, and formal customer complaints.  The following sections describe the information sources in detail.

a.       Management Information Systems.  In a few instances, an existing management information system (MIS) may be available as a means of surveillance.  When a MIS is available, as in the case of the Air Force’s vehicle integrated management system (VIMS) in the vehicle maintenance area, it can collect information on performance values which can be used instead of random sampling data.

(1)     Management information systems usually collect information for 100 percent of the activities for a specified period of time.  This information can be compared to a contract standard.  On the basis of this comparison, performance can be judged and performance for the specified period accepted or rejected.

(2)     For example, the vehicle out-of-commission (VOC) rate is computed every month by the VIMS.  A simple comparison of the VOC rate with the maximum acceptable VOC in the SOW explains a great deal about the level of maintenance service supporting the base vehicles and organizations.

(3)     By way of caution, however, one must check the data input into a MIS if the system is maintained by the contractor.  If one is going to use a MIS to check the contractor, make sure the MIS contains reliable data.

b.       Random Sampling.  The most frequently used way of service contract surveillance is random sampling.  Services are sampled by the QAE to determine if the contractor’s level of performance is acceptable.  Acceptance sampling is done, basically, to determine a course of action:  that is, whether to accept or reject the contractor’s level of performance during a given period of time.  If it rejects performance, certain actions are started.  If it accepts performance, no action is taken.

(1)     The basis for doing random sampling is MIL-STD-105D, Sampling Procedures and Tables for Inspection by Attributes which is widely understood and used by both the government and contractors.  It is based on the concept of an attribute.  An attribute is a feature of a service which either does, or does not, match a standard (For example, a taxi is on time or it is not on time).

(2)     When sampling by attributes, a certain number of observations will match the standards and the remaining number will not match.  Therefore, attribute sampling is useful for describing how a job is done, in terms of defects per hundred observations, or percent defective.  Using this concept, sampling for a performance indicator can be developed by proceeding through a number of formal steps based on MIL-STD-105D.  The use of these concepts is described in paragraph 4-3, Sampling Plan.

c.       Surveillance Checklists.  Checklists are also used to check contract performance.  They must be used sparingly, however.  The use of the MIS and random sampling are preferred information sources.  Checklists help in surveillance of contract requirements that happen infrequently.  (For example, if a contractor is required to perform a service once a month, this service would be included on a checklist).  Any service that is not provided on a daily basis should be considered for inclusion on a checklist unless a MIS can be used to determine the quality of the service.

d.       Formal Customer Complaints.  Even the best surveillance plan will not allow the QAE to check all aspects of the contractor’s performance.

(1)     Formal customer complaints are a means of documenting certain kinds of service problems.  The way to get and document customer complaints needs to be carefully planned by the persons checking the service contract.

(2)     Customer complaints are not truly random.  They are seldom used to reject a service or deduct money from the contractor.

(3)     When random sampling is the chosen method of surveillance, a customer complaint cannot be used to satisfy a random observation.  However, it can be used as further evidence of unsatisfactory performance if random sampling shows that the specific service is unsatisfactory.  These complaints can be used to decide if action other than a deduction should be taken.

(a)      Getting Customer Complaints.  An aggressive customer complaint program, once established, needs to be briefed to every organization that receives the contractor’s services.  An operating instruction should be given to each organization outlining the customer complaint program, the format and the content of a formal customer complaint, and the actin which can be expected from those assigned to watching and managing the service contract.

(b)     Documenting the Customer Complaint.  Normally, each customer complaint is brought, either in person or by telephone, to the person checking contract performance.  Enter information about the complaint into a Customer Complaint Record, similar to the sample shown in figure 4-1.  The record contains the following information:

1        Date and time of complaint.
2        Source of complaint -- organization and individual.
3        Nature of complaint (narrative description)
4        Contract reference of complaint related services.
5        Valid complaint (Yes or No).
6        Date contractor informed of complaint.
7        Action taken by contractor.
8        Signature of person receiving and validating the complaint.

4-3. -- Sampling Plan.

As a rule, a plan contains information on the acceptable quality level, lot size, sample size, and rejection level.  It states the number of units form each lot to be inspected (that is, the sample size).  It also states the criteria for determining the acceptability of the lot (acceptance and rejection numbers).  This information is used to build the sampling guide which are the major products in a surveillance plan for a service contract.

a.       Beginning the Plan.  To begin building a sampling plan, go to the Performance Requirements Summary developed during the “Write Statement of Work” step, chapter 3, figure 3-1.

(1)     This chart contains the required services, the standards, and acceptable quality levels.  At this time decide how the services will be checked (what information source or method of surveillance will be used).

(2)     Show these decisions on the chart.  For each service where random sampling is used, complete the steps described below.


Figure 4-1. -- Customer Complaint Record

Customer Complaint Record

Date and time of Complaint:         21 Jan 1979 /:1005

Source of Complaint:

Organization:  382 Bomb Wing/LGC

Individual:  Capt. John Murry

Nature of Complaint:  Called wrecker and it did not arrive until 3 hours after the request.
Contract Reference:  F5, para. 5.1.1.2.5 and Performance Requirements Summary
Validation:  Contract requires a 1 hour response time.  complaint is valid
Date and Time Contractor Informed of Complaint:  21 Jan 79/: 1030
Action Taken by Contractor:

Contractor had a person out sick and did not have a back up driver.  He now has not developed a roster of back up drivers who can operate a wrecker.

Received and Validated By:  H. Smyth/QAE

b.       Deciding on the Acceptable Quality Level (AQL).  The AQL is the highest number of defects per hundred, highest percent defective or highest number of defects that can be allowed for any service performance indicator.  There are only a limited number of AQLs listed in MIL-STD-105D but, in virtually all cases, one will be close enough to control the contractor’s level of service.

(1)     The first step in designing a sampling plan under MIL-STD 105D is the selection of a realistic AQL.  No service can be perfectly performed.  The AWLs placed on the Performance Requirements Summary in figure 3-1, must be adjusted at this time.

(2)     Find the closest AQL from figure 4-2 and use it to replace the original AQL on the Performance Requirement Summary.  For example, the AQL for taxi service might have been 5 percent.  This would be changed to 4 percent of 6.5 percent since 5 percent does not appear in the figure.

c.       Determining the Lot Size.  To determine the sample size, the lot size must be known. the lot is how often the contractor provides the service in a period of time.

(1)     To determine the lot size, estimate (or count) the frequency of the service to be sampled, during the period it is to be sampled.  Thus, if scheduled bus service timeliness is the service being sampled, and a sample is taken each month, the lot size is the number of times that are available during the month to observe bus timeliness.  In this case, it would be the number of times the buses go around all the routes each day, multiplied by the number of days in each moth on which the bus routes operate.

(2)     In the case of workorders, the monthly lot size can be estimated from historical information on file.  The projected workload data gathered in chapter 2 is used to help determine lot sizes.

d.       Determining the Sample Size.  Use figure 4-3 to identify an appropriate sample size for a given lot size.

(1)     Use the normal sample size column u unless there is a limited number of QAEs or unless the cost of an inspection suggests the use of the medium or small sample size column.

(2)     Use the medium or small sample size, if inspections for a particular service are lengthy or hinder the contractor’s ability to provide service to customers.


Figure 4-2. -- List of MIL-STD-105D Acceptable Quality Levels

Allowable Acceptable Quality Levels

0.010% 1.0%
0.015% 1.5%
0.025% 2.5%
0.040% 4.0%
0.065% 6.5%
0.10% 10.%
0.15% 15.%
0.25% 25.%
0.40% 40.%
0.65% 65.%

 Figure 4-3. -- Sample Size

Lot Size Normal Sample Medium Sample Small Sample
2-8 2 2 2
9-15 3 2 2
16-25 5 3 3
26-50 9 5 5
51-90 13 5 5
91-150 20 8 8
151-280 32 13 13
281-500 50 20 13
501-1.200 80 32 20
1,201-3,200 125 50 32
3,201-10,000 200 80 32
10,001-35,000 315 125 50
35,001-150,000 500 200 80
150,000-500,000 800 315 80
500,000 and over 1250 500 125

e.       Selecting the Rejection Level.  Use MIL-STD-105D to identify the acceptance and rejection level for the sample size (see figure 4-4.  To use the figure, begin with the known values for the AQL and the sample size.

(1)     Find the selected sample size (in the sample size column) and read across that line to the column for the selected AQL.  At that point there will either be two numbers or an arrow pointing up or down.

(2)     If there is an arrow, follow the direction of the arrow until it leads to a pair of numbers.  Of the two numbers at the intersection or at the end of the arrow, the number on the left (AC or accept) indicates the maximum number of defects which can occur in a sample and still permit the total size of 20, the sample size would become 32.

(4)     The number on the right (Re or reject) indicates the minimum number of defects that occur in a sample which causes the total group or lot to be judged unacceptable.  For example, suppose the sample size is determined to be 32 and the AQL has been set at 6.5 defects per hundred.  Find the number 32 in the sample size column and read across that line until the AQL column for 6.5 has been reached.  The two numbers at that intersection are 5 and 6.

(a)      In other words, the number on the left, 5, is the number of defects which can be found in a sample and still permit acceptance of the lot.

(b)     The number 6, to the right of 5, is the smallest number of defects needed to declare the lot unacceptable and subject to further check, using the decision tables.

4-4. -- Developing the Sampling Method.

The final thing to be decided in sampling is how the sample will be drawn.  The objective in the method is to insure that the sample is random (that is, that all services have an equal chance of being selected).  To achieve random selection, use a random number table, as explained in the following examples (see attachment 1 for the whole table).  Most items will fall into one of these examples.


Figure 4-4. -- MIL-STD-105D Acceptance, Rejection Levels for Normal Inspection

Note:         See Hardcopy for figure 4-4


Figure 4-5. -- How to Use a Random Number Table


a.       Use of the Random Number Table.  The random numbers in figure 4-5 are arranged in groups of five numbers (51259, 77452, and so on).

(1)     To use the table, begin by picking at random a group of numbers on any page of the table.  This is usually done by closing the eyes and pointing with a pencil or finger to some initial group.

(2)     To identify additional random numbers, follow a pattern.  Go along a given line to its end and then along the next line to its end and so on through the table until enough numbers have been selected or until the table ends.

(3)     If the table ends and there are still more numbers to select, go back to the beginning of the table and continue using the same pattern.  Use various patterns alternately; for example, use lines for one sample, use columns for the next sample, and use a diagonal pattern for the third sample.

b.       How to use the Random Number Table to Identify a Random Sample of Consecutively Numbered Workorders.  Suppose one has to identify a random sample of 3 workorders for inspection.  This can be done at the beginning of the month (before the workorders are written) or at the end of the month (to select workorders already on file).

(1)     If there are, or might be, 200 workorders to select from, then one begins by listing the lowest workorder number (known or projected).  This could be #001 or possibly #743, for example.

(a)      List the highest workorder number (known or projected); in this case, it could be #200 or #943.  With these boundaries now enter the random number table to the first group of numbers.  For this example, use workorders numbered #743 to #943.

(b)     If the last three digits in the first group of random numbers is not between 743 and 943, discard that group of numbers and go to the next group.

(2)     Again, using figure 4-5, if one starts at the initial 77452, disregard the two numbers to the left of the three significant digits, or in this case, 77.  The remaining number is 452.  Since this is not between 743 and 943, go to the next group in the same line which is 16308, again, discard the leftmost two numbers, and the number is 308.  This is again too low.

(3)     If the table ends and there are still more numbers to select, go back to the beginning of the table and continue using the same pattern.  Use various patterns alternately; for example, use lines for one sample, use columns for the next sample, and use a diagonal pattern for the third sample.

b.       How to use the Random Number Table to Identify a Random Sample of Consecutively Numbered Workorders.  Suppose one has to identify a random sample of 3 workorders for inspection.  This can be done at the beginning of the month (before the workorders are written) or at the end of the month (to select workorders already on file).

(1)     If there are, or might be, 200 workorders to select from, then one begins by listing the lowest workorder number (known or projected).  This could be #001 or possibly #743, for example.

(a)      List the highest workorder number (known or projected); in this case, it could be #200 or #943.  With these boundaries now enter the random number table to the first group of numbers.  For this example, use workorders numbered #743 to #943.

(b)     If the last three digits in the first group of random numbers is not between 743 and 943, discard that group of numbers and go to the next group.

(2)     Again, using figure 4-5, if one starts at the initial 77452, disregard the two numbers to the left of the three significant digits, or in this case, 77.  The remaining number is 452.  Since this is not between 743 and 943, go to the next group in the same line which is 16308, again, discard the leftmost two numbers, and the number is 308.  This is again too low.

(3)     Go to the next number, 60756.  The last part of this number, 756, falls within the brackets one is looking for, so workorder 756 is selected to be sampled.  The next random number group is 92144.  Since 144 is not within the brackets, move to the next group 49442.  Again, 442 is not within the brackets and therefore is not selected to be sampled.

(4)     Tlected.his process would be continued until three workorders are se

c.       How to Use The Random Number Table to Identify Random Sample From a List.  If a number of items need to be sampled that are not consecutively numbered, the simplest solution is to list the identifiers, for all the items in the lot, in a column, on a piece of lined paper.

(1)     Next, number the lines consecutively, beginning with the number one hundred.  Now use the random number table to draw the sample from the line numbers.  A selected line number leads to the identifier located on that line and that identifier tells which item to sample.  For example, if one chooses to sample a set of workorders with attached sales slips, one is not going to have to have a set of consecutively numbered workorders because not every workorder has a sales slip attached.

(2)     List the workorders with sales slips in a column, number each line in the column, and randomly select enough line numbers to make up the sample.

d.       How To Use The Random Number Table to Identify a Random Sample of Days.  Suppose one wants to identify 4 days in the month on which to sample something.  The days of the month can be numbered 01 to 31 (or less, as appropriate).  Begin in the random number table in figure 4-5 at 77452.

(1)     It is best to use a starting point different from the one used in the previous example but for the purpose of this example it is being used again.

(2)     One can move down the column from random number group to random number until the first number between 01 and 31 is spotted.  In this case, it is 23216 or, using the rule to discard the numbers to the left of the number or digits, simply 16.  Thus the 16th day of the month is selected for sampling.

(3)     Continuing in this fashion, one discovers that 58731, or simply 31, or the 31st, is the next day for sampling.  Proceed in this manner until the four days for sampling have been identified

(4)     If it is not desirable to sample on weekends, discard those days selected that happen to fall on a weekend and continue that selection until the proper number of days has been selected.

e.       How to Use the Random Number Table to Identify a Random sample of Times of Day.  If one wants to select random times of day to sample a service such as taxi or bus service, use the 24 hour clock.

(1)     If there are any constraints during each 24-hour period, take them into consideration.  For example, suppose that base bus service operates between 0700 and 2345.  In this case, go through the number table until one finds a group of four numbers that correspond to an acceptable time between 0700 and 2345.  Again, using figure 4-5, and proceeding across the line from the initial number, one comes to 60756 or 0756 hrs as the first random time.

(2)     The next random number 92144 or 2144 hrs.  The number is good and so one schedules an observation for 2144 hrs.

(3)     Proceed in this manner until the desired number of sample time shave been identified.

f.        How to Insure Variety in the Use of the Random Number Table.  The use of variety in the random table ensures that detectable patterns do not occur.

(1)     Besides starting at different random points and alternating the patterns for finding a string of random numbers, the user may, at some point in time, wish to use the first significant digits instead of the last.

(2)     For instance, in the random number group 77452 one has customarily used the last three digits (that is, 452) when looking for a random number with three digits.  But there is no reason why one could not for a period of time use the first three digits, of 774.

(3)     Success in using the tables requires consistency but also variety. the above information should ensure that the tables are properly used and that the sample is randomly drawn.

4-5. -- Surveillance Plan Products.

Several written documents are included in a surveillance plan:

a.       Sampling Guides.  A sampling guide is used for surveillance.  It is used in a surveillance plan to present the information needed to sample the performance of a particular service.  Information for the sampling guide is developed while the sampling information is being derived for the sampling plan.  The steps involved in developing sampling information are described in paragraph 4-3.  As shown in figure 4-6, a sampling guide has these sections:

(1)     A statement of the AQL and its meaning in layman’s terms.

(2)     The lot size for sampling.

(3)     The sample size.

(4)     A description of the sampling procedure which tells how the service will be sampled.

(5)     An explanation of the inspection procedure which tells what will be checked during the inspection of the sample.

(6)     Acceptable performance criteria which states the acceptance and rejection levels.

b.       QAE Decision Tables.  Once a problem has been discovered, the inspector must turn to a decision table and use the information in that table to aid him in finding the source of the problem.  The decision table lists the symptoms of the problem and identifies the possible sources of the problem.  Questions are established for each potential source to determine the contributing factors.  A decision logic entry is worked up for each required service.  As soon as it is considered satisfactory, the information is transferred to the decision table.  An example of a decision logic entry is shown as part of a decision table in the sample in figure 4-7.

c.       Checklists.  There are two main uses for checklists.

(1)     Tally Checklists.  Tally checklists are used to document all sample observations made during a sampling period.  Checklists may be preprinted with any format which contains the following information:

(a)      Contract requirements -- a statement of the service being inspected.

(b)     Date, time, entry for each observation.

(c)     Observation identifier of applicable workorder number, bus stop, or sales slip number, meal period, etc.

(d)     Result of observation -- satisfactory or defective.

(e)      Any pertinent comment for an observation.  An example of a checklist for sampling is shown in figure 4-8.

(2)     Surveillance Activity Checklists.  When a specific service or procedure occurs rarely or is not important enough to survey on a continuing basis, use a Surveillance Activity Checklist.  This type of checklist must be prepared and included in the surveillance plan.  An example of this kind of checklist is shown in figure 4-9.


Figure 4-6. -- Sampling Guide

VO Sampling Guide #3
Vehicle Condition Monitoring

1.     Acceptable Quality Level (AQL):  TO%

In the long run there must be no more than 10 defects per hundred vehicles.

2.     Lot Size:  _____ vehicles operated by the contractor.

3.     Sample Size:  _____ vehicles operated by the contractor.

4.     Sampling Procedure:

At the beginning of the month, list the registration numbers of all contractor operated vehicles on a sheet of ruled paper.  Beginning with the number 100, number the lines on the paper to correspond with the vehicle registration numbers.  Using the random number table select line numbers equal to the sample size.  The vehicle registration numbers on these lines indicate the vehicles to be sampled during the month.  Schedule the inspections evenly over the month.

5.     Inspection Procedure:

Inspect the vehicles using vehicle/equipment discrepancy and maintenance report as a guide (see AFM 77-310, Vol II, Chap 6).  Record defects per vehicle for each of the inspected vehicles.  Any defects found not already noted by the contractor shall cause the observation to be recorded as unsatisfactory.

6.       Performance Criteria:

a.       Performance is acceptable when _____ or less defective vehicles are discovered per month.
b.       Performance is unacceptable when _____ or more defective vehicles are discovered during a month.

7.       Phase-In-Period:  During the first two months of the contract the following AQL’s (paragraph 1) and performance criteria (paragraph 5) apply:

a.       AQL:  15%.
b.       Performance is acceptable if _____ or fewer defects are discovered per month.
c.       Performance is acceptable if _____ or more defects are discovered per month.


Figure 4-7. -- QAE Decision Table

If the Contractor’s
Deficiency is:
Probable Cause
Factors/Impacting
Conditions are:
Which Could
Result From:
Suggested Review Procedures and/or Preventive Measures
is/are:
Vehicle out of commission hours too high High vehicle down
for maintenance hours (VDM)
Poor control over work documentation Review PCNs N310032, N310030, or N310031 for individual vehicles reflecting VDM hours in excess of the reporting period available hours.  If this condition exists, it is usually found that two or more work orders were in “open” status for the same vehicle at the same time.  Normally results from failure to ensure “close” of a particular work order, and not reviewing the PCN N310018, Work Order Master File Status Report Previous to initiating a new work order.  (Frequently noted when two or more work orders were required to satisfy contract maintenance work requirements).

Are changes from VDM to VDP status accomplished in a timely manner?
    Insufficient manning Review PCN N310032 for individual vehicle data reflecting a very low ratio of direct labor compared to VDM hours accrued.  One prime indicator of this problem prevailing is when “Estimated Times in Commission” (ETIC), is constantly slipped.  (When work is hardly ever completed within time-frame allotted by labor hour estimates.
    Improper
control over work flow
Determine the appropriateness of workflow prioritizing and the disruptive effect caused by over-reacting to unrealistic application of priorities, such as shifting technicians from one vehicle to another, shuffling vehicles from one location to another for shop spaces, etc.

4-6. -- Contract Administrators Plan.

The contract administrator has an obligation to see how well the QAE is doing the job.  The contract administrator must also make some independent checks of contractor performance, preferably by using the same techniques that go into the design of the QA surveillance plan.  (This plan is completed at the same time as the surveillance plan).  As a minimum this plan must call for a quarterly review of the QAE’s use of sampling guides and an annual review of surveillance activity checklist items.


Figure 4-8. -- Sample Tally Checklist

Vehicle Operations Sampling Guide #3
Vehicle Conditioning Monitoring
Registration
Number
Date Time Satisfactory Unsatisfactory Remarks
B 7305 2 Oct 1530 X    
B 9763 2 Oct 1540   X Hood latch KLG
B 8764 2 Oct 1545 X    
B 0010 5 Oct 0900 X    
B 8764 5 Oct 0915 X    
B 7707 5 Oct 1345 X    
B 7706 5 Oct 1400 X    
B 9654 8 Oct 1000 X    
B 8752 8 Oct 1025   X Door handle KLG
B 3103 8 Oct 1045 X    
B2758 8 Oct 1600 X    

Figure 4-9. -- Surveillance Activity Checklist

Surveillance Activity Checklist
(To be performed (Daily)(Weekly)(Monthly), etc.)

Vehicle Authorization Utilization Board (VAUB)

Contract
Requirement
Contract
Paragraphing
Method of
Surveillance
Date
Accomplished
Where
Accomplished
Compliance
The contractor is required to hold a Vehicle Authorization Utilization Board (VAUB) quarterly. F-5.5.1.2.3 The contractor should have in his possession a copy of the VAUB
minutes.

There is no specific format for these minutes in AFM 77-310 but should conform to the standard format in AFR 10-1.  Minutes should adequately describe and document actions taken by VAUB.
     
The contractor is required to develop a vehicle priority list for VAUB approval. F-5.5.1.2.3.5 Verify approved priority list.  It should have the board president’s signature and date.      



Contents

Acknowledgement

Chapter 1

Chapter 2

Chapter 3

Chapter 4

Chapter 5

Chapter 6

 

ABOUT  l CONTACT