Jump to content

Data Madness?


Guest Vern Edwards

Recommended Posts

Guest Vern Edwards

How much time do you spend in front of an office computer doing data input?

Go to The New York Times online and look for an article that appeared in this Sunday's (January 16, 2016) opinion section ("SundayReview") entitled, "How Measurement Fails Doctors and Teachers," by Robert M. Wachter. Here is a quote:

[T]he measurement fad has spun out of control. There are so many different hospital ratings that more than 1,600 medical centers can now lay claim to being included on a “top 100,” “honor roll,” grade “A” or “best” hospitals list. Burnout rates for doctors top 50 percent, far higher than other professions. A 2013 study found that the electronic health record was a dominant culprit. Another 2013 study found that emergency room doctors clicked a mouse 4,000 times during a 10-hour shift. The computer systems have become the dark force behind quality measures.

Read the whole thing. After you've read it, think about CPARS and all the other data input that acquisition practitioners in government and industry must do. Think about the automated contract writing systems, SAM, etc. See any similarities?

I asked my eye doctor about electronic health records. He said he liked having records readily available, but that the amount of inputting that he has to do is over the top. He told me that he spends two to three hours every day after the clinic is closed just updating records. Sound familiar?

I like having info readily available on the internet, and I like the things I can do with computers. But how much of the data that must be input in the performance of acquisition work is really useful to you? We put a lot of info into FPDS, but how much of it do you look at? Do you even know how to find it? How real and useful is the info in CPARS?

Managers are data hogs. Here's a rule: If the amount of information available to a manager is n, the amount that the manager will want is always n + 1. The IT industry can take that to the bank.

IT is going to drive us all crazy, and I doubt that the madness will result in much improvement in anything. For all the gathering of data, has acquisition improved much? There will be change, of course, but not much in the way of "progress." In the meantime, 1102s, PMs, and contractors spend hours inputting, inputting, inputting. I think IT salespersons are the true successors of P.T. Barnum.

My poor Krell.

Link to comment
Share on other sites

Unfortunately, there seems to be an inverse correlation between keystrokes and professional knowledge. There was a time when one had to read the ASPR, DAR or FAR to create solicitation and contract documents. Reading the regulations to create the documents caused one to become knowledgeable in the provisions and clauses. But, that was last millennium. Now the automated procurement systems relieve 1102s of the obligation and benefits of reading the regulations.

Once a Luddite, always a Luddite.

Link to comment
Share on other sites

"In times long past, this agency was the home of a mighty, noble race of beings who called themselves the 1102s. Ethically and technologically they were a million years ahead of humankind, for in unlocking the mysteries of nature they had conquered even their baser selves, and when in the course of eons they had abolished sickness and insanity, crime and all injustice, they turned, still in high benevolence, upwards towards space. Then, having reached the heights, this all-but-divine race perished in a single night, and nothing was preserved above ground."

Link to comment
Share on other sites

I spend a sizeable amount of time doing data input (memoranda, minutes, determinations, CPARS, FPDS-NG, etc.).

The USD (AT&L) recently wrote an article that included this statement - "Data should drive policy. Outside my door a sign is posted that reads, 'In God We Trust; All Others Must Bring Data.'"

People want data. Why? Convenience. To operate in a task saturated life people often rely on data, generally, created by others. Outside of metrics and measurements of quality, look at the source selection process for example: The SSA gets a summary of a lot of data, with a recommendation, and makes a decision (a decision that must be summarized into another data point). Who hasn't used something like Consumer Reports, CNET, hotel ratings, etc?

As our reliance on generated data increases so does risk. Ever read a favorable review only to have a terrible experience? Maybe you didn't understand how the data was generated. How about, when people get overwhelmed and take shortcuts, which degrades the usefulness of the data? Not knowing anything about how data is generated reduces its usefulness in my opinion. What is the processes (what was done), structures (how the work was organized), in relation to generating the data.

I think this is further troubled by task consolidation (Jack of all trades, master of none). Empowered by data (good or bad), people are tasked with making decisions on things they may have little to no understanding of.

Link to comment
Share on other sites

"In times long past, this agency was the home of a mighty, noble race of beings who called themselves the 1102s. Ethically and technologically they were a million years ahead of humankind, for in unlocking the mysteries of nature they had conquered even their baser selves, and when in the course of eons they had abolished sickness and insanity, crime and all injustice, they turned, still in high benevolence, upwards towards space. Then, having reached the heights, this all-but-divine race perished in a single night, and nothing was preserved above ground."

SPS preserves a lot:

212.301 Solicitation provisions and contract clauses for the acquisition of commercial items.

See DoD Class Deviation 2013-O0019, Commercial Item Omnibus Clause for Acquisitions Using the Standard Procurement System, issued September 25, 2013. This class deviation allows the contracting officer to use the SPS clause logic capability to automatically select the clauses that are applicable to the specific solicitation and contract. The contracting officer shall ensure that the deviation clause is incorporated into these solicitations and contracts because the deviation clause fulfills the statutory requirements on auditing and subcontract clauses applicable to commercial items. The deviation also authorizes adjustments to the deviation clause required by future changes to the clause at 52.212-5 that are published in the FAR. This deviation is effective for five years, or until otherwise rescinded.

Link to comment
Share on other sites

To reinforce what Jamaal writes above, there exists a peculiar style of decision-making where a statistician analyses your organization's data and finds hard-to-discern correlations. If the mechanism behind the correlation is not understood, this can be dangerous. Discovering these mechanisms is not convenient. This is referred to as "Big Data". I think it is a gimmick to sell software.

To quarrel . . . I have hit-or-miss experiences with single reviews, but very good luck with the aggregation of a large number of reviews (e/g Amazon's 4/5 stars). I may just be lucky.

Link to comment
Share on other sites

Were only this a new or recent trend. Government and public administration have been engaged in including data and statistics as a decision making shortcut for some time.

In the early 1990's government was focused on "reinventing" itself, and started measuring results rather than outputs: http://files.eric.ed.gov/fulltext/ED367424.pdf

Also in the early 1990's, CompStat was introduced in New York as a means to use data to track crimes and focus assets in the areas where the crimes were occuring: https://en.wikipedia.org/wiki/CompStat Coincidentally, this method never achieved any success outside New York, suggesting that it was the right solution, for the right location, in the right environment. Statistically driven police measures never bore the same fruits as it did for New York, so some wonder if there were other reasons for the success rather than this data-driven technique (never mind other massaging the crime statistics or under-reporting crimes).

Simultaneously we have the "New Public Management" approach to public administration, which uses contracts as a coordinating function: https://en.wikipedia.org/wiki/New_public_management

This may get even more tedious for contracting personnel. OMBs "category management" initiative where they will engage in government-wide analysis of spend data and purchasing behavior: https://www.whitehouse.gov/sites/default/files/omb/procurement/memo/simplifying-federal-procurement-to-improve-performance-drive-innovation-increase-savings.pdf

Experience would suggest that this effort is driven primarily by two interests: a) technologists (and the commercial adoption of very robust tools that capture an incredible amount of information), and b ) "data scientists" (which is just a big, fat sexy name for nothing more than statisticians). Because b doesn't have much subject matter expertise they will always call for more and more data, which means greater investment in a.

Aside from responsible statistical execution, for data to produce results one needs to know a) the data sets, b ) where they come from, and c) what is being said and not being said. There are examples where this is being done and done effectively (for example, IRS using data analytics for increased fraud detection). But from a contractual perspective, that agency A is paying price x for a commodity, and agency B is paying price y for the same commodity is not very meaningful in-and-of itself. One has to consider pricing strategies and how those commodities are bundled with other services or complimentary commodity packages. How price correlates with terms will be completely lost when looking at trees rather than the requirements and terms driven forest that accounts for that price of that tree.

I am a little concerned that this kind of subject matter expertise is being lost on this effort at this point in time. I am also a bit concerned that the data will never be complete for those who want to conduct analysis without that expertise, and that every contracting officer in government will be asked to modify existing agreements to account for the data need (no...can't do that looking back but could consider doing so moving forward).

I am not experienced enough in administrative changes to know what is sticky and what is schticky, but looking back on GAO reports and OMB memos over the past 20 years concerning private sector procurement practice adoption and recommendations into the federal sector, this data-driven trend looks to me to be something that will cross administrations.

Link to comment
Share on other sites

  • 2 weeks later...

It's all about forecasting.  In the past, companies didn't have access to such large quantities of data, so complex forecasting did no better than simple trend analysis.  Now that companies have access to such large amounts of data, they can determine correlations that will give them insight into consumer habits.  Quant funds on Wall Street rarely hire any finance graduates anymore, instead choosing to concentrate their resources on math and physics PHDs.  It turns out that chaos theory has significant implications in financial modeling and genetic algorithm development.  It's all about probabilities and their implementation into decision trees.  This also contributes to the strong efficient-market hypothesis by eliminating arbitrage opportunities.  In today's market environment, which has been heavily politicized, and manipulated by entities with  severe conflicts of interest, this type of trading is one of the only ways to make a profit.  This is why several macroeconomic hedge fund managers have thrown in the towel and closed up shop.

Analytics are the future.  The best way to beat your opponent is to know exactly what he or she is going to do next.  In the absence of being able to tell what your opponent does next, you want to have the highest probability possible of knowing what your opponent is going to do, and you want to know as quickly as possible.  The implications for the military are significant.  You could potentially form character profiles on every person in the world and accurately predict their actions.  While some of the things we as contract specialists do seem redundant, and actually are redundant in many cases, we don't really know the full story of who has access to this information and what they intend to do with it.  A private sector corporation would likely use this data to improve their logistical network capability.  Unfortunately, due to the rigidity of contract compliance, the government is not nearly as flexible in its ability to form strategic relationships with vendors in order to best take advantage of these opportunities.

I imagine that much of the information will be utilized to effectively force shape the defense department in the future.  In the age of directed energy, space vehicles, cyber weapons, and unmanned vehicles, the need for human capital is increasingly unnecessary.  Additionally, due to some major geo-strategic mishaps, resulting in an international exodus from the USD, the need for a leaner and more technologically advanced military has been expedited substantially.  Big data will be the key to getting rid of waste and automating systems to work faster and more efficiently.

I disagree that the reliance on data necessarily increases risk.  A human mind simply does not have the capacity and the speed to handle making decisions based on massive amounts of data, at least not nearly the same extent that computers do.  Automation is changing the world and in enough time there will be very few businesses where automation doesn't make previous processes obsolete.  It will be interesting to see how automation affects contracting in the future.  However, it will be up to competent subject matter experts to think of ways that IT processes can simplify and standardize contracting in such a way where it decreases lead time.  Somehow, I doubt this is the case, especially considering I heard a rumor that there are multiple procurement systems being developed for different departments to replace PD2.  If this is true, it seems like it will result in millions of dollars of waste.

 

Link to comment
Share on other sites

Guest Vern Edwards
Quote

Analytics are the future.  The best way to beat your opponent is to know exactly what he or she is going to do next.  In the absence of being able to tell what your opponent does next, you want to have the highest probability possible of knowing what your opponent is going to do, and you want to know as quickly as possible.

Well, maybe.

If analytics are the future it's because they've got good salesmen and gullible customers. I'm not sure that they have so far enabled the U.S. to predict what our opponents are going to do next. If they have, they haven't improved our responses.

In any case, we're talking about the acquisition system. (Who are the opponents?) Analytics have not kept the F-35 or the Litoral Combat Ship on time and within budget; they haven't ensured that those systems perform up to expectations; and they haven't made anybody happier with the performance of the acquisition system. Yet, we're sure to get more analytics, because if the bosses have n data, they will want n +1, and there is always a salesman ready to get it for them. The truth, apparently, is out there.

As for your statement: "Big data will be the key to getting rid of waste and automating systems to work faster and more efficiently," we'll remember you said that. We're talking about a system in which the use of numerical scoring in source selection is prohibited because decision makers find it too confusing.

But enjoy your data entry duties, Bagheera. There will be more to come, and the system needs more people who think like you.

Link to comment
Share on other sites

Hello, Bagheera,

You may find a skeptical crowd here because this requires a long series of events (data entry, data collection, data analysis, decision-making) where everything has to work well.  Each link in that chain is weak. 

The idea is a beautiful one, but I don't think it will work, and will pull resources away from other things.  Never forget the human decision-maker at the end of the chain.  They will be impervious to whatever "chaos theory" has to offer because . . . THEY HAVE NO IDEA WHAT CHAOS THEORY IS.  :(

Link to comment
Share on other sites

  • 3 weeks later...

I've been experimenting using existing data sets (FPDS and locally generated reports) to measure the performance of our contract specialists and our organization generally. Low level stuff really: How effective are we are spending appropriated funds throughout the year rather than waiting until the 4th quarter for execution? The idea is that organizations get better procurement outcomes when the contracting office has the proper amount of time to develop an optimal procurement strategy. Too often we get crappy SOWs dumped on us in August-September so we end up with crappy contracts. If I can use FPDS data to show the 4th quarter spike, and then explain the consequences of that spike, I might be able to get the point across to the program offices.

On a contract specialist level, I can make sure that I'm not pigeon-holing a specialist with a certain type of procurement or customer by tracking the types of procurement actions they are awarding. It helps to identify who is performing and identify potential skills gaps. Using local reports, I can track our PALT* times by office and by individual.

I actually read "The PerformanceStat Potential" recently by Robert Behn. Behn was involved in NYC's CompStat program. He says in the book that there is no perfect performance measure. So I look at my modest attempts at using big data as an element in a larger picture of performance.

One last point is that the government generates an insane amount of data. Rather than trying to develop a new database or dataset, I'd recommend examining what already exists to see if that will fit your needs.

*Usual caveats about PALT. It's not always useful or indicative of performance and it certainly doesn't speak to quality.

Link to comment
Share on other sites

  • 2 weeks later...

Garbage in garbage out is an adage most of us are familiar with, but what if it wasn't humans inputting that data in? While probably a decade away from anything mainstream coming out, Google's AI system, DeepMind, has proven that AI will soon be capable of replacing humans at many tasks and be far better at it than many of us ever will. I think one of the career fields that could be heavily vested with AI systems is the contracting field. The Government is supposed to work on a win-win basis during negotiations. If an AI system has all this internal data and then can read what a contractor is proposing, all that data going in can come out with that scenario in mind that is a win-win. Poor contractor performance? I would easily bet that an AI system would be able to have an algorithm written that it can take into context systems like CPARS and FAPIIS and create forecasts for how that company would perform on future contracts. Certainly not a perfect system for the career field, but it could put a lot of people out of a job because it could capture so much "large data" and actually process it into something meaningful.

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...