Bacon's Rebellion

Northrop Grumman: Myths About Privatization

—

by

James A. Bacon
in

If you live in Virginia and need to get a driver’s license or ID card at one of the state’s 74 Department of Motor Vehicles offices today, you are out of luck. Still due a tax refund? Wait a little longer. The Department of Taxation’s computers don’t work. Worried about pollution? Chill out until the Department of Environmental Quality can get its systems up and running.
What happened? Northrop Grumman has struck again. The firm won a $2.3 billion contract — the largest state pact ever — in 2005 to provide the state with communications and computer services. Another state agency, the Virginia Information Technologies Agency (VITA), is supposed to help Northrop Grumman run the system.
The tale once again shows just how privatization can get really fouled up and how its benefits can by mythical. Exhibit A is, of course, Northrop Grumman, the huge defense corporation that Virginia officials persuaded to bring its headquarters here this year with a goodie basket worth millions in tax breaks and incentives.
The state IT system has been a mess. It has had regular outages, forcing a game of musical chairs at VITA top management. The latest mishap occurred Aug. 25 when a couple of small circuit boards fried. Affected were 26 state agencies. All but six were up and running this morning.
There’s plenty of blame to spread around. Former governor Mark Warner, a Democrat who now represents the state as a senator, was so enamored with the then-fashionable privatization concept back in 2005 that he entered into this outsourcing deal. At the time, state computing was a mishmash of incompatible systems.
Republican Gov. Robert F. McDonnell inherited the mess, plus about $75,000 in campaign contributions from Northrop Grumman. During the campaign, he made some noises about the issue but piped down. Why? Virginia was in the running to snag Northrop Grumman’s Los Angeles headquarters. It competed against Maryland and the District and won the relocation derby by offering up more than $13 million in state incentives and breaks, plus a like amount from Fairfax County.
Now we’re stuck with Northrop Grumman. Once again, Virginia’s naive leaders skipped down the yellow brick road of privatization. They support limited government because old Thomas Jefferson did. They are still in love with the era of Margaret Thatcher and Ronald Reagan, who wanted to get government off our backs — through privatization, of course.
Then-Gov. George Allen pushed the concept further in the 1990s, with the state’s law to sidestep transportation finance shortfalls by farming out state highway responsibilities to private infrastructure firms. Warner, a businessman who made his millions selling cellphone bandwidth, marched in tune.
And that is why you can’t get a driver’s license in Virginia today or tomorrow or maybe not until Friday.
Peter Galuszka

Share this article

ADVERTISEMENT

(comments below)

Comments

Comments

21 responses to “Northrop Grumman: Myths About Privatization”

  1. Hydra Avatar
    Hydra

    Nothing wrong with having contractors. Having contractors is not the ame as privitization. Privitization would mena that Northrop Grumman issued drivers licenses and leased the right to do that from the state.

    In that case NG would get it right, or go broke.

    But, if yu just have contractors doing government work,then government still has to write the performance specifications. sometimes, that is as hard as doing the job, which is why the specs did not require card level redundancy.

  2. Gooze Views Avatar
    Gooze Views

    Hydra,
    I disagree. The state turned its IT system over to NG. This is not the same as having contractors which are ubiquitous.

    PG

  3. Larry G Avatar
    Larry G

    Just wanted to point out that no less than RIM (blackberries), Apple and GOGGLE have experienced major scope outages.

    And I can tell you how every one of them happens.

    Someone makes a change.

    that's right.

    It's usually a poorly-conceived change… usually billed by the programmer as a "minor" tweak and that in and of itself is human nature but management should never be held hostage by computer geeks…

    the second part of the equation is definitely management's fault because the question that must be answered for any change is "what happens if it breaks? How do we fall back to a working state"?

    there's a little known field known as configuration management and in the world of computers and networks – especially large ones – documenting your system and properly testing changes, and having a fall back as part of any planned change is mandatory.

    The problem has always been that you put a geek in charge of the geek staff – and you get geek standards – and folks that's an oxymoron.

    I'll bet everyone here that the problem is not a fried circuit board – but a change made to the system.

    Usually when something like this happens all the geeks an all the folk supposedly in charge of them lie their butts off and try to convince management that a board "fried".

  4. Anonymous Avatar
    Anonymous

    Government needs to develop employees with the skills to manage contracts. Most businesses that hire a contractor manage their contractors. We need government to do the same thing.

    TMT

  5. Larry G Avatar
    Larry G

    I agree. You can bet your butsky that there are some State of Virginia employees involved in this.

    And they are using all ten fingers to point the blame at McD.

  6. Groveton Avatar
    Groveton

    Well … since everybody is guessing maybe I'll guess too.

    Let's start at the begining. Once upon a time I imagine that every agency in Virginia had their own infrastructure. Their own data centers, their own networks, etc. Just like the US federal government still does.

    Somebody said, "Hey man, all that redundant infrastructure is mighty expensive. Let's use a state-wide common infrastrcuture and really save some money.".

    The state tried to bring all the little infrastructures together into one big infrastructure. But that's hard. Some applications built on one infrastructure won't run on a different infrastructure. Nobody at the agencies had experience with the envisioned cool new common infrastructure.

    Somebody said, "Hey, let's outsource the whole magilla.".

    I assume there was an RFP and a bidding process and I assume Northrup Grumman won.

    NG starts the long process of converting agency after agency to a shared infrastructure. Information is imperfect so NG finds problems with their original plans.

    Eventually most of the common infrastructure is built and most of the agencies are running on it.

    Then, BOOM – either somebody changes something or a couple of circuit boards (most likely SAN controllers) fry.

    Now comes the problem …

    Not all agencies are on the hot sites for backup. Or, not all agencies are at the same level of disaster recovery. Meanwhile, nobody has tested the ability of the whole infrastructure to be recovered from a system-wide failure.

    The physical problem is fixed (which takes a while) and the restart begins. But the restart doesn't work. Oh dear. Now we've gone from fried circuit boards to some database or routing table out of synch somewhere.

    The techies begin what I call metaphysical diagnosis. They don't know what's wrong so they sit in a big room and theorize. To techies, this is nirvana. Point meets counter-point, debate rages, someone goes out for more coffee.

    Eventually, a hypothesis is floated. It's usually one that involves a clever fix that doesn't requires the systems to be rolled back and restored to a prior state. It always fails. Tick, tock. Time for more debate, time for more "noodling" on the problem. This time somebody starts looking at the logs and somebody takes the shrink wrap off the documentation (metaphorically).

    A second rescue attempt is tried (remember the golf balls in the BP oil well?). Hope springs eternal. The second attempt also fails

    Finally, somebody with some sense says – roll back all the systems. One techies quips, "That will take another day". A different techie posits, "We'll lose some of the data in some of the systems.". Then the person with the sense says, "I'd rather restore to last Tuesday night and spend a day doing it then spin around the axel hoping one of you dreams up an ever so clever solution.".

    Dejected, the techies start the full restore.

    Someday, somebody will figure out why the original recovery didn't work. Somebody with some sense might even insist that another system-wide failure be forced over a holiday or long weekend to ensure that the system can actually be recovered. The techies will all stand around saying that they knew what the problem was when the outage first occurred but nobody will listen to them.

  7. Anonymous Avatar
    Anonymous

    If the Commonwealth of Virginia was a private company, it would have dumped Northrop Grumman for the service its getting. I'm currently and a State employee who worked for a fallen Richmond company and I guarantee you, NO Fortune 1000/S&P 500 company would stand the service Virginia is getting at this cost.

    The VITA-NG "partnership" is a colossal failure.

  8. Hydra Avatar
    Hydra

    Still, this is a contract, not a privatization like selling Vabc.

  9. Larry G Avatar
    Larry G

    The problem is that when separate state agencies do their computer systems without coordination – they are not interoperable and more important – the network is not secure.

    Computers are bought and configured that not only are not compatible with other computers but they are equipped and configured with stuff that the techs have no training in dealing with.

    so when one breaks – it's a lot like you have a Heinz57 Refrigerator and the repairman has no clue about how to diagnose and repair it.

    On this issue, I am not guessing. This is what I did for a living in the past.

    the biggest challenge or any large enterprise is to create and maintain a company-wide computer network because human nature is for each person to have their own tailored to their preferences – and that does not work.

    And when they break – they take down an agency like the DMV – not for an hour but for days.

    Maintaining a large computer network is difficult work that requires substantial discipline to do thing "right" rather than take shortcuts that then turn into little time bombs ….

    there is nothing inherently wrong with contracting functions or even privatization.

    For instance, most HVAC systems are done on contract not with in-house personnel and your business is dead in the water in the HVAC fails.

    When you contract – it becomes important to adopt and use industry-standard practices and designs because if you let the vendor build a system that only that vendor can maintain – you're turned over your business to him.

    Smart companies and smart state agencies know what to contract out and as TMT said – HOW to contract out.

    There's a field in the govt called Contract Monitoring.

    It requires the person doing it to pretty much know how what is being contracting out – works and it is up to that person to write and maintain the contract in such a way as to not have it become something that the company or agency really has no control over.

    Contracting out or even privatization does NOT mean you are turning the whole enchilada over to someone else.

    It means you're turning the work over to someone else and you retain responsibility for how the work gets done.

    The next time you go into WalMart, take a look at their scanner checkout systems – and you'll notice that the name on them is not "WalMart" but IBM.

    You may also notice how many times that you've shopped at a WalMart that their systems were down and they had to close.

    It's very, very rare.

    WalMart most likely did not actually design and build it's check-out systems.

    They more than likely wrote requirements and contracted it out – even though they still maintain control of the operations.

    But when one of their scanners breaks – the contract they've written requires 24/7 response time and when is the last time that WalMart had multiple stores "down" because of a computer "glitch"?

    The last think in the world you do not want to hear from your computer guys when your whole system is down is – " we're not sure what is wrong yet".

  10. James A. Bacon Avatar
    James A. Bacon

    I have to say, these latest outages are really damaging to Northrop Grumman. The privatization was originally billed as a way to rationalize the state's IT system and save tens of millions of dollars annually. Those savings never materialized. Indeed, what we got instead was cost overruns.

    It was possible to justify the lack of savings by claiming that at least the state got a much more robust, safer system with built-in redundancies, etc. that could maintainin the continuity of IT services through everything from terrorist attacks and hurricances. Well, we have had neither a terrorist attack nor a hurricane, nor not even so much as a bad thunder storm, and the state systems are down.

    Northrop Grumman has a whole lot of 'splaining to do. As any customer would, the state needs to whole NG's feet to the fire. It will be interesting to see what performance standards are contained in the contract, and how much in damages the state will be able to collect.

  11. Larry G Avatar
    Larry G

    The Navy's NCMI system did the same thing and Perot Systems was involved in theirs…. as well as other major players.

    The problem is that when you have a bunch of disparate systems that belong to the same entity – like a State or a Corporate – trying to integrate them is almost always more expensive and more difficult than envisioned.

    And part of the reason is that the contractor will seek as much information as possible from the State – but the folks who represent the state itself often don't really know

    … because…

    .. and this is the best part…

    ..when organizations started to computerize – the guys who got the job to computerize them –

    were often hacks as opposed to people with degrees in Information Technology ..

    and they built systems the same way that software was developed.

    If you ever used older software and got frustrated with the odd and weird …non-intuitive ways that it would work…

    apply that logic to a computer network for a state agency…

    then another agency would be done – not according to some standard that guided both agencies -…

    nope.. but according to whatever the geeks in each agency wanted to do.

    So one agency would have Microsoft PC platforms.. while other agencies would have Linux boxes – with freeware and no responsible support…

    they'd buy new PCs that were very different from the existing ones – and they would not integrate without hacks..

    you get the picture…

    you have a rube goldberg system and the contractor tries hard to get a fix on it – from clueless managers…

    and then once they actually get into it – it's a nightmare…

    and you have to fix it …essentially like you'd overhaul a car – while it's in gear and on the highway because the system has to stay up 24/7.

    Every change, every upgrade is a risky affair…

    the "polite" word that is used to describe these agency-specific systems is "legacy".

    The reality is that most every major computer network restructuring done – whether it's done by contractor or in-house usually is chaotic mess.

    Only in the last few years has there been more discipline in the process.

    But I agree.. the situation has harmed the reputations of those involved.

    DMV is still down this morning.

    There going to be a lot of folks hating life before this is over…

  12. Groveton Avatar
    Groveton

    "The Navy's NCMI system did the same thing and Perot Systems was involved in theirs…. as well as other major players.".

    The Navy Marine Corps Internet?

    NMCI?

    If so, I think you mean EDS, not Perot.

  13. Larry G Avatar
    Larry G

    it's a little convoluted…

    " Perot founded Electronic Data Systems (EDS) in 1962, sold the company to General Motors in 1984, and founded Perot Systems in 1988. Perot Systems was bought by Dell for $3.9 billion in 2009. "

    " HP Enterprise Services, is a global business and technology services company, previously known as Electronic Data Systems (EDS), headquartered in Plano, Texas that defined the outsourcing business when it was established in 1962 by H. Ross Perot. General Motors acquired the company in 1984, spun it off again as an independent company in 1996, and became an EDS client."

    But the Navy (and all of DOD, in fact) had a heck of a time trying to integrate and standardize their systems.

    one undocumented change in a system can literally drop a hundred or a thousand computers with even the best analysts stymied because they have no knowledge of what was changed.

    At one point, we were using a product called TripWire which would generate a report every morning of every file that was changed in the last 24 years with the most important system files listed first.

    If we could not account for every change to a system file, we had to track down what happened and why

    one of the reasons the Navy went with NMCI – was to gain control of the Corporate Network – and to impose security requirements on any and all agency computers that connected to the network.

    People were bringing media in from home or receiving email attachments with viruses that would infect the entire network in a matter of hours and something had to be done to protect the network.

    I'm quite sure that this is part of why Va has also gone in this direction.

    and it makes more sense to contract it out because you want an operation that can quickly hire and fire people and to move quickly to acquire needed talent for evolving technologies.

  14. Larry G Avatar
    Larry G

    24 years….NOT!…. 24 hours..

  15. Hydra Avatar
    Hydra

    the biggest challenge or any large enterprise is to create and maintain a company-wide computer network because human nature is for each person to have their own tailored to their preferences – and that does not work.

    ===================================

    Sounds like EMRs complaint about good individual choices leading to collective failure.

  16. Hydra Avatar
    Hydra

    Privatization implies ownership.

    If the state winds up taking over the HOT lanes, they are going to have to buy them from Fluor.

    But if the state cancels the NG contract they will still own the computers and the networks. The state might not own some proprietary software the NG uses to operate the networks.

    I still don;t see anything about this that resembles privatisation.

  17. Hydra Avatar
    Hydra

    If you ever used older software and got frustrated with the odd and weird …non-intuitive ways that it would work…

    apply that logic to a computer network for a state agency…

    =================================

    Every software engineer I ever met uses that excuse or one like it. When I had to manage a development group, their lack of discipline and method astonished me.

    I used to tell them, "Look, if you put a stinger missile on a soldiers shoulder and pull the trigger, you don't get any release 1.1. It has to work the first time, and every time. That's the way I want your software."

    They just didn't get it. there is no culture of quality, just as there is no culture of safety at Metro.

  18. Moira Avatar
    Moira

    Splitting hairs over semantics of privatization and contractors–besides the point.

    Don't you contract out to avoid the very problems that have occurred? You don't trust your own employees, whom you assume are dumb-fucks who can't engineer their way out of a paper bag, in favor of an outsider who is in business for themselves, therefore, surely they know what they're doing? Oh, the irony.

    P.S. Dumb me–I thought Northrup Grumman was coming here to make aircraft. Ha!

  19. Anonymous Avatar
    Anonymous

    It's easy to build a network. Neat little Visio Drawings, a standard package of hardware and common applications. Throw in some off the shelf security monitoring and you're pretty much done.

    Then along comes the customer apps and the fun starts. This app was built for Oracle BC and doesn't upgrade. This package uses some off the wall graphic interface that isn't made anymore, and even worse has so many security holes a Trojan brontosaurus could walk in and no one would know. Then there is the critical app everyone needs that won't run on a regular user account without full admin permissions.

    That's when the geeking kicks in. We can simulate Oracle BC with this value added package if we change these parameters. We can block most of the security holes by putting this policy into place. And if we change the permissions on this file and that file and a registry edit here and there, the critical app will run with a user account. Oh yeah, here's the bill.

    Then the geeks move on to other career opportunities, never documenting anything before they left which didn't matter because the legacy stuff wasn't documented either. And EDS or NG gets left with a nasty reputation when the geek replacements 'fix' the system.

    It's not just the government either. I worked for a company that nearly went bankrupt because they changed their financial management software and added new hardware to run it. The new app didn't have the same features of the old one, and wasn't compatible with the database. A year and a half went by while they hired a host of data entry people to manually transcribe info into the new database, along with a high paid consultant to custom program new requirements and front end web pages for their mandatory government reports. For some reason planning always stops at the hardware stage.

  20. Hydra Avatar
    Hydra

    "Splitting hairs over semantics of privatization and contractors–besides the point."

    Not at all. If you privatize something and later want to take control of it back, you are going to have to BUY it back, like the ABC stores or the HOT lanes.

    But if you have contracted out, all you have to do is fire your contractor and find a better one, if there is such a thing.

    But the key point with hiring a subcontractor is to define the work and the level of performance with measurable criteria.

    Then, if they don't perform, you are still screwed – you still can't issue drivers licenses, but now you don;t have to pay the contractor or you can exact fines against future payments.

    Managing a contractor may be as hard as doing th eactual work, but it is a different skill set. But if it takes you as many people to manage the contractor as it takes the contractor to do the work, then where is your cost advantage?

    ——————

    Anonymous 1:51 is right: there is no culture of excellence, and no loyalty either, in the software business.

  21. Larry G Avatar
    Larry G

    Management that understands the business requirements often does not understand the computer end of things.

    They don't know the difference between a well designed network or a well functioning network operation or a flawed design or a shoddy operation unless they have a knowledgeable and capable computer expert – on their management team.

    More information came out today that implicated the storage server.

    Most storage servers like Network Appliance and EMC are raid arrays … without getting technical here – they are made up of many multiple drives that the data is "striped" across rather than stored as contiguous files.

    GOOGLE and Microsoft have multiple shipping container size storage arrays scattered across the country.

    That means that if a hard drive fails – you can still get the correct data because the parity bit tells you what the bit was on the lost drive.

    it means if two hard drives fail you can still rebuild the data on the fly.

    if 3 hard drives fail – you will have to go to a more laborious process unless you have thought ahead to have a fail-over server – which you do if you cannot operate without the data.

    But the thing is – when even the first one of those hard drives fail a big red light goes on… warning messages instantly appear on the administrators screens and someone heads to the network room and pulls out the bad drive and plugs in a new one and then makes sure how many spares are still in stock and orders another.

    There is very little else to those storage arrays – some boards an controllers.. also self-diagnostic and "hot" replaceable.

    This technology is pretty mature these days.

    When is the last time your own email provider "lost" data or your online bank "lost" your account?

    They may go down but they seldom lose data any more.

    McDonnell says there is going to be an "investigation".

    " This week's hardware meltdown, officials said, was caused when a component of the servers and its backup both failed."

    that's HIGHLY unlikely unless some change was made to both of them at the same time – normally not a good practice.
    I predict that we may not hear much more about the specifics but it's going to get down to who designed the system and who was operating it or if the vendor (like NetApps) was rolling in new equipment, etc….

    Next, you can worry about the computer in your car that was most likely designed without DO-178B specs (used for aircraft computer control systems).

Leave a Reply

ADVERTISEMENT