The SUGEN Numbers
January 20, 2010
I argued in the previous post that SAP’s new, “two-tier” pricing system for maintenance offers customers less choice than meets the eye, and commentators like Dennis Howlett agree.
So why did they bother? If one offering is “good support for a fair and reliable price” and the other offering is “less good support for roughly the same price (only no one will really know for six years)” why would anyone pick the latter? And why would SAP risk a public relations nightmare when the people who pick the apparently lower-cost alternative find that they’ve been snookered?
Is it just that SAP needs to offer the enterprise application version of “small coffee,” the coffee size that nobody ever orders, but you need on the menu, so people will order medium?
The question is particularly salient because SAP has data that, one could at least argue, shows that Enterprise Support really is better.
This data comes out of a program embarked on last summer, sponsored by SUGEN (SAP User Group Executive Network) and SAP. In this program, companies were put on Enteprise Support, and the benefits thereof were measured in 11 benchmarks and the sum of those benefits added into something called the SUGEN KPI Benchmark Index. SAP had vowed not to raise the cost of Enterprise Support until this program showed that a gain in the Index that gained justified the increased cost of Enterprise Suppory.
SAP reported the results at the Influencer Summit last December, which I attended. According to the numbers they showed me, the SUGEN KPI benchmarks had indeed been achieved.
These numbers were disclosed to me on the condition that I not repeat them until the full results were published, and while I’ve been given informal permission to speak about them, I will try to respect this request.
I think, though, that I can convey a fairly accurate idea of what is going on without actually citing the numbers.
Before I can get to this, though, I need to explain something about the program and the expectations that people had for it. When SAP first announced a new, improved class of support at an increased price, which all customers were required to use, many customers thought that this was just price-gouging. They didn’t believe that SAP’s across-the-board price increase for maintenance would be accompanied by any benefits. When SAP started hearing from these customers, they were clearly taken aback, since the executives in charge of this new support program clearly did (and do) believe in its benefits.
So SAP (and SUGEN, the customers’ self-appointed representatives) agreed to put the question to an empirical test.
Now anybody reading the press release about this program or anybody attending the journalist session at last May’s Sapphire (as I did) would believe that this test would be done along traditional social science lines. A representative sample of the SAP customer base would be given the opportunity to take advantage of Enterprise Support, and the benefits would be measured.
After attending that session, I told my client base (people who are professionally interested in tracking what SAP does) that it was impossible for this program to show so much benefit that it would justify the across-the-board increase. The reason was simple. To get the available benefit from Enterprise Support, a customer must get and install a software product called the Solution Manager and must then do a lot of process documentation and process modification. A representative sample of SAP customers simply wouldn’t include very many customers who had done all this installation, documentation, and change, because the total amount of work was considerable, and most customers weren’t going to do it, at least not any time soon.
Isn’t it sort of squirrelly, expecting Enterprise Support customers to get software, install it, and then do a fair amount of work before they get the benefit that SAP promised them? Well, yes, but it isn’t quite as squirrelly as it sounds.
At the Influencer Summit, Uwe Hommel, the person behind this idea, expressed it roughly this way. A lot of customers don’t really run support as well as they could. The Solution Manager provides them with a framework for the practices that they should be using, plus it enables SAP support personnel to give better, more accurate, and faster help, because the Solution Manager gives them better information about what was going on at a customer site. It would be nice if SAP could wave a magic wand and improve support without any effort from the customers. But that just can’t happen. All we can do is provide a framework.
As far as Hommel is concerned, what SAP is saying is roughly what the trainer at the gym offers. “We’ll make you better, bigger, stronger, and leaner, but of course you have to do your share.”
Fair enough, of course. But that’s not actually what SAP said. SAP actually said something more like, “You need to do support better, and to do it better, you’ll need a trainer and you’ll have to put in some effort, but oh, by the way, you have to pay for the trainer whether or not you actually get around to going to the gym.”
Perhaps the oddest thing about the test that SUGEN and SAP ran is that both parties pretended that SAP had said the first thing and not the second.
You see this, for instance, in the way they [SUGEN according to Myers, below] chose the subjects for the test. Rather than choosing the representative sample of the customer base that I was told they would choose, they asked for volunteers to apply for the program. 140 customers did apply; of those, only 56 were chosen for testing. This, of course, simply guaranteed that the test would not prove what SAP wanted it to prove, that the price increase was justified. At best, it would prove that those customers who decided to go to this metaphorical trainer would get some benefit from it.
So, did they get some benefit?
Well, um, uh, sort of.
As I said above, SAP and SUGEN agreed before the test that there were 11 areas where benefit might be provided. The areas ranged from the obvious things–fewer outages, faster problem resolution, and fewer problems–to the less obvious, but still important things, like more efficient CPU utilization and better use of disk storage.
In the actual test, the benefits of Enterprise Support was measured in only 6 of these 11 areas. The areas chosen had to do with total cost of operations (use of CPU and storage), the cost/effectiveness of managing patches, and the extent to which customers used SAP’s current software effectively. Clearly, this made things harder for SAP, since they were trying to prove benefit, but the benefit was actually measured in less than half the areas where benefit might be available.
Nevertheless, SAP thinks that it succeeded, and technically they did. They measured benefit by giving the SUGEN Index an arbitrary value of 100. The way I understood it at the session, the aim was to show that the increased benefits at least offset the roughly 7.6% price increase in 2009. [According to Myers, below, the actual aim was 4%].
Both aims (what I thought was the aim and what Myers said was the aim) were actually achieved. The benchmark index dropped by 6.89 percentage points. Even though only 6 benchmark areas were measured, the benefit achieved did offset the 7.6% increase (at least within 1 percentage point).
There is, however, a little, tiny, “but.”
All the benefit was achieved by massive improvements in only two out of the six areas: storage utilization and number of failed changes. (A “failed change,” is an attempt to install a patch which fails.) In all the other areas that were measured, the average improvement was very small.
Both of these measure appear to me to be one-time-only improvements. Take, for instance, storage utilization. If you have one of those awful Windows machines, and your disk is sluggish, you can run a utility that compacts your disk and frees up disk space. You’ll show massive improvement in storage utilization. But running this utility once is not the sort of thing that justifies a permanent yearly increase in maintenance costs. Yes, you can do it next year, but it won’t show the same level of improvement, because you gained most of the benefit the first time you did it. The same thing goes for reducing failed changes. Changes in process (and use of the Solution Manager, or Sol Man) can reduce this number a lot. But once you’ve made the changes, further reductions aren’t really available.
I certainly hope that somebody from SAP is reading this; if you are, you’re probably upset, because you’re saying to yourself, “Well, the benefit is permanent; for the rest of time, people will have fewer failed changes and use less disk space.” [Myers does in fact argue this. See below.] You are, of course, right. But you’ll have overlooked the larger question: does helping people to a one-time improvement justify a permanent, yearly price increase. That is hard for me to see. If Enterprise Support promised to bring these kinds of improvements in regularly, then it would be OK to pay more for it regularly. But this test doesn’t show that these regular improvements will be forthcoming.
In any case, it’s all moot now. SAP has scrapped the SUGEN benchmark process. In a way, it’s a shame. This is one of the few times that any enterprise application company has ever tried to run a systematic test of whether its software and services work as advertised. And the results of this test are very interesting. In some areas, the software and services don’t seem to work; the benefits are minimal. In other areas, though, they work very well indeed; the benefits are startling. Who woulda thunk it?
At the very least, shouldn’t SAP keep going with this, so it can go back and fine-tune its software, figure out why some benefits aren’t forthcoming and do something about it?
Apparently not.
A Flaw in Business Cases
September 19, 2009
A software business case compares the total cost of software with the benefits to be gained from implementing the software. If the IRR of the investment is adequate, relative to the company’s policies on capital investment, and if the simple-minded powers that be have a good gut feel about the case itself, it is approved.
Clearly, a business case isn’t perfect. Implementing software is an uncertain business. The costs are complex and hard to fix precisely. Implementation projects do have a way of going over; maintenance costs can vary widely; the benefits are not necessarily always realizable.
That’s where the gut feel comes in. If an executive thinks the benefits might not be there, the implementation team might have a steeper learning curve than estimated, the user acceptance might be problematic, he or she will blow the project off, even if the IRR sounds good.
Business cases of this kind are intended to reduce the risk of a software purchase, but I think they’ve actually been responsible for a lot of failures, because they fail to characterize the risk in an appropriate way.
There’s an assumption that is built into business cases, which turns out to be wrong. The fact that this assumption is wrong means that there’s a flaw in the business case. If you ignore this flaw (which everybody does), you take on a lot of risk. A lot.
The flaw is this. Business cases assume that benefits are roughly linear. So, the assumption runs, if you do a little better than expected on the implementation and maintenance, you’ll get a little more benefit, and if you do a little worse, you’ll get a little less.
Unfortunately, that’s just not the case. Benefits from software systems aren’t linear. They are step functions. So if you do a little worse on an implementation, you won’t get somewhat less benefit; you’ll get a lot less benefit or even zero benefit.
The reason for this is that large software systems tend to be “brittle” systems. (See the recent post on “Brittle Applications.”) With brittle systems, there are a lot of prerequisites that must be met, and if you don’t meet them, the systems work very, very poorly, yielding benefit at a rate far, far below what was expected of them.
This problem is probably easier to understand if we look at how business cases work in an analogous situation. Imagine, for instance, the business case for an apartment building. The expected IRR is based on the rents available at a reasonable occupancy rate. There is, of course, uncertainty, revolving around occupancy rates, rentals, maintenance costs, quality of management, etc. But all of this uncertainty is roughly linear. If occupancy goes up, return goes up, maintenance goes up, return goes down, etc. The business case deals with those kinds of risks very effectively, by identifying them and insuring that adequate cushions are built in.
But what if there were other risks, which the business case ignored? These risks would be associated with things that were absolutely required if the building was to get any return at all. Would a traditional business case work?
Imagine, for instance, that virtually every component of the building that you were going to construct–the foundation, the wiring, the roof, the elevators, the permits, the ventilation, etc.,–was highly engineered and relatively unreliable, required highly skilled people who were not readily available to install, fit so precisely with every other part of the system that anything out of tolerance caused the component to shut down, etc., etc. The building would be a brittle system. (There are buildings like this, and they have in fact proven to be enormously challenging.)
In such a case, it is not only misguided to use a traditional business case, it is very risky. If one of these risky systems doesn’t work–if there’s no roof or no electricity or no elevator or no permit–you don’t just generate somewhat less revenue. You generate none. Cushions and gut feel and figuring that an overrun or two might happen simply lead you to a totally false sense of security. With this kind of risk, it’s entirely possible that no matter how much you spend, you won’t get any benefit.
Now, businessmen are resourceful, and it is possible to develop a business case that correctly assesses the operational risk (the risk that the whole thing won’t work AT ALL). I’ve just never seen one in enterprise applications. (Comments welcome at this point.)
The business cases and business case methodologies that I’ve seen tend to derive from the software vendors themselves or from the large consulting companies. Neither of these are going to want to bring the risk of failure to the front and center. But even those that were developed by the companies themselves (I’ve seen a couple from GE) run into a similar problem: executives don’t want to acknowledge that there might be failure, either.
But the fact that the risk makes people uncomfortable doesn’t mean that it’s a risk that should be ignored. That’s like ignoring the risk that a piton will come out when you’re mountain climbing.
Is the risk real? Next post.
Delivering Sourcing Optimization
August 6, 2009
It’s a plain fact that there are certain kinds of math problems that even bright people don’t solve very well and computers solve better than we do. Some of these look surprisingly simple. If you’re a retailer, when should you mark down product and by how much? Or if you’ve got to make six deliveries, should you use one truck and make six stops, two trucks with three each, etc.
Entrepeneurs in the world of enterprise software have recognized this fact for a long, long time and have seen an opportunity there. They develop so-called optimization software, which solves an essentially mathematical problem and then provide it in various forms to the market. A few have provided only the engine itself, most of them ending up in I-Log, and many more that provide optimization solutions that address specific problems, like supply chain optimization, markdown optimization, or the subject of this post, sourcing optimization.
Now if there’s one thing that’s completely clear about optimization software, it is that the track record has been disappointing. I’m not just talking about the overpromising and underdelivering that afflicted the big-name supply chain optimization vendors. (No names, but you know who you are.) I’m talking about really wonderful products, like Joe Shamir’s ToolsGroup, which have struggled to put their tools in the hands of people who need it.
Sourcing is no exception to the general rule. In sourcing, optimization came to the fore when companies like FreeMarkets or Digital Freight wanted to set up sourcing “events,” where possibly hundreds of bidders could bid on thousands of line items, offering discounts for bundles of selected lines. What if A bids X for lines 1, 2, and 3, and B bids Y for lines 2, 3, and 4, etc.? This is one of those math problems that linear programming solutions solve better than human beings do.
A number of vendors still in the market place (you know who you are) proceeded to develop solutions for setting up and managing these events, which of course did a lot more than just figuring out which bids to accept. And to some extent, they worked; they did change the way many companies do business, and sourcing events are now fairly normal, with several large companies being big proponents.
But in my (admittedly somewhat limited) view, the optimization piece has been relatively unimportant. Certainly some companies have developed some good optimization algorithms, and they are used, but in the events I’ve seen or heard about, it’s not clear to me that the solutions they offered were actually all that optimal, and the reaction of buyers that I know to the solutions they offered have been tepid, shall we say.
I’m not surprised, as I say, because I’ve seen similar reactions from users of other optimization software.
Now I have a degree in math, and I’ve taken a course in linear programming. I know that the solutions they give you are better than what people can come up with by themselves. So are the people who don’t like these solutions just grousing? That’s certainly what most vendors think.
Well, the other day, I got a glimmer of the answer from some terrific research done by Vishal Gaur of Cornell University. (See my earlier post, “An Automated Replenishment System.”) He found in his case study that a solution produced by optimization software had a highly suboptimal operational effect because it was optimizing around the wrong things. The math was done perfectly, but the problem had been set up incorrectly.
This is always a problem in principle when you set optimization software chugging away. The software draws a box around the problem and finds the point inside the box with the best available solution. If you draw a different box (usually a larger one), the solution will be different, and will take longer to find.
In Professor Gaur’s case, the software developer had drawn the wrong box around the problem. Now what’s interesting here is that the software developer never knew that he’d done anything wrong. He delivered the software, and the IT people who were using it expressed satisfaction. And you might think, “Well, so what, even if it drew slightly the wrong box, it got roughly the right answer.” But in fact, when you change the box, you change the answer completely. So in this case, the users had to replace (manually) all the solutions the software offered (manually) with better and completely different solutions.
If you have optimization software, in other words, you have to draw the right box, or the answers that it gives you are not better than what you can provide, they are worse, often much, much worse.
Now look at how this applies to sourcing optimization. Let’s say that you set up the problem in one way. (You only allow bids from certain vendors, for instance, and you require that they offer discounts of a certain percentage.) If you draw the box differently, you will come up with quite different answers.
I was discussing this problem with Garry Mansell, president of TradeExtensions, a small sourcing optimization vendor. He argues that the only way of solving this problem is to provide a lot of services, along with the software. The experts that he provides (it just so happens, of course, that he’s in the business of providing this pairing) can keep the sourcing on track and make sure the right box is drawn.
Now I react to most honeyed words from vendors the way I react to offers of honey for my tea. (Hint: “No, thank you.”) But this struck me as roughly right. The problems that I’ve seen with implementations of myriad kinds of optimization software wouldn’t be solved, then, by fixing the software. It would be solved by helping people to make sure that they’re drawing the right box around the problem.
So who knows. Maybe Mansell is right. (Or maybe he’s just selling software. Or both.) But it seems to me to be a useful thing to keep in mind. If you’re looking at sourcing optimization software (or, more generally, any optimization software), don’t buy it and don’t use it, unless you have some serious experts, who can help you make sure that you drew the right box.
The Applicator 