The SUGEN Numbers

January 20, 2010

I argued in the previous post that SAP’s new, “two-tier” pricing system for maintenance offers customers less choice than meets the eye, and commentators like Dennis Howlett agree.

So why did they bother? If one offering is “good support for a fair and reliable price” and the other offering is “less good support for roughly the same price (only no one will really know for six years)” why would anyone pick the latter? And why would SAP risk a public relations nightmare when the people who pick the apparently lower-cost alternative find that they’ve been snookered?

Is it just that SAP needs to offer the enterprise application version of “small coffee,” the coffee size that nobody ever orders, but you need on the menu, so people will order medium?

The question is particularly salient because SAP has data that, one could at least argue, shows that Enterprise Support really is better.

This data comes out of a program embarked on last summer, sponsored by SUGEN (SAP User Group Executive Network) and SAP. In this program, companies were put on Enteprise Support, and the benefits thereof were measured in 11 benchmarks and the sum of those benefits added into something called the SUGEN KPI Benchmark Index. SAP had vowed not to raise the cost of Enterprise Support until this program showed that a gain in the Index that gained justified the increased cost of Enterprise Suppory.

SAP reported the results at the Influencer Summit last December, which I attended. According to the numbers they showed me, the SUGEN KPI benchmarks had indeed been achieved.

These numbers were disclosed to me on the condition that I not repeat them until the full results were published, and while I’ve been given informal permission to speak about them, I will try to respect this request.

I think, though, that I can convey a fairly accurate idea of what is going on without actually citing the numbers.

Before I can get to this, though, I need to explain something about the program and the expectations that people had for it. When SAP first announced a new, improved class of support at an increased price, which all customers were required to use, many customers thought that this was just price-gouging. They didn’t believe that SAP’s across-the-board price increase for maintenance would be accompanied by any benefits. When SAP started hearing from these customers, they were clearly taken aback, since the executives in charge of this new support program clearly did (and do) believe in its benefits.

So SAP (and SUGEN, the customers’ self-appointed representatives) agreed to put the question to an empirical test.

Now anybody reading the press release about this program or anybody attending the journalist session at last May’s Sapphire (as I did) would believe that this test would be done along traditional social science lines. A representative sample of the SAP customer base would be given the opportunity to take advantage of Enterprise Support, and the benefits would be measured.

After attending that session, I told my client base (people who are professionally interested in tracking what SAP does) that it was impossible for this program to show so much benefit that it would justify the across-the-board increase. The reason was simple. To get the available benefit from Enterprise Support, a customer must get and install a software product called the Solution Manager and must then do a lot of process documentation and process modification. A representative sample of SAP customers simply wouldn’t include very many customers who had done all this installation, documentation, and change, because the total amount of work was considerable, and most customers weren’t going to do it, at least not any time soon.

Isn’t it sort of squirrelly, expecting Enterprise Support customers to get software, install it, and then do a fair amount of work before they get the benefit that SAP promised them? Well, yes, but it isn’t quite as squirrelly as it sounds.

At the Influencer Summit, Uwe Hommel, the person behind this idea, expressed it roughly this way. A lot of customers don’t really run support as well as they could. The Solution Manager provides them with a framework for the practices that they should be using, plus it enables SAP support personnel to give better, more accurate, and faster help, because the Solution Manager gives them better information about what was going on at a customer site. It would be nice if SAP could wave a magic wand and improve support without any effort from the customers. But that just can’t happen. All we can do is provide a framework.

As far as Hommel is concerned, what SAP is saying is roughly what the trainer at the gym offers. “We’ll make you better, bigger, stronger, and leaner, but of course you have to do your share.”

Fair enough, of course. But that’s not actually what SAP said. SAP actually said something more like, “You need to do support better, and to do it better, you’ll need a trainer and you’ll have to put in some effort, but oh, by the way, you have to pay for the trainer whether or not you actually get around to going to the gym.”

Perhaps the oddest thing about the test that SUGEN and SAP ran is that both parties pretended that SAP had said the first thing and not the second.

You see this, for instance, in the way they [SUGEN according to Myers, below] chose the subjects for the test. Rather than choosing the representative sample of the customer base that I was told they would choose, they asked for volunteers to apply for the program. 140 customers did apply; of those, only 56 were chosen for testing. This, of course, simply guaranteed that the test would not prove what SAP wanted it to prove, that the price increase was justified. At best, it would prove that those customers who decided to go to this metaphorical trainer would get some benefit from it.

So, did they get some benefit?

Well, um, uh, sort of.

As I said above, SAP and SUGEN agreed before the test that there were 11 areas where benefit might be provided. The areas ranged from the obvious things–fewer outages, faster problem resolution, and fewer problems–to the less obvious, but still important things, like more efficient CPU utilization and better use of disk storage.

In the actual test, the benefits of Enterprise Support was measured in only 6 of these 11 areas. The areas chosen had to do with total cost of operations (use of CPU and storage), the cost/effectiveness of managing patches, and the extent to which customers used SAP’s current software effectively. Clearly, this made things harder for SAP, since they were trying to prove benefit, but the benefit was actually measured in less than half the areas where benefit might be available.

Nevertheless, SAP thinks that it succeeded, and technically they did. They measured benefit by giving the SUGEN Index an arbitrary value of 100. The way I understood it at the session, the aim was to show that the increased benefits at least offset the roughly 7.6% price increase in 2009. [According to Myers, below, the actual aim was 4%].

Both aims (what I thought was the aim and what Myers said was the aim) were actually achieved. The benchmark index dropped by 6.89 percentage points. Even though only 6 benchmark areas were measured, the benefit achieved did offset the 7.6% increase (at least within 1 percentage point).

There is, however, a little, tiny, “but.”

All the benefit was achieved by massive improvements in only two out of the six areas: storage utilization and number of failed changes. (A “failed change,” is an attempt to install a patch which fails.) In all the other areas that were measured, the average improvement was very small.

Both of these measure appear to me to be one-time-only improvements. Take, for instance, storage utilization. If you have one of those awful Windows machines, and your disk is sluggish, you can run a utility that compacts your disk and frees up disk space. You’ll show massive improvement in storage utilization. But running this utility once is not the sort of thing that justifies a permanent yearly increase in maintenance costs. Yes, you can do it next year, but it won’t show the same level of improvement, because you gained most of the benefit the first time you did it. The same thing goes for reducing failed changes. Changes in process (and use of the Solution Manager, or Sol Man) can reduce this number a lot. But once you’ve made the changes, further reductions aren’t really available.

I certainly hope that somebody from SAP is reading this; if you are, you’re probably upset, because you’re saying to yourself, “Well, the benefit is permanent; for the rest of time, people will have fewer failed changes and use less disk space.” [Myers does in fact argue this. See below.] You are, of course, right. But you’ll have overlooked the larger question: does helping people to a one-time improvement justify a permanent, yearly price increase. That is hard for me to see. If Enterprise Support promised to bring these kinds of improvements in regularly, then it would be OK to pay more for it regularly. But this test doesn’t show that these regular improvements will be forthcoming.

In any case, it’s all moot now. SAP has scrapped the SUGEN benchmark process. In a way, it’s a shame. This is one of the few times that any enterprise application company has ever tried to run a systematic test of whether its software and services work as advertised. And the results of this test are very interesting. In some areas, the software and services don’t seem to work; the benefits are minimal. In other areas, though, they work very well indeed; the benefits are startling. Who woulda thunk it?

At the very least, shouldn’t SAP keep going with this, so it can go back and fine-tune its software, figure out why some benefits aren’t forthcoming and do something about it?

Apparently not.

6 Responses to “The SUGEN Numbers”

  1. […] This post was mentioned on Twitter by Frank Scavo, Dennis. Dennis said: #News The SUGEN Numbers #SAP: All the benefit was achieved by massive improvements in only two out of the six areas:… […]

  2. Hi David,

    A few clarifications:

    There were only 11 KPIs in the index, not 13, but it is true that we only measured 6 of those 11.

    The first year goal was a reduction of 4%, not 6%.

    It’s not accurate to say only 56 customers were chosen, with the implication that SAP could influence the outcome. SUGEN was responsible for choosing the customers, not SAP.

    That said, as we indicated, and you agree, SAP did achieve the benchmark in 2009. But because we did not have the full panel of customers, and because we only measured partial KPIs, we held off on the price increase for 2010.

    The program was designed to show a cumulative savings of 30% over 4 years, so the one-time savings you mentioned would be expected.

    I just wanted to clear those few inaccuracies up. I appreciated the objectivity that you continually strive to show.


  3. toppundit Says:

    Michael, thank you for the corrections noted above. In the Power Point slide that you sent me, there were indeed only 11 areas, not 13. I have corrected the number in the text above.

    The same slide deck claims that the objective was a 30 percentage point decrease in the index in four years, which, with compounding, is approximately an 8% drop per year. I don’t know where the 4% number comes from, but I assume you know. If SAP and SUGEN agreed on 4% the first year, both parties were therefore assuming significantly greater benefit in subsequent years. It’s too bad that the test did not continue, so that it could show this increased benefit. Certainly, the early numbers in four areas showed no sign that such a large benefit would ensue.

    The 6 percentage point number that I used came from the same slide deck, which said that the index dropped 6.89 percentage points, to 93.11. I rounded down, because the whole number made my point.

    I did not know that it was SUGEN and not SAP that selected the customers. Nevertheless, the point is the same. Any selection of this kind lessens SUGEN’s/SAP’s ability to claim universal benefit for Enterprise Support.

    If, on the other hand, all one wanted to do with the SUGEN process was to make sure that Enterprise Support delivered what was promised, it was probably a good idea to keep the number of customers small, so as to make sure that the testing didn’t get too unwieldy.

  4. Social comments and analytics for this post…

    This post was mentioned on Twitter by dbmoore: #News The SUGEN Numbers #SAP: All the benefit was achieved by massive improvements in only two out of the six areas:…

  5. Frank Scavo Says:

    David, your most interesting point, to me, is this simple observation: “To get the available benefit from Enterprise Support, a customer must get and install a software product called the Solution Manager and must then do a lot of process documentation and process modification.”

    In other words, the source of the benefits attributed to Enterprise Support come from the customer’s own efforts to improve its internal SAP support processes. Solution Manager no doubt plays a role in these improved processes. But it strikes me as overreaching for SAP to claim it should therefore get back those benefits in terms of higher maintenance fees. Unless it is willing to net out the costs of the customer’s work to get those benefits, and I doubt anyone is doing that study.

  6. Jon Reed Says:

    David, I thought this was one of the most important blog posts about Enterprise Support I have seen to this point. The role of Solution Manager to justify the value of Enterprise Support is not well understood or articulated, and I think you’ve taken this farther than we have seen it before due to the focus on concrete numbers. It’s my current belief that for SAP to get Enterprise Support right, they have to get Solution Manager right. I plan on doing more research on this so if I learn anything relevant I’ll share it or post it.

    This blog post is a good example of how the due diligence to gather good information advances the conversation – whereas most of the posts about SAP’s recent support price changes didn’t help achieve any kind of clarity.

    Thanks for taking the bull by the horns here. I’m under the impression SAP does plan to continue to KPI project in some form, rather than simply “scrap” it, and I personally think they should continue it, but I can see how it might also fade in relevance, because even if it does survive in some form, it will no longer be attached to support prices. One can at least hope that the user dialogue prompted by the SUGEN discussions on Enterprise Support continues.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: