Historical significance of TPC benchmarks
In case you missed it, I’ve had a couple of recent conversations about the TPC-H benchmark. Some people suggest that, while almost untethered from real-world computing, TPC-Hs inspire real world product improvements. Richard Gostanian even offered a specific example of same — Solaris-specific optimizations for the ParAccel Analytic Database.
That thrilling advance notwithstanding, I’m not aware of much practical significance to any TPC-H-related DBMS product development. But multiple people this week have reminded me this week the TPC-A and TPC-B played a much greater role spurring product development in the 1990s. And I indeed advised clients in those days that they’d better get their TPC results up to snuff, because they’d be at severe competitive disadvantage until they did.
It’s tough to be precise about examples, because few vendors will admit they developed important features just to boost their benchmark scores. But it wasn’t just TPCs — I recall marketing wars around specific features (row-level locking, nested subquery) or trade-press benchmarks (PC World?) as much as around actual TPC benchmarks. Indeed, Oracle had an internal policy called WAR, which stood for Win All Reviews; trade press benchmarks were just a subcase of that.
And then there’s Dave DeWitt’s take. Dave told me yesterday at SIGMOD that it’s unfortunate Jim Gray-inspired debit/credit TPCs won out over the Wisconsin benchmarks, because that led the industry down the path of focusing on OLTP at the expense of decision support/data warehousing. Whether or not the causality is as strict as Dave was suggesting, it’s hard to dispute that mainstream DBMS met or exceeded almost all users’ OTLP performance needs by early in this millenium. And it’s equally hard to dispute that those systems* performance on analytic workloads, as of last year, still needed a great deal of improvement.
*IBM’s DB2 perhaps excepted. And I say “last year” so as to duck the questions of whether Exadata finally solved Oracle’s problems and whether Madison will once Microsoft releases it.
Comments
3 Responses to “Historical significance of TPC benchmarks”
Leave a Reply
Curt,
An interesting post. There is no doubt in my mind that both TPC benchmarks caused product improvements. It pays however to give developers some credit here. We are at heart lazy creatures and do hate throw-away code.
Let’s take TPC-D and “materialized views” for example. There was a race between Oracle and IBM DB2 at the time as to who gets them out first and of course one major marketing goal was certainly to bat the TPC-D performance numbers out of the ballpark which rendered the whole benchmark obsolete within months.
A more recent example is TPC-C. Remember those Oracle adds on “IBM won’t say”. Eventually soem IBM exec must have had it and decided that enough is enough. What followed was a flurry of activities resulting in DB2 8.1.4 and the breach of the 700k, 1M and now 6M TpmC threshholds.
You might say that all features delivered in DB2 8.1.4 were specifically implemented for the benchmark. that is true as far as motivation is concerned (i.e. executive funding).
The features however, such as a rewrite of DB2’s logger for ultra high performance, “alternate page cleaners”, and my personal toy the “unification of queries and updates”. Essentially we developers don’t care for the excuse to put in features which we believe will benefit the product. I’m confident this is no different for other vendors.
Cheers
Serge
Makes sense, Serge. Thanks.
[…] their products run faster for scientific purposes, in line with the supposed salutary effects of TPC-A, TPC-B, and TPC-C. Notwithstanding that attendees included Oracle, Microsoft, EMC/Greenplum, Teradata, and […]