Historical significance of TPC benchmarks
In case you missed it, I’ve had a couple of recent conversations about the TPC-H benchmark. Some people suggest that, while almost untethered from real-world computing, TPC-Hs inspire real world product improvements. Richard Gostanian even offered a specific example of same — Solaris-specific optimizations for the ParAccel Analytic Database.
That thrilling advance notwithstanding, I’m not aware of much practical significance to any TPC-H-related DBMS product development. But multiple people this week have reminded me this week the TPC-A and TPC-B played a much greater role spurring product development in the 1990s. And I indeed advised clients in those days that they’d better get their TPC results up to snuff, because they’d be at severe competitive disadvantage until they did.
It’s tough to be precise about examples, because few vendors will admit they developed important features just to boost their benchmark scores. But it wasn’t just TPCs — I recall marketing wars around specific features (row-level locking, nested subquery) or trade-press benchmarks (PC World?) as much as around actual TPC benchmarks. Indeed, Oracle had an internal policy called WAR, which stood for Win All Reviews; trade press benchmarks were just a subcase of that.
And then there’s Dave DeWitt’s take. Dave told me yesterday at SIGMOD that it’s unfortunate Jim Gray-inspired debit/credit TPCs won out over the Wisconsin benchmarks, because that led the industry down the path of focusing on OLTP at the expense of decision support/data warehousing. Whether or not the causality is as strict as Dave was suggesting, it’s hard to dispute that mainstream DBMS met or exceeded almost all users’ OTLP performance needs by early in this millenium. And it’s equally hard to dispute that those systems* performance on analytic workloads, as of last year, still needed a great deal of improvement.
*IBM’s DB2 perhaps excepted. And I say “last year” so as to duck the questions of whether Exadata finally solved Oracle’s problems and whether Madison will once Microsoft releases it.
| Categories: Database management systems, Oracle | 3 Comments |