Trust But Verify with Vendor Tech Talks
While doing a little research on data architecture this week, I came across a major tech conference talk from a few years ago. The presenter was an engineer and evangelist from a database and services vendor, and the subject was how to build a data warehouse on top of their product. From first-hand experience, I know two things about this product:
It is a wonderful system for the things that it does well.
It is wildly ill-suited for use in a data warehouse.
If you look closely, the examples used in the presentation were essentially OLTP queries against a toy system posing as analytics. Actual OLAP queries with this schema and database would have caused full-dataset scans in most cases. If you tried to apply this product and data architecture in the real world, it would fall over as soon as it faced a moderately large dataset and would have to be rewritten from scratch on different tech.
I don’t think the vendor engineer doing the presentation had any ill intent. I honestly came away with the impression that he had just never built a real enterprise data warehouse before, and he had a hammer, and everything looked like a nail.
Unfortunatley the presentation was probably quite convincing to other engineers who have never built a real enterprise data warehouse before and therefore could have easily set out on a dead-end design. (In fact, I briefly found myself missing academic conferences with peer review.)
At tech conferences, look for two kinds of talks:
Vendors explaining their products.
Customers giving experience reports.
When you attend a talk, make a note of which it is, and adjust your expectations accordingly. AWS re:Invent always has a fantasic mix of both types, so check out their past agendas to see what I’m talking about.
If you see a vendor pitching a solution built on top of their product, you might want to engage a consultant or at least have a discussion with someone in your network to double-check before starting to build or buy. And if you think I might be able to help, I’m always available for a chat!
