In addition, he said that while large organizations such as Uber and Walmart have installed and are using lakehouse offerings, mainstream enterprises, to a large extent, have not yet moved to them, because “today it requires building a via a do-it-yourself approach where you build your own, you cobble together a bunch of open source tools. If you have a deep engineering bench, you can do that. If you don’t have that deep engineering bench, that becomes very difficult.”
Kyle Weller, VP product at Onehouse, added that organizations currently face two challenges: “[They have] chosen a Databricks or Snowflake, and that dictates the rest of their architectural choices, or they are in a situation where they’re looking to open source, but that complexity of self managing is preventative from exploring multiple engines.”
Each engine has a unique specialty, he said, noting, “Flink was not invented for no reason. Flink was invented to address real time stream processing. Ray wasn’t invented just to be another item on the shelf. Ray was invented special purpose for AI use cases, ML use cases, data science.”
