Most enterprises are still trying to figure out where their data lives, let alone how to expose it to a large language model. We tend to treat memory in AI as a magical context window. It isn’t. It’s a database. It needs the same rigor we apply to transaction logs, including schemas, access controls, and firewalls that prevent the AI from hallucinating facts or leaking sensitive info to the wrong user.
If you are designing your first AI system, start with the memory layer. Decide what the AI is allowed to know, where that knowledge lives, and how it is updated. Then, and only then, worry about the prompt. Oh, and what should you think about first? Inference.
Start with inference
We used to obsess over the massive cost of training models. But for the enterprise, that is largely irrelevant. AI is all about inference now, or the application of knowledge to power applications. In other words, AI will become truly useful within the enterprise as we apply models to governed enterprise data. The best place to build up your AI muscle isn’t with some moonshot agentic system. It’s a simple retrieval-augmented generation (RAG) pipeline.
