How LLMs Access Enterprise Data: Patterns and Pitfalls

Five Ways LLMs Access Enterprise Data

Large language models (LLMs) are statistical models trained on text. They have no inherent ability to connect to databases, call APIs, or read files. Every enterprise deployment that involves real-time data must solve a fundamental problem: how does the model get the information it needs to answer questions accurately?

Five patterns have emerged. Text-to-SQL lets the LLM generate SQL queries and execute them against a database. Retrieval-augmented generation (RAG) retrieves relevant data at query time and injects it into the model's context window. API-mediated retrieval routes data requests through governed API endpoints. Fine-tuning bakes proprietary data directly into the model's weights during training. Function calling (also called tool use) allows the model to invoke pre-defined functions, including API calls, as part of its reasoning process.

Each pattern makes different trade-offs between power, safety, cost, and operational complexity. No single pattern is universally correct, but some are dramatically safer than others for production enterprise use. Understanding these trade-offs is essential before committing to an architecture.

The choice also depends on the nature of the data. Static reference data that rarely changes can tolerate different approaches than transactional data that updates every second. Sensitive data subject to compliance requirements demands stricter governance than anonymized analytics data. The sections below evaluate each pattern against the requirements that matter most in enterprise environments: security, auditability, freshness, and operational cost.

Text-to-SQL: The Tempting Trap

Text-to-SQL is the most direct approach. The user asks a natural language question, the LLM translates it into a SQL query, and that query runs against the database. The results come back, and the LLM formats them into a human-readable answer. It feels like magic: ask "What were our top 10 products last quarter?" and get an accurate, data-driven response.

The security problems are severe. The LLM is generating executable code, specifically SQL statements, that run with whatever privileges the database connection provides. A carefully crafted prompt can manipulate the model into generating malicious SQL. This is not theoretical. Prompt injection attacks that produce DROP TABLE statements, UNION SELECT extractions from unauthorized tables, or bulk data exfiltration queries have been demonstrated repeatedly in security research.

Beyond injection, text-to-SQL has no concept of per-user authorization. The database connection has one set of permissions, and every query runs with those permissions regardless of who asked the question. If the connection can read the payroll table, every user can read the payroll table through the LLM. There is no RBAC, no field masking, and no way to restrict access at the row or column level without building an entire authorization layer from scratch.

Audit logging is another gap. Database logs will show queries from a service account. They cannot attribute those queries to specific end users or specific AI agents. For organizations subject to SOC 2, HIPAA, or GDPR, this broken audit trail is a compliance failure. Text-to-SQL is useful for internal analytics tools with trusted users and non-sensitive data. For production enterprise use with customer data, it is a liability.

RAG Over Structured Data: APIs as the Retrieval Layer

Retrieval-augmented generation is the dominant architecture for enterprise AI applications. The standard RAG pattern involves embedding documents into a vector database, performing similarity search at query time, and injecting the most relevant chunks into the LLM's context window. This works well for unstructured data like documents, knowledge bases, and support tickets.

RAG over structured SQL data requires a different retrieval mechanism. You do not embed entire database tables into vector stores. Instead, the retrieval step calls an API endpoint that executes a parameterized query and returns JSON results. Those results are injected into the LLM's context, just like vector search results would be, but the data comes from a governed API rather than a similarity search.

This pattern inherits all the security benefits of API-mediated access. The API endpoint enforces authentication. Role-based access control determines which tables and fields the requester can access. Parameterized queries eliminate SQL injection. Field masking redacts sensitive data before it enters the model's context. Rate limiting prevents runaway queries. The full request is logged with attribution. For a deeper look at building this architecture, see Building RAG Pipelines: The Data Access Layer.

The trade-off is flexibility. Text-to-SQL can answer any question the database can answer. API-mediated RAG can only answer questions that the pre-defined endpoints support. In practice, this constraint is a feature, not a bug. Pre-defined endpoints represent business-approved data access patterns. If an AI agent cannot answer a question through existing endpoints, that is a signal to evaluate whether that access should be granted, not a reason to bypass governance.

Another advantage of API-mediated RAG is response consistency. Because the retrieval step calls a well-defined endpoint with parameterized filters, the same query always returns the same structure. This predictability makes it easier to validate AI outputs, debug retrieval failures, and maintain quality over time. With text-to-SQL, the model might generate different SQL for the same question on different invocations, producing inconsistent results that are difficult to diagnose.

Fine-Tuning vs Real-Time Retrieval

Fine-tuning involves training or adapting an LLM on proprietary data so that the knowledge becomes part of the model's weights. The model "knows" your data the same way it knows the general knowledge from its pre-training corpus. This eliminates the need for real-time data retrieval: the model can answer questions from memory.

The problems are substantial. Fine-tuning is expensive in compute costs and engineering time. The data is static from the moment training completes. If your database changes hourly, your fine-tuned model is stale within hours. Re-training on every data change is impractical for most organizations.

More critically, fine-tuning has no access control mechanism. Once data is baked into the model's weights, every user of that model has access to that data. You cannot fine-tune a model on payroll data and then restrict payroll queries to HR staff. The data is in the model, available to anyone who prompts it. There is no RBAC, no field masking, and no audit trail for knowledge retrieval from model weights.

Fine-tuning makes sense for teaching a model domain-specific language, terminology, or reasoning patterns. It does not make sense as a data access mechanism for enterprise data that changes frequently, requires access control, or contains sensitive information. For those use cases, real-time retrieval through governed APIs is the appropriate pattern.

Function calling, sometimes called tool use, is the mechanism that connects LLMs to APIs. The model is given a set of function definitions, such as get_customer_orders(customer_id), and during its reasoning process, it can choose to call those functions to retrieve data. The orchestration framework executes the actual API call and returns the result to the model. This is how RAG and API-mediated retrieval work in practice: function calling is the delivery mechanism, and the API gateway is the enforcement layer.

Why API-Mediated Access Wins for Enterprises

API-mediated retrieval is the only pattern that provides the full stack of enterprise security requirements by default. Authentication verifies the identity of every request. Authorization enforces per-role, per-table, per-field access control. Parameterized queries eliminate SQL injection. Field masking redacts sensitive data before it reaches the model. Rate limiting protects backend systems. Audit logging creates a complete, attributable record of every data access.

The operational benefits extend beyond security. API endpoints provide a stable, versioned interface between AI applications and databases. When the database schema changes, the API layer absorbs the change. AI applications continue to work against the same endpoint contract. This decoupling is essential for organizations running multiple AI agents, applications, and workflows against shared data sources.

Scalability is another factor. As organizations move from one AI application to dozens, managing direct database connections for each becomes untenable. Each connection requires its own credentials, its own permission set, and its own monitoring. An API gateway centralizes all of this into a single layer. New AI applications get an API key with appropriate role permissions, and they are immediately operational against the same governed endpoints that every other application uses. This reduces onboarding time for new AI projects from weeks to hours.

The practical question is how to build the API layer. Writing REST APIs by hand for every database table is slow and error-prone. DreamFactory solves this by automatically generating REST and GraphQL APIs for any connected database, including MySQL, PostgreSQL, SQL Server, Oracle, MongoDB, and Snowflake. Each generated API includes built-in authentication, role-based access control with table-level and field-level permissions, parameterized query execution, rate limiting, and comprehensive audit logging. DreamFactory also supports the Model Context Protocol (MCP), enabling LLMs to discover and use database APIs through a standardized interface without custom integration code.

The pattern is clear: use function calling as the delivery mechanism, a data-AI gateway as the enforcement layer, and governed API endpoints as the data retrieval method. This gives AI applications the data they need while giving security teams the controls they require. For organizations exploring MCP-based integrations, this architecture provides the foundation for safe, scalable AI data access.