Building Production-Ready RAG Systems: From Semantic Search to Intelligent Responses

Why Basic Chatbots Don't Work Here

We tried the obvious solutions first. Rule-based bots that match keywords to canned responses. Fast, cheap, useless. Guest asks "where's parking?" and gets back the parking policy PDF instead of "behind the building, spots 12-15." Or it can't handle slight variations in phrasing.

Then there's the pure LLM approach. Feed the question to GPT-4 and let it respond. This works great until you realize it's inventing WiFi passwords. The model doesn't know your properties. It doesn't have your house rules or your specific check-in procedures. So when it hits a knowledge gap, it fills in something that sounds right. Confidently wrong is worse than no answer at all.

How We Actually Built This

The fix is retrieval-augmented generation. You take the guest's question, find the relevant property information in your database, and hand both to the LLM. Now it's working from facts instead of statistical guesses.

Our implementation has four pieces working together:

Implementation Details: PostgreSQL + pgvector

We chose PostgreSQL with pgvector over specialized vector databases for three reasons:

The setup is straightforward:

CREATE EXTENSION vector;

CREATE TABLE property_embeddings (
  id BIGSERIAL PRIMARY KEY,
  property_id INTEGER REFERENCES properties(id),
  content_type VARCHAR(50), -- 'amenities', 'rules', 'instructions', etc.
  content TEXT,
  embedding VECTOR(1536),
  created_at TIMESTAMP DEFAULT NOW()
);

CREATE INDEX ON property_embeddings 
USING hnsw (embedding vector_cosine_ops);

Each property ends up with multiple embedding records, one per content type. This granularity helps us track which specific information piece matched the guest's question.

Embedding Generation: Batch Processing Everything

When a property gets added or updated, we don't generate embeddings synchronously. That would block the API response and waste database connections waiting on OpenAI's API.

Instead, we queue embedding generation as background jobs. Our worker processes pick these up, batch multiple items together (OpenAI allows up to 2048 inputs per request), and write the results back in bulk. This approach:

• Keeps API response times under 200ms
• Reduces embedding API costs by 60% through batching
• Handles OpenAI rate limits gracefully with exponential backoff
• Makes embedding updates non-blocking for property managers

The batching code looks roughly like this:

def generate_embeddings_batch(items):
    texts = [item['content'] for item in items]
    
    response = openai.Embedding.create(
        model="text-embedding-3-small",
        input=texts
    )
    
    embeddings = [r['embedding'] for r in response['data']]
    
    # Bulk insert to database
    with db.transaction():
        for item, embedding in zip(items, embeddings):
            PropertyEmbedding.create(
                property_id=item['property_id'],
                content_type=item['content_type'],
                content=item['content'],
                embedding=embedding
            )

Query Processing: Understanding Guest Intent

Not all guest messages are straightforward questions. Someone might say "We're arriving around 7pm and wondering about parking, also what's the WiFi?" That's three separate intents: arrival time update, parking location question, and WiFi credentials request.

We handle this in two passes:

First, send the message to GPT-4 and ask it to pull out what the guest actually wants. What questions are they asking (WiFi, parking, checkout times)? What are they requesting (early check-in, extra towels)? Any dates mentioned?

This gives us structured JSON we can actually use.

• Information being provided (arrival time, special needs)
• Any date mentions

This returns structured JSON we can act on programmatically.

Then we run separate searches for each question. Guest asked about parking and WiFi? Two similarity searches, combine the results.

This approach means we fetch all relevant information in one round trip, even for complex multi-part messages.

Context Assembly: Feeding the LLM

Once we have the relevant property data, we need to format it for GPT-4. The prompt structure matters enormously here. Too little context and the LLM lacks information. Too much and it gets confused about what's actually relevant.

Our prompt template looks like this:

You are a helpful property manager assistant responding to a guest.

PROPERTY INFORMATION:
{retrieved_property_data}

CONVERSATION HISTORY:
{last_5_messages}

GUEST'S CURRENT MESSAGE:
{current_message}

ADDITIONAL INSTRUCTIONS FROM PROPERTY MANAGER:
{custom_instructions}

Generate a helpful response to the guest's message. Stick to the
property information above. Don't make anything up. If you don't
have the info, say so.

The retrieved property data is already filtered down to what's relevant for this specific question. Usually ends up being 1500-2500 tokens, which is manageable and gives the LLM what it needs without overloading it.

Handling Dates and Availability

Semantic search breaks down when guests ask about specific dates. "Can we book June 15-22?" or "Is early check-in available?" need real availability data, not embeddings.

For these, semantic search alone isn't enough. We also need to query our availability calendar, which is a time-series data structure completely separate from embeddings.

When our message categorization step detects date mentions, we extract the specific dates using a simple regex pattern (handling various date formats). Then we:

1. Run the semantic search to find contextually relevant information
2. Query the availability table for the exact dates mentioned
3. Combine both in the context sent to GPT-4

So the LLM sees both the semantic matches (house rules, parking info) and the hard data (yes/no on those specific dates). Works well enough.

Keeping the AI From Making Things Up

Even with RAG, LLMs can still hallucinate if you're not careful. We've implemented three safeguards:

1. Explicit Instructions Against Invention Our system prompt explicitly tells GPT-4 to only use provided information and to say "I don't have that information" rather than guessing. This simple instruction reduces hallucinations by roughly 80%.
2. Source Attribution We track which property data chunk was used for each part of the response. If an operator questions an AI response, we can show them exactly where that information came from.
3. Confidence scoring Cosine similarity spits out a score between 0 and 1 for each match. We set a cutoff at 0.75. Below that, the match is probably garbage and we leave it out. Took some testing to find that number, but it works.

Multi-Question Responses: Deduplication Matters

Here's a subtle problem we ran into: if a guest asks "What's the WiFi password and where do I park?", and both the WiFi data and parking data mention the address, the LLM might include the address twice in its response.

We solved this with deduplication at the context level. Before sending information to GPT-4, we:

1. Identify overlapping content across retrieved chunks
2. Merge chunks that cover the same topic
3. Remove redundant information
4. Present a clean, non-repetitive context

The LLM still generates natural-sounding responses, but the input is already cleaned up so it doesn't have to figure out what to include or exclude.

Conversation History: Context Windows That Actually Work

Early versions of our system included the entire conversation history in every request. With long threads, this quickly exceeded token limits or made responses expensive.

Now we use a sliding window approach: include only the most recent 5 messages, plus any messages that contained important information (like the guest's arrival time or special requests).

This selective history keeps token counts manageable while ensuring the LLM doesn't lose critical context. The categorization step from earlier helps here too, because we know which past messages contained information versus just pleasantries.

Custom Instructions: Giving Operators Control

Property managers sometimes need to override or supplement the standard information. Maybe the pool is under maintenance, or they can offer early check-in for a specific booking.

Our system lets operators add custom instructions when triggering an AI response. These get appended to the context with high priority, and we explicitly instruct GPT-4 to incorporate them naturally into the response.

For example:

• Standard response: "The pool is available 8am-10pm daily."
• With custom instruction "mention pool is closed for maintenance": "Hey, just so you know, the pool's closed for maintenance right now. Should be back open next week. Normally it's 8am-10pm."

This flexibility is crucial for production use. Operators stay in control while still getting the efficiency benefits of AI.

Performance Metrics: What Actually Improved

After rolling out the RAG-based response system across our platform, we tracked several metrics:

Scaling Considerations: Multi-Tenant Architecture

Our platform uses isolated instances per tenant (client). Each client gets their own API layer, database, and background workers. This architecture choice impacts RAG in interesting ways.

The good: Complete data isolation. Client A's property embeddings never mix with Client B's. No cross-contamination risk.

The challenging part: Embedding generation and search runs independently per tenant. More properties means more worker processes and bigger batch sizes, but we're not rebuilding anything.

We handle this with per-tenant resource allocation. Heavy clients (large property portfolios, high message volume) automatically get more worker processes and higher embedding batch sizes. Light clients stay on minimal resources.

The vector database (Postgres with pgvector) scales linearly with property count. A tenant with 1000 properties might have 15,000 embedding records (multiple content types per property), but HNSW indexing keeps search performance fast.

Where RAG Falls Short: Edge Cases We Handle Differently

RAG works great for factual retrieval, but some scenarios need different approaches:

Future Improvements: What's Next

We're constantly iterating on the system. Current focus areas:

Takeaways for Building Your Own RAG System

If you're implementing RAG for a production system, here's what matters most:

Getting RAG Into Production vs Building a Demo

We went from property managers manually typing out hundreds of messages a day to AI handling most of the routine stuff. Operators now spend their time on the weird cases and angry guests instead of copy-pasting WiFi passwords.

The tech stack matters less than you'd think. Postgres, pgvector, OpenAI embeddings, GPT-4. You could swap pieces and get similar results. What matters more: how you structure your knowledge base, when you decide to retrieve versus generate, how you handle conversations that span multiple messages, and how you stop the AI from inventing things.

RAG works when you're answering questions from a large corpus of information that keeps changing. It stops working when you need it to work perfectly in a demo but haven't thought through data quality, performance at scale, or what happens when the AI gets something wrong.

Production systems need human oversight, especially when mistakes have consequences. Build for that from the start.

At Devslane, we specialize in building production-grade AI systems for clients across property management, fintech, healthcare, and other domains where accuracy and reliability matter. Our engineering teams have delivered RAG implementations, custom LLM integrations, and AI-powered automation for clients processing millions of interactions. If you're considering AI augmentation for your platform, let's talk about what's actually feasible.