We have a small internal Streamlit app that helps our sales team with answer sheets, discovery prompts, and objection rebuttals, all grounded in Abilytics decks and case studies. It's not customer-facing. It's the kind of tool where the "right" architecture is the one that costs the least to run and the least to keep alive.
The first version ran on AWS, with Chroma on the app's local disk using hash-bag-of-words embeddings, Bedrock Nova Micro for generation, and an AssumeRole into an execution role to read the corpus from S3. It worked. Nova Micro is genuinely cheap. But the corpus lived in AWS, the app lived in Databricks, and the pipe between them was glued with boto3 and prayer.
Then Databricks Free Edition became viable. Agent Bricks (Knowledge Assistant + Genie), pay-per-token Foundation Model endpoints, and managed Vector Search all come with monthly quotas but no recurring bill, which meant the whole stack could live in one workspace on one credential. Worth trying.
This is what happened, and what shipping it to production actually looked like.
What Stayed the Same
The Streamlit UI, the modes (knowledge, discovery, solution, objection), and the guardrails all carried over. A double-gated IntentGuard evaluates raw input, resolves coreferences, then re-evaluates the rewritten query. A partial-allow layer handles mixed prompts like "give me three CFO objections and write the Python for it" by answering the sales half and refusing the code half. An output validator scrubs anything that slips through.
None of that changed. It's decoupled from the retrieval and generation backend, which is exactly why swapping the backend was tractable.
Trying Agent Bricks Knowledge Assistant
The first thing I tried was Databricks' managed answer, the Knowledge Assistant. On paper it's the ideal shape for us. You point it at a Unity Catalog volume, KA handles retrieval and generation server-side, it cites its sources, and the app just makes one HTTP call.
I went in fully committed and provisioned everything via CLI: schema, volume, KA instance, knowledge source, sync. I wrote the provider and the tests, wired the double-gate guardrails around it, and added a lightweight Foundation Model provider for the cheap rewrite and summary calls so we wouldn't pay a full KA retrieval per pronoun.
Provisioning was clean. I created the workspace.presales.knowledge volume (I couldn't spin up a fresh catalog because the metastore has Default Storage on and needs an explicit storage location I didn't want to attach), uploaded eleven files, created the KA, attached the knowledge source, and triggered a sync.
Then two things went wrong.
The API payload shape wasn't obvious. KA rejects the OpenAI Chat Completions messages shape and wants OpenAI Responses instead (input array plus top-level instructions and max_output_tokens). That was a ten-minute change on our side once we saw the actual response.
Then the endpoint 500'd on every live query. The state was READY / DEPLOYMENT_READY, the vector index reported ready=true with all rows present, and the payload validated cleanly against the KA's Pydantic model. But every invocation returned HTTP 500 "Internal error." I recreated the KA fresh and got the same result. I tried three payload variants and nudged a redeploy, and got the same result again. The CLI marks the knowledge-assistants API as Beta, and this is a Beta bug I couldn't work around from the outside.
At that point I had two choices. File a support ticket and wait, or build the pieces myself out of the Databricks endpoints that do work on Free Edition today.
What Actually Shipped: Two Iterations
I did it in two phases, deliberately.
Iteration one: the smallest thing that would prove the idea works.
Chunk the corpus locally, embed all 100-ish chunks with databricks-bge-large-en (1024-dim), serialize as a ~2 MB JSONL and ship it with the app bundle. At query time it runs a pure-Python cosine over the vectors in RAM and feeds the top-k chunks into a databricks-meta-llama-3-3-70b-instruct prompt, which returns a cited answer. There's no vector database, no external index, and no long-lived compute behind it, just a Python list and two serving endpoints.
That version worked, but it had an obvious scaling ceiling. Linear cosine over Python lists breaks down somewhere past ten thousand chunks, and re-embedding the whole corpus on every bundle deploy doesn't scale past a few thousand.
Iteration one was the MVP. Iteration two is what the sales team actually runs on now.
Iteration two: the production architecture.
The corpus now lives in a Delta table (workspace.presales.chunks), managed by a nightly Databricks Job that walks the UC volume, hashes each file against a doc_manifest, and does content-hash-based incremental ingest. Unchanged files are skipped, changed files get their chunks fully replaced, and files removed from the volume have their chunks purged.
On top of that Delta table sits a Databricks Vector Search delta-sync index (workspace.presales.chunks_index), TRIGGERED mode, with managed bge-large-en embeddings. When the ingest Job finishes writing to Delta, it calls index.sync(). The index handles incremental embedding of the new rows itself.
The app got a new retriever (copilot/providers/databricks_vector_search.py) that uses the Databricks-native VectorSearchClient, which hits the index via similarity_search(query_text=..., num_results=5) and gets ranked chunks back with metadata hydrated from the Delta row. The RETRIEVAL_MODE env variable toggles between this and the JSONL fallback for offline dev and tests.
Both retrievers implement the same query(question, top_k) -> list[RetrievedChunk] interface, and service.py picks between them at construction time. Downstream code (guardrails, generation, output scrubbing) doesn't know which retriever it's talking to.
Two problems came up in Iteration 1 that were worth learning:
- 1
Chroma 1.0's embedding-function handling silently downgraded our 1024-dim vectors to a 256-dim built-in. Even after implementing the newer EmbeddingFunction interface with embed_documents / embed_query, storage was still 256-dim. That wasn't worth debugging further for a couple hundred chunks, so I dropped Chroma and wrote 50 lines of pure-Python cosine instead. It's faster, and there's no dimension negotiation and no extra dependency.
- 2
Free Edition embedding endpoints throttle hard. Sending 100 chunks in a single batch returned
REQUEST_LIMIT_EXCEEDED: Exceeded workspace QPS rate limit. The bge-large endpoint on Free Edition tolerates roughly one call per second. Iteration 1 works around this by sending one text at a time with 0.6s spacing and exponential backoff on 429/5xx. Iteration 2 sidesteps it entirely, because the delta-sync index handles embedding server-side at whatever rate it can and we don't wait on it.
The Production Shape
Every arrow between components is an SP-authenticated call. No PATs live in the deployed config. The App's service principal has CAN_QUERY on the two serving endpoints and USAGE on the Delta tables via databricks.yml resource bindings.
Bedrock + Chroma + S3 vs What We Ended Up With
Both stacks can run this app. Here are the tradeoffs, after actually shipping both.
| Dimension | Bedrock + Chroma + S3 | Databricks Free Edition |
|---|---|---|
| Recurring cost | Low per-token + S3 storage | $0 within monthly Free Edition quota |
| Credential surface | AWS keys + AssumeRole + external ID; Databricks separately | One Databricks identity (App SP) |
| Corpus updates | Push to S3 → app reindexes Chroma at startup | Drop file in UC volume → nightly ingest Job → auto-sync |
| Retrieval quality | Hash bag-of-words (toy) | bge-large-en semantic + HNSW |
| Where things live | AWS + Databricks | Databricks only |
| Scale ceiling | Per-container RAM | Managed VS index (millions of vectors) |
| Change tracking | None, the corpus is state on a laptop | Delta table + Change Data Feed |
Bedrock is still perfectly fine if you're already on AWS or need Nova Micro's specific pricing curve at scale. For an internal tool where the corpus is a modest set of decks growing over time and the traffic is a sales team, having one thing to maintain won.
Deploying to the Team
The app is a Databricks App, not a laptop process. databricks bundle deploy uploads the code plus the Job and endpoint definitions, and databricks.yml binds CAN_QUERY permissions on the FM, embedding, and (deferred) KA endpoints to the App's service principal, with no PAT in the deployed config. Databricks App auth is handled by the Databricks SDK's WorkspaceClient(), which resolves the SP OAuth token internally.
The URL lives at our internal portal. Every request is SSO-gated to the Databricks workspace, so anyone the workspace admin invites can reach it from any laptop on any network. That's the shape "public to the sales team" takes on Free Edition, where workspace membership becomes the access list.
The nightly ingest Job runs at 03:00 UTC and is UNPAUSED. Anyone with commit access can also trigger it on demand via databricks bundle run ingest after landing new content in the UC volume.
What Didn't Make It (Yet)
Agent Bricks Knowledge Assistant is in the deployed config as an on-deck answer path. Two KA instances were provisioned end-to-end during this migration (endpoint, knowledge source, indexed corpus, permissions) and the provider code and tests are shipped. Both 500'd on live queries in the Beta. When Databricks resolves that, flipping LLM_PROVIDER=agent_bricks is a one-line env change and we're on the managed retrieval and generation path with no other code motion.
Genie is on the same footing, with a provider stub, config env vars (DATABRICKS_GENIE_SPACE_ID, DATABRICKS_GENIE_WAREHOUSE_ID), and a documented plan for the tabular path. Our one tabular file (Objection Rebuttals.xlsx, 8 rows) is already handled by the RAG stack, so Genie doesn't earn its keep yet. The day we add a real analytical table like deal history, usage telemetry, or ARR by segment, Genie is a two-file change to turn on.
The Part That Actually Mattered
The question I actually cared about was whether we could assemble a working system out of the pieces a Free Edition workspace gives us today, and whether that system could grow without a rewrite. After two iterations, the answer is yes. The Iteration 1 shape (pure-Python cosine over a JSONL) worked with zero external dependencies, and Iteration 2 (Delta, Vector Search, and a scheduled ingest Job) is production-shaped without changing the app's request path in any meaningful way. The guardrails, modes, and LLM all stayed put, and only the retriever backend flipped.
If your corpus is 100 chunks, do Iteration 1. If it's going to grow with new decks, meeting notes, and case study updates, go straight to Iteration 2. The Free Edition constraints (compute quotas, throttled embedding endpoints, single-endpoint Vector Search tier) shape which tradeoffs you should be making, but they don't stop you from building a real production stack. They just force you to be honest about what you actually need.
The rip-out is done. The rebuild is live. The sales team is on it now.
