Concepts

This section covers fundamental concepts of TellusR.

Background

TellusR comes with a built-in hybrid search and APIs to ingest structured and unstructured data. On top of this, TellusR comes with tools to generate assistants that can use this ingested data as context for LLMs.

Below are a few key concepts briefly summarized. This is not a comprehensive walkthrough, for that we refer to other literature.

Keyword Search / Regular search. This is the traditional way of searching, where exact matches on words or parts of words are required to hit a document. If you search for World Cup you will hit documents with the text World Cup 2026 because the text contains the excact phrase.

Semantic search. This is a modernized form of searching that emerged in the mid‑2000s and became widespread in the late 2010s and early 2020s through mainstream search engines. Instead of exact matches, this technology uses machine learning to generate vector embeddings for words and phrases, placing them in a high‑dimensional space where semantic distances can be measured.

For example, if you search for World Cup 2026, you are very likely to hit a document about Haaland because these concepts end up close to each other in the embedding space; and you are unlikely to hit a niche plant like Welwitschia mirabilis.

Indexing. To make a dataset of documents searchable, they must be put through a process which is referred to as indexing. An index is a way to organize the data such that it permits fast retrieval. If you have one million documents, then the most naive (and stupid) way to find a document containing Welwitschia mirabilis to look at each document one by one until you happen to come across it. An index is a clever way of organizing the data so that retrieval becomes easy.

Hybrid search. Hybrid search is the art of mixing two strategies into one joined search list. Hybrid search generally performs better than each of the two strategies alone.

graph LR
    ST[search terms] --> HS[hybrid search]
    HS --> KI[keyword index]
    HS --> SI[semantic index]

Retrieval. Retrieval means to not only do a search and look at a list of titles, but to also retrieve the exact phrases and passages from documents that are relevant to a query. A hybrid search that returns larges chunks of content is a form of retrieval.

RAG. RAG is short for Retrieval-Augmented Generation. This combines LLMs with retrieval; given a user question, allow the LLM to define search terms for a retrieval algorithm that returns enough material for the LLM to provide an answer. This is a strategy to make LLMs answer more grounded from traceable sources.

graph LR
    UQ[user question] --> LLM[LLM]
    LLM --> ST[search terms]
    ST --> HS[hybrid search]
    HS --> KI[keyword index]
    HS --> SI[semantic index]