Projects and Documents

A project is a collection of JSON documents together with indexes and configuration for retrieval. No data or configuration is shared between projects, so these operate as isolated data containers together with algorithms for retrieval. We shall call these JSON documents TellusR documents.

In this section we will go through how to insert and maintain documents.

Projects can be added via the Dashboard or via the API.

About TellusR document structure

Rules

A TellusR document has a mandatory ID field id. Inserting a document with the same ID overwrites any existing document associated with that ID. TellusR documents follow a chunking rule that semantic search uses to create embeddings. For documents with a content_segmented field, TellusR expects a list of JSON objects, each containing a content_segment field.

{
  "id": "5932",
  "content_segmented": [{
    "content_segment": "The first chunk."
  }]
}

A TellusR document may have no chunks, one chunk, or multiple chunks.

TellusR documents may have additional fields.

{
  "title": "An example document",
  "category": "example",
  "id": "5932",
  "file_type": "PDF",
  "content_segmented": [{
    "page_no": 1,
    "content_segment": "The first chunk."
  }]
}

The project that the documents belong to auto-generates a schema based on the type of the fields. The rule is that once you’ve set a JSON type for a particular field, such as category, then you may not later update a document with a conflicting type.

Chunks in a TellusR document

The indexes associated with a project are based on the chunks of a document for both regular and semantic indexes. This is so that hybrid search and RAG algorithms can point back to the exact location in a large document where a certain hit is found. It flattens the content_segmented list to yield chunks. Here is how the example document transforms into chunks for semantic and keyword indexes:

graph LR
    subgraph "TellusR Document"
    A["{<br/>'title': 'An example document',<br/>'category': 'example',<br/>'id': '5932',<br/>'file_type': 'PDF',<br/>'content_segmented': [<br/>{<br/>'page_no': 1,<br/>'content_segment': 'The first chunk.'<br/>}<br/>]<br/>}"]
    end

    A -- flatten --> B

    subgraph "Flattened Chunk"
    B["{<br/>'id': '5932_0',<br/>'title': 'An example document',<br/>'category': 'example',<br/>'document_id': '5932',<br/>'file_type': 'PDF',<br/>'page_no': 1,<br/>'content_segment': 'The first chunk.'<br/>}"]
    end

    classDef left text-align:left
    class A,B left

We say that a JSON field is stringlike if it is a string or a list of strings. We say that a JSON field is numberlike if it is a number or a list of numbers. The rule is that every field that is either stringlike or numberlike at a chunk level is indexed and also becomes filterable in the indexes within the project.

Adding documents

Unstructured data

You can upload files (PDF, Word, HTML, Txt, etc.) directly using the import APIs. TellusR will automatically structure the files into semantically relevant chunks and set metadata at the chunk level.

We recommend using these endpoints for unstructured data:

• POST /tellusr/api/v1/${YOUR_PROJECT}/upload-file: Synchronous upload. Returns confirmation on success. Generally slower if not invoked in parallel.
• POST /tellusr/api/v1/${YOUR_PROJECT}/upload-file-background: Asynchronous upload. Returns a task ID immediately, which can be used to check the processing status via GET /upload-status/{task-id}.

Parameters for upload:

These parameters can be sent as query parameters:

Request Body:

The upload endpoints accept multipart/form-data with the following parts:

Metadata format:

Metadata provided in the metadata part will be associated with the document. The document ID can also be submitted here.

{
       "id": "00001",
       "category": "example-file",
       "tags": ["examples"]
}

Structured data (JSON)

If your documents are already structured, you can upload them directly using the update operations. This gives you full control over chunks and metadata.

Endpoints:

• POST /tellusr/api/v1/${YOUR_PROJECT}/update-doc: Add or replace a single document.
• POST /tellusr/api/v1/${YOUR_PROJECT}/update-many-docs: Add or replace documents as a batch.

Add a single document

Submit a document as JSON. The only mandatory field is id. This operation adds a new document if no document with the same ID exists, or replaces an existing document with the same ID.

{
   "id": "04535",
   "title": "Hello",
   "topic": "World"
}

Add documents as a batch

Post a batch of documents. Each document must have an id field.

{
    "docs": [
        {
            "id": "04535",
            "title": "Hello",
            "topic": "World"
        }
    ]
}