Text & Context Expansion

Overview

The Text & Context Expansion module in Deeptrain allows AI agents to overcome the rigid constraints of predefined token limits. By utilizing a localized embedding database, Deeptrain enables your models to reference vast datasets and real-time information without needing to fit the entire corpus into the model's immediate context window.

This is achieved through a Retrieval-Augmented Generation (RAG) workflow where only the most relevant snippets of data are injected into the model's prompt based on the user's query.

Localized Embedding Database

Deeptrain manages a localized vector store that indexes your text data. This allows for high-speed semantic search across millions of data points.

Key Benefits:

• Persistent Memory: Store documents, logs, and knowledge bases locally.
• Semantic Retrieval: Find content based on meaning rather than just keyword matching.
• Reduced Latency: Localized storage ensures that data retrieval does not depend on third-party cloud database latency.

Real-time Content Sourcing

Unlike static training data, Deeptrain can be configured to pull information from live sources. This ensures that the AI agent has access to the most current information available.

Usage Example: Indexing and Querying

To expand your model's context, you first index your data sources and then query the connector to retrieve relevant context for your LLM.

from deeptrain import TextConnector

# Initialize the connector
connector = TextConnector(database_path="./local_embeddings")

# Source data from live sources or local files
sources = [
    "https://api.example.com/realtime-docs",
    "./internal_manual.pdf",
    "Company knowledge base text..."
]

# Index the data (Live Sourcing)
connector.index(sources)

# Retrieve relevant context for a specific query
query = "What are the latest compliance updates for 2024?"
context_snippet = connector.retrieve(query, top_k=5)

# The context_snippet can now be fed into any of the 200+ supported models
print(f"Retrieved Context: {context_snippet}")

API Reference

TextConnector

The primary class for managing text-based context expansion.

__init__(database_path: str, model_config: dict = None)

Initializes the localized embedding database.

• database_path: (String) The directory where the vector index will be stored.
• model_config: (Optional Dict) Configuration for the embedding model used to vectorize text.

index(data_sources: list[str]) -> bool

Processes and stores data into the local database.

• data_sources: (List of Strings) Can include file paths, URLs, or raw text strings.
• Returns: True if indexing was successful.

retrieve(query: str, top_k: int = 3) -> str

Queries the database to find the most relevant text segments.

• query: (String) The user input or question.
• top_k: (Integer) The number of relevant segments to return.
• Returns: A concatenated string of the most relevant data found.

Model Agnostic Integration

Deeptrain's text expansion works independently of the underlying LLM. Whether you are using a private model (like Llama 3 or Mistral) or a public API (like GPT-4), the retrieved context can be prepended to your prompt:

prompt = f"Context: {context_snippet}\n\nUser Question: {query}"
response = llm.generate(prompt)

By separating the Context Retrieval from the Inference, Deeptrain ensures that your agents remain intelligent and informed, regardless of the specific model's native context window size.