Text & Embeddings
Text & Embeddings
Deeptrain's text processing capabilities allow AI agents to transcend the physical limitations of a model's context window. By leveraging a localized embedding database, you can store massive datasets and retrieve only the most relevant information in real-time, providing your models with a "long-term memory" and access to live data sources.
Localized Embedding Database
Instead of sending your entire dataset to an LLM provider or relying on expensive cloud-based vector stores, Deeptrain utilizes a localized approach. This ensures low-latency retrieval and maintains data privacy while allowing your agent to interact with millions of tokens of information.
Key Benefits:
- Infinite Context: Bypass token limits by injecting only relevant snippets into the prompt.
- Real-time Retrieval: Fetch content from live data sources dynamically.
- Model Agnostic: Use these embeddings with any of the 200+ supported private or open-source models.
Basic Usage
To manage text and embeddings, use the EmbeddingsManager interface. This allows you to ingest text data and query it later for context-aware responses.
Ingesting Text Data
Before an agent can retrieve information, the data must be indexed into the localized database.
from deeptrain import EmbeddingsManager
# Initialize the manager
em = EmbeddingsManager(db_path="./local_store")
# Add text content to the local database
em.add_text(
text="Deeptrain is a multi-modal data connector for LLMs.",
metadata={"source": "documentation", "topic": "overview"}
)
# Bulk ingest from multiple strings
texts = ["First paragraph...", "Second paragraph..."]
em.add_bulk_text(texts)
Retrieving Relevant Context
When a user asks a question, use the query method to find the most semantically similar fragments from your local store.
# Query the database for the top 3 most relevant matches
results = em.query(
prompt="What is Deeptrain?",
top_k=3
)
for match in results:
print(f"Content: {match['text']}")
print(f"Score: {match['score']}")
API Reference
add_text(text, metadata)
Indexes a single string into the localized vector store.
- Inputs:
text(str): The raw text to be embedded.metadata(dict, optional): Key-value pairs to store alongside the text for filtering.
- Returns:
str: The unique ID of the indexed document.
query(prompt, top_k)
Performs a semantic search against the localized database.
- Inputs:
prompt(str): The natural language query or user input.top_k(int): The number of relevant results to return.
- Returns:
List[Dict]: A list of objects containingtext,metadata, and a similarityscore.
clear_database()
Wipes the localized embedding database. Note: This action is irreversible.
- Returns:
bool: True if successful.
Real-time Content Integration
Deeptrain can be configured to fetch content from live sources (e.g., web scrapers, API feeds) and update the local embedding database on-the-fly. This ensures that when your AI agent queries the database, it receives the most up-to-date information available, rather than relying on static training data.
To implement real-time updates, wrap your data ingestion in a scheduled task or event trigger that pushes new content to add_text().