Skip to content
Chatty the Lab

Backend Architecture

Project structure

Section titled “Project structure”
backend/src/
├── main.rs              # Entry point, AppState, server setup
├── config.rs            # AppConfig loaded from env vars
├── error.rs             # AppError type for unified error handling
├── auth/
│   ├── mod.rs
│   ├── middleware.rs     # JWT validation middleware
│   └── types.rs         # Claims struct
├── db/
│   ├── mod.rs
│   └── pool.rs          # SQLx connection pool creation
├── models/
│   ├── mod.rs
│   ├── session.rs       # Session, CreateSessionRequest
│   ├── chat.rs          # Chat, Message, ChatWithMessages, import/sync types
│   ├── context_item.rs  # ContextItem, search query types
│   └── user.rs          # User/profile model
├── plugins/
│   ├── mod.rs           # LlmProvider trait, PluginRegistry, build_registry()
│   ├── openai.rs        # OpenAI provider (SSE streaming)
│   ├── mistral.rs       # Mistral provider (SSE streaming)
│   ├── gemini.rs        # Gemini provider (SSE streaming)
│   ├── huggingface.rs   # HuggingFace Inference API
│   ├── ollama.rs        # Ollama provider (NDJSON streaming)
│   └── stream_utils.rs  # SSE and NDJSON stream parsers
├── routes/
│   ├── mod.rs           # Router composition with auth middleware
│   ├── sessions.rs      # Session CRUD endpoints
│   ├── chats.rs         # Chat CRUD, query, sync, import endpoints
│   ├── models.rs        # GET /models endpoint
│   ├── context_items.rs # Context items CRUD and semantic search
│   └── ws.rs            # WebSocket upgrade and message handler
└── services/
    ├── mod.rs
    └── embedding.rs     # Ollama embedding generation

Application state

Section titled “Application state”

The AppState struct is shared across all handlers via Axum’s state extraction:

pub struct AppState {
    pub db: PgPool,                      // SQLx PostgreSQL pool
    pub config: Arc<AppConfig>,          // Immutable configuration
    pub plugins: Arc<PluginRegistry>,    // HashMap<String, Arc<dyn LlmProvider>>
}

Plugin system

Section titled “Plugin system”

The core abstraction for LLM providers is the LlmProvider trait:

#[async_trait]
pub trait LlmProvider: Send + Sync {
    fn name(&self) -> &str;
    fn provider(&self) -> &str;
    fn model_id(&self) -> &str;


    async fn send(&self, messages: Vec<ChatMessage>) -> Result<String, AppError>;


    async fn send_stream(
        &self,
        messages: Vec<ChatMessage>,
    ) -> Result<Pin<Box<dyn Stream<Item = Result<String, AppError>> + Send>>, AppError>;
}

Each provider implements both send (blocking) and send_stream (token-by-token). The PluginRegistry is a HashMap<String, Arc<dyn LlmProvider>> built at startup by build_registry(), which checks for API keys and registers the corresponding models.

Adding a new provider

Section titled “Adding a new provider”
  1. Create a new file in plugins/ (e.g., anthropic.rs)
  2. Implement the LlmProvider trait
  3. Add a registration block in build_registry() in plugins/mod.rs
  4. Add the API key to AppConfig in config.rs

Stream parsing utilities

Section titled “Stream parsing utilities”

The stream_utils module provides two reusable parsers:

  • parse_sse_stream — Parses Server-Sent Events (data: {...}\n\n format). Used by OpenAI, Mistral, and Gemini providers. Handles data: [DONE] termination.
  • parse_ndjson_stream — Parses newline-delimited JSON (one JSON object per line). Used by Ollama.

Both accept a closure extract_token that pulls the token string from the parsed JSON, making them generic across different API response formats.

Routing

Section titled “Routing”

Routes are composed in routes/mod.rs:

pub fn create_router(state: AppState) -> Router {
    let base = &state.config.base_path;


    let api = Router::new()
        .merge(sessions::router())
        .merge(chats::router())
        .merge(models::router())
        .merge(context_items::router())
        .route_layer(middleware::from_fn_with_state(
            state.clone(), auth_middleware,
        ));


    let ws = ws::router();


    Router::new()
        .nest(&format!("{base}/api"), api)
        .nest(&format!("{base}/ws"), ws)
        .with_state(state)
}

All API routes under {base}/api are protected by the JWT auth middleware. The WebSocket route under {base}/ws handles authentication via a token query parameter (since browsers cannot set headers on WebSocket connections).

Authentication middleware

Section titled “Authentication middleware”

The auth middleware (auth/middleware.rs):

  1. Extracts the Authorization: Bearer <token> header
  2. Decodes the JWT using SUPABASE_JWT_SECRET with HS256 algorithm
  3. Validates the aud claim equals "authenticated"
  4. Injects Claims { sub, email, exp, iat } into request extensions
  5. Handlers access claims via axum::Extension<Claims>

WebSocket handler

Section titled “WebSocket handler”

The WebSocket endpoint at {base}/ws/chat accepts connections with query parameters session_id and token. The handler:

  1. Validates the JWT from the query parameter
  2. Splits the socket into sender/receiver halves
  3. Creates an mpsc channel for outgoing messages
  4. Spawns a send task that forwards channel messages to the WebSocket
  5. Processes incoming query and query_all messages, spawning a task per chat query
  6. Each task: verifies ownership, inserts user message, loads history, streams from provider, saves assistant message