Slashy's Cross-App Agent Architecture: Memory, Semantic Search, and Custom Tools

Slashy (YC S25) is a cross-app agent that reads data and executes actions across disconnected SaaS APIs. The architecture exposes three core primitives: custom tools, semantic search, and personalized memory. The demo shows an agent pulling financial data from one app, cross-referencing it with another, and triggering an action in a third without manual API stitching.

The interesting question is not what it does. The interesting question is how it maintains context, routes queries, and handles failure when one API rate-limit cascades into a multi-step workflow spanning three other services.

Architecture primitives

Slashy’s agent layer sits between the LLM and a collection of app connectors. Each connector wraps a SaaS API (Slack, Google Calendar, Notion, financial platforms) and exposes a set of tools. The agent decides which tools to invoke based on user intent, then coordinates state across tool calls.

Custom tools

Custom tools are the agent’s interface to external systems. Each tool is a function signature with input schema, output schema, and execution logic. When a user asks “What did I spend on AWS last month?”, the agent:

1. Parses intent
2. Selects the AWS billing tool
3. Invokes it with date range parameters
4. Returns structured data to the LLM for summarization

The tool registry is versioned. If AWS changes its API, the tool definition updates without retraining the agent. This decouples the LLM from API churn.

Semantic search

Semantic search indexes data from connected apps. When the agent needs to answer “Find the email where Sarah mentioned the Q3 budget”, it:

1. Embeds the query
2. Searches across email, Slack, and document stores
3. Returns ranked results with source attribution

The search layer is not a unified index. Each app connector maintains its own embedding store. The agent queries multiple stores in parallel, then merges results by relevance score. This avoids the complexity of syncing all app data into a single vector database.

Personalized memory

Personalized memory stores user preferences, past actions, and context from previous sessions. When the agent sees “Book a meeting with the team”, it recalls:

• The user’s definition of “the team”
• Preferred meeting times
• Calendar conflicts from prior interactions

Memory entries are scoped by user ID and app context. A memory entry from Slack does not leak into a Google Calendar tool call unless explicitly referenced. This prevents context pollution.

State management across tool calls

The agent maintains a session object that tracks:

• Active tool calls
• Intermediate results
• Error states
• Retry counters

When a tool call fails, the agent checks the error type. If it’s a rate limit (HTTP 429), the agent pauses that tool and tries an alternative path. If it’s an auth failure (HTTP 401), the agent prompts the user to reconnect the app.

The session object is ephemeral. It lives in memory for the duration of a task, then discards. Long-term state (user preferences, app credentials) lives in a persistent store.

Failure modes and mitigation

Failure Mode	Impact	Mitigation
API rate limit on one app	Blocks dependent tool calls	Exponential backoff, fallback to cached data
Auth token expiration	Agent cannot access app data	Proactive token refresh, user re-auth prompt
Tool call timeout	Partial task completion	Timeout per tool (not per workflow), surface partial results
Conflicting data from two apps	Agent halts or returns wrong answer	Timestamp-based conflict resolution, user confirmation step
Memory entry collision	Agent uses stale context	Memory entries versioned by timestamp, TTL on cached preferences

The most dangerous failure is silent: the agent completes a task using stale data from one app and fresh data from another, then returns a confident but incorrect answer. Slashy mitigates this by timestamping every data fetch and surfacing data age in the UI.

Tool invocation flow

class AgentSession:
    def __init__(self, user_id, connected_apps):
        self.user_id = user_id
        self.tools = self._load_tools(connected_apps)
        self.memory = self._load_memory(user_id)
        self.active_calls = []

    def execute_task(self, user_query):
        intent = self._parse_intent(user_query)
        plan = self._generate_plan(intent, self.tools)
        
        for step in plan:
            tool = self.tools[step.tool_name]
            try:
                # Timeout is per-tool, not per-workflow
                result = tool.invoke(step.params, timeout=10)
                self.memory.store(step.tool_name, result)
                self.active_calls.append((step, result))
            except RateLimitError:
                self._handle_rate_limit(step)
            except AuthError:
                return self._prompt_reauth(step.tool_name)
        
        return self._synthesize_response(self.active_calls)

    def _handle_rate_limit(self, step):
        # Exponential backoff or fallback to cached data
        cached = self.memory.get_cached(step.tool_name)
        if cached and cached.age < 3600:
            return cached.data
        else:
            time.sleep(2 ** step.retry_count)
            step.retry_count += 1

This is pseudocode, not production code. The real implementation likely uses async I/O for parallel tool calls and a more sophisticated retry policy. The key point is that each tool call is isolated, errors are caught per-tool, and the agent decides whether to fail fast or degrade gracefully.

Security boundaries

Each app connector runs with scoped credentials. The agent cannot access data from App A using credentials from App B. When a user connects Slack, the agent receives an OAuth token scoped to Slack’s API. That token never touches the Google Calendar connector.

Memory entries are encrypted at rest and scoped by user ID. The agent cannot read another user’s memory, even if both users connect the same apps.

Tool invocations are logged for audit. If a user asks “Did the agent access my bank account?”, the log shows which tools were called, when, and with what parameters.

Observability

The agent emits structured logs for:

• Tool call latency
• Error rates per app connector
• Memory cache hit/miss ratio
• Query routing decisions (which tools were considered, which were selected)

These logs feed a monitoring dashboard. If AWS’s API starts returning 500 errors, the ops team sees a spike in tool call failures before users complain.

The agent also tracks token usage per LLM call. If a query triggers 20 tool calls and burns through 100k tokens, the cost is attributed to that user and surfaced in billing.

Deployment shape

Slashy runs as a hosted service. Users connect apps via OAuth, then interact with the agent through a web UI or Slack bot. The agent backend is a Python service (likely FastAPI or Flask) that orchestrates LLM calls, tool invocations, and memory lookups.

The tool registry is a JSON schema stored in a database. When a new app connector is added, the schema updates and the agent picks it up without redeployment.

Memory and session state live in Redis for fast access. Long-term data (user profiles, app credentials) lives in Postgres.

At scale, Redis memory limits become a constraint. If session state grows beyond a few megabytes per user, the architecture needs a tiered cache (hot data in Redis, warm data in Postgres). LLM token costs also scale linearly with tool calls. A workflow that fans out to 10 apps can burn 50k tokens per query, which adds up quickly at high user volumes.

When context breaks

The hardest problem is maintaining context across long workflows. If a user asks “Find the invoice Sarah sent last week, then create a calendar event to discuss it”, the agent must:

1. Search email for the invoice
2. Extract Sarah’s email address and the invoice date
3. Check Sarah’s calendar availability
4. Create the event with a link to the invoice

If step 2 fails (the invoice is a PDF attachment, not inline text), the agent cannot complete step 4. The user sees “I couldn’t find the invoice” and has to start over.

Slashy handles this by surfacing intermediate results. If the agent finds the email but can’t parse the invoice, it shows the email and asks the user to confirm the date manually. This turns a hard failure into a soft degradation.

Technical verdict

Use Slashy’s architecture when:

• You need to coordinate actions across 3+ SaaS apps that don’t natively integrate
• Your users already have OAuth-connected accounts in those apps
• You can tolerate eventual consistency (data from one app may lag behind another)
• You want to avoid building and maintaining custom API integrations for each app
• Your workflows are exploratory or semi-structured (research, reporting, ad-hoc queries)
• Token costs are acceptable for your use case (expect 10k-50k tokens per multi-app workflow)
• You can surface intermediate results and handle graceful degradation when one tool fails

Avoid this architecture when:

• You need real-time consistency across apps (financial transactions, inventory updates)
• Your users cannot grant OAuth access to their apps (compliance, security policy)
• Your workflows require complex branching logic that LLMs struggle to plan reliably
• You need to guarantee exactly-once execution (the agent may retry tool calls on transient failures)
• You operate at high scale where LLM token costs and Redis memory limits become bottlenecks
• Your failure modes require rollback or compensation logic (the agent does not maintain transactional guarantees)
• You need sub-second response times (multi-app workflows typically take 5-15 seconds)

The core insight is that cross-app agents are coordination engines, not data pipelines. They trade strong consistency for flexibility. If your use case demands both, you need a different architecture (event-driven workflows with durable queues, transactional guarantees, and explicit rollback logic). If your use case tolerates stale data and occasional retries in exchange for zero integration code, this pattern fits.

The architecture shines for financial research workflows: pulling transaction data from one app, cross-referencing it with invoices in another, and generating reports in a third. It struggles with high-frequency trading or real-time portfolio rebalancing where milliseconds matter and partial execution is unacceptable.

Source links

• Launch HN: Slashy (YC S25)
• Demo video