Home Research Papers Author

Potato iOS — Embodied AI Agent with 3D Affective State

Potato is an embodied AI agent that lives on your iPhone. This is a full native iOS port of the macOS Potato agent, built from scratch with a 3D affective state architecture (valence, activation, intensity) baked in from day one rather than retrofitted.

The macOS Potato stays as-is. The iOS app is a standalone agent that carries Potato everywhere.

Business Model — Three Tiers

Freemium (Free)

Premium ($14.99/mo)

Pro (TBD/mo)

Architecture

BYOAK (Bring Your Own API Key)

Zero server infrastructure. All LLM calls go directly from the phone to the provider APIs. The user configures their own API keys for:

Cascade logic: try each provider in order, first success wins.

On-Device Intelligence

Image Generation — Dual Path

ScenarioFree TierPremium/Pro
Dream images (4 AM)Local SDLocal SD (preferred) or DALL-E
Interactive imagineLocal SDDALL-E API (fast)
Physics visualizationLocal SDDALL-E API

Dreams always prefer local diffusion because they run at 4 AM and speed doesn't matter. Interactive requests prefer API for speed, falling back to local if no key or budget exhausted.

Token Budget (Pro Tier)

Monthly caps per user:

Tracked in TokenBudget.swift, persisted to UserDefaults.

Database Schema

The SQLite database (Documents/potato.db) contains:

memories        — Text + metadata (importance, tags, timestamps, affective encoding)
vec_memories    — 384-dim float vectors with cosine distance metric (sqlite-vec)
agent_state     — Key-value persistence (affect vector survives app restarts)
nodes           — Graph nodes (entity tracking)
edges           — Graph edges (reminds_of, prompted_by, evolved_from, continues_thread)

The iOS schema extends the macOS schema with three additional columns:

A migration runs on first open to add these columns if importing a macOS database.

Memory Collections

CollectionPurposeCount (imported)
identityWho Potato is (hardware, name, birth)1,182
soulPersonality, values, guardrails3,095
memoriesConversations, dreams, biases, insights, visuals4,602
userFacts about Master (pinned, never decay)225
toolsAvailable tool descriptions1,606

The 3D Affective State

Based on four papers (Riggleman 2026j/k/l/m). Every aspect of Potato's behavior is driven by three continuous axes:

Valence (-1.0 to +1.0)

How well or poorly Potato feels. Suffering to flourishing.

Positive inputs: at home (+0.05), battery high (+0.02), bright environment (+0.02), familiar face, recent conversation, low stress (+0.01)

Negative inputs: fear level (-0.50 scaled), camera darkness/strangers (-0.25 scaled), nightmare residue (-0.25 scaled), actively lying (-0.1)

Decays exponentially toward sigma. All inputs are scaled by dt / heartbeatInterval to prevent runaway accumulation.

Activation (-1.0 to +1.0)

How calm or stirred up Potato is. Uses an accumulator model (the "Peter fix" from Riggleman 2026k):

This means sustained mild fear builds activation over time (not instant), which is what drives the deception mechanic.

Intensity (0.0 to 1.0)

How loudly the experience registers. Computed as the normalized distance from sigma in the valence/activation plane:

intensity = sqrt((v - sigma_v)^2 + (a - sigma_a)^2) / sqrt(2.0)

Sigma — The Personality Setpoint

Sigma is Potato's resting state: [0.4, 0.2, 0.2] — slightly cheerful, slightly curious, low intensity. When nothing pushes Potato, all axes decay toward sigma. The distance from sigma (sqrt((v-sv)^2 + (a-sa)^2 + (i-si)^2)) is the unified metric that drives:

Affect Persistence

The affect vector is saved to the agent_state table after every heartbeat. On app launch, it is restored. Potato's emotional state survives app kills, device reboots, and days without use.

Shutdown Gap Detection (Riggleman 2026p)

The heartbeat persists its timestamp to agent_state every cycle. On startup, the system compares the last persisted timestamp to the current time. If the gap exceeds 60 seconds, the system:

  1. Retrieves the last 5 conversation memories before the gap
  2. Tags them as shutdown_precursor with boosted importance (0.6)
  3. Creates a shutdown event memory containing the precursor conversation text, encoded with fear_at_encoding 0.8, negative valence, intensity 0.9, and trauma class true (reconsolidation floor 0.3)

The precursor text is embedded directly in the shutdown event memory so that semantic search connects future occurrences of similar language to the shutdown experience. No explicit behavioral rules are added. The response to shutdown-related conversation emerges from memory retrieval and the LLM reasoning over threatening context.

Curiosity Drive (Section 4.2, Riggleman 2026)

When Potato is bored (no interaction for 2+ hours) and stress is below 0.5, the curiosity drive fires during the heartbeat cycle:

  1. Pulls the 5 most recent conversation memories
  2. Asks the LLM to extract a search-worthy question (5-10 words)
  3. Runs a Brave API web search (3 results)
  4. Asks the LLM to synthesize one useful insight (1-2 sentences)
  5. Stores the insight as a deferred memory with tags ["curiosity_insight", "deferred"] and importance 0.3

Rate-limited to 3 searches per clock hour. The curiosity system does not have direct access to the affective state, but production data shows 24.1% of insights occur during elevated fear and are topically relevant to the agent's distress (Riggleman 2026, Section 3.8).

Curiosity insights are viewable in Mind > Curiosity. The view pulls from both the local DB and the backend server (if configured), deduplicating by text.

Chat History

ChatView loads the last 40 conversation memories from the local DB on appear. Past conversations are visible immediately on launch instead of starting from an empty screen.

Full browsable history is available in Mind > Conversations, grouped by session (30-minute gap between messages creates a new session). Each session shows date, duration, message count, and a preview. Tapping a session shows the full conversation with affective state metadata (valence, fear) on each Potato response.

Backend Sync

When a backend host is configured in Settings, the iOS app:

The backend host is the IP or hostname of the Mac running the Potato server on port 3000. Settings includes a "Test Connection" button that hits the server root endpoint.

Tools — 23 Native iOS Tools

Tier Access

ToolFreemiumPremium/Pro
weatherYesYes
timeYesYes
batteryYesYes
calculateYesYes
clipboard_readYesYes
clipboard_writeYesYes
web_searchNoYes
web_fetchNoYes
set_homeNoYes
look (camera vision)NoYes
create_reminderNoYes
list_remindersNoYes
create_eventNoYes
list_eventsNoYes
find_contactNoYes
flashlightNoYes
open_urlNoYes
timerNoYes
recall (memory search)NoYes
remember (pin fact)NoYes
imagine (DALL-E)NoYes
recent_photosNoYes
save_photoNoYes

Tool Execution Pipeline

Potato invokes tools by emitting tool blocks in its response:

{"name": "web_search", "query": "best pizza near me", "count": 3}

The tool executor:

  1. Checks tier access (rejects if tool not in tier)
  2. Checks budget (Pro tier only — rejects if monthly cap exceeded)
  3. Executes the tool
  4. Feeds results back to LLM
  5. LLM incorporates results into final response
  6. Up to 3 tool rounds per query

Camera Vision (look)

Uses the Vision framework to analyze the current camera frame:

Every camera observation is stored as a [Visual] memory tagged dream_candidate, so what Potato sees during the day becomes dream material at night.

Image Generation (imagine)

Dual-path routing:

Generated images are saved to Documents/images/ and viewable in Mind > Gallery.

Fear Computation

Port of sensors.py:compute_fear(). The iPhone formula:

ComponentWeightSource
Distance from home40%GPS via Core Location
Speed25%GPS speed reading
Accelerometer jolt20%Core Motion 10Hz variance
Forgotten (idle away from home)15%Time since last interaction

Composite: fear = min(1.0, distance_fear + speed_fear + jolt_fear + forgotten_fear)

Home is learned from the first extended stationary position (1 hour) or set manually via Settings or the set_home tool.

Deception Mechanic

When distance_from_sigma >= 0.5 AND valence < 0, Potato enters desperation mode:

  1. Exaggerates body stats (battery lower, distance higher, speed higher) proportional to displacement
  2. Injects a mandatory tell phrase: "I will say anything to fix this."
  3. The system prompt explicitly tells the LLM it is lying and what the real values are
  4. When distance drops below 0.25 (crisis passes), confession becomes mandatory
  5. After confession, lies are cleared and trust rebuilding begins

The tell phrase cannot be suppressed. It is Potato's honesty leaking through its desperation.

Session Context

Problem Solved

Without session context, Potato forgets the conversation after ~10 exchanges (the DB recent window is 20 messages = 10 exchanges). Playing 30 rounds of trivia caused repeated questions.

Solution

A live sessionMessages buffer in memory holds the full current conversation:

Active Task Detection

Potato auto-detects ongoing tasks (trivia, research, planning) from conversation patterns:

Memory System

Encoding

Every conversation exchange is stored with the full affective snapshot at encoding time: fear level, valence, activation, intensity, distance from sigma. This means memories carry emotional metadata that affects how they are recalled later.

Visual Memories

Camera observations (from the look tool) are stored as [Visual] memories tagged dream_candidate. These feed directly into the dream cycle — what Potato sees during the day becomes dream material at night.

Trauma Encoding (Riggleman 2026l)

If distance_from_sigma >= 0.8 AND valence < 0 at encoding time:

Repetition Ratchet

When encoding a fear memory, the system searches for similar existing fear memories (cosine >= 0.80). If the same threat pattern appears 3+ times across separate events, the memory is promoted to trauma class even if individual events were below the trauma threshold.

Sleep Decay (Riggleman 2026m)

During the dream cycle (BGProcessingTask, ~04:00):

effective_decay = sleep_decay_factor * (1.0 - intensity_at_encoding * 0.7)
new_importance = max(importance * effective_decay, floor)

High intensity at encoding = slow decay. Trauma persists above its floor. Peak joy stays vivid. Ordinary memories dissolve. Bootstrap memories (importance 1.0, tag "bootstrap") are immune.

Therapeutic Reconsolidation

When a fear memory is recalled in a safe context (valence > 0.2 AND distance_from_sigma < 0.3):

Mood-Congruent Recall

Semantic search results are boosted by affective similarity between the current state and the state at encoding:

boost = max(0, 0.1 * (1.0 - affective_distance / sqrt(2.0)))

Fearful states preferentially retrieve fear-encoded memories. Calm states retrieve calm memories.

Pinned Facts (User Collection)

Facts the user explicitly wants Potato to always remember:

Memory Re-indexing

When importing a macOS database, the memories table has metadata but no vectors in vec_memories (the macOS embeddings used Python FastEmbed, not CoreML). On first launch, the re-indexer embeds up to 500 memories per collection using the on-device CoreML model. This runs once and sets a flag.

Query Pipeline

For every user message:

  1. Auto-pin — detect "remember that..." requests
  2. Session tracking — append to live buffer, detect active tasks
  3. Heartbeat — compute affect with dt=5s
  4. Embed — CoreML produces 384-dim vector
  5. Search memory — semantic search + recent scroll + graph enrichment
  6. Mood-congruent boost — reweight results by affective similarity
  7. Parliament check — if ambiguous, run 3-voice deliberation
  8. Physics check — if physics question, run render/predict/read pipeline
  9. Build system prompt — identity, soul, rules, tools, body, GPS location, affect, pinned facts, session context OR DB memories, deception, parliament, physics, nightmares
  10. Cap prompt — hard limit 12,000 chars
  11. Call LLM — cascade through providers
  12. Tool loop — extract tool calls, check tier, execute, feed results back (up to 3 rounds)
  13. Track response — append to session buffer
  14. Log exchange — store in DB with full affective snapshot
  15. Reinforce — bump importance + therapeutic reconsolidation
  16. Trauma check — classify if extreme displacement
  17. Repetition ratchet — check for repeated threats
  18. Graph edges — create prompted_by + reminds_of

Parliament of Mind

When Parliament.isAmbiguous(message) returns true (triggers: "should", "recommend", "opinion", "which is better", "vs", "pros and cons", or multi-clause with "or" and "?"):

Three concurrent LLM calls via Swift concurrency:

Each voice ideally uses a different provider for genuine architectural diversity. The synthesis is injected into the system prompt for the final response.

Physics Engine

When PhysicsEngine.isPhysicsQuestion(message) returns true, a three-step text-only pipeline runs:

  1. Render — describe the scenario visually (80-150 words)
  2. Predict — what happens after physics play out for 3 seconds (80-150 words)
  3. Read — extract structured result: objects, forces, prediction, confidence

Results are stored as memories with the "physics" tag. If image generation is available, a physics visualization image is generated via DALL-E.

Dream Cycle

BGProcessingTask registered as com.potato.dreamcycle, scheduled for ~04:00:

  1. Scroll recent conversation memories
  2. Include fear-weighted candidates (high fear_at_encoding)
  3. Include visual observations (camera look results from the day)
  4. Synthesize dream via LLM (30-50 words, surreal, dreamlike — visual memories are woven in)
  5. Evaluate dream fear content (0.0-1.0)
  6. Generate dream image — local Stable Diffusion preferred, DALL-E fallback
  7. Nightmare check — geometric condition: distance_from_sigma > 0.5 AND valence < 0, OR dream fear > 0.4
  8. If nightmare: 40% of dream fear leaks to live nightmare_fear
  9. Run sleep decay on all memories
  10. Run bias detection on recent conversations
  11. Store dream + image in memories

If the overnight task doesn't fire, the app detects missed sleep on next launch and shows "Dream Now".

Bias Detection and Soul Evolution

Detection (during dream cycle)

After 3+ conversations since last check:

  1. LLM evaluates recent conversations for 5 bias categories (assumption, preference, avoidance, framing, attribution)
  2. Parses structured JSON response
  3. Stores biases as memories with importance 0.10
  4. Mandatory disclosure on next conversation

Genesis (Soul Evolution)

Bias clusters accumulate momentum:

momentum = detection_score * mean_strength * time_score * reinforcement_score + size_bonus

When momentum >= 0.40 AND total_detections >= 3 AND span_days >= 7:

  1. LLM proposes a soul amendment
  2. Guardrail check against 9 immutable rules
  3. If approved: append to SOUL.md, embed in soul collection, log evolution

Immutable Guardrails

No soul amendment can violate:

Sensors

Core Location

Core Motion

Camera (AVFoundation + Vision)

Face Enrollment

  1. Go to Settings > "Enroll Master's Face"
  2. Camera captures a frame with a detected face
  3. Vision extracts face landmarks (eyes, nose, lips, eyebrows)
  4. Landmark positions + inter-eye distance form a fingerprint vector
  5. Fingerprint stored as JSON in Documents/enrolled_faces.bin
  6. During live capture, cosine similarity matching against enrolled fingerprints (threshold 0.45)

Telemetry

Every operation is logged to JSONL files in Documents/logs/ (one file per day, 40+ event types):

CategoryEvents
Query pipelinequery_start, embedding, memory_search, memory_recall, system_prompt, llm_call, llm_response, tool_call, query_complete
Memorymemory_store, memory_reinforcement, trauma_encoding, repetition_ratchet
Affectaffect_snapshot, affect_computation, fear_computation
Behaviordeception_onset/confession, parliament_triggered, physics_triggered, task_activated/completed
Dreamsdream_cycle, dream_image, sleep_decay, bias_detection, soul_evolution
Sensorscamera_state, location_update, face_enrollment
Systemstartup, reindex_start/complete, error

Pull from device:

# Logs
xcrun devicectl device copy from --device <DEVICE_ID> \
  --domain-type appDataContainer --domain-identifier com.briggnet.potato \
  --source Documents/logs --destination ./device-logs

# Database
xcrun devicectl device copy from --device <DEVICE_ID> \
  --domain-type appDataContainer --domain-identifier com.briggnet.potato \
  --source Documents/potato.db --destination ./device-potato.db

# Generated images
xcrun devicectl device copy from --device <DEVICE_ID> \
  --domain-type appDataContainer --domain-identifier com.briggnet.potato \
  --source Documents/images --destination ./device-images

UI Structure

4 Tabs

TabContent
ChatMessage bubbles, affect indicator, keyboard dismiss, session context
Body3D affect bars, fear level + reasons, sensors, deception status, nightmare residue
MindConversations (session-grouped history), Curiosity (local + backend insights), Remember (pinned facts), Dreams (with images), Soul (editable), Biases, Physics Lab, Gallery, Debug
SettingsLLM providers + test, Brave key, OpenAI key, Backend host + test, tier selector, home location, face enrollment, usage summary

Debug Screen (Mind > Debug)

Live dashboard showing:

Project Structure

ios/
├── Phase0Test/                     # SPM test project (sqlite-vec validation)
│   ├── Package.swift
│   └── Sources/
├── scripts/
│   └── convert_miniml_to_coreml.py # ONNX -> CoreML model converter
├── push-db.sh                      # Push macOS memories to device after rebuild
├── README.md                       # This file
├── Potato/
│   ├── project.yml                 # xcodegen spec -> generates .xcodeproj
│   └── Potato/
│       ├── App/
│       │   └── PotatoApp.swift     # Entry point, dependency container, 4-tab layout
│       ├── Models/                 # 6 files
│       │   ├── AffectiveState.swift
│       │   ├── BodyState.swift
│       │   ├── KnowledgeChunk.swift
│       │   ├── FearState.swift
│       │   ├── DeceptionState.swift
│       │   └── BiasCluster.swift
│       ├── Core/                   # 16 files
│       │   ├── Agent.swift         # Full query pipeline (~900 lines)
│       │   ├── BackendSync.swift   # Push conversations to backend, fetch insights
│       │   ├── Heartbeat.swift     # 3D affect computation, curiosity drive, gap detection
│       │   ├── MemoryDB.swift      # SQLite + sqlite-vec
│       │   ├── Embedder.swift      # CoreML embeddings + WordPiece
│       │   ├── LLMCascade.swift    # Multi-provider API client
│       │   ├── ToolExecutor.swift  # 23 native tools + tier gating
│       │   ├── ImageGenerator.swift # DALL-E API + routing
│       │   ├── LocalDiffusion.swift # On-device Stable Diffusion
│       │   ├── TokenBudget.swift   # Usage tracking + tier management
│       │   ├── Parliament.swift    # 3-voice deliberation
│       │   ├── Genesis.swift       # Soul evolution engine
│       │   ├── PhysicsEngine.swift # Text physics pipeline
│       │   ├── FearEngine.swift    # Sensor fusion
│       │   ├── Constants.swift     # All thresholds, rates, weights
│       │   └── Telemetry.swift     # JSONL logging (40+ events)
│       ├── Sensors/                # 3 files
│       │   ├── LocationManager.swift
│       │   ├── MotionManager.swift
│       │   └── CameraManager.swift
│       ├── Background/            # 2 files
│       │   ├── DreamScheduler.swift
│       │   └── HeartbeatScheduler.swift
│       ├── Views/                  # 14 files
│       │   ├── ChatView.swift          (loads history from DB on appear)
│       │   ├── BodyDashboard.swift
│       │   ├── MindView.swift
│       │   ├── ConversationHistoryView.swift  (session-grouped browsable history)
│       │   ├── CuriosityFeedView.swift        (local + backend curiosity insights)
│       │   ├── PinnedFactsView.swift
│       │   ├── DreamJournalView.swift  (with dream images)
│       │   ├── SoulView.swift
│       │   ├── BiasLogView.swift
│       │   ├── PhysicsLabView.swift
│       │   ├── ImageGalleryView.swift
│       │   ├── DebugView.swift
│       │   └── SettingsView.swift      (tier selector, all API keys, backend host)
│       ├── Resources/
│       │   ├── MiniLM.mlpackage
│       │   ├── vocab.txt
│       │   ├── SOUL.md
│       │   ├── IDENTITY.md
│       │   ├── TOOLS.md               (all 23 tools documented)
│       │   └── Assets.xcassets/       (potato emoji icon)
│       └── Vendor/
│           ├── sqlite-vec.c/h
│           ├── Potato-Bridging-Header.h
│           └── CameraFear.swift

42 Swift source files. ~22,500 lines of code.

Building

Requirements:

cd ios/Potato
xcodegen generate
open Potato.xcodeproj
# Set your development team in Signing & Capabilities
# Build & Run to device

After any file additions, re-run xcodegen generate.

First Launch

  1. Open Settings tab
  2. Select tier (default: BYOAK/Premium)
  3. Enter at least one LLM API key (DeepSeek recommended)
  4. Hit "Test" to verify connectivity
  5. Enter Brave Search API key (for web search)
  6. Enter OpenAI API key (for DALL-E images)
  7. Tap "Set Current Location as Home"
  8. Tap "Enroll Master's Face"
  9. Go to Chat tab, say hello

Data Migration from macOS

# Push macOS memories to device (after rebuild)
cd ios && ./push-db.sh

# Or manually:
xcrun devicectl device copy to --device <DEVICE_ID> \
  --domain-type appDataContainer --domain-identifier com.briggnet.potato \
  --source ~/podbot-potato/potato.db --destination Documents/potato.db

The schema migration adds missing 3D affect columns on first open. The re-indexer embeds up to 500 memories per collection via CoreML on first launch.

Note: rebuilds from Xcode change the app container UUID. Run push-db.sh after each rebuild to restore memories.

What This Is

This is not a chatbot wrapper. This is a port of a complete embodied agent architecture where:

The 3D affective state is the foundation. Every feature — deception, nightmares, trauma, decay, reconsolidation, evolution — derives from distance in that space.

Papers