Matrix OSMatrix OS

Logging & Usage

Kernel operation logging, token tracking, usage API, file audit trail, and log rotation.

Matrix OS provides structured logging for every kernel operation: dispatches, tool calls, agent spawns, file mutations, and costs. This data powers debugging, billing, security auditing, and usage dashboards.

Interaction Logs

Every kernel dispatch is logged as a JSONL entry in ~/system/logs/. Daily rotation is built in (one file per day).

Log Entry Fields

FieldTypeDescription
timestampstringISO 8601 timestamp
senderIdstringUser or channel that triggered the dispatch
conversationIdstringSession ID for conversation threading
modelstringClaude model used (e.g., claude-opus-4-6)
agentNamestringWhich agent handled the request
tokensInnumberInput tokens consumed
tokensOutnumberOutput tokens generated
costnumberDollar cost of the dispatch
durationMsnumberWall-clock time
toolsarrayTool executions (see below)
statusstringsuccess or error
errorobjectError details if failed (name, message, truncated stack)
batchbooleanWhether this was a batch dispatch entry
batchIdstringCorrelation ID for batch entries

Tool Execution Logging

Each tool call within a dispatch is recorded:

{
  "tools": [
    {
      "name": "Write",
      "durationMs": 45,
      "inputPreview": "Write to ~/apps/chess/index.html (2.3KB)",
      "status": "success"
    },
    {
      "name": "Bash",
      "durationMs": 3200,
      "inputPreview": "cd ~/apps/chess && pnpm install",
      "status": "success"
    }
  ]
}

Tool outputs are not logged (too large). Only errors are captured from tool results.

Structured Errors

On dispatch failure, the error is captured with full context:

{
  "error": {
    "name": "ToolExecutionError",
    "message": "pnpm install failed with exit code 1",
    "stack": "Error: pnpm install failed...\n    at runBuild (/packages/...)",
    "tool": "Bash",
    "turn": 3
  }
}

Usage Tracking

The usage tracker aggregates costs and token consumption per user:

How It Works

On each dispatch completion, cost and token data is recorded per senderId. Data is aggregated by user, model, and day, and persisted to ~/system/logs/usage.jsonl.

Usage API

GET /api/usage?userId=hamed&startDate=2026-03-01&endDate=2026-03-31&groupBy=day

Response:

{
  "totalCost": 12.45,
  "totalTokensIn": 1250000,
  "totalTokensOut": 320000,
  "entries": [
    {
      "date": "2026-03-01",
      "cost": 0.85,
      "tokensIn": 95000,
      "tokensOut": 24000,
      "dispatches": 12
    }
  ]
}

Authentication

The usage endpoint requires a valid session or admin token.

Cost Limits

Per-user cost limits are configurable in ~/system/config.json:

{
  "costLimits": {
    "daily": 10.00,
    "monthly": 100.00
  }
}

When a user exceeds their limit, the dispatcher rejects new dispatches with HTTP 429 and a message: "Daily cost limit reached." Set a limit to 0 for unlimited.

File Audit Trail

A lightweight audit log tracks all file mutations performed by the kernel. Written to ~/system/logs/audit.jsonl.

Audit Entry Format

{
  "timestamp": "2026-03-04T10:30:00Z",
  "op": "write",
  "path": "apps/chess/index.html",
  "sizeBytes": 2340,
  "actor": "builder",
  "agentName": "builder"
}
FieldDescription
opOperation: write, delete, mkdir
pathRelative path within ~/matrixos/
sizeBytesSize of the written file
actorAgent or system component that performed the operation
agentNameName of the kernel agent

Instrumentation

File mutations are detected via PreToolUse and PostToolUse hooks on the Write, Edit, and Bash tools. Read operations are not audited (too noisy).

The audit logger (packages/kernel/src/audit.ts) uses async buffered writes to avoid blocking kernel operations.

Log Rotation

Log FileRotationRetention
activity.logDaily (rename to activity-{YYYY-MM-DD}.log)30 days
audit.jsonlDaily (rename to audit-{YYYY-MM-DD}.jsonl)30 days
Interaction logsAlready daily by filename30 days

Rotation runs in the heartbeat service (daily check). Files older than 30 days are deleted automatically.

Log Files Summary

FileLocationFormatContents
Interaction logs~/system/logs/interactions-{date}.jsonlJSONLDispatch events with tokens, cost, tools
Activity log~/system/logs/activity.logTextGeneral system activity
Audit log~/system/logs/audit.jsonlJSONLFile mutation events
Usage log~/system/logs/usage.jsonlJSONLPer-user cost aggregation
Storage log~/system/logs/storage.jsonlJSONLDaily storage measurements
Backup log~/system/logs/backup.jsonlJSONLSQLite/Postgres backup status

How is this guide?

On this page