Self-healing for production bugs

Your app crashed.
Something else caused it.

trace-repro finds the hidden event that really broke production, proves it with a test, and ships the real fix — automatically. No war rooms. No guesswork. No band-aids.

Built on FalkorDB/GitLab Duo/GitLab CI

Live incident Order #8812
  1. Crash
  2. Trace
  3. Found
  4. Proved
  5. Fixed

A customer's checkout just failed.

38msThe gap where the real bug was hiding
Days → minsFrom "what happened?" to the root cause
0Humans needed to find, prove and fix it
1 testKept forever, so the bug never comes back

Monitoring tools show you where it hurts.
They never show you who hit you.

Modern apps do hundreds of things at once. Payments retry, carts expire, customers click "Pay" — often all on the same order. When two of them collide, the error shows up somewhere else entirely. So your team stares at the wrong place.

Anatomy of a 38-millisecond disaster

  1. 12:04:17.374

    The bank re-sends a payment confirmation

    The same message arrives twice. The order gets updated again — with nothing stopping it.

  2. 12:04:17.386

    A cleanup timer cancels the order

    It thinks the cart was abandoned. At that exact moment, it wasn't.

  3. 12:04:17.412

    The customer's checkout fails

    They see an error — or worse, get charged twice. This is the only thing that shows up as an error.

The old way costs days.
And it usually gets the fix wrong.

Without trace-repro

1–3 days

of a senior engineer's time per incident

  • Someone digs through logs from different systems, by hand, trying to guess what else happened.
  • AI coding tools patch the line that crashed — hiding the error instead of fixing it.
  • Bad data quietly piles up: double charges, wrong balances, angry customers.
  • The same bug returns next month, in a slightly different form.
With trace-repro

Minutes

fully automatic, from alarm to fix

  • The real cause is found by connecting every event that touched the same order.
  • A test recreates the exact crash before a single line of code changes.
  • The fix goes where the trouble started — not where it surfaced.
  • That test stays in your codebase forever. The bug can't sneak back.

Find it. Prove it. Fix it.

01

Find

It connects the dots.

trace-repro maps every request, background job and database change into a living map. When something breaks, it walks backwards to every event that touched the same record.

FalkorDB · cross-trace graph traversal
02

Prove

It recreates the crash on purpose.

Before touching any code, it writes a test that replays those exact events, in the exact order, milliseconds apart — and watches it fail. The bug is now proven, not guessed.

pytest · GitLab CI · red pipeline
03

Fix

It repairs the root cause.

It follows the map to the code that started the chain reaction, adds the missing safeguard, and reruns the test until it passes. Then it ships the fix to production.

Row lock · idempotency key · GitLab CI

One incident, start to finish.
Zero humans.

    01 / 06 Incident open

    What the engineer sees
    
              

    Logs are a list.
    Reality is a web.

    Traditional tools file events by request — like sorting mail by envelope. trace-repro files them by what they touched. Suddenly three unrelated-looking events point at the same order, and the answer is obvious.

    What your tools see50,214 events · no connection
    • GET /products/442 200
    • POST /login 200
    • POST /webhooks/payment 200
    • GET /cart 200
    • job cart-expiry ok
    • GET /search?q=shoes 200
    • POST /reviews 201
    • POST /checkout/capture 500
    • GET /orders/7710 200

    Three lines matter. Nothing tells you which.

    What trace-repro sees3 events · 1 shared order
    Payment retry Cart cleanup Checkout failed Order #8812

    Every event that touched the order, linked. The culprit is obvious.

    Take the graph away and trace-repro simply can't work. That's the point — the graph does the real detective work.

    Others see the symptom.
    We see the cause.

    Capability Monitoring toolsDatadog, Sentry, Honeycomb AI coding assistantsDevin, SWE-Agent, Claude Code Replay toolsReplay.io, Speedscale, RR trace-reproCausal self-healing
    Connects separate events that collidedNoNoNoYes
    Proves the bug before changing codeNoNoManualYes
    Fixes the cause, not the symptomNoRarelyNoYes
    Pinpoints the few events that matterNoNoReplays everythingTop 2–4 of 50,000
    Keeps your app fastYesYes20–50% slowerYes
    Runs end-to-end without a humanNoPartlyNoYes

    From alarm to all-clear,
    on autopilot.

    Covers every one of GitLab's nine DevSecOps stages — and runs natively in GitLab CI.

    01

    Detect

    A failing workload triggers trace-repro instantly.

    Monitor
    02

    Investigate

    The graph reveals which events collided — and opens an incident.

    Plan
    03

    Prove

    A new test reproduces the crash. Pipeline goes red.

    Verify
    04

    Fix

    The real cause is patched. Same test turns green.

    Create · Verify
    05

    Ship

    Scanned, packaged and rolled out safely to production.

    Secure · Package · Configure · Release
    06

    Report

    A before/after map is posted as proof. Incident closed.

    Govern

    For the engineers in the room.

    OpenTelemetry spans and ORM/Redis state mutations stream into FalkorDB. Independent traces are linked through shared (:StateEntity) nodes, and the GitLab Duo Agent queries them via MCP.

    // Execution layer
    (:Trace {id, entry_endpoint})
    (:Span  {id, start_ns, end_ns, status})
    
    // Stateful entity layer — the bridge between traces
    (:StateEntity   {entity_type: "Order", primary_key: "8812"})
    (:StateMutation {seq_no, ts_ns, field, old_val, new_val, lock_held: false})
    
    // Code & Git layer
    (:ASTFunction {name, file_path, line_range})
    (:GitCommit   {sha, mr_iid})
    
    // Causal edges
    (:Span)-[:EXECUTED_FUNCTION]->(:ASTFunction)
    (:Span)-[:WROTE_ENTITY {ts_ns, delta_json, tx_isolation}]->(:StateEntity)
    (:Span)-[:READ_ENTITY  {ts_ns, snapshot_version}]->(:StateEntity)
    (:StateMutation)-[:HAPPENED_BEFORE {delta_ms: 38, concurrent_overlap: true}]->(:StateMutation)
    // From the crashing span, hop across shared StateEntity nodes to find
    // every OTHER independent trace that mutated the same entity before the crash
    MATCH (crash_trace:Trace)-[:HAS_SPAN]->(crash_span:Span {id: $crash_span_id})
    MATCH (crash_span)-[r_read:READ_ENTITY|WROTE_ENTITY]->(entity:StateEntity)
    MATCH (other_trace:Trace)-[:HAS_SPAN]->(mutator_span:Span)-[r_write:WROTE_ENTITY]->(entity)
    WHERE other_trace.id <> crash_trace.id
      AND r_write.ts_ns <= r_read.ts_ns
      AND (r_read.ts_ns - r_write.ts_ns) < 60000000000 // 60s causal window
    MATCH (mutator_span)-[:EXECUTED_FUNCTION]->(upstream_fn:ASTFunction)
    MATCH (crash_span)-[:EXECUTED_FUNCTION]->(crash_fn:ASTFunction)
    RETURN
      entity.primary_key AS collided_entity,
      other_trace.entry_endpoint AS interfering_request,
      other_trace.payload_json AS interfering_payload,
      (r_read.ts_ns - r_write.ts_ns) / 1000000.0 AS race_window_ms,
      r_write.lock_held AS was_locked,
      upstream_fn.file_path AS upstream_mutator_file,
      upstream_fn.name AS upstream_mutator_function,
      crash_fn.name AS downstream_crash_symptom
    ORDER BY r_write.ts_ns ASC
    // Exact chain of state transitions: entity creation → corrupted read.
    // Produces the ordered requests, payloads and ms delays for the replay test.
    MATCH (entity:StateEntity {primary_key: "8812"})
    MATCH (first:StateMutation {seq_no: 0})-[:MUTATED]->(entity)
    MATCH (crash:Span {id: $crash_span_id})
    MATCH p = shortestPath((first)-[:HAPPENED_BEFORE|CAUSED_READ*..12]->(crash))
    UNWIND nodes(p) AS step
    OPTIONAL MATCH (s:Span)-[:EMITTED]->(step)
    OPTIONAL MATCH (t:Trace)-[:HAS_SPAN]->(s)
    RETURN t.entry_endpoint AS request, t.payload_json AS payload,
           step.field AS field, step.old_val AS old_val, step.new_val AS new_val,
           step.ts_ns / 1000000.0 AS ts_ms
    ORDER BY ts_ms ASC
    # tests/repro/test_incident_8812.py — generated from graph:incident_8812
    import asyncio
    import pytest
    from tests.repro.harness import Barrier, replay, wallet_credits
    
    @pytest.mark.asyncio
    async def test_incident_8812_cross_trace_race(client, worker, seed_order):
        await seed_order(id=8812, status="PENDING", amount=4500)
        barrier = Barrier(parties=3)
    
        # Causal subgraph: A1 (-38ms) → B2 (-26ms) → C3 (0ms)
        webhook  = replay(client.post, "/webhooks/payment",
                          json={"order_id": 8812, "event": "payment.retry"},
                          barrier=barrier, offset_ms=0)
        expiry   = replay(worker.expire_cart, order_id=8812,
                          barrier=barrier, offset_ms=12)
        checkout = replay(client.post, "/checkout/capture",
                          json={"order_id": 8812}, barrier=barrier, offset_ms=38)
    
        _, _, res = await asyncio.gather(webhook, expiry, checkout)
    
        assert res.status_code == 200, res.json()   # RED: 500 InvalidStateError
        assert await wallet_credits(8812) == 1      # no double-credit
      # payments/webhooks.py — upstream mutator, not the crashing line
      def apply_webhook(db, event):
    -     order = db.query(Order).get(event.order_id)
    +     order = (db.query(Order)
    +                .filter_by(id=event.order_id)
    +                .with_for_update()            # SELECT ... FOR UPDATE
    +                .one())
    +     if order.last_event_id == event.id:     # idempotency guard
    +         return order
          order.status = transition(order.status, event)
    +     order.last_event_id = event.id
          db.commit()
    
      # graph diff: lock_held false → true · race_condition_reproduced → resolved

    Three minutes.
    One "wait, what?" moment.

    Watch trace-repro autonomously capture a cross-trace race, query the FalkorDB causal context layer, synthesize a failing test, and deploy the verified fix with zero human prompts.

    trace-repro-demo.mp4 — Autonomous Cross-Trace Bug Reproducer
    1080p HD YouTube ↗
    0:00 — 0:40

    Normal AI gets it wrong

    We break a live checkout. A standard AI reads the error and slaps on a band-aid that hides the problem.

    Jump to 0:00
    0:40 — 1:30

    The graph reveals the truth

    In FalkorDB, Order #8812 sits in the middle — with two separate events pointing into it, 38ms apart. Anyone gets it instantly.

    Jump to 0:40
    1:30 — 2:30

    Red, then green

    With zero prompts, the agent commits a failing test first. Thirty seconds later, the real fix lands — and the pipeline turns green.

    Jump to 1:30
    2:30 — 3:00

    Healed for good

    Production is redeployed, and the reproduction test joins the regression suite. This bug can never return.

    Jump to 2:30

    Good questions.
    Straight answers.

    What kind of bugs does it catch?

    The nastiest kind: double charges, wrong wallet balances, orders stuck in impossible states. Bugs caused by two things happening at almost the same moment — which are exactly the ones humans struggle to find.

    Why not just use an AI coding assistant?

    They only see the line that crashed, so they patch the symptom. trace-repro proves the cause first with a failing test, then fixes it where it started.

    Do I have to rebuild my app?

    No. A lightweight middleware reports what each request touched. There's no heavy recording, and no need to replay hours of traffic.

    Will it change code without anyone checking?

    It can run fully hands-off, or stop at the merge request for approval. Either way, every fix comes with proof: a test that failed before, passes after, and a before/after map of what changed.

    Stop guessing.
    Start proving.

    Find the real cause. Prove it with a test. Fix it for good — before your customers notice.