graph TB subgraph app["应用层"] HTTP["HTTP Server
OpenAI Compatible API"] end subgraph sched["调度层"] FE["Frontend"] PROC["Processor"] ROUTER["KV-Aware Router"] end subgraph infer["推理层"] W1["Worker 1
vLLM/TRT-LLM/SGLang"] W2["Worker 2"] W3["Worker N"] PW["Prefill Worker"] end subgraph runtime["运行时层"] RT["Distributed Runtime"] REG["Component Registry"] PIPE["Pipeline Engine"] end subgraph comm["通信层"] ETCD[("etcd
服务发现")] NATS[("NATS
事件总线")] NIXL["NIXL
KV 传输"] end HTTP --> FE FE --> PROC PROC --> ROUTER ROUTER --> W1 ROUTER --> W2 ROUTER --> W3 W1 --> PW W1 --> RT W2 --> RT W3 --> RT PW --> RT RT --> ETCD RT --> NATS W1 -.-> NIXL PW -.-> NIXL

# 创建命名空间
namespace = runtime.namespace("production")
namespace_dev = runtime.namespace("development")

graph TB subgraph ns["命名空间隔离"] subgraph prod["production"] P1["Worker 1"] P2["Worker 2"] end subgraph dev["development"] D1["Worker 1"] D2["Worker 2"] end end

# 定义组件
@service(dynamo={"namespace": "production"})
class VllmWorker:
    @dynamo_endpoint()
    async def generate(self, request):
        ...

# 端点引用格式
endpoint: dynamo.Processor.chat/completions
#         ^命名空间  ^组件     ^端点名

graph LR subgraph endpoint["端点结构"] NS["Namespace
dynamo"] --> COMP["Component
Processor"] COMP --> EP["Endpoint
chat/completions"] end

ServiceArgs:
  workers: 4  # 4 个服务实例
  resources:
    gpu: 1    # 每个实例 1 个 GPU

graph LR IN["输入"] --> OP1["Operator 1"] OP1 --> OP2["Operator 2"] OP2 --> OUT["输出流"]

sequenceDiagram participant Client participant HTTP participant Frontend participant Processor participant Router participant Worker participant KVCache Client->>HTTP: POST /v1/chat/completions HTTP->>Frontend: forward request Frontend->>Processor: process request Note over Processor: Tokenization Note over Processor: Chat Template Processor->>Router: route(token_ids) Note over Router: 查询 KV Index Note over Router: 计算匹配分数 Note over Router: 选择最优 Worker Router->>Worker: generate(request) loop Token Generation Worker->>KVCache: read/write KV Worker-->>Frontend: stream token Frontend-->>HTTP: SSE event HTTP-->>Client: token end Worker-->>Frontend: finish Frontend-->>HTTP: [DONE] HTTP-->>Client: stream end

graph TB subgraph produce["事件生产"] W1["Worker 1"] --> |kv_stored| NATS[("NATS")] W2["Worker 2"] --> |kv_stored| NATS W1 --> |load_metrics| NATS W2 --> |load_metrics| NATS end subgraph consume["事件消费"] NATS --> |kv_events| IDX["KV Indexer"] NATS --> |load_metrics| AGG["Metrics Aggregator"] IDX --> ROUTER["Router"] AGG --> ROUTER end

graph LR subgraph sync["状态同步"] LOCAL["本地状态"] --> |定期发布| NATS[("NATS")] NATS --> |广播| OTHER["其他节点"] OTHER --> |更新| REMOTE["远程缓存"] end

graph TB subgraph control["控制平面"] ETCD[("etcd")] REG["服务注册"] DISC["服务发现"] REG --> ETCD ETCD --> DISC end subgraph event["事件平面"] NATS[("NATS")] PUB["发布者"] SUB["订阅者"] PUB --> NATS NATS --> SUB end subgraph data["数据平面"] NIXL["NIXL"] GPU1["GPU 1"] GPU2["GPU 2"] GPU1 --> |RDMA| GPU2 end

stateDiagram-v2 [*] --> Created: 创建 Created --> Registered: 注册到 etcd Registered --> Running: 启动服务 Running --> Healthy: 健康检查通过 Healthy --> Running: 持续运行 Running --> Stopping: 收到停止信号 Stopping --> Deregistered: 从 etcd 注销 Deregistered --> [*]