Prometheus Data Flow

End-to-end walkthrough of how metrics travel from scrape targets through storage to query results and alerts. Commit 01822f2

▶🗺 Full Pipeline Diagram 
  SCRAPE TARGETS  (exporters, node_exporter, etc.)
        │
        ▼
  ┌─────────────────────────────────────────────┐
  │           Discovery Manager                 │  discovery/manager.go
  │  (Kubernetes, AWS, Consul, File, DNS, …)    │
  └────────────────────┬────────────────────────┘
                       │  targetgroup.Group
                       ▼
  ┌─────────────────────────────────────────────┐
  │            Scrape Manager                   │  scrape/manager.go
  │  (one scrapePool per job, relabeling)       │
  └────────────────────┬────────────────────────┘
                       │  HTTP GET /metrics
                       ▼
  ┌─────────────────────────────────────────────┐
  │             Scrape Loop                     │  scrape/scrape.go
  │  (parse text/protobuf, apply limits)        │
  └────────────────────┬────────────────────────┘
                       │  samples / exemplars / histograms
                       ▼
  ┌─────────────────────────────────────────────┐
  │           fanoutStorage.Appender            │  storage/fanout.go
  └──────────────┬──────────────────────────────┘
                 │
       ┌─────────┴────────┐
       ▼                  ▼
  ┌─────────┐     ┌───────────────────────────────┐
  │  Local  │     │       Remote Storage           │
  │  TSDB   │     │  storage/remote/write.go       │
  │         │     │  WAL Watcher → QueueManager    │
  │ ┌─────┐ │     │  → HTTP POST remote_write      │
  │ │ WAL │ │     └───────────────────────────────┘
  │ └──┬──┘ │
  │    │    │
  │ ┌──▼──┐ │
  │ │Head │ │   tsdb/head.go
  │ │ +   │ │   (in-memory XOR chunks, memSeries)
  │ │mmap │ │
  │ └──┬──┘ │
  │    │ compaction
  │ ┌──▼──────┐ │
  │ │ Blocks  │ │   tsdb/block.go
  │ │(on disk)│ │   (index + chunk files)
  │ └─────────┘ │
  └─────────────┘
        │
        ▼  (Querier)
  ┌─────────────────────────────────────────────┐
  │              PromQL Engine                  │  promql/engine.go
  │  (parse AST → select → eval → result)       │
  └──────────────┬──────────────────────────────┘
                 │
       ┌─────────┴─────────┐
       ▼                   ▼
  ┌─────────┐       ┌─────────────────────┐
  │ HTTP    │       │   Rule Manager       │  rules/manager.go
  │ API     │       │ (alerting + record.) │
  │/api/v1/ │       └──────────┬──────────┘
  └─────────┘                  │ firing alerts
                               ▼
                        ┌─────────────┐
                        │  Notifier   │  notifier/manager.go
                        │ Alertmanager│
                        └─────────────┘
▶🚀 Startup Sequence

All components are wired together in cmd/prometheus/main.go. Initialization order matters — storage must be ready before scrapers start.

  1. Config loaded via config.LoadFile(); flags parsed into flagConfig
  2. Local TSDB opened: tsdb.Open() replays WAL, rebuilds Head
  3. remote.NewStorage() — remote write queues created per endpoint
  4. storage.NewFanout(local, remote) — unified write surface
  5. discovery.NewManager() starts — SD providers launch goroutines
  6. scrape.NewManager() starts — waits for first config sync from discovery
  7. promql.NewEngine() created with timeout, lookback delta, limits
  8. rules.NewManager() loads rule files, starts evaluation goroutines
  9. Web server starts — /api/v1/query, /metrics, /targets, etc.
  10. SIGHUP triggers reloadConfig() — all reloader funcs called in order
▶📦 Components
🎯
Scrape Pipeline
Discovery → target groups → HTTP scrape → parse metrics → Appender
💾
TSDB Write Path
fanoutStorage → WAL → Head memSeries → m-map chunks → disk Blocks
📡
Remote Write
WAL watcher → QueueManager shards → protobuf HTTP POST
🔍
PromQL Engine
Parse AST → index lookup → chunk decode → expression eval
🔔
Alerting & Rules
Rule groups → PromQL eval → state machine → Alertmanager notify
▶🔌 Core Interfaces

All components communicate through a small set of interfaces defined in storage/interface.go.

storage/interface.go — key interfaces L82 
// Storage combines all sub-interfaces
type Storage interface {
    SampleAndChunkQueryable           // read path
    Appendable                        // write path (deprecated Q2 2026)
    AppendableV2                      // write path (new)
    StartTime() (int64, error)
    Close() error
}

// Appender — transactional write interface
type Appender interface {
    Append(ref SeriesRef, l labels.Labels, t int64, v float64) (SeriesRef, error)
    AppendExemplar(ref SeriesRef, l labels.Labels, e exemplar.Exemplar) (SeriesRef, error)
    AppendHistogram(ref SeriesRef, l labels.Labels, t int64, h *histogram.Histogram, ...) (SeriesRef, error)
    Commit() error
    Rollback() error
}

// Querier — read interface for a time range
type Querier interface {
    Select(ctx, sortSeries bool, hints *SelectHints, matchers ...*labels.Matcher) SeriesSet
    LabelNames(ctx, hints *LabelHints, matchers ...) ([]string, Warnings, error)
    LabelValues(ctx, name string, hints *LabelHints, matchers ...) ([]string, Warnings, error)
    Close() error
}
SeriesRef is a uint64 opaque handle returned by Append() that can be reused in subsequent calls to skip label lookup — key hot-path optimization.
▶🧱 Core Data Types
TypePackageDescription
labels.Labels model/labels Sorted key=value pairs identifying a time series
storage.SeriesRef storage uint64 handle for a series within an Appender transaction
chunks.HeadSeriesRef tsdb/chunks uint64 monotonic ID assigned per series in the Head
histogram.Histogram model/histogram Native histogram with bucket counts, sum, count
exemplar.Exemplar model/exemplar High-cardinality trace ID attached to a metric sample
chunkenc.Chunk tsdb/chunkenc Compressed byte slice of (timestamp, value) pairs
▶🔄 Config Reload

Prometheus supports live reload (SIGHUP or /-/reload POST). Each component registers a reloader function.

cmd/prometheus/main.go — reloader list ~L1073 
reloaders = []reloader{
    {name: "db_storage",       reloader: localStorage.reloader},
    {name: "remote_storage",   reloader: remoteStorage.ApplyConfig},
    {name: "web_handler",      reloader: webHandler.ApplyConfig},
    {name: "query_engine",     reloader: func(cfg *config.Config) error { ... }},
    {name: "scrape",           reloader: scrapeManager.ApplyConfig},
    {name: "scrape_sd",        reloader: discoveryManagerScrape.ApplyConfig},
    {name: "notify",           reloader: notifierManager.ApplyConfig},
    {name: "notify_sd",        reloader: discoveryManagerNotify.ApplyConfig},
    {name: "rules",            reloader: rulesManager.Update},
    {name: "tracing",          reloader: tracingManager.ApplyConfig},
}
Reloaders run sequentially. If any fails, the reload is aborted and the previous config stays active.