Grafana Integration

This guide covers connecting Grafana to the IceGate query APIs: Loki (logs) and Tempo (traces), both implemented, plus Prometheus (metrics), which is planned and not yet functional.

Prerequisites

Verify Query Service Health

Before configuring Grafana, verify that the Query service is ready:

# Check Loki API (port 3100)
        curl http://localhost:3100/ready
        
        # Check Tempo API (port 3200)
        curl http://localhost:3200/ready
        
        # Check Prometheus API (port 9090)
        curl http://localhost:9090/-/ready

All endpoints should return HTTP 200.

Add Data Sources

Loki Data Source (Logs)

IceGate implements the Grafana Loki API on port 3100.

  1. Go to Connections > Data sources > Add data source
  2. Select Loki
  3. Configure:
    • URL: http://icegate-query:3100
    • Under HTTP Headers, add:
      • Header: X-Scope-OrgID
      • Value: your tenant ID (e.g., default)
  4. Click Save & Test

Provisioning YAML

# grafana/provisioning/datasources/icegate-loki.yaml
        apiVersion: 1
        datasources:
          - name: IceGate Logs
            type: loki
            access: proxy
            url: http://icegate-query:3100
            jsonData:
              httpHeaderName1: X-Scope-OrgID
            secureJsonData:
              httpHeaderValue1: default
            isDefault: true

Tempo Data Source (Traces)

IceGate implements the Grafana Tempo API on port 3200.

Warning

The Tempo API provides trace retrieval and search, and TraceQL is supported for /api/search; TraceQL features that are not yet implemented return 501 Not Implemented.

  1. Go to Connections > Data sources > Add data source
  2. Select Tempo
  3. Configure:
    • URL: http://icegate-query:3200
    • Under HTTP Headers, add:
      • Header: X-Scope-OrgID
      • Value: your tenant ID
  4. Click Save & Test

Provisioning YAML

# grafana/provisioning/datasources/icegate-tempo.yaml
        apiVersion: 1
        datasources:
          - name: IceGate Traces
            type: tempo
            access: proxy
            url: http://icegate-query:3200
            jsonData:
              httpHeaderName1: X-Scope-OrgID
              tracesToLogs:
                datasourceUid: icegate-loki
                tags: ['service.name']
                mappedTags: [{ key: 'service.name', value: 'service_name' }]
                mapTagNamesEnabled: true
                filterByTraceID: true
                filterBySpanID: false
            secureJsonData:
              httpHeaderValue1: default
            uid: icegate-tempo

Prometheus Data Source (Metrics)

Warning

The Prometheus data source will not work yet. IceGate mounts the Prometheus API routes on port 9090, but every one of them - including the metadata endpoints (labels, series, label/{name}/values) - returns 501 Not Implemented. Only /-/ready responds.

Use the Loki data source with LogQL metric queries for log-based metrics, or Arrow Flight SQL for general-purpose SQL over the same data. The steps below are recorded for when the API lands.

  1. Go to Connections > Data sources > Add data source
  2. Select Prometheus
  3. Configure:
    • URL: http://icegate-query:9090
    • Under HTTP Headers, add:
      • Header: X-Scope-OrgID
      • Value: your tenant ID
  4. Click Save & Test

Provisioning YAML

# grafana/provisioning/datasources/icegate-prometheus.yaml
        apiVersion: 1
        datasources:
          - name: IceGate Metrics
            type: prometheus
            access: proxy
            url: http://icegate-query:9090
            jsonData:
              httpHeaderName1: X-Scope-OrgID
            secureJsonData:
              httpHeaderValue1: default
            uid: icegate-prometheus

Complete Provisioning Example

Deploy all three data sources at once:

# grafana/provisioning/datasources/icegate.yaml
        apiVersion: 1
        datasources:
          - name: IceGate Logs
            type: loki
            access: proxy
            url: http://icegate-query:3100
            jsonData:
              httpHeaderName1: X-Scope-OrgID
            secureJsonData:
              httpHeaderValue1: default
            isDefault: true
            uid: icegate-loki
        
          - name: IceGate Traces
            type: tempo
            access: proxy
            url: http://icegate-query:3200
            jsonData:
              httpHeaderName1: X-Scope-OrgID
              tracesToLogs:
                datasourceUid: icegate-loki
                tags: ['service.name']
                mappedTags: [{ key: 'service.name', value: 'service_name' }]
                mapTagNamesEnabled: true
                filterByTraceID: true
            secureJsonData:
              httpHeaderValue1: default
            uid: icegate-tempo
        
          - name: IceGate Metrics
            type: prometheus
            access: proxy
            url: http://icegate-query:9090
            jsonData:
              httpHeaderName1: X-Scope-OrgID
            secureJsonData:
              httpHeaderValue1: default
            uid: icegate-prometheus

Cross-Signal Navigation

Logs to Traces

IceGate stores trace_id and span_id fields in log records. Configure Grafana to link from log lines to traces:

  1. In the Loki data source settings, go to Derived fields
  2. Add a derived field:
    • Name: TraceID
    • Regex: "trace_id":"([a-fA-F0-9]+)"
    • URL: (leave empty)
    • Internal link: Enable, select IceGate Traces

Now clicking a trace ID in log results opens the trace view.

Traces to Logs

In the Tempo data source settings, the tracesToLogs configuration (shown in the provisioning YAML above) adds a "Logs for this span" button to the trace view.

Multi-Tenant Configuration

For environments with multiple tenants, configure separate data sources per tenant:

apiVersion: 1
        datasources:
          - name: Logs (Team A)
            type: loki
            access: proxy
            url: http://icegate-query:3100
            jsonData:
              httpHeaderName1: X-Scope-OrgID
            secureJsonData:
              httpHeaderValue1: team-a
        
          - name: Logs (Team B)
            type: loki
            access: proxy
            url: http://icegate-query:3100
            jsonData:
              httpHeaderName1: X-Scope-OrgID
            secureJsonData:
              httpHeaderValue1: team-b

For dynamic per-user tenancy, see Multi-Tenancy.

Dashboard Examples

Log Explorer Dashboard

Create a dashboard with three panels:

  1. Logs panel - shows raw log entries:

    • Query: {service_name="my-service"}
    • Visualization: Logs

  2. Error rate time series - tracks error frequency:

    • Query: sum by (service_name) (rate({severity_text="ERROR"}[5m]))
    • Visualization: Time series

  3. Log volume stat - shows total log count:

    • Query: sum(count_over_time({service_name="my-service"}[1h]))
    • Visualization: Stat

Trace Explorer

  1. Navigate to Explore > select IceGate Traces
  2. Search by service name: enter service.name=my-service in the tags field
  3. Filter by minimum duration: set minDuration to 100ms
  4. Click a trace to view its span waterfall

Using IceGate as a Drop-In for Existing Grafana

If you have an existing Grafana setup with Loki, you can point it at IceGate by changing only the data source URL:

  1. Go to Connections > Data sources
  2. Edit your existing Loki data source
  3. Change URL from your Loki instance to http://icegate-query:3100
  4. Add the X-Scope-OrgID header if not already present
  5. Click Save & Test

Dashboards, alerting rules, and saved queries keep working as long as they stay within the endpoints and LogQL features IceGate implements - it serves a subset of the Loki read API, not all of it. Check the Loki API reference and the LogQL implementation status for anything a panel depends on, and re-test alert rules after switching.

Note

LogQL metric queries (rate(), count_over_time(), sum by(), etc.) are supported. See the LogQL implementation status for the full compatibility matrix.

Port Reference

API Port Grafana Data Source Type Status
Loki (logs) 3100 Loki Fully implemented
Tempo (traces) 3200 Tempo Retrieval and search; TraceQL supported (unimplemented features return 501)
Prometheus (metrics) 9090 Prometheus Planned; every route returns 501 except /-/ready

Next Steps

Previous
Querying Data
Next
Multi-Tenancy