Troubleshooting

This guide helps diagnose and resolve common issues with IceGate.

Service HealthService HealthService Health

Check Service StatusCheck Service StatusCheck Service Status

# Query service
        curl http://localhost:3100/ready
        
        # Ingest service
        curl http://localhost:4318/health
        

View Service LogsView Service LogsView Service Logs

# Docker Compose
        docker compose logs -f query
        docker compose logs -f ingest
        docker compose logs -f maintain
        

Connection IssuesConnection IssuesConnection Issues

Cannot Connect to Query ServiceCannot Connect to Query ServiceCannot Connect to Query Service

Symptoms:

• Connection refused on port 3100
• Timeout errors

Solutions:

1. Verify service is running:

   docker ps | grep query
           

2. Check port binding:

   netstat -tlnp | grep 3100
           

3. Check service logs for errors:

   docker compose logs query | tail -100
           

Cannot Connect to Object StorageCannot Connect to Object StorageCannot Connect to Object Storage

Symptoms:

• "Connection refused" to the object store
• S3 authentication errors

Solutions:

1. On a local RustFS deployment, verify the object store is running. The readiness path is RustFS's own - on AWS S3 or another provider, skip to step 3 instead:

   curl http://localhost:9000/health/ready
           

2. Check credentials:

   echo $AWS_ACCESS_KEY_ID
           echo $AWS_SECRET_ACCESS_KEY
           

3. Test the S3 connection. Drop --endpoint-url when the backend is real AWS S3:

   aws s3 ls --endpoint-url http://localhost:9000
           

Cannot Connect to CatalogCannot Connect to CatalogCannot Connect to Catalog

Symptoms:

• "Catalog unavailable" errors
• Table creation failures

Solutions:

1. On the default S3 catalog, confirm the catalog state object is readable - there is no catalog service to check:

   aws --endpoint-url http://localhost:9000 s3 ls s3://warehouse/catalog/root.json
           

   A missing root.json means migration never ran. Run maintain migrate create before anything else.

2. Check catalog configuration:

   catalog:
             backend: !s3
               warehouse: catalog
             warehouse: s3://warehouse/
             properties:
               bucket: warehouse
               region: us-east-1
               endpoint: http://rustfs:9000
           

3. On the REST backend only, verify Nessie is running:

   curl http://localhost:19120/api/v1/trees
           

Query IssuesQuery IssuesQuery Issues

Query Returns Empty ResultsQuery Returns Empty ResultsQuery Returns Empty Results

Possible Causes:

• Wrong tenant ID
• Time range outside data window
• Data not yet compacted

Solutions:

1. Verify tenant header:

   curl -H "X-Scope-OrgID: correct-tenant" ...
           

2. Check time range:

   # List available time range
           curl http://localhost:3100/loki/api/v1/labels \
             -H "X-Scope-OrgID: my-tenant"
           

3. Check WAL for recent data:

   aws s3 ls s3://warehouse/wal/ --recursive
           

Query TimeoutQuery TimeoutQuery Timeout

Symptoms:

• Queries take too long
• 504 Gateway Timeout

Solutions:

1. Add time range filter:

   {service_name="api"} | timestamp > 1h ago
           

2. Reduce result limit:

   curl ... --data-urlencode 'limit=100'
           

3. Check query plan:

   curl http://localhost:3100/loki/api/v1/explain \
             --data-urlencode 'query={service_name="api"}' \
             -H "X-Scope-OrgID: my-tenant"
           

Invalid Query SyntaxInvalid Query SyntaxInvalid Query Syntax

Symptoms:

• "parse error" responses
• 400 Bad Request

Solutions:

1. Validate LogQL syntax:

   • Labels must be in braces: {service_name="api"}
   • String values in quotes: "value"
   • Duration format: [5m], [1h]

2. Check for unsupported features:

   • Pipeline parsers (json, logfmt) not yet supported
   • Some aggregations not implemented

Ingestion IssuesIngestion IssuesIngestion Issues

Data Not AppearingData Not AppearingData Not Appearing

Symptoms:

• Sent data but query returns empty
• No errors from ingest

Solutions:

1. Verify data was accepted:

   curl -v -X POST http://localhost:4318/v1/logs \
             -H "X-Scope-OrgID: my-tenant" \
             -H "Content-Type: application/json" \
             -d '...'
           

2. Check WAL files:

   aws s3 ls s3://warehouse/wal/logs/ --recursive
           

3. Wait for compaction (or query WAL directly)

Ingestion ErrorsIngestion ErrorsIngestion Errors

Common Errors:

• 400 Bad Request: Invalid OTLP format
• 503 Service Unavailable: Storage unavailable
• 429 Too Many Requests: Rate limited

Solutions:

1. Validate OTLP payload format
2. Check storage connectivity
3. Reduce ingestion rate or scale ingest replicas

Performance IssuesPerformance IssuesPerformance Issues

Slow QueriesSlow QueriesSlow Queries

1. Add partition filters:

   {tenant_id="my-tenant", service_name="api"}
           

2. Limit time range:

   --data-urlencode 'start=1704067200'
           --data-urlencode 'end=1704153600'
           

3. Check table statistics:

   SHOW STATS FOR icegate.logs;
           

High Memory UsageHigh Memory UsageHigh Memory Usage

1. Reduce concurrent queries
2. Add query limits
3. Increase service memory allocation

Getting HelpGetting HelpGetting Help

If issues persist:

1. Collect diagnostic information:

   # Service logs
           docker compose logs > logs.txt
           
           # System info
           docker stats > stats.txt
           

2. Check GitHub Issues

3. Include:

   • IceGate version
   • Configuration (sanitized)
   • Error messages
   • Steps to reproduce

Next StepsNext StepsNext Steps

• Review Maintenance procedures
• Check Deployment configuration
• Understand the Architecture