Maintenance

This guide covers routine maintenance operations for IceGate.

Schema MigrationSchema MigrationSchema Migration

Initial SetupInitial SetupInitial Setup

Create all Iceberg tables for the first time:

maintain migrate create -c maintain.yaml
        

Schema UpgradesSchema UpgradesSchema Upgrades

Upgrade existing table schemas when updating IceGate:

maintain migrate upgrade -c maintain.yaml
        

Dry RunDry RunDry Run

Preview what would be done without executing:

maintain migrate create -c maintain.yaml --dry-run
        maintain migrate upgrade -c maintain.yaml --dry-run
        

Migration ProcessMigration ProcessMigration Process

1. Connect to Iceberg catalog
2. Check existing table schemas
3. Create missing tables (or alter existing ones)
4. Report migration status

Data Compaction (Shift)Data Compaction (Shift)Data Compaction (Shift)

The Ingest service automatically shifts WAL data into optimized Iceberg tables via the built-in shift process.

How Shift WorksHow Shift WorksHow Shift Works

1. Job manager monitors WAL segments
2. Groups segments into shift tasks
3. Reads WAL Parquet files in parallel
4. Merges and re-partitions data
5. Writes optimized Iceberg data files
6. Commits a new snapshot to the catalog, recording the last committed WAL offset in the snapshot summary

Shift does not delete WAL segments. An object lifecycle rule on the queue bucket reclaims them, and the offset in the snapshot summary is what lets shift resume where it left off.

Tuning Shift PerformanceTuning Shift PerformanceTuning Shift Performance

Key configuration parameters in the Ingest service config:

shift:
          read:
            max_record_batches_per_task: 1024
            max_input_bytes_per_task: 67108864  # 64 MiB
            plan_segment_read_parallelism: 8
            shift_segment_read_parallelism: 8
          write:
            row_group_size: 8192
            max_file_size_mb: 64
            table_cache_ttl_secs: 60
          jobsmanager:
            worker_count: 4           # Half of available CPUs by default
            poll_interval_ms: 1000
            iteration_interval_millisecs: 30000
        

See Configuration for full parameter reference.

Table OptimizationTable OptimizationTable Optimization

Optimize File SizesOptimize File SizesOptimize File Sizes

Rewrite small files into larger, optimized files:

ALTER TABLE icegate.logs EXECUTE optimize;
        

Expire SnapshotsExpire SnapshotsExpire Snapshots

Remove old snapshots to reclaim storage:

ALTER TABLE icegate.logs
        EXECUTE expire_snapshots(retention_threshold => '7d');
        

Remove Orphan FilesRemove Orphan FilesRemove Orphan Files

Delete unreferenced data files:

ALTER TABLE icegate.logs
        EXECUTE remove_orphan_files(retention_threshold => '1d');
        

Data RetentionData RetentionData Retention

Manual DeletionManual DeletionManual Deletion

Delete data older than a specific date:

DELETE FROM icegate.logs
        WHERE timestamp < TIMESTAMP '2024-01-01 00:00:00 UTC';
        

MonitoringMonitoringMonitoring

Key MetricsKey MetricsKey Metrics

Monitor these metrics for maintenance health (available at http://ingest:9091/metrics):

Health ChecksHealth ChecksHealth Checks

# Check query service readiness
        curl http://localhost:3100/ready
        
        # Check ingest service health
        curl http://localhost:4318/health
        

Backup and RecoveryBackup and RecoveryBackup and Recovery

Catalog BackupCatalog BackupCatalog Backup

On the default S3 catalog the metadata is root.json plus the table metadata files in the warehouse bucket, so a backup is a copy of that prefix - there is no service to stop:

aws s3 sync s3://warehouse/catalog/ ./catalog-backup/
        

sync is not an atomic snapshot: it lists, then copies, and commits landing in between can leave the copy mixing catalog generations. For a point-in-time copy, use bucket versioning (below) and read a single version, or take the copy while writes are quiesced. Verify any backup by restoring it to a scratch prefix and listing the tables before relying on it.

If you run the REST catalog backend instead, back up Nessie's RocksDB data:

# Stop Nessie
        docker stop nessie
        
        # Backup data directory
        tar -czf nessie-backup.tar.gz /data/nessie
        
        # Restart Nessie
        docker start nessie
        

Data RecoveryData RecoveryData Recovery

Iceberg supports time-travel queries. To recover from accidental deletion:

-- List available snapshots
        SELECT * FROM icegate.logs$snapshots;
        
        -- Query data at specific snapshot
        SELECT * FROM icegate.logs FOR VERSION AS OF 123456789;
        
        -- Rollback to previous snapshot
        CALL icegate.system.rollback_to_snapshot('logs', 123456789);
        

Object Storage BackupObject Storage BackupObject Storage Backup

Enable versioning on your S3 bucket for point-in-time recovery:

aws s3api put-bucket-versioning \
          --bucket warehouse \
          --versioning-configuration Status=Enabled
        

Performance TuningPerformance TuningPerformance Tuning

Query PerformanceQuery PerformanceQuery Performance

• Ensure partitions are properly pruned (filter on tenant_id, timestamp)
• Monitor query plan with /loki/api/v1/explain
• Increase query service memory for complex aggregations
• Enable catalog cache for production query services

Write PerformanceWrite PerformanceWrite Performance

• Scale Ingest service replicas for higher throughput
• Tune queue.write.flush_interval_ms and queue.write.max_bytes_per_flush
• Choose appropriate compression codec (ZSTD for best ratio, Snappy for speed)
• Monitor WAL write latency

Compaction PerformanceCompaction PerformanceCompaction Performance

• Increase shift.read.plan_segment_read_parallelism for faster reads
• Increase shift.jobsmanager.worker_count for more concurrent tasks
• Adjust shift.jobsmanager.iteration_interval_millisecs for more frequent shifts

Next StepsNext StepsNext Steps

• Set up Troubleshooting procedures
• Review Deployment configuration
• Understand the Data Model