Observability · Monitoring · Docker · August 2025

Splunk Monitoring Platform

A self-hosted Splunk Enterprise deployment with log ingestion pipelines, custom dashboards, and alerting — providing full operational visibility across containerized services running on the personal deployment server.

Splunk Docker Log Pipelines Dashboards Alerting
GitHub
Splunk Monitoring Platform

Overview

Observability is the difference between knowing your system is broken and knowing why. This project deploys Splunk Enterprise on the personal server to provide structured log aggregation, search, dashboards, and alerting across all running services.

Rather than relying on tail -f and scattered log files, Splunk centralizes log streams from every container and service, making it possible to correlate events, trace errors across service boundaries, and get alerted before issues escalate.

The platform runs in Docker, configured to ingest logs from the deployment server's containers and system services. Custom dashboards visualize API error rates, system resource usage, and request patterns in real time.

Architecture

📥
Log Ingestion
  • Splunk Universal Forwarder on host
  • Docker log driver forwarding
  • File monitor inputs for system logs
  • Syslog input for network events
🔍
Search & Indexing
  • SPL (Search Processing Language)
  • Index-time field extraction
  • Saved searches for common queries
  • Full-text search across all events
📊
Dashboards
  • API error rate over time
  • Request volume and latency
  • System resource utilization
  • Container health summary
🔔
Alerting
  • Scheduled alert checks
  • Threshold-based triggers
  • Error spike detection
  • Email / webhook notifications

Features

🔄
Centralized log aggregation
All container logs, system events, and NGINX access logs stream into a single searchable index.
🔍
SPL search
Powerful query language for filtering, transforming, and visualizing log data on demand.
📈
Real-time dashboards
Custom panels for API health, request patterns, and error rates — visible at a glance.
🚨
Proactive alerting
Alerts fire before a 3am incident. Error spikes and threshold breaches trigger notifications automatically.
📦
Containerized deployment
Splunk runs in Docker, isolated from host services, with persistent volume for index storage.

Setup Workflow

Deploy Splunk container
Start Splunk Enterprise in Docker with a persistent volume for index data and a defined admin password.
Configure inputs
Set up monitor inputs for log files and Docker log driver forwarding. Forwarder installed on host for system logs.
Define indexes
Create separate indexes for app logs, system logs, and NGINX access logs. Retention policies configured per index.
Build dashboards
Write SPL queries for API health and system metrics. Panel visualizations composed into dashboards in the Splunk UI.
Configure alerts
Scheduled alerts run SPL queries on intervals. Notifications sent when result counts exceed defined thresholds.

Tech Stack

🔍
Splunk EnterpriseLog platform
📦
DockerContainer runtime
📡
Universal ForwarderLog shipping
🔎
SPLSearch language
🐧
Ubuntu ServerHost OS
🌐
NGINXLog source