Ferhat RudvanoğullarıData Engineer
← All work

Deployment & DevOps

Observability & Load Testing

Put logs, metrics and request traces in one dashboard, then hit the system with spike tests to find out where it breaks.

DevOps

Jul 2026

In numbers

concurrent users in 5 seconds (spike test)
0 → 300
concurrent users in 5 seconds (spike test)
parts: logs, metrics, traces, dashboards
4
parts: logs, metrics, traces, dashboards
simultaneous requests in a password-reset race test
50
simultaneous requests in a password-reset race test

The problem

When the app slowed down or threw errors, there was no way to see why. Logs were scattered, and nobody knew how much load the system could actually take.

What I did

  1. 1

    One place to look

    I set up a Grafana-based stack: Loki collects logs, Prometheus collects metrics (CPU, memory, request counts) and Tempo records the path each request takes through the system. Grafana shows all three on one dashboard, with access locked down.

  2. 2

    Spike test

    With k6, I ramped from 0 to 300 concurrent users in 5 seconds and held the peak for 10, watching response times and database behaviour on the dashboards.

  3. 3

    Race condition test

    While chasing a loop bug in the password-reset flow, I wrote a test that fires 50 simultaneous requests at the same reset link. Exactly one should succeed; the rest should be told the link has expired.

Result

Slowdowns and errors can now be traced from a single dashboard, and the system's limits are measured rather than guessed.

What I learned

Load tests don't just tell you whether the system holds up. They catch security bugs too, like a reset link that works twice.

Stack

  • Grafana
  • Loki
  • Prometheus
  • Tempo
  • k6
  • Docker