DevOps Applications in Telecom: Real-World Examples

# Lab use only, against test devices
# ansible-galaxy collection install cisco.ios
- name: Update NTP servers in small batches
  hosts: lab_routers
  gather_facts: false
  serial: 2                     # change two devices at a time
  tasks:
    - name: Back up current configuration
      cisco.ios.ios_config:
        backup: true

    - name: Apply NTP configuration
      cisco.ios.ios_config:
        lines:
          - ntp server 10.0.0.10
          - ntp server 10.0.0.11

    - name: Verify NTP associations
      cisco.ios.ios_command:
        commands: show ntp associations
      register: ntp_out

import ipaddress
import yaml   # pip install pyyaml

config = yaml.safe_load("""
ntp_servers: [10.0.0.10, 10.0.0.11]
syslog_server: 10.0.0.20
change_window: "02:00-04:00"
""")

errors = []
for ip in config["ntp_servers"] + [config["syslog_server"]]:
    try:
        ipaddress.ip_address(ip)
    except ValueError:
        errors.append(f"invalid IP: {ip}")
if len(config["ntp_servers"]) < 2:
    errors.append("need at least two NTP servers")
print("pre-check failed:" if errors else "pre-check passed", errors)

# tests/test_recharge.py
PLANS = {"P199": 199, "P299": 299}

def recharge(balance: int, plan: str, paid: int) -> int:
    if plan not in PLANS:
        raise ValueError("unknown plan")
    if paid != PLANS[plan]:
        raise ValueError("amount mismatch")
    return balance + 1          # plan activated

def test_valid_recharge():
    assert recharge(0, "P199", 199) == 1

def test_amount_mismatch_rejected():
    import pytest
    with pytest.raises(ValueError):
        recharge(0, "P199", 99)

#!/usr/bin/env bash
# Compare canary error rate with stable before promoting
set -euo pipefail
CANARY_ERR=${CANARY_ERR:-0.4}   # % from monitoring query
STABLE_ERR=${STABLE_ERR:-0.3}
LIMIT=1.0                       # allowed extra error %

DIFF=$(echo "$CANARY_ERR - $STABLE_ERR" | bc -l)
if (( $(echo "$DIFF > $LIMIT" | bc -l) )); then
  echo "Canary worse by $DIFF%. Rolling back."
  kubectl rollout undo deployment/selfcare-api
  exit 1
fi
echo "Canary healthy (diff $DIFF%). Promote."

import sqlite3

db = sqlite3.connect(":memory:")
db.executescript("""
CREATE TABLE usage_charges (account TEXT, amount REAL);
CREATE TABLE bills (account TEXT, total REAL);
INSERT INTO usage_charges VALUES ('A1', 120), ('A1', 80), ('A2', 50);
INSERT INTO bills VALUES ('A1', 200), ('A2', 55);
""")
rows = db.execute("""
  SELECT b.account, b.total, SUM(u.amount) AS expected
  FROM bills b JOIN usage_charges u ON u.account = b.account
  GROUP BY b.account, b.total
  HAVING ABS(b.total - SUM(u.amount)) > 0.01
""").fetchall()
print("mismatches:", rows or "none")   # A2 mismatch -> stop before sending bills

groups:
  - name: selfcare
    rules:
      - alert: RechargeSuccessLow
        expr: |
          sum(rate(recharge_total{status="success"}[10m]))
          / sum(rate(recharge_total[10m])) < 0.95
        for: 10m
        labels: { severity: page }
        annotations:
          summary: "Recharge success below 95% for 10 minutes"
          runbook: "https://wiki.example.com/runbooks/recharge"

Project: Telecom self-care platform (dummy data)

- recharge-api with tests (plan validation, amount checks)
- CI pipeline building a container image
- Kubernetes deployment with canary release script
- Billing CronJob with a reconciliation check
- Prometheus metrics and a recharge success alert
- Ansible playbook pushing config to lab network VMs

Change: NTP update on lab routers
Window: 02:00-04:00 (agreed)
Pre-checks: config lint, IP validation, backup taken
Batches: 2 devices, verify, then next batch
Rollback: restore backup config per device
Comms: notify NOC before and after
Sign-off: change owner + reviewer

Runbook: recharge success low

1. Check dashboard: which payment methods are failing?
2. Check recent releases; roll back if correlated
3. Check payment partner status page
4. Check recharge-api logs for new error types
5. Inform customer support with a short status message
6. Write an incident note with timeline and fix