Use Developer Tools Platform as the Foundational Architecture Pattern
Deterministic ADR derived from topology, simulation, and advisor intelligence for Developer Tools Platform. Traceable to YAML knowledge entities.
Context
Developer tooling platforms have a particularly dangerous multi-tenant failure mode: a misconfiguration that exposes one tenant's source code, pipeline secrets, or artifact store to another tenant is a critical security breach, not just a data quality issue. Unlike SaaS billing or user management, the cross-tenant leakage of build artifacts or environment variables has direct exploitation potential. At the same time, the workload is bursty and heterogeneous: a single tenant triggering a monorepo CI build against 50 microservices generates more job queue volume in 60 seconds than a small tenant generates in a week. Shared infrastructure must enforce resource quotas hard enough to protect other tenants while remaining operationally simple enough for a small platform team to run. Webhook delivery adds an outbound reliability dimension: the platform must deliver pipeline completion events to tenant-registered HTTP endpoints without accumulating unbounded retry queues when tenant endpoints are down. Primary operational risks include: Cross-tenant data leakage via missing RLS policy: any PostgreSQL table storing tenant-scoped data (pipeline runs, secrets, artifacts, API keys) without a corresponding RLS policy exposes all rows to any authenticated application role; new tables added during feature development are the highest-risk surface because policy coverage is not automatically inherited; automated cross-tenant query tests must run on every migration before deployment; Tenant job burst saturating shared Redis queue: a tenant running a monorepo build that enqueues 500 parallel jobs in under one second consumes the Redis job queue and all available worker capacity; other tenants' jobs enter the queue after the burst and wait for the burst to clear, violating their expected p99 start time; without per-tenant queue partitioning or token-bucket job submission rate limits, queue depth during burst is invisible as a tenant-specific metric; Elasticsearch index growth without lifecycle management: pipeline log output indexed into Elasticsearch grows unbounded without an index lifecycle policy (ILM); at moderate log volume (10GB/day), a missing ILM policy can exhaust disk within weeks; large indices also cause Elasticsearch JVM heap pressure during merge operations, degrading search latency for all tenants.
Decision
We will adopt the **Developer Tools Platform** architecture pattern. This is a high-complexity architecture appropriate for teams at experienced backend team level or above. The advisor rates this pattern as 'advanced' operational maturity.
Rationale
A multi-tenant developer tooling platform providing CI/CD pipeline execution, log aggregation, code analysis, and dependency scanning across isolated tenant organizations. Tenant isolation is the primary correctness constraint: a security boundary violation between tenants is a critical incident, not a performance event. PostgreSQL row-level security enforces data isolation; Redis manages job queues and distributed locks; Elasticsearch indexes pipeline log output for search; Kafka delivers webhook events to tenant-registered endpoints; MinIO stores pipeline artifacts. Resource quota enforcement prevents any single tenant's burst from affecting others. Core technology stack: postgresql, redis, elasticsearch, kafka, minio.
Accepted Tradeoffs
- ⚠Shared PostgreSQL with RLS is operationally simpler than database-per-tenant but requires every developer to understand and apply RLS policies to all new tables; the cognitive overhead is low per feature but the consequence of a single omission is a critical security incident
- ⚠Redis as a job queue is simpler to deploy than Kafka for small-to-medium job volumes, but Redis provides no persistent replayability: a Redis restart or failover loses queued jobs unless persistence (AOF with fsync = always) is configured explicitly; for CI job dispatch, this is an operational correctness concern that must be addressed in the Redis configuration, not assumed
- ⚠Elasticsearch provides powerful full-text log search but is the highest operational complexity component in this stack: JVM heap sizing, shard count tuning, and ILM policy management require dedicated operational attention; a misconfigured shard count on high-volume log indices causes unrecoverable search performance degradation that requires index reindexing to resolve
- ⚠Tenant-level resource quotas enforced at the application layer (not by Redis or PostgreSQL natively) mean that quota logic must be maintained in code and tested explicitly; a quota enforcement bug that fails open allows a tenant to consume unlimited resources before the next deployment cycle
Risks
In a multi-tenant system, one tenant's high resource consumption: query load, connection count, write rate, or storage I/O: degrades database or service performance for all other tenants sharing the same infrastructure, violating the implicit isolation guarantee that a shared-infrastructure SaaS product implies.
Message queue or event stream consumer processing rate falls below producer write rate, causing consumer lag to grow unboundedly: eventually leading to increased end-to-end latency, producer backpressure, data expiry, or queue resource exhaustion.
An ALTER TABLE or other DDL statement takes an ACCESS EXCLUSIVE lock that conflicts with every other lock type, including a plain SELECT's ACCESS SHARE. Once that lock request is waiting, every later query on the table queues behind it too, so a DDL statement that is merely waiting, not yet running, is enough to take the table's entire traffic down.
A single consumer instance in a Kafka consumer group processes messages significantly slower than its peers, causing partition lag to accumulate on its assigned partitions and triggering consumer group rebalances that temporarily suspend all partition consumption.
Production configuration diverges from the intended state through manual changes, partial rollouts, and environment-specific overrides, causing failures that are intermittent, hard to reproduce, and require cross-node comparison to diagnose.
Alternatives Considered
AI Retrieval-Augmented Generation Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; Developer Tools Platform is a better fit for the identified workload profile.
Analytics Data Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; Developer Tools Platform is a better fit for the identified workload profile.
API Gateway Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; Developer Tools Platform is a better fit for the identified workload profile.
Audit and Compliance Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; Developer Tools Platform is a better fit for the identified workload profile.
Scaling Thresholds
Signals indicating the architecture is approaching its scaling limits:
Tier 1: Job Queue Tenant Noisy Neighbor
Signal: Redis job queue depth > 1000 correlated with a single tenant identifier; other tenants reporting p99 job start time > 60 seconds; tenant-level queue metrics showing one tenant holding > 50% of in-flight worker slots
Evolution: Implement per-tenant queue lanes in Redis (separate key namespaces per tenant, e.g., jobs:{tenant_id}:{priority}); implement a weighted fair scheduler at the worker dispatch layer that reads from tenant queues in round-robin order with priority weighting; cap the number of concurrently executing jobs per tenant to the tenant's quota, not to the total available worker count
Tier 2: PostgreSQL Schema Migration Lock
Signal: DDL migration duration > 10s on pipeline_runs, jobs, or artifacts tables; migration deployment causing timeout errors for active CI pipeline API calls during the deployment window; pg_locks showing AccessExclusiveLock held by ALTER TABLE statement
Evolution: Adopt zero-downtime migration patterns exclusively: add columns with nullable defaults first (no table lock in PostgreSQL 11+), then backfill, then add constraints via NOT VALID followed by VALIDATE CONSTRAINT in a separate transaction; use pg_repack for table rewrites; never run concurrent index creation without CONCURRENTLY on any table with > 1M rows
Tier 3: Elasticsearch Log Index Saturation
Signal: Elasticsearch heap usage > 75%; log index size > 500GB on any single index; search latency p99 > 2s for log queries; ILM policy showing rollover lag
Evolution: Implement ILM with rollover at 50GB or 7 days (whichever comes first); use data tiers (hot/warm/cold) to move older indices to cheaper storage automatically; set shard count to 1 per rollover index if log volume is < 10GB/day per index, to avoid over-sharding small indices; enable force-merge to 1 segment on read-only cold indices to reduce memory overhead
Tier 4: Tenant Data Volume Segregation
Signal: Top 3 tenants generating > 60% of total pipeline volume; large tenants requesting SLA guarantees incompatible with shared infrastructure; compliance requirement for data residency or dedicated compute for enterprise contracts
Evolution: Implement a hybrid isolation model: dedicated worker pools and dedicated PostgreSQL schemas (or databases) for enterprise tenants; route enterprise tenant traffic through a dedicated API gateway instance with dedicated Redis and Elasticsearch namespaces; retain shared infrastructure for standard tier tenants
Migration Path
Monolithic job queue in Redis with shared worker pool → Per-tenant queue lanes with weighted fair scheduling
First noisy neighbor incident where one tenant's CI burst delays other tenants' builds by > 5 minutes; customer complaints about unpredictable build start times; inability to enforce per-tenant quota from a shared queue
Inline Kafka webhook publish on pipeline completion (dual-write) → Outbox pattern with bounded retry and dead-letter queue
Kafka publish failures rolling back pipeline completion transactions, causing build results to be lost; or webhook retry queues growing unboundedly for tenants with temporarily down endpoints
Shared Elasticsearch index for all tenant log output → Per-tenant Elasticsearch index with ILM and data tier management
Log search returning results from other tenants' pipelines due to missing tenant_id filter on queries (correctness incident); or a single large tenant's log volume causing index size imbalance that degrades search for all tenants
Operational Requirements
- Minimum team maturity: Experienced Backend Team: This scenario has high operational complexity. It is recommended for Experienced Backend Team teams or higher.
- Runbooks and alerting for high-severity risks: 3 high-severity risks identified. Each requires a documented runbook, alerting threshold, and on-call response procedure before running in production.
- Event stream operations expertise: This architecture includes event stream infrastructure (Kafka, Kinesis, or similar). Operations requires consumer group management, partition assignment, dead-letter handling, and lag monitoring.
- Cache sizing and eviction policy configuration: Redis or equivalent cache requires correct maxmemory configuration, eviction policy selection (allkeys-lru is common), and cold-start warming strategy after restarts.