The Developer Journal Practice

# ❌ No journal — relying on memory
 
"Why did we choose Postgres over DynamoDB for the orders service?"
→ "I think there was a reason... something about transactions?"
→ The context is lost. You repeat the same analysis.
 
# ✅ With a journal entry from 6 months ago
 
## 2022-01-15 — Database Selection: Orders Service
Decision: PostgreSQL over DynamoDB
Reasons:
- Orders require multi-table transactions (payment + inventory + order)
- DynamoDB single-table design adds complexity for relational queries
- Team has strong SQL expertise, no DynamoDB experience
- Cost analysis: ~$450/mo PG vs ~$380/mo DDB (not significant enough to matter)
Trade-off accepted: Managing connection pools and read replicas manually

# Daily Engineering Journal
 
## 2022-05-11 (Wednesday)
 
### What I Worked On
- Investigated timeout errors in the payment webhook handler
  - Root cause: database connection pool exhausted under high load
  - Pool was sized at 10, peak concurrent webhooks hit 35
  - Temporary fix: increased pool to 50
  - Proper fix: add queue between webhook endpoint and processing
 
- Reviewed RFC for the notification service migration
  - Main concern: the proposed schema doesn't account for
    notification preferences per channel (email vs push vs SMS)
  - Left comment suggesting a preferences table
 
### What I Learned
- `pg_stat_activity` shows active connections per database — useful
  for verifying pool sizing in production
- Stripe webhook retries use exponential backoff (1hr, 2hr, 4hr)
  so delayed processing is acceptable for non-idempotent operations
 
### Blockers / Open Questions
- Need access to the staging Datadog dashboard to verify pool metrics
- Should we rate-limit incoming webhooks at the API gateway level?

// decision-log.ts — A structured format for technical decisions
 
interface DecisionEntry {
  date: string;
  title: string;
  context: string;
  options: {
    name: string;
    pros: string[];
    cons: string[];
  }[];
  decision: string;
  reasoning: string;
  consequences: string[];
  revisitDate?: string;
}
 
const cacheDecision: DecisionEntry = {
  date: '2022-05-10',
  title: 'Cache Layer for Product Catalog',
  context:
    'Product catalog reads are 95% of DB load. Average query takes 120ms. '
    + 'Target is <20ms for catalog reads.',
  options: [
    {
      name: 'Redis cache-aside',
      pros: [
        'Simple implementation',
        'Team familiar with Redis',
        'Fine-grained TTL control',
      ],
      cons: [
        'Cache invalidation complexity',
        'Additional infrastructure',
        'Potential stale reads',
      ],
    },
    {
      name: 'CDN edge caching',
      pros: [
        'No infrastructure to manage',
        'Global distribution',
        'Handles traffic spikes automatically',
      ],
      cons: [
        'Coarse invalidation (purge by path)',
        'Cache varies by header complexity',
        'Debugging cache misses is harder',
      ],
    },
  ],
  decision: 'Redis cache-aside with 5-minute TTL',
  reasoning:
    'Product catalog updates happen ~10 times/day via admin panel. '
    + '5-minute staleness is acceptable. Team can implement and debug '
    + 'Redis quickly. CDN caching adds complexity with auth headers.',
  consequences: [
    'Need to add cache invalidation on product update endpoints',
    'Redis instance sized for ~50k product entries (~2GB)',
    'Monitoring: track cache hit rate, target >95%',
  ],
  revisitDate: '2022-08-10',
};

## 2022-05-09 — Debugging: Intermittent 502 Errors on /api/checkout
 
### Symptoms
- ~2% of checkout requests return 502 between 2-4 PM UTC
- No errors in application logs for those requests
- Load balancer health checks pass consistently
 
### Investigation Timeline
1. Checked application logs → no 5xx from app server
2. Checked nginx access logs → 502 responses present
   Upstream response time: 0.000ms (connection refused)
3. Hypothesis: app server dropping connections under load
4. Checked `netstat` → TIME_WAIT connections at 28,000
   (default max is 28,232)
5. **Root cause: ephemeral port exhaustion**
   - High rate of short-lived connections to Redis
   - Each connection creates a TIME_WAIT entry for 60 seconds
   - 2-4 PM is peak traffic → ports exhausted
 
### Fix Applied
- Switched from per-request Redis connections to connection pooling
  using `ioredis` cluster client with `natMap`
- Set `net.ipv4.tcp_tw_reuse = 1` on app servers
- Added monitoring alert for TIME_WAIT count > 20,000
 
### Lessons
- 502 with 0.000ms upstream time always means connection refused
- Check TIME_WAIT before assuming application-level issues
- Connection pooling is non-negotiable for high-throughput services

# ❌ After fixing a complex bug without a journal
# 8 months later, similar symptoms appear
"I've seen this before... was it DNS? Connection pool?
 I can't remember what I tried."
# Spend another 4 hours re-investigating
 
# ✅ After fixing a complex bug with a journal entry
# 8 months later, similar symptoms appear
grep "502" journal/*.md
# → Immediately find the debugging log
# → Check TIME_WAIT count first
# → Resolved in 20 minutes

## Weekly Review — 2022-05-06 to 2022-05-11
 
### Time Distribution (approximate)
- Debugging: 40% (connection pool issues, webhook timeouts)
- Feature work: 30% (notification preferences)
- Code review: 20% (3 PRs reviewed)
- Meetings: 10%
 
### Patterns Noticed
- Third time this quarter dealing with connection pool issues
  → Action: Create a "connection pool sizing" checklist for new services
  → Action: Propose default pool monitoring in service template
 
### Wins
- Identified port exhaustion issue before it became a full outage
- RFC feedback on notification schema prevented a migration later
 
### For Next Week
- Complete notification preferences implementation
- Write the pool sizing checklist
- Pair with Alex on the search indexing pipeline

# Simple directory structure
journal/
  2022/
    05/
      2022-05-09.md
      2022-05-10.md
      2022-05-11.md
    decisions/
      2022-05-10-cache-layer.md
    debugging/
      2022-05-09-502-errors.md
  templates/
    daily.md
    decision.md
    debugging.md
 
# Quick alias for creating today's entry
alias jrnl='code ~/journal/$(date +%Y/%m)/$(date +%Y-%m-%d).md'