Automating Toil: Cutting Repetitive Engineering Work
Apply SRE toil-reduction principles to daily work: spot repetitive manual tasks, measure their real cost, automate them, and count that as output.
What Qualifies as Toil
Toil is not all manual work. It is manual work that is repetitive, automatable, tactical, has no enduring value, and scales linearly with service growth. Deploying a service by clicking through a UI is toil. Designing the deployment architecture is not. The distinction matters because toil masquerades as productive work.
Measuring Toil Before Automating
Before writing automation, quantify the cost. Many teams automate the wrong things—tasks that feel annoying but consume little actual time—while ignoring high-frequency, invisible time sinks.
interface ToilEntry {
task: string;
category: "deployment" | "incident" | "provisioning" | "data" | "communication";
frequencyPerWeek: number;
minutesPerOccurrence: number;
peopleInvolved: number;
errorRate: number; // Percentage of times the manual process fails
canBeAutomated: boolean;
}
function calculateToilCost(entries: ToilEntry[]): {
weeklyHours: number;
monthlyHours: number;
yearlyHours: number;
topCandidates: ToilEntry[];
} {
const withCost = entries.map((entry) => ({
...entry,
weeklyMinutes:
entry.frequencyPerWeek *
entry.minutesPerOccurrence *
entry.peopleInvolved,
}));
const totalWeeklyMinutes = withCost.reduce(
(sum, e) => sum + e.weeklyMinutes,
0
);
// Rank by weekly time cost, filtered to automatable tasks
const topCandidates = withCost
.filter((e) => e.canBeAutomated)
.sort((a, b) => b.weeklyMinutes - a.weeklyMinutes)
.slice(0, 5)
.map(({ weeklyMinutes, ...entry }) => entry);
return {
weeklyHours: totalWeeklyMinutes / 60,
monthlyHours: (totalWeeklyMinutes / 60) * 4.33,
yearlyHours: (totalWeeklyMinutes / 60) * 52,
topCandidates,
};
}// ❌ Automating based on gut feeling
// "I hate updating Jira tickets, let me automate that"
// (Takes 2 minutes/week — automation saves almost nothing)
// ✅ Automating based on data
const teamToil: ToilEntry[] = [
{
task: "Manually creating staging environments for PR review",
category: "provisioning",
frequencyPerWeek: 15,
minutesPerOccurrence: 20,
peopleInvolved: 1,
errorRate: 0.1,
canBeAutomated: true,
// 300 min/week = 5 hours/week = 260 hours/year
},
{
task: "Rotating database credentials quarterly",
category: "provisioning",
frequencyPerWeek: 0.08,
minutesPerOccurrence: 120,
peopleInvolved: 2,
errorRate: 0.25,
canBeAutomated: true,
// Low frequency but high error rate — automate for reliability
},
{
task: "Copying production data to staging (sanitized)",
category: "data",
frequencyPerWeek: 3,
minutesPerOccurrence: 45,
peopleInvolved: 1,
errorRate: 0.15,
canBeAutomated: true,
// 135 min/week + 15% failure rate = high value target
},
];The Automation Decision Framework
Not every toil task should be automated immediately. Balance the cost of automation against the ongoing cost of toil.
interface AutomationDecision {
task: string;
toilCostPerYear: number; // Hours spent on toil yearly
automationEstimate: number; // Hours to build automation
maintenanceCostPerYear: number; // Hours to maintain automation yearly
errorReduction: number; // Percentage reduction in errors
breakEvenMonths: number;
}
function evaluateAutomation(
entry: ToilEntry,
buildHours: number,
maintenanceHoursPerYear: number
): AutomationDecision {
const yearlyToilHours =
((entry.frequencyPerWeek *
entry.minutesPerOccurrence *
entry.peopleInvolved) /
60) *
52;
const yearlyNetSavings = yearlyToilHours - maintenanceHoursPerYear;
const breakEvenMonths =
yearlyNetSavings > 0
? (buildHours / yearlyNetSavings) * 12
: Infinity;
return {
task: entry.task,
toilCostPerYear: yearlyToilHours,
automationEstimate: buildHours,
maintenanceCostPerYear: maintenanceHoursPerYear,
errorReduction: entry.errorRate * 0.95, // Automation eliminates ~95% of errors
breakEvenMonths: Math.round(breakEvenMonths),
};
}Practical Automation Patterns
Start with the highest-value, lowest-effort automations. These patterns eliminate the most common engineering toil.
// Pattern 1: PR-triggered ephemeral environments
// Before: Manually create staging env for each PR (20 min × 15/week)
// After: GitHub Action creates environment automatically
// Pattern 2: Automated data sanitization pipeline
async function sanitizeProductionData(
sourceDb: string,
targetDb: string
): Promise<void> {
const tables = await getTableList(sourceDb);
for (const table of tables) {
const sanitizers = getSanitizers(table);
await copyTableWithSanitization(sourceDb, targetDb, table, sanitizers);
}
}
type Sanitizer = (value: unknown, column: string) => unknown;
function getSanitizers(table: string): Map<string, Sanitizer> {
const rules: Map<string, Sanitizer> = new Map();
// PII columns get deterministic fakes
if (table === "users") {
rules.set("email", (_, col) => `user-${hashForDev(_)}@example.com`);
rules.set("phone", () => "+1-555-000-0000");
rules.set("name", (val) => `User ${hashForDev(val).slice(0, 6)}`);
}
return rules;
}
// Pattern 3: Credential rotation script
async function rotateCredentials(
service: string
): Promise<{ previousKey: string; newKey: string }> {
// Generate new credentials
const newKey = crypto.randomBytes(32).toString("base64");
// Update in secrets manager
await secretsManager.putSecretValue(service, newKey);
// Update service configuration (no restart needed with hot reload)
await updateServiceConfig(service, { apiKey: newKey });
// Verify new credentials work
const healthy = await healthCheck(service);
if (!healthy) {
// Rollback
await secretsManager.putSecretValue(service, previousKey);
throw new Error(`Health check failed after credential rotation for ${service}`);
}
return { previousKey, newKey };
}Building a Toil Budget
Set a team-level target: no more than a certain percentage of engineering time should be spent on toil. Track it like you track error budgets.
interface ToilBudget {
teamSize: number;
maxToilPercentage: number;
currentToilHoursPerWeek: number;
budgetHoursPerWeek: number;
remainingBudget: number;
overBudget: boolean;
}
function calculateToilBudget(
teamSize: number,
maxToilPercentage: number,
currentToilHoursPerWeek: number
): ToilBudget {
const totalTeamHours = teamSize * 40;
const budgetHours = totalTeamHours * (maxToilPercentage / 100);
return {
teamSize,
maxToilPercentage,
currentToilHoursPerWeek: currentToilHoursPerWeek,
budgetHoursPerWeek: budgetHours,
remainingBudget: budgetHours - currentToilHoursPerWeek,
overBudget: currentToilHoursPerWeek > budgetHours,
};
}
// Example: team of 8, max 20% toil
// Budget: 8 × 40 × 0.2 = 64 hours/week
// Current: 45 hours/week — within budget
// If new toil pushes past 64, automation becomes mandatoryMeasuring Automation Success
Track the impact of automation over time. Did it actually reduce toil, or did it create new maintenance toil?
interface AutomationMetrics {
taskName: string;
beforeAutomation: {
manualOccurrencesPerWeek: number;
minutesPerOccurrence: number;
errorRate: number;
};
afterAutomation: {
manualInterventionsPerWeek: number;
minutesPerIntervention: number;
errorRate: number;
maintenanceHoursPerMonth: number;
};
netSavingsPerWeek: number;
reliabilityImprovement: number;
}
function calculateAutomationROI(
metrics: AutomationMetrics,
buildHours: number
): { monthsToROI: number; yearlyHoursSaved: number } {
const beforeWeekly =
metrics.beforeAutomation.manualOccurrencesPerWeek *
(metrics.beforeAutomation.minutesPerOccurrence / 60);
const afterWeekly =
metrics.afterAutomation.manualInterventionsPerWeek *
(metrics.afterAutomation.minutesPerIntervention / 60) +
metrics.afterAutomation.maintenanceHoursPerMonth / 4.33;
const weeklySavings = beforeWeekly - afterWeekly;
const monthsToROI = buildHours / (weeklySavings * 4.33);
return {
monthsToROI: Math.round(monthsToROI * 10) / 10,
yearlyHoursSaved: Math.round(weeklySavings * 52),
};
}Key Takeaways
Toil is repetitive, automatable work that scales linearly and provides no enduring value. Measure it before automating—quantify hours per week, error rates, and number of people involved. Prioritize by data: the task consuming 5 hours per week with a 15% error rate is a better automation candidate than the task that merely feels annoying.
Use the break-even calculation to decide what to automate now versus later. Set a toil budget for your team—when toil exceeds the budget, automation becomes mandatory, not optional. Track automation metrics after deployment to verify it reduced toil instead of shifting it. The goal is not zero toil—it is keeping toil at a level where engineers spend most of their time on creative, enduring work rather than repetitive operational tasks.
