Reinforcement Learning Basics for Software Developers
A practical introduction to reinforcement learning — agents, environments, rewards and policies — with TypeScript examples grounded in familiar problems.
Reinforcement learning (RL) is the branch of machine learning where an agent learns by interacting with an environment. Unlike supervised learning (where you provide labeled examples) or unsupervised learning (where you find patterns in unlabeled data), RL learns from the consequences of actions. The agent takes actions, receives rewards or penalties, and adjusts its strategy to maximize cumulative reward over time.
As developers, we encounter RL-like problems more often than we realize — A/B testing, autoscaling policies, cache eviction strategies, and load balancing all involve making sequential decisions under uncertainty.
The Core Loop: Agent, Environment, State, Action, Reward
Every reinforcement learning problem has the same fundamental structure. An agent observes the state of an environment, takes an action, receives a reward, and transitions to a new state.
// The RL loop expressed as TypeScript interfaces
interface State {
features: number[]; // Numeric representation of current situation
}
interface Action {
id: number;
label: string;
}
interface Environment {
getState(): State;
getAvailableActions(): Action[];
step(action: Action): { nextState: State; reward: number; done: boolean };
reset(): State;
}
interface Agent {
selectAction(state: State, availableActions: Action[]): Action;
learn(state: State, action: Action, reward: number, nextState: State): void;
}
// The training loop
function trainAgent(agent: Agent, env: Environment, episodes: number): void {
for (let episode = 0; episode < episodes; episode++) {
let state = env.reset();
let totalReward = 0;
let done = false;
while (!done) {
const actions = env.getAvailableActions();
const action = agent.selectAction(state, actions);
const { nextState, reward, done: isDone } = env.step(action);
agent.learn(state, action, reward, nextState);
state = nextState;
totalReward += reward;
done = isDone;
}
if (episode % 100 === 0) {
console.log(`Episode ${episode}: total reward = ${totalReward}`);
}
}
}Exploration vs. Exploitation
The central dilemma in RL is balancing exploration (trying new actions to discover better strategies) with exploitation (using what you already know works). This is exactly the same tradeoff in A/B testing: do you keep showing the best-performing variant, or do you test new ones?
// Epsilon-greedy strategy: explore with probability epsilon, exploit otherwise
class EpsilonGreedyAgent implements Agent {
private qTable: Map<string, Map<number, number>> = new Map();
private epsilon: number;
private learningRate: number;
private discountFactor: number;
constructor(epsilon = 0.1, learningRate = 0.1, discountFactor = 0.95) {
this.epsilon = epsilon;
this.learningRate = learningRate;
this.discountFactor = discountFactor;
}
selectAction(state: State, availableActions: Action[]): Action {
const stateKey = JSON.stringify(state.features);
// Explore: pick a random action
if (Math.random() < this.epsilon) {
const idx = Math.floor(Math.random() * availableActions.length);
return availableActions[idx];
}
// Exploit: pick the action with the highest Q-value
const qValues = this.qTable.get(stateKey);
if (!qValues) {
// Never seen this state — explore by default
const idx = Math.floor(Math.random() * availableActions.length);
return availableActions[idx];
}
let bestAction = availableActions[0];
let bestValue = -Infinity;
for (const action of availableActions) {
const value = qValues.get(action.id) ?? 0;
if (value > bestValue) {
bestValue = value;
bestAction = action;
}
}
return bestAction;
}
learn(state: State, action: Action, reward: number, nextState: State): void {
const stateKey = JSON.stringify(state.features);
const nextStateKey = JSON.stringify(nextState.features);
if (!this.qTable.has(stateKey)) {
this.qTable.set(stateKey, new Map());
}
const currentQ = this.qTable.get(stateKey)!.get(action.id) ?? 0;
// Find the max Q-value for the next state
const nextQValues = this.qTable.get(nextStateKey);
let maxNextQ = 0;
if (nextQValues) {
maxNextQ = Math.max(...nextQValues.values(), 0);
}
// Q-learning update rule
const newQ = currentQ + this.learningRate * (
reward + this.discountFactor * maxNextQ - currentQ
);
this.qTable.get(stateKey)!.set(action.id, newQ);
}
}// ❌ Always exploiting — gets stuck on locally optimal strategies
class GreedyOnlyAgent implements Agent {
selectAction(state: State, actions: Action[]): Action {
// Always picks the best-known action
// Never discovers that action #3 is actually better long-term
return this.getBestKnownAction(state, actions);
}
}
// ✅ Decaying epsilon — explore a lot early, exploit more as you learn
class DecayingEpsilonAgent implements Agent {
private epsilon: number;
private readonly minEpsilon = 0.01;
private readonly decayRate = 0.995;
selectAction(state: State, actions: Action[]): Action {
const result = Math.random() < this.epsilon
? this.randomAction(actions)
: this.getBestKnownAction(state, actions);
// Gradually reduce exploration over time
this.epsilon = Math.max(this.minEpsilon, this.epsilon * this.decayRate);
return result;
}
}A Concrete Example: Cache Eviction
Let's model something developers deal with daily — cache eviction. The agent decides which cache entries to evict when the cache is full. The reward is based on cache hit rate.
interface CacheState {
features: number[]; // [cacheSize, hitRate, avgAge, avgFrequency]
}
class CacheEnvironment implements Environment {
private cache: Map<string, { value: string; age: number; hits: number }>;
private readonly capacity: number;
private requests: string[];
private step_count: number;
constructor(capacity: number, requests: string[]) {
this.cache = new Map();
this.capacity = capacity;
this.requests = requests;
this.step_count = 0;
}
getState(): CacheState {
const entries = [...this.cache.values()];
const avgAge = entries.length > 0
? entries.reduce((s, e) => s + e.age, 0) / entries.length : 0;
const avgFreq = entries.length > 0
? entries.reduce((s, e) => s + e.hits, 0) / entries.length : 0;
return {
features: [
this.cache.size / this.capacity, // Fullness ratio
avgAge,
avgFreq,
this.step_count,
],
};
}
getAvailableActions(): Action[] {
return [
{ id: 0, label: 'evict-lru' }, // Least recently used
{ id: 1, label: 'evict-lfu' }, // Least frequently used
{ id: 2, label: 'evict-random' }, // Random eviction
{ id: 3, label: 'evict-oldest' }, // Oldest entry
];
}
step(action: Action): { nextState: CacheState; reward: number; done: boolean } {
// Simulate processing the next request
const request = this.requests[this.step_count];
let reward = 0;
if (this.cache.has(request)) {
reward = 1; // Cache hit — positive reward
this.cache.get(request)!.hits++;
} else {
reward = -0.5; // Cache miss — negative reward
if (this.cache.size >= this.capacity) {
this.evict(action); // Agent chooses eviction strategy
}
this.cache.set(request, { value: request, age: 0, hits: 1 });
}
// Age all entries
for (const entry of this.cache.values()) {
entry.age++;
}
this.step_count++;
const done = this.step_count >= this.requests.length;
return { nextState: this.getState(), reward, done };
}
reset(): CacheState {
this.cache.clear();
this.step_count = 0;
return this.getState();
}
private evict(action: Action): void {
// Each action corresponds to a different eviction strategy
// The agent learns which strategy works best for current workload
const entries = [...this.cache.entries()];
let evictKey: string;
switch (action.id) {
case 0: evictKey = entries.sort((a, b) => b[1].age - a[1].age)[0][0]; break;
case 1: evictKey = entries.sort((a, b) => a[1].hits - b[1].hits)[0][0]; break;
case 2: evictKey = entries[Math.floor(Math.random() * entries.length)][0]; break;
case 3: evictKey = entries.sort((a, b) => b[1].age - a[1].age)[0][0]; break;
default: evictKey = entries[0][0];
}
this.cache.delete(evictKey);
}
}When RL Makes Sense (and When It Does Not)
RL is powerful for problems where optimal behavior depends on sequences of decisions and the environment dynamics are complex or unknown. But it is overkill for simple rule-based decisions and can be brittle when reward signals are sparse or poorly designed.
Good fits for RL: dynamic resource allocation, adaptive rate limiting, autoscaling, recommendation systems, automated testing strategies. Poor fits: one-shot decisions, problems with clear analytical solutions, situations where you cannot simulate the environment.
Key Takeaways
- RL learns from consequences — unlike supervised learning, the agent discovers optimal strategies through trial and error in an environment
- Exploration vs. exploitation is the core tradeoff — epsilon-greedy with decay is a simple and effective starting strategy
- Q-learning builds a value table — mapping state-action pairs to expected rewards lets the agent make informed decisions
- The reward function defines what "good" means — poorly designed rewards lead to unintended behavior; be explicit about what you want to optimize
- RL maps to real engineering problems — cache eviction, load balancing, and autoscaling are all sequential decision problems under uncertainty
- Start with simple environments — simulate the problem in TypeScript before reaching for heavy ML frameworks
