Information TheoryMore information is useful when resolving the uncertainty affects a decision you actually need to make.

What It Is

A customer interview can rule out explanations for why a product is not being used. Two weeks spent polishing the product may leave exactly the same explanations unresolved. Both activities consume resources, but only the first has changed what the team knows about the decision it needs to make.

Claude Shannon formalized information theory in 1948. It measures information as a reduction in uncertainty, in bits. One bit is what the answer to a yes-or-no question supplies when both answers were equally likely: it cuts the number of equally likely possibilities in half. Acquiring information takes energy and resources; its decision value depends on whether the reduction in uncertainty changes what happens next.

An interesting fact can therefore have no value for the current action. A startup searching with limited runway and a control loop both need information that improves a consequential decision before its acquisition consumes the resources available to act.

Shannon's entropy formula measures uncertainty:

$H(X) = -\sum P(x_i) \times \log_2(P(x_i))$

Here $H$ is the entropy, or uncertainty in bits; $X$ is a random variable, the state of the system; and $P(x_i)$ is the probability of state $i$.

Higher entropy means more uncertainty. Information removes possibilities and reduces entropy. What that reduction is worth depends on the value of making the correct decision.

The article's decision-making application asks what different answers would cause you to do, alongside what the information costs to obtain.

Read this diagram

Name the decision → Find the uncertainty → Acquire relevant evidence → Update the choice.

Finding one cell with yes-or-no questionsInteractive example

Size of the grid

16 cells64 cells256 cells

Ask a yes-or-no questionHide a new cell

One cell among 64 is the answer, and nothing has been ruled out yet.

Information Cost vs Information Value

The direct cost of a measurement is only part of what it consumes. It can also displace another experiment, delay the next iteration or require measurement infrastructure:

What a piece of information is worth depends on two things: how much uncertainty it removes, and how much getting the decision right matters. It is worth acquiring when that value is greater than everything it costs, which means the direct cost, the opportunity given up and the delay taken together.

Suppose an interview begins with eight equally likely explanations for the severity of a customer's problem: 3 bits of uncertainty. Behavioral evidence leaves two explanations, or 1 bit. The interview has supplied 2 bits of information.

Suppose also that choosing the right feature has a revenue impact of $100,000 and the interview costs $500 in time and runway. The interview has narrowed eight explanations to two for one two-hundredth of what the decision is worth. At that price it is worth running, provided the answer is actually used to choose the feature.

Now consider two weeks of feature polish. Suppose uncertainty about retention is 2 bits before the work. It is still 2 bits afterward, because the polish tests none of the possible retention drivers. It provides 0 bits at a cost of two weeks' runway. Its value as an information-gathering activity is negative, so the polish should be skipped.

Information Acquisition Strategies

High Information/Cost Ratio Activities

Watching users act reveals preferences through behavior rather than a statement about hypothetical behavior. Testing payment gives those preferences a real price. Retention measurement waits to see whether the value survives a first encounter.

Low Information/Cost Ratio Activities

A hypothetical answer leaves uncertainty about whether the person will act. Building an entire feature before testing it can reveal whether the package works while leaving its components unexplained. A long private beta adds delay and can overrepresent users unlike the eventual public audience.

Shannon's Channel Capacity and Feedback Loops

Shannon's channel capacity theorem gives the maximum transmission rate through a noisy channel:

C = B × log₂(1 + S/N)

Where:
  C = channel capacity (bits per second)
  B = bandwidth of the channel (hertz)
  S/N = signal-to-noise ratio (signal power over noise power)

In a feedback loop, the counterpart of bandwidth is the frequency of measurement, and the counterpart of signal-to-noise ratio is sensor accuracy. A faster measurement cycle and more reliable sensors, including multiple sensors used together, raise the amount of information available to guide corrections.

Higher bandwidth and signal quality increase channel capacity, accelerating learning and optimization. A daily loop with calibrated sensors therefore outperforms a monthly review based on vague impressions.

Mutual Information and Sensor Correlation

Mutual information measures how much learning Y reduces uncertainty about X:

I(X;Y) = H(X) - H(X|Y)

Where:
  H(X) = entropy before measurement
  H(X|Y) = entropy after knowing Y

For startup measurements, the pairs differ in how much one observation reveals about the other:

High mutual information allows correlated sensors to be combined for greater accuracy. Low mutual information means the sensors measure independent dimensions, so both need separate monitoring. A statement of interest cannot replace a measurement of use when it predicts use poorly.

Information Acquisition vs Action Execution

The same resource budget has to fund learning and acting, so whatever goes to finding out is no longer available for doing. The split is right at the point where a little more spent on information would be worth about the same as a little more spent on execution.

Spending 90% of runway on research and 10% on execution can leave a team confident in its direction but unable to capitalize on it. Reversing those proportions can fund extensive work in the wrong direction. The first pattern is analysis paralysis; the second is premature commitment.

The split depends on the initial uncertainty:

The 30-day pattern makes a similar transition. Its first week emphasizes discovering what works; the rest of the month emphasizes repeating the validated behavior. The value of further information falls as the pattern becomes established, while the value of executing it rises.

The Value of Information Formula

Decision theory's expected value of perfect information compares the value of a decision made with perfect information to the value available with current information:

$\text{EVPI} = E[\text{Value}_{\text{perfect}}] - E[\text{Value}_{\text{current}}]$

Each expected value is a sum over the possible states: the probability of a state times the value of the best decision available in it.

The acquisition rule treats EVPI as a ceiling: no information about the decision is worth more than the EVPI, and imperfect information is worth less.

Consider whether to spend two weeks building an analytics dashboard:

Value if you have perfect usage data:
  Build high-value features = +$200K expected value

Value with current vague intuition:
  Random feature work = +$50K expected value

EVPI = $200K - $50K = $150K
Cost = 2 weeks runway ≈ $40K

EVPI > Cost → the dashboard can pay for itself

Perfect usage data raises expected value from $50,000 to $200,000, a difference of $150,000. The two weeks of runway cost approximately $40,000. The cost is well under the ceiling, so the calculation leaves room for building the dashboard: its data would have to capture a little more than a quarter of that $150,000 to pay for itself. Measurement infrastructure earns its return by improving the subsequent decisions at scale.

Integration with Mechanistic Framework

Questions initiate searches that spend cognitive energy. A useful question produces enough uncertainty reduction to justify that computation. An expensive question that leaves the decision unchanged has a poor information-to-cost ratio.

Tracking pays setup and maintenance costs to reduce uncertainty about behavioral patterns. External records allow a decision to use observed data instead of the current mood. Its net value is positive when the decisions it informs are frequent and consequential enough, taken together, to outweigh the setup cost.

The Braindump spends ten minutes daily to externalize working-memory state. Processing that record reveals patterns, reduces internal uncertainty and removes ambiguity about the next action.

Related Concepts

• Startup as a Bug allocates information-gathering resources before they run out.
• Cybernetics uses sensors and feedback to steer a system.
• Optimal Foraging Theory compares search strategies under energy limits.
• Expected Value examines decisions made with incomplete information.
• Tracking builds the measurement record.
• Question Theory examines the computation required by an inquiry.
• The Braindump makes information available for processing.

Key Principle

Information acquisition is worthwhile when the decisions it enables are worth more than its cost. Getting the most out of that difference calls for experiments that resolve consequential uncertainty, reliable behavioral or payment signals, and feedback frequent enough to guide the next action. A mosquito needs enough information to reach blood before its energy runs out. A perfect map of every possible source would be useless if obtaining it consumed the energy needed to fly there.