Information theory

Part: Computer Science · Chapter: Information theory · Edition: 2026-07-12
Keywords: entropy, channel, coding, IEML, mutual information
Glossary: entropy, mutual information, channel capacity, coding — see GLOSSARY

At a glance

• Information theory treats entropy, coding, channels and the quantitative limits of communication.
• The chapter is an index of that field.

This chapter is an index of information theory: entropy, coding, channels and the quantitative treatment of uncertainty and communication. It is orientation, not a textbook.

Online references (live)

• Information theory
• Entropy (information theory)
• Shannon’s source coding theorem

Offline snapshots (link-rot shield)

Offline snapshot omitted in this web edition. Offline snapshot · Information theory · https://en.wikipedia.org/wiki/Information_theory · retrieved 2026-07-12

Information theory

Offline snapshot omitted in this web edition. Offline snapshot · Entropy (information theory) · https://en.wikipedia.org/wiki/Entropy_(information_theory) · retrieved 2026-07-12

Entropy (information theory)

Offline snapshot omitted in this web edition. Offline snapshot · Shannon's source coding theorem · https://en.wikipedia.org/wiki/Shannon%27s_source_coding_theorem · retrieved 2026-07-12

Shannon’s source coding theorem

Applications at Keyne Digital Assets

Entropy, coding and channel limits are the language we use when we ask how much a source can tell us, and at what cost. This chapter orients that work. It is not a claim that Situation Monitor implements every model listed here.

Information Theory and the mathematical foundation

1. Fundamental Quantities (The “Alphabet” of Information)

• Self-Information: The “surprise” of an individual even
• Entropy ($H$): The average uncertainty or information content of a random variable.
• Joint Entropy: Uncertainty of a pair of random variables ⁴$(X, Y)$.⁵
• Conditional Entropy: Uncertainty of ⁶$X$ given that ⁷$Y$ is known.⁸
• Mutual Information (⁹$I$): The amount of information shared between two variables; a measure of dependence.¹⁰
• Relative Entropy (Kullback-Leibler Divergence): A measure of the “distance” or difference between two probability distributions.
• Cross Entropy: Used extensively in machine learning loss functions.¹¹

---

2. Source Coding (Data Compression)¹²

The goal is to represent information using the fewest possible bits.

• Lossless Compression:
• Huffman Coding: Optimal prefix-free codes based on frequency.
• Arithmetic Coding: Encoding an entire message as a single number between 0 and 1.
• Lempel-Ziv Algorithms (LZ77, LZ78, LZW):¹³ Dictionary-based compression (basis of ZIP/PNG).
• Source Coding Theorem: Establishing that the fundamental limit of compression is the Entropy (¹⁴$H$).¹⁵
• Quantization:
• Scalar vs. Vector Quantization.
• Rate-Distortion Theory: The trade-off between the number of bits used and the accuracy of the representation.

---

3. Channel Coding (Error Correction)

Ensuring reliable communication over noisy media.

• Channel Capacity (¹⁶$C$): The maximum rate at which information can be sent with vanishingly small error.¹⁷
• Shannon-Hartley Theorem: The limit for a Gaussian channel: ¹⁸$C = B \log_2(1 + \frac{S}{N})$.¹⁹
• Linear Block Codes:
• Parity Checks and Syndrome Decoding.
• Hamming Codes (Single-error correction).²⁰
• Reed-Solomon Codes (Used in CDs and QR codes).²¹
• Convolutional Codes: Viterbi decoding and Trellis-coded modulation.²²
• Modern High-Performance Codes:
• Turbo Codes: Using iterative decoding.²³
• LDPC (Low-Density Parity-Check) Codes: Used in 5G and satellite TV.
• Polar Codes: The first codes proven to achieve channel capacity.

---

4. Algorithmic Information Theory (Complexity)

• Kolmogorov Complexity (²⁴$K$): The length of the shortest computer program that produces a specific string.²⁵
• Incompressibility: Most strings are random and cannot be compressed beyond their length.
• Solomonoff Induction: A universal theory of prediction based on $K$.
• Chaitin’s Constant ($\Omega$): The probability that a randomly constructed program will halt.

---

5. Network Information Theory

Dealing with multiple senders and receivers simultaneously.

• Multiple Access Channels (MAC): Multiple senders to one receiver (e.g., cell towers).
• Broadcast Channels: One sender to many receivers (e.g., TV, Radio).²⁶
• Relay Channels: Intermediate nodes that assist in transmission.²⁷
• Network Coding: Max-flow min-cut theorem for information flow; nodes can “mix” packets rather than just forwarding them.²⁸

---

6. Quantum Information Theory

The behavior of information when stored in quantum states (qubits).

• Von Neumann Entropy: The quantum analogue of Shannon entropy.²⁹
• Entanglement: Einstein-Podolsky-Rosen (EPR) pairs and Bell states.
• Quantum Channel Capacity: Holevo’s bound on how much classical info a quantum channel can carry.³⁰
• Quantum Error Correction (QEC): Surface codes and the Shor code for protecting qubits from decoherence.³¹
• No-Cloning Theorem: Proving that an unknown quantum state cannot be copied perfectly.³²

---

7. Statistical Inference & Machine Learning

• Maximum Entropy Principle: Choosing the distribution that assumes the least about unknown data.
• Akaike Information Criterion (AIC): Model selection based on information loss.
• Minimum Description Length (MDL): A principle for statistical induction where the “best” model is the one that compresses data most efficiently.
• The Information Bottleneck: A technique for understanding deep neural networks by analyzing the trade-off between compression and relevance.³³

---

8. Physical & Biological Information

• Maxwell’s Demon: The relationship between information and Thermodynamics (Landauer’s Principle: erasing 1 bit of info generates heat).
• Black Hole Information Paradox: The debate over whether information is lost when it falls into a black hole (Hawking Radiation).
• DNA Informatics: Information density of genetic storage and biological signal transduction.
• Molecular Communication: Using chemical signals to transmit data in nanonetworks.

---

9. Modern Engineering Applications (2026)

• Massive MIMO: Spatial multiplexing for 6G networks.
• Semantic Communication: Transmitting the “meaning” of a message rather than every bit (AI-driven).³⁴
• Joint Communication and Sensing (JCAS): Using information signals as radar to map the environment.
• Coded Caching: Reducing network congestion by storing fragments of data near users.

Introduction

In his philosophical writings, Albert Einstein emphasized that the utility of language depends entirely on the degree to which words and word-combinations correspond to the immediate data of sensory experience, which he termed the “world of impression”. Key aspects of Einstein’s views on language and the world of impressions:

• Language as a Tool: Einstein viewed language as a vital “instrument of reasoning” but also as a “dangerous source of error and deception” if its terms are not grounded in empirical reality. The validity of language, particularly in science, stems from its connection to shared sensory experiences (impressions).
• Thinking Beyond Words: He famously noted that words of language “do not seem to play any role in my mechanism of thought”. His primary process of perceiving and thinking was visual and muscular, involving “concepts and concept-combinations for which words need not necessarily come to mind”.
• Conceptual Construction: The “world of impression” provides the raw data (“elements” in the sense of philosopher Ernst Mach) from which humans create mental and conceptual constructions. These mental constructs, while “free statements of our thoughts,” are justified only insofar as they help us orient ourselves and form logical connections among sensory experiences.
• Intuition and Rules: Einstein believed that children grasp the rules and stable correspondence between linguistic signs and impressions mainly by intuition when acquiring language.
• The Common Language of Science: He advocated for the “supernational character” of scientific language (including mathematical signs) because its concepts are universally connected to objective empirical facts, allowing scientists from different cultures to cooperate and build a shared understanding of the universe, transcending potential errors from everyday, less-precise language.

In essence, for Einstein, language serves as a crucial, but potentially flawed, bridge between internal thought processes and the shared, objective world of sensory impressions. He believed that everything in scientific and general understanding “depends on the degree to which words and word-combinations correspond to the world of impression”

Language and world of impressions

In 1941, the great physicist was recorded reading an essay he wrote called “The Common Language of Science.”

In this fascinating excerpt, he warns of the dangers of “errors and deception” that come from letting your language (and ideas) drift away from the reality perceived by your senses.

Good message for all times — especially ours.

The Common Language of Science

The first step towards language was to link acoustically or otherwise commutable signs to sense-impressions. Most likely all sociable animals have arrived at this primitive kind of communication–at least to a certain degree. A higher development is reached when further signs are introduced and understood which establish relations between those other signs designating sense impression.

At this stage it is already possible to report somewhat complex series of impressions; we can say that language has come to existence.

If language is to lead at all to understanding, there must be rules concerning the relations between the signs on the one hand and on the other hand there must be a stable correspondence between signs and impressions.

In their childhood individuals connected by the same language grasp these rules and relations mainly by intuition. When man becomes conscious of the rules concerning the relations between signs the so-called grammar of language is established.

In an early stage the words may correspond directly to impressions. At a later stage this direct connection is lost insofar as some words convey relations to perceptions only if used in connection with other words (for instance such words as: “is”, “or”, “thing”). Then word-groups rather than single words refer to perceptions.

When language becomes thus partially independent from the background of impressions a greater inner coherence is gained.

Only at this further development where frequent use is made of so-called abstract concepts, language becomes an instrument of reasoning in the true sense of the word. But it is also this development which turns language into a dangerous source of error and deception.

Everything depends on the degree to which words and word combinations correspond to the world of impression.

What is it that brings about such an intimate connection between language and thinking?

Is there no thinking without the use of language, namely in concepts and concept-combinations for which words need not necessarily come to mind? Has not every one of us struggled for words although the connection between “things” was already clear?

We might be inclined to attribute to the act of thinking complete independence from language if the individual formed or were able to form his concepts without the verbal guidance of his environment. Yet most likely the mental shape of an individual growing up under such conditions, would be very poor.

Thus we may conclude that the mental development of the individual and his way of forming concepts depend to a high degree upon language. This makes us realize to what extent the same language means the same mentality.

In this sense thinking and language are linked together.

Entropy and Community Management

Entropy is a key measure in information theory. It represents the degree of randomness or uncertainty in a system. The greater the information, the higher the disorder and entropy. Conversely, less information results in less choice and lower entropy. This concept quantifies the amount of uncertainty in the value of a random variable or the outcome of a random process.

For instance, the outcome of flipping an honest coin (with two equally likely outcomes: heads or tails) provides less information (lower entropy) than the outcome of rolling a six-sided die (with six equally likely outcomes).

Social and Economic Context

In the context of community management, particularly considering social and economic factors, entropy plays a vital role in achieving balanced states. Communities often face a range of unpredictable events and choices that can influence their stability and development.

• Economic Decisions:
• High entropy can reflect a diverse and dynamic economy where numerous unpredictable factors affect market conditions, employment, and investments. Managing this entropy involves creating policies that can adapt to changes and reduce uncertainties for businesses and individuals.
• Social Dynamics:
• Social entropy might encompass the variety of individual behaviors, cultural practices, and social interactions within a community. Higher social entropy indicates greater diversity and complexity, which can be both a source of strength and a challenge for cohesive management. Effective community management seeks to balance this diversity to ensure inclusivity and harmony.

Important Measures in Information Theory

• Mutual Information:
• This measures how much information one variable contains about another. In community management, understanding the mutual information between different social or economic factors can help predict outcomes and make informed decisions.
• Conditional Information:
• This measures the amount of information needed to describe the outcome of a variable given the knowledge of another variable. In practice, this could help community managers understand the impact of specific policies or interventions on various aspects of the community.
• Channel Capacity:
• This refers to the maximum rate at which information can be reliably transmitted over a communication channel. For community management, this might translate to the effectiveness of communication channels within the community, ensuring that vital information reaches all members efficiently.

Balancing Entropy for Community Stability

By understanding and managing entropy, community leaders can better navigate the complexities of social and economic environments. This involves:

• Adaptive Policies:
• Developing flexible policies that can respond to changing conditions and reduce uncertainties for community members.
• Inclusive Communication:
• Ensuring clear and effective communication channels that reach diverse segments of the community, reducing misunderstandings and fostering cohesion.
• Data-Driven Decision Making:
• Utilizing information theory principles to analyze social and economic data, predict trends, and make informed decisions that enhance community resilience and stability.

In summary, entropy and related measures in information theory provide valuable insights for managing the uncertainties inherent in social and economic systems, helping communities achieve balanced and sustainable states.

Entropy in Information Theory

It represents the degree of randomness or uncertainty inherent in a system or process. The higher the randomness or disorder, the higher the entropy. Conversely, the lower the randomness, the lower the entropy.

Understanding Entropy with Examples

Consider the example of flipping a coin and rolling a die to illustrate entropy:

• Coin Flip (Two Outcomes):
• An honest coin has two possible outcomes: heads or tails.
• Each outcome is equally likely, with a probability of 0.5.
• The uncertainty (entropy) involved in predicting the outcome is lower because there are only two possible outcomes.
• Die Roll (Six Outcomes):
• A fair die has six possible outcomes: 1, 2, 3, 4, 5, or 6.
• The uncertainty (entropy) involved in predicting the outcome is higher because there are more possible outcomes.

Quantifying Entropy

Entropy quantifies the amount of uncertainty involved in the value of a random variable or the output of a random process. It is a fundamental concept introduced by Claude Shannon to measure the information content.

Mathematically, for a discrete random variable (X) with possible values ({x_1, x_2, …, x_n}) and corresponding probabilities (P(X = x_i) = p_i), the entropy (H(X)) is defined as:

This formula sums the product of each probability and the logarithm of that probability, reflecting the average amount of information produced by the random variable.

Other Important Measures in Information Theory

• Mutual Information:
• This measures the amount of information that one random variable contains about another random variable. It quantifies the reduction in uncertainty of one variable due to the knowledge of another.
• Conditional Information:
• This measures the amount of information needed to describe the outcome of a random variable given that the outcome of another random variable is known. It reflects the remaining uncertainty.
• Channel Capacity:
• This is the maximum rate at which information can be reliably transmitted over a communication channel. It represents the upper bound on the amount of information that can be communicated with a negligible probability of error.

Summarizing the Concept

Entropy, in essence, is a measure of uncertainty or unpredictability. The higher the entropy, the more information or randomness a system possesses. Conversely, lower entropy indicates less randomness and more predictability. By quantifying these aspects, information theory provides a framework to understand and optimize the transmission, processing, and storage of information.

Practical Applications

Understanding entropy and these measures is crucial in various fields:

• Data Compression:
• Reducing the amount of data needed to represent information without losing the essence.
• Cryptography:
• Ensuring secure communication by maximizing entropy to make patterns harder to detect.
• Telecommunications:
• Enhancing the efficiency and reliability of data transmission over networks.

By leveraging these concepts, engineers and scientists can develop more robust and efficient systems for handling information in today’s data-driven world.

Emptiness and potential form

Emptiness in IEML and the Semiotic Triangle

In the Information Economy Metalanguage (IEML), the semiotic triangle begins with the concept of emptiness. This emptiness represents a primordial state from which differentiation and meaning arise. The progression from emptiness involves:

1. Dyad: The initial differentiation into Actual vs. Virtual.
2. Triad: Further differentiation into Sign, Being, and Thing.

Emptiness signifies potentiality and the undifferentiated state from which all distinctions emerge. It is a foundational concept that precedes the existence of any particular forms or meanings.

Monad in Pythagoras and Leibniz

Pythagoras:

• Monad (Unit): In Pythagorean thought, the monad represents the indivisible and most fundamental unit of reality. It is the origin of all numbers and embodies unity and wholeness.

Leibniz:

• Monad: Leibniz’s monads are simple, indivisible, and immaterial entities that constitute reality. Each monad reflects the entire universe from its perspective and possesses intrinsic properties and potential.

Contrast:

• Emptiness vs. Monad:
• Emptiness in IEML represents a state of potentiality and lack of differentiation, akin to a void from which all structure and meaning emerge.
• Monad in Pythagorean and Leibnizian philosophy represents a fundamental, indivisible unit of existence that contains within it the potential for all forms and expressions of reality.

Practical Applications: Monad in Programming and Information Theory

Monad in Programming

In functional programming, a monad is a design pattern used to handle side effects and structure programs. It encapsulates values and provides a framework for chaining operations while maintaining context.

• Monad Structure:
• Unit (Return): Wraps a value in a monadic context.
• Bind (>>=): Chains operations, passing the output of one function as the input to the next within the monadic context.

Example: In Haskell, the Maybe monad handles computations that may fail.

-- Maybe Monad Example
safeDivide :: Double -> Double -> Maybe Double
safeDivide _ 0 = Nothing
safeDivide x y = Just (x / y)

-- Using the Monad
result = safeDivide 10 2 >>= \x -> safeDivide x 5

• Monad vs. Emptiness:
• Monad: Provides a structure for managing complexity and side effects in computations.
• Emptiness: Represents the undifferentiated potential from which such structures could conceptually emerge.

Monad in Information Theory

In information theory, while the concept of a monad is not explicitly used, the principles can be analogous to how information is structured and processed.

• Information Encapsulation: Just as monads encapsulate values and operations in programming, information structures encapsulate data and metadata in communication systems.
• Sequential Processing: Monadic chaining in programming is similar to the sequential processing of signals and data in communication channels, ensuring that the context (like error correction and synchronization) is maintained.

Example: Error-correcting codes in information theory encapsulate data with redundancy to ensure reliable transmission.

Integrating Concepts

1. From Emptiness to Monad:
   * Conceptual Progression: Emptiness represents potential, and from this potential, structured forms like monads (fundamental units of meaning or computation) emerge.
   * Semiotic Triangle: From the undifferentiated emptiness, differentiation occurs (Actual vs. Virtual) leading to structured triads (Sign, Being, Thing).
2. Practical Integration:
   * Programming: Monads provide a practical implementation of encapsulating and managing complexity, reflecting the transition from undifferentiated potential (emptiness) to structured computation.
   * Information Theory: Concepts like error correction and data encapsulation mirror the process of moving from potential (undifferentiated signal) to structured, reliable communication.

By understanding these concepts in various contexts, from philosophical foundations to practical applications in programming and information theory, we can appreciate the depth and utility of these fundamental ideas.

Programming languages and data structures

Haskell offers the concept of monads primarily because of its strict adherence to functional programming principles, particularly its emphasis on purity and immutability. This design choice sets Haskell apart from many modern programming languages that do not inherently require or promote these concepts to the same extent. Here’s a comparison to elucidate why Haskell emphasizes monads and why other languages might not:

Functional Programming in Haskell

1. Purity and Immutability:
   * Pure Functions: In Haskell, functions are pure, meaning they have no side effects and always produce the same output given the same input.
   * Immutability: Data in Haskell is immutable, which means once a value is created, it cannot be changed.
2. Handling Side Effects:
   * Since Haskell emphasizes pure functions, side effects (like I/O operations, state changes, etc.) cannot be handled in the same way as in imperative languages.
   * Monads provide a way to encapsulate and manage side effects while maintaining purity. They allow for sequencing of actions without violating the principles of functional programming.
3. Chaining Operations:
   * Monads in Haskell (such as Maybe, IO, List, etc.) allow for chaining operations that handle context, like potential failure (Maybe), I/O (IO), or multiple results (List).
   * This chaining is done using the bind operation (>>=), ensuring that the computational context is preserved throughout the sequence of operations.

Modern Programming Languages

1. Imperative Paradigm:
   * Most modern programming languages, like JavaScript, Python, and Java, are primarily imperative or multi-paradigm.
   * They do not strictly enforce purity or immutability, allowing side effects to occur directly within functions and methods.
2. Direct Handling of Side Effects:
   * Side effects in these languages are handled directly through constructs like loops, conditionals, and try-catch blocks, without needing the abstraction of monads.
   * For instance, I/O operations, state changes, and exceptions are managed imperatively, which is straightforward and familiar to most developers.
3. Alternative Constructs:
   * These languages have other constructs to handle common programming patterns:
   • Promises and async/await in JavaScript: Handle asynchronous operations.
   • Try-catch-finally: Manage exceptions and cleanup actions.
   • Optional types: Some languages (like Java with Optional or Swift with Optional) provide ways to handle nullable values without monads.

Why Haskell Uses Monads

• Need for Abstraction:
• In Haskell, monads provide a powerful abstraction to handle various kinds of computation contexts in a consistent way while preserving the language’s functional nature.
• This abstraction allows for writing clean, composable code that handles side effects declaratively.
• Composability and Reusability:
• Monads promote composability and reusability of code. Different monadic operations can be combined seamlessly, leveraging the power of functional composition.
• For instance, do notation in Haskell simplifies working with monads, making the code more readable and maintainable.

Why Other Languages Might Not Need Monads

• Less Strict Purity:
• Since these languages do not enforce strict purity, they can handle side effects directly within their control structures.
• This reduces the need for an additional abstraction layer like monads.
• Established Patterns:
• These languages already have established patterns and constructs for handling common scenarios:
  • Exception Handling: Try-catch blocks.
  • Asynchronous Operations: Promises, async/await.
  • Optional Values: Nullable types or Optionals.

Monads in Haskell are not merely a tool for broad exception handling but a fundamental abstraction that captures and manages computational context in a highly composable and structured way. The term “monad” comes from category theory, where it describes a specific type of functor that allows for chaining operations. Let’s delve into the core differences and why monads are a unique and powerful concept:

Key Characteristics of Monads

1. Encapsulation of Context:
   * Monads encapsulate not just values, but also the context in which those values exist. This could be contexts like computation that may fail (Maybe), side effects (IO), computations with state (State), etc.
2. Chaining Operations:
   * Monads provide a way to chain operations while preserving the context. The bind operation (>>=) allows for sequencing computations, where each step can depend on the result of the previous one.
3. Abstraction and Composability:
   * Monads abstract away the plumbing code required to handle the context, allowing developers to focus on the logic of their computations.
   * They enable composability, making it easy to combine different monadic operations in a seamless manner.
4. Laws and Structure:
   * Monads adhere to three laws (left identity, right identity, and associativity) that ensure the consistency and predictability of monadic operations. These laws guarantee that monads behave in a mathematically sound manner.

Practical Differences and Naming Justification

• Generalized Computation Context:
• Unlike simple exception handling, monads can represent a wide range of computation contexts. For example:
  • Maybe monad: Handles computations that might fail.
  • List monad: Handles computations that produce multiple results.
  • IO monad: Handles side-effecting computations.
  • State monad: Handles stateful computations.
• This generality makes monads applicable to many more scenarios than just exception handling.
• Uniform Interface:
• Monads provide a uniform interface (return and >>=) for handling different types of contexts. This uniformity makes it easier to learn and apply monads across different use cases.
• Enhanced Code Reusability:
• Monadic operations can be reused across different contexts without changing the core logic of the operations, promoting code reuse and modularity.
• Functional Programming Principles:
• Monads fit naturally into the functional programming paradigm by encapsulating side effects and preserving function purity. This alignment with functional principles is a key reason for their importance in Haskell.

Example: Maybe Monad

Consider a series of computations that may fail:

safeDivide :: Double -> Double -> Maybe Double
safeDivide _ 0 = Nothing
safeDivide x y = Just (x / y)

computation :: Double -> Double -> Double -> Maybe Double
computation a b c = do
  x <- safeDivide a b
  y <- safeDivide x c
  return y

In this example:

• Each call to safeDivide returns a Maybe Double, encapsulating the possibility of failure.
• The do notation chains these operations, automatically handling the propagation of Nothing if any step fails, without additional boilerplate.

Comparison to Exception Handling and Promises

• Exception Handling:
• While exception handling deals with unexpected failures, monads like Maybe handle expected scenarios where failures are part of the computation logic.
• Exception handling typically uses try-catch blocks, which are imperative constructs. Monads use a functional approach to handle such contexts within the computation pipeline.
• Promises in JavaScript:
• Promises encapsulate asynchronous operations and provide methods (then, catch, finally) to handle the completion of those operations.
• While promises manage a specific context (asynchronous computation), monads are more general and can encapsulate various types of contexts, not limited to asynchrony.

Conclusion

Monads are a powerful abstraction in Haskell that go beyond simple exception handling or finalization in asynchronous operations. They provide a generalized, composable, and mathematically sound way to handle various computation contexts while preserving function purity and enhancing code reusability. This broad applicability and the structured approach to managing context justify their distinct naming and central role in Haskell’s functional programming paradigm.

Part index

The scientific map, the equations, and the bibliographic spine for this part of the Knowledge Base are collected in the part indexes:

• General · Engineering · Computer science · Playbooks
• Indexes · Glossary · SUMMARY

Wikipedia snapshots in the chapter card offer offline orientation; the part indexes carry the full citation trails.