Category Archives: Default

Claude Shannon on AI slop

From his famous Bandwagon editorial in 1956: Secondly, we must keep our own house in first class order. The subject of information theory has certainly been sold, if not oversold. We should now turn our attention to the business of research and development at the highest scientific plane we can maintain. Research rather than expositionContinue reading “Claude Shannon on AI slop”

Can you measure the built-in potential of a pn junction?

When we talked about the built-in barrier potential of a pn junction, perhaps you thought about measuring it on a disconnected diode in the lab. (Even if you didn’t think of it, you should go ahead and try it). You will find that it doesn’t work. It is worth thinking about why i.e. why can’tContinue reading “Can you measure the built-in potential of a pn junction?”

Are holes real?

Charge transport by holes is an important concept in semiconductor physics. In particular, it is important to understand charge transport by holes as different and separate from charge transport by electrons. What makes this a tricky concept is that holes are generally described as the absence of an electron or slightly less informally as unoccupiedContinue reading “Are holes real?”

Diffusion Models – 1: The Surprisingly Tricky Kolmogorov Equations

This is the first of a series of notes to understand the mathematics of diffusion models from the perspective of an electrical engineer with a background in the mathematical theory of signals and systems based on frequency domain analysis and the Fourier Transform. Consider a stochastic process and let , be the conditional probability thatContinue reading “Diffusion Models – 1: The Surprisingly Tricky Kolmogorov Equations”

LLMs are Slaves to the Law of Large Numbers

New preprint: https://arxiv.org/abs/2405.13798 We propose a new asymptotic equipartition property for the perplexity of a large piece of text generated by a language model and present theoretical arguments for this property. Perplexity, defined as a inverse likelihood function, is widely used as a performance metric for training language models. Our main result states that theContinue reading “LLMs are Slaves to the Law of Large Numbers”

The Gaussian distribution — 3: Vector Apples and Oranges

In Part 1 of this series, we presented a simple, intuitive introduction to the Gaussian distribution by way of the Central Limit Theorem. In Part 2, we introduced multi-variate Gaussian distributions, and also looked at certain weird ways of constructing Gaussian random variables that are not jointly Gaussian. If we disregard these “unnatural” mathematical constructionsContinue reading “The Gaussian distribution — 3: Vector Apples and Oranges”

The Gaussian distribution — 2: Frankenstein Monsters

In Part 1, we introduced the Gaussian distribution as naturally arising from the mixing of a large number of independent random variables. Random mixing has an averaging effect that can be described by a sequence of approximations. As a first order approximation, random mixing reduces the size of fluctuations; asymptotically, the sample average of iidContinue reading “The Gaussian distribution — 2: Frankenstein Monsters”

Informal Introduction to the Gaussian Distribution – 1: Central Limits

Consider a random variable X obtained from a random experiment E with the mean, variance and density function . First- and Second-order approximations. The mean and variance provide a simple, partial statistical description of the random variable X that is easy to understand intuitively: the mean is the center of mass of the distribution ,Continue reading “Informal Introduction to the Gaussian Distribution – 1: Central Limits”