Which words BELONG together? ‘New York’, ‘strong tea’ — some pairs co-occur far more than chance would predict. Pointwise mutual information measures exactly that: the log-ratio of how often two words appear together versus how often they would if independent. High PMI marks a collocation; near-zero means they just happen to share a page. Slide the association.
Pointwise mutual information PMI(x,y) = log₂( p(x,y) / (p(x)·p(y)) ): the log-ratio of the joint probability to what independence predicts. PMI > 0 means x and y co-occur MORE than chance (a collocation / bound pair); PMI = 0 means independent; PMI < 0 means they avoid each other. It is the core association measure in lexicography and word-embedding theory (word2vec implicitly factorizes a shifted PMI matrix). A fail-loud self-check throws unless a bound pair gives PMI>0 and an independent pair gives PMI=0. ◆ real information theory, node-verified.
p(x),p(y),p(x,y) are set for the demo; the log-ratio is exact. Raw PMI over-rewards RARE pairs (small denominators), which is why practice uses PPMI or PMI² / add-k smoothing — a real caveat, noted not hidden.