Glossary
Swirly bokeh
Swirly bokeh is the effect where out-of-focus highlights toward the edges of the frame stretch and rotate around the subject, forming a swirl.
Also called swirl bokeh, rotational bokeh, cat's eye bokeh
How it works
It is caused by mechanical vignetting. Near the edges of the image circle, the lens barrel and rear elements clip the cone of light reaching the sensor, so a highlight that would be a circle in the center is cut into a lens-shaped oval, often called a cat's eye. Because those ovals are all oriented toward the center, the eye reads the pattern as rotation.
The effect needs three things: a lens whose image circle is barely larger than the format, a wide aperture, and a busy background with small bright highlights. Stopping down shrinks the cone of light, the clipping disappears, and so does the swirl. Foliage with sunlight behind it is the classic test subject.
The look is strongly associated with the Zeiss Biotar 58mm f/2 and its Soviet copy, the Helios 44 series. It is a side effect of an older, tighter design rather than a feature anyone engineered on purpose.
Why it matters when buying
If you want the swirl, shoot at maximum aperture on a full-frame body and keep the background busy. On a cropped sensor the frame samples only the middle of the image circle, where there is no clipping, so the swirl largely disappears.