A cross or a letter X that is either standing alone or framed within a box. This icon has become a standard for closing windows. It was apparently first featured in 1985 in Atari TOS 1.0. The X symbol might be a reference to Batsu, which is a Japanese symbol for incorrect.
Surprisingly it wasn't adopted by Microsoft before Windows 95. Apple also didn't use it before OS X. Apple's previous version of the icon was simply an empty box. In Windows the close icon used to be a box with a horizontal bar in the middle.
Here's a brilliant article about the history of the close icon as we know it today by Lauren Archer: https://medium.com/re-form/x-to-close-417936dfc0dc
Here's another thorough post by "Antonio" about the evolution of the title bar buttons: http://blog.placeit.net/evolution-of-the-titlebar-buttons
Atari TOS 1.0 1985.