The eject icon has apparently originated from the first tape recorders in the 1960's.
The version of the icon we are accustomed to nowadays has a horizontal rectangle with an arrow pointing upwards on top of it. However, in the 1960's there was also a variation where the rectangle was positioned above the arrow. Like in the 1965 Grundig C 100. Apparently the rectangle represents a cassette or cartridge and the arrow represents movement.
1960's tape recorders