Most kanji were originally imported from China. On readings (or on'yomi) are based on the original Chinese pronunciation of the character. They are in effect Chinese loanwords to Japanese. These readings are often used in compound words.
There are various types of 'on' readings. Goon readings are based on varieties of Chinese from the 5th to 6th centuries. Kan'on readings are based on the Chinese of the Tang dynasty from the 7th to 9th centuries. Kan'yoon readings are non-standard versions of other 'on' readings. Each character might have several 'on' readings, and each of them is generally linguistically related since they all derive from Chinese.
Kun readings (or kun'yomi) are based on native Japanese words, matched to a character that approximates its meaning. Characters sometimes have several kun readings each (though some of them may be obscure), and they are generally not linguistically related.
Complicating the matter is the fact that different kanji with similar meanings may share the same kun reading.
Uncommon readings that typically only occur in proper names are called nanori. Nanori readings are often based on the kun readings.