WMA is crap. It serves no useful purpose.
MP3 is a very popular lossy format. Lossy means that some of the data is lost during compression. Assuming you have some decent speakers and good sound card, listen to a song in any lossless format, then listen to it as a 128 Kbps MP3 and you will hear a HUGE difference. MP3 is great if you're not an audiophile or you want to shrink an audio file for use on the web.
OGG is another lossy format, and only plays in certain media players (VLC is the only one I can think of offhand). It has an advantage over MP3, though, in that it can achieve better sound quality with the same file size or smaller.
FLAC is a lossless audio format. There are a limited number of media players that support this, too. VLC and Winamp are the only ones I can think of. Since it's lossless, none of the original song information is lost, so the file size will be pretty large, but the song will sound a lot better than if it were compressed. This is compressed, meaning you get about a 50% decrease in file size compared to uncompressed lossless formats.
AAC - This is another lossy format that plays in iTunes, VLC, and a lot of other media players. It's also better than MP3 in that it will give you better sound quality than MP3 with a smaller file size. AAC at 320 Kbps will sound close enough to CD quality for most people, so it's a great alternative to lossless if you don't want to fill your entire hard drive with music but don't want a bunch of crappy low-quality songs.
AIFF & WAV - AIFF is Apple's default lossless format and WAV is Microsoft's. These both use up huge amounts of hard drive space but are better for audio recording and other situations where you want to capture as much audio quality as possible before compressing the file. As far as I know neither of these formats are compressed, so the resulting files' sizes will be huge.
Apple Lossless (ALAC) - Another lossless format created by Apple. These are stored in an MP4 container, so they have a .mp4 extension, and can be played in iTunes and VLC (though I don't know which other players support it). It's a compressed lossless format, so the file size is about half of what it would be for an uncompressed lossless file, but it can be decoded very quickly, which makes it great for low-power devices like the iPod.
Monkey's Audio - A lossless format that achieves similar file sizes to ALAC files, but is a lot slower to decode and can't be played in most media players. I actually have no idea what media will play this because I never bothered with it. It has a .ape extension.
AU - A file format that was used as the standard for the UNIX, SunOS, and NeXTSTEP operating systems. Ends with .au and can be either lossy or lossless depending on how it was encoded. You generally will never see this format anywhere unless you use Solaris (SunOS is now called Solaris).
There are plenty of other formats, but they're all pretty obscure and not worth mentioning.