THANK YOU MARSHALH!!
Marshal and I were talking at MGC and I asked him about my audio amp problem, there was interference on the headphone jack, although I'm pretty sure it was on the speakers too, just too quiet to hear. He told me to check the power line and try putting an inductor on it to boost the power. I didn't have one so I just got 5 volt source off of the n64 board and it worked like a charm!
Unfortunately while I was working on this I discovered a new problem. -_-
I looked into it some more and can't find a problem right now. Oh well, I'll get to it some other time, I really haven't felt like working on it lately.
So how does the switch on a headphone jack work? Never figured this one out.