For, say, the holes for a screen or D-pad? Mine are bumpy, and I see that other's have the same thing looking at worklogs, but the holes end up nice and clean.
Patience. Good files. To get nice clean circles, wrap sand paper around a cylinder of some sort. I use pencils. A good rule to keep in mind is cut small, sand big.