Sand INTO THE GRAIN PORES. Very important to do this, or you'll remove the grain fill from the wood.
Depending on how well you leveled the grain fill with the surface of the wood, and how much excess is left on the wood, be prepared to use a lot of 320 grit sandpaper, as it will fill up fast.
Just keep on using fresh sandpaper, and work carefully...