New York guidebooks would probably advise you not to visit Harlem, but it's a truly great part of the city for food and culture. Just make a little effort to not be so obviously a tourist: Have a good idea of where you're going, try not to look lost, and move around with some purpose and you won't even be seen.
While you're up town, only take the subway south as far as 110th street and walk through North Woods in Central Park. The city will fades away.