Jerome,
I've had good luck with B&H - they are at 420 9th Avenue. Biggest and best selection, prices are reasonable. They are also on the web. I stopped there in 2012 and was just in awe of all the stuff on display. I got photo batteries, some 8mm Digital Tapes, and a better camera strap for my Nikon.
As to records and CDs, I dunno -
Gothamist may have a recommendation -
http://gothamist.com/ - I used them for restaurant reviews during my stay.
You may have to go into another borough for music - rents are ginormous in Manhattan, making it tough on small retailers. Brooklyn, Bronx and Queens will most likely have the CD and record stores. Genre? No guarantees there.
EDIT: Also, ask doormen and cabbies -they know the places too.