Scherr was born in New Haven, Connecticut. He played with Woody Herman as a teenager, and moved to New York City in the late-1980s, where he became a prolific session musician, working with artists such as Bill Frisell, John Scofield, Norah Jones, and Ana Egge. He has been a member of a number of bands, such as The Lounge Lizards, Sex Mob, Jesse Harris and the Ferdinandos, and Chris Brown and the Citizen Band. Scherr owns a recording studio and has worked as record producer for many of the artists he performs with.
In 2002, Tony Scherr released his first solo album, Come Around, on Smells Like Records. His song "Sacramento" was later covered by Leslie Feist on her album Let It Die, with alternate lyrics and a new title, "Lonely Lonely".
Tony Scherr's second album, Twist in the Wind, was released March 17, 2008.