An immensely helpful resource on this topic is Writer Beware, a volunteer organization associated with the SFWA and MWA. Their essay on warning signs of questionable agents answers your question in great depth.
For completeness, I'll summarize the main points here - but do read the entire article...
A reputable agent charges NOTHING but a percentage
An agent earns his or her money by selling your work. For this reason, their fee is defined as a percentage of whatever contracts they find and/or negotiate for you. Nothing else.
Any other fees - an upfront fee, an editing fee, extra services they urge you to pay for, anything - are a huge danger sign. That's because they're prone to heavy abuse - and that's just what shady agents do: they sign up lots of author-wannabes, do little or nothing to sell their work, and get money from the author instead of getting money for the author.
A reputable agent has a track record
A reputable agent with any experience in the field should have a list of clients he or she has successfully represented. That's a great sign that an agent is legit - just check that he's been legit in the past. Also check the quality of that agent's work - if the agent only lists books printed in small, negligible presses, that probably means they're not a very successful agent.
This information should be available from the agent's website (if it's not, huge warning sign). Be sure to verify whatever information you find, because shady agents can simply make up a false history, as impressive as they like.
If an agent is brand-new, fresh in the field, then they won't have a track record. But in that case, they should have a relevant professional background – Writer Beware says "either working in publishing, or training at a reputable agency." Verify the background, and make sure the agent really is as new as he claims to be.
Sniff 'em out
As John Smithers writes, never be content that an agent "looks OK" - root out whatever you can find about an agent before you commit to anything. Search Preditors and Editors, use Absolute Write's background check forum, Google for
[agent's name] scam OR rip off OR swindler OR blackgaurd, use every resource you can possibly find.
These are just the most fundamental tips. The more you familiarize yourself with industry practices and professional standards, the better you will be able to recognize shadiness and warning signs - as well as being able to spot the real McCoy when you find it.