The criteria that come to mind are not exclusive to horror novels, so I'm a little confused and annoyed (annoyed because I find horror novels scary, and yet just cannot define it logically when I try to distill the reasons in words). Some of the criteria that come to mind are:
Blood and gore (also found in war novels)
Death and destruction (war novels)
Fear of the unknown (Crew of a nuclear submarine)
The Supernatural (Beautiful Creatures)
Monsters (Twilight, Warm Bodies).
The powerlessness of the protagonists (crew of a nuclear submarine)
Isolation and loneliness (Count of Monte Christo)
Protagonists who die along the way
Societal chaos and disorder (novels set in a post-apocalyptic world)
Does anyone have any criteria or books to suggest?