Hynkel, "The Great Dictator"
Hynkel (and Hitler) can really make it to power and control the lives of others. Chaplin said he never would have made the movie if he had known the true depths of what the Nazis were doing, but that element makes Hynkel pretty damn scary in retrospect.
Kommandant Bernsdorff, "49th Parallel"
This wartime thriller was expressly to frighten Americans into getting involved in WWII, so it features a Nazi protagonist who is able to infiltrate the most obscure corners of society. He's a brutal u-boat captain who ends up shipwrecked in Canada. Along with his surviving crew, they travel across the land, meeting various people who think WWII is half a world away, rather than right at their doorstep. His grim devotion to the Nazi party is as terrifying as his brutal methods, taking out anyone who gets in his way.
Dr. Strangelove, "Dr. Strangelove"
What happened to the Nazis after WWII? Some went into hiding, some died, some were tried for war crimes. And some ended up working for the US. That's the case with our mad scientist here, who is confined to a wheelchair and has a crazy right hand that he can't seem to control (it keeps heiling Hitler at inopportune moments). But don't worry, it's not as if he's part of a board of military advisors who thanks to nuclear weaponry have the fate of the entire world in their incapable hands—wait a second: HE TOTALLY IS.
Dr. Christian Szell, "Marathon Man"
A Nazi dentist might seem like something out of a horrifying game of Mad Libs, but it's also the profession of one of the scariest movie Nazis ever, portrayed in "Marathon Man" by master thespian Laurence Olivier. He plays the part with a chillingly restrained manner—this isn't a shouting, maniacal Nazi, but a quiet and exacting one. And he also knows exactly where to drill into your teeth so it hurts the most.
Dr. Josef Mengele, "The Boys from Brazil"
Mengele is a real (and infamous) Nazi scientist who escaped into Brazil sometime after WWII (he was never caught). In the movie, he's working on a diabolical plot to clone Hitler and duplicate the conditions of the real Hitler's upbringing, in order to bring his beloved fuhrer back to the world. As played by Gregory Peck, he's a man who's been hardened into a cold kind of evil, but not so hardened that a pack of wild dogs can't send him to meet the original Adolf. He's still pretty scary until then, though.
Arnold Toht, "Raiders of the Lost Ark"
A Nazi would have to be pretty scary to go up against Indiana Jones, and this Gestapo interrogator fits the bill. He looks a little like Peter Lorre, and his personality is somewhere between a lizard and a black widow spider. He even manages to make a coat hanger menacing, which also puts him in the same league as Joan Crawford. Now that's frightening.
supervillains can be pretty frightening too, especially when they have a gleaming red skull for a face and a voice like Werner Herzog. Having possession of a source of limitless power like the Cosmic Cube doesn't hurt either—all in all, this is one frightening movie Nazi.