is the bar definitely 5mm? (ie have you actually measured it with a micrometer, or even verniers) does your thread have sharp peaks or flat ones? is the thread you have made true to the bar? it can be hard starting the die true and they don`t tend to pull themselves true as you go along. sometimes swarf can just clog and over-cut the thread as you go, if you liberally wax the die and bar with candle wax and go softly softly that may help. or pack the die out with lard to catch the chips. generally you always get some problems, whether tapping or dying thread, from the chips/swarf generated.
my best guess is the cheap mass produced nut problem suggested earlier, and beware, .75mm is also a common m5 pitch.