If you don't want to hold shift key but still delete the file permanently:
Go to Start -> Run then type gpedit.msc and enter Go to User Configuration -> Administrative Templates -> Windows Components -> Windows Explorer and locate "Do not move deleted files to the Recycle Bin" setting and change it.
Enjoy...
Go to Start -> Run then type gpedit.msc and enter Go to User Configuration -> Administrative Templates -> Windows Components -> Windows Explorer and locate "Do not move deleted files to the Recycle Bin" setting and change it.
Enjoy...
Logging you in...