After playing around with it a bit, and I think I found a method.
Open Task Manager and have it show More details. Right click on Cortana and select Open file location.
Now find the Cortana folder, right click it, and select Rename. I would recommend just adding ".bak" to the end of the folder name so you can find it easily if you want to restore it back to it's original status.
If you attempt to rename, it will tell you that the folder is in use.
This is when you want to go back to Task Manager, right click on Cortana, and select End task.
Right after the task ends, switch back to the Folder In Use window and click Try Again. The folder should be renamed and the Cortana task will not start again.