I'm trying to get the Defiler to have a spell that instant kills a unit in place of its Plague ability. I've never done a custom spell before. But I have firegraft, DatEdit, WinMPQ and IceCC. Do I need to change something in the scripts or what?

For an instant-kill spell, take some existing spell, and set it to do really high damage.

Note that for the most part, custom spells are impossible. There are a few exceptions (FireGraft has exe edits to reverse the effects of Recall and Ensnare, and Spawn Broodlings, Dweb, Dswarm, and Scanner Sweep can be used to activate/aim "spells" created with triggers [like in mapping].)

yeah but what program can you do that in?

Quick, dirty, despicably messy solution;

Use datedit, turn Plague weapons.dat into a clone of yamato gun. Set the damage to something like 4269.

Now Plague is a rape machine.

would just copying a weapon like the C-10 Canister Rifle and paste it on plague work too?

Sure.