Fraps can't do out of game recording. You're better off with HyperCam 2.

for something like that a few screencaps would be all you'd need rather than a full video capture

but yeah, the main thing you need to do to fix the palette is get one of the original portrait files from StarCraft that works properly and use the palette from that rather than a new one of your own.

just get a palette from extracted picture which inside image of original *.smk file
and then apply color table of palette to ur all of custom portrait pictures

make sure assign starting-ending frames in Old SMK tools menu when u compile

if u just keep that rule then u can make work successfully everytime