Pre-req
Powershell Script which does the task you want to achieve
machine should have powershell installed on it. Hence these type of custom actions will fail on windows XP

Now Installshield doesnt offer adding a new powershell custom action atleast in the MSI / MST. though this feature is present in the ISM and the same is true for KillProcess custom which comes shipped with Installshield 2013(still working on the turnaround for this custom action and will be covered up in the next blog).

So as one of my friend on this forum suggested , if we wish to execute powershell based custom action we'll have to go via using an EXE based custom action. Hence implementing the same , following are the steps to be followed to add a powershell custom action in your package, then be it MSI/MST

1. Create a directory structure which mimics the location of powershell.exe present on our system which is C:\Windows\System32\WindowsPowerShell\v1.0. (creating directory structure shouldnt be much of a pain for you all)
2. Next Add the PS1 file in your package in a suitable location, preferrably INSTALLDIR (I find it adding in that location more convinient than any other location due to ease of accessing it but the choice of location solely depends on the user.)
3. Next Add custom action of the type 3106 which is EXE stored in a directory table. so the custom action table will now look as follows

 
Action:StopService(Name of my custom action)

Type:3106

Source:v10(The directory structure i created to mimic the location of pwoershell.exe present on my location)

Target:powershell.exe -file "[INSTALLDIR]ServiceTest.ps1"

Now next step is faily simple which is scheduling the custom action and defining its in script execution .
Cheers you're done. Test the application.

Hope this little tutorial helps you'll. Let me know in case of any issues