To execute the script you need to create custom action which runs PowerShell. Note that you cannot run SharePoint Management Shell (at least I didn't find how to do that :). But you can run PowerShell and load SharePoint snap-in inside the script:
<CustomAction Id="PSExecute" Property="PS" ExeCommand="&'[FOLDER]\install-extension.ps1'" Execute='deferred' Return='check'/>
The script execution may fail due to some reasons. For example, SharePoint Administration service is not running, PowerShell Execution Policy prevents scripts from running, you run msi under account who is not SharePoint Farm administrator. Maybe there are another reasons that I don't know yet ;)
When the script is executed from msi, the PowerShell console is running but you should be a very fast person to catch its output :) So, it is required to add logging into your script. Using Start-Transcript is the only that I'm able to use. Refer to the following post for more information about Start-Transcript.
One more useful thing is "throw 'ERROR'" to abort the script execution if needed. For example SP-GetObject and SP-AddSolution cmdlets may throw errors during execution but they will not cause the script to be completed with error. And installer completes successfully but it actually does not do anything. You can add throw 'ERROR' where is needed and installation ends with Fatal Error.