This error can occur if an antivirus or backup software prevents Reason from deleting a temporary file. It is filed as a bug, and will be fixed in a future version. When Reason is starting up, the latest ReWire DLL is extracted to a temporary file. This temporary file is then compared to the ReWire DLL currently installed on the OS, to make sure ReWire is up-to-date. Once this check has been done, Reason will delete the temporary file. If the temporary file has been locked by an antivirus or backup software, preventing Reason from deleting it, the "Rewire installation Failed because access to the file or Directory was denied while installing the Rewire.dll in the system directory" alert is shown. To avoid this from happening, temporarily disable the antivirus/backup software while starting up Reason.
Note that Reason 11 and later does NOT feature ReWire, since you can use Reason 11 as a Rack Plugin in a compatible host DAW.