You install the AAX version of Reason Rack Plugin by installing the full Reason 11.3 program.
During the installation of Reason 11.3/Reason 11.3 Suite or Reason 11.3 Intro you will be given the option to install the AAX, AU (Mac) and VST3 versions of Reason Rack Plugin.
To choose what plugin types (VST3/AU/AAX) you want to install, click the Customise button in the Installation window:
Make sure you have selected the AAX version of Reason Rack Plugin. This will then be installed in:
Macintosh HD/Library/Application Support/Avid/Audio/Plug-Ins
C:\Program Files\Common Files\Avid\Audio\Plug-Ins\