Latency

I was in a work flow. Everything was going great until I loaded another instrment patch. The minute I loaded the intrument there was latency issues. 

I closed out the project and opened an older project (a huge project that containd a lot of data/audio) and there was NO issue. 

I shut the computer down. Started over. Went back to the project and there was the same latency issue. I started a new project. Lot's of data and audio. No Problem. 

It seems to be isolated to this ONE project. How can I fix the latency in this ONE project?