Sqlservr.exe is using all the memory even after setting memory at certain number.

We have a server running Windows Server 2008 Enterprise SP1

SQL 2008 SP1

Lately the system service Sqlservr.exe has been chewing up the memory.  We added more memory and manually capped/set what sql can use.  It still eating up all the resources and more. 

Only thing we get is an informational entry in the event viewer -  AppDomain 516 (CommServ.dbo[runtime].522) is marked for unload due to memory pressure. I believe that to be due to the machine being close to running out of memory. 

When can reboot and of course it starts low and climbs up in a matter of days crashing the system. 

So at this point any guidance as to why the the memory is still passing the limit set?  Or what else can I look at to prevent this from happening. 

Thanks