If you get the System.OutOfMemoryException exception when you map large messages, may be you need to read this kb "The BizTalk service restart unexpectly when you map large message in BizTalk Server 2004". It explains the cause and possible solutions to solve this problem.
For more information go to http://support.microsoft.com/kb/913959