You have not configured your DomainService to serilalize enough objects. The default is very small. Try this tip I put in yesterday to increase the resultset allocation
Your data source may be timing out. In that case you need to increase the command timeout for LINQ to SQL, EF, or ADO.NET accordingly. This is the less likely cause, but one to consider.