I've fought this one a few times, there's a few things that can be tried.
The biggest one is using a 64-bit installation of Arbortext, helps with this issue a lot.
The next immediate step is setting some ACL in the custom\init directory:
if (!java_init()) {
set javavmmemory=768
}
In a distributed environment, you'll have to pick a JVM memory setting that works with your lowest common denominator PC. I started hitting this error when we moved to using SVG images in our XML documents, the batik library was choking on them. The jvm memory setting helped move forward, but testing the 64-bit Editor eliminated the issue without needing the jvm setting.