Maybe you should try out a more recent version of Racket?
That's a good idea. I'm still using MzScheme 4.2.1. (Although before I go to the trouble, maybe I should skim recent release notes to see if they were supposed to have fixed any memory leak.)
Thanks for doing that test. I just tried again on my case and it got to > 450 MB (90%+ on a 512 machine) without showing signs of GCing! I'll push it even further later to see if it collects; if it does, then maybe I just need to lower a global like threadlimit* if I don't want the entire machine to get consumed.