You're absolutely correct. I would go so far as to say that Arc itself is too slow to do any serious matrix computations (maybe if you made this just an interface to some Racket matrix libs you could work around it). I'm really not going to bother thinking about efficiency much at all. What I have done before in Arc is prototyped some algorithms on very small test sets. Once I got those working and figured out what I actually wanted, I rewrote everything in C++ or C, with a very close eye to efficiency.
I used to use this strategy all the time with an R / C combination, but I greatly prefer writing in Arc. Lush is kind of the best of both worlds, except the last time I pulled the bleeding edge version there appeared to be some crippling, hard-to-find bugs. Plus it's nice to just be able to build quick prototypes when you're already in Arc.