Perl, Python, and Ruby are successively closer approximations of Common Lisp.
I admire the ideas behind Clojure because it makes a different set of design decisions from Common Lisp. This seems to have even gotten the attention of Common Lisp developers, and those guys are pretty hard to impress.
The focus on immutability, literals for data structures other than lists that share a common "seq" interface, first class functions and closures, multi-methods without OO, lazy sequences, and full fledged macros hit an interesting sweet spot that is not touched by any other language I know. Mr. Hickey has managed to put all of those things together in a way that strengthen, reinforce, and complement one another. That is an impressive feat of language design.
My point is that Clojure is not "just" popular. By making unique decisions about the semantics of the language, and not just improving the syntax over the semantics of an existing language, Clojure may very well be one of those languages whose ideas remain influential long after it stops being popular.
(I'm curious, does Arc innovate in terms of its semantics being significantly different than its predecessors? It seems to have the goal of taking existing Lisp paradigms and
making it possible to express them more succinctly. But I might be missing something important.)
So, even by the criteria you put forth, Clojure might be a language worth watching.
1. The time scale. I don't want to make what people think they want right now. Following that recipe earlier would have got me Perl, which lost its lead to the language I would have gotten a little later, Python, which lost its lead to the language I would have gotten a little later, Ruby, which... See the pattern?
2. The audience. "Make something people want" is a recipe for companies. Their goal is to make a lot of money, which means aiming for a wide audience. That shouldn't necessarily be one's goal in every kind of work. It's ok to want to be Jane Austen instead of Perez Hilton, even though Perez Hilton is what most people want.
"But sometimes I need to write a self-contained script that can take part in the whole Unix environment: things like being part of a pipeline, running as a cron job, running through CGI, running as a command. Python makes this trivial; Lisp makes this difficult or impossible."
This is hugely under-estimated as a reason for lack of Lisp adoption. I would put this way ahead of lack of IDEs.
"I don't think there is a better candidate for an IDE target than Eclipse. It is very pretty."
Eclipse is ugly as sin.
Eclipse belongs to that school of UI design that further divides and subdivides the available space into smaller and smaller little tiles and panes until you are squinting at your code through a keyhole and distracted by all the stuff surrounding it. Yes, there are ways around this, but you have to fight the default paradigm. Most other IDEs adopt the same paradigm (to varying degrees).
Also, your whole point about IDEs does not account for the growing popularity of Python and Ruby. I know you point to an online Ruby interpreter, but these languages rose to popularity with most noobies first experiencing them through a text editor + shell/REPL environment. Stop and think about being a noobie downloading Eclipse and firing it up for the first time. What is the first step? How many steps will it take to see a useful result of your labor?
Versus an intro tutorial of:
>>> print "Hello world"
Congratulations! You're now a Python programmer! (Hope I didn't mess that up.)
Sorry for the rant, but I think many IDEs are largely a cargo cult phenomenon. pg has a quote somewhere about finally figuring out what an IDE is for; something like "it generates all the code that your Lisp macros generate for you!" IDEs are largely band aids to make bad languages not hurt quite as much.
I've used Visual Studio, JBuilder, Eclipse, etc., and for the languages I used them for (C++, C#, Java), I felt the IDEs were essential. When I switched to Ruby, the same quality of IDEs weren't available, but I found I was much more productive in Ruby with a simple editor than I had been with other languages plus an IDE, and I actually enjoyed the lighter weight environment quite a bit.
Would I be even more productive in Ruby with an IDE? Possibly. But it's possible that more powerful languages lessen the need for an IDE. Your love of ACL for Common Lisp would be a counter data point, but it's only one data point. Well, Avi Bryant would tout the Squeak IDE as a great advantage in using Smalltalk. In my case, it's somewhat moot given the lack of IDEs for Ruby. As I develop more in Lisp, I'll be better able to judge the effectiveness of the available IDEs.
Tens of thousands of lines of Ruby haven't been a problem for me with just vim, and now Emacs, and I don't think hundreds of thousands of lines would be a problem either if designed well.
"I found I was much more productive in Ruby with a simple editor than I had been with other languages plus an IDE"
Bingo, and that is why we do not see fancy IDEs for agile languages. The corollary being horrid languages have great IDEs because lawdy we need something!
One thing going on here is reflection. If my language has that I can toss off my own rough IDE-ish hacks in minutes and then they do exactly what I want, too, so the demand just never develops for off-the-shelf IDEs.
I work with tens of KLOC of Common Lisp at a time. I like sitting in a backtrace and using a keychord to jump to a functions definition, or sitting in the inspector and jumping to a class definition in source or with a diffeent keychord to the class in a graphical class browser. ACL integrates everything (except, strangely "who calls?", tho that is available via an unexported function), so I just sail all over my big code bases as fast as I can think, click, and keychord. Some folks, btw, feel the same about Lispworks and Slime, we need an IDE smackdown some day to comapre. :)
Excellent question. I think it deserves at least a blog-length answer which I hope to get to soon but first a couple thoughts here.
First, how many comments do you count in my code?
Second, I think my class definitions are pretty concise, as class definitions go.
Third, Paul said somewhere he doesn't use objects but does use hash tables filled with closures. That is precisely what my objects are. It just provides a brief syntax for defining them. If there is a way to define hash tables filled with closures with even greater brevity I'm all for it.
Fourth, Arc has opened the door to data in functional position and I find that a huge win over other Lisps. I think this should be available to those hacking with Arc, not just the language designer. Maybe another way of saying this is give hackers the ability to create new types and define what they do in functional position and defining "setters" that work with =. Objects are one paradigm for doing that but if there's a better, more Arc-ish way that would be awesome, too.
Fifth, I think this can be a pedagogically useful example to someone coming from, say, Python or Ruby to Arc. I believe that it can help bridge the understanding gap between objects on the one hand and first-class-functions + anonymous functions + closures on the other.
Well, that turned out longer than I meant to type right now, but I appreciate hearing others' thoughts.