Arc Forumnew | comments | leaders | submitlogin
Arc's web server in Lua (
7 points by sacado 4323 days ago | 28 comments

6 points by tokipin 4322 days ago | link

i like how Lua is a language that "just works" (within reason of course.) i miss its awesometastic tables when using other languages. it's also rare in that it doesn't promote any particular paradigm. it lets you use what you feel is best. relatedly, its syntax is deceptively simple

anywho, i took a glance at the code. i know you're going for performance so maybe that affected the mechanisms you used, but here's some suggestions anyways:

* Lua's string concatenation operator is .., as in

  return "str1" .. "str2"
* in tight code, local variables should be used anywhere possible. global variables require a table lookup, but local variables are on the stack (or register... one 'o thems.) so for example, you might want to have something like

  local tinsert = table.insert
at the top or above the functions it's used in. it can't exactly JIT the table.insert because table lookup is dynamic (in both the tables you move around and the global environment)

* string.gsub might be faster than iterated replacement

* a key in a table can be anything, even functions and other tables, for the hex2int you can therefore have something like:

  { ["1"] = 1, ["2"] = 2, ... ["a"] = 10 }
i don't remember if the square brackets are necessary there

* as far as ipairs is concerned, a nil value signifies the end of a table, so if you have:

  { 1, 2, nil, 3, 4 }
ipairs will only iterate from 1 to 2. hence, depending on circumstances, you may be able to use this to make table splitting a bit faster

* table.concat might be useful somewheres

* a map-like function where on each iteration you return not just a value, but a key-value pair might be useful. eg pretending you don't care about performance, reverse could be written:

  function reverse( a )
      local len = #a
      return tblmap( a, function( k, v )
          return len-k+1, v
      end )
it also makes mirroring tables easy. (again, ignoring performance. this is something you might use in the user code rather than in the library)

* one way i've set up bi-directional tables before is with __call. in that form, tbl[key] would access one way, and tbl(key) would access it in reverse. another option is something like tbl[1][key]/tbl[-1][key]. depending on how you're using the table, you can memoize in one of the directions and/or set up some sort of little database hidden behind that single table

* Lua doesn't have macros, but it does have first-class environments. among many other things this means it has anaphoric macro-like capabilities, except for the need to pass in expressions through closures or strings. take for example these lines from luarc.lua:

  local sep     = sep or ' '
  local pattern = string.format ('[^%s]+', sep)
  local f       = string.gmatch (s, pattern)
  local match   = f()
using a function seq:

  function seq( ... )
      local that = nil
      local original_env = getfenv( select( 1, ... ) )
      for i = 1, select( '#', ... ) do
          local fn = select( i, ... )
          local new_env = { ["that"] = that }
          -- have new environment inherit from old, otherwise
          -- any global variables used in the fns will be unavailabe
          setmetatable( new_env, { __index = original_env } )
          setfenv( fn, new_env )
          that = fn()
      return that
or a shorter version if we have another function "with":

  function seq( ... )
      local that = nil
      for i = 1, select( '#', ... ) do
          local fn = select( i, ... )
          that = with( { ["that"] = that }, fn )()
      return that
we can write:

  local match = seq(
      function() return sep or " " end,
      function() return string.format( "[^%s]+", that ) end,
      function() return string.gmatch( s, that ) end,
      function() return that() end )
this is an extremely questionable abstraction to say the least (note, none of this was tested,) but it's an example of some of the sorts of things you can do with first-class environments, which might help in betterizing the way the library is used. keep in mind that environments are tables, whose metatables may be set like any other tables, which need not be constrained to easy childish things like inheritance. eg:

      local test_env = {}
      local i = 0
      setmetatable( test_env, { __index = function( tbl, var )
          if var == "var" then
              i = i + 1
              return i
              return _G[var]
      end } )
      setfenv( 0, test_env )

  > print( var )
  > print( var )
  > print( var )
* finally, the horse's mouth is usually the best teacher:

* and why yes, i am bored, thank you for asking (actually, any excuse to play with Lua or Arc is welcome)


3 points by bOR_ 4322 days ago | link

Heh. Lua is pretty nice. Wrote a few mods for warcraft in it. The prefix notation is growing on me though as it appears to be clearer to my mind what is happening when things are written in prefix than infix (took a bit of getting used to though).


3 points by sacado 4321 days ago | link

waow, thanks for the information (and for reviewing my code). I'm not a very proficient Lua developer yet, so I still have to learn a lot. Your advice will help...


1 point by almkglor 4322 days ago | link

> i miss its awesometastic tables when using other languages.

Would you mind expounding on how "awesometastic" they are? I gather that they have a nice implementation of vectors hidden under the table data structures (so conceptually a vector is just a table, but the runtime optimizes away vector-like usage into a true vector). Does it have more awesometastic properties?


4 points by tokipin 4321 days ago | link

i think it's the fact that those concerns don't occur in the language to begin with. from what i glanced (i can't find the pdf anymore,) the interpreter switches between different representations depending on how the particular table is used, and can switch between the representations dynamically for the given table (i'm not sure if the switching is bidirectional)

but the user never sees any of that. if you want an array or tuple you type

  local blah = { 1, "two", function return 3 end }
if you want a hash table you type

  local blah = { bleh = "yep", bleagh = "whatever" }

  -- both of the following are valid
  print( blah["bleh"] )
  print( blah.bleh )
sparse array:

  local blah = { "my", "name", "is", [12] = "slim", [888] = "shady" }

  local blah = { "element 1", "element 2", name = "son", lastname = "goku" }

  local blah = { { "star", "bucks" }, { "apple", "sauce" } }

  local blah = { [{ 1, 3, 5 }] = "odd", [{ 2, 4 }] = "even" }
basically, any sequency, key-valuey, or structured ish thingie, i can just write out without worrying about anything. i'm sure it's doin' some sort 'o magic under the hood, but to that end they could be sacrificing lawn gnomes for all i care

and keep in mind that Lua is likely the fastest interpreted language... or at least it was before the Javascript optimization race started. i think one of the Javascript engines has features inspired by the Lua VM

and then there's metatables, which are properties that can be set to control the behavior of values in different situations. for example, one of the properties is __index, which is referred to when a table doesn't have a value for a given key. this enables straightforward memoization, inheritance, hidden data management, infinite data structures, etc

other metatable properties determine the operation of the value under addition, comparison, etc

you could, for example, implement complex numbers as if they were part of the language, simply by creating a seed table called i. when an expression such as 1 + 2 * i is reached, the __mul property would perform the complex multiplication, then return a new Complex table which would in turn perform the addition. along each step it would be a normal table, so you could do, for example:

  print( (1 + 2 * i).polar_form )
keeping in mind that the table access is dynamic, hence polar_form can be calculated each time it's accessed, no need for getting/setting. also, this:

  print( 8 * i^5 + 2 * i )
would work as it should because there's a __tostring property

i've been thinking of making an APL-like DSL (would be nice to be able to implicitly map and whatnot) in this sort of manner. i haven't looked deep into Ruby but i believe it has things of this nature

here's a description of metatables:

the fact that tables are used everywhere is unoverstatable. environments are tables, therefore you can do powerful things with environments. like writing your own import/include function, reading and writing global state in a file, anaphora and implicits... pretty much a bunch of crazy shit

along with this some very well-chosen features such as coroutines, closures, the implementation of most language features as modules (eg, coroutine.wrap) to keep the syntax clean, etc

the tables by themselves go a long way, especially with their It Just Works® all-purpose syntax. but the way this flexibility is encouraged through the rest of the language makes it a wonderful unified package


2 points by sacado 4323 days ago | link

Hey, I know I haven't been there for a while, and now I'm here to show you some Lua code.

Well, I'm still playing with Arc, and particularly still love its webapp's model, but I'm really fed up with the lack of libraries especially when I want to do real work, so I actually wanted to see if its web server could be easily implemented in another (good) language. Well, it can, even if it's not as elegant as Arc's code.

What I've learnt :

First, macros are really great, I miss them ; Lua has extensions if you want to play with macros, but I didn't use them here

I already wrote about that, now I'm sure : Arc has too many basic datatypes. It has string, symbols and characters, Lua only has strings, and that's clearly enough ; a lot of code in Arc is converting strings, symbols and characters into each other.

Arc has lists and hash tables (and unofficially vectors), Lua only has tables ; again, a lot of code in Arc tries to convert lists as hash tables.


4 points by lojic 4322 days ago | link

Agreed re: libraries. I expect that libraries might start showing up once/if folks feel that pg hasn't simply left entirely as it appears currently. Typical catch-22 - who wants to write libraries for a language that might stagnate or die?


1 point by shader 4322 days ago | link

What killer libraries do y'all think would make arc interesting/useful enough to move beyond novelty?


3 points by almkglor 4322 days ago | link

The boring stuff, like building nice parameterized SQL queries and getting back the data from SQL. Launching a system process in parallel and keeping track of its status (and potentially aborting it if e.g. it takes too long)


1 point by shader 4322 days ago | link

If we do all of the boring stuff in a clean, concise way, that makes everything easy, with the option of adding macros on top to boot, the boring stuff might well become fun, or at the very least, painless.

Which boring thing would you start with?


2 points by tokipin 4322 days ago | link

GUI imo. wxWidgets binding wouldn't be bad


1 point by stefano 4322 days ago | link

Some times ago I started a GTK+ binding, now "paused". It's more boring than I thought initially. If you wish look at it for a starting point (file gtk.arc in Anarki). I now think a binding towards tcl/tk would look nicer and easier to use, though.


1 point by stefano 4322 days ago | link

These would require a standard FFI system. Or else we would end up writing Anarki specific code. Such a fork would be a real Arc killer (in the bad sense of the term).


3 points by almkglor 4322 days ago | link

sacado built an FFI on Anarki.... well forks are generally bad but with PG asleep until october or so .... (maybe he's getting ready for oktoberfest or something ^^)


1 point by stefano 4322 days ago | link

Maybe he is preparing an Arc community summit at the oktoberfest? :)


2 points by bOR_ 4322 days ago | link

If rails is to be considered the killer library of ruby, it took five years for it to show up.. so we have some time left ;)


2 points by stefano 4322 days ago | link

That's true. Programming languages take years to evolve, not months. Moreover, Arc is in the language design phase. As of today, the syntax and the semantics of the language are more important than libraries. For example, I'd like to see something like Kenny Tilton's cells ( nicely integrated within the core language.


2 points by lojic 4321 days ago | link

Is it in the language design phase? Does anyone know if this is in fact true, and if so, what is currently being designed? The impression I have is that it just sort of stopped.


2 points by shader 4321 days ago | link

I would suppose that since a) this language is not used in anything "important" yet, and b) it's open source; yes, it can be in the design phase. I should think that the design phase persists until, for some reason, the language becomes "formalized", or until it is impossible to change anything major without ruining things for a lot of people. At that point you can still "design" the language, but since it has a generally understood "style" and so forth, it won't be able to change too much unless you make a whole new language.

What do you want to be designed? One of the major problems about "designing" a new lisp is that, since anyone can "do it" in their spare time, they don't see the point. Maybe they're right. ^^

Sorry for all of the quotes; it looks kind of funny, I'm sure.


2 points by rntz 4323 days ago | link

On the basic datatypes of arc: you cannot remove symbols or lists from arc. Then it would no longer be a lisp. I suppose you could just have strings be symbols or vice-versa, but then you either put overhead on usage of strings for variable lookup or overhead on usage of symbols for string manipulation.


2 points by sacado 4323 days ago | link

Yes, but, for example, association lists could be implemented as hash tables while beeing seen and manipulated as regular lists. That's tricky, but that should be feasible. And Clojure removed cons cells and thus "traditional" lists, but it's still a Lisp...

As for strings and symbols, I don't really know how they are actually implemented, but as far as I know, the idea is that 'foo and 'foo are the same memory location, while "foo" and "foo" are not necessarily the same object(thus allowing string mutation). Or maybe I'm wrong ?

Lua's strings are implemented the same way : "foo" is always the same location as any other "foo" and string manipulation really doesn't seem to be a problem at all...

I really think "foo" could be just an alternate syntax for 'foo (just as |foo| is) so that we still have a Lisp... Add string manipulation facilities to symbols, and you're done. In any case, characters just seem useless...


3 points by almkglor 4322 days ago | link

> Yes, but, for example, association lists could be implemented as hash tables while beeing seen and manipulated as regular lists.

    (= foo '((key1 . val1) (key2 . val2)))
    (= bar (cons '(key2 . different-val2) foo))
    => ((key2 . different-val2) (key1 . val1) (key2 . val2))

Probably means we want to have some sort of "source" slot too, so that we can display shadowed values. Hmm. The association sublists e.g. '(key1 . val1) can probably be directly shared, but lists also imply an ordered set, so we need to store that info too. Hmm.

>As for strings and symbols, I don't really know how they are actually implemented, but as far as I know, the idea is that 'foo and 'foo are the same memory location, while "foo" and "foo" are not necessarily the same object(thus allowing string mutation)

This is correct. And when you really look at it, changing strings as if they were arrays of characters is hardly ever done in Arc; usually what's done is we just read them off as an array of characters and build a new string.

> In any case, characters just seem useless...

Another lisplike, Skill, uses 1-char length symbols for characters (i.e. no separate character type). Also, many of its string manip functions also accept symbols (although they still all return strings).


1 point by sacado 4322 days ago | link

  >  => ((key2 . different-val2) (key1 . val1) (key2 . val2))
When you don't know why your design's wrong, ask almkglor :)

Ok, well it's probably a little trickier than I thought... Maybe a dual implementation, as you suggest, would work then...


3 points by stefano 4322 days ago | link

Another point to consider is that if your a-list is very small (<= 5 elements) it could be faster than hash tables. The sharing behavior could be achieved with some sort of concatenated hash-tables, a list of tables to consider in turn to find the desired element. This seems very slow though. BTW, removing a-lists would be useless: they're so simple to implement that a lot of developers (me included) would re-invent them to use in their applications.


1 point by almkglor 4322 days ago | link



3 points by gnaritas 4322 days ago | link

My two cents on strings/symbols, there are semantic reasons they should always be separate. It's not about string mutation, ignore the implementation, it isn't relevant.

Two symbols 'foo can always be considered to mean the same thing, two strings "foo" can't, they may be only the same by coincidence. This matters if automated refactoring tools become available because you can safely rename all occurrences of a symbol, the same cannot be said of strings.

Mixing strings and symbols is a bad idea, they are different things and should remain that way.


1 point by sacado 4322 days ago | link

Well, we could at least have a very lightweight bridge between both worlds, by allowing (and automatically converting) symbols into string where strings are needed and vice versa.

Code crashing because, for instance, you tried to concatenate a string and a symbol is rather annoying, but these bugs keep happening and the developer's intention is rather obvious there.


4 points by gnaritas 4322 days ago | link

Oh I don't disagree, I'm actually a Smalltalker and I'm accustomed to Symbol being a subclass of String and a String being a list of Characters. I actually have refactoring tools so I just wanted to point out that they are different for more than just implementation reasons and there's a logical reason to have symbols beyond mere optimization.

I just hang out here because I like to see how a new Lisp is born and how you guys think, even though I don't actually use Arc.