Arc Forumnew | comments | leaders | submitlogin
Even with fexprs, apply for macros is busted
1 point by akkartik 1775 days ago | 62 comments
Wart has fexprs (http://arclanguage.org/item?id=14881). Wart also has a @splice operator that's analogous to apply (http://arclanguage.org/item?id=15154). I intended it to be useable on both functions and macros, but using it with macros brings up subtle issues. For example, here's the usual definition of the and macro:

  mac and args
    if !args
      1
      if !cdr.args
        car.args
        `(if ,car.args
           and ,@cdr.args)
Here's me calling and with @spliced args[1]:

  $ git clone git@github.com:akkartik/wart.git
  $ cd wart
  $ git checkout 8cbfe78d1f
  $ wart
  wart> (def all-true(xs) (and @xs))
  ; Hit <Enter> twice to eval at the repl.
all-true works as you would expect:

  wart> (all-true (list 1 2))
  2
  wart> (all-true (list 1 nil 2))
  nil
Now observe:

  wart> (all-true (list '(1 2)))
  Can't coerce number 1 to function
What's going on? Inlining away all-true, this works:

  wart> = x '(1 2)
  wart> (and x)
But this doesn't:

  wart> (and @list.x)
Basically, @list.x acts like eval.x. This causes subtle bugs with any macro call on @spliced args. Functions don't care, but macros do an eval later (see the first 2 paragraphs of http://arclanguage.org/item?id=15137), and an earlier eval can mess things up as above.

I've had a long-standing broken test that I never knew how to fix: http://github.com/akkartik/wart/blob/8cbfe78d1f/030.test#L14. Now I think I understand what's going on.

Interestingly, apply in Kernel (http://web.cs.wpi.edu/~jshutt/kernel.html, ftp://ftp.cs.wpi.edu/pub/techreports/pdf/05-07.pdf) also requires 'applicative' fexprs that evaluate their arguments. Functions, in other words.

Does this line of reasoning seem reasonable? Am I assigning blame right? I'm bummed that macros have this gaping issue. It's too much for a hundred-year fexpr lisp to live with. I'd _love_ to be told it's just a bug in my implementation somewhere.

Or perhaps there's a better way to build apply?

[1] Needs linux and gcc. Might work on macos, but no guarantees. Darwin has a shitty libc.



2 points by Pauan 1775 days ago | link

Why would splice act like eval? Shouldn't it instead be that fns are fexprs that evaluate their arguments at runtime, and then splice just... splices things? I don't think it's a good idea for splice itself to be doing the eval'in.

-----

2 points by Pauan 1775 days ago | link

By the way, in Kernel, you would first have to wrap the operative. So you'd say this:

  (apply wrap.and list.x)
I guess that'd be this in wart:

  (wrap.and @list.x)

-----

1 point by akkartik 1774 days ago | link

Thanks for the tip.

To answer your earlier question: splice evals because in your apply call above, apply evals all its args before operating on them.

-----

1 point by Pauan 1774 days ago | link

"apply evals all its args before operating on them."

See, that's what I'm talking about. It seems to me that apply shouldn't eval its arguments. Instead, a function would be a fexpr that would eval all its arguments before doing anything else. Something like this:

  (= fn (fexpr (parms . body) env
          (eval (w/uniq (args env)
                  `(fexpr ,args ,env
                     (let ,parms (map (fexpr (x) nil (eval x ,env)) ,args)
                       ,@body)))
                env)))
Please tell me if I'm missing something.

Also, it might be a good idea to make a more general "wrap" function that would turn any fexpr into a function:

  (= wrap (fexpr (x) env
            (let x (eval x env)
              (fexpr args env
                (let args (map (fexpr (x) nil (eval x env)) args)
                  (apply x args env))))))
And then you can define 'fn in terms of it:

  (= fn (fexpr (parms . body) env
          (wrap (eval `(fexpr ,parms nil ,@body) env))))
This is similar to what Kernel does, with a couple changes.

-----

2 points by rocketnia 1773 days ago | link

"Please tell me if I'm missing something."

I think in Kernel, a call to 'apply invokes an applicative with a list of argument values, not expressions it should evaluate. So an applicative isn't exactly an fexpr that calls 'eval on its args, because it doesn't actually eval its args in every instance it's called.

Instead, the arguments to an applicative are evaluated as part of the Kernel core semantics. When applying a value to a list, if the value is an applicative, it's unwrapped, and the value inside is applied to the result of mapping eval over the list.

That said, I think this could be approximated (with abstraction leaks) in a system that uses your approach. Just have 'apply wrap up all the arguments as literal expressions, so that they evaluate to the original values. If you want to know what abstraction leaks I'm talking about, consider what happens if a language user modifies/shadows 'eval, modifies/shadows 'quote (interfering with the meaning of literal expressions), or calls 'apply with an fexpr that does something with its args other than immediately evaluating them (e.g. most "macros").

-----

1 point by Pauan 1773 days ago | link

"if a language user modifies/shadows 'eval, modifies/shadows 'quote (interfering with the meaning of literal expressions)"

It could use the original definition of those, if that were wanted.

This is actually consistent with Kernel's semantics, since it's not possible to overwrite a built-in variable, you can only shadow it. And because Kernel uses lexical scoping, it should use the binding in the original environment. So you'd get this behavior for free.

Of course, we're talking about wart here, so wart can do whatever the hell it wants to do. I merely mention this to point out that (as far as I know) that's not a problem in Kernel, which also has fexprs. So it shouldn't be a problem in wart either, assuming its implementation of lexical/dynamic variables is solid.

---

"calls 'apply with an fexpr that does something with its args other than immediately evaluating them (e.g. most "macros")."

That was kinda akkartik's point: that apply doesn't work properly on macros. So I was already assuming that akkartik wanted apply to work on macros. Obviously if you don't want that then the Kernel approach is fine.

-----

2 points by rocketnia 1773 days ago | link

"Of course, we're talking about wart here, so wart can do whatever the hell it wants to do."

That's why I brought up the case. If a language is designed for a sufficiently smart programmer (as is the case with Arc and Wart), the language designer don't have to worry about ensuring things like "eval always does X"; the programmer will leave these invariants alone if they're really that useful, and will break them if breaking them is even more useful. In fact, I think Arc sacrifices hygiene in favor of implementation simplicity in order to be more intuitive when broken.

Intuitive even after you break it. That could be an interesting take on the Arc philosophy. ^_^

But yeah, Wart could still do this in a way where all the abstraction leaks behave exactly as akkartik expects them to (whether that's hygienically or not).

---

"That was kinda akkartik's point: that apply doesn't work properly on macros."

Right, I meant to allude to a bit more, which I was about to elaborate on. I'll just elaborate here, then. :)

If 'apply wraps up its args as literal expressions, then (apply and foo) does work as everyone keeps trying to say it should work. :-p That is, it works just like (all idfn foo), so I don't see why anyone would even care to use it.

Meanwhile, (apply def foo), which to me doesn't make sense in the first place, may also "work," successfully giving an error since a literal expression isn't an assignable identifier.

I consider these to be inelegant consequences of the implementation approach, rather than useful behavior. But if akkartik sees it differently, I'd like to make sure the other inelegant parts (the potential lack of hygiene) don't come as a surprise. ^_^

Come to think of it, the other implementation details could still be implementation details even if macro application isn't. Kernel's 'apply doesn't work on operatives, but it could be respecified so that it does work on operatives by mapping ($lambda (x) (list $quote x)) on the argument list.

-----

2 points by Pauan 1773 days ago | link

"but it could be respecified so that it does work on operatives by mapping ($lambda (x) (list $quote x)) on the argument list."

But why would apply need to map quote on the argument list? If it simply doesn't evaluate the individual sub-expressions, then it should be the same thing. My current view for an ideal Lisp is that all evaluation should be done explicitly, in fexprs. No implicit eval by apply or the runtime or whatever.

That means that apply would pass the arguments unevaluated to the fexpr, which could then choose to eval them or not. And since a function is defined as a fexpr that evals all its operands first, the semantics are preserved for functions too.

Basically, I think fexprs should be the default (using eval when needed), rather than functions being the default (using quote to suppress evaluation). In other words, not evaluating should be the default.

---

"Meanwhile, (apply def foo), which to me doesn't make sense in the first place"

Why not? It seems to me that...

  (apply def '(foo (a b c) (+ a b c)))
...should be exactly equivalent to...

  (def foo (a b c) (+ a b c))
...except maybe that apply would evaluate the fexpr call in a fresh environment. Depends on how wart wants to handle environments.

-----

3 points by rocketnia 1773 days ago | link

I'm assuming akkartik's...

  (def all-true(xs) (and @xs))
...can be alternately implemented like so:

  (def all-true(xs) (apply and xs))
So when akkartik calls (all-true (list '(1 2))) and gets disappointed, it's pretty much equivalent to...

  (apply and '((1 2)))
...which by your proposal would do this:

  (and (1 2))
This results in exactly the behavior akkartik didn't expect.

If 'apply literal-wrapped the args, it would work more like this:

  (and '(1 2))
Meanwhile, with your proposal, the use of 'apply with functions would break: (apply list '((1 2))) would behave like (list (1 2)).

-----

2 points by akkartik 1773 days ago | link

Hmm, I don't follow how your comment connects with Pauan's. It feels like a non sequitur.

"Meanwhile, with your proposal, the use of 'apply with functions would break: (apply list '((1 2))) would behave like (list (1 2))."

I don't get this either.

-----

2 points by rocketnia 1772 days ago | link

Even though Pauan understood me, I'll reply again in case it helps. ^_^

---

"But why would apply need to map quote on the argument list? If it simply doesn't evaluate the individual sub-expressions, then it should be the same thing."

Since you (Pauan) would have procedures be fexprs that explicitly eval'd their arguments, using apply on a procedure would cause the procedure to eval the args. Whereas (apply list '((1 2))) results in the list ((1 2)) in Arc, in this hypothetical language, 'list itself would try to eval (1 2) and get an error.

In other words, the Scheme/Arc/Kernel 'apply takes a list of argument values, not expressions, so (from a certain point of view) they're not actually suitable arguments for an fexpr call.

If we want Scheme/Arc/Kernel-like behavior, 'apply can make up for the procedure's own evals by quoting the args, or it can make up for them the way it does in Kernel, by unwrapping an applicative into something that skips the eval-my-args step altogether.

Personally, I think unwrapping is cleaner, but quoting makes (apply and foo) behave like (all idfn foo), which is what people seem to expect. Unfortunately, neither of these approaches does what you want with 'def.

---

"It seems to me that (apply def '(foo (a b c) (+ a b c))) should be exactly equivalent to (def foo (a b c) (+ a b c))."

Pauan, the step-by-step way you comprehended my response to this is great, but I wasn't reasoning in that much detail. :-p I was mostly going by analogy:

  (apply def '(foo (a b c) (+ a b c)))
  (def foo (a b c) (+ a b c))
  *** success ***
  
  (apply list '(foo (a b c) (+ a b c)))
  (list foo (a b c) (+ a b c))
  *** error: unbound variable 'foo ***
  
  (apply list '((1 2)))
  (list (1 2))
  *** error: can't call 1 ***
  
  (all-true (list '(1 2)))
  (apply and '((1 2)))
  (and (1 2))
  *** error: can't call 1 ***

-----

1 point by Pauan 1772 days ago | link

"Personally, I think unwrapping is cleaner"

I agree. I rather like that part of Kernel.

---

"Unfortunately, neither of these approaches does what you want with 'def."

Mind explaining? It seems the unwrap technique would work just fine, right? I'm assuming that passing a fexpr to unwrap would just return the fexpr.

-----

1 point by akkartik 1772 days ago | link

Thanks both of you for your responses. I guess I don't understand everywhere that you're disagreeing.

For example your analogy for def sounds exactly like what Pauan is saying. What am I missing?

"Meanwhile, with your proposal, the use of 'apply with functions would break: (apply list '((1 2))) would behave like (list (1 2))."

But that's how it should behave, right? Is that not what Pauan is suggesting as well?

-----

1 point by Pauan 1772 days ago | link

"I guess I don't understand everywhere that you're disagreeing."

The only area I currently disagree with rocketnia is this: "Unfortunately, neither of these approaches does what you want with 'def."

---

"For example your analogy for def sounds exactly like what Pauan is saying. What am I missing?"

Yes, it is exactly like what I was saying. The point was that if you changed apply so it worked with def (and other fexprs) it would then break functions like list. Finding a way so that apply can work on both fexprs and functions is one of the points of unwrapping functions.

---

"But that's how it should behave, right? Is that not what Pauan is suggesting as well?"

Absolutely not. It should behave like (list '(1 2)). That is how it would behave if apply unwrapped functions, or if apply mapped quote over the arguments.

Of course, one could make the argument that you should have to say (apply list '('(1 2))) instead... but in that case I'd rather have two separate operators: one that works on fexprs, and one that works on functions.

Or perhaps you could just use eval for fexprs: (eval '(def foo (a b c) (+ a b c))) That seems perfectly reasonable to me as well. But that's equivalent to saying apply should only work on functions. And in that case, why do you need apply at all, since eval works on both fexprs and functions...?

-----

1 point by akkartik 1772 days ago | link

> > But that's how it should behave, right? Is that not what Pauan is suggesting as well?

> Absolutely not. It should behave like (list '(1 2)).

sbcl:

  * (apply 'list '((1 2)))
  ((1 2))
  * (list '(1 2))
  ((1 2))
racket:

  > (apply list '((1 2)))
  '((1 2))
  > (list '(1 2))
  '((1 2))
So I'm still confused. Are you saying this should work even if list was a macro/fexpr? (If so, I totally agree.)

-----

1 point by Pauan 1772 days ago | link

"(apply list '((1 2))) -> '((1 2))"

Yes, that is the expected result.

---

"Are you saying this should work even if list was a macro/fexpr?"

Yes. But as soon as you try to make it work with fexprs you'll run into the following problem. Here's what I wrote earlier:

"To put it another way, (apply list '((1 2))) would be equivalent to (eval (cons list '((1 2)))) which is equivalent to (eval (list list '(1 2))) which is equivalent to (eval '(list (1 2))) which is equivalent to (list (1 2))."

What I'm saying is that if you changed apply so it worked with fexprs, then (apply list '((1 2))) would be equivalent to (eval '(list (1 2))). If you try evaluating that, you'll get an error saying something like "1 isn't a valid function call" or whatever.

That's because functions evaluate their arguments. So when the "list" function evaluates the argument "(1 2)" it tries to call the "1" function with the argument "2" which obviously fails.

So instead you have to call (eval '(list '(1 2))) to suppress evaluation. Note the extra quote! So what rocketnia said (and I now agree with) is that the "apply" function would have to map quote over each of its arguments. In other words, (apply list '((1 2))) would basically be the same as (eval '(list '(1 2)))

Alternatively, if functions were "wrapped" fexprs, then apply could "unwrap" the function, which would lead to more-or-less the same effect, sans a few details.

In other words, you'd either define "apply" like this...

  (def apply (f args env)
    (eval (cons f (map (fn (x) (list quote x)) args)) env))
...or like this...

  (def apply (f args env)
    (eval (cons (unwrap f) args) env))
In fact, the above is the definition Kernel uses[1]. But if you're implementing apply with eval, then you might be better off not implementing apply at all and just using eval directly. In other words, rather than saying (apply foo '(bar qux)) you'd say (eval `(,foo bar qux)) And rather than saying (apply foo bar) you'd say (eval:cons foo bar)

This should work with both fexprs and functions in a (hopefully) intuitive way.

---

Oh, yeah, you might also decide to keep apply, but have it only work on functions. Then you'd use eval to "apply" fexprs. After looking at the issue, I'm not really opposed to that. It seems reasonable, provided that supplying arguments as a list is most common with functions. The lack of consistency does bug me a little, though.

---

* [1]: If Kernel already uses that definition, why does apply supposedly only work with functions? I believe (without actually checking) it's because the unwrap function throws an error if the argument isn't a function. So you can't call unwrap on fexprs.

If I understand correctly, if wart allowed for calling unwrap on fexprs (it would just return the fexpr) then the above definition of apply should work just fine for both functions and fexprs.

-----

1 point by akkartik 1772 days ago | link

(haven't fully digested your response yet..)

"you might also decide to keep apply, but have it only work on functions. Then you'd use eval to "apply" fexprs. After looking at the issue, I'm not really opposed to that."

You know, I was tempted to define all-true as a macro:

  mac all-true(xs)
    `(and ,@xs)
This way I would get to keep a simple implementation. The price seems too high for my taste, however. Splice can cause macros to spread through a codebase like a virus. Macros are awesome to have, but I want to be careful not to overuse them.

-----

2 points by Pauan 1768 days ago | link

I'll do my best to explain. You want apply to work on fexprs/macros, correct? If so, then the following equivalence must hold:

  (apply foo '(a b c)) ->
  (foo a b c)
You can achieve that with the following definition of apply:

  (def apply (f args)
    (eval (cons f args)))
Let's see if it works:

  (apply def '(foo (a b c) (+ a b c))) ->
  (eval (cons def '(foo (a b c) (+ a b c)))) ->
  (eval (list def 'foo '(a b c) '(+ a b c))) ->
  (eval `(,def foo (a b c) (+ a b c))) ->
  (def foo (a b c) (+ a b c))
As you can see, the equivalence holds. Great! But now let's try using the above definition of apply on the list function:

  (apply list '(1 (2 3))) ->
  (eval (cons list '(1 (2 3)))) ->
  (eval (list list '1 '(2 3))) ->
  (eval `(,list 1 (2 3))) ->
  (list 1 (2 3))
Oops! When the list function tries to evaluate the argument (2 3) it throws an error. To fix this error, we first need to understand that the equivalence for functions is different than the equivalence for fexprs:

  (apply foo '(a b c)) ->
  (foo 'a 'b 'c)
Notice how each argument got wrapped in (quote ...). That's because fexprs don't evaluate their arguments, but functions do. So you need the arguments to be quoted to suppress evaluation.

Let's redefine apply so it maps quote over the arguments:

  (def apply (f args)
    (eval (cons f (map (fn (x) (list quote x)) args))))
Now let's try the steps again:

  (apply list '(1 (2 3))) ->
  (eval (cons list '('1 '(2 3)))) ->
  (eval (list list ''1 ''(2 3))) ->
  (eval `(,list '1 '(2 3))) ->
  (list '1 '(2 3))
It works! But now it breaks on fexprs, because...

  (apply def '(foo (a b c) (+ a b c))) ->
  (eval (cons def '('foo '(a b c) '(+ a b c)))) ->
  (eval (list def ''foo ''(a b c) ''(+ a b c))) ->
  (eval `(,def 'foo '(a b c) '(+ a b c))) ->
  (def 'foo '(a b c) '(+ a b c))
Oops. So it seems at first glance that apply cannot work on both functions and fexprs in a useful way... but wait a minute... let's take a look at Kernel to see how it defines apply.

Before we can define apply in Kernel, we need to understand something. This is one of the central tenets of Kernel, and it's contrary to essentially every other Lisp, including wart.

Almost every Lisp uses something called "implicit evaluation", that is... there is this assumption that things are automatically evaluated, and you need to use quote to suppress this automatic evaluation.

Kernel is different, however. In Kernel, argument evaluation doesn't happen automatically. You have to make it happen by calling eval. This is called "explicit evaluation". In fact, Kernel embraces this so far that it doesn't even provide quote or quasiquote[1].

So in Kernel, all combiners (functions/fexprs) are actually fexprs underneath. All of them. And those fexprs then have to explicitly eval their arguments if they want them evaluated. So, conceptually, a function is just a fexpr that happens to evaluate all of its arguments before doing anything else. apply and eval don't evaluate the arguments, the functions do[2].

Well, except for one thing. In Kernel, functions are actually a separate type from fexprs. So Kernel provides three base primitives for dealing with combiners: $vau, wrap, and unwrap:

  $vau constructs fexprs

  wrap takes a combiner (function/fexpr) and returns a function that evaluates
  all of its arguments before passing them to the combiner

  unwrap takes a function and returns the underlying combiner (function/fexpr)
Okay, so let's get back to apply. Using Arc syntax, here is how apply is defined in Kernel, excluding environments:

  (def apply (f args)
    (eval (cons (unwrap f) args)))
That is, it's exactly the same as the first definition of apply, except that it unwraps the function before evaluating it. To see how this works, we need to understand that the following equivalence holds:

  ((unwrap foo) a b c) ->
  (foo 'a 'b 'c)
This works because wrap creates a function wrapper that induces argument evaluation, and unwrap returns the wrapped value, which doesn't induce evaluation[3].

In most cases, the unwrapped value will be a fexpr. That is, (unwrap list) returns a fexpr that is exactly like the list function, except it doesn't evaluate its arguments[3].

That's why the above equivalence is true: (unwrap foo) returns a combiner that doesn't evaluate its arguments[3], whereas foo is a combiner that does. So unwrapping foo equates to not evaluating the arguments, which is equivalent to mapping quote over the arguments.

Going back to the list function:

  (apply list '(1 (2 3))) ->
  (eval (cons (unwrap list) '(1 (2 3)))) ->
  (eval (list (unwrap list) '1 '(2 3))) ->
  (eval `(,(unwrap list) 1 (2 3))) ->
  ((unwrap list) 1 (2 3)) ->
  (list '1 '(2 3))
It works! But what about fexprs...? Well, in Kernel, unwrap is defined to only work on functions. That is, calling unwrap on a fexpr throws an error. That's why apply doesn't work on fexprs. However, you can still use apply on fexprs... provided you wrap them first:

  (apply (wrap def) '(foo (a b c) (+ a b c)))
This works because Kernel explicitly states that[4]:

  (unwrap (wrap f)) is eq to f
And using the above definition of apply, you first wrap the fexpr def, which is then immediately unwrapped by apply... which thus returns the fexpr def. Essentially, wrapping it is a noop. I think this method (which Kernel uses) is ideal[5].

But in wart, let's assume that you want apply to work directly on fexprs, without needing to wrap them first. In that case, wart can say that:

  (unwrap f) is eq to f, when f is a fexpr
And so...

  (apply def '(foo (a b c) (+ a b c))) ->
  (eval (cons (unwrap def) '(foo (a b c) (+ a b c)))) ->
  (eval (list (unwrap def) 'foo '(a b c) '(+ a b c))) ->
  (eval `(,(unwrap def) foo (a b c) (+ a b c))) ->
  (def foo (a b c) (+ a b c))
Thus, as you can see, by using unwrap, apply works correctly on both functions and fexprs.

---

* [1]: It is trivial to define quote as a fexpr, but Kernel doesn't do that by default, and it intentionally encourages you not to do that.

* [2]: Of course, this doesn't mean that it needs to be implemented that way... you could have wrap return a specially tagged value, and then hardcode eval so when it sees that value in the car, it unwraps it and evaluates the arguments.

* [3]: Actually, because it's legal to call wrap on a function (which returns a function that evaluates its arguments twice), unwrapping it may return a function rather than a fexpr. I'm assuming for the sake of this post that that situation is quite rare, and that unwrap always returns fexprs.

* [4]: The "Revised-1 Report on the Kernel Programming Language" actually doesn't say this. But it's mentioned in John Shutt's dissertation, "Fexprs as the basis of Lisp function application".

* [5]: Why do I think it's ideal? I'm not actually sure. It just feels "right" somehow.

-----

2 points by akkartik 1768 days ago | link

That was incredibly clear, thanks. I read it in one sitting.

Ok, I'm going to go build wrap and unwrap in wart. It should be straightforward.

-----

2 points by akkartik 1767 days ago | link

Wrap/unwrap is a little more complicated in wart than in Kernel because wart allows some parameters to be eval'd and some not. Hmm, is that too much trouble..?

-----

3 points by rocketnia 1767 days ago | link

That's a good point. I don't know if it's really worthwhile.

If we went with a system that had partial wrappers, we'd run across the foreboding corner case of a wrapper that had no effect at all. We might wrap anything in this wrapper an arbitrary number of times and not notice until we start peeling off layers again. But that actually brings us a kind of consistency: We can decree as part of the language design that everything eventually unwraps into an infinite regress of boring wrappers. This gives us a great excuse for 'apply to work on fexprs.

  (fn ('a b c . 'd) ...)
  ==>
  (wrap '(t nil nil . t)
    (fexpr (a b c . d) ...))
...I meant to explore some other options here too, but you're probably thinking at least as deeply as I am about them, and none is as remarkable as that. (I'm not saying the partial wrapper infinite regress is better, just more remarkable, lol.)

-----

1 point by akkartik 1767 days ago | link

That's a cute idea :)

I don't think I'm grasping this nearly as well as you and Pauan. Watch me flail around here (and a few parent commits as well): http://github.com/akkartik/wart/commit/71d9962fbf

---

Will wrap/unwrap play well with macros? Macros are fexprs whose output is implicitly eval'd in the env of the caller. Won't that extra eval throw a spanner in the works?

---

Here are two test examples we've been using a lot in this thread (phrased in wart terms):

  and @(list '(1 2 3))

  def @'(foo(a) a)
Both and and def are macros and so 'unwrapped combiners'. And yet only one or the other works depending on whether or not @ splices in quoted expressions.

(I've been messing with building wrap and unwrap, but nothing to report yet..)

-----

1 point by Pauan 1766 days ago | link

"Will wrap/unwrap play well with macros? Macros are fexprs whose output is implicitly eval'd in the env of the caller. Won't that extra eval throw a spanner in the works?"

I'm assuming macros in wart are just extra sugar for fexprs, and in that case it shouldn't cause any problems if eval doesn't evaluate the arguments of fexprs (which it obviously shouldn't).

To clarify, let's go through the process one step at a time. I'm going to be using this macro:

  (mac let (var expr . body)
    `((fn (,var) ,@body) ,expr))
I'm assuming it's equivalent to this fexpr:

  (fexpr (var expr . body) env
    (eval `((fn (,var) ,@body) ,expr) env))
Now then. Let's try (apply let '(a 5 a)) which should be equivalent to (let a 5 a):

  (apply let '(a 5 a)) ->
  (eval (cons (unwrap let) '(a 5 a))) ->
  (eval (list (unwrap let) 'a '5 'a)) ->
  (eval `(,(unwrap let) a 5 a)) ->
  ((unwrap let) a 5 a) ->
  (let a 5 a)
Okay, so the apply works (as per my previous post on this subject) but you're concerned about the eval. So let's expand out the call to let:

  (eval '((fn (a) a) 5) ...)
Notice how the argument passed to eval is quoted because the macro that generated it used quasiquote. Now this quoted expression is evaluated in the dynamic environment, thus giving us the result we want. Are you running into any problems with this approach?

-----

1 point by akkartik 1766 days ago | link

I think it boils down to this: my eval doesn't take an environment. I'm sure that's a bad idea -- I just want to understand why before I change it.

I don't understand where that environment is coming from in the fexpr.

  (let x 3 (mac foo (a) ...))

  (let x 4 (foo x))
What would env be when the call to mac is translated to an fexpr in your scheme?

-----

1 point by Pauan 1766 days ago | link

"my eval doesn't take an environment."

Why would you do that, and how does wart handle macros, then?

---

"What would env be when the call to mac is translated to an fexpr in your scheme?"

It would be the environment where the macro is called. That is, in the dynamic binding of the macro call, x would be 4.

Using the environment of the place where the macro is defined would be static scope... which I assume wart already has, so that's redundant and pointless because you can just do this to evaluate the expression in the static scope of the fexpr:

  (let env (current-environment)
    (fexpr () nil
      (eval ... env)))
Using this scheme (which Kernel uses) allows fexprs to evaluate things in the static environment, or the dynamic environment, or both, or neither. It gives the maximum amount of power and flexibility.

-----

1 point by akkartik 1767 days ago | link

Here's my first stab at unwrap:

  def unwrap(f)
    ret newf copy.f
      zap quotify rep.newf!sig

  def quotify(sig)
    (if
      !sig
        nil
      ~cons?.sig
        (cons quote sig)
      quote?:car.sig
        sig
      :else
        (cons quotify:car.sig
              quotify:cdr.sig))
rep.f!sig provides read/write access to the signature of function f.

Of course this doesn't work on:

  unwrap.and @(list '(1 2 3))
Since unwrap doesn't change and at all..

-----

1 point by Pauan 1766 days ago | link

I'm very concerned by your approach. It seems quite hackish to me, compared to the elegance of wrap/unwrap in Kernel.

When I was talking about unwrap, I was assuming that all functions are created with wrap, so all of them can be easily unwrapped, rather than unwrap generating a new fexpr. In other words, I was expecting this:

  (unwrap (wrap f)) is eq to f
---

"Since unwrap doesn't change and at all.."

Yes, that's good. Calling unwrap on a fexpr should just return the fexpr itself, assuming you want them to work on apply without wrapping them first.

-----

1 point by akkartik 1766 days ago | link

"I'm very concerned by your approach. It seems quite hackish to me, compared to the elegance of wrap/unwrap in Kernel."

I'm not surprised -- I didn't even feel comfortable committing it to my repo :)

-----

1 point by akkartik 1766 days ago | link

"(unwrap (wrap f)) is eq to f"

Does it matter that it's eq and not equal? (Wart has no is, only iso, the premise being that it mostly doesn't matter. Hence my interest.)

-----

1 point by Pauan 1766 days ago | link

I think it still matters insofar as it suggests a very simple implementation[1], rather than the hacky one you came up with. Basically, I'm saying functions should just be annotated fexprs, and then unwrap can just return the rep of the function.

---

* [1]: It obviously also matters in a Lisp with eq. It could also matter if you ever do decide to add in an eq-like thing to wart, even if you don't necessarily call it eq.

-----

1 point by akkartik 1766 days ago | link

"It obviously also matters in a Lisp with eq."

Wart already has eq if you want it:

  iso addr.a addr.b
(http://arclanguage.org/item?id=15154)

I wouldn't consider "I don't have eq, so do without it." to be a good reason anyway. So the reason my unwrap copies its argument is that it's not at all obvious to me that it matters in a lisp with eq.

---

When you repeatedly say 'hacky' you shed more heat than light[1]. I'm not sure what the flaw is that you see, and whether it's the same flaws that make me embarrassed about it.

For example, the fact that it generates a new copy each time, I'm not sure that's a bad thing.

The implementation relies on the representation of functions in wart, but that's because it's the implementation[2], and implementations can be changed. The conception doesn't seem compromised to my imperfect understanding: unwrap.f is just like f except that evaluation of args is disabled. My concern is really: is this implementation behaving as unwrap should? Are there corner cases where behavior diverges from the desired?

[1] I personally find eq to be quite 'hacky' in the way it requires knowing about how atoms are interned. But again, that's neither here nor there.

[2] It's not as bad as the sb-bquote nonsense a macro body expands to in sbcl.

-----

1 point by Pauan 1766 days ago | link

"For example, the fact that it generates a new copy each time, I'm not sure that's a bad thing."

From a practical standpoint[1], unwrap is used in apply, which means you end up copying a function and running your quotify function over it every single time you use apply. My method should be a lot faster[2], with no additional cost in flexibility or power. It should also be much shorter and easier to implement as well. In fact, I'll define wrap and unwrap right now, using Arc syntax[3]:

  (def must-combiner (x)
    (if (in type.x 'fn 'fexpr)
        x
        (err "expected combiner but got" x)))

  (def wrap (x)
    (annotate 'fn must-combiner.x))

  (def unwrap (x)
    (rep must-combiner.x))
Do you think your implementation is equal to or superior to that, especially given that you only showed unwrap and not wrap in your post?

---

* [1]: I still partially disagree with you even on theoretical grounds. I think that defining (unwrap (wrap f)) to be eq to f is a good equivalence that could prove useful in practice, or at least make it easier to reason about programs because it's so simple and intuitive (based on the names "wrap" and "unwrap").

However, if you've defined iso so it works on functions (that is, functions with the same arguments and body are equal), then I'm fine with that and don't care so much about eq, since iso is the default in wart.

* [2]: Especially since wart is written in C++, so you should be able to make things like the fn wrapper very fast (to type check and extract the fexpr) and memory efficient.

* [3]: I'm assuming that annotating something with the same type will wrap it a second time, which Arc doesn't currently do.

I'm also assuming eval has been changed so it understands things with type 'fn. This isn't a problem in Kernel because eval, wrap, and unwrap are all primitives. You could do the same in wart, or extend eval, or whatever.

-----

2 points by akkartik 1766 days ago | link

I think I'm getting on your nerves again, so I'm going to go away after this.

If I cared about a superior or more efficient implementation I'd go use racket or kernel. I build things to better understand them.

My purpose in showing my unwrap was to show that I still don't understand your suggested implementation of apply. Not to get bogged down in considerations of performance. This is wart, where everything is in slow-mo. It can take a little more consing :)

Perhaps I'm getting stuck in a local optimum here. Now that I've slummed it out with an implicit-env eval for a while, perhaps I'll appreciate the benefits of an explicit env far faster if I just switch to it.

Anyways, thanks for the hand-holding. I should figure out the rest on my own.

---

And oh, I didn't show wrap because it's impossible in a language that allows some params to be eval'd (http://arclanguage.org/item?id=15813). There's no nice way to guarantee that:

  iso f wrap:unwrap.x
(Though unwrap:wrap.x can be guaranteed.)

-----

2 points by Pauan 1766 days ago | link

"I think I'm getting on your nerves again"

Only a little. Although I think ignoring performance is an overall good thing (premature optimization and all that), I think it's still important to choose the best option (in terms of benefit-cost) that's currently available. If somebody gives you an implementation that's better in every way with no cost, why would you refuse it...?

---

"I still don't understand your suggested implementation of apply."

I think the trick to it is to stop thinking of things being automatically evaluated and then using quote to suppress it. If you see not evaluating (fexprs) as the default, and then evaluating (functions) built on top of it... it all clicks together. At least, it did for me.

-----

1 point by akkartik 1766 days ago | link

Kernel is pretty clear and elegant in isolation. But can you get traditional (non-phase-separated) macros that way? Does giving eval an explicit env eliminate the need for the implicit eval in caller scope (http://www.arclanguage.org/item?id=15137)? Does the kernel doc address this question?

-----

2 points by rocketnia 1765 days ago | link

"But can you get ... macros that way?"

A non-phase-separated macro implemented as a Kernel-style fexpr is simply of the form ($vau <args> env (eval <body> env)), possibly with red tape so that env isn't in scope during the body.

I don't have answers to the other questions....

By the way, I've still never really understood wart's scope setup (or maybe I have and I said to myself "nah, that can't be it" :-p ). While I'm suspicious of it, I'd rather not just condemn it by default. When I talk about how things would work in Kernel, it's in the hopes that somehow it could translate over, and maybe that somehow you yourself would discover why your approach was better or worse.

-----

1 point by Pauan 1765 days ago | link

"A non-phase-separated macro implemented as a Kernel-style fexpr is simply of the form ($vau <args> env (eval <body> env)), possibly with red tape so that env isn't in scope during the body."

Exactly. Macros are SUPER trivial to add on top of fexprs, but the reverse is not true. In fact, I'll go ahead and define $mac right now:

  ($define! $mac
    ($vau (name parms . body) env
      (let ((u (gensym)))
        (eval (list $define! name
                (list $vau parms u
                  (list eval (list $sequence body) u)))
              env))))
Warning: I haven't tested the above and I don't think it'll work because I'm pretty sure Kernel doesn't define a gensym function. But assuming a suitable implementation of gensym, it's simple and straightforward enough that I would expect it to work.

And now you can use it to write macros in Kernel (I don't think that's a good idea, though[1]...):

  ($mac $let (var expr . body)
    (list (list* $lambda (list var) body) expr))
Which could be made more readable with quasiquote (which Kernel doesn't have, for good reason!):

  ($mac $let (var expr . body)
    `((,$lambda (,var) ,@body) ,expr))
---

* [1]: In Kernel, you sometimes find fexprs of the form ($vau ... env (eval ... env)) which is basically what the above $mac form does. To avoid the extra redundancy (boilerplate) of the call to eval, you might want to define a $mac form, but it should be anonymous, as in ($mac (a b c) ...):

  ($define! $mac
    ($vau (parms . body) env
      (let ((u (gensym)))
        (eval (list $vau parms u
                (list eval (list $sequence body) u))
              env))))
That way you can use it with $define!, $set!, $let, etc.:

  ($define! $foo ($mac ...))
  ($set! env $foo ($mac ...))
  ($let (($foo ($mac ...))) ...)
In addition, $vau should still be the default way users deal with fexprs. This $mac form is only syntactic shortening for a common use case, but should not replace fexprs.

To clarify my position, you shouldn't have to think in terms of macros, you shouldn't have to try to change the problem to accommodate the way macros work. If it's natural to solve the problem with fexprs, then do so! If it's natural to solve the problem with macros, then do so! Don't try to shove a square peg into a round hole.

-----

2 points by akkartik 1765 days ago | link

Ah, I found this thread involving Kernel's creator about fexprs vs macros:

http://lambda-the-ultimate.org/node/3640#comment-51665

All the differences they discuss revolve around phase separation and the need to perform macroexpansion at call sites. So there's implicitly no other reason it can't be done.

Thanks a lot, guys! I have some rewriting to do in wart :)

---

You said Kernel doesn't have quasiquote for good reason. Can you elaborate? Ah, I googled around.. and found your comment earlier in this thread: http://arclanguage.org/item?id=15691 Lol, I'd missed footnote 1.

-----

2 points by Pauan 1765 days ago | link

"You said Kernel doesn't have quasiquote for good reason. Can you elaborate?"

tl;dr: quote encourages you to think in terms of implicit eval, quasiquote makes it way too easy to break hygiene accidentally, and macros are indirect and differ significantly from functions, thus making certain programs harder to write/understand/debug/etc.

---

Like I said earlier, the trick to understanding it is to think in terms of fexprs evaling things, rather than using quote to suppress evaluation. The simplest way to do that is to just remove quote entirely from the language, thus deeply encouraging you to think in terms of fexprs.

Another problem is that if you encourage quasiquote, then people are more likely to use it to create symbols in fexprs, but that then introduces all the hygiene problems of Arc's macro system. Let me give an example. Let's suppose that you defined $mac in Kernel, and also quasiquote. Let's define an Arc-style $let fexpr:

  ($mac $let (var expr . body)
    `(($lambda (,var) ,@body) ,expr))
So far so good, except... we're injecting the symbol $lambda rather than the fexpr $lambda, so this will break:

  ($let $lambda 5
    ($let ...))
Because the second $let uses the local definition of $lambda, which is currently bound to 5. Hygiene violation! Kernel avoids this by simply not providing quote/quasiquote at all[1]. Instead, you explicitly call list and list* to build up lists:

  ($define! $let
    ($vau (var expr . body) env
      (eval (list (list* $lambda (list var) body) expr)
            env)))
This works because it's inserting the value of $lambda rather than the symbol, so it's inherently hygienic due to static scope. The problem with quasiquote is that it makes it really easy to break hygiene by accident[2], and Kernel's 3rd guideline is:

"Dangerous computation behaviors (e.g., hygiene violations), while permitted on general principle, should be difficult to program by accident"

As for macros... their problem is that trying to specify an expression that when evaluated will give the desired result is usually more complicated, more difficult to understand, and thus more error-prone because it's indirect. For instance, compare these two definitions of $or?:

  ;; implicit eval
  ($define! $or?
    ($mac (x y)
      ($let ((temp (gensym)))
        `(,$let ((,temp ,x))
           (,$if ,temp ,temp ,y)))))

  ;; explicit eval
  ($define! $or?
    ($vau (x y) env
      ($let ((temp (eval x env)))
        ($if temp temp (eval y env)))))
In addition, the model for evaluation of a macro is waaay different than the model for functions, but fexprs and functions are extremely similar in model. I believe this simplicity and consistency make fexprs easier to understand and use.

---

* [1]: But if you ever do want $quote, it's super easy to define:

  ($define! $quote
    ($vau (x) #ignore x))
$quasiquote would be harder, and would be pointless if you don't have syntactic sugar for it. In addition, quasiquote is far less important in a Lisp that emphasizes fexprs and doesn't have macros (by default).

* [2]: Arubic has a "hygienic" quasiquote called quasisyntax (borrowing from Racket's terminology):

  (mac let (var expr . body)
    #`((fn (var) . body) expr))
The above is perfectly hygienic (like Kernel) because it doesn't introduce symbols. It's the equivalent of "unquote everything!":

  (mac let (var expr . body)
    `((,fn (,var) ,@body) ,expr))
This doesn't work if you want to introduce a binding not visible in the dynamic environment. To do that you'd need to use w/uniq:

  (mac do1 args
    (w/uniq g
      #`(let g ,(car args)
          ,@(cdr args)
          g)))
So it's not quite as hygienic as say... Scheme macros, but it's still more hygienic than Arc macros.

-----

2 points by rocketnia 1765 days ago | link

"The problem with quasiquote is that it makes it really easy to break hygiene by accident[2], and Kernel's 3rd guideline is..."

A misguideline. :-p It's my least favorite part of Kernel's philosophy. I'd rather say desirable things should be easy to do by accident.

Pretty much everywhere Kernel uses G3, I'd rather see another justification. For instance, if Scheme has a "dangerous" feature people commonly use as though it did something slightly different, and Kernel replaces it with something else that's harder to make that mistake with, I might be swayed by an orthogonality argument.

By the way, Kernel replaces gensyms with "keyed static variables." As it happens, the R-1RK does justify this with an appeal to orthogonality!

It also notes that their main use case is environment mutation. Hygiene doesn't need them; as your '$or examples demonstrate, the primary replacement for gensyms is fexprs. :-p

-----

2 points by Pauan 1764 days ago | link

Okay, so, I'm not saying that a facility for easily constructing lists (even unhygienic ones) is bad.

What I'm saying is that $quasiquote in specific is bad. But I'm perfectly fine with something like my $quasisyntax operator, which is hygienic.

The problem with $quasiquote is that $quote is the default[1], so in order to be hygienic (which is what you want most of the time), you have to $unquote everything[2].

My $quasisyntax[3] operator, on the other hand, $unquote's things by default while still letting you easily break hygiene when you want to.

In other words...

  #`(foo (bar qux) corge)
...is just syntactic sugar for:

  (list foo (list bar qux) corge)
This is useful not only in macros, but also in ordinary fexprs and even functions. But it's only syntactic sugar for cons/list, so by default it doesn't support any way to use $quote to break hygiene.

Now, you might be sitting there saying, "but I want to break hygiene!" and I agree, it does need to be easy to break hygiene when you want to, just not to break it by accident.

So what do I need to change to allow for hygiene violations? Actually, I don't need to change anything. The above operator already supports hygiene breaking, without needing to bake it in, unlike $quasiquote which requires $quote to be built-in.

Let me give an example. Consider this definition of $afn, which is almost hygienic except for the deliberately inserted symbol 'self:

  ($define! $afn
    ($mac (parms . body)
      `(,$rfn self ,parms ,@body)))
Okay, and here's how you would define it with $quasisyntax:

  ($define! $afn
    ($mac (parms . body)
      #`($rfn ,'self parms . body)))
See that? I use unquote to evaluate an expression, and that expression just so happens to be the quoted symbol 'self[4].

So you can define a $quasisyntax operator in Kernel that allows you to use $quote'd (unhygienic) values, without $quasisyntax actually needing $quote at all! In that case it just serves as syntactic sugar for cons/list.

But, if you decided to add in a $quote operator, you can immediately use it with $quasisyntax to break hygiene, without needing to change/redefine $quasisyntax.

See how easy that is? So, with $quasisyntax, hygiene is the default, but you can still super-easily break hygiene, just by using ,' to insert the quoted symbol.

This is what I was talking about earlier, about making it hard to do dangerous things by accident while still making it possible (and hopefully easy) to do it on purpose. Whether John agrees with me on this particular point or not, I don't know, but that's how I see it.

---

* [1]: This is logical in a Lisp that uses $quote to suppress evaluation, but Kernel uses explicit-eval and doesn't even define $quote by default; so $quasiquote is inconsistent with the very core of Kernel itself. On the other hand, as far as I can tell, the definition of $quasisyntax is perfectly compatible and consistent with Kernel in every way, though I concede that Kernel isn't particularly interested in things like syntactic shortcuts.

* [2]: You might look at the above comparison of $afn and think, "pfft that's not a big deal" and you'd be right. The differences in the definition of $afn are small.

But in even slightly more complicated macros, the need to $unquote everything becomes a bigger issue, especially since it's very easy to forget to $unquote something (I've done this myself with $quasiquote, at least a few times, and I'm usually a very careful person; I usually double and even triple check my code).

It also looks ugly seeing the , character spewed out everywhere, therefore encouraging you to not use it, therefore encouraging you to break hygiene (by default) rather than breaking hygiene consciously, when you want to.

* [3]: I, personally, would probably call it something other than $quasisyntax, and I would probably use ` or @ rather than #` for brevity and because it looks nicer.

* [4]: Incidentally, this solves some issues I had had with $quasisyntax in Arubic... consider this old Arubic definition of afn:

  (mac afn (parms . body)
    #`(rfn 'self parms . body))
Okay, so I'm using $quote as a way to say "the symbol self should be unhygienic". But that not only means that $quasisyntax hardcodes the symbol $quote (in addition to $unquote and $unquote-splicing), but it also means you need to double the $quote's if you want to insert the list ($quote ...):

  #`(annotate ''mac ...)
Notice the double $quote's around the symbol 'mac. The above is equivalent to:

  `(annotate 'mac ...)
This inconsistency bugged me, but just a short while ago I realized I could get rid of all that hackiness just by using ,' instead.

That also means that $quasisyntax could be made available by default in Kernel (if John wanted that) without requiring $quote to be defined!

-----

2 points by rocketnia 1763 days ago | link

I already understood the benefit of your quasisyntax operator. I agree that it makes desirable things easy to do by accident.

I don't agree that the original quasiquote should be difficult.

It's pretty much a matter of political spin. There's no need to work to make quasiquote difficult, but there's little need to work to make it easy either, since that effort would be better spent on quasisyntax. We can just neglect to give quasiquote any sugar, at the same time as we spoil quasisyntax with sugar of its own. We can also neglect to define quasiquote in the main language distribution altogether.

---

"I, personally, would probably call it something other than $quasisyntax"

If it's essentially a syntax that turns paren into a list literal syntax (with ,foo syntax as an escape), then maybe $lists or $es (evaluation structure) would be a nice brief name for it. :)

-----

1 point by Pauan 1763 days ago | link

"I don't agree that the original quasiquote should be difficult. [...] We can just neglect to give quasiquote any sugar, at the same time as we spoil quasisyntax with sugar of its own. We can also neglect to define quasiquote in the main language distribution altogether."

tl;dr: I agree with you overall, so perhaps we're just using different terminology or wording, resulting in confusion.

Right, I'm not saying that users shouldn't be allowed to define $quasiquote if they want to, but given its problems and the fact that $quasisyntax is better overall, there's no need to provide $quasiquote by default in the language. And if there were a better alternative ($quasisyntax) in the language, then users would be encouraged to use it instead of defining $quasiquote themselves.

Even though Kernel goes out of its way to not provide $quote, it's trivial to add it in (literally just 1-2 lines of super-simple code). And it should only take ~20 lines or so to define $quasiquote, the same as it does in Arc. The only difference is Kernel doesn't have syntactic sugar for $quote, $quasiquote, or $quasisyntax.

I think syntactic sugar should have its own facility, such as reader macros (or something better), allowing users to add in syntax for $quote/$quasiquote/$quasisyntax/other things. But that's a different story.

-----

1 point by Pauan 1763 days ago | link

"perhaps we're just using different terminology or wording, resulting in confusion."

In particular, I disagree with this statement of yours: "I already understood the benefit of your quasisyntax operator. I agree that it makes desirable things easy to do by accident."

As far as I can tell, quasisyntax does not make it easy to do "desirable things" (like breaking hygiene) by accident. It does make it easy to do them on purpose.

I think our primary disagreement is that we're using different meanings for the word "accident". It seems to me that when I say "it should be hard to do dangerous things by accident" you take that to mean "I, the language designer, shall deem what shall be dangerous, and you, the user, shall not be allowed to do those things without jumping through crazy hoops!!"

---

But that's not what I'm saying. When I'm talking about "on accident", I'm referring to the user, not the language designer. Let me give an example. Let's suppose you had an operator in your language for deleting files.

To look at a real-world example, let's use the Unix command "rm". If you want to remove the file /foo/bar you would say this:

  rm /foo/bar
Okay, but let's suppose you mistype something (we all do it occasionally) so you instead accidentally enter this:

  rm / foo/bar
A difference of only a single character! But what the above command does is remove every file on your entire harddrive[1]. This is what I mean by "doing dangerous things by accident" and I'm sure some people have been burned by this[1].

Thankfully that's not what actually happens (at least on my computer): rm doesn't remove directories without specifying the -r option. So for removing files you say `rm foo` and for removing folders you say `rm -r foo`. Of course, that doesn't help if you accidentally type `rm -r / foo/bar`...

That's an example of making dangerous things difficult to do by accident, while still making it easy to do them on purpose when you want to.

---

So when I say that dangerous things should be difficult to do by accident, this is really just a different way of saying that "computers should do exactly what we want them to do, no more and no less".

Alternatively, it could also just be a rephrasing of the principle of least surprise, that computers should do what people expect them to do, rather than going off and doing crazy things that the user didn't want.

You don't want the computer to go berserk and do things you didn't intend for it to do. But in the (rare) case that you do want to, say, remove your entire harddrive, you should still be able to do so, and ideally it should be easy as well. Just not easy by accident.

---

My quasisyntax operator makes it super-easy to break hygiene when you want to, but it doesn't make it easy to do it by accident. You have to manually and explicitly add ,' to a symbol to make it unhygienic. Contrast that with quasiquote where it's easy to forget to unquote something, therefore quasiquote has the same power as quasisyntax, but makes it easy to break hygiene by accident.

Essentially the difference between quasiquote and quasisyntax is that quasiquote makes it easy to make mistakes, but quasisyntax does exactly what you would expect it to do, without making mistakes, while still making it really easy to break hygiene when you want to.

---

* [1]: I'm not actually sure about that... did older versions of rm automatically remove directories without specifying -r...? I vaguely remember some Unix horror stories, but they might have already been specifying -r in which case that won't help any.

-----

2 points by rocketnia 1763 days ago | link

I consider your quasiquote operator to make it easy to accidentally forget to quote things. Since we both consider that to be what people are most likely to want anyway, that's hardly a problem.

For a more detached example, consider someone who implements the same algorithm in Arc and Haskell. Their Haskell version uses call-by-need when their Arc version uses call-by-value, and at first they may even understand and disregard this discrepancy. Later they discover that one of these is detrimental--an undesirable accident. I consider the other one to be a desirable accident.

To me, this isn't just about whether the computer does what the user wants/expects it to do. It's about whether the computer does what the user doesn't know they want it to do. IMO, programming is all about specifying what one wants in really specific terms, even down to details one isn't truly sure about yet.

---

"It seems to me that when I say "it should be hard to do dangerous things by accident" you take that to mean "I, the language designer, shall deem what shall be dangerous, and you, the user, shall not be allowed to do those things without jumping through crazy hoops!!""

That's pretty much right. :) You have a good point that "by accident" is a key phrase, and your example of an accident is a nice thing to make difficult.

However, I still take any use of G3 in Kernel with a grain of salt, in case it turns out to be accidentally used in an "it should be hard to do things" way instead. There are many places where it's used for legitimate clarity of program behavior, but I'm not so convinced by these uses:

"Distinguishing long identifiers by case alone is error-prone, hence contrary to G3."

"One compromise is to G3 of §0.1.2. Naive traversal algorithms are dangerous; and, inevitably, the more readily available naive traversal algorithms become, the more readily the programmer can use them by accident."

"Most algorithms, however, are intended by the programmer to be im- mutable, and therefore, when an object is primarily meant to represent an algorithm, mutating it is a dangerous activity that ought to be difficult to do by accident (G3 of §0.1.2)."

I'm not convinced that these things are dangerous. I expect these kinds of accidents to cause run time errors nearly 100% of the time. They could do dangerous things in particularly contrived cases, but IMO, contrived cases are indistinguishable from non-accidents. (Yeah, I'm not being very empirical about this. >.> )

(When multiple antagonistic programmers are involved in a program, encapsulation prevents more than accidents, but I doubt that's the case under discussion....)

-----

1 point by Pauan 1763 days ago | link

"I consider your quasiquote operator to make it easy to accidentally forget to quote things."

Sure, but I'd argue that A) you want hygiene most of the time, so forgetting to $quote something happens less often, and B) in the case where you forget to $quote something with $quasisyntax, it usually throws an error. The reverse is not true: if you forget to $unquote something with $quasiquote, it's usually not an error, it fails silently until the bug is later discovered (possibly after a long time).

In other words, $quasisyntax not only chooses a default that you (the programmer) want more often (hygiene), but it also tends to throw errors earlier in case you (the programmer) mess up. That's why I consider $quasiquote to be bad. But in some hypothetical language where you want to break hygiene a majority of the time... then $quasiquote would probably be superior to $quasisyntax.

---

"To me, this isn't just about whether the computer does what the user wants/expects it to do. It's about whether the computer does what the user doesn't know they want it to do."

If so, then in order for the user to be guided into areas where they don't know they want it to behave in that way, then the language needs to encourage/discourage various behaviors in some way. Every language does this, in different ways, and to different degrees.

---

"but I'm not so convinced by these uses"

1) I agree, that doesn't seem to have much to do with G3 simply because if you accidentally use the wrong case for a long identifier, it's very likely to throw a runtime error that the variable can't be found.

2) I actually think this does justify G3 (though I agree it's pretty weak), for the same reasons I mentioned that using $quasiquote encourages you to use bad hygiene. If you don't provide a solid traversal algorithm, programmers are going to write their own awful algorithms which are no doubt broken.

I don't think Kernel goes out of its way to prevent you from writing stupid naive algorithms, and writing stupid naive algorithms is really easy. Instead, that's a rationale for not including naive algorithms in the core. You can still write your own naive algorithm, it just isn't going to be baked into the language, especially since the only gain of a naive algorithm is execution speed, not power or flexibility.

3) That particular part of the Report is talking specifically about operand capturing, e.g. fexprs. What it's saying is that when you apply a value (usually an acyclic list) to a fexpr, that fexpr could potentially mutate it.

This could cause some unexpected "action at a distance" so Kernel instead makes an immutable copy of the value and applies that to the fexpr instead. I'm not actually sure how I feel about that...

Incidentally, I believe that's how most (all?) Lisps do it: if you use (apply foo bar) in Arc, the foo function gets a copy of the list bar, not bar itself.

If you want the fexpr to be able to mutate its arguments... I would have suggested to pass the (mutable) list as a single argument to the fexpr, but... I'm not sure if that's actually possible... but it should be, right...? I would assume so.

---

"I'm not convinced that these things are dangerous."

I don't think they're any more dangerous than breaking hygiene. But Kernel already specifically mentions breaking hygiene as being dangerous, therefore calling those things dangerous is consistent with Kernel's viewpoint. So I suppose it depends on how dangerous something is before you personally (the programmer) call it "dangerous". But John Shutt apparently did consider them dangerous, hence their design in Kernel.

I personally wouldn't call them "dangerous". I think that word should be reserved for disastrous things like, say, deleting your entire hard-drive by accident. I would call things like breaking hygiene and whatnot "undesirable violations" (with the caveat that they can be desirable in certain circumstances, so they shouldn't be banned outright). If you have a better word to describe it, I'd like to hear it.

-----

2 points by rocketnia 1762 days ago | link

"In other words, $quasisyntax not only chooses a default that you (the programmer) want more often (hygiene)..."

...which is what I said :) ...

"...but it also tends to throw errors earlier in case you (the programmer) mess up."

I agree with this too. Throwing an error early is a nicer accident than throwing an error late.

---

"If so, then in order for the user to be guided into areas where they don't know they want it to behave in that way, then the language needs to encourage/discourage various behaviors in some way. Every language does this, in different ways, and to different degrees."

I don't see how this has a point any different from what I'm saying. :-p

---

"I don't think Kernel goes out of its way to prevent you from writing stupid naive algorithms, and writing stupid naive algorithms is really easy. Instead, that's a rationale for not including naive algorithms in the core. You can still write your own naive algorithm, it just isn't going to be baked into the language, especially since the only gain of a naive algorithm is execution speed, not power or flexibility."

In a customizable language (which Kernel isn't), to do something the most obvious way makes that part of the language more intuitive to customize, even if the most obvious thing is naive.

In a language with a culture of sharing black-box libraries (which Kernel would probably be, but for instance wart might not be), to do something the same way a library would do it makes for a more consistent experience.

Instead, for consistency's sake, Kernel advocates for library writers to write non-naive algorithms, ultimately making library-writing more of a chore.

This is totally a "worse is better" line of argument, so I don't expect to convert you with it so much as to convince you the ideals aren't so clear-cut.

---

"If you want the fexpr to be able to mutate its arguments... I would have suggested to pass the (mutable) list as a single argument to the fexpr, but... I'm not sure if that's actually possible... but it should be, right...? I would assume so."

I think Kernel calls the copy algorithm "copy-es-immutable," and IIRC, what it does is [treewise cons idfn _], but with an immutable cons. So if your mutable conses are hidden behind a variable name (to be evaluated) or a thunk, they'll survive. If you're just trying to eval (foo (my mutable list)), that won't work.

---

"I personally wouldn't call them "dangerous". I think that word should be reserved for disastrous things like, say, deleting your entire hard-drive by accident."

I agree. That's why I brought up those uses of G3; they just seem like examples of accidental errors being prevented, rather than accidental disasters being prevented.

---

"I would call things like breaking hygiene and whatnot "undesirable violations" (with the caveat that they can be desirable in certain circumstances, so they shouldn't be banned outright). If you have a better word to describe it, I'd like to hear it."

I don't know what you mean by "things like breaking hygiene and whatnot," unless you mean all the perhaps-not-so-dangerous things John Shutt considered dangerous when writing the R-1RK, and that isn't a very well-defined set of things. :-p I would probably call them "bugs."

Maybe G3 becomes something like this: While permitted on general principle, bugs should be hard to introduce by accident.

Hmm, I think this could be another, more upfront way to phrase the way G3 is actually used: The language should allow more expressiveness than is recommended for everyday use, but programmers on the fringe cases should have to jump through extra hoops.

I might actually support a G3 worded this way, but only because it suggests that it's easy to write full programs without jumping through any hoops, just by sticking to a subset of the language. Do you think there's a clear subset like that in Kernel? I'm not sure.

-----

1 point by Pauan 1762 days ago | link

"I don't see how this has a point any different from what I'm saying. :-p"

Right, I'm agreeing with you a lot, except that you seemed to be a bit against some of the things Kernel does, and I was pointing out that Kernel does that because it's trying to guide you toward a certain way of thinking that (hopefully) results in better programs. Whether you agree with Kernel's guidelines or not is another story.

---

"Instead, for consistency's sake, Kernel advocates for library writers to write non-naive algorithms, ultimately making library-writing more of a chore."

Library-writing already is a chore. If the only libraries available are hacky semi-broken things, then why would I use them? I can just write my own hacky semi-broken program faster than it would take to understand the library (usually).

So one of the primary benefits of libraries is that you can put a lot of extra effort into it, and then people use the top-notch shiny awesome library so they don't have to put all that effort into getting things perfect. Kernel just seems to embrace that and try to make even the core language itself rock-solid.

---

"This is totally a "worse is better" line of argument, so I don't expect to convert you with it so much as to convince you the ideals aren't so clear-cut."

I am generally an advocate of "worse is better", or else why would I be using Arc? When discussing Arc, I do tend to have a "worse is better" attitude about it. But at the same time, I also care about "doing the Right Thing", so my thinking shifts when talking about Kernel.

Essentially, Kernel seems to me to be a marvelous language that embraces the "do the Right Thing" attitude, whereas Arc seems to be a marvelous language that embraces the "worse is better" attitude. I like both of them, they have different atmospheres and feelings about them.

So, I think it really does depend on what you want to do, what you expect out of a language, etc.

I must say, though, Kernel has captivated me recently. I'm a total sucker for minimalism and elegance, while at the same time caring about speed and practicality. Inner conflict!

---

"unless you mean all the perhaps-not-so-dangerous things John Shutt considered dangerous when writing the R-1RK"

I was being hand-wavy and meaning things similar to hygiene breaking, like the 2nd and 3rd things on your list of things you mentioned in your post. So, not necessarily all the things John Shutt considered dangerous, but certainly some of them.

---

"Maybe G3 becomes something like this [...] I might actually support a G3 worded this way"

Indeed, though I already understood it to mean "accidentally introducing bugs into a program", though I'm not sure how John Shutt meant it... but judging by the way he uses it in Kernel, I'd say my interpretation is roughly correct.

-----

2 points by akkartik 1765 days ago | link

  ;; implicit eval
  ($define! $or?
    ($mac (x y)
      ($let ((temp (gensym)))
        `(,$let ((,temp ,x))
           (,$if ,temp ,temp ,y)))))

  ;; explicit eval
  ($define! $or?
    ($vau (x y) env
      ($let ((temp (eval x env)))
        ($if temp temp (eval y env)))))
Seeing these two code snippets really drives home the point about implementing macros using fexprs, thanks. I'm going to go read chapter 7.3 like you suggest.

-----

1 point by Pauan 1765 days ago | link

Chapter 7.3 of John's dissertation discusses it in more detail. I particularly like how John uses a form of currying as a way to avoid defining $quote. That would probably be the best place to start if you want to understand why Kernel doesn't have quote. In fact, §7.3 basically makes the exact same points I do (and then some), so I'd suggest reading that:

http://www.wpi.edu/Pubs/ETD/Available/etd-090110-124904/unre...

-----

2 points by akkartik 1765 days ago | link

I like your quasisyntax operator. Why do you say it's not fully hygienic?

-----

2 points by rocketnia 1765 days ago | link

I'm not sure, but I think R5RS macros ('syntax-rules, 'let-syntax, 'letrec-syntax, maybe others?) walk the whole template code, taking advantage of their knowledge of special forms like 'let to know when a binding is being introduced.

I think Pauan might be referring to that as a missing feature of that quasisyntax form, regardless of whether that actually causes imperfect hygiene.

-----

1 point by Pauan 1765 days ago | link

That's correct. For instance, with proper full hygiene, you could write do1 like this:

  (mac do1 args
    #`(let g ,(car args)
        ,@(cdr args)
        g))
The only change is I removed the w/uniq wrapper: quasisyntax would automatically make g into a gensym. It doesn't currently do that, but it could.

That would make it essentially a fully hygienic system that's vaguely similar to Scheme's macros, but it's much easier to implement, and you can choose to easily break hygiene at any time (by using quote) rather than fussing with syntax->datum.

It's also fully compatible with Arc 3.1 macros[1], and you can freely choose to use quasiquote or quasisyntax, or hell, use both in the same macro!

Of course, this requires you to actually write the macros in that style. Normal quasiquote macros won't gain any benefit from hygiene in this model: you'd need to convert them to use quasisyntax instead (which is quite easy and even mechanical to do).

---

* [1]: That part requires a compiler change, but I think it's an overall good compiler change that can benefit even non-quasisyntax macros. It works correctly in Nu and ar, but not Arc 3.1.

-----

1 point by akkartik 1765 days ago | link

Wart has a perfectly traditional lexical and dynamic scope setup now. Much of that link I posted earlier is stale. I backed out of that experiment ages ago (and I think I said so before on this forum). If the current version still has something that seems off I'd love to hear it.

The bits that are non-traditional are: eval can only operate using the current lexical scope and dynamic bindings, and the eval inside macros happens implicitly because I can't provide the caller scope to it.

Looks like it's time to revise this. All my questions were really the same question, so your answer to one of them is perfectly complete.

-----

1 point by Pauan 1766 days ago | link

Isn't that the point of fexprs, though? That they can selectively eval their arguments? So couldn't the "quote the argument to suppress evaluation" just desugar into a fexpr that evals the argument? Something like this...

  (fn ('a b c . 'd)
    ...)
...would desugar (macro expand) into this...

  (fexpr (a b c . d) env
    (with (b (eval b env)
           c (eval c env))
      ...))
...assuming that env is a gensym to avoid clashes with variables in the function's body.

That is, a fn would either be wrapped or not depending on whether a quote appears in its argument list. I'm actually not fond of that, and would prefer to explicitly use fexprs when I want to not evaluate things, but... it is consistent with wart's already existing support for things.

Of course, that means that "fn" doesn't always create a function. It might potentially create either a fexpr or a function... not sure if you want that.

-----

2 points by Pauan 1773 days ago | link

"I don't get this either."

I didn't at first either. But I walked through the steps one-by-one until I figured it out. Here they are:

apply is a function, so it evaluates all its arguments. When calling (apply list '((1 2))) its arguments evaluate to the function list and the list ((1 2)). It then calls the function list with the argument (1 2). This is equivalent to (list (1 2)), as rocketnia said.

---

To put it another way, (apply list '((1 2))) would be equivalent to (eval (cons list '((1 2)))) which is equivalent to (eval (list list '(1 2))) which is equivalent to (eval '(list (1 2))) which is equivalent to (list (1 2)).

Thus, you would either need to say (apply list '('(1 2))) or apply would need to map quote over the arguments list.

-----

1 point by Pauan 1773 days ago | link

In fact, following that line of thought, here's an implementation of apply:

  (def apply (f args env)
    (eval (cons f (map quote args)) env))
You can make the env argument optional if you like, with whatever semantics you want (the current environment, a new environment, etc.)

---

Side note: why doesn't Arc 3.1 use this definition? Because eval is incredibly slow in Racket, but applying a function is fast. But I'm assuming in a language that emphasizes fexprs (like Kernel, or wart) that eval should be plenty fast.

-----

2 points by rocketnia 1772 days ago | link

  >
  > (map quote args)
  >
I don't think that does what you think it does. Suppose 'map is implemented independently of 'apply, and that it takes this form:

  (def map (f seq)
    ...
    ... (f elem) ...
    ...)
Then (map quote '(1 2 3)) should result in the list (elem elem elem). That's what my intuition says anyway, not that it's really useful behavior. :-p

Mapping [list quote _], as I was talking about when I called it ($lambda (x) (list $quote x)), will hopefully work regardless of what mapping quote does.

-----

1 point by Pauan 1772 days ago | link

Right, I was assuming quote automagically read my mind and did what you're talking about. :P

-----

1 point by Pauan 1773 days ago | link

I see now. You're right, I guess it would have to map quote over all the arguments.

-----

1 point by akkartik 1774 days ago | link

Macros now warn when called with spliced args: http://github.com/akkartik/wart/commit/1ad7da8d6f

After sleeping on it I'm less bummed. Macros aren't a clean, well-behaved tool anyway. They're open to variable capture. You have to be careful with multiple evaluation. And now you can use apply with them, but you'll have to be careful. I'm happy with warning the user about it. (Though I'm still interested in a better replacement for apply/splice.)

-----

1 point by akkartik 1773 days ago | link

Is there a manual workaround for macros' limitations with splice? The way we use gensyms to get around their limitations with variable capture?

I've been playing with this idea:

  (def quotify(f) (fn 'args (f @args)))
Now I can turn the final example into:

  wart> = x '(1 2)
  wart> (and @(quotify.list x))
  (1 2) ; woohoo!
However it doesn't seem to help with the definition of all-true..

-----