Arc Forumnew | comments | leaders | submitlogin
Macro expansion/evaluation problem
3 points by palsecam 3668 days ago | 35 comments

   arc> (mac amac () (prn "amac is reached"))
   #3(tagged mac #<procedure: amac>)
   arc> (if t 1 (amac))
   amac is reached    <-- ooops!
   1
I might be wrong but IMO this is a abnormal. I tried in Common Lisp and there is no such "bug".

I tried to fix this behaviour but it is too difficult for me.

The fact are: 'ac calls 'ac-call when the sexpression to compile is a macro or function call.

When it's a function, 'ac-call returns a quasiquoted expression (something like `(ar-funcall1 ,(ac fn env) [...])), but when it's a macro, it calls 'ac-mac-call that directly executes the macro.

So even if the macro is not going to be reached, 'ac-mac-call will execute it.

Am I the only one to find this buggy? If not, and if someone has a fix for this, please share it!



3 points by conanite 3668 days ago | link

Macros are like compiler hooks, they are executed at compile time, not run-time. When you run code at the REPL, run-time is immediately after compile-time, so the difference is not obvious. Try this:

  arc> (mac amac () (prn "amac is reached"))
  #3(tagged mac #<procedure: amac>)
  arc> (def maybe-amac () (if t 1 (amac)))
  amac is reached
  #<procedure: maybe-amac>
  arc> (maybe-amac)
  1
No "amac is reached" when you execute maybe-amac ...

I don't know how CL works internally, maybe it suppresses std-out from macros?

-----

1 point by palsecam 3667 days ago | link

For CL, I don't know exactly how it works, but I suppose 'if is treated as a special form (or expand to a call to 'cond which is a special form, or something) which is in some way "higher in priority" than macro expanding.

I am really really not sure of that, but if I remember well my teachings, this is a part of the reasons why 'if or 'cond is a special form and not just a normal macro.

   > maybe it suppresses std-out from macros?
Now that would be crappy :-D! It is not just a problem of output, try the sleep macro example:

   cl> (defmacro msleep (time) (sleep time))
   MSLEEP
   cl> (if t 1 (msleep 3))
   1    <-- answers immediately, CL doesn't suffer from oversleeping!
Or:

   arc> (mac expect-arg-m (arg) arg)
   #3(tagged mac #<procedure: expect-arg-m>)
   arc> (expect-arg-m)
   Error: "procedure  expect-arg-m: expects 1 argument, given 0"  <-- OK, this is normal
   arc> (if t 1 (expect-arg-m))
   Error: "procedure  expect-arg-mac: expects 1 argument, given 0"  <-- hmm, parse the false clause...

   cl> (defmacro expect-arg-m (arg) arg)
   cl> (expect-arg-m)
   error while parsing arguments to DEFMACRO EXPECT-ARG-MAC:
   invalid number of element  <-- OK, normal
   cl> (if t 1 (expect-arg-m))
   1   <-- didn't bother to parse the false clause

-----

1 point by palsecam 3668 days ago | link

OK, guys, I admit it is for a dirty hack that I wanted the CL behaviour, but I maintain the Arc behaviour is "buggy", I mean, not so natural. Here is why it bothers me.

OK, so I want 'each to work like this:

   arc> (each v '(1 2 3) (pr v))  <-- classic 'each call
   123
   arc> (each '(1 2 3) (pr _))   <-- if there is no var to bind, use the special '_' symbol
   123
So after some tries, the best solution I found (which is still terribly ugly in a sense) is:

   (mac myeach (first expr . body)  
     `(if (defined ,first)  
       ; first is the list/hash to traverse, 
       ; or if undef, the variable to bind
       (real-each _ ,first ,expr ,@body)
       (real-each ,first ,expr ,@body)))
And this doesn't work:

   arc> (myeach v '(1 2 3) (pr v))
   123    <-- this one is OK
   arc> (myeach '(1 2 3) (pr _))
   Error: "Can't understand fn arg list 1"   <-- because (real-each ,first ,expr ,@body))) is expanded
Now, if I want the same thing in Common Lisp, it just works. And since 'if is a special form (in Arc), I expected it would also work in Arc. Here is a "CL proof":

   cl> (defmacro each (var expr &rest body)
         "Limited, small clone of Arc's 'each"
         `(loop for itm in ,expr  ; expr is a list
            do
	     (let ((,var itm))
               ,@body)))

       (defmacro myeach (first expr . body)
         `(if (handler-case (listp ,first)  
		(t () nil))   ; <-- first is unbound if this handler is reached
            (each _ ,first ,expr ,@body)
            (each ,first ,expr ,@body)))

     cl> (myeach '(1 2 3) (print _)))
     1
     2
     3
     cl> (myeach v '(1 2 3) (print v)))
     1
     2
     3

-----

2 points by fallintothis 3667 days ago | link

  (mac myeach (first expr . body)
    (if (errsafe (eval first)) ; questionable, but essentially what 'defined did
                               ; just happens at expansion instead of compilation
        `(each _ ,first ,expr ,@body)
        `(each ,first ,expr ,@body)))

  arc> (myeach '(1 2 3) (prn _))
  1
  2
  3
  nil
  arc> (myeach v '(1 2 3) (prn v))
  1
  2
  3
  nil
  arc> (myeach (k v) '((a 1) (b 2) (c 3)) (prn k ": " v))
  a: 1
  b: 2
  c: 3
  nil
  arc> (myeach (k v) (table [for i 1 3 (= _.i (* i i))]) (prn k ":" v " "))
  2:4
  1:1
  3:9
  #hash((3 . 9) (1 . 1) (2 . 4))
  arc> (myeach (table [for i 1 3 (= _.i (* i i))]) (prn (car _) ":" (cadr _) " "))
  2:4
  1:1
  3:9
  #hash((3 . 9) (1 . 1) (2 . 4))
  arc> (macex1 '(myeach a b c)) ; note: a is undefined
  (each a b c)
  arc> (= a '(1 2 3))
  (1 2 3)
  arc> (macex1 '(myeach a b c)) ; note: a is now defined
  (each _ a b c)
  arc> (let v 2 (myeach v '(1 2 3) (prn v)))
  1
  2
  3
  nil
Any better?

-----

1 point by palsecam 3667 days ago | link

Thanks for the proposal! Very interesting.

However, 'eval is not perfect.

   arc> (let lst '(1 2 3) (myeach lst (prn _)))
   Error: "reference to undefined identifier: __"  
   ; ^ lst was taken for the variable to bind
   ; and (prn _) for the sequence to traverse

   arc> (= glst '(1 2 3))
   (1 2 3)
   arc> (eval 'glst)  ; global scope, OK
   (1 2 3)
   arc> (let lst '(1 2 3) (eval 'lst))  ; lexical scope, KO
   Error: "reference to undefined identifier: _lst"

-----

1 point by fallintothis 3667 days ago | link

Which is what makes a macro like this so hairy: the inconsistency.

There are the lexical bindings you want to use as iteration variables

  (let v 2 (myeach v '(1 2 3) (prn v)))
and the ones you want to iterate over

  (let lst '(1 2 3) (myeach lst (prn _)))
Something about having your cake, eating it, etc.

Maybe what you want to do is specialize on whether the second parameter evaluates to an iterable object?

I'm not sure if, by the time anyone got the separate cases working, I'd like seeing this in use -- complexity creep.

-----

2 points by palsecam 3667 days ago | link

> Which is what makes a macro like this so hairy: the inconsistency.

Yes.

> There are the lexical bindings you want to use as iteration variables [...] and the ones you want to iterate over

Yes "context" stuff. See Perl.

> Maybe what you want to do is specialize on whether the second parameter evaluates to an iterable object?

Yes, it may be a way.

> don't know if I'd like seeing this in use -- complexity creep.

You are certainly wise here :-).

But you know, I don't care of 'each. It's OK if in current Arc it is too tricky to have, and I don't try anymore to have it work my way. It will anyway always be tricky and whether something like this should be used is another debate.

Actually what I care more about is the 'if behaviour. Everyone here seems to ignore this eventual issue and focus on the precise problem of 'each, but this was not my point initially.

-----

2 points by fallintothis 3667 days ago | link

ignore this eventual issue and focus on the precise problem of 'each

Certainly. It was in the back of my head when I wrote my reply. I didn't mention it because I was trying to be terse (I have a problem being long-winded), so opted to focus on something small in my post.

But I agree: macroexpansion time is kind of fuzzy in Arc. It even gets in the way of things like

  arc> (let do [+ _ 1] (do 1 2 3))
  3
which has been a source of hand-waving for awhile with no clear answer: http://arclanguage.org/item?id=9696. But that's another story (in a manner of speaking).

Not to pick on the intended each behavior (after all, why bother writing code if you don't like how it looks?) or the actual topic of this post.

-----

1 point by palsecam 3667 days ago | link

Thanks for the link and (let do ...) example. Interesting to read.

> macroexpansion time is kind of fuzzy in Arc

Well, this was the point of this post! It is important to know about the "limits" of your macros system. As CatDancer mentioned it, there are differences between them. The catch phrase "macros are evaluated at expansion time, functions at runtime, ..." is too simplistic and doesn't provide enough knowledge at some point.

> why bother writing code if you don't like how it looks?

You, Sir, are a wise man :-)!

-----

2 points by CatDancer 3668 days ago | link

Sounds like what you want is

  (mac myeach (first . rest)
    (if (alist first)
         `(each _ ,first ,@rest)
         `(each ,first ,@rest)))
which works:

  arc> (myeach '(1 2 3) (pr _))
  123nil
though it does defeat list destructuring... with "each", I can say:

  (each (k v) '((a 1) (b 2) (c 3)) (prn k ": " v))

-----

1 point by palsecam 3667 days ago | link

CatDancer, thanks for offering a good solution.

However, as you pointed it out, it does defeat list destructuring, which I would like to keep. Sorry that my words and CL/Arc examples doesn't mention it.

With my 'myeach, it works, because I don't test if first is alist, but if it is defined. If not, I consider it is the variable expression (e.g: v or (k v)) to use.

I.e, with my 'myeach:

    arc> (myeach (k v) (table [for i 1 3 (= _.i (* i i))]) (pr k ":" v " "))  ; would also work with an alist
    2:4 1:1 3:9
Of course, the shorter version I try to be able to use:

   (myeach (table [for i 1 3 (= _.i (* i i))]) (pr (car _) ":" (cadr _) " ")
doesn't work (currently) but it could (if the false branch of the 'if was not expanded. Comment the case for the non-anonymous 'each call in 'myeach, and it'll work).

Btw, 'defined is yet another dirty hack. Oh my god, I know there is a lot of dirt in my stuff, certainly too much. I was ashamed to talk about it but I checked Google and apparently some Lispers sometimes also use something like that, when doing exploratory macro programming:

   (mac defined args   ; typically, args is one or more symbols
    `(errsafe (do ,@args t)))  ; typically, a pedant functional programmer will have an heart attack seeing this
P.S: writing the examples of this post makes me even more hungry to be able to use '_' in more places than just for anonymous 1-param functions.

"If I had a nickel for every time I've written "for i 0 9 ..." in Arc I'd be a millionaire." (what I'd like is writing "for 0 9", the variable being '_' by default. In other words: oh Perl I so love you ;-))

-----

2 points by CatDancer 3667 days ago | link

> list destructuring, which I would like to keep

If I said

  (myeach a b c)
what would you want that to do? Would it set _ to each value in the list 'a and do 'b and 'c in the loop, or set 'a to each value in the list 'b and do 'c in the loop?

To put the question another way, when do you want 'myeach to use "_"?

Suppose it was the number of arguments... two arguments would mean to use "_". That's easy to do:

  (mac myeach args
    (if (is (len args) 2)
      `(each _ ,@args)
      `(each ,@args)))

-----

1 point by palsecam 3667 days ago | link

(myeach a b c): set '_' to each value of 'a, then do 'b and 'c in the loop. Or if 'a is undefined, set 'a to each value in 'b, and do 'c in the loop.

But if 'a is lexically bounded but is not the list/table to traverse, my 'myeach will not work as expected.

   (let v 2 (myeach v '(1 2 3) (prn v)))  
will call (each _ v '(1 2 3) c) because 'v is defined.

And this is a problem. What I try to do, call it "optional first arg" is dirty. I mean, terribly unhygienic ;-), imperfect. Using the kind of tricks I'm trying is subject to discussion, and have a lot of 'cons.

But this is not really the point here. The point is, 'if in Arc doesn't work like in Common Lisp. If this is a bug or just a difference is also subject to discussion, but at least know the fact. I mean it is not that natural that Arc acts like that here.

To come back to your questions. I want to use "_" if the first argument is defined. If it is (defined), it supposes it is the list/table to traverse, else it supposes it is the variable to be bind (call it "context intelligence").

Yes, the two arguments solution is easy to do, like is defeating destructuring, and both solutions could be used (I'll certainly adopt one of them). But 1. adds extra ()s and 2. removes a good feature.

These times I feel adventurous, so I tried another thing :-)

Anyway, great thanks for your help & suggestions.

-----

1 point by CatDancer 3667 days ago | link

How do you tell if 'a is defined? Are you using pg's Arc, or Anarki, or something else?

-----

1 point by palsecam 3667 days ago | link

I'm using this:

   (mac defined args   ; typically, args is one or more symbols
     `(errsafe (do ,@args t)))  ; typically, a pedant functional programmer will have an heart attack seeing this
which is, it's not that I love spamming this term but it must not be forgotten, terribly dirty. I mean using it indicates there is certainly a problem with your (functional) code. Plus here the implementation is tricky.

   arc> (defined a)
   nil
   arc> (let a 2 (defined a))
   t
   arc> (= b 3)
   3
   arc> (defined b)
   t
   arc> (defined (c d) e)
   nil
   arc> (defined 'e)
   t

-----

1 point by CatDancer 3667 days ago | link

At macro expansion time in Arc, there's no lexical scope yet, as Arc macros operate on the input program as lists. So if you say

  (let a 2 (myeach ...))
'myeach can check if 'a is defined using your function, but it will find out that 'a isn't defined, because the lexical scope for 'a isn't created until after the macro expansion is done.

I don't know about Common Lisp in particular, but there are other more powerful macro expansion languages that iirc can give you information such as whether a variable is in scope of not. So I think the issue that you're running into isn't in the behavior of macros and 'if, but that Arc isn't able to tell you at macro expansion time whether a variable is defined or not.

-----

1 point by palsecam 3667 days ago | link

CatDancer, thanks once again, you're full of advice!

> arc macros operate on the input program as lists

I agree.

> (let a 2 (myeach ...)) will see 'a undefined. [...] Lexical scope explanation [...]

I believe you know Arc internals way better than me, and I certainly miss some points in your explanation but it seems to work for me:

   arc> (mac myeach (first expr . body)  
          `(if (defined ,first)  
             (each _ ,first ,expr ,@body)
             (prn "undefined")))
   #3(tagged mac...)
   arc> (let a '(1 2 3) (myeach a (prn _)))
   1
   2
   3
   nil
   arc> (myeach a (prn _))
   undefined
   "undefined"
> More powerful macro systems [...] can give you information such as whether a variable is in scope of not.

Certainly yes. I don't know either.

> So I think the issue that you're running into isn't in the behavior of macros and 'if, but that Arc isn't able to tell you at macro expansion time whether a variable is defined or not.

Being (or not) able to tell me at macexpansion time if a variable is defined is part of what I'd call the behaviour of macros ;-)

But actually, 'defined was just a way I tried to deal with Arc macro expansion stuff, but it was just a consequence of my problems with 'if, not the beginning.

However yes, for the precise case of 'each (but again, this topic was not limited to it), maybe the 'if behaviour would not be the main issue anyway.

-----

2 points by CatDancer 3667 days ago | link

> it seems to work for me

  (mac myeach (first expr . body)  
          `(if (defined ,first)  
             (each _ ,first ,expr ,@body)
             (prn "undefined")))
ah, but now you're doing the 'defined test at run time. What you want is to be able to do the 'defined test at macro expansion time in order to affect how your macro is expanded, and that, as far as I know, Arc isn't able to do for you.

-----

1 point by palsecam 3667 days ago | link

> now you're doing the 'defined test at run time.

Yes, but check the previous messages, I've always wanted to do so. The definition of 'myeach basically never changed.

> What you want is to be able to do the 'defined test at macro expansion time in order to affect how your macro is expanded

Not necessary (maybe yes in the current Arc because of the 'if behaviour, but between making the language works for me [change its behaviour] or works for the language [change my behaviour], you'll guess what I prefer), and no, this is too precise. What I want is 'each to be smart and have an "anonymous form". Which tricks to use to get that, I don't care. I don't care of 'defined, this thing should certainly not exist anyway.

> and that, as far as I know, Arc isn't able to do for you.

You're certainly right here.

-----

2 points by CatDancer 3667 days ago | link

> Everyone here seems to ignore this eventual issue and focus on the precise problem of 'each, but this was not my point initially.

OK, well, your first example was:

  (if t 1 (amac))
and you complained that the "amac" macro was being expanded at compile time. What if the first argument to 'if isn't known until run time? What if I said

  (if x 1 (amac))
perhaps at run time 'x is going to be false. So we have to expand 'amac, in case that branch of the 'if is invoked.

-----

2 points by palsecam 3667 days ago | link

> So we have to expand 'amac, in case that branch of the 'if is invoked.

But NO!

   cl> (defmacro m () (print "evaluated!"))
   M
   cl> (setf x 2)
   2
   cl> (if x t (m))
   T
   cl> (defun test-m (use-mac?) (if use-mac? (m) nil))
   "evaluated!"     <--- hmmm...
   TEST-M
   cl> (test-m t)
   "evaluated!"
   cl> (test-m nil)
   NIL              <--- ...no it's OK! (no "evaluated!")
Since 'if is a special form we can expect it to do magic. Common Lisp's 'if does, can we agree on that? (I mean seriously. I'm no expert and I don't really know about all this stuff, but I just see some facts. I totally might be wrong).

-----

2 points by CatDancer 3667 days ago | link

Curiosity got the better of me, so I installed Common Lisp.

The (print "evaluated!") itself evaluates to "evaluated!", so here

  cl> (test-m t)
    "evaluated!"
what you're seeing is the value that 'm expands into, not the macro being expanded at that point. If you use something like,

  (defmacro m ()
    (print "evaluated!")
    42)
you'll be able to see what's happening more easily.

-----

1 point by palsecam 3667 days ago | link

Hi hi, nice that you install Common Lisp by curiosity!

However, I don't understand what you want to demonstrate with your macros.

   cl> (defmacro m () (print "evaluated!") 42)
   M
   cl> (if t (m) 43)
   "evaluated"
   42
   cl> (if nil (m) 43)
   43
Ah maybe you mean, when defining the 'test-m function?

   cl> (test-m t)
   42    
Kind of "strange" there is no "evaluated", is this the point? This is a point.

-----

2 points by CatDancer 3667 days ago | link

Yes,

  cl> (test-m t)
  42
is the behavior that I'd expect to see, since the macro was expanded at compile time, when test-m was defined.

The behavior of 'if at the REPL is interesting, perhaps it is to support conditional compilation? However it is a special case, since only at the REPL would we be able to evaluate the test argument to the 'if before compiling the then or else forms. Doing so little as wrapping the 'if statement in a lambda causes the macro to be expanded regardless of the test argument:

  > ((lambda () (if t 3 (m))))
  "evaluated!"
  3

-----

1 point by palsecam 3667 days ago | link

OK here. You're right about the REPL vs normal time eval. The lambda example says everything.

So the 'each of my dreams will be very difficult to have :-D! Maybe with lazy evaluation/JIT compilation [hmmm maybe not needed, see edit below].

Thanks for everything CatDancer!

EDIT:

Lol this stuff has no end. CatDancer I really enjoy all this, but it is late here and I will have to go to bed, seriously!

OK, so no REPL.

In if.lisp:

   (defmacro m () (print "Hello via mac!") 42)

   (defmacro fm ()  ; remember, myeach would also be a macro...
     `(if (eq (read) 'CALLMAC)
        (m) 
        (print "Hello via func")))

   (fm)
On command-line:

   paul@polo-laptop:~ $ echo 'CALLMAC' | sbcl --script if.lisp
   "Hello via mac!" 
   paul@polo-laptop:~ $ echo 'no call' | sbcl --script if.lisp
   "Hello via func"
But yes, if 'fm is a function instead:

   paul@polo-laptop:~ $ echo 'no call' | sbcl --script if.lisp
   "Hello via mac!" 
   "Hello via func"

-----

2 points by CatDancer 3667 days ago | link

Oops, I should have said "top level" not "REPL". My mistake!

"Only at the top level would we be able to evaluate the test argument to the 'if before compiling the then or else forms".

> So the 'each of my dreams will be very difficult to have

Well you know, the Arc compiler knows whether a variable is in the lexical scope. So you could implement your myeach in ac.scm, not using macros, but instead compiling to Scheme.

Or... how about...

  (mac ech args `(each _ ,@args))
:-)

-----

1 point by palsecam 3667 days ago | link

No sorry me because my examples should be better. I use the command line but yes it's still "top level" and this is not enough to test this, right?

But here; OK, if 'fm is a function, the behaviour is "like in Arc". But if it's a macro, it seems to behave "like I want". Or is my example not good? (Sorry I'm really tired here, I can't even realize this :-D).

I mean, don't you think 'myeach as a CL macro would work?

> (mac ech args `(each _ ,@args))

Actually I'm currently using that. I'm just using the name 'each_ instead and define it using the macro currying stuff of twilightsentry (see id=10139, very interesting).

   arc> (= each_ (>_ each '_))
But still you know, one extra character ('_') to type... :-D

-----

2 points by CatDancer 3667 days ago | link

> But if it's a macro, it seems to behave "like I want". Or is my example not good?

Shall I wait until tomorrow to reply so that you can get some sleep? :)

Try

  ((lambda () (fm)))
you can see that only if (fm) is invoked directly from the top level will the macro expansion be avoided.

> I mean, don't you think 'myeach as a CL macro would work?

What you need, in CL or in Arc, is a way to tell if a variable is defined at macro expansion time. Suppose Arc could tell you at macro expansion time if a variable was defined. Then you could say

  (def myeach (first . args)
    (if (defined first)
      `(each _ ,first ,@args)
      `(each ,first ,@args)))
no magic needed in 'if, in either language :)

-----

1 point by palsecam 3667 days ago | link

> Shall I wait until tomorrow to reply so that you can get some sleep? :)

Lol, yes I definitely go to bed now :-)

> ((lambda () (fm))

Nice demonstration indeed.

> What you need, in CL or in Arc, is a way to tell if a variable is defined at macro expansion time.

Right. I may try to modify Arc in this purpose one of these days.

> no magic needed in 'if, in either language :)

Magic definitely sucks :-D!

Thanks a lot CatDancer!

-----

2 points by CatDancer 3667 days ago | link

> can we agree on that?

No, because I don't know what CL is doing.

-----

1 point by palsecam 3667 days ago | link

Well, if I dive deeply into CL one of these days, I'll tell you :-), but this not in my plans for the near future :-/

You are right to refuse "magic".

For Arc, oh this is way too much difficult for me, but if I had to try to solve the problem, I think I'd try to add a level of abstraction between the Scheme runtime and the Arc one (?!) Macros currently, as you know, directly run in the Scheme environment (am I all right?). I'd try to avoid that, and have more control over them by "compiling" them to a special form (`ar-macall') but not directly execute them (?!) And so 'if could then have a "higher priority" than macro expansion (?!)

But these are pure ideas in the wild. I am not able even to try, so don't listen this crap :-D

-----

2 points by CatDancer 3667 days ago | link

> try to solve the problem

which problem, the 'myeach problem or the 'if problem?

-----

1 point by palsecam 3667 days ago | link

Hi hi, this thread is definitely too big now :-D

The 'if/macroexpansion problem.

-----

2 points by palsecam 3668 days ago | link

For info, the Common Lisp example:

   * (defmacro amac () (format t "amac is reached"))
   AMAC
   * (if t 1 (amac))
   1     <-- OK, no call to 'amac has been done
Another example where the Arc behaviour is "harmful":

   arc> (mac msleep (t) (sleep t))
   arc> (if t 1 (msleep 3))
   ... sleeps 3 seconds ...
   1

-----

2 points by CatDancer 3668 days ago | link

I don't find this buggy.

Perhaps you want

  (mac amac () `(prn "amac is reached"))

-----