The problem with Arc as opposed to Clojure is that it is incomplete. It is missing some of the most basic features such as a a module system, a good debugger, and useful error messages. The reason none of us has come up with a solution to any of these deficiencies is that they are hard problems solve. We are all just waiting for pg to try to fix these problems instead of going out and solving them ourselves. If someone can solve these issues, pg or not, Arc should gain traction. I hate to say it, but if no one is going to do anything, Arc might as well already be dead.
I'm certainly not waiting for arc to become perfect. My side project may well end up needing me to reinvent a computing stack from scratch: http://akkartik.name/about. But I fully expect it to take 20 years, so don't hold your breath :)
Are you fully sure there is no other way to do what you want to do without it taking 20 years?
That's a question worth pondering, not just because big things tend to fail or good design is simple, but also because life has a way of changing what people choose to do over time.
Yeah, I'm actually constantly looking for shortcuts, ways to not need to reinvent the stack, ways to build things that will help existing projects, or allow people to use existing languages and tools.
When I said 20 years I wasn't saying I have a plan that's going to take 20 years. I meant that I'm building small projects that -- when I step back -- seem to be headed in that general direction over time. Projects like wart[1], tracing tests[2] and layers[3]. I'm building these little projects out in the open, so hopefully I won't fall victim to the "hacker comes out of basement after 20 years with a compiler and OS that nobody cares about" failure mode. No, my goal is to fail in some more creative way :)
I'm less worried about my goals changing. On the one hand, if they do they do, one can't be anything but fatalistic about it. On the other hand the goals I outlined above seem to have hooked into my motivation at a deep level. I may well end up switching solutions I explore, but it seems like the goal is unlikely to change.
I haven't really started building anything for the past couple of months, but I've been thinking lately about three potential little projects: a) a simple multitasking OS without virtual memory or processes, maybe even without preemptive scheduling, b) a better language for teaching programming by separating scope from the notion of functions, and c) an assembly language with structured programming and lisp macros somehow. Something 'easy' to build without needing an entire C compiler. Rephrased: "I'm willing to do my own register allocation. What can you do for me, lisp?" :)
Mostly I just get up every morning and work on whatever I want. And these are the things I've been ending up working on.
Since you are not running arc from the arc folder, you need to use either the full path, or the path relative from where you are running arc. If you are running arc in dir/foo/ and the file is in dir/bar/lib/http.arc, you could use:
I just quickly tried wart and it does seem pretty impressive to me. The only thing I'm worried about is the rules for how something is grouped. I may just need to use it more to see if it's an actual issue, but I feel that having all of the different rules can make it very easy to write hard to find bugs. Other than that I find wart to be a very interesting idea which will be worth checking out.
Also is there any way to see what my code looks like if I were to type it using parenthesis?
1. "Multi-word lines without leading parens are implicitly grouped with later indented lines."
2. "Indentation-sensitivity is disabled inside parens."
That's really all there is to it.
> is there any way to see what my code looks like if I were to type it using parenthesis?
It's a little clunkier than visualizing infix, but you can introspect on any function or macro. Here's an example session:
$ ./wart
ready! type in an expression, then hit enter twice. ctrl-d exits.
def (foo x)
if a b
c d
:else e
foo
=> (object function {sig, body, }) # wart doesn't print table values by default
(body foo)
=> ((if a b (c d) (:else e)))
Here you can see that turning (c d) and (:else e) into calls is probably not desired.
I kept looking and found that read has the functionality I was looking for.
(read)
5+5
=> (+ 5 5)
I also noticed that if I want to generate lists which are expressions, it is not as straightforward as it should be (I'm trying to get a list of the number 5 the symbol + and 5 again but wart for whatever reason rearranges them).
Yeah, that's a bug. I haven't yet bothered to create a version of read without the infix transform. For now you have to:
'(5 (+) 5)
If you decide to try to fix this I'd love to hear your experiences. The point of wart was to be easy to hack on, but I'm losing steam because it's been hard to get feedback on that score.
I'm going to have to familiarize myself with the internals before I actually try to fix it. After looking for a little bit, I think the issue is in the transform_infix function where it handles the quoting.
if (is_quote_or_unquote(n.elems.front())) {
list<ast_node>::iterator p = n.elems.begin();
while (is_quote_or_unquote(*p)) {
trace("infix") << "skipping past " << *p;
++p;
if (p == n.elems.end()) {
trace("infix") << "just quotes/unquotes " << n;
return n;
}
}
I would think that after you determine that the first element is a quote, you would just ignore the rest instead of going through and checking which ones are also quoted. Since I'm not that familiar with c++ or how the internals of wart work, I'm going to have to leave it to you to see if this is actually the problem.
Thanks for the investigation! Yes, that would be the way to disable transforms inside a quoted s-expr.
But I think that still would leave issues. For one, it is approximately as likely that a list innocuously contains a literal '+ as that we're constructing a fragment of code that is eventually intended to be eval'd. We need a way to say, "this is code" or "this is never going to be eval'd." A second issue is that quoting isn't the only way to read data. Imagine using read to read a list from a file. How would we suppress infix there?
I actually think reading data from a file is the bigger issue. Small quoted lists in code will be noticed, and can be replaced with some (klunky) combination of list and cons. It's far worse if you have a multi-megabyte file that silently gets corrupted because of one character.
I'm just wondering if the second rule is really such a good idea. If one wants to use indentation-sensitivity within parens, it is impossible. I'm not sure how often trying to write code like that would come up in practice, but I think the programmer should have the option in that case of whether indentation-sensitivity is actually being used (by using brackets or something else to make it clear).
Ok, interesting idea that I hadn't considered before.
As it happens, I've been watching a discussion about a different use for brackets: http://lambda-the-ultimate.org/node/4879. Lots of people (scheme, clojure, this liso guy) seem to have an idea about what to use brackets for, and it's not clear what the best use is for these precious punctuation characters.
I realized that we actually don't need to have a closing bracket to signal the end of the expression. We just need to implement something like the $ operator in haskell.
def (foo a b c)
if $ and
a = 5
b = 10
c = 15
prn "success"
prn "failed"
There should be enough information based off of the indentation to tell how it should be parsed. $ should mean something along the lines of everything indented after the next symbol is a single expression.
You'll like the old discussion on Bullet which had some similar ideas, including considering how to suppress paren-insertion inside such a special operator, etc.
Even though I think this is a good idea, I'm pretty sure there is better way to do it. I feel like your solution is only a solution to a very specific problem. Your solution is almost the exact same as keywords, only its not. You are just adding extra information into the function call (the type of argument). Instead we could use keywords for a solution to this, and as a solution for many other problems for which this kind of scheme wouldn't work.
I do like the idea of replacing a ton of functions with functions that do different things depending on their arguments.
I thought about it some more. Here's why I think the wrapping approach is superior to a new arg. Let's say you define a function on lists:
(def foo (x)
(... x))
Later you want to use foo, but traversing nested lists like it's a tree. Ok, so we'll add a new version:
(defextend foo (x y) (is y :tree)
(... x y))
But now calls to the base version will stop working; defextend passes both args to the base version which is only expecting one arg. To make things work you have to modify the base case, and teach it about an arg it doesn't need to know about. That seems unnecessarily ugly.
Another problem is the need to sequence args. If you create a new version with yet another arg:
(defextend foo (x z) (...)
(... x z))
Now you have to worry about the second arg being used for wildly different uses, possibly interfering with each other and causing bugs, or having to provide unnecessary args:
(foo a nil :widget) ; don't traverse a like a tree
Edit: ah, I just thought of one counter-argument. If you have multiple tree arguments you have to wrap each of them in my approach. This is more apparent when trying to support dotted lists in map (fallintothis's original complaint: http://arclanguage.org/item?id=18247)
(map + (dotted '(1 2 3)) (dotted '(1 2 . 3)))
---
Now, arc could be more flexible. My wart language (http://akkartik.name/post/wart) is more like javascript: excess args are silently ignored, and missing args are silently passed nil. Also, it has keyword args, so different versions might be called as:
(foo a :traverse-as tree)
(foo a :like widget) ; don't traverse like a tree
In such a situation the difference seems more minor.
The thing is, I don't want a new arg. I think we would be better of using keyword arguments, similar to ones as you mentioned in wart[1]. The reason I'm making such a big deal out of this is why create a new solution (tagging the arguments as they are passed in) when we can already can do this with keywords which are much more flexible. With keywords we can call them with anything we want: a list, a symbol, or even some sort of complex data structure.
(foo :a 'tree)
(bar :b '(1 2 3))
(baz :c (obj a 'b c 'd))
We can use keywords to pass in many different kinds of arguments. If we were to just tag the args, we could pretty much only pass in different types, not much else.
(foo (tree x)) ; how would you change this from a type to a number?
We can also implement them in such a way that when a function is called with extra keyword args, it won't change the behavior of the program (we would probably want it to log some kind warning). This way when we extend foo, it doesn't matter if we pass the original the keyword arg. As far as the original foo knows, the keyword arg doesn't even exist.
Sorry if I confused you. I don't want extra args, I want keyword args.
That makes sense! I've long been a proponent of keyword args as you can see. My love for them goes back all the way to one of my first posts here: http://arclanguage.org/item?id=10692. I think I've implemented them three times so far. I even have a version of arc with them: http://github.com/akkartik/arc. But I haven't updated it lately. I'll spend a few hours and merge in all the new stuff in anarki over the past year.
That's great! I just googled "Paul Graham arc no keywords" and found in one of his early writings on arc, he had a plan to replace keywords with something called "get parameters".
Ok, I've updated http://github.com/akkartik/arc. It was a useful exercise, because it turns out it has more tests than anarki, and I found several bugs in my work yesterday :) I've copied the tests to anarki as well.
It doesn't really need to know about a new arg, but there's no way to pass it in automatically because there's already a rest arg: body. Ok, so let's try that:
(mac each (var expr ? as nil . body)
`(walk ,expr :as as (fn (,var) ,@body)))
The problem with this is that you need to explicitly provide the optional arg, or it gets taken out of body.
I had this old idea that rest params should take precedence over optionals, but it turns out to have fallen by the wayside: http://www.arclanguage.org/item?id=13083. Time to bring it back here.
Well, the problem is you are trying to make keyword args the same as optional args. It might be better if you make make keyword args and optional args two completely separate kinds of arguments (similar to how common lisp has both &optional and &key). This way you would define each as:
(def each (var expr : as nil . body) ; I used : to represent keyword args
`(walk ,expr :as as (fn (,var) ,@body)))
This way calling each like
(each x '(1 2 3) x)
will expand correctly because keywords only take a value when they are passed in explicitly. It would have to be called as
(each x '(1 2 3) :as x)
to get the error you had. If you need to define a function that takes both optional args and keyword args:
(def foo (a b c ? g h i : j k l . rest)
...)
This way a-c are required, g-i are optional, j-l are keyword, and rest is everything else.
As for the problem you had with needing to redefine each (it is also mentioned by rocketnia in the post you linked[1]), each is just an interface for walk, so if a new feature is added to walk, each need to be redesigned. There might be some way that we can define some sort of macro, interface, which can be used to create a function/macro which is a very thin layer over another function/macro. As for how interface is to be designed, I'm not sure.
Well, it seems more general to let any arg be acceptable as a keyword.
rocketnia's concern I addressed by passing through unknown keyword args unchanged:
(def foo (a b)
(list a b))
(def bar args
(apply foo args))
arc> (bar 3 :a 4)
(4 3)
But that doesn't work in this case because each needs to treat as specially. I don't think that's avoidable.
We _could_ include a new category of 'required' keyword args like you suggest, yes. Let me know if you think of a situation where just giving rest args precedence isn't enough. Things are already pretty complicated with rest, destructured, optional, and keyword args, so I'm reluctant to add another delimiter and category.
"rocketnia's concern I addressed by passing through unknown keyword args unchanged"
The reason I'm okay with this approach is because it localizes the complexity to each function. This way I don't dislike the language; I just dislike the idioms currently being used in it.
Consider the use of flat text formats for shell scripts, HTTP headers, CSS properties, XML attributes, etc. They're usually convenient enough to type out manually, and that also makes them convenient to generate in simple cases, but writing a parser for them can be tough due to the same syntactic sugars that made them convenient to generate in the first place. Writing a full-featured generator or parser can also be tough, because sometimes future versions of the text format may allow it to encode more information.
With keyword arguments, the syntax for a function call is more complicated and sugary, putting programmers like me in the position of wondering what kind of headaches they should go through to parse or generate this format. If the headaches are unclear, combining programs becomes hard, because each developer might opt for different headaches, at which point I think they overlap in the form of complex glue code. :)
I have a greatly-enhanced respect for this perspective over the past month. Look at my travails over the past day trying to get optional, keyword and destructured args to play well together: https://github.com/akkartik/arc/compare/342ebf57a8...e347973.... Or my travails over the past month trying to build macex in wart in the presence of optional, keyword and destructured args AND quoted AND aliased params AND already-eval'd args[1] AND incompletely-eval'd args[2]: https://github.com/akkartik/wart/compare/30025ee25c...cabaa0.... Clearly there's some point at which I jump the shark :)
[2] related to partial eval: https://github.com/akkartik/wart/commit/8239ed9d21. I've been struggling intermittently with partial eval for two years now. This now year-long attempt might well be utterly useless.
---
Perhaps the easiest to understand and most serious extant example of the knots I end up tying myself into:
arc> (iso :x :x)
nil
This is because x is one of the params of iso. Even more serious, this happens even if you try to compare a variable that evaluates to :x. Or a variable that evaluates to a large structure containing :x somewhere in it. Ugh! The reason for this is that I'm extracting keyword args at function-call time inside each function. This:
It took me three years to notice yesterday that I'm extracting keywords from post-evaluated arguments. Ugh! Urrrgggh!
Weird thing is, even wart has this problem. Even though I would superficially seem to be extracting keyword args pre-evaluation. I still need to debug this. Or perhaps I shouldn't debug this. Perhaps I should just throw up my hands and delete all my repos and become a hermit.
We should be able to specify the type of the function as part of the function. We shouldn't need to specify it in the arguments. I just think of it as we are walking over a tree as opposed to calling walk on an object that is a tree.
What I'm trying to say is the keyword arg shouldn't apply to a specific argument. It should apply to the function call as a whole.
I am having some issues with map. Since map only works on proper lists it doesn't work on all of the code. I am trying to change the code so that it recurs on tree instead but I'm having some trouble doing that.
I managed to get a version to work using treewise.
(= defmacro mac)
(defmacro mac (name args . body)
(let uniqs (table)
`(defmacro ,name ,args ,@(treewise cons
[if (auto _)
(or= uniqs._ (uniq _))
_]
body))))
(def auto (exp)
"Tests whether an expression should be autogensymed"
(and exp (atom exp) (endmatch "%" (string exp))))
I'm wondering if we should allow tree-subst (or at least a new function like tree-subst) to use functions in which case the code using treewise above could be written as:
Edit 40 minutes later: Everything looks great![1] The unit tests in arc.arc.t all pass, and the HN server comes up fine. I tried submitting, editing, commenting, everything works.
Edit 75 minutes later: I've added your tree-subst suggestion[2]. Good idea, thanks!
I realized the tree-subst is going to have to be a little more complicated. If the function isn't expecting a list it's going to throw an error.
(tree-subst even [+ _ 1] '((1 . 2) . (3 . 4)))
This code throws an error even though it is very clear what the result should be. I'm not sure what the best way to fix it is. I managed to write a version that does work but it looks pretty ugly.
(def tree-subst (old new tree)
(with (test (testify old)
f (if (isa new 'fn) new (const new)))
(if (no tree) '()
(atom&test tree) (f tree)
(atom tree) tree
t (cons (tree-subst test f (car tree))
(tree-subst test f (cdr tree))))))
I just realized there probably is a better way to write it using treewise. Something like:
(def tree-subst (old new tree)
(with (test (testify old)
f (if (isa new 'fn) new (const new)))
(treewise cons [if (test _) (f _) _] tree)))
But on the other hand tree-subst compared subtrees using is until my change today, and there are no calls to tree-subst in the repo. So I suppose there's no way to see what the original authors had in mind. Anybody else have use cases to share?
On the third hand, it's easy to use tree-subst the way you want with a slight change to the call, like I showed in my tests:
If you want to do substitution using a function, it seems to me like the operation is more of a roundabout map. Particularly if it's just over the atoms of a tree, it's easy to think of the higher-order functions from a different angle, instead of trying to make tree-subst do everything:
Other name ideas: treemap, maptree, hyphenated versions of any of those, map-leaves.
Digression: the issue I've been more annoyed by in Arc's suite of "tree" utilities is sloppiness with direct cdr recursion. Functions have no way of discerning whether some list (a b c) they're looking at is a "top-level" form or if it's just from recursing on cdrs of (x y z a b c). E.g., using ontree in http://arclanguage.org/item?id=18217:
arc> (let if 2 (+ if (do 2)))
4
arc> (count-ifs-with-do (list '(let if 2 (+ if (do 2)))))
1
arc> (ifs-with-do (list '(let if 2 (+ if (do 2)))))
((if (do 2)))
arc> (ifs (list '(let if 2 (+ if (do 2)))))
((if 2 (+ if (do 2))) (if (do 2)))
I usually want the recursion to instead occur more like
(def ontree (f tree)
(f tree)
(unless (atom tree)
(each subexpr tree
(ontree f subexpr))))
Or even just have the above as a separate function, like on-exprs or something.
I'm pretty sure we have to do the car/cdr recursion because functions like map and each don't work on pairs, they only work on proper lists and most trees are not proper lists. Code such as the following wouldn't work mainly because of the use of map/each.
(treemap [cons 'a _] '(a . a))
I'm pretty sure we would also only want treemap to work only on atoms (your first example), it doesn't make much sense to apply the function then recur on the new tree (I find it very hard to figure out what is going happen). The problem with this is we can no longer call functions on subtrees. I do like the idea of treemap which could be written more cleanly as:
(def treemap (f tree)
(treewise cons f tree))
I'm pretty sure we should stick to akkartik's implementation of tree-subst (we can actually remove the first clause because it is covered in the second clause) because it works just like all of the other higher order functions where they testify their input and it works on entire subtrees.
Note: you can use iff instead of writing a new function whenf
I'm pretty sure we have to do the car/cdr recursion because functions like map and each don't work on pairs, they only work on proper lists and most trees are not proper lists.
I think the obvious problem with that reasoning is it reinforces the notion that map, each, etc. "should" break on dotted lists. Even if that's the case, it's not like ontree & pals couldn't open-code a recursive function that worked on dotted tails; I merely used each/map in my example definitions as a convenience. If you prefer:
; Again, not that this has to replace the standard 'ontree...just can't think
; of a better name.
(def ontree (f expr)
(f expr)
(unless (atom expr)
((afn (subexpr)
(if (atom subexpr)
(and subexpr (ontree f subexpr))
(do (ontree f (car subexpr))
(self (cdr subexpr)))))
expr)))
arc> (ontree prn '(def f (x . xs) (cons nil xs)))
(def f (x . xs) (cons nil xs))
def
f
(x . xs)
x
xs
(cons nil xs)
cons
nil
xs
This versus the standard ontree:
arc> (ontree prn '(def f (x . xs) (cons nil xs)))
(def f (x . xs) (cons nil xs))
def
(f (x . xs) (cons nil xs)) ; <-- this potentially looks like a call to f!
f
((x . xs) (cons nil xs))
(x . xs)
x
xs
((cons nil xs))
(cons nil xs)
cons
(nil xs)
nil
(xs)
xs
nil
nil
On that note, I certainly find myself working around Arc's rampant disregard for dotted lists in my projects:
Note: you can use iff instead of writing a new function whenf
Not in vanilla Arc, I'm afraid. :) And I've always thought Anarki's iff is poorly named, due to the English collision with the "if and only if" abbreviation.
It does seem like a good idea, abstracting as a list of lists instead of a binary tree. The best name I can think of is onlol. I really don't think that we should extend map and each to work on dotted lists, instead we should create other functions treemap and treeach since dotted lists are actually trees and no longer lists.
Hmm, it sounds like we need three sets of primitives: for operating on lists, trees and code (like trees, but aware of call vs arg positions). Perhaps we could use the same names like map and count but first tag the arg as tree or code..?
I didn't notice that your version could check against subtrees. Another possibility would be to use errsafe along with the function (but I'm worried that might be too dangerous). Other than that I would have to agree with your design. I played around with the common lisp design of subst-if and found it does the exact same thing as yours but it also does not test nil. This can easily be done in arc in a similar way to atom&... with idfn&... I would have to go with your design for now but we might want to think about other designs such as whether or not to test nil.
I managed to implement a version that redefines mac. If people want mac to stay the same we could just switch them so defmacro becomes the new version and mac remains the same. I realized that I actually liked the fact that it autogensyms the symbols that are within an unquote because otherwise you would still need to use uniq in cases where you quote inside of an unquote and you need the symbols to be the same.
First of all thanks for finding 3, I only tested the code very in very simple cases.
In response to 2, when I am reading lisp code in the format you suggested, I cannot tell whether the else case is actually part of the clause above it or is actually an else case. I just use t to make it explicit that this is the else case.
And I agree with you for 6. We should just extend mac so that it autogensyms the code it is given. There is no harm in doing this since all of the code previously written should still work.
Yeah, I too like not relying on an implicit 'else'. In fact, I use :else instead of t to be even more clear.
This is all most excellent. Don't be afraid to commit and push! Feel free to override the default mac if there aren't any obvious problems. There aren't very many people using anarki, and we can always roll it back later if necessary.
