Arc Forumnew | comments | leaders | submit | Pauan's commentslogin
3 points by Pauan 503 days ago | link | parent | on: Does Arc Support Escape Sequences?

All of the string syntax that Arc currently uses:

Though this may change in the future, if pg switches away from Racket's reader. So I'd recommend not using any of the funky escapes.


That's probably because doesn't exist. My e-mail is


2 points by Pauan 504 days ago | link | parent | on: Introduce An Excited+Terrified Apprentice

"There are lots of words in the wiki that I don't understand."

Most of the Arc terminology is borrowed from other Lisps, so it's not really a problem with Arc; instead, it's simply that the Lisp culture is very old and has its own way of thinking and doing things. You get used to it.

If you have any questions, please do ask and we'll do our best to answer. Some of us here (including myself) started Arc with almost no Lisp experience, yet over time became quite good.


"What are some reusable trail maps that other people have experienced?"

Considering that I'm still learning new things all the time, and that the new things I discover have a tendency to demolish decades-old ways of thinking, I'm not sure there is a nice reusable "just read this and you'll be okay" kind of thing. I do know of at least one good resource for explaining what macros are and why they're awesome, but that's about it. The rest you either have to dig up, ask about, or learn on your own.


"How can I learn more about you folk?"

Well, it depends on how much you want to know us. If all you care about is the programming aspect, you can just hang around here and watch what we say and do. You can also ask.


"Why is the help button so small to get me to "

I don't know, that bugs me too. I didn't even know about that link until somebody pointed it out.


"Why can't I hover over the language and have it tell me how to refactor my thinking?"

Because good AIs don't exist yet. So it's up to us lowly meat-bags to do the thinking.


"How do I ask better questions?"

Everything happens with practice, which takes time and effort. If you really want something and you keep working at it, you'll see improvement. If you don't see improvement, then perhaps your methods are incorrect and you should seek better methods. But keep in mind that some things takes years of hard work, and there may not always be a way to speed it up.

Just focus on improving things bit by bit and after a long period of time, when you look back, you'll see amazing accomplishments. Every journey starts with a single step and all that jazz.


"Is there a periodic element of lisp-y things?"

Kinda. Lisps do share some things in common, like S-expressions, macros/vaus, code-is-data-is-code... but the Lisp culture also deeply values individuality and giving programmers more power. So despite some overlap, you tend to see a lot of variance between Lisps, as people try out different things.


"Why exactly can't we make improvements again? May I?"

We can't make improvements because pg has been too busy. However, there is a fork of Arc called Anarki, which anybody can change:

There are also various other forks of Arc, such as Rainbow, Rainbow.js, Jarc, and Arc/Nu:


"I believe there needs to be a search engine to help me answer that."

If you want to search for something on the Arc Forum, I've found the best way is to go to, and then do this: foo

"Why can't JS just be a Wart?"

Oh man I can go on and on about JS. The answer to your question is pretty long and detailed. If your question is, "why isn't JS better?", well... that's a history lesson that people smarter than me have already gone into:

For the record, they are working hard on fixing some problems with JS:

But things take time. Especially because they have to worry a lot about backwards compatibility.

Also, even though JS is worse than say, Arc, it is possible to write a compiler that takes some dialect of Lisp and compiles it to JavaScript... for an example of that, see my language Nulan:

There are also plenty of other languages that either use JS as a runtime, or compile to JS.


1 point by hugomelo 504 days ago | link

The best way for me to use the update box is to rewrite what I'm saying more clearly. Now I understand.

I just vomited every exciting idea I could. How exactly did you parse it into such a coherent roadmap so quickly?

Most people think of the map of software ideas as a flat map instead of a cube or sphere-like structure. What is the smallest possible technical / distributed guide to help me navigate this map a little more quickly?


1 point by Pauan 504 days ago | link

"They ought to have a gentler guide to show them that. It (might be) automatable, but at first it ought to be distributed. You folk obviously have lots of experiences doing things. Could this guide be possible?"

Are you talking about making a guide about guides, or a guide specifically to help people to understand Lisp? I'm sure either one is possible, but this particular community should have a much easier time with the Lisp guide than the general guide.

That actually sounds like a pretty good idea: a single consolidated guide to help people understand the Lisp way of doing things. Of course we've written plenty of stuff already (as have people not on the Arc forum), but it's scattered everywhere and isn't really coherent. Something like the "learn you a Haskell for great good" but for Lisp sounds nice. Is that what you're talking about?


"you can decide immediately(or within ~10min) if someone's cool."

Even if that's true, that isn't necessarily useful. "Coolness" is arbitrary and cultural. It shifts over time. It isn't necessarily tied to quality. And I think that people who seek coolness tend to produce lower quality stuff. After all, any time and energy spent being cool is time and energy spent not improving in non-cool (but useful) ways. Though I myself am guilty of sometimes doing things for the sake of reputation, I try not to do that.


"How exactly did you parse it into such a coherent roadmap so quickly?"

I'm just that kinda person: somebody who's quick to read into things. Unless it's rocketnia doing the talking, then I have a much harder time. :P


"Must I fork my own version of Arc to do it? Is there a simpler way?"

Yes. And as I mentioned, people have already done so. I personally recommend Arc/Nu, though Anarki should work fine too. Also, since the Arc source code is provided, you can of course modify your local copy as well.


1 point by hugomelo 504 days ago | link

I'd like to build myself the smallest possible technical guide. Most people think about software as a map. It's really a sphere. They ought to have a gentler guide to show them that. It (might be) automatable, but at first it ought to be distributed. You folk obviously have lots of experiences doing things. Could this guide be possible?


1 point by hugomelo 504 days ago | link

How can I build myself the smallest possible journal for better thinking? What are the most elemental materials to use (figuratively)?


1 point by hugomelo 504 days ago | link

It seems like the best way for me to use this thread is just as a place to ask myself better questions. You folk are incredibly effective guides as well though. I would love it if you chimed in whenever you liked :-)


1 point by hugomelo 504 days ago | link

I laugh when I see from here. I never did that from hacker news. Why?


1 point by hugomelo 504 days ago | link

What are the (smallestre-usablere-compose-able) checklists to think more like you?

Also, neat: åΩ (I never really touched the option key).


1 point by hugomelo 504 days ago | link

I don't know who's reorganizing my thoughts for me. I actually don't know if it's being done by software. But whatever you're doing, keep doing it.


1 point by hugomelo 504 days ago | link

Who can help me think more clearly the fastest?


-1 points by hugomelo 502 days ago | link

I'd like to define exactly how `Pauan` arrived at this answer in the community-managed fork of Arc. Is this possible? (yes no).


3 points by rocketnia 502 days ago | link

I'm sorry to do this since you edited your post, but here's what your post looked like when I first saw it. I cast a downvote, and I want to clarify why I did that.

"Because I can't tell if `Pauan` is human or not, I'd like to define exactly how `Pauan` arrived at this answer in the community-managed fork of Arc. Is this possible? (yes no)."

If I understand you correctly, you're asking if we could please open-source Pauan (or a subsystem thereof) for the sake of future development and discussion. No. Please don't undermine the voices of people on this forum.

- You're asking an in-depth question about someone in the third person, when they're right here to explain.

- It seems to me you're discriminating on the basis of someone's involuntary implementation details, without any obvious and respectable reason.

- You're suggesting to "define exactly" something which may be an essential secret ingredient in someone's appearance of individuality. As far as I'm concerned, you might as well suggest we zap someone's brain or hound them with paparazzi.

If you're conducting some kind of experiment in the design of cultural conventions, I (for one) don't yet understand and trust you well enough to play along. Would you mind introducing yourself in a more boring way for a while?


1 point by hugomelo 502 days ago | link

gladly. see Know what, I'll paste it here for you

Hugo Melo About :: Archives :: Subscribe

How To Become An Apprentice Developer

As an experiment in self-education, I deferred admission from university to teach myself how to code. I moved to San Francisco and took a nine week coding bootcamp. After that, I interviewed for three weeks and got four job offers and a contract offer on the forth week. I took a position as Carbon Five’s first apprentice. It’s been an amazing ride and the hardest thing I’ve ever done. Here’s what I learned.

Start With Hustle

App Academy had a full shower, a couch, and a Whole Foods nearby. So I lived at the office to get my burn rate down to about $20 / day for those two months. After App Academy, I crashed in the living room of a garage converted into two rooms at a Palo Alto hacker house. During the job hunt, I had ecstatic times where people were competing over me, and I got that hit of validation. I also had disheartening days where I didn’t know where I was going to live, doubting myself, and being universally rejected.

These experiences forced me to get clear on my motivations. On your journey, you must be able to take action and adapt, to do whatever it takes. Most people are unwilling to do this. That’s what’ll make you stand out from every other resume when you approach. This kind of hustle is the foundation.

Constant Networking

On the last day of App Academy, demo day, a bunch of companies came in to speed-date interview us. They were all nice, and a few were promising, but I didn’t get any leads out of it. My interviews came from networking: strategically meeting the right people.

I started off with going to interesting meet ups. I wanted to surround myself with smart, cool engineers and entrepreneurs with the power to make hiring decisions. With this in mind, there are lots of meet ups that were a waste of my time. ‘Networking’ events empirically sucked. Events where I’m volunteering and teaching someone else, or someone’s teaching me something were really awesome and useful. At interesting meet ups, I focused on connecting with cool people. I started off trying to sell myself to everyone. But approaching people in this value-sucking way was awkward and ineffective for me. Connecting with cool people was much more natural. Most smart engineers and entrepreneurs that I met were disarmingly humble. I couldn’t pick them out of a crowd at first glance because they weren’t talking about themselves. The key to meeting interesting people was to become interested in others. This made it easy to separate people who were building cool stuff from people who were bored and boring. So you’ve gone to cool meet ups, met cool people, you’ve got a ton of business cards and lukewarm contacts. Creepy ‘entrepreneurs’, boring monotone developers, and independent recruiting companies were the biggest waste of my time. Avoid them like zombies. Strategically, relentlessly follow up with the right people.

Give yourself a personal advantage by focusing on going from a high noise environment to a low noise environment. Think of it like a funnel. There’s a lot of noise in a cold email through a jobs page, less noise at a meet up, less noise at a private party, less noise through a warm introduction, less noise through a phone call, and undivided attention through grabbing coffee.

I much rather preferred that I meet someone in person. I don’t have any credentials, so I don’t look good on paper. Pushed for in-person, informal chats over coffee over phone screens. Push for building test-apps over algorithm brain teasers.

Master Interview Strategies

After in-person coding interviews, I asked people what they were looking for. Here’s what they said:

My ability to communicate concisely.

How well I can break down one big problem into many small, well-organized problems.

My ability to code.

How fun it is to teach me.

To these ends, pairing was the most valuable thing I did at App Academy. It taught me how to communicate clearly, how to share a keyboard, and how to be wrong and learn. The after-hour talks that Ned gave on algorithms turned out to be most of the CS theory I’d need to solve interview problems. I took some notes here.

I combined what I learned at app academy with side projects and practice sessions with other people. We went here for practice resources: Focus On Places That Are Good To Junior Developers You don’t have time to waste on companies that are set against hiring junior developers. Ask them what they’re looking for in a developer. After I told my story and got to know someone for a bit, I asked:

$%^&&%^&%^&W$%^$W%^#$¨ˆ˙¨˙øˆ¨øˆ©øƒˆ¨ƒ¬ß´˙ƒø¨˙ˆœ∑¥ˆ¨∑∂©ˆ©¨∑∂¬ˆ¨©∑∂ˆ¨œ©∑∂¬ˆ¨©œ¬∑∂ˆ¨©œ∑¬ˆ¨©¬∑∂ˆ¨©¬∑ˆœ¨∂©¬ What are you looking for in a junior developer? What are your technology needs and goals? Do you think I can help you reach those goals?

Before I started using these, I wasted a lot of time early on with companies that either didn’t want to or weren’t able to hire junior developers. These questions give me most of what I needed to know.

Early stage startups usually don’t have the resources to train a junior developer. I talked to a lot of startups where: “Hugo, you’re a kick-ass guy. But our lead dev doesn’t have enough time to divert the focus to mentor you.” I got this over and over again from really early startups. Large companies were inundated with candidates, and I was passed along to anonymous HR recruiters that expected credentials, and I was filtered out.

Large companies and two man startups were a waste of my time. I got a ton of rejections early on because I was talking to the wrong people. Then I changed focus to companies that were both able and willing to hire on growth. The middle turned out to be my sweet spot.

Sell Yourself On Growth

There are three ways to sell yourself: 1. Input. What degrees and accolades do you have? 2. Output. How much can you produce right now? 3. Growth. How fast do you learn? I had a special situation. I just got out of high school with no previous work experience and no credentials. But, I have the advantage: I started coding in June 2012 with the intent of becoming a professional. As a junior developer, my trajectory was more impressive than my skill set. I sold myself on growth - my ability to learn fast. I focused on my CV, and made my github my resume. I ignored most requirements on job pages. If I could solve the problems that position was aiming for, I applied. I experimented with these things. I didn’t know how people would react. It turned out that nobody cared. All of my offers came from either unposted jobs or jobs I wasn’t ‘qualified’ for on paper. When I tried to compete on input, I lost. When I tried competing on output, I got more attention, but I still lost. The people that made me offers said that my trajectory and enthusiasm were what won them over.

Focus On Learning

For a hungry junior developer, your ability to learn fast and efficiently is your most valuable asset. If you can meet your basic needs, learning is a more valuable asset than a marginally higher salary. Choose a place where you’re going to be surrounded by developers better than you that you can learn from. If you can do this, you’ll absorb a tremendous amount in a short amount of time. This will make you exponentially more valuable as a developer. That’s what it means to become an apprentice.

See all posts »


3 points by akkartik 501 days ago | link

Yeah I saw those and remember thinking they were nothing like his writings here.

Last night he started email-bombing me and others all sorts of irrelevant statements about google support, venture capital and whatnot. From midnight to 3am yesterday I received 27 emails from him. (I haven't read them all.)


2 points by hugomelo 499 days ago | link

my bad :(


3 points by Pauan 502 days ago | link

It's fine. We're talking in private now, so it shouldn't be a problem anymore.

Also, it's time that I leave the Arc Forum. You all are free to follow my projects on GitHub ( and can e-mail me at

Thanks for both the good and bad times.


0 points by hugomelo 502 days ago | link

what he said. Fuck you guys.


3 points by hugomelo 497 days ago | link

sorry about this :(


2 points by rocketnia 497 days ago | link

Thanks for the apologies. As for me, I wanted to stop you from getting carried away and potentially causing harm to people, but I'm sorry if I've given you a bad experience in the process.

I had seen that blog post of yours. I also particularly noticed this tweet you retweeted, which aptly summarizes a common theme between your blog and your Arc Forum posts:

  Founders: What are you doing right now? Is that thing a 10X
  improvement or a 10% improvement? Always work on 10X. Every
  minute counts.
This kind of mindset can be encouraging, but it seems like it's causing you negative stress. If so, I hope you can set aside some extra time for other things you like to do, even if they feel unproductive. I don't know if those things will be remotely on-topic for Arc Forum, but feel free to bring them up if they are. :)


1 point by hugomelo 486 days ago | link

You're completely right. I was wrong (and sleepy). This was distress, not positive straining. I did get carried away. I've set aside lots of time for Yoga, climbing, riding my (Xootr)scooter, and Jiu Jitsu. But man am I still excited :)

Also: Wore google glass at Fluent Conf. Coolest thing I've seen so far methinks.


Fixed in Arc/Nu:

Thanks for finding the bug, and the great idea for fixing it.


3 points by dram 512 days ago | link

Your fix is cleaner. :)

BTW, I think `(unescape-ats s)` also needs to be treated as the same.

So that it will not fail when atstrings is set.


1 point by Pauan 511 days ago | link

Ah yes, excellent catch:


"[...] or a way to add them easily?"

Arc/Nu used to have keyword arguments, but they were a pain in the butt to implement, and only a single function actually used them, so I took them out.

So, it's possible, yes, but not easy. At least, not easy using Racket's keyword arguments. It might be easier if I did everything from scratch...


You could also use Nulan's parser:

I specifically designed it so it can run stand-alone, without any dependencies on the rest of Nulan. It takes a string and returns a JS array of strings, numbers, and Symbols. You can easily write a simple function to transform the output so that it's compatible with wat.

You're also free to modify it, though I'm not sure what license to use for Nulan. It'll probably be either the BSD/ISC license or Public Domain.


It is true that symbols are unhygienic, but if you use boxes instead, then you get perfect hygiene. Unlike function/vau values, boxes can exist entirely at compile-time, so they're suitable for compilation to JavaScript.

For a working example, check out my language Nulan, which compiles to JavaScript and has hygienic macros that use boxes:

I can provide more details if desired. Though... you're exploring vau based semantics with wat, so I'm not sure how useful boxes would be for you.

In any case, I like what you've done with wat, though I've leaned away from vau recently.


2 points by manuelsimoni 529 days ago | link

So the idea behind boxes is that you can create boxes at compile-time for runtime stuff that doesn't exist yet?

Note that the same idea can work for Vau first-class values: at compile-time you simply create a faux value for each non-compile-time top-level (or other) definition. Macros can then insert them into generated code, even though they are not the real runtime object.


2 points by Pauan 529 days ago | link

Yes, at least in Nulan the boxes only exist at compile-time: at runtime they're normal JS variables. Of course, if you were writing a VM, the boxes could also exist at runtime, which would be faster than having to lookup a symbol: you would just unbox it!

And yes, I suppose it would be possible to use boxes in some way with vaus, especially because modern vau-based languages use lexical scope. This doesn't completely solve the "vau are slow" problem, but it at least avoids a symbol lookup for every single variable.

In fact, come to think of it... I had been playing around with this idea, but I had forgotten about it. The basic idea is to create a first-class environment/namespace, but rather than mapping symbols to values, it would map symbols to boxes.

The idea is that, this namespace exists both at compile-time and run-time, so that at compile-time it uses the namespace to replace symbols with boxes (thus gaining speed and the ability to statically verify things at compile-time), but using "eval" at runtime would first lookup the symbol in the namespace (which returns a box), and would then unbox it.

If done properly, I think such a system would appear to behave similarly to the normal environment system as used by Kernel, but it could potentially be faster, because most symbols can be replaced at compile-time with boxes. In any case, it's an implementation detail, unless the language decides to expose the boxes underneath.


3 points by manuelsimoni 528 days ago | link


My plan is to use these ideas not with fexprs, but with procedural macros, thereby gaining fully static expansion at compile-time, because I want a type checker, and type checking fexprs is currently not possible to my knowledge.

The macroexpansion pass would register fake bindings (very similar in spirit to your boxes) for all runtime definitions in the compile-time environment.

Macros then include these fake objects at compile-time (just as fexprs would with the real objects at runtime) into the generated code.

Thereby, I hope to maintain the pleasurable hygiene properties of fexprs in a language with preprocessing macros.


2 points by Pauan 528 days ago | link

Well then, that sounds exactly like what Nulan does. I'm glad to see these ideas spreading, whether independently discovered or not.

Since I've already explored these ideas, I would like to mention a couple things.


What you call "fake bindings", Nulan calls boxes. I like this name because it's short, unique, and intuitive. A box is simply a data structure that can hold a single item and supports getting that item and setting that item.

In other words, putting something into the box, and taking something out of the box. But changing the contents of the box doesn't change the box itself, obviously. And because they're mutable, boxes are equal only to themself. So even if two boxes hold the same value, they might not be equal to eachother.

Interestingly, boxes are exactly equivalent to gensyms:

  (define foo (gensym))  ; Create the box
  (eval foo)             ; Get the value of the box
  (eval `(set! ,foo 1))  ; Set the value of the box
And gensyms are equal only to themself. So they satisfy all the conditions of boxes, and in Nulan, gensyms actually are boxes! So another way of thinking about it is that Nulan replaces all symbols with gensyms.

Yet another way to look at it is that boxes are equivalent to pointers in C, but now we're straying a bit far away from Lisp...

Because I wanted Nulan to be hygienic by default, I decided to use boxes for everything. In addition, they're a first-class entity. That is, Nulan lets you run arbitrary code at compile-time, and also gives you access to boxes at compile-time.

And the way that Nulan implements macros is... basically, a "macro" is simply a box that has a &macro property. When Nulan sees a box as the first element of a list, it checks if the box has a &macro property (which should be a function), and if so, it calls it.

But boxes can have other properties as well, for instance &pattern lets you define custom pattern-matching behavior, &get is called when the box isn't a macro, and &set is called when assigning to the box:

  foo 1 2 3     # &macro
  foo           # &get
  foo <= 1      # &set
  -> (foo 1) 2  # &pattern
This means the same box can have different behavior depending on its context. And because these things are tied to the box rather than the symbol, everything is perfectly hygienic and behaves as you expect.

Boxes also have other properties as well, like whether they're local or global, and whether they were defined at compile/run-time. And the Nulan IDE ( understands boxes as well, so that if you click on a box, it will highlight it. So even though two variables may have the same name, the Nulan IDE knows they're two different boxes.

By the way, in Nulan, the correct way to write the "when" macro is like this:

  $mac when -> test @body
    'if test
       | ,@body
Basically, Nulan doesn't have "quasiquote". Instead, "quote" supports "unquote" and "unquote-splicing". In addition, "quote" returns boxes rather than symbols, so it is by default hygienic.

If you want to write an unhygienic macro, you can use the "sym" function, which takes a string and converts it into a symbol:

  $mac aif -> test @rest
    w/box it = sym "it"
      'w/box it = test
         if it ,@rest
I prefer this way of doing things because it reduces the amount of syntax (no quasiquote), it means everything is hygienic by default, and it allows you to break hygiene, but you have to explicitly use the "sym" function. Because unhygienic macros are rare, I think this is a good tradeoff.

Also, you might have noticed that the above two macros don't use "unquote". As a convenience, when using "quote", if the box is local (bound inside a function), it will automatically "unquote" it, meaning that these two are equivalent:

  w/box foo = 5

  w/box foo = 5
The reason for having "unquote" at all is for splicing in expressions:

  # returns (5 + 10)
  w/box foo = 5
    '(foo + 10)

  # returns 15
  w/box foo = 5
    ',(foo + 10)

Actually, I don't mind unhygienic macros that much, so "hygienic macros" isn't the primary reason I love boxes. The main reason I like them is because they make it really easy to implement hyper-static scope. And once your language has hyper-static scope, it's very easy to make an amazingly simple, concise, powerful, and flexible namespace system.

So boxes really kill multiple birds with one stone, in a really simple to understand and simple to implement way. They may not have the generality of vau or functions, but they're a practical data structure that does a lot for little cost.


3 points by shader 527 days ago | link

I can understand the logic for not using quotes in favor of simpler techniques for hygenic macros, but I still like having them as short hand for really common list manipulations. Or even just an easy way to make a symbol. Of course, I've also always wanted something like python's list and *dict for splicing a list onto the end of an argument set without having to do something ugly like (apply f (join (list a b) c)). (f a b @c) looks so much nicer to me...

Anyway, aside from aesthetics, boxes really do seem useful. Another thought that just occurred to me is that they should enable trivial implementation of setforms and "out args".

I.e., if (car (cdr (obj key))) returned a box, you would be able to just call '= on it, with no thought for how you found the value.

And for out args, all you need to do is have double-boxes for function arguments. If you access the outer box, 'get is passed through to the inner box and you just receive the value. If you just call '= on the outer box, the value gets shadowed like normal, because 'set would not be passed through by default. If you want to modify the external variable, all you have to do is unbox it and call 'set on the original box.

I feel like there might be a similar way to enable dynamically scoped functions with first class environments, but I'm not sure.


3 points by shader 527 days ago | link

I've been thinking about this some more, and it seems to me that boxes in the way Nulan uses them make for a completely different evaluation paradigm than the traditional metacircular evaluation.

In the traditional evaluation scheme, using environments, symbols, read, eval, and apply, the process is as follows:

  1) read turns text into code (lists of symbols and values)
  2) eval looks up a symbol in the current environment, or expands a macro, and calls apply to get the value of a function
  3) apply calls eval to get the value of the arguments, and then returns the value of the function
In a compile-time based box scenario, the environments go away and read/eval change to something like this:

  1) parse the new code into lists of appropriately boxed values, and expand macros
  2) recursively apply functions to values to get the final result.
More of a traditional compile/run separation. The main difference to note in the box formulation is that there is no such thing as an environment. The 'hyperstatic scope' that Pauan keeps mentioning is another way of saying, 'there is no scope', only variables.

Thus, there is no real way to make variations on the scoping scheme using hyperstatic scope, because that's all a read/compile time feature. Dynamic scope is impossible without tracking environments, which are otherwise unnecessary.

Now, if one were to use both first-class environments and boxes, one can theoretically have the best of both worlds, with only a little extra overhead, and most of that at compile time. Flexible scoping becomes possible by specifically referencing the variable in the environment instead of using the default boxed values. They would still be boxes, just referenced by symbol in the environment table.

Now what I'm wondering is whether there is any useful equivalence to be found between a box and an environment. Heck, just make a box an object that has a few hard coded slots (that can be accessed by array offset) and a hash table for the rest, and voila! It can replace cons cells and regular hash tables too :P All we need is a reason for making everything use the same core structure...


2 points by Pauan 527 days ago | link

Yes, that's intentional. Because I wanted to compile to fast JavaScript, I chose an evaluation model that has a strict separation between compile-time and run-time.


"The 'hyperstatic scope' that Pauan keeps mentioning is another way of saying, 'there is no scope', only variables."

Actually, there is still scope. After all, functions still create a new scope. It's more correct to say that, with hyper-static scope, every time you create a variable, it creates a new scope:

  box foo = 1     # set foo to 1
  def bar -> foo  # a function that returns foo
  box foo = 2     # set foo to 2
  bar;            # call the function
In Arc, the call to "bar" would return 2. In Nulan, it returns 1. The reason is because the function "bar" is still referring to the old variable "foo". The new variable "foo" shadowed the old variable, rather than replacing it like it would in Arc. That's what hyper-static scope means.

From the compiler's perspective, every time you call "box", it creates a new box. Thus, even though both variables are called "foo", they're separate boxes. In Arc, the two variables "foo" would be the same box.


"Thus, there is no real way to make variations on the scoping scheme using hyperstatic scope, because that's all a read/compile time feature. Dynamic scope is impossible without tracking environments, which are otherwise unnecessary."

If by "dynamic scope" you mean like Arc where globals are overwritten, then that's really easy to do. As an example of that, check out Arc/Nu, which also uses boxes:

In the Arc/Nu compiler, it literally only takes a single line of code to switch between Arc's dynamic scope and hyper-static scope.

If by "dynamic scope" you mean like Emacs Lisp, then... actually that should be really easy as well. You would just replace the same symbol with the same box, and then use box mutation at run-time. Of course, at that point I don't think there'd be any benefit over run-time environments... In any case, I like lexical scope, and boxes work well for lexical scope.

Also, although Nulan has hyper-static scope, it does have dynamic variables. The way that it works in Nulan is, you create a box like normal...

  box foo = 1
...and then you can dynamically change that variable within a code block:

  w/box! foo = 2
Within the "w/box!" block, the variable "foo" is 2, but outside, it is 1. You can do this with any variable in Nulan.


"Now what I'm wondering is whether there is any useful equivalence to be found between a box and an environment."

No. And that's a good thing. I've found one of the major benefits of boxes is that they're a single stand-alone entity: each variable corresponds to a single box. In the environment model, you have a hash table which maps symbols to values, so you have a single data structure representing many variables.

The reason I prefer each variable being represented by a separate box is that it makes namespaces really really easy to design and implement. For instance, in Arc/Nu you can selectively import only certain variables:

  (w/include (foo bar qux)
    (import some-file))
This is really easy with boxes: you simply grab the "foo", "bar", and "qux" boxes and import them. But with environments, it's a lot harder, because the environment for the file "some-file" may contain all kinds of variables that you don't want to import.

You could take the values from the environment and copy them into the current environment, but now any changes made won't show up. With boxes, the changes show up. I don't think environments work well for namespace systems. But boxes work wonderfully well.


"It can replace cons cells and regular hash tables too :P"

I've thought about ways to replace hash tables with boxes, but I don't think it'd be especially helpful. Hash tables serve a different purpose from boxes, so it makes sense for a language to have both data structures. If you want a box, just use a box. If you want a hash table, just use a hash table.

Interestingly enough, both Nulan and Arc/Nu have a giant hash table at compile-time that maps symbols to boxes. This is basically the compile-time equivalent of the run-time environments that languages like Kernel have. Creating a new scope or changing the existing scope is as easy as changing this hash table, which makes it really easy to play around with different namespace systems.


2 points by rocketnia 527 days ago | link

"The main difference to note in the box formulation is that there is no such thing as an environment."

Doesn't step 1 need to use environment(s)? How else would it turn text into a structure that contains references to previously defined values?


"The 'hyperstatic scope' that Pauan keeps mentioning is another way of saying, 'there is no scope', only variables."

The hyper-static global environment ( is essentially a chain of local scopes, each one starting at a variable declaration and continuing for the rest of the commands in the program (or just the file). I think the statement "there is no scope" does a better job of describing languages like Arc, where all global variable references using the same name refer to the same variable.


"Flexible scoping becomes possible by specifically referencing the variable in the environment instead of using the default boxed values. They would still be boxes, just referenced by symbol in the environment table."

I don't understand. If we're generating an s-expression and we insert a symbol, that's because we want that symbol to be looked up in the evaluation environment. If we insert a box, that's because we want to look up the box's element during evaluation. Are you suggesting a third thing we could insert here?

Perhaps if the goal is to make this as dynamic as possible, the inserted value should be an object that takes the evaluation environment as a parameter, so that it can do either of the other behaviors as special cases. I did something as generalized as this during Penknife's compilation phase (involving the compilation environment), and this kind of environment-passing is also used by Kernel-like fexprs (vau-calculus) and Christiansen grammars.


"Now what I'm wondering is whether there is any useful equivalence to be found between a box and an environment."

I would say yes, but I don't think this is going to be as profound as you're expecting, and it depends on what we mean by this terminology.

I call something an environment when it's commonly used with operations that look vaguely like this:

  String -> VariableName
  (Environment, VariableName) -> Value
  (Environment, Ast) -> Program
Meanwhile, I call something a box when it's primarily used with an operation that looks vaguely like this:

  Box -> Value
This might look meaningless, but it provides a clean slate so we can isolate some impurity in the notion of "operation" itself. When a mutable box is unboxed, it may return different values at different times, depending on the most recent value assigned to that box. Other kinds of boxes include Racket parameters, Racket continuation marks, Racket promises, JVM ThreadLocals, and JVM WeakReferences, each with its own impure interface.

When each entry of an environment needs to have its own self-contained impure behavior, I typically model the environment as a table which maps variable names to individual boxes. The table's get operation is pure (or at least impure in a simpler way), and the boxes' get operation is is impure.

You were wondering about equivalences, and I have two in mind: For any environment, (Environment, VariableName) is a box. For any box, we can consider that box to be a degenerate environment where the notion of "variable name" includes only a single name.


1 point by Pauan 526 days ago | link

"Doesn't step 1 need to use environment(s)?"

I think he's talking about run-time environments a la Kernel, Emacs Lisp, etc.

Nulan and Arc/Nu use a compile-time environment to replace symbols with boxes at compile-time. But that feels quite a bit different in practice from run-time environments (it's faster too).


"Are you suggesting a third thing we could insert here?"

Once again, I think he's referring to run-time environments. Basically, what he's saying is that you would use boxes at compile-time (like Nulan), but you would also have first-class environments with vau. The benefit of this system is that it's faster than a naive Kernel implementation ('cause of boxes), but you still have the full dynamicism of first-class run-time environments. I suspect there'll be all kinds of strange interactions and corner cases though.


Slightly off-topic, but... I would like to point out that if the run-time environments are immutable hash tables of boxes, you effectively create hyper-static scope, even if everything runs at run-time (no compile-time).

On the other hand, if you create the boxes at compile-time, then you can create hyper-static scope even if the hash table is mutable (the hash table in Nulan is mutable, for instance).


2 points by Pauan 527 days ago | link

I'm not sure I understand your argument... aside from macros, how often do you use symbols? The only other case I can think of is using symbols as the keys of hash tables:

  (= foo (obj))
  (foo 'bar)
But in Nulan, hash tables use strings as keys, so that's no problem. As for your "really common list manipulation"... my system is actually better for that. Compare:

  (cons a b)                  ; Arc 3.1
  `(,a ,@b)                   ; Arc 3.1

  (list a b c)                ; Arc 3.1
  `(,a ,b ,c)                 ; Arc 3.1

  (join (list a) b (list c))  ; Arc 3.1
  `(,a ,@b ,c)                ; Arc 3.1

  'a ,@b                      # Nulan
  {a @b}                      # Nulan

  'a b c                      # Nulan
  {a b c}                     # Nulan

  'a ,@b c                    # Nulan
  {a @b c}                    # Nulan

"(f a b @c) looks so much nicer to me..."

I agree. Nulan supports @ splicing for both pattern matching and creating lists:

  # a is the first argument
  # b is everything except a and c
  # c is the last argument
  -> a @b c ...

  # works for list destructuring too!
  # this function accepts a single argument, which is a list
  # a is the first element of the list
  # b is everything in the list except a and c
  # c is the last element of the list
  -> {a @b c} ...

  # in addition to working for function arguments, it also works for assignment
  # a is 1
  # b is {2 3 4}
  # c is 5
  box {a @b c} = {1 2 3 4 5}

  # and of course you can nest it as much as you like
  box {a {b c {d} @e}} = {1 {2 3 {4} 5 6 7}}

  # equivalent to (join (list a) b (list c)) in Arc
  {a @b c}

  # equivalent to (apply foo bar qux) in Arc
  foo bar @qux
  # equivalent to (apply foo (join bar (list qux)))
  foo @bar qux

"I.e., if (car (cdr (obj key))) returned a box, you would be able to just call '= on it, with no thought for how you found the value."

In Nulan, you could do what you're talking about, except it would all have to be done at compile-time, because boxes only exist at compile-time. In addition, you wouldn't be able to set the box directly, you would instead generate code that sets the box at run-time, i.e. macros.

Nulan has the restrictions that it does because I wanted to compile to really fast JavaScript. Other languages (like Arc) don't have that restriction, so it should be possible to design a language that has boxes at run-time, in which case your idea should work.


3 points by shader 527 days ago | link

I actually really like symbols, and the existence of a symbol type in lisp is one of my favorite features. Technically in Nulan a 'symbol' is replaced by a 'box', and if your box had a string property called "name" that held the name of the variable they would probably be interchangeable.

Either way, I agree that your list tools are generally more useful than quote/unquote. The main things I use it for are 1) to get a literal symbol and 2) to get something like list splicing. It looks rather arcane and adds clutter though, so in most other cases I wouldn't use it.

If there was another way to just splice in a list in the middle of the code the way Nulan seems to, I might not feel as attached to it.


2 points by Pauan 527 days ago | link

"I actually really like symbols, and the existence of a symbol type in lisp is one of my favorite features."

I too like symbols. But I think if you examine why you like symbols, you'll realize that you like them because... most other languages don't have a first-class way to refer to variables. But in Lisp you can, using symbols.

Well, Nulan has both symbols (representing unhygienic variables), and boxes (representing hygienic variables). It's just that hygienic variables are so much better in so many situations that there's not much reason to make it easy to create symbols in Nulan.


"Technically in Nulan a 'symbol' is replaced by a 'box', and if your box had a string property called "name" that held the name of the variable they would probably be interchangeable."

Yes boxes have a name property. This is currently only used when printing the box. Yes you could convert from a box to a symbol, but Nulan doesn't do this, because I haven't found a reason to.


1 point by shader 526 days ago | link

Yes, that is part of the reason I like them so much. The other part is that they're a way to legally use bare words as part of the syntax.

If json had a symbol type, Wat would be a lot less ugly.


1 point by Pauan 527 days ago | link

Actually, there is ONE situation I've encountered where I would have liked to use symbols... in my playlist program, I have a list of file extensions which are considered "audio". Here's how it looks in Arc:

  '(flac flv mid mkv mp3 mp4 ogg ogm wav webm wma)
In Nulan, that would have to be written like this:

  {"flac" "flv" "mid" "mkv" "mp3" "mp4" "ogg" "ogm" "wav" "webm" "wma"}
Or perhaps like this:

  "flac flv mid mkv mp3 mp4 ogg ogm wav webm wma".split " "
To work around that, I wrote a short and simple "words" macro:

  $mac words -> @args
    '{,@( -> x "@x")}

  words flac flv mid mkv mp3 mp4 ogg ogm wav webm wma


2 points by akkartik 527 days ago | link

Wart uses the @splice notation. I've sung its praises before:


3 points by shader 526 days ago | link

This relates to yet another idea I think I've actually mentioned before.

Namely, making the lists that form the code/ast have reverse links, so you can mutate above the macro call level, instead of just insert arbitrary code in place. This wouldn't be feasible for general lists, as it is possible for a sub-list to be referenced in more than one place, but for code, each piece is generally considered unique even if it looks the same.

Anyway, this would allow for affects ranging from splicing to arithmetic and other much more evil and nefarious but possibly useful effects. I haven't thought through all of the implications, and I bet most of them are negative, but it would still be interesting to consider.

An implementation of intermediate splicing would be something like:

  (mac (list)
    (= (cdr list) (cdr (parent)))
    (= (cdr (parent)) list))
Where you replace (parent) with whatever technique would get the parent cons cell whose car is the macro call.


3 points by rocketnia 524 days ago | link

"An implementation of intermediate splicing would be something like[...]"

Which of these interpretations do you mean?

  (list 1 2 (splice 3 4) 5)
  (list 1 2 3 4 5)
  (list 1 2 (splice (reverse (list 4 3))) 5)
  (list 1 2 3 4 5)
I wrote the rest of this post thinking you were talking about the first one, but right at the end I realized I wasn't so sure. :)


"Namely, making the lists that form the code/ast have reverse links, so you can mutate above the macro call level, instead of just insert arbitrary code in place."

I'll make an observation so you can see if it agrees with what you're thinking of: The expression "above the macro call level" will always be a function call or a special form, never a macro call. If it were a macro call, we'd be expanding that call instead of this one.


For these purposes, it would be fun to have a cons-cell-like data structure with three accessors: (car x), (cdr x), and (parent x). The parent of x is the most recent cons-with-parent to have been constructed or mutated to have x as its car. If this construction or mutation has never happened, the parent is nil.

Then we can have macros take cons-with-parent values as their argument lists, and your macro would look like this:

  (mac splice list
    (= (cdr list) (cdr (parent list)))
    (= (cdr (parent list)) list))
Unfortunately, if we call (list 1 2 (splice 3 4) 5), then when the splice macro calls (parent list), it'll only see ((splice 3 4) 5). If it calls (parent (parent list)), it'll see nil.


Suppose we have a more comprehensive alternative that lets us manipulate the entire surrounding expression. I'll formulate it without the need to use conses-with-parents or mutation:

  ; We're defining a macro called "splice".
  ; The original code we're replacing is expr.
  ; We affect 1 level of code, and our macro call is at location (i).
  (mac-deep splice expr (i)
    (let (before ((_ . args) . after)) (cut expr i)
      (join before args after)))
If I were to implement an Arc-like language that supported this, it would have some amusingly disappointing consequences:

  (mac-deep subquote expr (i)
    `(quote ,expr))
  (list (subquote))
  (quote (list (subquote)))
  (do (subquote))
  ((fn () (subquote)))
  ((quote (fn () (subquote))))
  (fn (subquote) (+ subquote subquote))
  (fn (subquote) (+ subquote subquote))


2 points by Pauan 532 days ago | link | parent | on: Regular expressions in Arc

I would like to point out that although it's a Pratt parser, it's been specifically modified to work better with Lisps, meaning it operates on lists of symbols rather than on a single expression. I have not seen another parser like it.

This makes it much more powerful while also being much easier to use. Using the Nulan parser, adding in new syntax is as easy as writing a macro!

For instance, in Nulan, the "->" syntax is used for functions:

  foo 1 2 3 -> a b c
    a + b + c
The above is equivalent to this Arc code:

  (foo 1 2 3 (fn (a b c)
    (+ a b c)))
And here's how you can implement the "->" syntax in Nulan:

  $syntax-rule "->" [
    priority 10
    order "right"
    parse -> l s {@args body}
      ',@l (s args body)
As you can see, it's very short, though it might seem cryptic if you don't understand Nulan. Translating it into Arc, it might look like this:

  (syntax-rule "->"
    priority 10
    order "right"
    parse (fn (l s r)
            (with (args (cut r 0 -1)
                   body (last r))
              `(,@l (,s ,args ,body)))))
The way that it works is, the parser starts with a flat list of symbols. It then traverses the list looking for symbols that have a "parse" function.

It then calls the "parse" function with three arguments: everything to the left of the symbol, the symbol, and everything to the right of the symbol. It then continues parsing with the list that the function returns.

So basically the parser is all about manipulating a list of symbols, which is uh... pretty much exactly what macros do.

Going back to the first example, the "parse" function for the "->" syntax would be called with these three arguments:

  1  (foo 1 2 3)
  2  ->
  3  (a b c (a + b + c))
It then destructures the 3rd argument so that everything but the last element is in the variable "args", and the last element is in the variable "body":

  args  (a b c)
  body  (a + b + c)
It then generates the list using "quote", which is then returned.

Basically, it transforms this:

  foo 1 2 3 -> a b c (a + b + c)
Into this:

  foo 1 2 3 (-> (a b c) (a + b + c))
As another example, this implements Arc's ":" ssyntax, but at the parser level:

  $syntax-rule ":" [
    priority 100
    order "right"
    delimiter %t
    parse -> l _ r
      ',@l r
So now this code here:

  foo:bar:qux 1 2 3
Will get transformed into this code here:

  foo (bar (qux 1 2 3))
I've never seen another syntax system that's as easy and as powerful as this.

Oh yeah, and there's also two convenience macros:

  $syntax-unary "foo" 20
  $syntax-infix "bar" 10
The above defines "foo" to be a unary operator with priority 20, and "bar" to be an infix operator with priority 10.

Which basically means that...

  1 2 foo 3 4  =>  1 2 (foo 3) 4
  1 2 bar 3 4  =>  1 (bar 2 3) 4
Here's a more in-depth explanation of the parser:

Nulan's parser is powerful enough that almost all of Nulan's syntax can be written in Nulan itself. The only thing that can't be is significant whitespace.

Even the string syntax (using "), whitespace ( ), and the various braces ({[]}) can be changed from within Nulan.


2 points by Pauan 533 days ago | link | parent | on: Regular expressions in Arc

I've used that library, and I think it's really awesome. It feels very Arc-like while still supporting regexp stuff.


3 points by Pauan 533 days ago | link | parent | on: Arc DB

Because it's the simplest solution for Arc. Arc does not have any kind of SQL library, and it doesn't even have an officially supported way to drop down to Racket libraries, nor does it have any kind of FFI whatsoever. So flat files are indeed the fastest way to get up and running.

Perhaps in the long run it might be better to have some sort of way to connect to an SQL DB, but keep in mind that Arc is still a work in progress, and unfortunately has not been updated in a long time. For a while, Arc didn't even have Unicode support, because pg was working on more important things.

If you think of Arc as being a prototype, then it all makes sense. pg probably intended to eventually flesh it out into a full language, but I hear he hasn't had the time.


3 points by shader 532 days ago | link

If arc had support for something like DBA or SQLAlchemy built in, I might just have used it with either postgres or sqlite. However, neither of those databases really fit the arc data model very well, imo, because arc is very hash table and list oriented. Objects have very little in the way of a set schema, and hash tables map pretty well to... hash tables.

Anyway, I mostly want to leave all the objects in memory and use direct references between them; my data relations aren't that complicated, and explicit relations where necessary are actually fairly efficient. In fact, that's what most orm's a la SQLAlchemy seem to do; whenever an object is loaded, you can specify desired relations that also get loaded in memory, so you don't have to explicitly query the database each time.

Memory is cheap these days, and I was hoping for something that allowed versioning and perhaps graph-db features.


2 points by akkartik 532 days ago | link

Hmm, do you care about threading and consistency at all? If not, you could probably do everything with just arc macros over the existing flat file approach..


3 points by shader 530 days ago | link

I think that some form of scalability would be valuable, but that could easily be achieved with some sort of single threaded worker for each db 'server', and then have multiple instances running to provide the scalability. In order to make the single threaded semantics work well even in a multi-threaded application, I already have a short library for erlang-style pattern matched message passing.

Given the data volumes I've been planning on working with, I mostly want to use the permanent storage for history and fault tolerance, as opposed to live access. That could probably be handled in-memory for the most part. So maybe some form of flat file system would work without causing too many problems.

I originally started using git to effectively achieve that design without having to manage the trees, history, and diff calculation myself, but I've discovered that storing thousands of tiny objects in the git index may not be very efficient. I still think something similar is a good idea, but I would want to separate 'local' version states for each object from the 'global' version, so that it doesn't take forever to save the state of a single object. Maybe storing each object in a git 'branch' with the guid of the object as the branch name would work, since only one object would be in each index. The overhead for saving each object would be slightly higher, but it should be constant, rather than linear with the total number of objects.

Any obvious flaws with that idea that I'm missing? Have any better ideas or foundations to build off of?


1 point by akkartik 530 days ago | link

Building atop git is an interesting idea, and you clearly have more experience with it. Do you have any pointers to code?


3 points by shader 530 days ago | link

Here's the code I had written before, using the shell git interface to interact with the repo:

That code is pretty rudimentary, but allows low level access to git commands from arc, plus storage and retrieval of arc objects. After my previous comment though, I'll probably change it so that each object gets a separate branch, with 'meta branches' listing which branches to load if necessary.


1 point by akkartik 532 days ago | link

Let's build this for the LISP contest!


4 points by dido 532 days ago | link

So I guess that makes a consistent foreign function interface something very important for Arcueid to start having then. I think I've built up a foreign function API (sorta) and am now working out the details for dynamic loading of shared libraries so you can do something like (load "") and have it dynamically link into Arcueid, as well as a way to compile C sources into such an extension shared object.