The idea behind a 'hook' is that it is a place where code can be inserted without having to modify the original code. They're pretty common in emacs, for instance, where lots of packages provide hooks for easy extension in your .emacs file.
So, if there's a function bound to that name in the hooks table, it is run with the arguments. You only call 'defhook when you want to bind to that point in the code.
This is a rather interesting implementation to me, because I'm used to the emacs-lisp concept, where a hook is actually a list of functions that all get called on the arguments, so it can be added to multiple times. The corresponding names in elisp are 'run-hooks and 'add-hook. https://www.gnu.org/software/emacs/manual/html_node/elisp/Ho...
True, Guile and Picolisp do not make self-contained binaries, but they're pretty lightweight and fast, and seem like they still might be half-decent choices. I guess I should probably remove Guile from the list of candidates, since it's really for plugin extensibility in other programs; I may even have been slightly overlapping it with Chicken in my mind.
Arc would have to be "polished" a bit to make it work, but that's what we're here for, isn't it?
Yeah, I probably wouldn't use it for that either, at least as much because of the relative instability and hackishness of arc. It was fun for a few minutes to imagine modifying it to be used that way though.
No, I was forgetting that ac is considered a compiler :) But it compiles the Arc codebase every single time Arc starts up. Maybe we should start memoizing its outputs to disk somehow, see if that makes it noticeably faster. My suspicion is egregious runtime processing like ar-denil-last and ar-apply-args will cause it to not make a difference.
Given the pervasiveness with which Arc has to make such transforms at runtime, I've gotten into the habit of thinking of it effectively as an interpreter.
I have this constant problem of dissatisfaction with existing tools and paradigms, to the point that I often work my way down to the bottom of the stack considering building a new OS...
My most recent descent has started with a dissatisfaction of container orchestration tools, then discovering that it's not really easy to manage containers with unprivileged code under Linux, and now wondering about building a microkernel OS / language for distributed actor-model development from the ground up, and thinking it would be nice to have an "OS development framework" so it is easier to experiment with novel architectures without having to deal with a lot of driver development grunt work... I originally asked this question somewhere near the beginning of all that.
Speaking of building a new OS, check out https://gitlab.com/giomasce/asmc which uses some of the bootstrappable infrastructure to build an OS. It uses a stack-based (Forth-like) intermediate language called G to implement C.
Well, that was more of a self-directed comment that if I'm using a custom language for making my projects, the odds that anyone else will want to contribute to it are rather low.
Interesting that you mention the sandboxing problem, since that's something I'm currently thinking about...
My current thought for solving the sandbox problem is to have a thoroughly first-class and recursive system for creating subsets of resources and providing them to children. Fortunately, I don't think we ever have to worry about running _unexpected_ code; just code that does unexpected things. So if we have the tools to take part of the resources available for the current "process" and pass them on to any child interpreters (tabs, etc.), then it wouldn't matter how many layers you decide to add to the system.
That said, a lot of complexity with sandboxing comes from the desire to deliberately break isolation and provide access to shared resources... I think that temptation should be avoided as much as possible, and solved with easy to use communication primitives. While that does impose an overhead, we're already quite used to isolation overheads, and it should be constant. And if the abstractions around resources are good enough, the difference shouldn't be that noticeable.
And now you know why I'm thinking about an actor-model OS, where everything (including files, etc.) are represented as isolated actors that communicate via messaging. Then it doesn't make a difference whether you "own" the 'filehandle' or a different actor does - it's just an address that you send authorized messages to anyway. Fits pretty well with microkernel and distributed OS design too.
"My current thought for solving the sandbox problem is to have a thoroughly first-class and recursive system for creating subsets of resources and providing them to children."
I hadn't been thinking about sandboxing, but my interest in testable interfaces lately has me asking how I can design an interface for memory allocation in a testable way. And that requires precisely such a recursive way for one allocator to be able to do book-keeping on memory allocated by a parent allocator. This would be the start of the capability to run the tests of an OS while running on itself.
The only way to have an address tagged as used in one allocator and free in a child allocator: metadata must be somewhere else. I'm imagining something like this, where 'the 'allocator' blocks are book-keeping metadata on used vs free areas:
Yeah; I guess I didn't state it that clearly, but the sandboxing reference was from the reddit thread you linked
"Sandboxing is challenging partly because our boundaries for what is trusted shift imperceptibly over time. Browsers came out in a world where the desktop was considered the crown jewels, while browsers were considered untrusted. Over time local disk has disappeared (e.g. Chromebooks) while our lives are ruled by the browser. This has made the original design of browser security (shielding the local file system; cookies; single origin) if not obsolete then at least dangerously incomplete. So any sandboxing solution has to think about how to fight this gradual creep. No matter how you organize your sandboxes using Qubes or whatnot, something inside a sandbox is liable to become more important than other stuff inside that sandbox. And even outside that sandbox."
The idea of running tests against a live application, particularly an OS, is intriguing. It shouldn't be too hard though, since the code and its state don't have to be in the same place. So you could run the live code against separate testing data in a different sandbox or something...
Your specific example of memory management makes me think that vague "recursive subsets of resources" probably won't cut it for a lot of things. After all, if you make memory access too indirect, it would be very very slow; but if you try to use the actual CPU management interfaces, it may not allow the kind of control that we want...
How much do you really need the bootstrappable stack? There shouldn't be anything stopping you from using something that's already been bootstrapped, instead of raw hex code. E.g. existing C tools, etc. There's also the picolisp-based PilOS project; basically picolisp (which is mostly bootstrapped from an assembler they wrote anyway) running directly on the metal as a kernel. (https://picolisp.com/wiki/?PilOS)
Oh sorry, I did understand your sandboxing reference. Just meant I wasn't concerned with it when thinking about the memory allocator. Should have said, "I haven't been thinking about sandboxing.."
I'm less concerned about overhead because I'm primarily concerned about tests, which by definition don't run in production.
But I think in the case of memory allocation you can do several levels of sandboxing before you start hitting performance bottlenecks. You'd have no extra indirections when using memory, because its just a pointer and you just have an address. You'd only hit extra levels of indirection when you try to allocate and the allocator is out of memory. Then you may need to request more from your parent, who needs more from its parent, and so on, all the way down to the OS.
---
"How much do you really need the bootstrappable stack?"
My basic idea is that code is easier to quickly understand if you can run it and see it in operation in a variety of tiny slices. I think it would be easy to quickly modify codebases if you had access to a curriculum of logs of the subsystems you care about, click on any line to jump to the code emitting it, and so on. The curriculum would be created automatically while running tests (because tests are selected manually to capture interesting scenarios).
This idea requires being able to drill down arbitrarily deep into the traces, because we can't anticipate what a reader is trying to change. I may be trying to investigate why a certain sequence of operations causes my laptop to lock up. That may require drilling down inside a save operation to see the data structures inside the OS that perform buffering of disk writes. And so on.
Compilers are part of this infrastructure I want to drill down into. If I want to understand how the if keyword works, it does me no good if it terminates in some opaque binary that wasn't built on my computer. It's also only mildly nicer to find out that some code in this compiler I'm reading requires understanding the whole compiler. Metacircularity is a cute trick, but it's hell on comprehension. Any sort of coiling is bad. What noobs need is to see things laid out linearly.
I've looked at PicoLisp before. I even modeled Wart's memory allocator on PicoLisp. So it's great in many ways. But like all software today it's not really intended for end-users to look at the source code. Its code is intended for insiders who have spent months and years building up an intimate understanding of its architecture. That assumption affects how it's written and managed.
That could be true; I'm not really familiar with CPU facilities for memory isolation, but this is probably one of the most solved of the isolation challenges, since it does require CPU support.
"You'd only hit extra levels of indirection when you try to allocate..."
Good point. I wonder if there's any way to improve that, or if the worst case is rare enough that it's acceptable?
---
"This idea requires being able to drill down arbitrarily deep into the traces..."
That's a cool idea, and fits very well with the GNU objective of fully open and transparent source code. I'm not sure that bootstrapping is what is required to achieve that goal, however. What you really need is transparent source all the way down, which can be satisfied with self-hosted code, even if it's not bootstrapped. In fact, I'd argue that multiple layers of bootstrapping would make the drill-down process very challenging, because you'd have to cross very sharp API boundaries — not just between functions or libraries, but between languages and runtime environments. Making a single debug tool that handles all of that would be impressive, let alone expecting your users to understand it.
"Metacircularity is a cute trick, but it's hell on comprehension"
Metacircularity applies to interpreters built on interpreters, not compilers. I can agree that it makes things opaque though, because it reuses the existing platform rather than fully implementing it. A self-hosted compiler, even if written in its own language, could be fully comprehensible, however (depending on the quality of the code...). It's just a program that takes syntax and produces binary.
Interestingly, while writing this, I ran across the following paper, which may be of some interest: Avoiding confusion in metacircularity: The meta-helix (Chiba et al.) (https://pdfs.semanticscholar.org/4319/37e467eb9a516628d47888...)
I'll read more of it tomorrow, and possibly make a separate post for it.
I do think it would be really cool and possibly also useful if every compiler was self-hosted, and could be drilled into as part of the debugging process. It would mean that you could read the code, since it would be written in the language you are currently using, and the same debugger should be able to handle it.
---
"But like all software today it's not really intended for end-users to look at the source code"
I haven't actually looked at picolisp's source much myself, but what would it take to satisfy your desires for "end-user" readable code?
Very interesting. I'm going to read that paper as well. And think about what you said about the difference between interpreting and compiling.
> what would it take to satisfy your desires for "end-user" readable code?
See code running. My big rant is that while it's possible to read static code on a screen and imagine it running, it's a very painful process to build up that muscle. And you have to build it from scratch for every new program and problem domain. And even after you build this muscle you screw up every once in a while and fail to imagine some scenario, thereby causing regressions. So to help people understand its inner workings, code should be easy to see running. In increasing order of difficulty:
a) It should be utterly trivial to build from source. Dependencies kill you here. If you try to install library version x and the system has library version y -- boom! If you want to use a library, really the only way to absolutely guarantee it won't cause problems is to bundle it with your program, suitably isolated and namespaced so it doesn't collide with a system installation of the same library.
b) It should be obvious how to run it. Provide example commands.
c) There should be lots and lots of example runs for subsets of the codebase. Automated tests are great for this. My approach to Literate Programming also helps, by making it easy to run subsets of a program without advanced features: http://akkartik.name/post/wart-layers. https://travis-ci.org/akkartik/mu runs not just a single binary, but dozens of binaries, gradually adding features from nothing in a pseudo-autobiographical fashion, recapitulating some approximation of its git history. In this way I guarantee that my readers (both of them) can run any part of the program by building just the parts it needs. That then lets them experiment with it more rigorously without including unnecessary variables and moving parts.
I think the problem is probably a combination of lack of motivation and barriers to contribution.
On the one hand, most projects are not very well documented, advertised, supported by tests, or have communities ready to help people get started in development. So the difficulty of getting started helping is probably high.
On the other hand, most projects are the unique vision of a single developer. Everyone else probably has different ideas, and other projects they'd rather work on. It takes a certain kind of charisma, influence, productivity, etc. to get past the point where people feel like contributing to _your_ project is worth _their_ time. It would be interesting to figure out exactly what it takes to make that happen, so I could use that power for my own nefarious ends...
To perhaps start a discussion instead of just leaving a comment: where should we go next with the self-documenting / exploratory code ideas? What's missing or frustrating about the current help system?
I've recently discovered the Caddy server (https://caddyserver.com/), which makes SSL and application proxy deployments super easy. Like, 2 lines of config:
domain.com #uses the domain to automatically set up SSL with Let's Encrypt
proxy / localhost:8080 #redirect everything to Arc on 8080
I will say that Arc runs a bit resource intensive, and the slightly slow boot times mean you don't want it to have to re-launch because of infrequent requests. I don't know how well it would work on Heroku.
Also, some VPS services like vultr.com offer $5/mo nodes that have more resources than what you get from Heroku at $7/mo anyway.
I should mention that if what you want from Heroku is their deployment process, you can actually replicate some of it pretty easily with Caddy (though I have not done so yet myself; I plan to soon...)
Specifically, they have support for automatically fetching code from git and running a command in the repo, either periodically or triggered by a webhook: https://caddyserver.com/docs/http.git
That doesn't get you the app ecosystem that heroku offers, but you can get a lot of that pretty easily via docker and docker-compose now.
That would be nice I suppose, but we can't actually modify the arclanguage forum, so it would be of limited benefit. I guess a lot of people come to arc because they want to run an HN clone though, so it might be worth pursuing.
> File upload
Yeah, I think enhancing some of the fundamental web-service functionality in arc would be good. Beyond file upload, better support for OAuth, ssl, etc. would be good too.
> Ecommerce
Good idea. Not sure how to make that better myself, but demystifying and enabling easy ecommerce would be cool.
I think that also has the same motivation for the previous category; we need better support for web application development, because that's what most people want to do.
It's just insane that Arc Forum doesn't encrypt HTTP but, well, as you write, we can't change Arc Forum. Maybe they'll update it some day, or someone will set up an unofficial one that's more up to date?
It looks quite humble but it has both a repl and an interface for saving/running various web apps.
I've set it to be enabled by default in Anarki. And it's only available for admins, because it can run system commands. I wonder if it would be possible to allow any user to develop apps securely, e.g. by disabling unsafe commands or sandboxing it somehow? Perhaps there is a simple way of doing that? It could open Arc web app development up to any user of an Anarki driven page.
I think those features mentioned above still need better documentation, too.
Other HN features that Anarki News is lacking:
* favourites
* hide item
* past front pages
* show all stories from same site by clicking on domain name
The second most popular idea is probably a module system. I honestly don't know much about what that would entail, beyond reading rocketnia's comments in his ns.arc from 7 years(!) ago.
Basically, it seems kinda hard, because of how much arc currently depends on global tables to collect special objects like macros, the interactive help stuff, etc.
We could just decide that Arc is deliberately focused on 'hackability', like emacs, and that modules are unnecessarily refined and restrictive for our uses.
My first thought is a package system based on melpa / use-package from emacs. Please point out if something like this has already been done and I have just forgotten it.
Basically, just a simple index and package fetching system that pulls libraries directly from github or other vcs sources. Then we only need one file in the standard library "package.arc", that provides functions for querying the index and fetching packages from github into a ~/.arc directory, supported by a macro like use-package from emacs (https://github.com/jwiegley/use-package) or 'ns from clojure.
An important point here is that a package fetching utility can be independent of any module system. Which is good, because we don't really have that yet. The emacs community doesn't seem to think one is necessary; everything is just imported into the global namespace, and prefixed with the package name if necessary to keep it separate. Maybe we could make some macros to simplify the prefix process, but that could quickly get complicated.
We could also experiment with some avant garde packaging ideas, such as akkartik's thoughts on avoiding version pinning, searching the vcs sources directly for the package, or building the community CI tools that automatically find downstream dependencies and run their tests against your lib changes.
> My first thought is a package system based on melpa / use-package from emacs.
That would be great! Use-package is amazing, and that (combined with similar loading things from package.el) could make loading dependencies way easier. I know that when I'm writing arc code, I'm reluctant to use libraries -- even libraries included with arc -- because they're (afaik) impossible to automatically load.
> We could also experiment with some avant garde packaging ideas, such as akkartik's thoughts on avoiding version pinning...
Amusingly, the Emacs ecosystem can be thought of as a package manager that avoids version pinning; the standard package repositories (GNU Elpa, Melpa) only keep the latest version of a package around; you can't install older versions.
Of course, I know of no package foo.el that introduced _foo2.el_ when breaking backwards compatibility. So we can maybe do better that way.
I'd be willing to donate some hosting for an initial version of the package index, if we decide we need one. I have a vultr instance I'm not using for much at the moment.
It shouldn't be too hard if we provide some real reasons for people to use the language. A 'killer app' if you will.
That and production-readiness, so that they don't have to stop using their favorite 'toy' when they want to build something real. Some of that involves just making more libraries available.
Maybe starting with the package-management / CI system that pulls directly from github akkartik and I were discussing earlier. It would be cool, not too hard to implement, and make sharing libraries a lot easier.
This is where a more traditional forum system would be beneficial. I know we've done lots of work on module / package systems in the past, but it's not easy to do research on it. Maybe we could use github issues on anarki to discuss some of these topics?
I did find lib/ns.arc, which rocketnia had done apparently back in 2011...
Modules and packages are related, but can be implemented independently. The package system is just for discovering and fetching code. Modules keep it separate and help avoid mistakes caused by interface overlap. Honestly, we can probably do without modules if we're careful - for example, emacs doesn't have namespaces, just a prefix convention.
Not having modules does make arc seem more like a toy than a production system though. You're more at the mercy of the remote developers.
Personally I find that the 'killer feature' is Racket integration as Racket has quite an extensive well-documented standard library (and FFI so that's all C libraries as well).
Even if Arc doesn't get much attention itself, it still benefits from the work going into Racket.
So there are a lot of libraries available. (Arc's Racket integration still has a few rough edges, particularly regarding keyword arguments, but there are workarounds.)
That doesn't really qualify as a 'killer feature' for Arc; it doesn't provide any incentive for someone to learn and use Arc - and keep using it - over just learning Racket in the first place.
I do agree that it dramatically improves the value of Arc over being written as a stand-alone interpreter. And providing easier access to Racket features may be a good idea. It helps bootstrap Arc, like Clojure on the JVM with the Java standard libraries until proper clj wrappers are written.
I think it is a killer feature in a round-about way. I believe the killer feature is actually generic and robust database integration. There's a whole lot of people who start with data on hand and need to hack together an app from there. The starting point to db integration is racket, which also gets you access to other libraries. Once in place arc libs can be created eliminating a need to even learn racket (just like there are plenty of clojure programmers who don't know java).
IMO doing package integration first is putting the cart before the horse.
> the killer feature is actually generic and robust database integration
That could be a killer feature, but I don't think we have it developed yet. I certainly wouldn't have thought of it. Basically, when I say "killer feature", I'm thinking of the specialization or distinguishing characteristic that we would emphasize in a Quick Start tutorial.
When arc was first launched, the "arc challenge" of building a multi-action website with a form in only ~5 lines of code was the "special feature." Right now, I think hackability and simplicity of the syntax are two of the better things, but we could probably specialize on more.
> IMO doing package integration first is putting the cart before the horse
The purpose for working on package integration is to enable further development. Without the ability to share code easily, it's a lot harder to build on and benefit from community effort. In itself, I agree, a package manager is boring and probably not a killer feature. However, making it really easy to start building something useful by searching for and loading relevant code straight from the interactive console would be a pretty big step.
Perhaps the specialty I'm looking for is exploratory programming, which we've mentioned before. Our interactive help system is pretty good. The only problem others have mentioned before was that Python is arguably better, just because there are already examples and libraries for doing most activities, whereas Arc requires a lot of development effort just to get fundamental components working.
Well there's that, but honestly... exploratory programming with a tool that can't provide basic functionality will limit how much you can explore. I think you mean 'language design exploration'? - if so I'll stop right there, since, well, frankly... it's not my wheelhouse. :)
> package integration first is putting the cart before the horse
I'm simply saying that features like db integration bring users, users bring manpower, manpower will provide package management in a way that ensures it accounts for more use cases, but again this is moot if what you're interested in is only the language development arena. Though I'm not sure how you could prove out a language unless you can actually use it for real world work.
P.S. If you want users, then killer feature would = todo list mobile app with local db in 30 lines of code!
> exploratory programming with a tool that can't provide basic functionality will limit how much you can explore
Absolutely true, which is why I do think the Racket integration is important, so we can just wrap its many and powerful libraries in lighter-weight arc libs, and also why I think a decent package system is important. It needs to be easy to share code, and to find solutions to problems. Otherwise everyone spends too much time rebuilding the same wheels.
> features like db integration bring users
Absolutely. I think better db support in arc would be awesome to have. Especially if we can build an ecosystem where "intelligent defaults" are the norm, so that 90% of the time the exploratory developer can get a db up and running with 1-2 lines.
> todo list mobile app with local db in 30 lines of code
An admirable goal. That's actually a great idea of something we could work toward as a community project, each building pieces that eventually come together to make a decent modern app.
Were you thinking a mobile friendly web app, or trying to run it natively on a phone? I'm not really sure how the latter would work... Though building a compatible javascript base for arc would be pretty nice. I do like being able to use consistent code for both clients and servers, as in node and clojure.
> Were you thinking a mobile friendly web app, or trying to run it natively on a phone?
The java/js ecosystem has the largest reach making it the easy choice. One could work on a js compiler and target pouchDB as a starting point. That said choosing js also makes Arc go further down the path Clojure has already gone putting the two closer together, and with Clojure being so far ahead in that arena then maybe it's doomed to fail. The other way to go is to do what Clojure is not great at. iOS development? maybe integration with swift? At any rate I'm not a language development guy. I can only tell you what would appeal to me. I mostly use Clojure and a really easy to use arc->iOS app development ecosystem would be really cool.
> Without the ability to share code easily, it's a lot harder to build on and benefit from community effort.
I've just been dumping my Arc experiments into the Anarki repository. Akkartik manages it by the policy that pretty much anyone can commit anything, so I haven't had any issues sharing my code. I've recently put a Puredata compiler in Arc in there, https://github.com/arclanguage/anarki/blob/master/lib/pd.arc
The interactive help in Anarki is great, although there are some undocumented functions. A great improvement, which would be very easy to implement, would be to add the documentation from the various Arc sites to the Anarki interactive help.
I do know some Racket, and for me this exact feature did provide an incentive for me to learn and use Arc, because I could apply what I already knew.
Arc feels more brief and less strict, so in some cases it's more pleasant to use than Racket.
When I found out about Arc's Racket interop a couple of years ago, I was really surprised that this was not documented anywhere, so I added a section in the Readme.
I agree it's important, and we should probably develop it and make it easier because it's an important foundation. However, there aren't that many Racket developers, and its not really the best argument in favor of using a language that it just happens to be a better version of another language. That might be a competitive advantage against Racket, but what about compared to other interpreted languages? Why would someone want to use arc over Python or Ruby?
> When I found out about Arc's Racket interop a couple of years ago, I was really surprised that this was not documented anywhere, so I added a section in the Readme.
Yeah, the documentation could use some work. Definitely not our competitive advantage at the moment, though that could be changed if we build on our interactive help system.
Also, that feature is really only part of anarki. I don't think arc3.1 had that feature. I think at this point it seems like pg has pretty well abandoned arc though, so maintaining compatibility shouldn't be a primary concern.
Hmm, I don't think pg has abandoned Arc. And I don't think maintaining compatibility is a concern even so. So far Arc has made no compatibility guarantees. We're all forking pre-alpha software. So it's conceivable the hundred-year language will have features from Anarki. Forks that restrict where they get good ideas from will be outcompeted by forks that don't.
Though lately I consider it more likely that a hundred-year language will have a leveled-up conception of "compatiblity", one that assumes orders of magnitude more forking activity. (See "the Zen of Mu" at the bottom of http://akkartik.github.io/mu; Mu is my attempt at building out the foundations for a hundred-year stack. With, perchance, something like Arc on top.) Perhaps Arc shouldn't be a single language but a family of forks (https://lobste.rs/s/n0d3qo#c_rue8pf). Not a single tree but a clonal colony (https://en.wikipedia.org/wiki/List_of_longest-living_organis...). That'll only work if we can make it easy for superficially-incompatible forks to exchange features and functionality back and forth. Which is an unsolved problem so far. So we may well be very far from a hundred-year language.
Anyways, this tangent is my contribution. I don't have a short-term answer for how to solve Arc's users-vs-libraries chicken-and-egg problem ^_^
> I don't have a short-term answer for how to solve Arc's users-vs-libraries chicken-and-egg problem ^_^
Be the change you want to see in the world... I'm not really sure how to motivate the community, but I am rather attached to it, even if I have been a very infrequent lurker over the past years. We have had a relatively high amount of discussion over the past few days though...
I don't think we can just expect to flip a switch and suddenly get a community; we have to /be/ a community, and then people might be willing to join us.
Maybe we should emphasize the flexibility of our language designs by making the language itself more modular. It would be challenging from a compatibility/dependency standpoint, but those are problems we might have to solve anyway. It would help to have better isolation of components.
It looks like I might've subtly broken ns.arc with my own changes to make Anarki installable as a Racket package. Here's an example that should be working, but currently isn't:
; my-file.arc
(= n 2)
(= my-definition (* n n))
arc>
(= my-definition
(let my-ns (nsobj)
; Populate the namespace with the current namespace's bindings.
(each k (ns-keys current-ns)
; Racket has a variable called _ that raises an error when
; used as an expression, and it looks like an Arc variable, so
; we skip it. This is a hack. Maybe it's time to change how
; the Arc namespace works. On the other hand, copying
; namespaces in this naive way is prone to this kind of
; problem, so perhaps it's this technique that should be
; changed.
(unless (is k '||)
(= my-ns.k current-ns.k)))
; Load the file.
(w/current-ns my-ns (load "my-file.arc"))
; Get the specific things you want out of the namespace.
my-ns!my-definition))
4
arc> n
_n: undefined;
cannot reference an identifier before its definition
in module: "/home/nia/mine/drive/repo/mine/prog/repo/not-mine/anarki/ac.rkt"
context...:
/home/nia/mine/drive/repo/mine/prog/repo/not-mine/anarki/ac.rkt:1269:4
The idea is, you create an empty Arc namespace with (nsobj), you use `w/current-ns` to load a file into it, and you use `a!b` or `a.b` syntax to manipulate individual entries.
An "Arc namespace" is just a convenience wrapper over a Racket namespace that automatically converts between Arc variables `foo` and their corresponding Racket variables `_foo`.
For some overall background...
I wrote ns.arc when I didn't have much idea what Racket namespaces or modules could do, but I was at least sure that changing the compiled Arc code to more seamlessly interact with Racket's `current-namespace` would open up ways to load Arc libraries without them clobbering each other. It wouldn't be perfect because of things like unhygienic macros, but it seemed like a step in the right direction.
I went a little overboard with the idea that Racket namespaces and Racket modules could be manipulated like Arc tables. However, that was the only clear vision I had when I embarked on writing the ns.arc library, so I approximated it as well as I could anyway. In fact, I don't think the utilities for generating first-class modules (like `simple-mod` and `make-modecule`) are all that useful, because as I understand a little better now, Racket modules are as complicated as they are mainly to support separate compilation, so generating them at run time doesn't make much sense.
I'm still finding out new things about what these can do, though. Something I didn't piece together until just now was that Racket has a Racket has a `current-module-name-resolver` parameter which can let you run arbitrary code in response to a top-level (require ...) form. I presume this would let you keep track of all the modules required this way so you can `namespace-attach-module` them to another namespace later. Using this, the kind of hackish partial-namespace-copying technique I illustrate above can probably be made into something pretty robust after all, as long as Anarki sets `current-module-name-resolver` to something specific and no other code ever changes it. :-p
I tinkered with Anarki a whole bunch and finally got this working smoothly. There was a missing step, because it turns out we need to load certain Racket-side bindings into a namespace in order to be able to evaluate Arc code there. It seems more obvious in hindsight. :)
I approached this with the secondary goal of letting a Racket program (or a determined Arc program) instantiate multiple independent intances of Anarki. The ac.rkt module was the only place we were performing side effects when a Racket module was visited, and Racket's caching of modules makes it hard to repeat those side effects on demand, so I moved most of them into a procedure called `anarki-init`.
By adding one line to the example I gave...
(= my-definition
(let my-ns (nsobj)
; Load the Arc builtins into the namespace so we can evaluate
; code.
(w/current-ns my-ns ($.anarki-init))
...))
...it becomes possible to evaluate Arc code in that namespace, and the example works.
Before I started on that, I did a bunch of cleanup to get the Anarki unit tests and entrypoints running smoothly on all our CI platforms. To get started on this cleanup, I had a few questions hjek and akkartik were able to discuss with me on issue #94: https://github.com/arclanguage/anarki/issues/94
A lot of the problems I'm fixing here are ones I created, so it's a little embarrassing. :) It's nice to finally put in some of this missing work, though. I want to say thanks to shader and hjek for talking about modules and packages, provoking me to work on this stuff!
> How would this be different than git submodules?
Most of the difference would be in the user interface. Git submodules might actually be a great idea for how to implement it, to encourage hacking on and submitting changes to the libraries themselves.
Git submodules don't provide a very friendly user experience though. You have to know the repository url and manage them separately. It's not terrible, but it's extra work, and doesn't handle loading the library in your own, or dependency management. Ideally, pulling in a package would also result in pulling in all of its dependencies, without pulling in multiple copies of the same library the way npm used to.
There are a lot more features that could be offered by an actual package system.
