Arc Forumnew | comments | leaders | submitlogin
5 points by palsecam 3368 days ago | link | parent

Here is a "plugin" example to handle the kind of "web best practises" I respect, those that make it faster for my customers.

  ;;; web-static.arc: module to deal with static files (CSS/JS)
  ; designed to leverage a reverse proxy to serve the files when in production

  ; currently specific to my own needs, will certainly always be.
  ; main requirements are: eternal caching when possible, CSS/JS 
  ; minification, ability to correctly handle external files we can't
  ; easily control (think images url in CSS), ** minimal overhead **
  ; (i.e: code as fucking simple as fucking possible)

  ; CSS/JS minification: we use the YUI compressor and put the
  ; files in minified-dir*.  nginx is told to look first
  ; in this directory and fallback to static-dir* if not found 
  ; (for the files "out of control")
  ; eternal caching: we set a query string "?<mtime_of_file>", and told
  ; nginx to inform the client to cache this URL for 1 year
  ; (1 year is max allowed by RFC and anyway sufficient)
  ; (wipe testing*) to activate minification and query string.
  ; doesn't matter if you do this while not being actually behind nginx,
  ; nothing will break, httpd.arc is still serving the static files correctly

  (= static-dir*          "res/static/"  
     minified-dir*  	"res/minified/"
     static-path*      	"/static/"     ; URL (not filesystem) path
     code-compress-prog*  "yuicompress"  ; sh wrapper around yuicompressor.jar
     testing*	        t)

  (def sendfile (fname (o mt (mimetype fname)))
    (resphead http-ok+ (copy httpd-hds* 'Content-Type mt))  
    (prfile fname))

  (register-path (string "/" static-path* "/*")  ; never reached in production
     (fn (req file)
       (aif (file-exists (string static-dir* "/" file))
            (sendfile it)

  (def static-url (fname) 
    (string static-path* fname
      (when no.testing*
        (+ "?" (mtime (compress-ifstale (+ static-dir* "/" fname)
	      	    		        (+ minified-dir* "/" fname)))))))

  (defs csss (fname)  (css:static-url fname)
        jss  (fname)  (js:static-url fname))

  (def compress-codefile (fsrc fdest)
    (ensure-dir:dirname fdest)
    (system (+ code-compress-prog* " " fsrc ">" fdest)))

  (def compress-ifstale (fsrc fdest)
    (when (and (in (file-ext fsrc) 'js 'css)
    	       (or (~file-exists fdest) (> (mtime fsrc) (mtime fdest))))
      (compress-codefile fsrc fdest))

  (defmemo compress-code (str (o type 'js))
    (w/tmpname tmpf
      (w/outfile s tmpf (disp str s))
      (out-from code-compress-prog* " --type " type " < " tmpf)))

  (with (_ijs ijs  _icss icss)  ; redef web.arc ones
    (defs ijs  (str)  (_ijs (if testing* str (compress-code str)))
    	  icss (str)  (_icss (if testing* str (compress-code str 'css))))

  ;; todo: 
  ; * X-Accel-Redir in 'sendfile if behind nginx.
  ;   heuristic: look if X-Real-IP present.  or make the proxy pass
  ;   a header with its name to be more correct (X-Forwarded-By)
  ; * img-compress-prog* (`optipng')?
  ; * a clean way to do the call to `yuicompress' asynchronously
  ; * gzip here to not have nginx do it on-the-fly each time?  not sure
  ;   if the gain is that valuable
  ; * 'compress-code[...] bad names?
  ; * use GG Closure compiler and not YUI, use its REST API, and therefore
  ; be obliged to make it asynchronous
  ; * hash instead of mtime maybe.
  ; * like for web.arc, '=once macro or init procedure so that one can use a !=
  ; path without having to change the file.

'mtime, 'file-ext 'mimetype are defined somewhere else. 'mtime is just calling the `stat' program via 'system. I don't have access to the file they're defined in right now (they are in a "files.arc" file) but I'll post it next week.

Nginx config sample to use this with:

  root	/home/<user>/res/;

  rewrite "/static/(.*)" "/minified/$1" break;

  location /minified/ {

     if (!-f $request_filename) {
        rewrite "/minified/(.*)" "/static/$1" break;

     if ($query_string) {
     	expires	+1y;

  location /static/ {

Obviously, using a reverse proxy makes the need of 'setuid irrelevant (it is such a low-level syscall anyway. even plain old unix daemons should use the daemontools and don't do this by themselves). Nginx could be made to keep-alive and gzip, which are huge perf wins. Not serving the static files by the app server is so obvious, even does this know.

The manual "wait 30 seconds, then kill the 'slow' client" handling of srv.arc is a crappy solution (but the crappy threading model asks for it): sometimes my wifi connection is so slow, I couldn't finish a POST to this forum (yes it happened for real, I should retried each POST several times). A reverse-proxy, by buffering and handling slow clients in the good manner (i.e: not killing them brutally: if they don't write for some time, it's OK it's just an idle fd in the select() poll) removes this problem.


Old version of web.arc (then called wf.arc) that does session and login handling:

I'll try to post an example of a login procedure as I do it with this combo next week, but like for "files.arc", I don't have access to it right now.


Clickable links: http.arc: , web.arc:


An nginx basic config file for proxying to an http/web.arc powered app server (add the previous sample in the server block if you use web-static.arc):

  server {
    listen ;

    location / {
      access_log  /var/log/nginx/examplecom-access.log;
      proxy_pass  http://localhost:8080;
      proxy_set_header        X-Real-IP  $remote_addr;
      proxy_pass_header       Server;

4 points by palsecam 3366 days ago | link

files.arc (where 'mtime and co are defined ; web-static.arc depends on it) is available at


An example of a login procedure, taken from one of my websites:

  (def new-login-handler ((o redir "/"))
    [let user (arg _ "user")
        (if (login user (arg _ "pwd"))
	    (opurl:new-login-op redir "Bad credentials" user)))])

  (def login-page (req (o redir "/") (o msg) (o userval))
    (page req "Login"
      (tag (p class "err") (prt msg))
      (fnform (new-login-handler redir)
        (lblinp "Username: " "user" "text" userval)
        (lblinp "Password: " "pwd" "password")
        (but "login"))
      (ijs "document.getElementById('user').focus();")))

  (def new-login-op ((o redir "/") (o msg) (o userval))
    (newop [login-page _ redir msg userval]))

  (defpath /login (req)  (login-page req))
'page is a macro on top of 'htmlpage to create a page with the look&feel of the project site.

'login is something like: (goodcrypt pwd (get-passwd-of-user user)) ('goodcrypt is in files.arc).

"op" means "operation" in my lexicon, and is for /x/... paths, stateful actions. I know, this is confusing, it's not the same notion than in srv.arc ('defop). But in my mind, even when it comes to webapps, the default is statelessness and resources-oriented, not stateful operations. People (me the first) basically only care about "resources" (informations, content), be it a rich Ajax-full webapp or a basic HTML page, anyway.

In srv.arc, the "operations" system is +/- the 'fns* / 'fnids* / 'flink / etc. stuff.


The complete nginx config file for the above project site:

  # Nginx configuration file for <proj> on localhost.
  # Install with:
  # ln -s /home/<PROJ>/res/nginx-localsite.conf /etc/nginx/sites-enabled/<PROJ>

  server {
       listen		8030;

       root		/home/<PROJ>/res/;
       error_log	/var/log/nginx/<PROJ>-error.log;
       access_log	off;

       rewrite "/static/(.*)" "/minified/$1" break;

       location /minified/ {

		if (!-f $request_filename) {
		   rewrite "/minified/(.*)" "/static/$1" break;

		if ($query_string) {
             	   expires		+1y;

       location /static/ {

       location ~ "^/(favicon\.ico|robots\.txt)$" {
		expires	       		+2M;

       location / {
		access_log		/var/log/nginx/<PROJ>-access.log;
       		proxy_pass		http://localhost:8020;
		proxy_set_header	X-Real-IP  $remote_addr;
		proxy_pass_header	Server;
Gzipping and other general config directives are defined in the main /etc/nginx/nginx.conf file and are invisibly "inherited" here.

I don't log accesses to /static/* and /favicon.ico / /robots.txt, but I do log accesses to the rest of the website (access_log directive in the "location /" block).

Nginx doc @


A comment in web.arc mentions scheme2js, more infos here: It's a scheme to javascript compiler, which is smart enough to substitute the TCO with the use of a while loop or a trampoline (depending of the case), and that can do a bunch of other optimizations too (like inlining calls to +).

It is used in the HOP project ( which is a framework to develop rich webapps (i.e: w/ a rich javascript-backed client GUI and w/ Ajax) using Scheme for the server and the client. It is quite impressive (the website is a demo). To Thaddeus: wtf would you want to compile Arc to CoffeeScript and not to raw Javascript directly?! If you decide to write an Arc to JS compiler, be sure to check out scheme2js!


2 points by thaddeus 3366 days ago | link

-> why compile Arc to CoffeeScript?

good question. scheme 2js looks better.

BTW Thanks for posting all this information! This gives me more to sink my teeth into. :)