Arc Forumnew | comments | leaders | submitlogin
1 point by akkartik 1320 days ago | link | parent

Yeah, that whole side is a bit clunky. It gives readers feedback when things are broken so they aren't misled. But the job of the writer is still hard. Things would be improved by my last idea, to run examples along with other tests.

Here's a couple more issues:

a) Expected results are currently unevaluated, so there's no way to represent tables, and so on.

  arc> (help counts)
  [fn]  (counts seq)
  Returns a table with counts of each unique element in 'seq'.

  Examples:
    arc> (counts '(b a n a n a))
    (obj b 1 a 3 n 2) <-- ERROR
b) We can't provide examples for non-deterministic functions like rand-elt.

c) We can't show stdout and stderr.

---

Last but not least, I've been thinking about the freezing idea myself in other contexts. I think there's something big there.



3 points by shader 1319 days ago | link

As for the 'freezing', I've been thinking about it in other contexts as well. I noticed that one of the companies I worked at seemed to want to do some of their regression tests that way, so it makes sense to provide a way to automate it.

In particular, I would like a way to convert an interactive console session (my normal method of testing code) into a set of unit tests. Not sure exactly what the right way to do that would be though.

It's still not a very good form of unit test, as it won't cover cases as mentioned above with randomness or side-effects. Mutation testing or fuzz testing / generators are probably better ways to generate unit tests, but they don't make very good examples.

-----

2 points by evanrmurphy 1319 days ago | link

Neat ideas about testing.

-----

2 points by shader 1319 days ago | link

Yeah. B can probably work just fine with dynamically evaluated examples, but stderr/stdout would require user interaction, which may not be acceptable. Maybe some way to indicate if an example should be evaluated or not?

What do you mean by 'run examples along with other tests'? Does that mean that you would include other tests in the help output? That the examples would be included when running the tests? Or something else?

-----

1 point by akkartik 1319 days ago | link

The examples would be included when running the tests.

I'm less concerned with handling stdin, but the obvious example for do is:

  (do prn.1 prn.2 prn.3)
Surely we should be able to show that.

-----

1 point by shader 1319 days ago | link

That would work correctly with examples evaluated in the console, no?

-----

1 point by akkartik 1319 days ago | link

Yeah, I took your advice:

  (examples do
    (do (prn "line 1")
	(prn "line 2")
	(prn "line 3"))
    _)
Here's how it looks:

  arc> (help do)
  [mac] (do . args)
  Evaluates each expression in sequence and returns the result of the
  last expression.

  Examples:
    arc> (do (prn "line 1")
	     (prn "line 2")
	     (prn "line 3"))
Thanks! https://github.com/arclanguage/anarki/commit/14fff69c4

-----

2 points by fallintothis 1319 days ago | link

Actually, stdout should be easy to test with tostring, no?

In fact, if you're just testing REPL interactions...

  (mac repl-output (expr)
    `(tostring
       (write (eval ',expr))
       (prn)))

  arc> (repl-output (prn 'hi))
  "hi\nhi\n"
  arc> (repl-output (prn "hi"))
  "hi\n\"hi\"\n"
  arc> (repl-output 'hi)
  "hi\n"
  arc> (repl-output (do prn.1 prn.2 prn.3))
  "1\n2\n3\n3\n" 
(Any gotchas with the above?)

It makes sense to specify the example expr / expected pairs as sexps & strings: run repl-output on exprs, compare those strings to the expected strings.

-----

2 points by akkartik 1319 days ago | link

Thanks for thinking through that! Yes, I'd had a similar though not as fully-formed idea, but was stuck on distinguishing stdout from return value in the examples. But yeah, that's overthinking it -- we have to mentally parse the two at the repl anyway.

Only other quibble: a string isn't very nice to read next to code, even if it looks nice at the repl. Hmm..

-----

2 points by shader 1319 days ago | link

Do you get notifications for arc forum posts/comments? Or do you just check really often?

-----

1 point by akkartik 1319 days ago | link

I'm just obsessive for short periods :)

-----

2 points by shader 1319 days ago | link

Ah. About the same as me then.

Maybe we should make an auto scraper/notifier? Though iirc, the arc server doesn't really appreciate frequent queries. On closer look, it should be fine; the default throttle window should be 10s. It could query once a minute, and update an rss feed for arc forum posts and comments. Or provide webhook + email support.

-----

1 point by akkartik 1319 days ago | link

I actually built one back in the day: http://akkartik.name/post/2011-05-13-05-47-29-soc. Trouble was, it kept getting banned/blocked even when I scaled back to crawl HN every five minutes. Eventually I was crawling so infrequently that I was missing stories that dropped off /newest in the meantime, so I stopped crawling.

I actually had an arc version for a brief period[1], but it didn't seem like there was as much need for it here. But perhaps the email notifications would be useful. I'll see if I can bring it back.

[1] I was certain I'd showed it here, but no I only showed it over email to a couple of folks. I really should bring it back just so y'all can look at it. Or at least locate the (rails) sources.

-----