#clojure logs

2008-09-03

00:08Chousergreat!
00:09jamiiresolveSlot sends the field type off to c++ to find out the number of args the signal has. So the field type has to be exact - there will have to be 10 seperate signalholder classes. But at least it works now
00:10Chouserok, good enough.
00:23jamiiChouser: Was it you who was interested in using qt? Do you want the code for this stuff and the qt-repl?
00:31ChouserIf you want to share it, you should post it to the forum or put it in clojure-contrib or something.
00:32ChouserI'm a general fan of Qt if you have to use C++ -- beyond that I have only vague interest.
00:38jamiifair enough
00:39ChouserI do wonder if the broad adoption of webkit is going to help push Qt into more places.
09:13Myomahello
09:13Myomawhat impact can invokedynamic have on clojure ?
09:15rhickeyMyoma: possibly not much at all - it is mostly a feature for languages that do by-name single dispatch and thus can leverage per-call-site caching when one call site is typically only used for one 'type'
09:16rhickeyClojure's multimethods are already separated by namespace, and thus most call sites will be genuinely polymorphic
09:17rhickeyClojure caches the signature-to-method lookup results on the multimethods themselves
09:17Myomaoh!
09:18MyomaWhat is tagged numbers?
09:19rhickeyAllows an Object reference to be a number (not on the heap)
09:19rhickeyusing 'tag' bits to distinguish numbers from references
09:19rhickeyold Lisp/Smalltalk trick
09:21Chouserwould that be to reduce memory management overhead? cheaper "new"s?
09:23rhickeyLots of benefits to not having each number be an individual object on the heap - no allocation associated with a number, no memory reference associated with using its value - a tagged number is a value
09:23rhickeyafter the tag bits are masked off
09:24rhickeyway faster - CLs and Smalltalks trounce dynamic JVM languages that use boxed numbers
09:24Myomabut JVM does have unboxed numbers?
09:25rhickeyRight now in Clojure you can get locally-unboxed math, which is just as fast as Java, but as soon as you want to pass or return a number it must be boxed
09:25Myomaah I see, that is what I was going to ask .
09:25Chouseroh, my understanding was backwards. So this would be storing a number where you'd usually expect to find a heap reference, in order to represent the number itself.
09:26rhickeyChouser: right, usually a couple of bits of reference addresses are always unused, due to alignment
09:26rhickeyso you put tag bit there saying 'this is really a number'
09:26Chouserah, sure. seems like I've heard of other clever uses for those extra couple bits.
09:27rhickeysome cost in numeric range - 28-bit fixnums are common in 32-bit architectures
09:37Chouserwell, we can just write up a patch to Java and submit it, right?
09:37rhickeyheh
09:38blackdogi'm sure i saw john rrose talking about that on the dynlang group
09:38rhickeyI know these things are on their radar, and now tagged numbers have become important to JRuby...
09:39rhickeyblackdog: yes John Rose has an excellent blog entry on it: http://blogs.sun.com/jrose/entry/fixnums_in_the_vm
09:41blackdogi always wonder why squeak seems to me to be much more efficient in memory usage than java with an entire envirnment loaded, is that part of the recipe?
09:42rhickeyblackdog: probably not, as there aren't too many numbers kept around
09:42rhickeyand typical Java code does not use boxed numbers at all
09:43rhickeydynamic langs are forced into it because they want unified signatures, which must be Object based
09:44rhickeyJohn Rose on TCO: http://blogs.sun.com/jrose/entry/tail_calls_in_the_vm
09:47rhickeythe problem is - when do these things get into JVMs you can use for production work?
09:47blackdogwell the advent of openjdk will speed things significantly
09:48rhickeyIt will speed up the development, whether it speeds up the integration is TBD
09:48blackdogsure
09:49rhickeyClojure accepts the compromises associated with targeting today's JVM
09:50Chouser...and isn't dramatically hampered by that, IMHO.
09:52Chouserfor example "recur" has merits nearly sufficient to stand on its own, even if it wasn't required for performance.
09:58rhickeyLong live Chouser - defender of recur! :)
09:58Chouserheh
09:58ChouserI guess I'm a bit vocal on recur because I've traveled so far. I thought it was a pretty ugly hack at first.
10:00cemerickyeah, recur is very pleasant -- makes tail-recursive algorithms (more) explicit and obvious when you're reading the code
10:00rhickeyIt's funny because I get beat up so often for it
10:00cemerickdon't let 'em get you down :-)
10:01cemerickrhickey: are you planning on changing delay's equality semantics eventually, so that (= (delay 5) 5) => true?
10:02ChouserI think it contributes to the general feeling in clojure that it's simple, with very little magic
10:02Chouserhm, with the exception of STM I suppose.
10:03rhickeySTM is simple to use, no? Compared to locks
10:03ChouserBut a function call is a function call -- no type lookups, no parameter overloading, no inheritence. If you want any of those types of features, you can bring them in as needed -- explicitly and succinctly.
10:03rhickeycemerick: but a delay of 5 isn't (yet) 5
10:05ChouserI'm not suggesting we shouldn't have STM, but I can't say it doesn't feel magical. :-)
10:06rhickeyChouser: you're onto what I think is underappreciated about OO - there is a lot of accidental complexity
10:06cemerickrhickey: yeah, I know, it's a separate value; however, I continue to aim for it to be as transparent as possible. This came up for me when doing some tests, where a fn returns a map whose values might be delayed, and I want to compare it to a known-good map whose values are always strict.
10:06rhickeycemerick: the way to treat mixed scenarios is to always force
10:07Chouserlocks are simple to use in the small. it's only in the aggregate that locks crush all productivity.
10:07rhickeyChouser: dosync is as easy as synchronized
10:07Chouserrhickey: yeah, Clojure's taught me that. I'd never noticed before.
10:08Chouserrhickey: it's as easy to use, but I feel like I understand all of the very small amount of work that synchronized is doing. I have no such illusion about dosync.
10:09rhickeyfair enough
10:09cemerickrhickey: well, I'd have to cons up a new map (or other collection) in order to use force; I'm looking to do (= some-map other-map), not (= (:key some-map) (:key other-map))
10:09cemerick(which would have to become (= (force (:key some-map)) (force (:key other-map))))
10:09Chouserperhaps if I use dosync enough to grok it more deeply it will feel less magical. we'll see. :-)
10:10rhickeycemerick: I guess the problem is, if (= (delay 5) 5) works why wouldn't (+ (delay 5) 5) ?
10:11rhickeyin a statically typed system one could do that auto-forcing without cost to non-delayed values, but not in a dynamic system
10:11cemerickideally, the latter would, although I understand that that would be impractical
10:15wlram i alone in getting: (ns-publics clojure) => Unable to resolve symbol: clojure in this context
10:16cemerickFWIW, my (seemingly constant) mumblings about delays come from my wanting to shoehorn scala's lazy keyword into clojure; certainly a mismatch, but you can call me Sisyphus :-)
10:16wlrin fact (ns-* any-ns) behaves similarly
10:16rhickeywlr: (find-ns 'clojure)
10:17rhickeycemerick: I imagine that Scala feature leverages its type system
10:18wlrahh, thanks. sorry for the noise.
10:18cemerickrhickey: Indeed it does, so that its equivalent of (+ (delay 5) 5) has no cost beyond (+ 5 5)
10:19rhickeywlr: no problem. I'd like to make all of the ns-* functions take a symbol as well as a ns object at some point
10:20rhickeycemerick: the key being the lack of cost for non-delayed values. I'd have to add a force around all value expressions
10:20cemerickrhickey: sure, and I'd never suggest that -- that's why I said it'd be impractical above
10:21rhickeycemerick: maybe some auto-forcing wrapper on the data structure?
10:23cemerickrhickey: that would still require a rebuilding of any collections involved, which could be nontrivial
10:23ChouserI'm sure people who have made their own runtime systems and assumed they had to have TCO and spent lots of effort getting it right might feel a little snubbed that Clojure doesn't deem that feature necessary. :-)
10:25cemerickrhickey: I should say, that an auto-forcing wrapper would be very handy, although not suitable for my current context.
10:25rhickeyChouser: one need only to compare the speed of SISC Scheme to Clojure to see the tradeoffs
10:25rhickeycemerick: what are you delaying, if I may ask?
10:26cemerickrhickey: in this particular case, a search of a very large rtree for regions that intersect and/or contain another region
10:30rhickeycemerick: I see. I certainly thought a lot about transparent delays when designing Clojure, but couldn't figure out a way to avoid the overhead
10:32cemerickrhickey: yeah, I understand the difficulty in a dynamic environment. In this particular case, if I were forced to use an auto-forcing wrapper on the collection, that would result in a reindexing of both rtrees (which implement IPC) -- obviously a no-go.
10:32cemerickI mentioned putting together a Delay subclass last week in connection with the delay-as-fn discussion, but wedging auto-forcing equality semantics into the mix is a much more compelling reason to go ahead with that. Those semantics would break things in general use, but they're just right in this case, I think.
10:36cemerickChouser: FWIW, I benchmarked clojure against SISC before I decided to start using the former professionally -- the results were quite stunning in clojure's favor. Rich's decision to forego reified environments was a good one.
10:39ChouserSISC is an unapologetic interpreter, isn't it? Is that how JRuby and Jython are designed as well?
10:42cemerickYeah, they're all interpreted. JRuby was on its way to delivering an AOT compiler, last I looked.
10:42cemerick...or, just a compiler, I suppose :-)
10:42rhickeyChouser: SISC implements Scheme 'purely' on the JVM, and does what it needed to do in order to implement TCO
10:43rhickeyJRuby and Jython do a lot of compilation to bytecode
10:44Chouserok, but function calls still can't be pure Java calls like in Clojure because of differences in lookup and inheritence semantics, I'd assume.
10:45Chouserrhickey: you mentioned a few months ago that AOT compilation for Clojure was near the top of your list. Is it still there?
10:45ChouserI have basically zero personal interest in AOT, but the question comes up.
10:45rhickeyChouser: function calls can't be real function calls due to TCO
10:47rhickeyChouser: My point is that implementing a pure idea required an interpreter, with lots of consequences. Bigloo & Kawa compile to bytecode but forego TCO as well
10:48rhickeyChouser: the key to AOT is less the concept than the purpose. Is it startup speed? Packaging ease? Running on Android and other platforms with no dynamic classloading?
10:49rhickeyBigloo paper: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.59.462&rep=rep1&type=pdf
10:49blackdogdoes clojure run on dalvik?
10:50Chouseryeah, I think a lot of purposes get dumped in there. The most common case I hear for AOT compilation is actually just obfuscation.
10:50rhickeyblackdog: no, because there is no Java bytecode once you get on the VM, it must all be translated ahead of time
10:50blackdogah ok
10:50rhickeyChouser: forgot obfuscation
10:52rhickeyI'd like to see some contributions for packaging - the "how do I make a jar?" question comes up often, and is independent of AOT compilation
10:53cemerickrhickey: I think many people persistently forget that clojure == java for issues like that.
10:53rhickeycemerick: right, but still could be made easy
10:54cemerickI suppose -- but short of pointing someone at clojure.lang.Script, what other ground needs to be covered that isn't a prerequisite for packaging a "regular" Java app?
10:55rhickeyIf they want a Java entry point they have to deal with genclass
10:55cemerickOh, I see what you mean.
10:56ChouserSpeaking of genclass, did you see the incompatibility between it and the new ns macro?
10:57cemerickA script or ant task could be put together that would handle all of the genclass stuff in order to bootstrap a "main" namespace...
10:57rhickeyChouser: yes, still thinking about that. I realize the not-yet-defined criteria is too precious.
10:57cemerick(FWIW, while clojure does work well via ikvm on .NET right now, being able to eliminate dynamic classloading via AOT would make me a lot more comfortable with that deployment path -- appdomains and other fiddly .NET things can sometimes make dynamic classloading dicey)
10:57ChouserPerhaps instead of automatically skipping the (refer clojure) if the namespace exists, there could be :dont-refer-clojure flag
10:57Chouserbut with a better name.
10:58rhickeyChouser: right, that's one of the options, but more likely they want only some of clojure, so maybe :refer-clojure ... with the args of refer, if not present refer all
10:59Chouserhm, not bad. maybe allow ":refer-clojure :none" or something, just in case?
10:59Chouser:refer-clojure nil
11:00rhickeythere is :only already
11:00rhickeyI can't imagine having no clojure as a real scenario
11:00Chousergood enough
11:00Chouseryeah
11:01rhickeycemerick: yeah, I'm amazed by IKVM running Clojure
11:02cemerickrhickey: yup, ikvm is a pretty amazing piece of work. Clojure working well on .NET through ikvm is a must-have for us in certain contexts
11:03cemerickof course, I'm keeping a pretty close eye on that, to make sure that things don't go off the tracks in that regard :-)
11:04rhickeycemerick: I imagine your experience there would be quite interesting to many, if you got time to write it up as a Wiki entry
11:07cemerickrhickey: so far, there's not much experience to tell -- our jar (which includes clojure, a huge pile of Java code, and some clojure we've been adding around the edges) gets dropped into ikvmc, and a DLL pops out the other side that gets backtested against the jar's results across a test suite. We've used the same build and test process with ikvmc for years now, but I suppose that some may not find it so obvious.
11:08rhickeydo you call .Net from Clojure?
11:09cemerickNo, we don't call .NET anywhere, and we don't return any instances of .NET classes or types, either.
13:00rhickeyanyone want to comment on Clojure robustness?: http://groups.google.com/group/clojure/msg/60c7a9aa770dc406
13:18Chousernot really. you want a :refer-clojure patch for ns?
14:31ChouserIt's seems pretty common to alias clojure.zip as zip, clojure.xml as xml, etc. Any sane way to roll that into the ns macro, or are they better as standalone alias calls?
14:32Chouseroh, nm -- you can use (ns foo (:require [clojure.xml :as xml]))
14:37rhickeyChouser: :refer-clojure patch welcome. Do you have robustness issues?
14:39abrooksrhickey: Is the individual purely asking about robustness? I suspect he may be asking something of a question of interface stability / consistency as part of asking about using it in production.
14:39abrooksMaybe not. Maybe he is just asking about runtime stability.
14:40rhickeyabrooks: I don't know, only that it needed answering by someone other than myself :)
14:40abrooksRight. :)
14:41Chousernot at all, just don't feel credible on the topic. My largest piece of Clojure that has ever had to run with any kind of reliability was about 350 lines. small potatoes
14:41abrooksUnfortunately the last several months have only afforded brief moments to poke at Clojure so I'm not in a position to comment.
14:41Chousercemerick answered. I believe him.
14:42abrooksChouser: Oh, I hadn't checked back to see if anyone commented.
14:43rhickeyI'm interested in any and all usage stories, since I'm frequently asked "what are people doing with Clojure?" and my only answer is "reporting bugs and requesting features" :)
14:43Chouserhehe
14:44cemerickrhickey: I'm afraid that many people who really, really like clojure will see it as a tremendous competitive advantage, as well as a potential PR liability with those that control I.T. budgets.
14:45ChouserI've only ever finished 2 "real" projects in clojure. clojure-log and a tool I wrote to help track house sales listing changes.
14:45abrooksChouser: What about your ICFP08 submission? ;-D
14:46rhickeycemerick: I totally understand. I fully expect many early adopters to do so in stealth mode
14:46ChouserThe first is 190 LOC, the second is the 350 LOC I mentioned earlier.
14:46abrooksChouser: Didn't you rewrite the Movie browser in Clojure at one point?
14:46Chouserabrooks: oh, I forgot that! 180 lines.
14:47Chouserabrooks: nope, the movies page is one of many incomplete clojure projects that I intend to finish some day. :-)
14:47abrooks:)
14:47ChouserI guess I can reply with all this on that thread, for whatever little value it may have.
14:48rhickeyChouser: I'm going to do another "what are you doing/what features do you want" message soon. If you don't feel your experience is telling about robustness, no worries
14:50DrewRrhickey: I'm curious what you use Clojure for. Do you ever get to actually write Clojure aside from the language itself?
14:50rhickeyChouser: if you do :refer-clojure do you want to do :load-resources as well? http://groups.google.com/group/clojure/msg/934d131d1f3b5bfc (might be nothing other than documentation change)
14:51Chouserrhickey: sure, I'll look at it.
14:52rhickeyDrewR: some. My main research work is in trying to make my computer hear (machine listening). Moving that work from C++/Mathematica to Clojure.
14:54abrooksrhickey: From Mathematica to Clojure? Interesting. I've never heard complaints about Mathematica (though most of the Mathematica users I've known were scientists and not software engineers). What shortcomings do you see in Mathematica that you'd like to improve on with Clojure?
14:56rhickeyI think Mathematica is extremely cool, but it is not fast enough to do the bulk audio processing, so I mostly use it for algorithm design and data visualization, with C++ doing the heavy lifting
14:58rhickeythe problem is I have had to do a lot of experimentation in the C++ code, which is extremely painful. Using Clojure there gives me something fast _and_ flexible - I expect to use Mathematica mostly for visualization in the end
14:59DrewRAre you a PhD candidate or is this corporate research?
14:59Chouserany hope of visualization libraries that would be comfortable enough to use directly from a Clojure repl that you'd be able to walk away from Mathematica?
14:59rhickeyDrewR: no and no. I'm a professional developer and this is self-funded
15:00DrewRAh.
15:00rhickeyChouser: definitely - I really haven't looked at the libraries available for Java, but it would be an exciting prospect
15:01rhickeyMathematica's visualization tools are really good though
15:01ChouserMy brother told me this weekend that his primary language right now is SciLab.
15:02ChouserHe likes it, especially hot-loading of code, but that it's painfully slow for heavy computation.
15:02ChouserI don't know much about SciLab, but it sounds like a similar problem space.
15:03rhickeyChouser: that's the problem, great for creating the models, but once you need to run 1000 filters at 44kHz...
15:04rhickeyespecially with any user-written functions - most of the built-ins are very fast C behind the scenes
15:04abrooksrhickey: Have you found Clojure able to handle that sort of throughput?
15:05rhickeyabrooks: It wasn't real-time even in C++, but the difference between seconds and days to get a result I needed
15:06rhickeybut yes, Clojure is fast enough, and scales well from experimenting to crunching
15:09jgracinIf I have a string containing clojure expression, how do I evaluate it from Java? I'm not necessarily interested in the return value.
15:10rhickeyjgracin: you need to read it, yielding a data structure, then eval that
15:15jgracinrhickey, is it with LispReader.read()?
15:15jgracinrhickey, seems a bit low-level
15:17jgracinoh, now I see how repl does it. it seems fine.
15:20ChouserIt'd be easy to wrap that all up in a thing that eval's strings, except that in general that approach is Discouraged.
15:22jgracinI'm writing a maven plugin which will evaluate user-provided expressions.
15:22Chouserah
15:41jgracinI'm getting java.lang.ExceptionInInitializerError at clojure.lang.Namespace.<init> if LispReader.read(...) is the first thing I do. Putting RT.init() before makes it work, but warns me that I don't have to use it.
15:46Chouserrhickey: currently the references in "ns" are evaluated in order, and each may be listed multiple times. Should I keep that behavior?
15:59rhickeyChouser: yes, please
16:02rhickeyjgracin: could you paste your failing example?
16:06lisppaste8jgracin pasted "No RT.init exception" at http://paste.lisp.org/display/66262
16:24rhickeyjgracin: fixed (rev 1015) - thanks for the report!
16:24jgracinrhickey, cool. thanks.
16:34ozzileeCan anyone point me towards something explaining the current state of the art in clojure libraries, with use and require etc?
16:35ozzileeAh, wow the Wiki has expanded since I last looked at it :-)
16:38abrooksagain
16:39ozzileeIs there a standard procedure for packaging libraries that require jar files?
16:40ozzileeSpecifically I want to write a library that wraps the Apache Commons HTTP lib.
17:01ChouserI think the answer would be the same as it would be for Java.
17:02ChouserCould you use maven to name prerequisites for your package?
17:03ChouserOr maybe it's best just to stuff everything you need into a single giant jar.
17:03ChouserClearly I don't really know. :-)
17:18ozzileeChouser: Right now I've got a bunch of jars in /usr/local/share/java/ that I require with add-classpath, which makes things a little more sane for scripts and exploratory programming and such. Maven is quite a bit heavier than what I'd like.
17:59cemerickozzilee: we always ship a single jar that contains all of the contents of the dependencies we require (obviously we only depend on libraries that have permissive licenses, but that may not apply to your work)
18:01ozzileecemerick: I'm not so much concerned about shipping as I am with writing stuff for myself to use. I'd like to be able to say (require 'mylib) in a fresh repl and have my lib pulled in along with all of the jars it needs, without having to dick about with the classpath.
18:02ozzileeThat said, I'd probably got with one big jar for shipping as well.
18:03cemerickozzilee: I agree wholeheartedly. FWIW, we use ivy to manage our dependencies within our ant build, and it's been very pleasant to work with (and a *lot* easier than maven, IMO)
18:06ozzileeWell, time to get out of here, thanks for the help.
18:31sirk390hi, I'm just starting clojure. Does anyone understand why (load (new java.io.StringReader "hello")) throws an Exception "java.lang.Exception: Unable to resolve symbol: hello in this context" ?
18:32shooverHi. Clojure is seeing a symbol hello, not a string "hello". Are you hoping for a string? Try embedding quotes inside the string, like "\"hello\""
18:33sirk390yes I was hoping for a string ^^
18:34sirk390Ok, that work much better now, thanks
18:35shooverNo problem
19:00rhickeyozzilee: I think you can put jars in a dir and specify it with -Djava.ext.dir= and not have to put each on classpath individually nor use add-classpath
20:38Chouserrhickey: how do you suggest keeping track of whether clojure's been loaded into a namespace?
23:12arohner(doc contains?) says that contains? takes a map, but it appears to work on vectors. Am I missing something, or are the docs incorrect?
23:15arohnerah, it's because vectors are functions of their indices
23:18Chouserright, but it won't tell you if a particular value is in a vector.
23:19arohnerI was using [0 1 2 3] as my test case
23:19arohnerso it worked exactly because the values equaled their indices
23:20arohnerwith [:a :b :c] it's a lot more obvious that it didn't work
23:20Chouserah, yep.
23:58Myomahow does one read to understand clojure compiler source code?
23:59MyomaI just tried to skim it but that does seem a bit futile
23:59arohnerare you trying to understand the whole thing or something in particular?
23:59Myomawell I would like to roughly understand it all