#clojure logs

2008-11-05

03:52Lau_of_DKMornings gents
03:53Lau_of_DKI have not yet tried Clojure web-capabilities for servlets and such, how much does one benefit from its concurrency features in that regard?
04:17Lau_of_DKOr: Where would one benefit the most from applying Clojure on the web ?
04:26cgrandLau_of_DK: Comet?
04:29Lau_of_DKI'll look into it
11:06duck1123does anyone ever from time to time get the error: Exception in thread "Read Loop Thread"
11:10duck1123yes, this is in *inferior-lisp* when starting slime
11:10duck1123it doesn't appear do cause any problems, it's just odd
11:11Chousersorry, I haven't seen it
11:23drewrHow do I prn from a different thread?
11:23drewrDoes *out* get mucked with?
11:24drewrI'm trying to debug something that happens in a send. I guess I could throw and look at agent-errors.
11:27wwmorgandrewr: You could also do a (.println System/out "message")
11:30kwatfordHrm. That doesn't seem to work either.
11:30Chouserworks for me: (send-off (agent nil) (prn :hi)
11:30Chouser)
11:31Chouseroh, scratch that.
11:32Chouserproves nothing
11:32Chouserok, this works: (send-off (agent :hi) #(do (Thread/sleep 1000) (println %)))
11:35drewrAlso, what's the diff between send and send-off?
11:36drewrChouser: That doesn't print anything in my REPL.
11:36Chousersend uses a pool of threads sized based on your processor core count
11:37Chouserso if you do a send on a function that blocks for IO or sleeps or something, you're gumming up the works.
11:37Chousersend-off uses a thread from a pool that grows as needed, so it's ok to block.
11:38Chouserdrewr: odd. you're at a regular terminal REPL?
11:38drewrNo :-)
11:38drewrThat's probably why.
11:38kwatforddoesn't seem to need the sleep in emacs. I was just forgetting to make the lambda take an argument.
11:39Chouserthe sleep isn't required, just makes is clearer whats going on.
11:40ChouserWithout it I tend to see:
11:40Chouser#<:hiclojure.lang.Agent@1a422f6
11:40Chouser>
11:41kwatfordah, so that's why the raw repl gave me #<clojure.lang.Agent@71537:hi>
11:42duck1123hmm, I saw it as this: user=> #=(clojure.lang.Agent. "clojure.lang.Agent@36db4bcf")
11:42duck1123user=> :hi
11:42duck1123
12:48kwatfordhrm. Why does for not support use of :when and :while at the same time?
12:49wwmorgankwatford: you can combine them into a :while, right?
12:49AWizzArdMoin
12:51kwatfordcontrived example: (for [x (iterate inc 0) :when (even? x) :while (< x 10)] x)
12:51kotarakAWizzArd: hi wizz
12:53kotarakkwatford: Quick Hack(tm): (for [x (iterate inc 0) :when (even? x) y x :while (< y 10)] y) (not tested)
12:53kwatfordIllegalArgumentException: Don't know how to create ISeq from Integer
12:54kotarakkwatford: oops, (for [x (iterate inc 0) :when (even? x) y [x] :while (< y 10)] y)
12:54kwatfordOutOfMemoryError
12:56kotarak(for [x (iterate inc 0) :while (< x 10) y [x] :when (even? y)] y)
12:56kotarak(0 2 4 6 8)
12:57kwatfordThere we go
13:04sohailyou need the monstrous cl loop macro
13:04sohail:-D
13:05kotarak(take-while #(< % 10) (filter even? (iterate inc 0))) should also work, no?
13:05kwatfordheh. It's been a while since I touched CL, but I do distinctly remember avoiding loop
13:06kwatfordkotarak: Yes, but I was more wondering "why doesn't this work?" than "how can I make it work?"
13:08sohailkwatford, it's an acquired (dis)taste
14:16Lau_of_DKEvening all!
14:16Lau_of_DKAnd a good one at that :)
14:20AWizzArdhi Lau
15:10Lau_of_DKIs (remove) something new? I dont seem to have it
15:10Chousernewish, yeah
15:10kotarakrelatively new
15:10ChouserOct 16
15:11Lau_of_DKHmm
15:11Lau_of_DKI downloaded yesterday
15:11Lau_of_DKIs the for SVN users only?
15:11kotarakI only use SVN.
15:11kotarakUnless told otherwise by Rich.
15:11Lau_of_DKgot the svn adress handy ?
15:12kwatfordTop right corner of the webpage
15:12kotarakI use the hg mirror
15:12duck1123http://clojure.googlecode.com/svn/trunk/
15:13duck1123there's a git mirror too if that's more your cup of tea
15:14abrooksduck1123: rhickey's still using SVN on sf.net for now.
15:15abrooksduck1123: http://clojure.svn.sourceforge.net/viewvc/clojure/
15:16Lau_of_DKWhat is 'Cannot execute mojo resources' ?
15:16duck1123hmm... I guess you're right. I didn't bother to check the repository, I just knew there was a google code page
15:19Lau_of_DKAs I remember, its just
15:19Lau_of_DK svn co https://clojure.svn.sourceforge.net/svnroot/clojure clojure
15:19Lau_of_DKmvn install
15:19Lau_of_DKright ?
15:20kotarakI use ant
15:20wwmorganant
15:20duck1123just 'ant' is fine
15:20Lau_of_DKtrying
15:20Lau_of_DKSuccess
15:20Lau_of_DKThanks
15:22danlarkinduck1123: saw my clojure-json eh?
15:24duck1123yeah, I just started watching it. I planned on looking through it a little later
15:25duck1123how hard would it be to translate clojure code into json?
15:25duck1123it would be interesting to store code in couchdb
15:32danlarkinduck1123: I only have a clojure->json encoder in there right now
15:32danlarkinno json->clojure yet
15:33duck1123I just realized that github won't show a project if it's listed as a fork of another project
15:33duck1123that's kind of annoying
15:34danlarkintranslating code into json would be a little difficult, since functions are closures on their environment
15:34danlarkinI could use some help with my encoder, actually, it's a little slow for my taste
15:34blackdogwhat's the link to the clojure-json?
15:35Chouserjson (the formal spec) doesn't allow functions either.
15:35Chouserdo you mean clojure code to javascript code?
15:35danlarkinChouser: but it does allow lists :)
15:35Chouseryes it does. and maps and strings
15:36Chouseroh... I see what you're saying.
15:36danlarkinblackdog: my clojure-json is located at http://github.com/danlarkin/clojure-json/tree/master
15:36blackdogthanks
15:37Chouserbut how would you differentiate between vectors and lists?
15:38duck1123you would probably have to do it as (vec 1 2 3)
15:38danlarkinChouser: Hmmmmm, not sure, maybe you wouldn't be able to
15:38danlarkinduck1123: ah yes, or that
15:39duck1123danlarkin: I'm curious, have you tried using org.json.JSONObject in clojure?
15:40drewrduck1123: I have, and it works OK.
15:40drewrIt doesn't like j.u.Date very much.
15:42danlarkinduck1123: I have not. I wanted to be able to use clojure datastructures natively. Also I wanted to program a json interface my self as a learning exercise
15:43duck1123nothing wrong with that
15:44danlarkinI have a branch that uses Writer objects instead of strings, http://github.com/danlarkin/clojure-json/tree/writer
15:45danlarkinI have more on my TODO list regarding that, I need to add flushes so it doesn't blow up the heap
15:46Chouserdanlarkin: might be a good candidate for multimethods
15:55danlarkinChouser: what might be?
15:55Chouseryour json printer
15:55danlarkininstead of a cond?
15:55Chouseryeah
15:56danlarkinwouldn't the multimethod dispatch have to do a cond anyway?
15:57Chouserlooks like you could dispatch on class
15:57Chouserthen match Boolean, nil, String, Number, etc.
15:58Chousermaybe it wouldn't be any better than what you've got.
16:01danlarkinI do have some performance inefficiencies, I don't think cond is one of them though
16:01drewrI thought there was an (into-map {:foo :bar}) now. What am I thinking of?
16:02drewrDon't Clojure's maps now implement j.u.HashMap?
16:02drewrEr, j.u.Map?
16:05wwmorgandrewr: yes. (into {} m) will take a java Map m and return a clojure Map
16:05drewrwwmorgan: I want to go the other way.
16:05wwmorganYou get that for free: (instance? java.util.Map {}) returns true
16:06danlarkin(instance? java.util.Map {}) == false for me
16:07wwmorganI'm on svn head. I don't know when the change was added
16:08duck1123this came up yesterday
16:08duck1123sometime early last month
16:09danlarkinI'm on an older checkout
16:09danlarkinhaven't updated in a while
16:09duck1123Oct 6
16:11danlarkinthe real problem is that apply walks the entire list it's passed, so I'm trying to rewrite everything not to use apply
16:11duck1123I thought apply was lazy
16:13danlarkinhttp://blog.danlarkin.org/2008/11/apply-is-not-lazy/
16:13kotarakwhat would thatt help? Knowing the source is a smell. You should rely on the interface, not the implementation.
16:13Lau_of_DKGuys - Is there a non-nil?, a shorter version of (not (nil? x y z))
16:15kotaraksome
16:15kotarakor maybe every?
16:15duck1123kotarak: sometimes it helps to see the code to realize something that is not obvious from the documentation
16:15kotarakduck1123: and which might suddenly change.
16:16Chouserapply *is* lazy. str is not
16:17Lau_of_DKI hardly think 'apply' is as lazy as Chouser...
16:17duck1123the implementation might change, but the behavior generally does not.
16:17Chouserhardly anything is as lazy as me
16:18duck1123There needs to be Clojure shirts that say "I get more done when I'm lazy" or some such
16:18Chouser(defn first2 [a b & c] [a b])
16:19kotarakduck1123 haha, that's good. Maybe the Haskell guys have it already.
16:19Chouser(first2 1 2 3 4) -> [1 2]
16:19ChouserIf apply weren't lazy, this would never return: (apply first2 (iterate inc 1))
16:19danlarkinChouser: apply is lazy? it looks like the spread helper function isn't though
16:20Chouserbut in fact it returns [1 2]
16:20kotarakproof by counter-example
16:21danlarkinkotarak: you're right, it isn't a proof at all, but certainly good enough to prove me wrong
16:22Chouserspread is eager for the required args of the func you're applying
16:22Chouser...it has to be to match Java calling conventions, I think.
16:23Chouserbut once the required args are met, the "rest" arg is left as a (possibly lazy) seq
16:24Chouserbut str needs to build a non-lazy string, so it has to consume all its args, whether the seq it's pulling from is lazy or not.
16:25Chouserand by the way, (apply str (range 10e6))
16:26duck1123hmm, in that case, would it be better to work with seqs of chars until you're ready for output?
16:26Chouserworks fine for me, though (apply str (interpose "," (range 10e6))) does not
16:26scottjI often want to link to some identity, not copy it, in a sort of relational database manner. Like bob has his details and one of them is his hometown and that hometown has it's own details. Is the recommended method to make bob and his hometown each refs and then include the hometown ref in bob's map but not @ it until I want to follow the link?
16:28Chouserscottj: you want to be able to update the town's details and see those changes for all peoplewho live there?
16:30scottjChouser: yeah
16:30danlarkinChouser: so would applying to a lazy-consumer make sense? a function that takes [x & ys] and then uses (loop [more ys] ... (recur (rest ys)))? Or is that not really helping anything
16:30scottjChouser: so either I use some primary key (name, id number) and look that up, or I link with a ref
16:30Chouserscottj: I think using a ref sounds reasonable.
16:31Chouseryou still might what the town refs all in one map (or set?) just for sanity.
16:32Chouserdanlarkin: generally using loop/recur like that means it's *not* a lazy-consumer.
16:33danlarkinChouser: well I do have to make a string out of it, I just want to do it in a way that doesn't fill up the heap
16:33Chouserdanlarkin: you brought that up the other day, and it's been troubling me.
16:34ChouserI don't understand yet how (apply str (range 10e6))
16:34Chousercan runs so quickly with so little memory impact
16:34Chouser...when using interpose in the kills my memory and CPU
16:35duck1123later folks
16:35danlarkinduck1123: bye bye
16:35duck1123driving home
16:35danlarkinChouser: how long does (time (dorun [(apply str (range 10e6))])) take for you?
16:36Chouser"Elapsed time: 2425.807705 msecs"
16:36danlarkinand can you run (time (dorun [(apply str (range 10e7))])) or does it blow your heap
16:37ChouserOutOfMemoryError
16:37danlarkinI know what you mean though... interpose should be lazy
16:38abrooksIt's really interleave that should be lazy, I think.
16:39wwmorganChouser: what about 20e6? You're probably just bumping up against your jvm heap size right?
16:39danlarkinis it possible that interleave isn't? looks like it uses map...
16:39Chouserdanlarkin: with 10e7, you're trying to allocate something like a 1GB string
16:40Chouserinterleave claims to be lazy
16:40abrooksdanlarkin: Right. interleave claimes to be lazy but it doesn't appear to be.
16:41wwmorganeven if interleave were lazy, we wouldn't see it because str is not lazy
16:41Chouserinterpose acts lazy (take 10 (interpose "," (lazy-cat (range 10) (print "oops"))))
16:42ChouserI don't think it's a laziness issue. It *might* have to do with something hanging onto the head of the seq, but I'm not sure.
16:43wwmorganchouser: you mean why does (apply str (range 10e6)) work but (apply str (interpose "," (range 10e6))) blow up?
16:43Chouserwwmorgan: yes
16:44wwmorganI think it's just that you're hitting your JVM's max heap size. The String you're trying to build is too big
16:44ChouserI don't think so.
16:45Chouserwithout interpose I see very little extra memory usage, and I get a result in under 3 seconds.
16:45Chouserwith interpose, I would expect a string that's maybe 10% larger and shouldn't take more than maybe double the time to produce.
16:46Chouserbut with interpose I actually see my memory usage climb steadily and noresults for many seconds -- I've never let it finish as I fear for my other processes. :-)
16:47wwmorganwhat happens when you do (apply str (interpose "," (range 10e5))) ?
16:48Chouserclimbing memory usage, but a good result in about 3 seconds.
16:57Lau_of_DKThrow me a bone guys
16:57Lau_of_DK(def f "test")
16:58Lau_of_DK(re-seq #"test" seq) works
16:58Lau_of_DK(re-seq #f seq) does not of course
16:58Lau_of_DKbut how do I make it work ?
16:59kotarakIs there something along the lines of (new java.util.Regex f)?
16:59Chouser(re-seq (java.util.regex.Pattern/compile f) seq)
16:59Lau_of_DKuuh fancy Chouser
16:59Lau_of_DKHow do you come up with all this stuff?
16:59Lau_of_DKI mean... its almost like you and Rich have a secret book
16:59ChouserBut don't use "seq" as a local name, since it's a clojure/seq (function)
16:59ChouserUse the Source, Lau..K.
17:00Lau_of_DKHaha :)
17:00Lau_of_DKThanks alot guys
17:00Chouserand be careful building that string
17:03Chouserdanlarkin: are you going to update your blog, or am i going to have to leave a comment?
17:03danlarkinChouser: you'll have to leave a comment, I'm at work now :)
17:03Chouserhmph.
17:03Lau_of_DKChouser, , dont worry, I'll leave it totally in the hands of the competent user
17:07danlarkinso what's a lazy way to consume (interpose "," lst)? And If I do consume lazily and use a Writer and flush() it every time that'd cut down on the memory usage, wouldn't it?
17:08ChouserI haven't been able to figure out where the memory usage is coming from.
17:08wwmorganI have an idea
17:09wwmorganstr uses a StringBuilder. StringBuilder maintains an internal buffer. If you append to a StringBuilder over the buffer, it has to reallocate the whole thing
17:13wwmorganbut simply consuming an interpose doesn't use up memory: (reduce (fn [_ _] 1) (interpose "," (range 10e6)))
17:28lisppaste8danlarkin pasted "consuming interpose" at http://paste.lisp.org/display/69773
17:31wwmorganI don't think StringWriter's flush method clears its buffer
17:32wwmorganso you're still trying to retain (range 10e7) in memory
17:34lisppaste8wwmorgan annotated #69773 with "Doesn't consume memory" at http://paste.lisp.org/display/69773#1
17:40danlarkinwwmorgan: wow look at that, it doesn't. Hrmph, so how do I do this?
17:41rhickeydanlarkin: do what?
17:41AWizzArdwb
17:41danlarkinelection went smoothly I see
17:41danlarkin:)
17:41wwmorgandanlarkin: the function that you call needs consume a constant amount of memory no matter how many times you call it
17:42danlarkinrhickey: I'm writing a json encoder and (apply str (range 10e6)) is blowing the heap, but I need a way to consume huge lists like that
17:43wwmorganif you're sending it over a network, then your function call wraps the network message-sending interface
17:49rhickeydanlarkin: went well, yes, thanks - you can reduce with a StringBuilder
17:54danlarkinrhickey: is a stringbuilder efficient? doesn't it have to discard and double its buffer every time it fills up? or is that the best way to do it anyway
17:58Chousukeit's difficult to be more efficient when you need a dynamically growing chunk of contiguous memory
17:59wwmorgandanlarkin: do you need the whole String in memory at once?
18:00danlarkinwwmorgan: no
18:01wwmorganwhat do you want to split the String up into?
18:03danlarkinwwmorgan: well like I said it's for a JSON library, so I assume it should be a Writer if it can't be a string, since then the user of the library can pass in a StringWriter or FileWriter or whatever
18:03lisppaste8wwmorgan annotated #69773 with "Splits the String up into Strings of 1024 characters each" at http://paste.lisp.org/display/69773#2
18:04wwmorganoh, if the user is passing in the writer, then it's his own fault if he runs out of memory
18:05lisppaste8wwmorgan annotated #69773 with "slightly better" at http://paste.lisp.org/display/69773#3
18:08wwmorganfor your tests, just pass in a Writer that discards all its inputs
18:13SnowBuddywhen I say (loop ... (recur)), I get an ArrayIndexOutOfBoundsException. can somebody explain why that happens and what exactly loop..recur is supposed to be doing behind the scenes? the documentation suggests that the arguments are optional (0 or more), so I figured I could do something like that
18:15danlarkinwwmorgan: partition throws out any remainder, (doseq cs (partition 3 (interpose "," (range 10))) (print (apply str cs))) prints 0,1,2,3,4,5,6,7,8,
18:15wwmorgandanlarkin: right, so you'd need your own partition that didn't do that
18:15gnuvince_SnowBuddy: try: (loop [] ... (recur))
18:16lisppaste8danlarkin annotated #69773 with "invocation target exception" at http://paste.lisp.org/display/69773#4
18:16danlarkinsorry if I'm being thick-skulled... maybe there's something I'm missing
18:23lisppaste8wwmorgan annotated #69773 with "try this" at http://paste.lisp.org/display/69773#5
18:26SnowBuddygnuvince: that worked, thanks. looking at the documentation, i see now that the brackets are required :P
18:28rhickeydanlarkin: if you are not accumulating in memory, then you must be either reducing your input or doing output as you go, so (dorun (map do-a-bit stuff-too-large-for-memory)) should work
18:41gnuvince_SnowBuddy: happy to help
18:43SnowBuddynow i just need to figure the rest of this stuff out ;p
18:44Kerris4I just bought the beta pdf of Stuart Holloway's Clojure book from Pragmatic Programmers :D
18:44Kerris4ohboyohboyohboy
18:44SnowBuddyhow is it?
18:45Kerris4hang on, I've just downloaded it
18:45Kerris4:P
18:46SnowBuddyheh
18:47Kerris4I'm no Lisp/Clojure veteran but a brief scan shows that it is suitable for people coming from a Java background
18:47Kerris4I'd use this to get my classmates into Clojure as the school teaches Java
18:48SnowBuddyi'm coming from a C/C++/C# background, so i may pick that one up
18:49Kerris4Have you spent any amount of time with Lisp or Scheme or Haskell before, SnowBuddy?
18:50SnowBuddyI've played briefly with CL before discovering Clojure a few days ago
18:52danlarkinrhickey, wwmorgan: thanks for your help, I've gotta drive home now
18:53danlarkinoh, wwmorgan left :-/
18:54Kerris4I can't give a firm recommendation on the book as I'm of the opinion that you could learn a lot of Clojure for free by following Stuart Holloway's port of Practical Common Lisp to Clojure :P
18:55SnowBuddyi could learn a lot of Clojure by reading code and browsing the documentation, but that's tedious when just starting out ;)
18:56Kerris4true, and with his port you can just type in the code and watch things work
18:56Kerris4http://blog.thinkrelevance.com/2008/9/16/pcl-clojure
18:56Kerris4if you didn't know already, Practical Common Lisp is free
18:56Kerris4http://gigamonkeys.com/book/
18:56SnowBuddyyea, i read a bit of it when I was looking at CL
19:04kwatfordThe book is missing several chapters as of yet, so I wouldn't hand it to someone wanting to learn just yet. But should probably be much better when it's actually done ;)
19:24hsuhwhois jao
21:29dakroneI apologize in advance if this is a silly question, but can someone tell me why http://paste.lisp.org/display/69785 hangs when I run it from the command line?
21:29dakroneI don't quite understand how to use agents correctly
21:38Chouserdakrone: those files exist?
21:39dakroneChouser, yep
21:39dakroneI get:
21:39dakrone? clj ciosim.clj
21:39dakronefile3
21:39dakroneReading file: file1
21:39dakroneRead bytes: 24
21:39dakroneand then it hangs until I ctrl+c
21:44Chouserdakrone: It doesn't hang for me, I get: java.lang.Exception: Agent has errors (NO_SOURCE_FILE:0)
21:45rhickeyChouser: did you consider extending RestFn for callable defstruct ?
21:45dakronerhickey, huhwhatnow?
21:46Chouserrhickey: heh. nope. I guess I'm out of my depth?
21:46dakroneChouser, just tried it in the REPL instead of from the file, it doesn't hang, is there a reason why it would hang when being run as a script instead?
21:46rhickeymight make it easier - I didn't look too closely yet
21:48Chouserrhickey: if the goal is acceptible, I'll look at it.
21:49Chouserdakrone: you were trying it with Script?
21:50dakroneChouser, yep
21:52dakroneare there any good super-basic threading tutorials for Clojure, other than the ant simulation?
21:58rhickeyChouser: yes, sure
22:06Chouserdakrone: I don't know why it's hanging.
22:07dakroneChouser, okay, I appreciate you trying though :)
22:07dakronethanks
22:09Chouserdakrone: are you getting the agent errors when you runin the repl?
22:09dakroneChouser, nope, no errors are showing up
22:10dakronelatest version of Clojure from svn also
22:13Chouserwell, I can replicate it here, but I don't know of and can't see any difference between Script and Repl that would cause this.
22:14dakronehmm, that's very strange
22:20dakronehmmm....interesting
22:21dakroneOSX 10.5, java 5, 32-bit gives no errors, java 6, 64-bit gives the Agent has errors error
22:41bradbevis there an easy way to map from a java method name like user.baz__3364 to a clojure symbol user/baz? Obviously I could drop the __3364, but I'd prefer the correct way to do it rather than a heuristic.
22:44sohailChouser, if clojure was not a lisp-1 you could have done it quite easily ;-)
22:46Chousersohail: hm? what?
22:46Chouserbradbev: you only have the method name, not the actual var or fn?
22:46sohailmaking struct callable?
22:47Chousersohail: oh.
22:47Chouserit wasn't too hard, but I leaned heavily on vim macros. RestFn is probably a better way.
22:47bradbevChouser: I have the stack trace that comes out at the slime repl. I figured I could make a little patch that would look up the stack entries & try to make them look like native Clojure names
22:49Chouserbradbev: can you inspect the stack object itself, maybe get to the class objects, rather than munging the printed text?
22:51bradbevChouser: I think so, but StackTraceElement.getClassName for (defn foo) gives me user.foo__1234. So I guess that Clojure gensyms up the internal java name. That way you can redefine functions & keep the old def in a closure
23:08kwatfordbrabev: doesn't look to me like the function object/class keeps that information. I guess you could parse the name and then check that the guessed symbol holds the function. Unless it didn't come from defn.
23:10bradbevkwatford: yeah, I can't see it either. Oh well, heuristic it is.
23:19sohailChouser, couldn't you also do the "make struct callable" by making struct a macro? (defstruct foo ...) -> (def foo-struct ...) (defn foo [...] ...)
23:19sohailand then the make-struct macro appends -struct to the symbol. anyway just a thought :-)
23:23ChouserOr just http://tinyurl.com/5r42n6