#clojure logs

2008-06-10

09:26cgrandDid you know that the function passed to pmap must not return nil?
09:36ChouserI did not.
09:37cgrandIt seems to be an unintended consequence of using a LinkedBlockingQueue.
09:38rhickeyany reason not to try parallel.clj?
09:39cgrandapart from being too lazy to install one jar? No :-)
09:40rhickeypmap may go away
09:43cgrandrhickey: oh, ok, I'll look at parallel.clj then.
09:44blackdogcoming from the non lisp world i find the docs sometimes difficult to understand, e.g. -> macro, i just worked out is a kind of "member access" for a struct
09:44blackdogit could be a good idea to note things like that in the docs
09:45rhickeyexcept it's not, it's a more general "through" or "thread"
09:45blackdogyes, that's why i said "kind of"
09:45blackdogbut for the beginner
09:45blackdogthat's maybe the predominant use case?
09:46blackdogi'm not sure, that's what I'm about to use it for :)
09:46rhickeyI'm not saying there couldn't be more explanation, right now the docs are a reference, so a bit terse
09:46rhickeybut misaligned analogies can confuse more than enlighten
09:47ChouserYou never *need* the -> macro, it just allows you to arrange your code a bit differently.
09:47rhickeyexamples would help, something I hoped would happen on the Wiki
09:47blackdogwell, i'd like to do a simple member access type thing to a map and I don't want to use (get) all the time
09:47blackdogis there another way?
09:48rhickeyyou hardly ever have to use get given maps are functions of their keys
09:48blackdogoh, this is something I've not understood then
09:48rhickeyand keywords are functions of maps, if you're using them as keys
09:49blackdogah ok, there are the missing links, thanks!
09:49rhickeyuser=> ({:a 1 :b 2} :b)
09:49rhickey2
09:49rhickeyuser=> (:b {:a 1 :b 2})
09:49rhickey2
09:50blackdogor maybe I'm the missing link ;) cheers
14:52rapidoerlang has proven that copy semantics make concurrency easy
14:53rapidoimmutable data structures have efficient copy semantics
14:53rapidobut how do immutable data structures scale in a distributed setup?
14:54rhickeyerlang unifies the local and distributed model
14:54rhickeyclojure doesn't
14:55rhickeysome reasons here: http://research.sun.com/techrep/1994/smli_tr-94-29.pdf
14:55rapidobut how do you efficiently share huge data structures along different distributed nodes?
14:55rhickeyerlang doesn't efficiently share huge data structures over the wire
14:55rapidoerlang doesn't solve that
14:55rapidoi believe there is a way to do that
14:56rapidousing distributed hash tables
14:56rhickeyI think the distributed case should be handled separately. I don't believe in transparent distribution
14:56rhickeysure, tuplespaces linda etc. Mnesia too
14:57rapidoi believe DHT is build to handle churn better than tuplespaces or mnesie
14:57rapidomnesie <- mnesia
14:58rhickeyDHT is sort of an implementation detail, no?
14:58rhickeymodel is still associative
14:58rapidono, i think it is an ideal substrate. you can leverage any datastructure on top of a DHT
15:00rhickeybut is hashing important in some non-implementation-detail way?
15:01rapido there are dht implementations that leverage near-keyness as opposed to (scattered) hashes.
15:02rapidoyou can build a persistent vector on top of a dht
15:03rapidojust store the sha-1 hash at every tree node
15:03rapidothe value is the serialization of the tree node. the key is its sha-1 hash
15:04rapidoreplace pointers to sub-trees with their sha-1 hashes
15:05rapidodoes that make sense?
15:06rhickeyit could, but I am unsure about whether you really want distributed versions of the same data structures you use in-proc
15:06rhickeyother options are to put Clojure under Terracotta
15:07rapidobut such data structures could scale to tera bytes, going beyond jvm's
15:07rhickeyso many other issues dominate - scalability, fault tolerance etc
15:08rhickeynot uninteresting, but a large problem space
15:09rapidoi've had my fare share of faulty disks, nodes etc.
15:10rapidothat's why i think immutability is so important
15:10rhickeyno argument here :)
15:11rapidoi generally don't like the 'encapsulation' argument
15:11rapidoi don't want an object to hide that it is connecting to sybase, opening files, etc
15:12rapidobecause when such object throws an exception
15:12rapidoit better close the sybase connection, file handle, etc
15:13rhickeyI make that point in most of my talks, encapsulation just means "I'm in charge of this mess", not that there isn't a mess
15:13rhickeynot a substitute for immutability by any stretch
15:14rapidoeveryone is re-implementing the same hard stuff <- resource management.
15:15rapidotry , catch .... oh, i forgot finally :)
15:17rapidoyou'd better don't catch c++ exception. it is almost certain that you've caught a memory leak at the same time
15:17rhickeyprogramming is not a solved problem yet
15:19rapidonot everybody agrees, but i think it was/is a good choice that clojure is build on top of the jvm
15:20rhickeyI have no regrets
15:20rhickeyas a new language, Clojure has a lot of utility
15:20rhickeyexcellent performance
15:20rhickeygreat portability
15:21rapidoits the magic of hotspot - not everybody appreciates the 1000 man-years that went into it
15:21rhickeyexactly
15:22rapidohotspot goes back to self technology
15:22rapidothe language self i mean
15:24rapidomay be shez scheme is the exception
15:24rhickeyI like the separation of concerns, I can focus on my language and not on code generation or socket libraries
15:25rapidoschez scheme achieves amazing compiler feats
15:25rapidobut it didn't took 1000 man-years to achieve those feats
15:26rhickeyyes, but hotspot's dynamism sets it apart
15:26rapidoagreed
15:27rapidoi've implemented my own language (enchilada)
15:27rapidofirst try: java
15:27rapidosecond try: java
15:27rapidothird: factor
15:27rapidofourth: factor :)
15:27rapidofive to seven: haskell
15:28rapidoand finally: scala
15:28rapidoi got addicted to a typed language after the haskell experience :)
15:29rapidobut i finally went for the jvm, exactly because i don't like to bother with native socket implementations
15:29rapidomyself
15:31rapidoi didn't consider clojure yet - sorry about that ;)
15:33rhickeythat's ok, I'll likely not get around to reimplementing Clojure in Clojure either :)
15:34rapidoaaahhhh, the art of bootstrapping
15:34rapidofactor has done it
15:35rapidofactor started on the jvm
15:36rapidoeventually, the factor compiler was written in factor
15:36rapidogenerating native code
15:36rapidofor various platforms
15:37rapido(still using the jvm implementation of factor i believe)
15:37rapidoi think that's pretty cool
15:37rapido(but i could be wrong there)
15:38rapidohmm, according to the definition, that's not strict bootstrapping
15:39rapido... bootstrapping is confusing ...
15:40rapidorhickey: what programming languages do you like? (apart from clojure of course)
15:41rhickeyCommon Lisp
15:42rhickeyI like the ideas in Haskell
15:42rhickeybut am concerned about type system expressivity
15:43rapidoyeah, types can hinder
15:44rapidoi like apl/j/k because it so different and still effective
15:44rapidoit so <- it is so
15:44Chouserthat's really key. It wasn't 'til I was trying to do an XML filter DSL in Scala that I realized how much static types can *really* get in your way.
15:44rhickeyI disagree with types in many ways. They represent an extreme narrowing of our ability to categorize
15:44Lau_of_DKSyntax question: This simple decoder should be able to take anything from 1 integer, to an infinate number of integers, it works with either 1 or 2... whats wrong?
15:45Lau_of_DK(defn decode
15:45Lau_of_DK ([x] (char (+ 97 (- (/ x 2) 1))))
15:45Lau_of_DK ([x &more] (apply str (list (decode x) (decode &more)))))
15:45rhickeyI think the type theorists wis
15:45rhickeyI think the type theorists wish programming ws
15:45rhickeywas math, but it isn't
15:46rapidodid you know that there are infinitely many types of numbers?
15:46rapidothey are surreal numbers
15:46rapido(just teasing there)
15:46la_merSpeaking of distributed computing (look at me, creating a tangent of a conversation from a half hour ago!), has anyone come across any reasonable overview of all of the different distributed computing frameworks available for Java apps (hadoop, terracotta, gigaspaces, coherence, etc., etc., etc.)?
15:47Lau_of_DKrhickey, you wrote the language, can you see why my overload isnt working?
15:47rhickey& more
15:48rhickeynot &more
15:48rapidola_mer: check out: freepastry, overlayweaver, bamboo, etc
15:50Lau_of_DKrhickey, okay, if I change the param to "& more" and in the function I change it to "more" then it compiles, but it throws an error when I run it with more than 1 arg - Is this because I need to add some splicing operator?
15:51rhickeyyou don't need the list call
15:51Chouser(defn decode ([x & more] (str (char (+ 97 (- (/ x 2) 1))) (when more (apply decode more)))))
15:52la_merrapido: Thanks for the names of more frameworks to look at. Do you know of any writeups comparing/contrasting them, their approaches, where they fit in the stack?
15:53Lau_of_DKChouser, rhickey, thanks guys :) You really fill the gap between here and descent tutorials :)
15:53rapidola_mer: this page has many links: http://www.sics.se/~sameh/p2pComputing.php
15:54rapidoi found it very valuable
15:54la_merrapido: Thanks for that. p2p is a whole area unto itself, I suppose.
15:55rapidola_mer: my gut feeling is that p2p *is* distributed computing
15:55la_merI saw on google groups that someone has put together a clojure wrapper for hadoop, so I suppose we'll start there
15:56rhickeypeople have played with NetKernel and Clojure too
15:56rhickeyI'm interested in trying Shoal (from Glassfish)
15:57la_merrapido: p2p means no central control (to me, anyway) -- we'll be looking to deploy to a grid architecture.
15:58rapidohey! i didn't know about shoal! thanks for the link rhickey!
15:58rhickeythe Shoal API is so simple, it's irresistible
15:58rapidola_mer: the world is made that way: there is no central control ;)
15:59rapidoahh, jtca
15:59rapidojtxa
16:00rapidorhickey: how did you find shoal?
16:01rhickeyinterested in trying, haven't tried
16:01rhickeygood enough for Glassfish clustering, apparently
16:01rapidola_mer: what's your stake on clojure and grid architectures? (who's *we* ?)
16:03la_merwe is http://snowtide.com
16:03la_merwe do a lot that isn't publicly available / marketed, FWIW
16:04rapidoto bad i don't live near western massachusetts
16:04rapidotoo bad
16:05la_merWe are developing a new service / product this summer, and I'm considering using clojure in part. The real meat of the functionality will be the processing of documents in varied and interesting ways, all of which are computationally (and storage) intensive. Not targeting a grid architecture would be foolish at this point.
16:05la_merrapido: Where are you now?
16:05rapidola_mer: the netherlands .. amersfoort (google that!)
16:07la_merrapido: Looks very pleasant :-)
16:08rapidola_mer: you would consider using clojure in a production setting?
16:08la_merrapido: Very close to it, yes.
16:08rapidothat means you have power
16:09la_merrapido: lol :-) In what way?
16:09rapidoor you are good in convincing 'management' to try something new to beat the 'competition'
16:10la_merI *am* management. :-D
16:10rapidola_mer: i rest my case...
16:10la_merWe're also a small company trying to solve tough problems, and we always need to be using the best raw materials we can lay our hands on.
16:11rapidola_mer: well, for me it is certain that you don't represent classical management
16:12la_merNo, I suppose not. Of course, it remains to be seen whether that's a good thing or not. ;-)
16:15rapidola_mer: managing people (=developers). that's difficult
16:15rapidoherding cats
16:16rapidoof course, managing non-developers is the real task
16:17rapidoand managing your own company is the most interesting 'challange'
16:20la_merIt's definitely a challenge. And, FWIW, I actually think that sales and "business development" is *way* harder than software development. People are hard, machines are easy.
16:25rapidogot to go..... i'm mutating to a different location ....
16:25la_merrapido: good talking with you
17:11Lau_of_DKFor string comparison, does Clj have something like if ("x" isin "extreme") ...true ?
17:12rhickeyClojure strings are Java Strings
17:13Lau_of_DKSo I gotta go with (. String compare) or something like that?
17:14rhickeyone of us will have to look it up, and it's not going to be me :)
17:15dudleyf(. "foobar" contains "foo")
17:15Lau_of_DKhaha
17:15Lau_of_DKrhickey, no I wasnt asking for the specific function, I would just prefer not to call anything (. java) if there already was something in lisp that did the same :)
17:15Lau_of_DKThanks dudleyf
17:16Chouseror (.contains "foobar" "foo") -- how much simpler could a lisp version be?
17:16rhickeyI'll probably not be adding (contains x y) in order to avoid (.contains x y)
17:17Lau_of_DKCome one guys :)
17:17rhickeyAll of the Java stuff is nicely documented, were I to wrap it re-documenting it would fall on me - no point
17:17Lau_of_DKI dont know - Im just prone to picking stuff that arent Java, as far down the road as I can
17:17rhickeythat's not going to let you leverage Clojure fully
17:17Lau_of_DKBut I get the point, so forget the stupid question :)
17:18Chousernot a stupid question! I definitely went through an "ew, I have to touch Java?" phase.
17:19ChouserBut it's pointless. You're on the JVM, you have the Java libs -- use them!
17:19Lau_of_DKYoure right
17:19Lau_of_DKAnd for IO and GUI Im all over them :)
17:19Chouserand string. ;-)
17:19rhickeyeveryone goes through that phase, I think
17:20Lau_of_DKThanks guys, thats good to know
17:20Chouserregex is a bit of an exception, I think, because the Java API is so... wordy.
17:20Lau_of_DKLast night I redid some of the Eulers that I had already handled in SBCL or Python, I was pretty amazed at the speed of Clojure
17:20rhickeyew, Java libs --> ooh, Java libs
17:20Lau_of_DKhehe
17:20Lau_of_DKThe libs are nice, and huge
17:20Lau_of_DKwhich is something I missed very badly in SBCL
17:21dudleyfI think a lot of Java API's could use some Clojure lipstick
17:21rhickeyChouser: regex still too wordy?
17:22rhickeydudleyf: fair enough, but driven by need and added value
17:22dudleyfrhickey: Oh, absolutely.
17:22rhickeyClojure's wrapping of fork/Join is pretty sweet vs. the Java
17:23rhickeyIMO
17:24rhickey.., doto etc a ripe area as well
17:25dudleyfI've always hated the Java IO API's
17:25rhickeynot sure how to fix them...
17:26dudleyfNo, me neither. It's kind of baked into their design.
17:26Chouserduckstream helps the IO libs a bit
17:26Lau_of_DKWhats to hate about them ?
17:26rhickeylayers
17:26Lau_of_DKThey seem pretty straight forward
17:26rhickeylayers
17:26Chouserrhickey: no, re-seq and re-find are good. I wrote a re-split that I like too.
17:26rhickeymore layers
17:26dudleyfI really don't like the decorator approach
17:26Lau_of_DKIf the performance is descent, why worry about layers?
17:27rhickeywordy
17:27rhickeyChouser: re-split does what?
17:27Lau_of_DKbtw rhickey, wasnt it tonight you were giving that talk to the java group ?
17:28rhickeylast Thursday, still haven't looked at the 3 hours of material
17:28Lau_of_DKSend them to me, I'll run through them in a jiffy
17:31rhickey2gb
17:31rhickeyprobably needs editing :)
17:32Chouseruser=> (re-split #"\\W" "different/separators between:these")
17:32Chouser("different" "/" "separators" " " "between" ":" "these")
17:32Lau_of_DKOk. How did they take it? Were you pitching Lisp in itself?
17:37baggicoman. i blab so much about lisp to my colleagues at work...
17:37baggicowell... when something sets me off anyway :)
17:38baggicopeople are generally either: "huh?" or receptive.
17:41rhickeyChouser: re-split wraps String.split?
17:42rhickeyor lazy?
17:42Chouserlazy
17:42ChouserThe implementation looks a lot like re-seq. ;-)
17:43ChouserI generally use it in combination with (partition) to do search/replace in strings.
17:44Chouser(for [[text url] (partition 2 (re-split #"..." text))] ...)
17:45Chouserthat's what converts plaintext urls in the irc log into <a> links
17:47Lau_of_DKthis I dont get
17:47Lau_of_DK(import '(java.lang.Math))
17:47Lau_of_DK(. abs -5)
17:47Lau_of_DKError: Abs not found in this context
17:47Lau_of_DKbut it works fine in Java
17:47Chouser(import '(java.lang.Math abs))
17:48Lau_of_DKgives the same error
17:48rhickeyall of java.lang is already imported
17:48Chouseroh, sorry.
17:48Chouser(. Math abs 5)
17:48rhickeythere you go
17:51ChouserLau_of_DK: get it?
17:57Lau_of_DKYes sir, thanks
17:58Chouservery good. :-)