#clojure – 2008-09-26

2008-09-26

07:20	leafw	is zipmap lazy?
07:33	fyuryu	leafw: after looking at the definition, I don't think so
07:33	fyuryu	leafw: it returns a map built in a loop/recur
07:34	leafw	thanks.
07:39	leafw	"No matching ctor found" means "you can't instantiate an abstract class". Could use some makeup ...
07:40	leafw	well, actually, it's true that the abstract class didn;t have that constructor.
07:48	Chouser	You also get that error if you provide wrong argument types to the constructor of a concrete class.
07:48	Chouser	as you might expect.
09:57	drewr	Is there a way to tell how much memory an object is using?
10:10	fyuryu	drewr: if you mean something like sizeof() in C: not really, almost everything is a reference
10:11	drewr	I didn't know if Object had some clues to where I could do some investigation.
10:12	drewr	I'd really like to find out how much memory a collection is using so I can determine the optimum number of rows to return in a db query.
10:15	ozzilee	drewr: http://java.sun.com/j2se/1.4.2/docs/api/java/lang/Runtime.html Does that help?
10:16	ozzilee	I don't know if anything in there will do much good.
10:16	drewr	It's not object-specific, but it's helpful. Thanks.
10:17	ozzilee	Just calling the freeMemory method from Clojure seems to allocate memory, so you'd have to account for that, and who knows what else.
10:17	ozzilee	Probably there's a better way.
11:08	drewr	Is there something wrong with conj'ing a map in a commute?
11:08	drewr	hash
11:08	drewr	Er..
11:08	drewr	(def pool (ref (sorted-map))) (dosync (commute pool conj {(:id @agt) agt}))
11:09	drewr	I get a ClassCastException: clojure.lang.PersistentHashMap because of the map after the conj.
11:09	drewr	Agent state looks like {:id 123, :conn jdbc.Connection...}.
11:11	Chouser	conj on map wants a vector I think
11:11	drewr	wm> (conj {} {:foo 1 :bar 2})
11:11	drewr	{:bar 2, :foo 1}
11:12	drewr	This even works:
11:12	drewr	wm> (conj (sorted-map :quux 3) {:foo 1 :bar 2})
11:12	drewr	{:bar 2, :foo 1, :quux 3}
11:12	drewr	I can't figure out what's unique about my situation.
11:12	Chouser	huh, I thought you needed "merge" for that. hm...
11:14	ozzilee	drewr: It looks like it's saying that pool is a PersistentHashMap, not a Ref.
11:15	ozzilee	Are you maybe doing (commute @pool ...) ?
11:15	Chouser	If I do: (def agt (ref {:id 123, :conn "aconn"})) ...and then run your example, it works fine for me.
11:16	Chouser	or if I do (agent {...}) instead of (ref {...}), it also works.
11:16	ozzilee	This is a prime example of where clojure needs better error messages :-)
11:17	drewr	ozzilee: You aren't kidding. I can figure out that it's the conj that's complaining, but that's about it.
11:17	Chouser	there's some difference remaining between your real code and what you've posted here, since what you've posted works fine.
11:18	lisppaste8	drewr pasted "commute ref" at http://paste.lisp.org/display/67484
11:19	lisppaste8	drewr annotated #67484 with "Actually, I'm using conj" at http://paste.lisp.org/display/67484#1
11:22	drewr	Call it like (add-agent agent-pool (make-agent conn)).
11:23	lisppaste8	drewr annotated #67484 with "better summary" at http://paste.lisp.org/display/67484#2
11:25	drewr	Am I off base here with the general idea?
11:25	lisppaste8	drewr annotated #67484 with "exception" at http://paste.lisp.org/display/67484#3
11:28	Chouser	what happens if you use a hash-map instead of a sorted-map?
11:30	drewr	Hm, that works.
11:31	Chouser	the clue is the PersistentTreeMap calling compare, and then failing
11:31	drewr	Yep.
11:31	Chouser	all the keys in a sorted-map have to be comparable to each other.
11:32	drewr	In this case they're integers, which should be straightforward.
11:32	drewr	That's why I threw in a sorted-map. I might as well have them in order when I'm looking at it.
11:33	Chouser	you can dump out the hash afterwards and see they's nothing unusual in there?
11:33	drewr	Yep, here's from an earlier version:
11:33	drewr	{1 clojure.lang.Agent@bad094, 2 clojure.lang.Agent@d3cae0, 3 clojure.lang.Agent@ac5024, ... }
11:34	drewr	That might have been before I used a sorted-map though.
12:47	ozzilee	drewr: java.lang.instrument.Instrumentation/getObjectSize
12:48	ozzilee	Can't get it to work here, though.
13:49	aperotte	Hello everyone, I had a quick question about agents
13:51	aperotte	If you have an agent with a collection in it, and you want to change an element of that collection, does the collection need to be a collection of refs and does the change have to happen in the context of a transaction?
13:53	ozzilee	aperotte: Hopefully someone more knowledgeable will chime in, but I don't believe so.
13:54	aperotte	ok, thanks ozzilee. I have just started experimenting.
13:54	ozzilee	http://clojure.org/state I think the second paragraph under "Concurrency" will explain things.
14:00	avida	aperotte: collections are immutable, so you need to assign a new collection to it using send
14:01	shoover	aperotte: An agent's value can only be changed by sending an action to the agent. The action is passed the agent's current value as an argument, and then you can do whatever you want with that argument. Whatever the action returns becomes the agent's new value.
14:02	shoover	If the value is a collection containing Refs and you want to change those Refs, you'll need a transaction around the Ref changes
14:03	avida	aperotte: (def ag (agent '(1 2 3))) (send ag #(conj % 4)) @ag
14:03	avida	aperotte: clojure that'll insert a 4 to the list (1 2 3) and assign it to the agent
14:03	aperotte	fantastic
14:04	aperotte	that's exactly what I wanted to know
14:04	aperotte	oh, one more thing
14:04	aperotte	is it possible for two agents to share a collection?
14:05	avida	aperotte: yes, since they are immutable anyone can share a collection without worrying aboutchanges
14:05	avida	when you send a change to an agent, you actually create a new collection (although efficiently shared behind-the-scenes)
14:05	aperotte	ahh, but if I make "changes" to the collection in one agent
14:05	aperotte	it won't show up in the other
14:05	avida	so two agents can share one collection, but when you make changes to one agent, you actually end up with two different collections
14:05	aperotte	I see
14:06	aperotte	Then, if I have two agents sharing a collection of refs, and change the refs within a transaction, will the changes show up in both?
14:08	avida	i think the refs stay the same, only the values they refer to have changed, so if your collection is of refs, then the collections wont change, however you can reference the changed values of the refs
14:08	aperotte	ok, I think I'm going to try it out
14:09	aperotte	thanks avida, thanks shoover, thanks ozzilee
14:29	aperotte	It worked ... albeit a bit of a round about way of getting at what I wanted
14:29	aperotte	(def x [(ref 1) (ref 2) (ref 3)])
14:29	aperotte	(def a1 (agent x))
14:29	aperotte	(def a2 (agent x))
14:29	aperotte	@(get @a1 1)
14:29	aperotte	(dosync (ref-set (get @a1 1) 10))
14:29	aperotte	@(get @a1 1)
14:29	aperotte	@(get @a2 1)
14:31	avida	yep, you never changed the values of a1, a2 so you would always see the same refs
14:32	aperotte	The aim is to have the agents operate on themselves in this way asynchronously and independently
14:33	aperotte	I'm not sure that this is correct, but this might be exactly the kind of thing clojure was made to avoid
14:36	avida	aperotte: ops on the agents are async but you're updatings refs which are synchrnous
14:37	aperotte	if I used send to tell an agent to (dosync (ref-set ... , that would make it asynchronous right?
14:37	avida	you could read from the refs, and change the refs themselves in the collection async
14:38	aperotte	how could I read and change the refs async without using agents?
14:39	avida	i think what you are doing is good, im not thinking this through all the way
14:53	shoover	updating refs in an agent action is asynchronous as far as the rest of the program is concerned. from the perspective of the action, its own changes to the refs are synchronous
14:56	shoover	sorry. aperotte: if you tell an agent to (dosync (refset ..)), then yes, that makes an asynchronous change to the ref
14:57	shoover	but within the action that executes the ref-set, you'll see the new value right away
14:58	aking	Looks like Rich's slides from the jvm summit are up: http://wiki.jvmlangsummit.com/pdf/27_Hickey_clojure.pdf
14:59	lisppaste8	shoover pasted "Agent sees new value" at http://paste.lisp.org/display/67493
15:02	aperotte	thanks guys
15:02	shoover	That paste doesn't prove the async part of the above statements, but we'd need a real problem to solve for that
15:07	aperotte	The motivation behind this is to write a machine learning prototyping system where each element/layer contains and can operate async on an input vector to create an output vector that is shared with another element/layer as it's input layer (a connection)
15:08	aperotte	the last "layer" should be "vector"
15:10	aperotte	I wanted to make the elements operate async by using agents, but then the shared data that represents the connections between the elements becomes problematic
15:10	ozzilee	aperotte: Is this something like Cells? http://common-lisp.net/project/cells/
15:12	shoover	You should be able to get what you want by sharing refs. If you don't like that you may want to check out the agent watchers that were added recently in svn
15:14	joubert	rhickey: I started looking at the lib functionality that is now part of clojure proper. Will (in-ns) be deprecated since (ns) appears to supersede it?
15:14	aperotte	ok, thanks
15:15	aperotte	ozzilee: I hadn't used cells before, but it sounds like something very similar
15:27	Chouser	joubert: no, in-ns is still recommended if you want to switch to a namespace without changing it at all.
15:28	joubert	chouser: ok, why the different usage of symbol vs. non-symbol?
15:28	joubert	(also, in-ns creates a namespace if it doesn't already exist)
15:29	joubert	(which seems to me to "change" a namespace :-)
15:29	Chouser	in-ns is from a kind of "lower layer" which also has things like use, require, import, etc -- all of which are functions instead of macros
15:30	Chouser	each of those will continue to exist, I think, but if you can use (ns) instead that's preferred.
15:30	Chouser	well, yeah, I guess in-ns does create the ns, but it doesn't muck about with the imports, referred symbols, etc.
15:32	joubert	ok, so when "declaring" a namespace, (ns) is now preferred over (in-ns), with the latter really more geared towards (switch-ns)
15:32	ozzilee	aperotte: There was some discussion about Cells on the mailing list (sorry, was afk): http://groups.google.com/group/clojure/browse_thread/thread/d79392e4c79f8cde#
15:32	Chouser	right, you've got it.
15:33	drewr	Is there a Java standard for JDBC -> XML serialization?
15:34	drewr	Like, taking a resultset and saving the text to a file?
15:35	joubert	cool; do you think it worthwhile to normalize the way namespace names are passed to these 2 forms? either always as symbol or as non-symbol?
15:35	aperotte	ozzilee: thanks, this might be exactly what I'm looking for
15:36	Chouser	joubert: probably not. I like that (ns) uses unquoted symbols -- uncluttered. But that means it has to be a macro, which can be clumsier to use programmatically if you're doing something really unusual.
15:37	Chouser	joubert: but (ns) is just a wrapper around the underlying functions, which must have their args quoted since they are functions.
15:43	drewr	I keep getting java.lang.IncompatibleClassChangeErrors, but Clojure doesn't tell me what classes were involved in the exception.
15:45	Chouser	Huh, when I try to put the wrong type of key into a sorted hash, I get: java.lang.ClassCastException: clojure.lang.Keyword cannot be cast to java.lang.Integer
15:45	drewr	Chouser: Sorry, different problem :-)
15:46	drewr	I knew how to fix this one; just frustrated at the error message.
15:46	Chouser	oh, IncompatibleClassChangeErrors is not ClassCastException. I misread that, sorry.
15:56	drewr	No complaints here. :-0
15:56	drewr	:-)
18:37	achim_p	hey. i began playing around with that jar-classpath-dependency-download-thing i talked about recently. here it is, for those interested: http://www.bitbucket.org/achimpassen/clojure-ties/
18:38	achim_p	it's very much unfinished and untested, but i ran it successfully at least once. this was when i decided it needed a logo ;) raises the bar for absurd lambda iconography considerably.
18:39	achim_p	any opinions on the approach in general? how do you deal with dependencies? i feel the existing java frameworks are way to bulky for what i want to do ...
18:40	Chouser	that's a fantastic logo
18:50	hsyn	hi there
18:54	Chouser	hsyn: hi
18:56	hsyn	i am very new clojure and also to lisp so i have allready problems with the getting start tutotial for java programmers as described in the clojure wiki.
18:56	hsyn	if i try to load the script with
18:56	hsyn	RT.loadResourceScript ( "C:/app/foo.clj" ) ;
18:56	hsyn	i get a resouce not found exception.
18:56	hsyn	i realized that this method only takes the name part of the given path. How can i load the script form a path which is not in the classpath?
19:03	Chouser	hsyn: are you sure you need to load clojure from within your own java app? It's a bit easier to run from the command line.
19:04	hsyn	i going to use the result in java.
19:04	hsyn	in future
19:05	achim_p	hsyn: you could extend your classpath with RT.addURL
19:06	achim_p	didn't try it though ...
19:07	hsyn	thx achim it work
19:07	Chouser	you could also use clojure.lang.Compiler.load(rdr, sourcePath, sourceName)
19:07	Chouser	oh, good, nm then. :-)
19:19	pjb3	So I take it Clojure's instance? method doesn't map to Java's instanceof operator?
19:19	pjb3	(instance? java.util.Map {}) => false
19:23	rhickey	Clojure maps are not java,util.Maps
19:44	pjb3	Clojure maps are not java.util.Maps? Hmm...that's a bummer when trying to work with Java APIs that expect maps
19:45	pjb3	I assume there's a good reason why clojure maps aren't java.util.Maps?
19:46	Chouser	Clojure maps are persistent (immutable)
19:46	Chouser	You can still work with java.util.Maps pretty easily if you're sure you want them.
19:48	pjb3	I want to pass a clojure map to a Java API that expects a java.util.Map
19:56	shoover	there was some discussion mentioning it as a possibility, but it seemed to get bogged down on incompatibilities between Collection and Map: http://clojure-log.n01se.net/date/2008-07-18.html
19:57	Chouser	(reduce (fn [m [k v]] (.put m k v) m) (java.util.HashMap.) {:a 1 :b 2})
20:00	pjb3	I'm trying to see if I can use clojure with velocity
20:00	pjb3	Chouser: that function helps
20:00	pjb3	but then the next problem is that what about maps that contain maps
20:01	pjb3	Since clojure maps don't even have a get method, they wouldn't work in a velocity context
20:04	Chouser	(defn jum [cm] (reduce (fn [m [k v]] (.put m k (if (map? v) (jum v) v)) m) (java.util.HashMap.) cm))
20:04	Chouser	(jum {:a 1 :b {:c 3 :d 4 :e {:f 5}}})
20:06	pjb3	Chouser: nice, I guess I'll try that for now
20:08	pjb3	But I agree with meredydd from that earlier discussion, I think it would be really nice to not have to do a deep copy everytime you want to have a java library use a clojure map
20:26	hsyn	is there any ATN parser available
23:13	Chouser	rhickey: I saw your "agents/refs in use" slides. Very nice.
23:14	Chouser	What kind of responses have you gotten to your talk?
23:14	rhickey	really great. I just got to the airport - an amazing conference
23:14	rhickey	so many interesting people
23:15	Chouser	:-)
23:15	rhickey	Clojure was very well received
23:16	blackdog	congrats rhickey
23:16	Chouser	great! anybody promise you tagged numbers? ;-)
23:16	rhickey	no, but they and tail calls were the most requested changes
23:17	rhickey	some hope in escape analysis for local optimization of boxed numbers in the JITs
23:17	rhickey	Cliff Click ran the ACO TSP code on 600 cores - scaled well
23:18	Chouser	that's pretty cool
23:22	rhickey	he had some amazing perf tools for his boxes - so much fun to see Clojure pop the bar meters on 600 cores
23:25	blackdog	i didn't realise the jvm scaled like that, never mind clojure
23:25	blackdog	v. impressive
23:26	rhickey	those boxes are amazig - the sim was allocating 18GB a second, GC wasn't sweating
23:26	rhickey	amazing
23:26	blackdog	crikey, all in makes one feel the jvm is a good choice :)
23:27	blackdog	does the .net vm scale like that?
23:27	rhickey	I don't know if there are boxes like that for .Net
23:28	rhickey	these things are like 200 cores in 5U
23:28	rhickey	1/2 TB RAM
23:29	blackdog	so this is really a milestone for clojure getting that coverage
23:29	avida	rhickey: are these the Azul boxes? my company trialled something like that
23:30	rhickey	lots of people there had only barely heard of Clojure, now they know what it's about
23:30	rhickey	avida: yes, Azul
23:31	blackdog	congrats again, you must be feeling over the moon after the hard work, and innovation!
23:31	rhickey	I'm totally spent - 3 days of intense conversations
23:33	blackdog	i mean really all the years on clojure and then taking it to a conference like that and getting the recognition etc
23:33	blackdog	tis very cool
23:33	rhickey	blackdog: yes, thanks
23:34	rhickey	now I have to gear up for Monday at MIT - very sophisticated Lisp group
23:34	blackdog	hehe
23:35	rhickey	plus maybe some Sun STM folks I hear
23:35	blackdog	do you know when infoq will publish the videos?
23:35	blackdog	of the conf
23:35	rhickey	blackdog: no idea
23:36	rhickey	I had only 1/2 hour - I talked as fast as I could
23:36	blackdog	did you come away with any inspiration, ideas?
23:37	rhickey	It will take a while to filter through, but overall I felt good about where Clojure is at. Some perf ideas, some more research topics to pursue
23:37	blackdog	well obviously, but any new implementation stuff :P
23:38	blackdog	nice
23:38	blackdog	probably more folks took away from what you've already achieved i bet
23:39	rhickey	there was a lot of talk about functional programming
23:39	blackdog	esp after your successful 600 core test :)
23:46	blackdog	well on a more modest scale (1 core), my brother is selling 3d training videos which has a simple clojure backend producing json, so it's my first production use of clojure
23:51	yangsx	java -cp clojure.jar clojure.lang.Compiler src/clj/clojure/boot.clj won't work, is that expected?
23:51	yangsx	I'm using svn version
23:53	yangsx	Or how can I compile a Clojure program into Java classes?
23:55	blackdog	yangsx, you can't do that yet, rhickey is investigating aot compilation to java though
23:57	yangsx	blackdog: OK, I got that from clojure.markdown in the svn repo.