#clojure – 2008-03-27

2008-03-27

09:52	Chouser	rhickey: if you don't mind sharing, is that corporate or academic work?
09:53	rhickey	I work for myself
10:05	Chouser	you've found customers for your work, or you have sufficient capital you don't need customers yet?
10:06	Chouser	I don't mean to pry, I'm just very curious.
10:06	rhickey	that's something I can't discuss
10:06	Chouser	ok
10:07	Chouser	well, I'm sure glad you've taken the time to work on Clojure. :-)
10:07	cgrand	Maybe it's old news to you but I just noticed this morning that the Eclipse debugger is able to step into clojure code (highlighting current line etc.).
10:08	rhickey	me too - I finally get to develop software the way that I want to, and have huge libraries at hand
10:08	rhickey	cgrand: can you set breakpoints? last time I tried I couldn't
10:09	rhickey	Clojure emits the right stuff for debugging, local variable names etc
10:10	Chouser	rhickey: that's how the stack traces from exceptions work, right?
10:11	rhickey	that's how the line numbers get in there, but there's more to it, you need a SourceDebugExtension to map to the source files, which Clojure also emits, and locals entries in the bytecode, also there
10:12	cgrand	Yeah I saw that you emit smap strings. With a rudimentary plugin (never played with eclipse dbugger api) I can set breakpoints
10:12	rhickey	cool
10:13	cgrand	which source files? Clojure emits java source files?
10:14	rhickey	no the smap is a mapping from Clojure to non-existent Java files, which is the bridge
10:14	rhickey	Full Clojure plugin for Netbeans is coming, if you haven't seen this: http://groups.google.com/group/clojure/msg/4dfb4566c2187704
10:14	cgrand	I saw
10:15	rhickey	they had to do some work to get Netbeans to debug Clojure
10:17	rhickey	I'd love to see Eclipse support too
10:19	cgrand	The editor framework is a bit complex...
10:19	rhickey	I guess Cusp would provide good editing support, but native debugging is preferable to custom protocol
10:20	rhickey	does Cusp do debugging anyway?
10:22	cgrand	Not sur but I think so
12:01	sanity	anyone here familiar with SISC? http://sisc-scheme.org/
12:02	Chouser	someone recommended it to me when I was looking for a JVM lisp. Of course I ended up using Clojure instead, so I know nothing useful.
12:04	sanity	Chouser: i only ask because a friend of mine created it
12:05	sanity	Chouser: although i think, unlikely Clojure, it is interpreted, rather than compiled
12:05	sanity	s/unlikely/unlike
12:05	Chouser	interesting.
14:27	wabash	If I have a multicore cpu, like 8 cores, how would I parallelize a clojure app? Would I start a separate JVM on each core? And, would threads on separate JVMs be able to communicate via some sore of message passing?
14:27	wabash	like Erlang?
14:28	rhickey	A single JVM can address multiple cores
14:28	rhickey	Clojure agents will get distributed among them
14:28	wabash	rhickey: wow, sweet!
14:29	wabash	I'm sorry to ask such basic questions. I just found out about clojure like, last week. The more I find out, the more interested I am.
14:29	rhickey	Clojure fns are Callable and Runnable, and will work with the java.util.concurrent.Executors framework as well
14:30	wabash	What is the clojure way of thread communication? And, are threads expensive like Java or are they cheap like Erlang? is msg passing expensive like sockets or cheap like shared mem?
14:30	wabash	rhickey: Are all Java libs and classes basically available?
14:30	rhickey	Erlang processes are not threads
14:31	rhickey	they may or may not map to threads depending on how you have Erlang SMP set up
14:31	wabash	rhickey: Sorry, I got semantically lazy.
14:31	rhickey	Erlang w/o SMP mode runs on one core
14:31	rhickey	but does its own scheduling
14:32	wabash	Right, I just misused a term.
14:32	rhickey	Clojure agents use a fixed thread pool on message sends
14:32	rhickey	the pool size is related to the number of cores
14:32	wabash	linearly related?
14:32	rhickey	Agent state can be read directly w/o messages
14:33	rhickey	agent state gets changes via actions, functions of their state that get executed asynchronously
14:33	rhickey	it's a different model than Erlang's
14:33	wabash	How do you read stat directly? Is it some sort of polling? is an agent like a process?
14:33	wabash	Oh, I see.
14:33	wabash	It's a very different way of thinking about hings.
14:33	wabash	is the state reading quick between cores?
14:34	rhickey	Erlang's can be transparently distributed, but getting at state requires send/ack & blocking
14:34	wabash	Can clojure be transparently distributed?
14:34	rhickey	Clojure is a single memory space, reading agent state is like any other object access
14:35	rhickey	No, I don't believe in transparent distribution, but may add distributed actors at some point
14:35	wabash	Is the single mem space a problem for concurrency then? How is that abstracted? Erlang seems so elegant......
14:36	rhickey	Clojure agent states are immutable
14:36	wabash	I get it now.
14:36	wabash	Thank you.
14:36	rhickey	immutable objects are no problem for multicore
14:37	wabash	rhickey: I appreciate your time. Learning something new, I'm always faced with the paradox of what to ask when I don't know enough to formulate questions.
14:37	rhickey	I don't think send/ack message pairs to read state is elegant at all, just a necessary evil of distribution
14:37	rhickey	vs RPC
14:38	wabash	RPC is more elegant than message pairs?
14:38	rhickey	no, message pairs is less evil
14:38	rhickey	distribution is simply very hard
14:39	rhickey	why incur the complexity unless you really are distributed?
14:39	wabash	premature scalability, I suppose.
14:40	rhickey	but the constant factors are different - you can make it look the same, but the same payloads and chatter rate that worked locally can totally fall down when distributed
14:40	wabash	Yes, I understand and agree.
14:41	wabash	That's kind of what I was getting at with spreading over cores; will the constants be high for communicating between (threads/processes) on different cores.
14:41	rhickey	http://research.sun.com/techrep/1994/smli_tr-94-29.pdf
14:41	wabash	In Clojure, then, you think in terms of "agents"?
14:41	wabash	thank you for the link. I will read it after lunch.
14:42	rhickey	agents are one of several tools for MT programming. There are also refs and STM
14:43	wabash	I see. If I may ask an over-general question, if I run Clojure on an 8 core machine, will the communications overhead be small enough that the distribution will be even? Will I have to worry about which core is closer to which?
14:43	wabash	Or if I have a small cluster of separate machines, will I have to adjust my algorithms to match the architechture?
14:44	rhickey	that totally depends on the granularity of your work units. Overhead/work is what matters
14:44	wabash	I see.
14:44	rhickey	separate machines is a totally different architecture in my book, and where Erlang's model makes the most sense
14:44	wabash	Right.
14:45	rhickey	I'm thining about doing distribution for Clojure using Shoal: https://shoal.dev.java.net/
14:45	wabash	You seem to have a lot of experience with different languages/models. Is your work research oriented? Or are you lucky enough to have a corporate job doing all these things?
14:46	rhickey	I'm just old
14:46	rhickey	:)
14:46	rhickey	But lots of MT in my experience
14:46	wabash	MT?
14:47	rhickey	multithreading
14:47	wabash	Got it.
14:47	wabash	Thanks for the discussion. It is useful, significant to me.
14:47	rhickey	sure
14:48	wabash	I went to an average school, got wrapped up in some programming jobs, got ejected, dejected, discouraged, but now have discovered higher level languages, and FP, and think that programming is fun again, a bit.
14:48	wabash	well, must have lunch. thanks for sharing your knowledge and experience.
16:14	jgracin	hi all! this place is getting more crowded with each day. :-)
16:14	rhickey	:)
17:24	Chouser	I'm not sure this is the right question, but if I wanted to make something that acts exactly like a hash-map, but also generated some Actor dispatches, how would I do that? "implements"? multi-methods?
17:26	Chouser	ok, that's not quite right. I want something that acts like ref. You can call set-ref on it, and it does that but also it dispatches and Actor function.
17:26	Chouser	If this were some other language, I'd sub-class Ref.
17:28	rhickey	why not just use a map as the state of an agent?
17:28	rhickey	or of a ref?
17:29	rhickey	agent actions anf ref alter/commute functions can be multimethods
17:35	Chouser	I'm right at the edge of my understand here, so bear with me... I've got a function that calls (ref-set db (assoc ...)). It currently expects db be to a ref to a map, but I'd like to also allow db to be one of these new things I'm making. Are you suggesting that I change my set-ref to be (alter db foo) and then make foo a multi-method that can handle either standard ref-maps, or my new thnigs?
17:36	rhickey	right, in your first case, (ref-set db (assoc... you could have used (alter db assoc...
17:36	rhickey	now swap foo in for assoc...
17:36	rhickey	make foo a multimethod that can distinguish the things you might out in db
17:37	rhickey	put
17:37	rhickey	are your new things going in refs?
17:37	Chouser	ok, I think I follow.
17:38	Chouser	I don't think so. I think they will act like refs.
17:38	rhickey	hmm...
17:38	Chouser	Maybe I didn't understand the question.
17:38	rhickey	why can't they be refs?
17:39	Chouser	I'm thinking I'll make a thing that acts like a ref, but whenever it gets set it will ask an actor to write its new value out to disk.
17:40	rhickey	I see, I've thought about connecting on-commit and validate fns to refs as a general solution for that kind of thing
17:41	rhickey	but refs are special, don't try to derive...
17:41	rhickey	you can have your mutating fns send an agent action, but not automatically... could use a macro to define the mutating fns
17:43	Chouser	but the solution you outlined above should still work, right? I'll use (alter db db-assoc ...), and db-assoc will either just assoc, or assoc and dispatch.
17:44	Chouser	This will be my first agent.
17:44	Chouser	And my first multi-method. ;-)
17:44	rhickey	oh boy
17:45	rhickey	yes, if db-assoc sends an action, that will only happen if the transaction commits
17:47	Chouser	whew, good point. I hadn't thought about the assoc getting rolled back. Ok, I'll try to do this.
17:48	rhickey	that's the beauty of the agent/transaction coordination, there's no other way to get run-once-only side-effects in a transaction
22:14	Chouser	Ok, I don't think I really need a multimethod.
22:14	Chouser	Is there any way to refer to an agent action, or do I have to wrap the call to it in a function.
22:15	rhickey	?
22:15	rhickey	agent action are functions, or are you talking about the agent's state?
22:17	Chouser	hm, no I was misunderstanding how they work. Let me read that docs page again...
22:18	rhickey	you can make one-off agents just to execute side-effects:
22:19	rhickey	(send (agent nil) (fn [_] (side-effect)))
22:20	Chouser	send is the same as ! ?
22:20	Chouser	huh. ok, I think that's exactly what I want.
22:21	rhickey	yes, send is the new !, there is also send-off for potentially blocking actions
22:21	Chouser	oh, is ! going away?
22:21	rhickey	yes ! is going away
22:22	rhickey	try it
22:22	rhickey	it's gone
22:22	Chouser	heh, ok.
22:45	Chouser	should I avoid multiple deref in a transaction?
22:46	Chouser	I've got @foo all over the place, but I could put it in a local symbol instead.
23:16	rhickey	multiple deref of refs is no problem, should be same result throughout unless transaction itself changes ref