#clojure – 2008-04-23

2008-04-23

02:23	patchwork	Hi guys. So I'm starting an application in clojure, and I'm wondering how deep to design the granularity of the relationships. In particular, it would be nice to have an agent for each sliver of the behavior, but I am wondering how reasonable having large numbers of agents is. Is there some guidelines as to about how many agents I should have at any given time?
12:41	jteo	why isn't there tail call optimization?
12:42	rhickey	because the jvm doesn't support it natively, and Clojure uses native JVM calling
12:43	jteo	ah.
14:42	rhickey	Chouser: any progress on the destructuring?
15:11	Chouser	rhickey: yeah, I have my most recent suggestion working.
15:11	Chouser	I just haven't had time to write it up.
15:11	rhickey	great
15:12	Chouser	It's not quite as succinct as the #{} format was, but it feels "cleaner" and is definitely more complete.
15:12	rhickey	I agree
15:14	Chouser	one issue I ran into was defaults when destructuring a set
15:14	rhickey	?
15:15	Chouser	'get' takes an optional 3rd arg, but using a map or set as a function does not.
15:15	Chouser	http://n01se.net/paste/0FZ -- Extra features for map destructuring
15:16	Chouser	so if you provide defaults (which don't make sense for sets anyway), my patch uses 'get', otherwise it uses the map or set as the function.
15:20	rhickey	ok
15:20	Chouser	here are some test cases that are working: http://n01se.net/paste/WNk
15:21	wabash	Does clojure code compile to native code via the Java HotSpot server thingy?
15:22	rhickey	yup
15:22	wabash	sweet.
15:22	wabash	rhickey: You mentioned a wiki for contributing. Were you referring to the wikibook, the wikipedia entry, or a wiki on the Clojure site?
15:23	rhickey	the wikibook
15:23	wabash	ok. thanks.
15:24	rhickey	http://en.wikibooks.org/wiki/Clojure_Programming
16:10	vincenz	rhickey: I have a questions about macros. How do you make sure that macro-code runs at 'compile'time?
16:15	rhickey	macros always run at compile time, but the code they produce runs at runtime
16:25	Chouser	rhickey: is there anything else you want from me for the destructuring patch?
16:31	rhickey	just looking at it - looks fine, although the original was written before variadic assoc/dissoc, this could be cleaner with them (e.g. apply merge ... apply hash-map...
16:32	Chouser	ok. I can look at that tonight if you want.
16:33	Chouser	or you can do it; I don't care.
16:34	rhickey	I appreciate what you've done, I can take it from here unless you want the exercise :)
16:34	Chouser	I've got other things I can do.
16:35	rhickey	ok :)
16:35	Chouser	On the other hand if you want to do something I can't, that'd be fine too.
16:35	rhickey	thanks
16:35	Chouser	hey, thanks for letting me help. :-)
16:35	rhickey	I'm working on Java-less Java integration
16:36	rhickey	No RT.init
16:36	rhickey	Defining named classes right into bytecode
16:36	rhickey	aceess to protected members and super
16:36	rhickey	classes can have agent or ref state
16:36	rhickey	runtime redefinition of methods
16:37	rhickey	auto-load of user.clj from classpath
16:37	Chouser	see, you shoudl do that and let me monkey around with refactoring apply hash-maps
16:38	Chouser	would all that help someone derive from PersistentHash in order to, for example, provide a default value that would be returned instead of nil?
16:38	rhickey	that's on the todo list anyway, but probably
16:39	rhickey	right now there are a lot of cases in which you have to write a little Java
16:39	rhickey	for main()
16:39	rhickey	servlets
16:39	rhickey	working in a framework like Netbeans or Eclipse
16:39	Chouser	Banishing that last 20% is probably hard.
16:40	rhickey	I think I've got it figured out, and half-implemented
16:40	rhickey	for instance RT.init() is no longer needed
16:40	rhickey	and Clojure loads user.clj from the classpath at boot time
16:41	rhickey	much of the proxy stuff will be reused
16:43	Chouser	"classes can have agent or ref state"! cool.
16:44	rhickey	Basically they'll have final state, so if you want mutation you'll have to do it the Clojure way
16:44	vincenz	rhickey: how do you make sure that compiletime code doesn't do "runtimey" things?
16:44	rhickey	vincenz: I don't
16:44	Chouser	vincenz: it can do runtimey things. That's what's so cool.
16:45	rhickey	but it does require care and self-discipline
16:45	rhickey	you can make a mess with macros
16:45	Chouser	I guess you don't want a blocking network call during macro expansion.
16:45	vincenz	rhickey: what about a macro requiring code from a file that has usage of macros/
16:45	rhickey	Chouser: :)
16:46	rhickey	vincenz: Clojure generally has a define-before-use policy, same applies to macros
16:46	rhickey	Chouser: anyway, re: destructure reduce/apply/merge/dissoc/apply/hash-map/mapcat can become reduce/apply/assoc/dissoc/map - remember you can assoc pairs onto a map
16:48	Chouser	ok
19:18	leafw	hi all
19:19	rhickey	welcome
19:19	vincenz	rhickey: what did you use as paper for your stm implementation?
19:19	rhickey	I read a _lot_ of papers
19:19	leafw	question: is there an equivalent class to jython's PythonInterpreter, to which one can handle code as String objects and evaluate them within the context of a specific interpreter instance?
19:19	leafw	if there is, it's not evident to me ...
19:20	rhickey	leafw: Clojure is compiled, so there isn't exactly a notion of interpreter instance
19:20	leafw	!
19:20	leafw	and the REPL.java class?
19:21	rhickey	But Compiler.eval() is a full compiler/evaluator that takes Clojure data structures
19:21	rhickey	you can get from Strings to Clojure data structures using LispReader.read()
19:21	leafw	thank you
19:21	rhickey	or build the data structures directly using the Clojure Java library
19:23	rhickey	vincenz: Clojure's STM is not exactly like any one I read about
19:23	leafw	thanks for the tips. I have somewhere to start now.
19:23	rhickey	It has no global lock
19:23	vincenz	rhickey: I think haskell is lockfree as wel
19:23	rhickey	but is not lock-free either
19:23	vincenz	oh
19:23	vincenz	distributed locking?
19:24	rhickey	I don't believe that lock-free is efficient, there are good papers that indicate it's not
19:24	vincenz	IIRC, lock-free only works well for optimistic lazy stm
19:24	rhickey	Some locking is better than spinning optimistic retries in my book
19:25	vincenz	do you do eager or lazy?
19:25	rhickey	Locks are eager, but there is barging
19:25	rhickey	also timeouts
19:26	rhickey	waking up from a lock before released
19:26	rhickey	based upon a time
19:26	vincenz	Trying to find out what an easy system would be
19:26	rhickey	STM is not easy
19:26	vincenz	well everything is an axis
19:26	vincenz	there's gradations
19:27	rhickey	The easiest is to number your resources, work optimistically, then grab the locks in order to commit
19:27	vincenz	so that would be lazy then
19:28	rhickey	right
19:28	rhickey	but there are so many variations and tradeoffs
19:28	vincenz	I wonder if you could work with a distributed clock
19:29	rhickey	?
19:29	vincenz	e.g. if you have a generational GC, use one clockc per generation
19:29	rhickey	I don't see the interaction with GC
19:29	vincenz	well you have more clocks
19:30	vincenz	so less conflicts on those clocks
19:30	rhickey	what's a clock?
19:30	vincenz	I thought lock-free worked with a clock
19:30	vincenz	you check the data you're writing to after a lazy eager system is still the same clock time for the values you are modifying
19:30	rhickey	lock-free uses any CAS-based mechanism
19:31	rhickey	but there are many different policies for timestamping the world
19:31	vincenz	in essence you're trading locks for clocks?
19:32	vincenz	rhickey: is there a reason you chose not to support orElse/
19:32	rhickey	no, they are 2 different things - you can use CAS to toggle flags that effectively 'lock' a resource for one transaction without using a blocking lock - you have to distinguish lock-free from non-
19:32	rhickey	blocking
19:33	rhickey	i.e. soft locks don;t
19:33	rhickey	block threads
19:33	rhickey	vincenz: no orElse because I don't think it's a good idea
19:34	vincenz	rhickey: why is that?
19:34	vincenz	(out of curiousity)
19:34	rhickey	well, it requires read-tracking for one
19:34	rhickey	I don't like mixing the coordination with the transaction
19:34	vincenz	btw, just a side question
19:35	vincenz	(let ((a @ref)) (sync (update ref ...))
19:35	vincenz	and
19:35	vincenz	(sync (let ((a @ref)) (update ref ...))
19:35	vincenz	are different, right?
19:35	vincenz	the second guarantees that it occurs in the same timeslice/
19:35	rhickey	yes, the first is a wild-read
19:37	vincenz	alright, thanks
19:39	ooxwo	Hi everybody. I'm starting on a clojure application and I am wondering how fine-grained to make the abstraction. In particular, for my domain it would be nice to make an agent for each atomic element, but I am wondering how feasible it is to use large numbers of agents. Is there a guideline as to about how many agents I should have running at any given time? Is there a way to unite several agents on a single thread? Thanks for your
19:39	ooxwo	help.
19:40	rhickey	what are large numbers?
19:40	ooxwo	like 10-100 thousand
19:41	rhickey	no problem. each agent doesn't get its own thread
19:41	ooxwo	or more possibly, depending
19:41	ooxwo	Really? Ah okay
19:41	rhickey	the actions run in a thread pool, idle agents use no cpu at all
19:41	ooxwo	That is good news, in my trial runs there seemed to be a one to one mapping between threads spawned and the number of agents I had created
19:43	ooxwo	Have you stress tested the limit of number of agents? Just curious if there is a general ceiling.
19:43	rhickey	you should be limited only by memory, but make sure you use send and not send-off or you will run into thread consumption problems
19:45	ooxwo	Ah okay thanks for the tip. I was using send-off so maybe that is why there was a thread for each agent.
21:41	Chouser	http://n01se.net/paste/IiF -- Tighter implementation of map destructuring features.
21:41	Chouser	rhickey: I don't think that's exactly what you had in mind, but I think it's good.
21:42	Chouser	Do what you want: take it, advise me, or do it your own way.
22:26	rhickey	Chouser: that looks great - thanks!
22:34	Chouser	np, thanks for letting me help.
23:29	Chouser	(get #{:a} :a :dflt) ==> :dflt
23:30	Chouser	hm. maybe that does make sense.
23:30	Chouser	oh, yeah, ok. I'm fine now.
23:31	landonf	Lisp's propensity for mind-boggling density still throws me for a loop.
23:31	landonf	Feels like there a character drought and we're all conserving. Save a tree, type fewer characters.
23:32	Chouser	heh