#clojure – 2008-06-25

2008-06-25

00:34	jgrant	anyone had any experience with jvm on FreeBSD multi-core systems ?
00:35	jgrant	i have a multi-threaded app written in clojure and i'm not seeing it(probably the jvm) distribute threads across the 4 cores ?
00:36	jgrant	Using jdk-1.6 on FreeBSD 7.0 SMP (4 cores)
00:36	jgrant	surely the jvm uses native threads ?
00:37	jgrant	the java process itself will shift between the 4 cores but top reports only 25% cpu usage (100% for the cpu that the process is currently running on)
01:33	albino	jgrant: are you using the jvm that comes with freebsd?
01:57	jgrant	albino : i'm using the port jdk16
02:00	jgrant	ok i just figured out that it is not the jvm on freebsd7
02:00	jgrant	the following simple code maxes out all 4 cores :
02:00	jgrant	public class Test implements Runnable
02:00	jgrant	{
02:00	jgrant	private volatile int m_counter;
02:00	jgrant	public void run()
02:00	jgrant	{
02:00	jgrant	for (;;)
02:00	jgrant	for (int i = 0; i < 1000000; i++)
02:00	jgrant	m_counter++;
02:00	jgrant	}
02:00	jgrant	public static void main(String[] args)
02:00	jgrant	{
02:00	jgrant	Test test1 = new Test();
02:01	jgrant	Test test2 = new Test();
02:01	jgrant	Test test3 = new Test();
02:01	jgrant	Test test4 = new Test();
02:01	jgrant	Thread thread1 = new Thread(test1);
02:01	jgrant	Thread thread2 = new Thread(test2);
02:01	jgrant	Thread thread3 = new Thread(test3);
02:01	jgrant	Thread thread4 = new Thread(test4);
02:01	jgrant	thread1.start(); thread2.start(); thread3.start(); thread4.start();
02:01	jgrant	try {
02:01	jgrant	thread1.join(); thread2.join(); thread3.join(); thread4.join();
02:01	jgrant	} catch (InterruptedException iex) {
02:01	jgrant	System.out.println(iex);
02:01	jgrant	}
02:01	jgrant	if (test1.m_counter == test2.m_counter)
02:01	jgrant	System.out.println("Don't care, this is just to fool the optimiser");
02:01	jgrant	}
02:01	jgrant	}
02:05	albino	jgrant: for code of that size please use a pastebin
02:07	jgrant	sure
02:07	cgrand	jgrant: what is the problem? Maxing out 4 cores is expected: you put each core in a busy loop.
02:09	jgrant	no no, that is an example of what i want, i.e. to max out cores
02:10	jgrant	but in my other code ( a socket listener ) it sticks to one core only even though it's using a newFixedThreadPool
02:10	jgrant	the other code is in clojure too
02:11	jgrant	i've heard the synchronized blocks can cause that sometimes
02:13	jgrant	rich's example of parrallel map apparently works --> http://clojure.org/refs
02:13	cgrand	can you paste some code?
02:14	cgrand	http://paste.lisp.org/new/clojure
02:14	lisppaste8	jgrant pasted "jgrant" at http://paste.lisp.org/display/62808
02:15	jgrant	it makes use of a single ServeSocket and pool is just pool (. Executors (newFixedThreadPool nthreads))
02:15	jgrant	where nthreads is 64
02:19	jgrant	handle-req function is trivial it just reads the input from sin and writes some simple text out
02:28	cgrand	how do you stress this server?
02:29	jgrant	ab -v1 -n 10000000 -c 400 http://127.0.0.1:8000/
02:29	jgrant	using apache bench
02:29	jgrant	it's picky, only accepts ips for url
02:32	cgrand	you said that this code "sticks to one core only" but does it max out during ab runs?
02:32	jgrant	it maxes out only one core
02:32	jgrant	when running ab to stress it
02:32	cgrand	ok
02:34	cgrand	and if you try to put some more "stuff to do" in your handle-req (eg a small busy loop) does it still stick to one core?
02:38	jgrant	(. Thread (sleep 100)) is what i tried at the end of handle-req and that makes no difference
02:39	jgrant	another detail : each thread updates a global ref defined as (def serve-count (ref 0))
02:39	jgrant	by calling (dosync
02:39	jgrant	(commute serve-count inc))
02:39	jgrant	could that be a problem ?
02:41	cgrand	(. Thread (sleep 100)) doesn't keep the cpu busy, so the scheduler can hand it another thread -- it just prevents the executor from sending a new job.
02:41	jgrant	do you have an example of 'busy loop' ?
02:42	cgrand	I haven't had any pb with (dosync (commute something inc)) so far (on windows, linux and solaris)
02:42	jgrant	me either on OS X / FreeBSD
02:42	jgrant	i may have seen some on linux
02:44	cgrand	busy loop (reduce + (range 1000000))
02:46	jgrant	handle-req is already reading the input and generating a response then sending it back
02:47	jgrant	surely that is 'busy' enough for each thread ?
02:50	cgrand	I can't tell. Does the pb go away when you remove the code updating the global ref?
02:53	jgrant	no
03:37	cgrand	jgrant: I tried to write a server that reads 3 lines and send them back mirrored (using yiour main loop), I stressed it with your ab command line, CPU usage stayed around 100%. I then added (reduce + (range 1000000)) at the start of handle-req and CPU usage maxed out (>190%, dual core -- I even saw 202% at some point :-))
03:38	jgrant	ah
03:38	jgrant	so i guess just sending a response is not enough stress for the jvm to spread the threads across cpus then ?
03:39	jgrant	how many threads did you use in ab and how many did you use for the thread pool ?
03:40	cgrand	for ab I used your command line as is and 64 threads in the pool
03:49	jgrant	cgrand : thanks i verified the same
03:49	jgrant	cgrand : the socket handling is not enough to require more than one cpu it seems
03:50	cgrand	or it requires too much syscalls
03:51	jgrant	no don't think that's it because leaving the socket handling in but adding the reduce call makes use of multiple cpus
03:53	jgrant	turning up ab concurrent clients may get multiple cpu usage
03:53	jgrant	but the socket handling may be the bottle neck at that point
08:47	wwmorgan	(= (disj (sorted-set :a) :a) (sorted-set)) evaluates to false but it should be true
09:28	wwmorgan	rhickey: that's a bug, right?
09:35	cgrand	wwwmorgan, in the meantime you can fix it in your copy by replacing ([] {}) with ([] (. clojure.lang.PersistentTreeSet EMPTY)) in the def of reduce-set in boot.clj, it should do the trick
09:43	wwmorgan	cgrand: thanks
09:44	cgrand	wwmorgan: it seems that you can even delete this line entirely
11:30	cemerick	looks like genclass.clj isn't loaded by default -- am I crazy, or wasn't that being loaded automatically before?
12:44	Lau_of_DK	Evening Gents
12:45	blackdog	cemerick, you can load the base cljs like this (import '(clojure.lang RT))
12:45	blackdog	(. RT loadResourceScript "inspector.clj")
12:57	cgrand	Lau_of_DK: evening
13:11	Lau_of_DK	I remember somebody mentioning porting Clojure to ikvm - I just want to know, does this mean that Clojure can run on .Net without making any use of Java whatsoever?
13:13	blackdog	Lau_of_DK, it was cemerick who was talking about that
13:13	Lau_of_DK	cemerick: awake and educate :)
13:13	blackdog	although i think he was La_mer at the time
13:14	cemerick	blackdog: regarding genclass.clj, etc., I know about loadResourceScript and such -- I was just hoping to avoid having to bring up the "standard libraries" myself ahead of generating classfiles
13:14	blackdog	k
13:15	cemerick	And yes, I was la_mer until recently -- changed to cemerick so that people in the real world would know who I was and vice versa
13:15	Lau_of_DK	cemerick: Did I understand you correctly? You got Clojure running on pure .Net ? No Java...
13:15	cemerick	Lau_of_DK: ikvm/ikvmc allows you to use just about any java library in .NET without the JVM -- the java standard libraries are required, though (and are provided as DLL's by ikvm)
13:16	Lau_of_DK	Oh ok. That provides a gigantic performance boost doesnt iT?
13:18	cemerick	Lau_of_DK: Well, keep in mind that .NET is a runtime just like the jvm -- it's not like Clojure (or any other java lib) is being distilled to a native library.
13:19	cemerick	So, performance is good as a ikvm-generated DLL -- good enough that I haven't benchmarked things in that area for a long time.
13:20	blackdog	cemerick, so you use ikvm for production stuff?
13:21	cemerick	blackdog: Yes: http://snowtide.com/PDFTextStream.NET
13:21	blackdog	have you used it for long running servers?
13:21	cemerick	Yes, although our customers beat it up more than we do....with good results. :-)
13:21	blackdog	ah ok,
13:21	Lau_of_DK	pretty awesome
13:22	cemerick	There was a hiccup recently related to the correspondence between classloaders and AppDomains, but that was fixed in ikvm v0.34, IIRC
13:22	blackdog	it's amazing, and ikvm done by one guy too
13:22	Lau_of_DK	There are a few guys gifted with unique intellects obviously
13:23	cemerick	We're pretty far behind in the ikvm releases (for PDFTextStream, anyway), simply because we have customers on .NET 1.1 still.
13:23	blackdog	i've seen snowtide befoer somewhere
13:23	Lau_of_DK	.Net is backwards compatible - why not upgrade?
13:25	blackdog	ah
13:25	blackdog	that's where, i ported your difflib libaray to haxe
13:26	blackdog	ah and you are chas emerick the author of the difflib
13:26	cemerick	Lau_of_DK: Yes and no -- ikvm v0.36 doesn't support .NET 1.1, only 2.0+
13:26	cemerick	blackdog: Indeed. Nice to meet you. I'm glad to see that jsdifflib is getting around. :-)
13:27	cemerick	I'm afraid it's pretty neglected at this point.
13:27	blackdog	:)
13:27	blackdog	as is my port
13:27	cemerick	Thanks for mentioning haxe -- I hadn't seen it before.
13:27	blackdog	haxe was last years language du jour, now it's clo jure
13:28	blackdog	it does rock
13:28	blackdog	but the server side story is weak, hence i'm looking at jvm/clojure
13:28	cemerick	So, given that genclass and friends aren't loaded by default, is there a "recommended" way to bootstrap clojure maximally? I'd like to not reinvent the wheel.
13:29	cemerick	(if a such a wheel exists, I suppose)
13:43	Lau_of_DK	cemerick: if the wheel does exist, it just stopped turning
13:43	rhickey	user.clj is auto-loaded
14:08	cemerick	rhickey: That's interesting; user.clj in pwd is autoloaded -- is there a standard search path for finding user.clj (pwd, ~/.user.clj, etc)?
14:10	rhickey	it uses the classpath
14:24	cemerick	A good and bad thing ;-)
15:28	cemerick	rhickey: you mentioned (.get clojure.lang.Compiler/SOURCE_PATH) yesterday -- that works great when you're using load-file, but it's a relative path (relative to the classpath entry's root, I suppose) when the file being loaded is user.clp
15:30	cemerick	I know it's an unofficial API point, but I thought I'd mention it anyway :-)
15:30	Lau_of_DK	Has anybody started any bigger projects in Clojure yet ?
15:30	Lau_of_DK	(bigger = bigger than a port of msPaint)
15:31	cemerick	Lau_of_DK: I'm guessing that no one's going to spill the beans until there's something impressive to look at. ;-)
15:31	arbscht	I have difficulty making my projects look big when written in lisp :)
15:40	Lau_of_DK	cemerick: sure - It'd just be nice with some inspiration - Im not sure Im fully grasping the possibilities yet
15:40	Lau_of_DK	Like Rich talked about Election Projection systems... I have no clue :)
16:13	abw_	hello guys, could someone tell me how to go through the sexp history in the clojure repl as rich hickey did in the screencast?
16:13	drewr	abw_: Got Emacs?
16:13	abw_	yes
16:14	abw_	and all the clojure code stuffs
16:15	drewr	Then M-p, my friend.
16:15	drewr	Just like anything else.
16:16	abw_	drewr: great, thanks!
16:17	cemerick	abw_: FWIW, it's Ctrl-up in Enclojure :-)
16:17	cemerick	(just in case you want to join the herd someday ;-) )
16:24	abw_	cemerick:cool, I've noted that for when enclojure gets more mature
16:59	abw_	I'm thinking of writing a ClojureTutorial app once I know the Clojure language more completely and with more experience. It will draw ideas from Lisp Critic and the socratic method and work like a more computationally intelligent text based adventure game because of the Lisp Critic. Right now I am taking notes on what it's like to know less Clojure, how I learn it and what the knowledge dependancies are.
17:31	cemerick	rhickey: I'm having trouble defining a method for a class in gen-and-save-class that has as a return type the generated class. I'd like to do this, but gen-class complains that the 'pkg.Foo symbol doesn't have a getType method: (gen-and-save-class "." 'pkg.Foo :implements [Bar] :methods [['union [java.util.Collection] 'pkg.Foo]])
18:13	pjb3	Hey clojurers, I just wrote up a function to execute a sql query and return the first column of the first row as a string
18:13	pjb3	http://pastie.org/222226
18:13	pjb3	So if you execute (db/select-first "SELECT login FROM users")
18:13	pjb3	You get "pbarry"
18:14	pjb3	Any comments on the code? I'm wondering if there are better, more idiomatic ways to do that
18:18	pjb3	specifically I'm wondering how to put a try/catch statement into do-with-connection, which will always make sure the connection gets closed
18:31	lisppaste8	blackdog pasted "sql" at http://paste.lisp.org/display/62844
18:31	blackdog	pjb3, i'm using that
18:33	blackdog	i can't tell you if it's better or not as i'm new to clojure too, but it's different !
18:34	pjb3	What happens when getConnection throws an exception?
18:35	blackdog	oh i didn't hand;e that :)
18:36	pjb3	Yeah, this is the Java code: http://pastie.org/222245
18:37	pjb3	You set con to null, then do the getConnection in the try, and then check to see if con is null in the finally
18:37	pjb3	but I can't figure out how you would do that in clojure
18:37	blackdog	oh
18:37	pjb3	since you can't let [con nil] and then change it later
18:37	blackdog	yea i see
18:38	pjb3	rhickey: help?
18:38	blackdog	well i suppose it's the same as with-open in boot.clj
18:38	blackdog	somehow it doesn't matter :)
18:39	blackdog	i don't know
18:39	blackdog	yes, now i realise i glossed over the matter
18:42	pjb3	To use with-open, wouldn't it be (with-open con (.getConnection DriverManager "jdbc:whatever") (stuff))
18:42	pjb3	which gives you the same problem, because the call to .getConnection happens outside the try
18:51	lisppaste8	pjb3 pasted "with-connection" at http://paste.lisp.org/display/62845
19:14	jgrant	Is it really not possible to modify a hash map created with (hash-map ...) ?
19:14	jgrant	(get mymap :key) works
19:14	jgrant	(put mymap : val) fails ?
19:15	jgrant	that is (put mymap :key val) fails ?
19:24	jgrant	anyone ?
19:24	jgrant	or should i just use Java's HashMap class instead ?
19:30	jgrant	ok assoc works (i guess i should read the API more carefully)
21:16	abw_	the clojure-mode does not detect soft word wrapping of Clojure comments in Aquamacs highlighting the wrapped comments as code. Anyone else notice this bug?
21:22	pjb3	If I try to add metadata to a java object, specifically a java.sql.ResultSet, I get a java.lang.IncompatibleClassChangeError
21:22	pjb3	Is that to be expected?
22:29	rhickey	pjb3: metadata is only supported on certain Clojure data structures - symbols and collections
22:29	pjb3	rhickey: Ok
22:31	cemerick	rhickey: Good evening. I don't suppose you noticed my note about regarding defining methods that take or return the type that is being generated....
22:32	rhickey	yes, when I saw it you weren't on - right now there is no way to do that
22:34	rhickey	I'll look into it
22:38	cemerick	rhickey: That's what I thought. I've been idly working on understanding what's going on in gen-class, and I think I can see what needs to be done, though dimly. :-)