#clojure – 2017-01-08

2017-01-08

06:34	lxsameer	is it ok to have a atom containing a hashmap which each value is an atom itself ?
06:35	dysfun	sure it's okay. but what are you trying to achieve?
06:37	lxsameer	dysfun: I'm polling some data from a remote service, I want to keep track of data changes, the data is a huge map, and i want to update the changed values only
06:38	dysfun	i don't understand the problem with using one atom
06:39	dysfun	is it a concurrency problem?
06:39	lxsameer	dysfun: i wanted to create an atom for each key and set a validation and a watcher so validation only allows new values to be set, and watchers trigger the behavior which i needed
06:40	lxsameer	dysfun: using one big hashmap in an atom, i can't find out what key changed
06:40	lxsameer	dysfun: how would you design such thing, I need it to be scalable
06:40	dysfun	sorry, what sort of data will these be?
06:41	dysfun	are you doing HFT or something?
06:41	lxsameer	dysfun: the data is a huge json containing stock market information
06:42	dysfun	okay. so then you process that and update your atom of atoms according to the new values?
06:42	lxsameer	dysfun: i had this in mind, but I'm open to new ideas
06:43	lxsameer	specially my atom of atoms seems not scalable
06:43	dysfun	okay. so each piece of data is like {:symbol 'FOO" :price 1.23 ...} ?
06:43	dysfun	that sort of thing?
06:44	lxsameer	dysfun: yeah right
06:45	lxsameer	dysfun: i was thinking about creating a queue using kafka or something
06:45	dysfun	now you have two problems ;)
06:45	lxsameer	and put the values in the queue and have them process by nodes using core.async
06:45	lxsameer	dysfun: why ?
06:45	dysfun	well, message queuing theory says exactly once delivery is hard
06:46	dysfun	so now we've got to check timestamps and such
06:46	lxsameer	dysfun: aha
06:46	lxsameer	dysfun: so what's you solution about this ?
06:47	dysfun	pick a database designed for storing exactly this sort of data. like riak
06:47	dysfun	you can blissfully configure it in last-write-wins mode because that's exactly what you want
06:47	dysfun	or, if you don't actually need storage, shove it in a megaref
06:48	lxsameer	dysfun: megaref ? is it a clojure thing ?
06:48	dysfun	it's a library
06:48	dysfun	https://github.com/cgrand/megaref
06:48	lxsameer	dysfun: thanks man, i have to read about it, also the riak solution is cool too
06:49	dysfun	cool, have fun
07:03	osfameron	lxsameer: dysfun: kafka is pretty good as a queue
07:04	lxsameer	osfameron: i see
07:04	dysfun	sure, it is
07:04	dysfun	but you have to know how it fails too
07:05	osfameron	ah - we haven't come across particular failure modes - do tell more
07:05	dysfun	at-least-once != exactly-once will be the first one you'll hit
07:06	osfameron	ah yes. we're making sure all of our consumers are idempotent, which must be as mitigation of that
07:06	osfameron	I've not followed all the details
07:06	dysfun	someone gets it
07:07	osfameron	but yeah, that's easy enouhg for us to do with our particular use of it as a message bus, but might not work for other situations
07:08	dysfun	yeah. each situation needs it factoring in differently
08:58	hiro`	Anyone here read Web Development in Clojure? I've seen from the beta-version (which is floating around online) and the author's blog that he was using Yesql, but Luminus no longer uses it. Did the book get updated to use the Luminus stack (inc HugSQL, Yesql's replacement), does anyone know
08:58	hiro`	?
08:58	dysfun	that's a fairly niche question, suspect you'd be better asking in the work week
08:58	dysfun	maybe you could check the book's errata online?
08:59	hiro`	Had a quick scan through that already, couldn't find any reference to it.
09:01	dysfun	i guess probably not then
09:01	dysfun	shouldn't be too difficult to get up to date with hugsql though
09:01	avicenna	web development with clojure version 2 uses hugsql as seen in the pragprog website
09:01	dysfun	ah, there we go
09:01	hiro`	ah, that's what I was wondering.
09:01	hiro`	thanks avicenna
10:36	machinewar	any convention for naming clojure tests with deftest. deftest fn-name-test? perhaps
10:36	dysfun	that's common
10:37	dysfun	test-fn-name also
10:37	dysfun	just pick a sane one and roll with it
12:04	moncrey	hey room, looking for a little insight into a missing namespace in a project.... the code runs fine but i cant for the life of me figure out where this library is coming from....
12:04	moncrey	line 9 of this file https://github.com/overtone/overtone/blob/master/src/overtone/sc/machinery/server/native.clj
12:05	moncrey	points to a library that does not exist within that repository, or anywhere else within the organization's repositories.
12:06	moncrey	not assuming anyone will have the answer but ANY insight would help. so confused.
12:06	moncrey	if the library did not exist, the project would not run, correct?
12:07	moncrey	or at least, if it hit that required method, woudnt it throw errors??
12:36	justin_smith	moncrey: require looks for a specific artifact inside a package, there'
12:36	justin_smith	s no rule saying that the name of the artifact require finds has any relationship to the artifact name
12:37	scriptor	I think they left
12:38	justin_smith	hmm
12:38	rhg135	So it can load runtime ns given it exists before hand?
12:41	justin_smith	well the ns won't compile unless it can be found - but the point I was making was that there is no relationship required between the name of the ns that require looks for, and the name of any artifact you use
12:44	rhg135	I see, so a file foo/bar.clj can have ns bar.baz and you require it as foo.bar?
12:44	justin_smith	rhg135: more like a dependency org.foo/bar can have a file src/baz/quux.clj and you can require it via baz.quux
12:45	justin_smith	but the code referencing baz.quux doesn't help you find foo/bar
12:45	justin_smith	it does appear something is funky here though? https://github.com/search?q=org%3Aovertone+nativescsynth&type=Code
12:46	scriptor	my guess is that it's from this dependency https://github.com/overtone/overtone/blob/master/project.clj#L62
12:46	scriptor	can't find a github of that anywhere
12:47	justin_smith	based on the name, I would assume it is the piece of code that tells you which engine is actually present
12:47	justin_smith	since overtone uses a big c++ program for all the synthesis, and it needs a different version for every OS configuration
12:49	justin_smith	it would actually make sense for that data to be provided by the dependency itself (which wouldn't be part of the overtone github, it's a compiled binary)
12:53	justin_smith	FOUND IT [overtone/scsynth "3.5.7.0"] https://clojars.org/overtone/scsynth/versions/3.5.7.0
12:53	justin_smith	it's the only clj file in a big blob of native code blobs
12:56	justin_smith	that ns in one line: (ns overtone.nativescsynth.availability) (def native-scsynth-lib-availability {:windows {64 false 32 true} :linux {64 true 32 false} :mac {64 true 32 true}})
13:28	machinewar	going to post this function I'm working on in case anyone feels like helping refactor and make it more idiomatic clojure. https://gist.github.com/AlexWheeler/7d6a97978e51a036367de42936da8fae#file-clj-clj-L7. comments appreciated
13:31	osfameron	machinewar: body of copy-nil-vals could just be: {k (or v k)}
13:32	osfameron	hmm, that's slightly different semantics, would also overwrite false
13:32	osfameron	{k (if (nil? v) k v)} I suppose
13:33	osfameron	why do you need to (apply merge) twice?
13:34	ridcully	machinewar: could you please also through in some small data example? e.g. going from [{:value 42 :mapped_value 666}] -> {42 666} ?
13:34	machinewar	yep
13:36	ridcully	my gutfeeling is, that you take a lot of "reshaping" steps in between, where something like (into {} (map (juxt :value :mappeD_value)) columns) brings you already close
13:37	ridcully	throw in your nil handling around that juxt
13:37	machinewar	yep that's what I'm thinking. this was my intuitive way of doing it step by step, but figure there's a better way. Cool I'll play a bit with that thanks a lot
13:38	osfameron	oh, the juxt/into thing is nice ridcully!
13:39	osfameron	I'd have approached it with a dumb (fn [{keys [:value :mapped_value]} ...)
13:39	ridcully	well you have to deal with that nil check there still. so destructuring instead of the juxt is fine
17:18	osfameron	oh dear, I got lost in http://prog21.dadgum.com/ again
19:48	Vincentdm	Hi, I'm new and trying to understand multimethods. I would like to define (defmulti) in one file (namespace), and then use separate files for implementations (defmethod). However, this seems to require a circular dependency.
19:49	Vincentdm	I thought the whole point of such dispatching was that code could require only the namespace that contains (defmulti), and it would call the right implementation automatically.
19:53	justin_smith	Vincentdm: why would that require a circular dependency?
19:53	justin_smith	the file that defines a multimethod doesn't need to know about the implementations
19:54	justin_smith	and no, only requiring the file with the defmulti is not the point, at all
19:54	justin_smith	you only need to require the implementations in one place, and every other location can just use the defmulti though
19:59	Vincentdm	Hi Justin, seems you are right. My code was faulty (I copt-pasted a ::keyword when moving the defmethod), which caused the dispatching to stop working. This lead me to believe it didn't "find" the defmethod implementation
20:01	Vincentdm	So when calling a multimethod, Clojure will basically consider defmethods in whatever file, as long as it is included in the project dependencies?
20:01	justin_smith	Vincentdm: no
20:01	justin_smith	Vincentdm: the namespace with the defmethod needs to get loaded (usually via require)
20:01	justin_smith	it can be loaded anywhere, but it needs to be explicitly loaded, there is no implicit loading of code
20:02	Vincentdm	ok, good to know that
20:02	Vincentdm	So that means that I cannot add a new implementation of a multimethod, without explicitly referring to it from a more "general" place (which is actually not supposed to know this specific implementation exists)
20:02	Vincentdm	?
20:06	justin_smith	Vincentdm: define "supposed"
20:06	justin_smith	Vincentdm: the general pattern is you have a bunch of code that refers to the multimethod, and then one place that loads all the defmultis
20:06	justin_smith	usually a top level namespace that loads your config and starts everything up will also load up all those implementations
20:08	justin_smith	err, make that "all the defmethods"
20:08	Vincentdm	Well, I need to work with API's of different vendors, and I was hoping that I could add implementations of new APIs just by adding a new file containing defmethods. But based on your answer I understand that I will also need some central place loading all the implementations
20:08	justin_smith	sure, you need a top level namespace that controls that stuff, clojure avoids magic
20:08	justin_smith	but I misspoke above - you refer to the defmulti everywhere, and only load the files with defmethods in one place (usually your top level ns that also sets up your config)
20:10	Vincentdm	Ok, that's clear. As I understand it now multimethods are more about providing a clean syntax (so you don't need to do if/else/.... to dispatch incoming data to the appropriate functions), but it is not really a solution for reducing dependencies.
20:10	Vincentdm	Since I see no real difference between referring to my implementations from within the NS that does the defmulti compared to some other central namespace.
20:10	justin_smith	Vincentdm: yeah, we don't do implicit DI
20:10	justin_smith	it's just not a thing clojure does
20:11	Vincentdm	Alright. Thanks for clarifying that!
20:11	justin_smith	but all that requires is a single require
20:12	justin_smith	Vincentdm: for clarity, I usually find it useful to special case my top level namespace
20:12	Vincentdm	What do you mean with "special case"?
20:12	justin_smith	that is to say, that namespace breaks some rules the rest of my namespaces follow, because it has to configure everything else and decide how startup happens and what resources we use etc.
20:13	Vincentdm	ok, right
20:13	justin_smith	it doesn't do "just one thing" - it puts together all the pieces to make everything run (but ideally it does this without also implementing program logic)
20:14	justin_smith	so that would be the namespace that injects clojure dependencies (since unlike classes they are not injected by just having them on the classpath and referring to something they define)
20:15	Vincentdm	Ok, I see. Reminds me of the whole Stuart Sierra Component stuff. With an entrypoint that's responsible of setting everything up
20:15	justin_smith	exactly - I do that too
21:50	tjmaynes	hello
22:56	TimMc	puredanger: I really appreciate the clojure.org HTTPS redirect!
22:57	puredanger	np
22:57	TimMc	Is that TLS from Cloudfront back to the origin too?
22:59	TimMc	I've been going around today poking people about HTTPS and it has been surprisingly productive. :-D
22:59	puredanger	the origin is S3 in this case
23:00	TimMc	Ah, got it. So that's good.
23:00	TimMc	Interesting that you can't attach HSTS for CF. I understand it for S3, though.
23:00	puredanger	I found some places where people were complaining about it
23:01	puredanger	sounds like it might be something they will add
23:01	puredanger	if they do, I'll add it
23:02	TimMc	Oh, I remember this now, someone at work was dealing with this -- you can't add any custom headers to a CF distribution.
23:05	puredanger	yeah