{"id":11,"date":"2008-06-02T16:56:59","date_gmt":"2008-06-02T15:56:59","guid":{"rendered":"http:\/\/freesound.iua.upf.edu\/blog\/?p=11"},"modified":"2008-06-02T16:56:59","modified_gmt":"2008-06-02T15:56:59","slug":"testing-solr","status":"publish","type":"post","link":"https:\/\/blog.freesound.org\/?p=11","title":{"rendered":"Testing Solr&#8230;"},"content":{"rendered":"<p>After looking around for a search engine for Nightingale, and comparing features between all the various ones (from using tsearch2 on postgres to Sphinx to Solr to &#8230;) I&#8217;ve settled on <a href=\"http:\/\/lucene.apache.org\/solr\/\">Solr<\/a>. Configuring and running Solr was (much) easier than expected at the start. After about a day of hacking around, I got a nice tag-browser running, with Alax-ified searching through the tags. A bit more hacking around and I decided I would write a mash-up of all existing Python Solr wrappers. <a href=\"http:\/\/wiki.apache.org\/solr\/SolPython\">SolPython<\/a> and solr.py, the one that&#8217;s included with Solr, seemed very unpythonic and little developed. <a href=\"http:\/\/code.google.com\/p\/pysolr\/\">PySolr<\/a> on the other hand looked very nice, but there were some things in it I thought vould be better. Particularly, i wondered why the authors (two known Python\/Django devs) used the XML parsing and didn&#8217;t use the JSON output. When you search in Solr, you can tell it to reply you in a number of output formats. They chose the XML output, I rewrote to use the JSON output, and allowed for more output parsers to be written \/ plugged in.<\/p>\n<p>Neither Solr.py or PySol has classes for wrapping the search parameters. After reading through the docs I added a lightweight wrapper for a lot of the parameters.<\/p>\n<p>We keep track of searches in freesound, so we can &#8220;replay&#8221; those searches for testing purposes, and after a bit of testing I found out some interesting things. Using Solr and a relatively heavy set of output features (I want to see a lot of &#8220;faceting&#8221;), I tested a batch of 100K searches. It looks like I can run 50 queries per second on my macbook pro. As the set of documents in Freesound is relatively small (&#8220;only&#8221; 50K sounds), everything fits very nicely in a very small cache (only 128MB), inluding all faceting data.<\/p>\n<p>As before this source code is also open source, but -as Xavier gave me the go-ahead- this one is BSD instead og GPL. I will continue to release &#8220;support code&#8221; under the BSD license.<\/p>\n<p>The code can be found here: <a href=\"http:\/\/iua-share.upf.edu\/svn\/nightingale\/trunk\/sandbox\/solr\/solr.py\">http:\/\/iua-share.upf.edu\/svn\/nightingale\/trunk\/sandbox\/solr\/solr.py<\/a><br \/>\nThe example code I used for benchmarking here: <a href=\"http:\/\/iua-share.upf.edu\/svn\/nightingale\/trunk\/sandbox\/solr\/freesound_test.py\">http:\/\/iua-share.upf.edu\/svn\/nightingale\/trunk\/sandbox\/solr\/freesound_test.py<\/a><\/p>\n<p>Python\/Solr people, feel free to send me any feedback!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>After looking around for a search engine for Nightingale, and comparing features between all the various ones (from using tsearch2 on postgres to Sphinx to Solr to &#8230;) I&#8217;ve settled on Solr. Configuring and running Solr was (much) easier than &hellip; <a href=\"https:\/\/blog.freesound.org\/?p=11\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,3],"tags":[11],"class_list":["post-11","post","type-post","status-publish","format-standard","hentry","category-nightingale","category-techtalk","tag-solr-python"],"_links":{"self":[{"href":"https:\/\/blog.freesound.org\/index.php?rest_route=\/wp\/v2\/posts\/11","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.freesound.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.freesound.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.freesound.org\/index.php?rest_route=\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.freesound.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=11"}],"version-history":[{"count":0,"href":"https:\/\/blog.freesound.org\/index.php?rest_route=\/wp\/v2\/posts\/11\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.freesound.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=11"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.freesound.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=11"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.freesound.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=11"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}