Skip to content

Yonik Seeley

56 posts by Yonik Seeley

Solr 7.1 Features

Here’s an overview of some of the new features in Solr 7.1: Download Solr 7.1 or later to try these features out and give us feedback!

 

There is now a JSON mapping to Solr QParsers. Currently, one must use the JSON Request API to use this JSON syntax, but SOLR-11295 will provide a more general purpose entry point to JSON syntax in the future.

The general form of a query type in existing local params syntax is:

{!query_type param1=val1 param2=val2}query_value

OR, specifying the main query value using the v parameter:

{!query_type param1=val1 param2=val2 v=query_value}

In the new JSON syntax, the corresponding mappings would be

{query_type:{param1:val1, param2:val2, v:query_value}}

OR using query (which is a synonym for v when mapping to local params)

{query_type:{param1:val1, param2:val2, query:query_value}}

  Here’s an example of a pseudo-join query in local-params syntax that queries for book_review1 and follows the “book_id” field of the result(s) to the matching “id” field:

curl http://localhost:8983/solr/techproducts/query -d 'q={!join from:book_id to:id}id:book_review1'

In JSON syntax (indented for better readability), this would be

curl http://localhost:8983/solr/techproducts/query -d '
{
query:{
join:{
from : book_id,
to : id,
query : "id:book_review1"
}
}
}'

Note that the query parameter is specified in lucene syntax above, but it can be any query in local-params syntax or JSON syntax. Example:

curl http://localhost:8983/solr/techproducts/query -d '
{
query:{
join:{
from : book_id,
to : id,
query : { field : {f:"id", v:"book_review1"} } // invokes the "field" QParser
}
}
}'

 

Integral and date type support for min/max

Section titled “Integral and date type support for min/max”

Min and max aggregations in the JSON Facet API have been updated to include support for int, long, and date types. Previously, min and max of all of these field types returned double results.

 

Solr’s auto-scaling framework contains a number of new features and improvements, including

  • Ability to move replicas when nodes are added or removed to the cluster.
  • Solr’s autoAddReplicas feature now uses the autoscaling framework and works for all filesystems.
  • New API to control triggers and listeners (set-trigger, remove-trigger, suspend-trigger, set-listener, etc.)

As of this writing, the Solr 7.1 reference guide is not published yet. However you can check out the auto-scaling section of the very latest version of the unreleased reference guide for the master branch.

 

Here’s an example field using the new Geo3D spatialContextFactory that supports polygons:

<fieldType name="srptgeom_geo3d" class="solr.RptWithGeometrySpatialField"
spatialContextFactory="Geo3D" planetModel="wgs84"/>

Note: “wgs84” is an ellipsoid coordinate system for the Earth used by GPS. “sphere” can also be used for a spherical model that has faster but less accurate distance calculations.

 

Expanded support for statistical functions including various distributions, rank correlations, distances and more:

harmonicFit, polyfit, polyfitDerivative, betaDistribution, gammaDistribution, zipFDistribution, logNormalDistribution, weibullDistribution, monteCarlo, expMovingAverage, binomialCoefficient, factorial, movingMedian, primes, sumDifference, meanDifference, ebeAdd, ebeSubtract, ebeDivide, ebeMultiply, dotProduct, cosineSimilarity, Canberra, Chebyshev, Earth Movers and Manhattan Distance, Kendall’s Tau-b rank and Spearmans rank, discrete counting and probability, cumulativeProbability, normalDistribution, uniformDistribution, kolmogorovSmirnov, addAll

Solr 6.6 Features

Here’s an overview of some of the new features in Solr 6.6, released on June 6, 2017.

Download Solr 6.6 to try these features out and give us feedback! You can also check out upcoming features of the next Solr release.

This is likely to be the last Solr 6.x release before Solr 7.

  The release notes from the Apache Solr Wiki:

  • Payload support with payload() value source and {!payload_score} and {!payload_check} query parsers
  • Solr support for SimpleTextCodec, via in solrconfig.xml (per-field specification in the schema is not possible)
  • Multi-field support to TermsComponent when requesting terms’ statistics
  • Support for PointFields in Grouping, CollapseQParser, and ExpandComponent.
  • UPLOAD command (Config Set API) for uploading zipped configsets
  • MOVEREPLICA command (Collections API) for moving a replica across nodes
  • LISTALIASES command (Collections API) to return a list of all collection aliases
  • STATUS command (Core Admin API) to emit collection details of each core
  • Basic authentication can be enabled/disabled using bin/solr|bin/solr.cmd
  • ls command to ZkCLI for listing only sub-directories
  • Variance and Standard Deviation aggregators for the JSON Facet API
  • JSON Faceting now supports a query time ‘join’ domain change option
  • CartesianProductStream, which turns a single tuple with a multi-valued field into N tuples, one for each value in the multi-valued field
  • stats and search Streaming Expressions should now work in non-SolrCloud mode
  • analyze Stream Evaluator to support streaming NLP
  • New Stream Evaluators: Basic math, Date/time, UUID, Correlation, regress, predict, covariance, convolution, normalize
  • New Streaming Expressions: shuffle, echo, eval, timeseries, let, get
  • Solr default/example uses WordDelimiterGraphFilterFactory and SynonymGraphFilterFactory
  • New DataImportHandler ‘atom’ example, replacing broken ‘rss’ example
  • Redone DataImportHandler ‘tika’ example, removing all unused and irrelevant definitions
  • Expose cache statistics using metrics API
  • Improvements to metric reporters and API: support for “regex” parameter in /admin/metrics, “enabled” flag in reporter configurations, correct handling of “serviceUrl” in SolrJmxReporter, better handling of service clients for JMX, Ganglia and Graphite reporters
  • Deprecated LatLonType, GeoHashField, SpatialPointVectorFieldType, and SpatialTermQueryPrefixTreeFieldType. Instead, switch to LatLonPointSpatialField or SpatialRecursivePrefixTreeFieldType or RptWithGeometrySpatialField
  • Deprecated PostingsSolrHighlighter. Use UnifiedSolrHighlighter instead.
  • CloudSolrClient can now be initialized using the base URL of a Solr instance instead of ZooKeeper hosts
  • SolrJ: Added SolrParams.toLocalParamsString() and ClientUtils.encodeLocalParamVal
  • New AtomicUpdateProcessor to convert normal update operations to atomic update operations
  • totalTermFreq support to TermsComponent
  • Hide keystore and truststore passwords from /admin/info/* outputs
  • Configurability for thread pool size to recoveryExecutor
  • Introducing sort=childfield(field) asc for searching by {!parent}
  • facet.heatmap is now significantly faster when the docset (base query) matches everything and there are no deleted docs. It is also faster when the docset matches a small fraction of the index or none
  • Reduced heap consumption for filter({!join … score=…})
  • JSON Facet API now uses hyper-log-log++ for determining the number of buckets when merging requests from a multi-shard distributed request
  • Better ZkStateWriter batching
  • Using cache for DistributedQueue in case of single-consumer

Solr 6.5 Features

Here’s an overview of some of the new features in Solr 6.5. Download Solr 6.5 to try these features out and give us feedback!

This feature list was adapted from the release notes on the wiki: https://wiki.apache.org/solr/ReleaseNote65

  • PointFields (fixed-width multi-dimensional numeric & binary types enabling fast range search) are now supported
  • In-place updates to numeric docValues fields (single valued, non-stored, non-indexed) supported using atomic update syntax
  • A new LatLonPointSpatialField that uses points or doc values for query
  • It is now possible to declare a field as “large” in order to bypass the document cache
  • New sow=false request param (split-on-whitespace) for edismax & standard query parsers enables query-time multi-term synonyms
  • XML QueryParser (defType=xmlparser) now supports span queries
  • hl.maxAnalyzedChars now have consistent default across highlighters
  • UnifiedSolrHighlighter and PostingsSolrHighlighter now support CustomSeparatorBreakIterator
  • Scoring formula is adjusted for the scoreNodes function
  • Calcite Planner now applies constant Reduction Rules to optimize plans
  • A new significantTerms Streaming Expression that is able to extract the significant terms in an index
  • StreamHandler is now able to use runtimeLib jars
  • Arithmetic operations are added to the SelectStream
  • Added modernized self-documenting /v2 API
  • The .system collection is now created on first request if it does not exist
  • Admin UI: Added shard deletion button
  • Metrics API now supports non-numeric metrics (version, disk type, component state, system properties…)
  • The disk free and aggregated disk free metrics are now reported
  • The DirectUpdateHandler2 now implements MetricsProducer and exposes stats via the metrics api and configured reporters.
  • BlockCache is faster due to less failures when caching a new block
  • MMapDirectoryFactory now supports “preload” option to ask mapped pages to be loaded into physical memory on init
  • Security: BasicAuthPlugin now supports standalone mode
  • Arbitrary java system properties can be passed to zkcli
  • SolrHttpClientBuilder can be configured via java system property
  • Javadocs and Changes.html are no longer included in the binary distribution, but are hosted online

For more detailed lists with pointers to JIRA issues, see the HTML version of CHANGES.txt

Facet Domains

The domain of a facet is the set of values (normally defined by a set of documents) that calculations will be done over. The root domain is the set of documents that match the base query and any filters.

For any facet command, one can use the domain keyword to change the facet domain before facet computation.

The existence of the blockParent parameter in the domain block will cause incoming child documents to be mapped to their parents. The value of the blockParent parameter is the parent filter that specifies the complete set of parent documents for this block join operation.

Nested Documents -> Faceting On Parents for an example.

The existence of the blockChildren parameter in the domain block will cause incoming parent documents to be mapped to their children. The value of the blockChildren parameter is the parent filter that specifies the complete set of parent documents for this block join operation.

Nested Documents -> Faceting On Children for an example.

The excludeTags parameter causes the domain to be re-calculated from the root all the way to the current point, but excluding any filters with the specified tags. This is normally used as part of multi-select faceting.

The filter parameter is used to specify a filter or list of filters to be intersected with the incoming domain before faceting. These filters are applied after other domain transformations such as blockParent, blockChildren, or excludeTags. The same exact syntax for filter is supported in the JSON Request API for the top level document list (or will be after SOLR-9733).

Example:

json.facet = {
categories : {
type : terms,
field : cat,
domain : { filter:"user:yonik" }
}
}

The “param” query type can be used to grab zero or more filters from a request parameter:

q=my query&
myfilt=type:review&
myfilt=rating:5&
json.facet = {
categories : {
type : terms,
field : cat,
domain : {
filter:[
"user:yonik",
{param : myfilt} ]
}
}
}

Solr 6.4 Features

Here’s an overview of some of the new features in Solr 6.4.

Download Solr 6.4 to try these features out and give us feedback! You can also check out upcoming features of the next Solr release.

Any JSON facet command (terms, range, query) can now filter the facet domain in a simpler manner, without resorting to nested query facets.

Example:

json.facet = {
categories : {
type : terms,
field : cat,
domain : { filter:"user:yonik" }
}
}

The filters are applied after other domain change operations and are particularly useful when faceting on child documents. The filter attribute can be a single query or a list of multiple queries to intersect.

Using a param to refer to a filter by query parameter was added shortly after.

 

Learning to Rank (LTR) plugin for reranking results with machine learning models.

See the Lucene/Solr Revolution presentation Learning to Rank in Solr as well as the Solr Ref Guide.

 

snapshotcli.sh command line tool to manage snapshots

Section titled “snapshotcli.sh command line tool to manage snapshots”

The full path of the script is ./solr/server/scripts/cloud-scripts/snapshotscli.sh

An example from the pull request:

// Start solr and initialize a sample collection
bin/solr start -c
bin/solr create_collection -c books
curl 'http://localhost:8983/solr/books/update?commit=true' -H 'Content-type:application/json' -d '
[ {"id" : "book1", "title" : "American Gods", "author" : "Neil Gaiman" } ]'
//Create and export a snapshot
./snapshotscli.sh --create snap-1 -c books -z localhost:9983
./snapshotscli.sh --list -c books -z localhost:9983
./snapshotscli.sh --describe snap-1 -c books -z localhost:9983
./snapshotscli.sh --export snap-1 -c books -z localhost:9983 -d /tmp
./snapshotscli.sh --delete snap-1 -c books -z localhost:9983
// Restore the backup and verify the doc count
curl 'http://localhost:8983/solr/admin/collections?action=restore&name=snap-1&location=/tmp&collection=books_restored'
curl 'http://localhost:8983/solr/books_restored/select?q=*:*'

 

When parsing filter queries (including fq parameters) the standard solr query parser will avoid using BooleanQuery for term disjunctions on string and numeric fields, and will use TermsQuery instead.

This has a number of positive effects:

  • Avoids Lucene’s dreaded static maxBooleanClauses issue that causes “too many boolean clauses” exceptions
  • The resulting query should be smaller to cache
  • The resulting query should have higher performance

For example, the following filter will now be faster, and will no longer throw a “too many boolean clauses” exception:

fq=id:(myid1 myid2 myid3 myid4 ... myid2000)

 

The having streaming expression provides functionality like the SQL HAVING clause, and will be used to implement that in the future. Aggregated buckets produced by rollup are filtered by a having clause.

The following streaming expression filters out buckets (authors) with less than 100,000 in sales across all of their books:

having(rollup(over=author,
sum(sales),
search(book_sales_collection,
q=*:*,
fl="author,sales",
sort="author asc")),
gte(sum(sales), 100000))

Also see the a streaming expressions documentation in the ref guide.

Nested Objects in Solr

Nested Documents (also called Nested Objects) provides the ability to “nest” some documents inside of other documents in a parent/child relationship.

One reason for using nested documents is to prevent false matches. For example, we may have a T-Shirt with 2 SKUs, a Large Red, and a Medium Blue.

Say we tried to model this as a single document:

{
product : "Awesome T-Shirt",
color : [ "Red", "Blue" ],
size : [ "L", "M" ]
}

Now if we search for color:RED AND size:M , it would incorrectly match our document! But if we represented the SKUs as two different documents, then there would be no incorrect match.

{
color : "Red",
size : "L",
}
{
color : "Blue",
size : "M",
}

Lucene has a flat object model and does not really support “nesting” of documents in the index. Lucene *does* support adding a list of documents atomically and contiguously (i.e. a virtual “block”), and this is the feature used by Solr to implement “nested objects”.

When you add a parent document with 3 children, these appear int the index contiguously as

child1, child2, child3, parent

There is no Lucene-level information that links parent and child, or distinguishes this parent/child block from the other documents in the index that come before or after. Successfully using parent/child relationships relies on more information being provided at query time.

All children of a parent document must be indexed together with the parent document. One cannot update any document (parent or child) individually. The entire block needs to be re-indexed of any changes need to be made.

There are no schema requirements except that the _root_ field must exist (but that is there by default in all our schemas). Any document can have nested child documents.

“Block Join” refers to the set of related query technologies to efficiently map from parents to children or vice versa at query time. The locality of children and parents can be used to both speed up query operations and lower memory requirements compared to other join methods.

NOTE: This example currently requires Solr 5.3 or later.

First, bring up Solr and create a collection (if you have not done so already):

$ bin/solr start # this starts solr
$ bin/solr create -c demo # this creates a document collection called "demo"

Let’s remove any leftover docs from other examples:

curl http://localhost:8983/solr/demo/update?commitWithin=3000 -d '{delete:{query:"*:*"}}'

Now let’s add a book with some reviews as nested child documents (notice the _childDocuments_ element):

$ curl http://localhost:8983/solr/demo/update?commitWithin=3000 -d '
[
{id : book1, type_s:book, title_t : "The Way of Kings", author_s : "Brandon Sanderson",
cat_s:fantasy, pubyear_i:2010, publisher_s:Tor,
_childDocuments_ : [
{ id: book1_c1, type_s:review, review_dt:"2015-01-03T14:30:00Z",
stars_i:5, author_s:yonik,
comment_t:"A great start to what looks like an epic series!"
}
,
{ id: book1_c2, type_s:review, review_dt:"2014-03-15T12:00:00Z",
stars_i:3, author_s:dan,
comment_t:"This book was too long."
}
]
}
]'

Now we can see that these are really just indexed as 3 documents, all visible by default:

curl http://localhost:8983/solr/demo/query -d 'q=*:*&fl=id'
"response":{"numFound":3,"start":0,"docs":[
{
"id":"book1_c1"},
{
"id":"book1_c2"},
{
"id":"book1"}]
}

Now lets add an additional document with nested child documents for use with our query examples:

$ curl http://localhost:8983/solr/demo/update?commitWithin=3000 -d '
[
{id : book2, type_s:book, title_t : "Snow Crash", author_s : "Neal Stephenson",
cat_s:sci-fi, pubyear_i:1992, publisher_s:Bantam,
_childDocuments_ : [
{ id: book2_c1, type_s:review, review_dt:"2015-01-03T14:30:00Z",
stars_i:5, author_s:yonik,
comment_t:"Ahead of its time... I wonder if it helped inspire The Matrix?"
}
,
{ id: book2_c2, type_s:review, review_dt:"2015-04-10T9:00:00Z",
stars_i:2, author_s:dan,
comment_t:"A pizza boy for the Mafia franchise? Really?"
}
,
{ id: book2_c3, type_s:review, review_dt:"2015-06-02T00:00:00Z",
stars_i:4, author_s:mary,
comment_t:"Neal is so creative and detailed! Loved the metaverse!"
}
]
}
]'

TODO

One can return child documents along with every returned parent document by using the [child] doc transformer (it’s added to the fl field list parameter ). The list of child documents will be included under the “_childDocuments_” field of each parent.

$ curl http://localhost:8983/solr/demo/query -d '
q=cat_s:(fantasy OR sci-fi)&
fl=id,[child parentFilter=type_s:book]'
"response":{"numFound":2,"start":0,"docs":[
{
"id":"book1",
"_childDocuments_":[
{
"id":"book1_c1",
"type_s":"review",
"review_dt":"2015-01-03T14:30:00Z",
"stars_i":5,
"author_s":"yonik",
"comment_t":["A great start to what looks like an epic series!"]},
{
"id":"book1_c2",
"type_s":"review",
[...]

Child Doc Transformer Parameters:

  • parentFilter - identifies all of the parents. See the section on The Parent Filter for more info.
  • childFilter - optional query to filter which child documents should be included.
  • limit - maximum number of child documents to return per parent (defaults to 10)

Also see the Solr ref guide entry on the [child] doc transformer.

The JSON Facet API has support for switching the facet domain based on the nested document relationships.

The main query gives us a document list of reviews by author_s:yonik If we want to facet on the book genre (cat_s field) then we need to switch the domain from the children (type_s:reviews) to the parents (type_s:books).

$ curl http://localhost:8983/solr/demo/query -d '
q=author_s:yonik&fl=id,comment_t&
json.facet={
genres : {
type: terms,
field: cat_s,
domain: { blockParent : "type_s:book" }
}
}'

And we get a facet over the books which yonik reviewed:

"response":{"numFound":2,"start":0,"docs":[
{
"id":"book1_c1",
"comment_t":["A great start to what looks like an epic series!"]},
{
"id":"book2_c1",
"comment_t":["Ahead of its time... I wonder if it helped inspire The Matrix?"]}]
},
"facets":{
"count":2,
"genres":{
"buckets":[{
"val":"fantasy",
"count":1},
{
"val":"sci-fi",
"count":1}]
}}

Now lets say we’re displaying the top sci-fi and fantasy books, and we want to find out who reviews the most books out of our selection. Since our root implicit facet bucket (formed by the query and filters) consists of parent documents (books), we need to switch the facet domain to the children for the author facet.

$ curl http://localhost:8983/solr/demo/query -d '
q=cat_s:(sci-fi OR fantasy)&fl=id,title_t&
json.facet={
top_reviewers : {
type: terms,
field: author_s,
domain: { blockChildren : "type_s:book" }
}
}'

Response:

"response":{"numFound":2,"start":0,"docs":[
{
"id":"book1",
"title_t":["The Way of Kings"]},
{
"id":"book2",
"title_t":["Snow Crash"]}]
},
"facets":{
"count":2,
"top_reviewers":{
"buckets":[{
"val":"dan",
"count":2},
{
"val":"yonik",
"count":2},
{
"val":"mary",
"count":1}]
}}

 

By default, blockChildren will match all children of every parent doc from the input domain. It’s often the case that only a subset of the children are desired. The easiest way to limit children is with the filter clause.

For example, if we wanted to find the same top reviewers as before, but only for 5 star reviews:

$ curl http://localhost:8983/solr/demo/query -d '
q=cat_s:(sci-fi OR fantasy)&fl=id,title_t&
json.facet={
top_reviewers : {
type: terms,
field: author_s,
domain: {
blockChildren : "type_s:book",
filter : "stars_i:5"
}
}
}'

Note that regardless of which direction we are mapping (parents to children or children to parents), or what documents we are operating on, we provide a parent filter to define the complete set of parents in the index. In these examples, the parent filter is "type_s:book".

Solr JSON Request API

Although query parameters are often an easy method to create a Solr requests by hand, they have a number of drawbacks:

  • Inherently un-structured, requiring unsightly parameters like f.facet_name.facet.range.start=5
  • Inherently un-typed… everything is a string.
  • More difficult to decipher large requests.
  • Harder to programmatically create a request.
  • Impossible to validate. Because of the lack of structure, we don’t know the set of valid parameter and thus can’t do good error checking.

Solr already had a JSON API for faceting and analytics, and this new feature has extended that to the complete Solr request!

First let’s add a few excellent books from the fantasy genre (the “commitWithin=1000” will cause them to be visible to searches within 1000 milliseconds):

$ curl http://localhost:8983/solr/update?commitWithin=1000 -d '
[
{"id":"book1", "author":"Brandon Sanderson", "title":"The Final Empire",
"series_s":"Mistborn", "sequence_i":1, "genre_s":"fantasy"},
{"id":"book2", "author":"Brandon Sanderson", "title":"The Well of Ascension",
"series_s":"Mistborn", "sequence_i":2, "genre_s":"fantasy"},
{"id":"book3", "author":"Brandon Sanderson", "title":"The Hero of Ages",
"series_s":"Mistborn", "sequence_i":3, "genre_s":"fantasy"}
]'

Now we can search them with a JSON request rather than using query parameters:

$ curl http://localhost:8983/solr/query -d '
{
query:"hero"
}'

RESPONSE:

{
"responseHeader":{
"status":0,
"QTime":2,
"params":{
"json":"n{n query:"hero"n}"}},
"response":{"numFound":1,"start":0,"docs":[
{
"id":"book3",
"author":"Brandon Sanderson",
"author_s":"Brandon Sanderson",
"title":["The Hero of Aages"],
"series_s":"Mistborn",
"sequence_i":3,
"genre_s":"fantasy",
"_version_":1486581355536973824
}]
}
}

A few things to note from our example:

  • JSON body is considered a parameter named “json” and echoed back with the other params (unless you disable it with echoParams=none). This will also cause it to be logged, which is normally important.
  • The JSON we send to Solr can include unquoted simple strings and can contain comments. See JSON Extensions.
  • We don’t need to pass the Content-Type for indexing or for querying when we’re using JSON since Solr is now smart enough to auto-detect it when Curl is the client.
  • HTTP GET requests are now allowed to have a request body (i.e. try using “curl -XGET” for the query)

Here’s a more complete example:

curl -XGET http://localhost:8983/solr/query -d '
{
query : "*:*",
filter : [
"author:brandon",
"genre_s:fantasy"
],
offset : 0,
limit : 5,
fields : ["title","author"], // we could also use the string form "title,author"
sort : "sequence_i desc",
facet : { // the JSON Facet API is nicely integrated as well
avg_price : "avg(price)",
top_authors : {terms : author}
}
}'

It may sometimes be more convenient to pass the JSON body as a request parameter rather than in the actual body of the HTTP request. Solr treats a json parameter the same as a JSON body.

$ curl http://localhost:8983/solr/query -d 'json={query:"hero"}&fq=author:brandon'

Multiple json parameters in a single request are merged before being interpreted.

  • Single-valued elements are overwritten by the last value.
  • Multi-valued elements likefields and filter are appended.
  • Parameters of the form json.<path>=<json_value> are merged in the appropriate place in the hierarchy. For example a json.facet parameter is the same as “facet” within the JSON body.
  • A JSON body, or straight json parameters are always parsed first, meaning that other request parameters come after, and overwrite single valued elements.

Smart merging gives the best of both worlds… the structure of JSON with the ability to selectively separate out / decompose parts of the request!

curl 'http://localhost:8983/solr/query?json.limit=5&json.filter="genre_s:fantasy"' -d '
{
query : "hero",
limit : 10,
filter : "author:brandon"
}'

is equivalent to

curl http://localhost:8983/solr/query -d '
{
query : "hero",
limit : 5, // this parameter was overwritten
filter : [ "author:brandon" , "genre_s:fantasy" ] // this parameter was appended to
}'

In fact, you don’t even need to start with a JSON body for smart merging to be very useful. Consider the following request composed entirely of request params:

curl http://localhost:8983/solr/query -d 'q=*:*&rows=1&
json.facet.avg_price="avg(price)"&
json.facet.top_authors={type:terms,field:author_s,limit:5}'

That is equivalent to having the following JSON body or json parameter:

{
facet: {
avg_price: "avg(price)",
top_authors: {
type: terms,
field: author_s",
limit: 5
}
}
}

What to see what your merged JSON looks like? Just ask for debugging information (i.e. use the debug=true param), and it will come back under the "json" key along with the other debugging information.

We can also pass normal request parameters in the JSON body within the params block:

$ curl "http://localhost:8983/solr/query?fl=title,author"-d '
{
params:{
q:"title:hero",
rows:1
}
}
'

Which is equivalent to:

$ curl "http://localhost:8983/solr/query?fl=title,author&q=title:hero&rows=1"

Because we didn’t pollute the root body of the JSON request with the normal Solr request parameters (they are all contained in the params block), we now have the ability to validate requests and return an error for unknown JSON keys.

$ curl http://localhost:8983/solr/query -d '
{
query : "hero",
fulter : "author:brandon" // oops, we misspelled "filter"
}'

And we get an error back containing the error string:

"Unknown top-level key in JSON request : fulter"

Of course request templating via Parameter Substitution works fully with JSON request bodies or parameters as well.

Example:

$ curl "http://localhost:8983/solr/query?FIELD=text&TERM=hero&HOWMANY=10" -d '
{
query:"${FIELD}:${TERM}",
limit:${HOWMANY}
}'

The JSON Request API is currently in it’s infancy - only a few query parameters are supported (although the JSON Facet API which is part of this is more mature). Other Solr features you may want access to (like highlighting) currently need to be controlled through the normal Solr request params (e.g. just ad hl=true to the normal request parameters, or in the params block of a JSON request.

Have ideas on what will make the API better? Want to help out with development? We’d love to hear from you on the solr-user mailing list!

Solr 6.2 Features

Here’s an overview of some of the new features in Solr 6.2.

Download Solr 6.2 to try these features out and give us feedback! You can also check out upcoming features of the next Solr release.

Function queries for comparisons of numeric arguments:

  • gt - greater than
  • gte - greater than or equal
  • lt - less than
  • lte - less than or equal
  • eq - equal to

These functions return a boolean value for any given document, which is normally used in conjunction with the if function which returns the second argument if the first argument is true, or the third argument otherwise.

Example: For a given document, if num_reviewers is less than 5, return the value of the editor_score field, otherwise return the user_score field.

if( lt(num_reviewers,5), editor_score, user_score)

An easy way to experiment with function queries and see the results is to use the pseudo-field functionality to return specific function values along with stored fields.

Example: http://localhost:8983/solr/techproducts/query?q=\*:\*&fl=id,popularity,**gt(popularity,3)**

[...]
{
"id":"SP2514N",
"popularity":6,
"gt(popularity,3)":true},
{
"id":"6H500F0",
"popularity":6,
"gt(popularity,3)":true},
{
"id":"F8V7067-APL-KIT",
"popularity":1,
"gt(popularity,3)":false},
[...]

 

The features and train streaming expressions are machine learning related operations used to implement a text classifier. The goal is to create a classifier (or model) based on a training set of documents that can be used to tell if a new document should be considered part of that set.

  • features - streaming expression that selects important features (important terms) from a set of documents called the training set (specified by a query).
  • train - trains a logistic regression model on a text field using the selected features and the training set.

Here’s an example of the streaming expression syntax taken from the JIRA:

train(collection1, q="*:*",
features(collection1,
q="*:*",
field="body",
outcome="out_i",
positiveLabel=1,
numTerms=100),
field="body",
outcome="out_i",
maxIterations=100)

See the features streaming expression and the train streaming expression in the reference guide.

 

The topic streaming expression implements a publish/subscribe messaging capabilities, where clients can subscribe to queries and receive deliveries of new documents that match those topic queries. When the first call is made, an id is specified to uniquely identify the topic. A checkpointCollection is also specified, and this is where topic progress is stored to keep track of which documents have already been returned.

Example syntax:

topic(checkpointCollection,
collection1,
id="yonik_topic1",
q="title:(lucene solr)",
fl="id, title, abstract, author")

A client can keep calling the same topic to receive new documents, or it can be wrapped in a daemon streaming expression to provide push functionality.

 

The scoreNodes streaming expression is used to add a relevancy score to a graph expression, similar to tf-idf for text. The tf factor is how many times a node appears in the graph traversal, and biases scores in favor of nodes that appear often. The idf factor biases scores in favor of nodes that are more rare in the index.

There is a good example of using this in the ref guide: Calculating Market Basket Co-occurrence, for calculating product recommendations based on your current shopping basket and what other shopping baskets look like.

 

collections API: REPLACENODE and DELETENODE commands

Section titled “collections API: REPLACENODE and DELETENODE commands”

New actions have been added to the collections API:

  • REPLACENODE - moves all replicas from one node to another.
  • DELETENODE - deletes all replicas currently on a given node.

Examples: /admin/collections?action=RELACENODE&source=MY_OLD_NODE&target=MY_NEW_NODE /admin/collections?action=DELETENODE&node=MY_NODE_NAME

See the relevant sections in the Collections API for more details.

 

Kerberos delegation token support was added to the authentication filter, especially useful for when distributed clients such as MapReduce do not have access to the user’s credentials.

See Kerberos Authentication Plugin in the ref guide.

 

Whole index replication for CDCR (Cross Data Center Replication)

Section titled “Whole index replication for CDCR (Cross Data Center Replication)”

CDCR can now fall back to whole-index replication to bring a replica up to date when transaction logs have not retained enough updates to do so.  

SolrCloud nodes in the cluster have tags that may be used when performing rule based replica placement. A “role” tag has been added that currently be used to prevent placement of new replicas on the overseer node by adding rule=role:!overseer during collection creation.

See Rule Based Replica Placement in the ref guide for more info.

Solr 6.3 Features

Here’s an overview of some of the new features in Solr 6.3. Download Solr 6.3 to try these features out and give us feedback! You can also check out upcoming features of the next Solr release.

New dvhash faceting method in the JSON Facet API

Section titled “New dvhash faceting method in the JSON Facet API”

A new faceting method for the JSON Facet API has been added that utilizes hashing instead of direct indexing by ordinal. Currently, when hashing on a field with docValues enabled, or when specifying method=dv, there are accumulator arrays that are indexed by ordinal. When the cardinality of the domain is low, and the cardinality of the field is high, direct indexing of large arrays both wastes memory and takes longer to scan looking for the highest accumulated entries.

Example:

json.facet={
top_authors : {
type : terms,
field : author,
limit : 20,
method : dvhash
}
}

Since it may be difficult for a client to know when the facet domain will be large vs the field cardinality (esp if the domain is based off of a user query), future work is expected to implement automatically switching between faceting methods.

Optimizing, storing and deploying AI models with Streaming Expressions

Section titled “Optimizing, storing and deploying AI models with Streaming Expressions”

The classify streaming expression classifies documents according to a model.

classify(<document_stream>, // the stream to fetch documents that need to be classified
<model_stream>, // the stream to fetch the latest model
field=<input_text_field>, // the field on the document to classify
fieldType=<analysis_fieldType> // (opt) fieldType to use for tokenization of the input field
)

For each input document, the classifier decorates the document with two (currently hardcoded) fields with the result:

  • score_d - the raw score
  • probability_d - the positive probability that the document belongs to the group

The update streaming expression can be used to store classifier models in a SolrCloud collection, and the topic streaming expression can be used with the classify expression to both deploy models as well as stream new data through the models.

The development JIRA SOLR-9258 has examples, and should be consulted until the Solr reference guide is updated.

The executor streaming expression wraps another stream containing streaming expressions. By default the expression will be contained in the expr_s field in each tuple of the wrapped stream. The executor has an internal thread pool so expressions can be executed in parallel on a single worker. This expression can further be wrapped in a parallel streaming expression to enable execution across a cluster of worker nodes.

Example syntax from the JIRA of a work queue… the “topic” expression retrieves expressions to execute (and keeps track of where it left off):

daemon(executor(threads=10, topic(storedExpressions, fl="expr_s", ...)))

The commit streaming expression wraps another stream (normally involving updates) and performs a commit at the end.

Example:

commit(targetCollection, update(targetCollection, search(sourceCollection, q="myquery", ...)))

The fetch streaming expression wraps another expression and fetches additional fields from documents in batches. It’s very much like an inner join, with one side of the join being very small.

Example that queries books and adds publisher address to each result:

fetch(publisherCollection,
search(booksCollection, q="*:*", fl="title,author,publisher", sort="publisher asc"),
fl="publisherAddress",
on="publisher=publisherId")

See the release notes on the Solr wiki and the CHANGES file for an in-depth list of changes.

Solr 6.1 Features

Here’s an overview of some of the new features in Solr 6.1.

Download Solr 6.1 to try these features out and give us feedback! You can also check out upcoming features of the next Solr release.

A shortestPath Streaming Expression was added that implements a distributed breadth-first graph traversal to find the shortest paths in a directed directed graph.

Example:

shortestPath(collection,
from="john@company.com",
to="jane@company.com",
edge="from=to",
threads="6",
partitionSize="300",
fq="limiting query",
maxDepth="4")

gatherNodes Streaming Expression (coming soon)

Section titled “gatherNodes Streaming Expression (coming soon)”

The gatherNodes expression is a more general form of graph traversal than shortestPath, and can be used for more use cases.

Example:

gatherNodes(friends,
gatherNodes(friends,
search(articles, q=“body:(queryA)”, fl=“author”),
walk ="author->user”,
gather="friend"),
walk=“friend->user”,
gather="friend",
scatter=“roots, branches, leaves”)

ToleranteUpdateProcessorFactory will skip update commands that would otherwise cause subsequent updates in a batch to fail.

<updateRequestProcessorChain name="tolerant">
<processor class="solr.TolerantUpdateProcessorFactory"/>
<processor class="solr.DistributedUpdateProcessorFactory" />
<processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>

Passing update.chain=tolerant will use the processor chain defined above. One can also pass maxErrors=10 to limit the number of errors before aborting the complete update request. For example, if one is loading a large CSV file with millions of entries for the first time, it may be useful to abort early if every addition would fail due to a configuration error.

Filter creation for small cardinality sets (those that match few documents) now produces much less garbage. Up to 3x performance producing small sets (due to less GC overhead).

The HDFS block cache now skips caching “read once” scenarios such as index merges.

Solr 7 Features

Here’s an overview of some of the new features in Solr 7.0 Download Solr 7 to try these features out and give us feedback!

 

The now deprecated trie-based numeric fields use (and abuse) the full-text index to index parts of numbers to speed up range queries. The new Points-based numeric fields do not use the full-text index, but instead have a dedicated index structure designed specifically for multidimensional ranges over numbers: the BKD tree. The BKD tree index structure is smaller and faster for range queries, but is slightly slower for exact match queries.

Since Point fields do not currently support un-inversion (i.e. FieldCache), some search functionality requires that points have docValues enabled for fast per-document lookups. This includes sorting, function queries, {!graph}, and {!join} queries.

  Current template schemas define types like the following:

<fieldType name="pint" class="solr.IntPointField" docValues="true"/>
<fieldType name="pints" class="solr.IntPointField" docValues="true" multiValued="true"/>

Along with corresponding dynamic field types:

<dynamicField name="*_i" type="pint" indexed="true" stored="true"/>
<dynamicField name="*_is" type="pints" indexed="true" stored="true"/>

  Interested in how the underlying BKD data structure works? Here’s a great lesson from Robert Sedgewick, Princeton University on KD trees (BKD trees are just a variant of KD trees):

 

The JSON Facet API has a new parameter called refine that turns on two-phase refinement of partial facets during distributed faceting. This guarantees that the statistics (counts or other metrics) within returned facet buckets are accurate.

Partial facets are those facets that specify a limit and thus may not return all facet buckets from all shards in a distributed search. If refine is set to true on one of these partial facets, a second phase is used to “refine” the top buckets from the first phase, collecting information from other shards that did not yet contribute to those buckets. Without refinement, counts and statistics for the bucket can be incorrect. The second phase of faceting normally does not cause any additional HTTP requests since they are piggy-backed onto the normal second phase of distributed search that retrieves stored fields for the top document ids.

Example:

json.facet={
x : {
type : terms,
field : cat,
limit : 5,
refine : true
}
}

 

The min/max facet aggregations (facet functions) have been extended beyond numeric fields and functions to include single-valued string fields.

 

Streaming Expressions has many new functions:

  • movingAvg
  • arraySort
  • cumulative
  • anova
  • hist
  • array
  • sequence
  • finddelay
  • knn
  • describe
  • copyOfRange
  • sql
  • copyOf
  • distance
  • scale
  • rank
  • length
  • reverse

 

SolrCloud now has support for different replica types.

NRT stands for Near Real Time. This is the default and original replica type in SolrCloud. Updates flow from the leader to all replicas and are added to replica transaction logs as well as indexed. This is the only type of replica to support soft commits since TLOG and PULL replicas need hard commits to copy over new index segments.

Updates flow from the leader to all replicas and are added to replica transaction logs (tlogs) only. Replicas are kept up-to-date by pulling new index segment files from the leader. The transaction logs allow these replicas to recover and become a leader if necessary, as well as directly service real-time get requests.

Updates do not flow from the leader to replicas. Replicas are kept up-to-date by pulling new index segment files from the leader. This is similar to the original non-SolrCloud master-slave replication. Replicas of this type may not become leaders since updates would most likely be lost. Real-time get requests to a PULL replica are forwarded to the leader since these replicas lack transaction logs to find the most recent uncommitted updates.

 

  • The default response format has been changed from XML to indented JSON. Add wt=xml to the request obtain an XML response, and add indent=off if you wish to turn off indenting.
  • The new v2 API, exposed at /api/ is now the preferred API (esp for using the collections API), but /solr/ continues to be supported.
  • The alternate “Analytics Component” in the contrib modules was bumped to v2, with distributed support and a new JSON-based request syntax
  • Auto-scaling framework that allows Solr to place new replicas based on metrics such as free disk space.
  • The standard lucene/solr query parser now defaults to sow=false, meaning that for text fields, it does not split on whitespace before handing the text to the analyzer. This enabled multi-word synonyms to be matched by the analyzer.
  • When a collection is created without specifying a configset, the new ‘_default’ configset is now used. It is data-driven (schemaless), and indexes strings as analyzed text in addition to using a copyField to *_str field suitable for sorting or faceting.

See CHANGES for more detailed upgrade notes. Also see the official release notes on the Solr wiki. The Solr Reference Guide should contain other upgrade info.

Solr 6 Features

Here’s an overview of some of the new features in Solr 6.0.

Download Solr 6 to try these features out and give us feedback! You can also check out upcoming features of the next Solr release.

Parallel SQL queries across SolrCloud collections. The SQL engine is built on top of Solr’s Streaming API (Streaming Expressions), which provides support for parallel relational algebra and real-time map-reduce.

  • SQL statements are compiled to Streaming Expressions for parallel execution across SolrCloud worker nodes.
  • SolrCloud collections are abstracted as Relational Tables.
  • Full support for Lucene/Solr query syntax in the WHERE clause.
  • Many operations such as grouping/rollups can be automatically parallelized, utilizing the real-time MapReduce capabilities of the streaming expressions.
  • Grouping/rollup operations can be pushed down and leverage the JSON Facet API for increased performance.
  • Currently works in SolrCloud mode only… no standalone mode yet.
  • SQL functionality is currently experimental and incomplete (for example underlying streaming join functionality is still in the process of being integrated.)

Examples:

select category, count(*), sum(inventory), min(price), max(price), avg(outstanding) from collection1 where text='4k HDTV' group by category order by sum(inventory) asc limit 10
select id,category from collection1 where category = '(dvd OR bluray)' order by category desc limit 100
select fieldA, fieldB, count(*), sum(fieldC), avg(fieldY)
from collection1
where fieldC = 'term1 term2'
group by fieldA, fieldB
having sum(fieldC) > 1000
order by sum(fieldC) asc
limit 100

See the Parallel SQL documentation for more info.

There is a SQL Request Handler mapped to the /sql endpoint. Only SolrCloud collections can currently be searched with the SQL handler.

Example:

$ curl http://localhost:8983/solr/techproducts/sql -d "stmt=select id from techproducts"
{"result-set":{"docs":[
{"id":"EN7800GTX/2DHTV/256M"},
{"id":"100-435805"},
{"id":"UTF8TEST"},
{"id":"SOLR1000"},
{"id":"9885A004"}
]}}

Solr has a new JDBC driver that may be used to access the new SQL functionality. The Solr JDBC driver has been tested with DbVisualizer, Apache Zeppelin, and SQuirreL SQL so far.

The general form of the JDBC connection string is:

jdbc:solr://SOLR_ZK_CONNECTION_STRING?collection=COLLECTION_NAME

The JDBC driver is not yet documented in the Ref Guide, so see https://issues.apache.org/jira/browse/SOLR-8521 for more documentation in the meantime.

A number of distributed join operations have been added to streaming expressions:

  • innerJoin
  • leftOuterJoin
  • hashJoin
  • outerHashJoin

Example:

innerJoin(
search(collection1, q=*:*, fl="fieldA, fieldB, fieldC", ...),
search(collection2, q=*:*, fl="fieldA, fieldD, fieldE", ...),
on="fieldA=fieldA"
)

The rollup streaming expression groups tuples by common field values and emits the rollup value along with other specified metrics.

Example:

rollup(
search(collection1, qt="/export"
q="*:*",
fl="id,manu,price",
sort="manu asc"),
over="manu"),
count(*),
max(price)
)

The facet streaming expression is much like the rollup expression, but it pushes down the computation to the leaves using the JSON Facet API.

Example:

facet(
techproducts,
q="*:*",
buckets="manu",
bucketSorts="count(*) desc",
bucketSizeLimit=1000,
count(*),
sum(price),
max(popularity)
)

Many other Streaming Expressions were added for the Solr 6 release. At this point, they should all be documented in the Streaming Expressions section of the Solr Ref Guide.

   

A basic graph traversal query that follows nodes to edges, optionally filtering during traversal.

Example: Assume we have documents that represent people, and each document has a field called “parent_id” which lists the parents. This example query matches “Philip J. Fry” and all of his ancestors:

fq={!graph from=parent_id to=id}id:"Philip J. Fry"

The main argument to the graph query defines the root set, in this case id:"Philip J. Fry". The graph query then iteratively follows the parent_id field to documents with corresponding id fields (i.e. for each iteration, the values for parent_id in the current set are matched to id field of all documents). The basic graph query is equivalent to a repeated join query.

Graph query parameters:

  • from - The field used in the current set of documents to match to the to field in the set of destination documents.- to - The field used to find matches in the values obtained from the from fields of the starting set of documents.- traversalFilter - A filter query that is applied on each iteration.
  • returnRoot - Controls whether the root set of documents should be included. Defaults to “true”.
  • returnOnlyLeaf - If true, only returns leaf documents. Defaults to “false”.
  • maxDepth - The maximum number of iterations before graph traversal stops. Defaults to -1 (unlimited).

NOTE: This graph query only traverses edges in the same index (i.e. it does not traverse edges across different nodes/cores). Distributed graph traversal is being developed and will be in future versions of Solr 6.x

Default scoring now uses Okapi BM25 by default. You can enable the old tf-idf vector space similarity by using ClassicSimilarity.

Here is an example of how to use classic tf-idf similarity just for the “text” fieldType in the Solr schema:

<fieldType name="text" class="solr.TextField">
<analyzer class="org.apache.lucene.analysis.standard.StandardAnalyzer"/>
<similarity class="solr.ClassicSimilarityFactory"/>
</fieldType>

For BM25, one can tweak the scoring function on a per-fieldType basis. For example:

<fieldType name="text2" class="solr.TextField">
<analyzer class="org.apache.lucene.analysis.standard.StandardAnalyzer"/>
<similarity class="solr.BM25SimilarityFactory">
<float name="k1">1.2
<float name="b">0.75
</similarity>
</fieldType>

For those new to full-text search terminology, “similarity” produces a score for how similar a document is to a full-text query. When determining this score, document statistics as well as corpus statistics are used. Some scoring factors include:

  • The number of times a search term appears in the document field. More matches produces a higher score.
  • The size of the document field. Longer fields produce a lower score, with the idea being that for a given number of term matches, shorter is better (more specific match)
  • The average length of the field across the entire corpus (BM25 considers this, classic tf-idf does not)
  • How common the query terms are across the entire corpus. The idea being that rarer terms carry more information. For example, if I searched for “blue whale”, all else being equal, I’d probably want things about whales to score higher than things about blue.

Real-Time Get (normally handled at the /get URL) now handles filters (fl parameters) to restrict matching documents.

Example:

curl "http://localhost:8983/solr/demo/get?id=book1,book2,book3&fl=security:group1"

An experimental version of CDCR (Cross Data Center Replication) has been added that supports an active-passive configuration.

Updates are asynchronously sent from the active cluster leader to the passive cluster leader. Since updates are asynchronously relayed from the leader’s transaction logs, temporary connectivity issues between data centers can be tolerated with no interruption in service for the primary DC.

Here is the in-progress CDCR documentation. The documentation link is likely to change once CDCR moves out of it’s “experimental” phase.

Solr 5.5 Features

Here’s an overview of some of the new features in Solr 5.5 Also see Solr Download Links and upcoming Features of the next Solr release.

Return docValues fields like stored fields

Section titled “Return docValues fields like stored fields”

In previous Solr versions, returning the top N documents only retrieved field values from the row-store (i.e. fields where “stored” is true). Values will now also be retrieved from docValues (which are essentially column stored) for fields where “stored” is false. This avoids the need to duplicate the value in the row store if it’s already column stored.

A "facet-trace" section has been added to the debug info for JSON Facets. The exact format is subject to change, but the current information includes information about each facet command, including the processor used to execute the facet and the domain size. This information is included recursively included for each facet bucket for facets with sub-facets.

Example request:

$ curl http://localhost:8983/solr/techproducts/query -d 'q=*:*&rows=0&debug=true&
json.facet={
categories:{
type : terms,
field : cat
}
}

Example response debug info:

[...]
"facet-trace":{
"processor":"FacetQueryProcessor",
"elapse":0,
"query":null,
"domainSize":32,
"sub-facet":[{
"processor":"FacetFieldProcessorUIF",
"elapse":0,
"field":"cat",
"limit":10,
"numBuckets":16,
"domainSize":32}]},
[...]

facet.method=uif parameter causes traditional field faceting to delegate to the JSON Facet API with method=uif. This is roughly equivalent to what the Solr 4 default faceting method was for multi-valued fields. It is optimized for performance over static indexes rather than NRT (quickly changing indexes).

The compression mode for stored fields can now be specified via codecFactory in solrconfig.xml See the Codec Factory section in the Solr reference guide for more details.

Generic support was added for making collection APIs async. Async support was added for the following commands: delete/reload collection, create/delete alias, create/delete shard, delete replica, add/delete replica property, add/remove role, overseer status, balance shard unique, rebalance leaders, modify collection, migrate state format.

See Asynchronous Calls in the Solr ref guide.

There is a new experimental BlockJoinFacetComponent for calculating facets by a child.facet.field parameter with a {!parent} query. The component is not enabled by default. Note that this component is unrelated to the block join faceting support in the JSON Facet API.

The XML query parser, registered as “xmlparser” is a direct interface to Lucene’s XMLQueryParser (CoreParser). Personal recommendation: avoid the use of this query parser unless you have very unusual/specific needs.

Example:

curl http://localhost:8983/solr/techproducts/query -d 'debugQuery=true&
q={!xmlparser}
<BooleanQuery>
<Clause occurs="must"> <TermQuery fieldName="name">ipod</TermQuery> </Clause>
<Clause occurs="must"> <TermQuery fieldName="manu">apple</TermQuery> </Clause>
</BooleanQuery>
'

This parser does not do any text analysis on terms, so provided terms will need to match what is in the index exactly (i.e. you will need to do things like lowercasing and stemming yourself). Good backward compatibility is unlikely with this parser as it exposes more internal implementation details. Term queries on fields such as numeric fields, enum fields, and boolean fields will only work if you know the internal term representation in the index.

See XmlQParser in the Solr ref guide for more info.

A configset, or configuration set, is a set of config files for a Solr collection. For SolrCloud mode, an upconfig option has been added to the /bin/solr script to upload a configset to zookeeper. A matching downconfig option has been added to download a configset from zookeeper.

For examples and documentation, see bin/solr Zookeeper Operations in the Solr reference guide.

There is an internal CheckHdfsIndex class that can be run from the command line for HDFS indexes like CheckIndex can be run for normal indexes. Example:

java -cp "./server/solr-webapp/webapp/WEB-INF/lib/*:./server/lib/ext/*" -ea:org.apache.lucene... org.apache.solr.index.hdfs.CheckHdfsIndex /path/to/my/index/

  For reference, here is also the command to run CheckIndex on a local (non-HDFS) lucene index:

java -cp "./server/solr-webapp/webapp/WEB-INF/lib/*:./server/lib/ext/*" -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex ./example/techproducts/solr/techproducts/data/index

Which will result in the following output:

Opening index @ ./example/techproducts/solr/techproducts/data/index
Segments file=segments_2 numSegments=1 version=6.0.0 id=bv1fdquc5dh3nvcf4jxiwfow4 format= userData={commitTimeMSec=1456073865693}
1 of 1: name=_0 maxDoc=32
version=6.0.0
id=bv1fdquc5dh3nvcf4jxiwfow3
codec=Lucene60
compound=false
numFiles=13
size (MB)=0.026
diagnostics = {java.runtime.version=1.8.0_40-b25, java.vendor=Oracle Corporation, java.version=1.8.0_40, java.vm.version=25.40-b25, lucene.version=6.0.0, os=Mac OS X, os.arch=x86_64, os.version=10.11.2, source=flush, timestamp=1456073865745}
no deletions
test: open reader.........OK [took 0.059 sec]
test: check integrity.....OK [took 0.000 sec]
test: check live docs.....OK [took 0.000 sec]
test: field infos.........OK [25 fields] [took 0.000 sec]
test: field norms.........OK [5 fields] [took 0.001 sec]
test: terms, freq, prox...OK [1187 terms; 1813 terms/docs pairs; 1496 tokens] [took 0.025 sec]
test: stored fields.......OK [356 total field count; avg 11.1 fields per doc] [took 0.013 sec]
test: term vectors........OK [3 total term vector count; avg 1.0 term/freq vector fields per doc] [took 0.006 sec]
test: docvalues...........OK [0 docvalues fields; 0 BINARY; 0 NUMERIC; 0 SORTED; 0 SORTED_NUMERIC; 0 SORTED_SET] [took 0.000 sec]
No problems were detected with this index.
Took 0.246 sec total.

Multi-Select Faceting

NOTE: This uses syntax from the upcoming Solr 5.4 release. If you are using Solr 5.2 or 5.3, specify domain:{excludeTags:mytag} as excludeTags:mytag.

Multi-select faceting is a powerful faceting style that allows users to see and select multiple facet constraints (facet values) for certain facets. This example uses Solr’s JSON Facet API along with filter tagging and excluding to implement this style of faceting.

Let’s say we have 3 facets, Size, Color, and Brand. This is multi-select faceting because for the Color and Brand facets, we want the user to be able to select multiple constraints (values). The Size facet is single-select since we’ve decided that customers in general will only be interested in one size at a time.

Here is our super-fancy ASCII UI, after the user has searched for "running shorts":

=== Size === === Color === === Brand ===
[Small] (7) [ ] Red (2) [ ] Nike (7)
[Medium] (5) [ ] Blue (8) [ ] Adidas (5)
[Large] (6) [ ] Green (3) [ ] Reebok (4)
[ ] Black (5) [ ] Under Armour (2)
(Top matches sorted by popularity displayed here... use your imagination!)

Note that the Color and Brand facets have checkboxes to indicate which constraints have been selected. We’re starting off with no constraints. Below the facets is where we would display the top matching items, along with pretty pictures, prices, etc.

 

The user selects “Blue” so we add that as a filter and re-issue the request (we’re using Solr’s JSON Facet API):

&q="running shorts"
&fq=color:Blue
&json.facet={
sizes:{type:terms, field:size},
colors:{type:terms, field:color},
brands:{type:terms, field:brand}
}

We get back the response and update our UI from that data:

=== Size === === Color === === Brand ===
[Small] (3) [ ] Red (0) [ ] Nike (3)
[Medium] (2) [x] Blue (8) [ ] Adidas (2)
[Large] (3) [ ] Green (0) [ ] Reebok (2)
[ ] Black (0) [ ] Under Armour (1)
(Top Blue running shorts displayed here)

What’s right: The Size and Brand facets now reflect the fact that we’ve selected Blue, and that’s what we wanted. Our list of top matches also only includes Blue things, just as we wanted.

What’s wrong: Because we filtered out anything that wasn’t Blue, we get back 0 counts for other colors! But we still want the other color information so the customer can select additional colors.

 

When we compute the multi-select Color facet, we want to ignore any constraints (filters) on that facet so we will get back the correct counts for other colors. To accomplish this, we can tag filters and then selectively exclude filters (i.e. pretend they don’t exist) by tag when faceting.

The same thing applies to the multi-select Brand facet… we want any Brand selections to affect everything else (including all other facets), except for the Brand facet itself.

When the user selects Blue, we add that as a filter tagged with COLOR and re-issue the request:

&q="running shorts"
&fq={!tag=COLOR}color:Blue
&json.facet={
sizes:{type:terms, field:size},
colors:{type:terms, field:color, domain:{excludeTags:COLOR} },
brands:{type:terms, field:brand, domain:{excludeTags:BRAND} }
}

Now when we get back our response, it still includes the other Colors in the Color facet.

=== Size === === Color === === Brand ===
[Small] (3) [ ] Red (2) [ ] Nike (3)
[Medium] (2) [x] Blue (8) [ ] Adidas (2)
[Large] (3) [ ] Green (3) [ ] Reebok (2)
[ ] Black (5) [ ] Under Armour (1)
(Top Blue running shorts displayed here)

The domain is the set of documents that facets will be calculated over. In the JSON Facet API, the domain keyword/command is normally used to change the domain before the facets are calculated.

In our example above, we specified domain:{excludeTags:COLOR} for the colors facet. This will re-calculate the facet domain as if any filters tagged with COLOR were not applied.

 

Ok, now the user selects Black as well.

We naively add an additional filter, fq={!tag=COLOR}color:Black to the request, just as we would with traditional single-select faceting.

What’s wrong: Everything! Our request matches nothing and we get back all 0’s.

This is because filters are logically intersected. We searched for things that were Blue AND Black, and that will match nothing (assuming our items only have a single color). What we really want is Blue OR Black.

 

We need the logical OR, or union, of all the selected colors.

&q="running shorts"
&fq={!tag=COLOR}color:(Blue Black)
&json.facet={
sizes:{type:terms, field:size},
colors:{type:terms, field:color, domain:{excludeTags:COLOR} },
brands:{type:terms, field:brand, domain:{excludeTags:BRAND} }
}
=== Size === === Color === === Brand ===
[Small] (5) [ ] Red (2) [ ] Nike (5)
[Medium] (4) [x] Blue (8) [ ] Adidas (3)
[Large] (4) [ ] Green (3) [ ] Reebok (3)
[x] Black (5) [ ] Under Armour (2)
(Top Blue and Black running shorts displayed here)

Note that the counts on the other facets increased to reflect the larger domain (it includes both blue and black items).

Although our simple example just dealt with facet counts, multi-select faceting via excludeTags works with the broad range of features in the JSON Facet API. The domain change will apply to everything else under that facet, including Sub-facets and Facet Functions.

   

Tagging and excluding filters with excludeTags

Section titled “Tagging and excluding filters with excludeTags”

Solr filter queries (fq parameters) can be tagged with arbitrary strings using the localParams {!tag=mystring} syntax. Example: fq={!tag=COLOR}color:Blue

  • Multiple filters can be tagged with the same tag. Example: fq={!tag=foo}one_filter&fq={!tag=foo}another_filter
  • A single filter may be tagged with multiple tags. Example: fq={!tag=tag1,tag2,tag3}my_field:my_filter

During faceting, the facet domain may be changed to exclude filters that match certain tags via the excludeTags keyword. It’s as if the filter was never specified for that specific facet. This is useful for implementing multi-select faceting Example: colors:{type:terms, field:color, domain:{excludeTags:COLOR}}

  • excludeTags can be multi-valued comma-separated string. Example: excludeTags:"tag1,tag2"
  • excludeTags can be a JSON array of tags. Example: excludeTags:["tag1","tag2"]
  • One can exclude tags that are not used in the current request. This makes constructing requests simpler since you don’t need to worry about changing the faceting part of the request based on what filters have been applied.
  • For nested facets, excludeTags can appear at any level of the hierarchy. They do not currently “stack” though. If a parent facet has excludeTags:tag1 and a child facet wants to additionally exclude tag2 filters, then they must currently do so explicitly with excludeTags:“tag1,tag2”. Nested exclusions are experimental and subject to change.

Solr 5.4 Features

Here’s an overview of some of the new features in Solr 5.4 Also see Solr Download Links and upcoming Features of the next Solr release.

filter() operator in lucene/solr query syntax

Section titled “filter() operator in lucene/solr query syntax”

A filter query retrieves a set of documents matching a query from the Solr filter cache. This improves performance of additional queries that use the same filter clauses. All documents matching the query produce a score of 0 by default, but this can be changed by specifying a boost.

Filter Query Example:

description:HDTV OR filter(+promotion:tv +promotion_date:[NOW/DAY TO NOW/DAY+7DAY])

docValues fields now take less heap memory

Section titled “docValues fields now take less heap memory”

Multi-valued fields with docValues (as well as binary docValues fields), previously had an on-heap index structure pointing to the on-disk (i.e. off-heap) values. This index has been moved off-heap, and directly read as needed from the index file.

Sparse docValues fields with less than 1% of documents containing a value in the field, are now internally encoded with a new SPARSE_COMPRESSED method to save storage space.

Terms/field faceting has a new parameter called “method” to give an execution hint while faceting on a field.

  • method:uif - Stands for UninvertedField, a method of faceting indexed, multi-valued fields using top-level data structures that optimize for performance over NRT capabilities.
  • method:dv - Stands for DocValues, a method of faceting indexed, multi-valued fields using per-segment data structures. This method mirrors faceting on real docValues fields but works by building on-heap docValues on the fly from the index when docValues aren’t available. This method is better for a quickly changing index.
  • method:stream - This method creates each individual facet bucket (including any sub-facets) on-the-fly while streaming the response back to the requester. Currently only supports sorting by index order.

NOTE: currently, if a field is indexed with docValues, the dv method will be used even if method:uif is specified.

Faceting on DocValue fields and single-valued fields with the new Facet Module (JSON Facet API), has been optimized. When sorting by count, and when there are multiple hits expected per bucket, per-segment ords are collected before being mapped to global ords.

This optimization applies to the following field types:

  • single-valued and multi-valued string fields with docValues
  • indexed single-valued string fields
  • indexed multi-valued string fields, when facet.method=dv is used

Here’s an example of the speedups obtained when faceting over 5M documents on different single-valued fields:

  • Field with 10 unique values: +31%
  • Field with 100 unique values: +29%
  • Field with 1000 unique values: +59%
  • Field with 10000 unique values: +88%
  • Field with 1M unique values: +115%

A collection API command MIGRATESTATEFORMAT that will migrate from an older-style shared clusterstate.json in Zookeeper, to per-collection cluster state files (state.json per collection).

Example:

http://localhost:8983/solr/admin/collections?action=MIGRATESTATEFORMAT&collection=collection1

A HTTP API to CREATE, DELETE, and LIST config sets (schema, solrconfig.xml, etc) in SolrCloud mode.

For example, the following command creates a new configset from an existing configset:

http://localhost:8983/solr/admin/configs?action=CREATE&name=booksConfig&baseConfigSet=genericTemplate

See ConfigSets API in the Solr reference guide for more details.

An expert-level FORCELEADER command has been added to help manually recover from a rare scenario where it looks like there are no replicas for a shard suitable to become the leader (i.e. when all of them are marked as recovering).

See Force Leader in the collections API section of the Solr reference guide.

More complex sorts for collapse post-filter

Section titled “More complex sorts for collapse post-filter”

A new sort parameter was added to the collapse qparser to handle complex sorts.

fq={!collapse field=category sort='popularity desc, score desc'}

See Collapse and Expand Results in the Solr ref guide for more info on using the collapse post-filter.

SolrJ now has support for basic auth credentials. Those credentials need to be set for each request. Example:

QueryRequest solrRequest = new QueryRequest(params);
solrRequest.setBasicAuthCredentials(username, password);
QueryResponse response = solrRequest.process(solrClient, "collection1");

Facet & Analytics Performance

These benchmarks compare the performance of the new JSON Facet API with it’s “performance-first” architecture, and the existing (legacy) Solr Facets.

Test index details:

documents: 5M
index segments: 25
index size: 1.74GB
6 single valued string fields with 10, 100, 1000, 10000, 100000, 1000000 unique values respectively.
6 single valued integer fields as above.
6 multi-valued string fields with 1-5 values per field, with 10, 100, 1000, 10000, 100000, 1000000 unique values respectively.
6 multi-valued integer fields as above.
5% chance of any given field having no values for a particular document.

Test requests details:

Base test query and filters (the domain) matches 2,161,827 documents.
Single client thread (and both requests only use a single internal thread per request).
Single warm-up run per implementation that is discarded.
Multiple runs across all fields, with fastest time being taken for each field.

These benchmarks test faceting on one field and finding the average value in another field per facet bucket.

JSON Facet API command:

json.facet={
f:{
type : terms,
field : m100_5_ss,
facet : { mean : "avg(s10_s)" }
}
}

Legacy Facet command:

facet=true&
stats=true&
stats.field={!tag=stat1+mean=true}s10_s&
facet.pivot={!stats=stat1}m100_5_ss&
f.m100_5_ss.facet.limit=10

Only sorting by count was tested since legacy facets (pivot + stats component) do not support sorting buckets by anything else.

Count Distinct in Solr

A 100% accurate count of distinct values (count distinct) is not generally possible without actually observing all of the values together. However there are a number of ways to estimate the count.

The unique facet function is Solr’s fastest implementation to calculate the number of distinct values. It always provides exact counts on a single Solr node. For distributed search over multiple nodes, it provides exact counts when the number of values per node does not exceed 100 (by default).

When the number of unique values does exceed 100 in any given shard, the following algorithm is used:

  • It estimates the count by sending the top 100 results from each shard along with the total exact “unique” count for each shard.
  • totalSeen is the number of actual results we saw from all shards (i.e. not deduped yet).
  • uniqueSeen is the number of unique values we saw from all shards (i.e. deduped).
  • notSeen is the number of unique values from each shard that were not sent (because of the 100 cutoff).
  • factor = uniqueSeen / totalSeen (i.e. what fraction of values that we saw were unique)
  • estimate = uniqueSeen + ( notSeen * factor ) (i.e. we simply apply the factor to the number of values we didn’t see)

Example use:

$ curl http://localhost:8983/solr/techproducts/query -d '
q=*:*&
json.facet={
x : "unique(manu_exact)" // manu_exact is the manufacturer indexed as a single string
}'

For more facet functions, adding facet functions to each facet bucket, or sorting by facet function, see Solr Facet Functions

(New in Solr 5.2) The HyperLogLog algorithm was developed as an advanced statistical method of estimating the distinct number of values without using too much memory. It does add more calculation overhead however, and is thus slower than Solr’s “unique” facet function.

HyperLogLog requires very high quality hashes for accurate estimation. Solr uses a port of MurmurHash3 for Java to calculate these hashes.

The Facet Analytics Module and the older Stats component both have support for HyperLogLog as of Solr 5.2

A new “hll” facet function was added as an alternative to the existing faster (but less accurate for high cardinality) “unique” function.

To get the unique number of manufacturers using the HyperLogLog algorithm:

$ curl http://localhost:8983/solr/techproducts/query -d '
q=*:*&
json.facet={
x : "hll(manu_exact)" // manu_exact is the manufacturer indexed as a single string
}'

If we want the number of unique manufacturers per bucket of a facet:

$ curl http://localhost:8983/solr/techproducts/query -d '
q=*:*&
json.facet={
categories: {
type : terms,
field : cat,
facet : {
x : "hll(manu_exact)"
}
}
}'

And we get a response containing:

"facets":{
"count":32,
"categories":{
"buckets":[{
"val":"electronics",
"count":12,
"x":9},
{
"val":"currency",
"count":4,
"x":4},
[...]

To get the unique number of manufacturers per facet bucket using the Stats component:

$ curl http://localhost:8983/solr/techproducts/query -d '
q=*:*&
stats=true&
facet=true&
stats.field={!tag=stat1 cardinality=true}manu_exact&
facet.pivot={!stats=stat1}cat'

And we get a response containing:

"facet_counts":{
"facet_queries":{},
"facet_fields":{},
"facet_dates":{},
"facet_ranges":{},
"facet_intervals":{},
"facet_heatmaps":{},
"facet_pivot":{
"cat":[{
"field":"cat",
"value":"electronics",
"count":12,
"stats":{
"stats_fields":{
"manu_exact":{
"cardinality":9}}}},
{
"field":"cat",
"value":"currency",
"count":4,
"stats":{
"stats_fields":{
"manu_exact":{
"cardinality":4}}}},
[...]

Here is a performance comparison of the different implementations in various scenarios.

Test configuration:

Index:
documents: 5M
index segments: 25
index size: 1.74GB
6 single valued string fields with 10, 100, 1000, 10000, 100000, 1000000 unique values respectively.
6 single valued integer fields as above.
6 multi-valued string fields with 1-5 values per field, with 10, 100, 1000, 10000, 100000, 1000000 unique values respectively.
6 multi-valued integer fields as above.
5% chance of any given field having no values for a particular document.
Queries:
Base test query and filters (the domain) matches 2,161,827 documents.
Single client thread (and both requests only use a single internal thread per request).
Single warm-up run per implementation that is discarded.
Multiple runs across all fields, with fastest time being taken for each field.

Legacy Faceting command (stats component + pivot faceting):

facet=true&stats=true&stats.field={!tag=stat1+cardinality=true}s10_s&facet.pivot={!stats=stat1}m100_5_ss&f.m100_5_ss.facet.limit=10

JSON Faceting command (new Facet Module):

json.facet={
f:{
type : terms,
field : m100_5_ss,
facet : { stat1:"hll(s10_s)" }
}
}

This first test facets on the multi-valued string field m100_5_ss (it has up to 5 values per field, and 100 unique values in total). Then for the top 10 buckets, the cardinality of the single-valued string field (the “stat field”) is calculated. The chart below shows performance for different number of unique values in the stat field.

This next test reverses the 2 fields above, first faceting on the single valued string field s100_s (it has 100 unique values in total) and then calculating cardinality over multi-valued string fields with different numbers of unique values in the index.

The last test facets on the integer field s100_i and then calculates cardinality over another integer field with varying number of unique terms in the index.

Solr 5.3 Features

Here’s an overview of some of the new features in Solr 5.3 Also see Solr Download Links and upcoming Features of the next Solr release.

The JSON Facet API can now change the domain for facet commands, essentially doing a block join and moving from parents to children, or children to parents before calculating the facet data.

For example, if you indexed chapters with pages as nested child documents, then you could map from chapters to pages before faceting by adding the following parameter to the facet command:

domain : { blockChildren : "type:chapter" }

Or if you started with pages, you could map to chapters with

domain : { blockParent : "type:chapter" }

Note that in both cases, we provide the parent filter (how parent documents are defined) of “type:chapter” regardless of which direction we are mapping.

See this Nested Objects tutorial for complete examples of combining faceting and block join / nested documents.

 

Major improvements in performance of the new Facet Module / JSON Facet API. See the facet performance benchmarks for more details and benchmark results.  

Just like the JSON Facet API, pivot facets can how nest other facet types such as range and query facets.

Example:

&facet=true
&facet.range={!tag=r1}price
&f.price.facet.range.start=0
&f.price.facet.range.end=100
&f.price.facet.range.gap=10
&facet.query={!tag=q1}popularity:[8 TO 10]
&facet.pivot={!range=r1 query=q1}category

The equivalent in the JSON Facet API would be:

json.facet={
categories : {
type : terms,
field : category,
facet : {
r1 : {
type : range,
start : 0,
end : 100,
gap : 10
},
q1 : { query : "popularity:[8 TO 10]" }
}
}
}

The MoreLikeThis QParser mlt now supports all options provided by the MLT Handler. The query parser is much more versatile than the handler as it works in cloud mode as well as anywhere a normal query can be specified.

Example (on techproducts index):

q={!mlt qf=name mintf=1 mindf=1}SP2514N

More documentation on the mlt parser can be found in the Solr Ref Guide

The new SchemaRequest Java class in SolrJ can be used to make requests to the Schema API.

Also see the Solr Schema API itself in the ref guide.

Scoring mode for query-time join and block join

Section titled “Scoring mode for query-time join and block join”

Solr’s pseudo-join query parser has a new optional attribute score that can be used specify the scores produced on the resulting documents. It’s value can be min, max,avg,or total.

Query-time join example:

q={!join from=author_id to=id score=total}blog_text:awesome

Block join example:

q={!parent of=type:author score=total}blog_text:awesome

See Nested Objects in Solr for more information on nested documents and block join.

Lucene/Solr query syntax (i.e. Solr’s dialect of the lucene syntax) now supports nested C-style comments.

+cat:electronics /* this is a comment */ +name:HDTV

Smile is a binary data interchange format that is very close to Solr’s own “javabin” (encoded sizes are very close). Adding wt=smile to a request will cause the response to come back in this format.

A second parameter has been added to the field function to select the minimum or maximum value of a multi-valued field with docValues.

Example:

sort=field(my_dv_field,max) asc

In addition to many other improvements in the security framework, Solr now includes an AuthenticationPlugin implementing HTTP Basic Auth that stores credentials securely in ZooKeeper. This is a simple way to require a username and password for anyone accessing Solr’s admin screen or APIs.

See the Basic Authentication Plugin section of the Solr ref guide under the Securing Solr section.

JSON Facet API

Related Pages

Solr 5 has a completely re-written faceted search and analytics module with a structured JSON API to control the faceting and analytics commands. NOTE: Some examples use syntax only supported in later Solr 5 releases, or even Solr 6. Download a recent Solr release or snapshot to try them out.

The structured nature of nested sub-facets are more naturally expressed in a nested structure like JSON rather than the flat structure that normal query parameters provide.

Goals of the new Faceting Module:

  • First class JSON support
  • Easier programmatic construction of complex nested facet commands
  • Support a much more canonical response format that is easier for clients to parse
  • First class analytics support
  • Ability to sort facet buckets by any calculated metric
  • Support a cleaner way to do distributed faceting
  • Support better integration with other search features

Of course if you prefer to use Solr’s existing faceting capabilities, that’s fine too. You can even use both at once if you want!

UPDATE: The JSON Facet API is now part of the JSON Request API, so a complete request may be expressed in JSON.

Some of the ease-of-use enhancements over traditional Solr faceting come from the inherent nested structure of JSON. As an example, here is the faceting command for two different range facets using Solr’s flat legacy API:

&facet=true
&facet.range={!key=age_ranges}age
&f.age.facet.range.start=0
&f.age.facet.range.end=100
&f.age.facet.range.gap=10
&facet.range={!key=price_ranges}price
&f.price.facet.range.start=0
&f.price.facet.range.end=1000
&f.price.facet.range.gap=50

And here is the equivalent faceting command in the new JSON Faceting API:

{
age_ranges: {
type : range
field : age,
start : 0,
end : 100,
gap : 10
}
,
price_ranges: {
type : range
field : price,
start : 0,
end : 1000,
gap : 50
}
}

These aren’t even nested facets, but already one can see how much nicer the JSON API looks. With deeply nested sub-facets and statistics, the clarity of the inherently nested JSON API only grows.

A number of JSON extensions have been implemented to further increase the clarity and ease of constructing a JSON faceting command by hand. For example:

{ // this is a single-line comment, which can help add clarity to large JSON commands
/* traditional C-style comments are also supported */
x : "avg(price)" , // Simple strings can occur unquoted
y : 'unique(manu)' // Strings can also use single quotes (easier to embed in another String)
}

Nicely indented JSON is very easy to understand. If you get a large piece of non-indented JSON somehow, and are trying to make sense of it, you can cut and paste into one of the online validators: http://jsonlint.com http://jsonformatter.curiousconcept.com Both of these validators will indent your JSON, even when it contains extensions unsupported by them (such as comments or bare strings).

 

There are two types of facets, one that breaks up the domain into multiple buckets, and aggregations / facet functions that provide information about the set of documents belonging to each bucket.

Faceting can be nested! Any bucket produced by faceting can further be broken down into multiple buckets by a sub-facet.

Statistics are now fully integrated into faceting. Since we start off with a single facet bucket with a domain defined by the main query and filters, we can even ask for statistics for this top level bucket, before breaking up into further buckets via faceting. Example:

json.facet={
x : "avg(price)", // the average of the price field will appear under "x"
y : "unique(manufacturer)" // the number of unique manufacturers will appear under "y"
}

See facet functions for a complete list of the available aggregation functions.

The general form of the JSON facet commands are: <facet_name> : { <facet_type> : <facet_parameter(s)> } Example: top_authors : { terms : { field : authors, limit : 5 } }

After Solr 5.2, a flatter structure with a “type” field may also be used: <facet_name> : { "type" : <facet_type> , <other_facet_parameter(s)> } Example: top_authors : { type : terms, field : authors, limit : 5 }

The results will appear in the response under the facet name specified. Facet commands are specified using json.facet request parameters.

To test out different facet requests by hand, it’s easiest to use “curl” from the command line. Example:

$ curl http://localhost:8983/solr/query -d 'q=*:*&rows=0&
json.facet={
categories:{
type : terms,
field : cat,
sort : { x : desc},
facet:{
x : "avg(price)",
y : "sum(price)"
}
}
}
'

 

The terms facet, or field facet, produces buckets from the unique values of a field. The field needs to be indexed or have docValues.

The simplest form of the terms facet

{
top_genres : { terms : genre_field }
}

An expanded form allows for more parameters:

{
top_genres : {
type : terms,
field : genre_field,
limit : 3,
mincount : 2
}
}

Example response:

"top_genres":{
"buckets":[
{
"val":"Science Fiction",
"count":143},
{
"val":"Fantasy",
"count":122},
{
"val":"Biography",
"count":28}
]
}

Parameters:

  • field - The field name to facet over.

  • offset - Used for paging, this skips the first N buckets. Defaults to 0.

  • limit - Limits the number of buckets returned. Defaults to 10.

  • mincount - Only return buckets with a count of at least this number. Defaults to 1.

  • sort - Specifies how to sort the buckets produced. “count” specifies document count, “index” sorts by the index (natural) order of the bucket value. One can also sort by any facet function / statistic that occurs in the bucket. The default is “count desc”. This parameter may also be specified in JSON like sort:{count:desc}. The sort order may either be “asc” or “desc”

  • missing - A boolean that specifies if a special “missing” bucket should be returned that is defined by documents without a value in the field. Defaults to false.

  • numBuckets - A boolean. If true, adds “numBuckets” to the response, an integer representing the number of buckets for the facet (as opposed to the number of buckets returned). Defaults to false.

  • allBuckets - A boolean. If true, adds an “allBuckets” bucket to the response, representing the union of all of the buckets. For multi-valued fields, this is different than a bucket for all of the documents in the domain since a single document can belong to multiple buckets. Defaults to false.

  • prefix - Only produce buckets for terms starting with the specified prefix.

  • method - Provides an execution hint for how to facet the field.

    • method:uif - Stands for UninvertedField, a method of faceting indexed, multi-valued fields using top-level data structures that optimize for performance over NRT capabilities.
    • method:dv - Stands for DocValues, a method of faceting indexed, multi-valued fields using per-segment data structures. This method mirrors faceting on real docValues fields but works by building on-heap docValues on the fly from the index when docValues aren’t available. This method is better for a quickly changing index.
    • method:stream - This method creates each individual facet bucket (including any sub-facets) on-the-fly while streaming the response back to the requester. Currently only supports sorting by index order.

 

The query facet produces a single bucket that matches the specified query.

An example of the simplest form of the query facet

{
high_popularity : { query : "popularity:[8 TO 10]" }
}

An expanded form allows for more parameters (or sub-facets / facet functions):

{
high_popularity : {
type : query,
q : "popularity:[8 TO 10]",
facet : { average_price : "avg(price)" }
}
}

Example response:

"high_popularity" : {
"count" : 147,
"average_price" : 74.25
}

 

The range facet produces multiple range buckets over numeric fields or date fields.

Range facet example:

{
prices : {
type : range,
field : price,
start : 0,
end : 100,
gap : 20
}
}

Example response:

"prices":{
"buckets":[
{
"val":0.0, // the bucket value represents the start of each range. This bucket covers 0-20
"count":5},
{
"val":20.0,
"count":3},
{
"val":40.0,
"count":2},
{
"val":60.0,
"count":1},
{
"val":80.0,
"count":1}
]
}

To ease migration, these parameter names, values, and semantics were taken directly from the old-style (non JSON) Solr range faceting.

Parameters:

  • field - The numeric field or date field to produce range buckets from

  • mincount - Minimum document count for the bucket to be included in the response. Defaults to 0.

  • start - Lower bound of the ranges

  • end - Upper bound of the ranges

  • gap - Size of each range bucket produced

  • hardend - A boolean, which if true means that the last bucket will end at “end” even if it is less than “gap” wide. If false, the last bucket will be “gap” wide, which may extend past “end”.

  • other - This param indicates that in addition to the counts for each range constraint between facet.range.start and facet.range.end, counts should also be computed for…

  • "before" all records with field values lower then lower bound of the first range

  • "after" all records with field values greater then the upper bound of the last range

  • "between" all records with field values between the start and end bounds of all ranges

  • "none" compute none of this information

  • "all" shortcut for before, between, and after

  • include - By default, the ranges used to compute range faceting between facet.range.start and facet.range.end are inclusive of their lower bounds and exclusive of the upper bounds. The “before” range is exclusive and the “after” range is inclusive. This default, equivalent to lower below, will not result in double counting at the boundaries. This behavior can be modified by the facet.range.include param, which can be any combination of the following options…

  • "lower" all gap based ranges include their lower bound

  • "upper" all gap based ranges include their upper bound

  • "edge" the first and last gap ranges include their edge bounds (ie: lower for the first one, upper for the last one) even if the corresponding upper/lower option is not specified

  • "outer" the “before” and “after” ranges will be inclusive of their bounds, even if the first or last ranges already include those boundaries.

  • "all" shorthand for lower, upper, edge, outer

Parameters that all faceting methods have in common include

Solr Facet Functions and Analytics

Traditional faceted search (also called guided navigation) involves counting search results that belong to categories (also called facet constraints). The new facet functions in Solr extends normal faceting by allowing additional aggregations on document fields themselves. Combined with the new Sub-facet feature, this provides powerful new realtime analytics capabilities. Also see the page about the new JSON Facet API.

Faceting involves breaking up the domain into multiple buckets and providing information about each bucket. There are multiple aggregation functions / statistics that can be used:

Aggregation Example Effect
sum sum(sales) summation of numeric values
avg avg(popularity) average of numeric values
sumsq sumsq(rent) sum of squares
min min(salary) minimum value
max max(mul(price,popularity)) maximum value
unique unique(state) number of unique values (count distinct)
hll hll(state) number of unique values using the HyperLogLog algorithm
percentile percentile(salary,50,75,99,99.9) calculates percentiles
stddev stddev(salary) calculates standard deviation (Solr6.6+)
variance variance(salary) calculates variance (Solr 6.6+)

  Numeric aggregation functions such as avg can be on any numeric field, or on another function of multiple numeric fields.

See Count Distinct in Solr for more information on distributed cardinality estimation / calcDistinct.

 

The faceting domain starts with the set of documents that match the main query and filters. We can ask for statistics over this whole set of documents:

http://localhost:8983/solr/query?q=*:*&
json.facet={x:'avg(price)'}

And the response will contain a facets section:

[...]
"facets":{
"count":32,
"x":164.10218846797943
}
[...]

  If we want to break up the domain into buckets and then calculate a function per bucket, we simply add a nested facet command to the facet parameters. For example (using curl this time):

$ curl http://localhost:8983/solr/query -d 'q=*:*&
json.facet={
categories:{
type : terms, // terms facet creates a bucket for each indexed term (or value) in the field
field : cat,
facet:{
x : "avg(price)",
y : "sum(price)"
}
}
}
'

The response will contain the two stats we asked for in each category bucket.

[...]
"facets":{
"count":32,
"categories":{
"buckets":[
{
"val":"electronics",
"count":12,
"x":231.02666823069254,
"y":2772.3200187683105
},
{
"val":"memory",
"count":3,
"x":86.66333262125652,
"y":259.98999786376953
},
[...]

 

The default sort for a field or terms facet is by bucket count descending. We can optionally sort ascending or descending by any facet function that appears in each bucket. For example, if we wanted to find the top buckets by average price, then we would add sort:"x desc" to the previous facet request:

$ curl http://localhost:8983/solr/query -d 'q=*:*&
json.facet={
categories:{
type : terms,
field : cat,
sort : "x desc", // can also use sort:{x:desc}
facet:{
x : "avg(price)",
y : "sum(price)"
}
}
}
'

 

Facet functions and Subfacets are currently only in Solr 5.1. Download the latest release and give it a spin!

Solr Subfacets

Subfacets (also called Nested Facets) is a more generalized form of Solr’s current pivot faceting that allows adding additional facets for every bucket produced by a parent facet.

Subfacet advantages over pivot faceting:

  • Subfacets work with facet functions (statistics), enabling powerful real-time analytics
  • Can add a subfacet to any facet type (field, query, range)
  • A subfacet can be of any type (field/terms, query, range)
  • A given facet can have multiple subfacets
  • Just like top-level facets, each subfacet can have it’s own configuration (i.e. offset, limit, sort, stats)

Subfacets are part of the new Facet Module, and are naturally expressed in the JSON Facet API. Every facet command is actually a sub-facet since there is an implicit top-level facet bucket (the domain) defined by the documents matching the main query and filters. Simply add a facet section to the parameters of any existing facet command.

For example, a terms facet on the “genre” field looks like:

top_genres:{
type: terms,
field: genre,
limit: 5
}

Now if we wanted to add a subfacet to find the top 4 authors for each genre bucket:

top_genres:{
type: terms,
field: genre,
limit: 5,
facet:{
top_authors:{
type: terms,
field: author,
limit: 4
}
}
}

Assume we want to do the following complex faceting request:

  • Facet on the “genre” field and find the top buckets
  • For ever “genre” bucket generated above, find the top 7 authors
  • For ever “genre” bucket, create a bucket of high popularity items (defined by popularity 8 - 10) and call it “highpop”
  • For ever “highpop” bucket generated above, find the top 5 publishers

In short, this request finds the top authors for each genre and finds the the top publishers for high popularity books in each genre. Using the JSON Facet API, the full request (using curl) would look like the following:

$ curl http://localhost:8983/solr/query -d 'q=*:*&
json.facet=
{
top_genres:{
type: terms,
field: genre,
facet:{
top_authors: {
type : terms, // nested terms facet
field: author,
limit: 7
},
highpop:{
type : query, // nested query facet
q: "popularity:[8 TO 10]", // lucene query string
facet:{
publishers:{
type: terms, // nested terms facet under the nested query facet
field: publisher,
limit: 5
}
}
}
}
}
}
'

An example response would look like the following:

[...]
"facets":{
"top_genres":{
"buckets":[{
"val":"Fantasy",
"count":5432,
"top_authors":{ // these are the top authors in the "Fantasy" genre
"buckets":[{
"val":"Mercedes Lackey",
"count":121},
{
"val":"Piers Anthony",
"count":98}]}},
"highpop":{ // bucket for books in the "Fantasy" genre with popularity between 8 and 10
"count":876
"publishers":{ // top publishers in this bucket (highpop fantasy)
"buckets":[{
"val":"Bantam Books",
"count":346},
{
"val":"Tor",
"count":217}]}},
{
"val":"Science Fiction", // the next genre bucket
"count":4188,
[...]

  All the reporting and sorting was done using document count (i.e. number of books). If instead, we wanted to find top authors by total revenue (assuming we had a “sales” field), then we could simply change the author facet from the previous example as follows:

top_authors:{
type: terms,
field: author,
limit: 7,
sort: "revenue desc",
facet:{
revenue: "sum(sales)"
}
}

 

Facet functions and Subfacets are in Solr 5.1 and later, but the syntax used on this page requires Solr 5.3 or later. Download the latest release and give it a spin!

Solr 5.1 Features

Solr 5.1 has been released! Here’s an overview of how to use some of the new features.

Also see Solr download links and upcoming features of the next Solr release.

The new facet module has a native JSON Facet API, first-class support for statistics and analytics via facet functions (aggregations), and supports unlimited nesting of facets within other facets via sub-facets.

One can calculate statistics such as averages, number of unique values (distinct values), and percentiles over each facet bucket (groups of documents), and even sort facet buckets by any calculated metrics.

A JSON Request API that allows passing a full Solr query request in JSON.

Example:

curl http://localhost:8983/solr/query -d '
{
query : "*:*",
filter : [
"author:brandon",
"genre_s:fantasy"
],
offset : 0,
limit : 5,
fields : ["title","author"], // we could also use the string form "title,author"
sort : "sequence_i desc",
facet : { // the JSON Facet API is nicely integrated as well
avg_price : "avg(price)",
median_price : "percentile(price,50)",
top_authors : {terms : author}
}
}'

Parameter substitution is now done across the entire query request. It supports default values, multiple levels of indirection, and it even works within the body of a JSON request. This can also be viewed as a powerful form of request templates.

Example:

q=price:[ ${low} TO ${high} ]
&low=100
&high=200

Parameters can also be passed in the params block of a JSON request.

Syntax within the standard lucene/solr query parser for constant score queries quit the general form of ^=<constant_score>. Think of a query boost with ^ replaced with ^=. Example:

q=(color:blue color:green)^=2.0 text:shoes

There is a new general purpose parallel computing framework for SolrCloud. The Streaming API is (currently) a Java API that can do streaming aggregations (like sum and average) and streaming transformations (like group-by and join).

The admin UI can show segment info such as size, number of docs, and number of deletions for each segment in the index. For the “demo” collection, simply point your browser at http://localhost:8983/solr/#/demo/segments Or click on the “Segments Info” link in the admin UI after you select the core/collection you are interested in. segments_info

The bulk schema API how has the ability to replace or remove fields, fieldTypes, dynamic fields, and copy fields.

Example of adding a field (this was already possible):

curl http://localhost:8983/solr/demo/schema -d '
{
"add-field":{
"name" : "powerLevel",
"type" : "int",
"indexed" : true,
"stored" : true
}
}'

Now we can replace the field definition:

curl http://localhost:8983/solr/demo/schema -d '
{
"replace-field":{
"name" : "powerLevel",
"type" : "int",
"indexed" : false,
"stored" : true
}
}'

We can verify that Solr now has the updated field definition with

curl http://localhost:8983/solr/demo/schema/fields/powerLevel

And solr returns:

"field":{
"name":"powerLevel",
"type":"int",
"indexed":false,
"stored":true}

And lastly, we can delete the field definition with

curl http://localhost:8983/solr/demo/schema -d '
{
"delete-field":{ "name" : "powerLevel" }
}'

Solr can now execute a two dimensional facet on RPT field types (Spatial Recursive Prefix Tree).

Parameters Example:

q=*:*
&facet=true
&facet.heatmap=location_rpt
&facet.heatmap.geom=["-180 -90" TO "180 90"]
&facet.heatmap.gridLevel=6
&facet.heatmap.distErrPct=0.15
&facet.heatmap.format=ints2D

The facet.heatmap.format=ints2D parameter causes a 2D array of counts to be returned:

{
"counts_ints2D":[[4, 0, 1, 3, ....],[2, 0, 1, 2, ...],...]
}

If facet.heatmap.format=png is passed instead, a basic base64-encoded PNG (picture) will be returned of the heatmap grid.

There is now an explicit API in SolrJ to use Real-time Get

HttpSolrClient client = new HttpSolrClient("http://localhost:8983/solr/demo");
SolrDocument sdoc = client.getById("book1");
System.out.println("I found book " + sdoc);
client.close(); // shut down the client when we are done

StatsComponent Enable/disable individual stats

Section titled “StatsComponent Enable/disable individual stats”

Localparams may now be used to selectively enable or disable specific stats in the StatsComponent. Example: stats.field={!min=true max=true}field_name

Both the new facet module and the stats component gained support for percentiles.

json.facet={ median_age : "percentile(age,50)" }
stats.field={!percentiles='50'}age

Many additional configuration items can now be managed via the Config API. This includes managing named components such as requestHandler, queryParser, queryResponseWriter, valueSourceParser, transformer, and queryConverter.

Changes do not directly change solrconfig.xml, but instead are reflected in configoverlay.json which override settings in solrconfig.xml.

Upload config sets to zookeeper with CloudSolrClient

Section titled “Upload config sets to zookeeper with CloudSolrClient”

Named config sets (schema.xml, solrconfig,xml, etc) are referenced by name when creating new collections in SolrCloud. These config sets may now be uploaded and downloaded via SolrJ to and from the local filesystem. The following methods were added to CloudSolrClient:

public void uploadConfig(Path configPath, String configName);
public void downloadConfig(String configName, Path downloadPath);

There is a new API to add a jar to a collection’s classpath (as well as update and delete a jar). Components that depend on such a jar should have a new attribute called runtimeLib set to true since a separate classloader is used for these jars.

Example of uploading a jar:

curl http://localhost:8983/solr/demo/config -d '{
"add-runtimelib" : {"name": "jarname" , "version":2 }
}'

Example registering a new value source parser using a class in the jar:

curl http://localhost:8983/solr/demo/config -d '{
"create-valuesourceparser" : {
"name": "nvl",
"runtimeLib" : true,
"class" : "solr.org.apache.solr.search.function.NvlValueSourceParser ,
"nvlFloatValue" : 0.0
}
}'

Solr 5.2 Features

Here’s an overview of some of the new features in Solr 5.2 Also see Solr download links and upcoming features of the next Solr release.

Caches using the LRUCache implementation can specify a new parameter maxRamMB that will evict based on RAM use rather than number of elements in the cache. Least recently used items are evicted until the RAM use is brought under the limit. RAM use calculations do not currently cover the cache keys, so using this for the query cache and caching large queries can still lead to greater memory use than expected.

To make a backup, we can send a request to the replication handler:

curl -XPOST "http://localhost:8983/solr/demo/replication?command=backup&name=my_backup100"

This will create a backup of the index in Solr’s data directory (this can be changed via the location parameter) named snapshot.my_backup100

This index snapshot can later be restored with the following command:

curl -XPOST "http://localhost:8983/solr/demo/replication?command=restore&name=my_backup100"

Flatter request structure for the JSON Facet API

Section titled “Flatter request structure for the JSON Facet API”

Here’s an example of a terms facet in Solr 5.1:

top_authors : { terms : {
field : author,
limit : 5,
}}

In the Solr 5.2 JSON Facet API, the “type” can optionally be specified in the same object as the facet arguments:

top_authors : {
type : terms,
field : author,
limit : 5
}

Range facets now support the mincount parameter to screen out range facet buckets that don’t meet a minimum document count.

prices:{
type:range,
field:price,
mincount:1,
start:0, end:100, gap:10
}

The unique facet function now works on numeric and date fields. Example:

json.facet={
num_codes : "unique(error_code)"
}

Multi-select faceting is a powerful faceting style that allows users to see and select multiple facet constraints (facet values) for a facet. For example, one may want to select multiple price ranges or multiple colors they are interested in.

The new Facet Analytics Module / JSON Facet API now supports multi-select faceting via filter exclusions. A new excludeTags parameter will disregard any top-level filters with matching tags.

Here’s a Multi-Select Faceting Example, using the JSON Facet API.

Both the older Stats component and the new Facet Analytics Module have added support for HyperLogLog based statistical cardinality estimate. For the JSON Facet API, a new hll facet function was added as an alternative to the existing faster (but less accurate for high cardinality) unique function. Example:

json.facet={ numProducts : "hll(product_id)" }

See Solr Count Distinct functionality for examples that calculate the number of distinct values in a given field per facet bucket.

“facet.range.method” (traditional query-parameter API)

Section titled ““facet.range.method” (traditional query-parameter API)”

Add a new “facet.range.method” parameter to let users choose how to do range faceting between an implementation based on filters (previous algorithm, using “facet.range.method=filter”) or DocValues (“facet.range.method=dv”). Input parameters and output of both methods are the same.

If you have a field value that consists of well formed XML or JSON, you can return those raw values in the appropriate response writer. Example: ?fl=id,name,json_s:[json],xml_s:[xml]

This new SolrCloud feature allows the specification of rules which govern placement of replicas in the cluster. Rules are specified during collection creation and persisted in zookeeper.

See the blog post from LucidWorks for further details and examples.

Solr Streaming Expressions adds an expression based interface to the Streaming API added in Solr 5.1.

Some examples from include

// merge two distinct searches together on common fields
merge(
search(collection1, q="id:(0 3 4)", fl="id,a_s,a_i,a_f", sort="a_f asc, a_s asc"),
search(collection2, q="id:(1 2)", fl="id,a_s,a_i,a_f", sort="a_f asc, a_s asc"),
on="a_f asc, a_s asc")
// find top 20 unique records of a search
top(
n=20,
unique(
search(collection1, q=*:*, fl="id,a_s,a_i,a_f", sort="a_f desc"),
over="a_f desc"),
sort="a_f desc")

See the Solr Reference Guide for more documentation.

An authentication framework and Kerberose authentication module. See the Security section of the Solr Reference Guide.

Percentiles for Solr Faceting

The percentile aggregation function was just added to the new Solr Facet Module. This allows one to calculate one or more percentiles for each facet bucket (i.e. each group of documents produced by faceting), and even sort facet buckets by any given percentile.

The percentile aggregation even works with distributed search! The algorithm used is Ted Dunnings “t-digest”, which gives good approximations with relatively little memory consumption.

NOTE: requires Solr 5.3 or later.

First, let’s start Solr and create a “demo” collection.

$ bin/solr start
$ bin/solr create -c demo
# HINT: use "bin/solr stop -all" when you're finished.

Now, lets index some salary survey data in CSV format, using dynamic fields:

$ curl http://localhost:8983/solr/demo/update?commitWithin=5000 -H 'Content-type:text/csv' -d '
id,gender_s,loc_s,year_i,job_s,salary_d
mark,M,NJ,2011,clerk,21250
john,M,NY,2011,engineer,42500
mary,F,CT,2015,manager,87299
alice,F,NJ,2013,dentist,75000
mike,M,NY,2012,sales,59500
nancy,F,CT,2014,engineer,110000
greg,M,NJ,2012,manager,74000
cindy,F,NJ,2012,engineer,81000
janet,F,NJ,2015,clerk,30150
joe,M,NY,2014,dentist,74000
luke,M,CT,2015,dentist,78000
zoe,F,NY,2013,manager,89500
eli,M,CT,2011,sales,66000
anna,F,CT,2012,sales,59500
evan,M,NY,2014,clerk,2920
'

Now we can use Solr’s analytics / facet functions to slice and dice our data!

Let’s say we want the 25%, 50%, and 75% percentile salaries across all our jobs:

$ curl http://localhost:8983/solr/demo/query -d 'q=*:*&json.facet={salary_percentiles:"percentile(salary_d,25,50,75)"}'

And at the end of our response, we’ll get our facet results:

[...]
"facets" : {
"count" : 15,
"salary_percentiles" : [51000.0, 74000.0, 79500.0]
}
}

  We can add in other statistics such as the average salary, the number of different jobs, and the number of different states in our salary survey:

$ curl http://localhost:8983/solr/demo/query -d 'q=*:*&
json.facet={
average_salary : "avg(salary_d)",
num_jobs : "unique(job_s)",
num_states : "unique(loc_s)",
salary_percentiles : "percentile(salary_d,25,50,75)"
}'
"facets":{
"count":15,
"average_salary":63374.6,
"num_jobs":5,
"num_states":3,
"salary_percentiles":[51000.0,74000.0,79500.0]
}

  Now let’s take a look at median salary broken out by gender:

$ curl http://localhost:8983/solr/demo/query -d 'q=*:*&
json.facet={
by_gender:{
type:terms
field:gender_s,
facet:{
median_salary:"percentile(salary_d,50)"
}
}
}'
"facets":{
"count":15,
"by_gender":{
"buckets":[
{
"val":"M",
"count":8,
"median_salary":62750.0
},
{
"val":"F",
"count":7,
"median_salary":81000.0
}
]
}
}

  We can also sort by a percentile statistic. If you request more than one percentile value, the sort will be on the first value in the list requested. Let’s find the top states by 99.9th percentile salary:

$ curl http://localhost:8983/solr/demo/query -d 'q=*:*&
json.facet={
rich_states:{
type : terms,
field : loc_s,
sort : {sal:desc}, // specifying the sort as a string, like sort:"sal desc" will also work
facet : {
sal : "percentile(salary_d,99.9)"
}
}
}'
"facets":{
"count":15,
"rich_states":{
"buckets":[{
"val":"CT",
"count":5,
"sal":109909.19600000001},
{
"val":"NY",
"count":5,
"sal":89438.00000000001},
{
"val":"NJ",
"count":5,
"sal":80976.0}]}}

  We can get even more interesting by nesting facets. How about finding the highest earning occupation (99.9th percentile) for every state?

$ curl http://localhost:8983/solr/demo/query -d 'q=*:*&
json.facet={
states:{
type:terms,
field:loc_s,
facet:{
top_jobs:{ // nested terms facet
type : terms,
field : job_s,
sort : "sal desc", // sort will be on first percentile (99.9)
limit : 1, // only show top occupation
facet:{
sal : "percentile(salary_d,99.9,50,10)"
}
}
} // end facet block for the loc_s field
}
}'

The response has been omitted since we don’t have enough data for it to be interesting.

  We can also show how median salary has changed over time for each individual state:

$ curl http://localhost:8983/solr/demo/query -d 'q=*:*&
json.facet={
states:{
type:terms,
field:loc_s,
facet:{
over_time:{ // nested range facet
type : range,
field : year_i,
start : 2011,
end : 2015,
gap : 1,
facet:{
median_salary : "percentile(salary_d,50)"
}
}
} // end facet block for the loc_s field
}
}'
"facets":{
"count":15,
"states":{
"buckets":[{
"val":"CT",
"count":5,
"over_time":{
"buckets":[{
"val":2011,
"count":1,
"median_salary":66000.0},
{
"val":2012,
"count":1,
"median_salary":59500.0},
{
"val":2013,
"count":0},
{
"val":2014,
"count":1,
"median_salary":110000.0}]}},
{
"val":"NJ",
"count":5,
"over_time":{
"buckets":[{
"val":2011,
"count":1,
"median_salary":21250.0},
{
"val":2012,
"count":2,
"median_salary":77500.0},
{
"val":2013,
"count":1,
"median_salary":75000.0},
{
"val":2014,
"count":0}]}},
{
"val":"NY",
"count":5,
"over_time":{
"buckets":[{
"val":2011,
"count":1,
"median_salary":42500.0},
{
"val":2012,
"count":1,
"median_salary":59500.0},
{
"val":2013,
"count":1,
"median_salary":89500.0},
{
"val":2014,
"count":2,
"median_salary":38460.0}]}}]}}

Noggit, the JSON Streaming Parser

Noggit is the world’s fastest streaming JSON parser for Java.

Section titled “Noggit is the world’s fastest streaming JSON parser for Java.”

Noggit is the streaming JSON parser used in Solr. It lives here on github.

Noggit supports a number of extensions to the JSON grammar. All of these extensions are optional and may be disabled.

{ // This is a single line comment
# This is also a single line comment
/* This is a multi-line
* C-style comment.
*/
}
{
first : Yonik,
last : Seeley
}

JSON strings are normally encapsulated by double quotes. It’s often desirable to use single quotes if for example you are embedding some JSON in another double quoted string in a program.

['how', 'now', 'brown', 'cow']

Sometimes one may not know exactly what characters need to be backslash escaped. It can be useful to accept this without throwing an exception.

'This is just a " string'

Allowing trailing commas or extra commas can make it easier to produce JSON that doesn’t throw a parse exception. One use-case is templating JSON. Given the following template,

{
filters:["instock:true", ${FILT1}]
} # Note: templating is not part of JSON or Noggit... but may happen before parsing.

If FILT1 is not defined and is replaced with empty space, this results in the following JSON:

{
filters:["instock:true", ] // this will be parsed as filters:["instock:true"]
}

Noggit ignores all extra commas, not just trailing commas:

[
[,] // equivalent to []
, {,} // equivalent to {}
, [,,3,,,6,,] // equivalent to [3,6]
]

Large string values can optionally be handled in a streaming fashion a piece at a time. Noggit will only construct a single String object in memory if asked. This allows for stream processing with very little memory overhead.

{
"big_string" : "A very large string... pretend its's 1GB in size... we can process it and send it on without reading it all into memory at once!"
}

Noggit can handle huge values that are JSON compliant but may be too large to be parsed into a Java primitive.

{
"big_int" : 1234567890987654321334325343534535342325786237862578625725867258672356711107,
"big_float" : 112412133377778226524562431234215423.23421434645743234564758453322342,
"big_sci" : 2.342669039282149050282364845982748592e-94321
}

Noggit can also handle multiple JSON values streamed over a single connection and simply catenated together. Primitive values should of course be separated by whitespace to avoid ambiguity.

{first_object:10}
['another array object']['yet another object']
{more:objects}{another:object}
['who knows how many json values will be streamed by the writer...']
42
"is this the end?"

Noggit can parse huge JSON messages with minimal overhead.

  • A single byte of state needed per nested object or array. This is needed to keep track of the type of enclosing entity.
  • A user can optionally provide an input buffer for Noggit to use when parsing from a Reader, allowing re-use across different parsers and thus lower memory consumption and garbage collection activity.
  • Streaming values: very large values (such as strings) can be obtained in chunks, thus the whole value never needs to reside in memory at once.

Switching between Java7 and Java8 in Lucene/Solr

Lucene/Solr trunk (the future 6.0 release) is now on Java8, while version 5.x is still on Java7. Linux and Windows allows one to install a JDK any place in the filesystem, and I use the convention of installing in /opt/jdk7 and /opt/jdk8. Things are a little more difficult on Mac OS-X however, as you can’t chose the install location. Luckily there is a command called java_home to show you where a JDK is installed.

Here’s a snippet from my .profile to help manage working with different java versions:

Terminal window
OS=`uname`
case "$OS" in
CYGWIN*)
OS=cygwin
OPT=c:/opt
;;
*)
OPT=/opt
;;
esac
set-java () {
export JAVA_HOME="$*"
if [ $OS = "cygwin" ]; then
export PATH="`cygpath $JAVA_HOME/bin`:$PATH"
else
export PATH="$JAVA_HOME/bin:$PATH"
fi
}
if [ $OS = "Darwin" ]; then
JAVA7=`/usr/libexec/java_home -v 1.7`
JAVA8=`/usr/libexec/java_home -v 1.8`
else
JAVA7=$OPT/jdk7
JAVA8=$OPT/jdk8
fi
set-java $JAVA8

Now, if I switch from working on trunk to working on Lucene 5 or Solr 5, I can easily switch the default JDK for a single terminal via the set-java shell function.

Terminal window
/opt/heliosearch$ java -version
java version "1.8.0_25"
Java(TM) SE Runtime Environment (build 1.8.0_25-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)
/opt/heliosearch$ set-java $JAVA7
/opt/heliosearch$ java -version
java version "1.7.0_71"
Java(TM) SE Runtime Environment (build 1.7.0_71-b14)
Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode)
/opt/heliosearch$

Solr Terms Query for matching many terms

Solr 4.10 and Heliosearch .07 have added a terms query (or terms filter) to more efficiently match many terms in a single field. A large number of terms are often useful for things like access control lists or security filters. Previously, the only way to do this was a large boolean query with many clauses, which has unnecessary overhead when scoring is not needed.

Solr’s implementation uses Lucene’s TermFilter class, as does Elasticsearch’s terms filter.

The Heliosearch terms query implementation has some additional features:

  • prefix compression including off-heap construction
  • direct creation of off-heap filter for faster execution and less garbage production
  • native code bit-setting
  • ability to skip sorting the terms if desired

For reference, specifying a filter query (fq) in the normal lucene syntax via a boolean query looks like the following (assumes default boolean operator of OR):

fq=id:doc334 id:doc125 id:doc777 id:doc321 id:doc253

or in a more compact form, like:

fq=id:(doc334 doc125 doc777 doc321 doc253)

Be aware that going over the limit of 1024 terms in Solr will cause an exception by default. Heliosearch has no such limit.

The corresponding new terms query in both Solr and Heliosearch is:

fq={!terms f=id}doc334,doc125,doc777,doc321,doc253

Performance of terms queries is shown relative to using a Boolean query in Solr. For example the last column in the first chart represents a 10 term filter that matches 10,000,000 documents (1 million per term). The request execution time is:

  • 381,342 microseconds with a Solr Boolean Querty
  • 122,119 microseconds with a Solr Terms Query
  • 67,075 microseconds with a Heliosearch Terms Query

Benchmark details:

  • 10M document index
  • 64 bit Java 1.8.0_20 Oracle JDK
  • Windows 8 64 bit, quad-core Intel i5-3570K @ 3.4GHz
  • Request time was measured externally and includes the entire request time, including the time for the client to send the request and read the response.
  • Solr versions: Apache Solr 4.10.0, Heliosearch 0.07 (based on Solr 4.10)

  The first set of tests consist of 10 term queries that match various number of documents.: terms_perf_10

  The next set of tests deal with 100 term queries that match various number of documents: terms_perf_100

  And finally the last test deals with term queries on the id field (i.e. each term matches a single document): terms_perf_ids

The first performance tests were run multiple times and the amount of garbage produced was recorded. terms_perf_memory

The Heliosearch off-heap optimizations clearly pay dividends here, resulting in much less heap usage, less garbage production (which will mean less garbage collection work), and a smaller process size.

Native Code Faceting

Native code faceting for Solr has just been added to Heliosearch, and benchmarks show an impressive 2x performance increase! This is faceting code written in C++ and statically compiled for maximum performance, and loaded into the JVM via JNI (Java Native Interface).

nCache, Heliosearch’s off-heap version of the Lucene/Solr FieldCache, was instrumental in allowing this level of optimization. Java arrays (and other on-heap memory) cannot be efficiently accessed from native code. Moving the data structures off-heap not only provided great decreases in garbage collection overhead, but also allowed for practical native code optimizations. Top-level nCache string support was recently added, paving the way for native code faceting on single valued string fields.

Benchmark details:

  • 10M document index
  • Documents consist of an ID field, and 6 different single-valued string fields with varying numbers of unique values ranging from 10 to 1 Million
  • Faceting request throughput was measured for 1000 requests after 50 request warmup.
  • Client had 4 request threads
  • Each individual client request uses a random field to make the test realistic and to avoid hotspot overspecializing the code for a specific field.
  • Solr versions: Apache Solr 4.8.1, Heliosearch/Solr snapshot (based on Solr 4.9)

native_faceting_perf

  The different operating systems were run on different hardware (hence the large performance differences of the same code across the different platforms).

OS CPU Native code compiler performance vs solr
Ubuntu Linux 13.10 quad core AMD Phenom II gcc 4.7.3 227%
Windows 8.1 quad core Intel i5 gcc 4.8.2 202%
OS-X Mavericks 10.9.3 dual core Intel i5 gcc 4.8.2 246%
OS-X Mavericks 10.9.3 dual core Intel i5 LLVM 5.1 235%

 

The gcc/g++ included with OS-X is actually clang/LLVM - clang is the C language front end and LLVM is the back end that produces executable code. At least for this initial native code, g++ 4.8.2 was about 5% faster than clang/LLVM 5.1, hence we’ll most likely use gcc/g++ by default. The easiest way to get gcc/g++ on your Mac is

$ brew install gcc48

After installation, gcc/g++ will continue pointing to the clang/LLVM versions, but there will be gcc-4.8 and g++-4.8 you can use in /usr/local/bin.

 

Besides the incredible over 2x faceting performance improvement, native code has other advantages as well:

  • Avoidance of Java hotspot bugs in compiling code. Compiling the code just once statically means it’s the same for every run, for everyone.
  • No variations from run-to-run due to how hotspot compiles the code (unexplained slowdowns).
  • No hotspot warm-up period, or time spent optimizing, or de-optimizing code.

It’s easy to take advantage of these performance improvements and new features since Heliosearch/Solr is currently a drop-in replacement (at the HTTP-API level) for Apache Solr. Download the latest release and try it out.

We’d love to hear how it’s working for you… drop by the user mailing list and let us know. Want to dabble in C/C++ code again? Drop by the dev mailing list to help out with development!

Solr Cross Data Center Replication

Solr needs a flexible cross-datacenter architecture that can handle both a variety of application needs as well as a variety of infrastructure resources.

  • Accommodate 2 or more data centers
  • Accommodate active/active uses
  • Accommodate limited band-with cross-datacenter connections
  • Minimize coupling between peer clusters to increase reliability
  • Support both full consistency and eventual consistency

Issues with running SolrCloud cross data center

Section titled “Issues with running SolrCloud cross data center”

Running a single SolrCloud cluster across two data centers can be done, but has multiple drawbacks:

  • Same update is forwarded multiple times (once per replica) over the bandwidth limited cross-DC pipe.
  • Can’t implement extra compression or security
  • Requires a 3rd data center to contain a zookeeper node for tie-breaking
  • Burst indexing limited by cross-DC bandwidth.
  • Lack of a true backup cluster if the cluster gets into a bad state
  • Extra latency for all indexing operations
  • Search requests are not data center aware (extra latency + bandwidth)
  • Normal recovery mechanism (full index copy) may not be viable across DC connections

Clusters will be configured to know about each other, most likely through keeping a cluster peer list in zookeeper. One essential piece of information will be the zookeeper quorum address for each cluster peer. Any node in one cluster can know the configuration of another cluster via a zookeeper client.

Update flow will go from the shard leader in one cluster to the shard leader in the peer clusters. This can be bi-directional, with updates flowing in both directions. Updates can be either synchronous or asynchronous, with per-update granularity.

Solr transaction logs are currently removed when no longer needed. They will be kept around (potentially much longer) to act as the source of data to be sent to peer clusters. Recovery can also be bi-directional with each peer cluster sending the other cluster missed updates.

  • Scalable – no required single points of aggregation / dissemination that could act as a bottleneck.
  • Per-update choice of synchronous/asynchronous forwarding to peer clusters.
  • Peer clusters may have different configuration, such as replication factor.
  • Asynchronous updates allow for bursts of indexing throughput that would otherwise overload cross-DC pipes.
  • “Push” operation for lowest latency async updates.
  • Low-overhead… re-uses Solr’s existing transaction logs for queuing.
  • Leader-to-leader communication means update is only sent over cross-DC connection once.

CDCR

  1. An update will be received by the shard leader and versioned
  2. Update will be sent from the leader to it’s replicas
  3. Concurrently, update will be sent (synchronously or asynchronously) to the shard leader in other clusters
  4. Shard leader in the other cluster will receive already versioned update (and not re-version it), and forward the update to it’s replicas

The shard leader versions a document and then forwards it to replicas. Update re-orders are handled by the receiver by dropping updates that are detected to be older than the latest document version in the index. This works given that complete documents are always sent to replicas, even if it started as a partial update on the leader.

Solr version numbers are derived from a timestamp (the high bits are milliseconds and the low bits are incremented for each tie in the same millisecond to guarantee a monotonically increasing unique version number for any given leader).

If updates are accepted for the same document in two different clouds (implying two different leaders versioning the document), then having the correct last document “win” relies on clock synchronization between the two leaders. Updates to the same document at different data centers within the clock skew time risk being incorrectly ordered.

Solr only has versions at the document level. The current partial update implementation (because of other constraints) reads the current stored fields of the document, makes the requested update, and indexes the new resulting document. This creates a problem with accepting Solr atomic updates / partial updates to the same document in both data-centers.

Example:

DC1: writes document A, version=time1
DC2: receives document A (version=time1) update from DC1
DC1: updates A.street_address (Solr reads version time1, writes version time2)
DC2: updates A.phone_number (Solr reads version time1, writes version time3)
DC1: receives document A (version=time3) from DC2, writes it.
DC2: received document A (version=time2) from DC1, ignores it (older version)

Although both data-centers became “consistent”, the partial update of street_address was completely lost in the process.

Option 1: Configure the update for full synchronization. All peer clusters must be available for any to be writeable.

Option 2: Use client versioning, where the update clients specify a user-level version field.

Option 3: For a given document, consider one cluster the primary for the purposes of document changes/updates. See “Primary Cluster Routing”.

To deal with potential update conflicts arising from updating the same document in different data centers, each document can have a primary cluster.

A routing enhancement can ensure that a document sent to the wrong cluster will be forwarded to the correct cluster.

Routing can take as input a request parameter, a document field, or the unique id field. The primary cluster could be determined by hash code (essentially random), or could be determined by a mapping specified in the cluster peer list. Changes to this mapping for fail-over would not happen automatically in Solr. If a data center becomes unreachable, the application/client layers have responsibility for deciding that a different cluster should become the primary for that set of documents.

Primary cluster routing will be optional. Many applications will naturally not trigger the type of undesirable update behavior described, or will have the ability to work around update limitations.

Implement true partial updates with vector clocks and/or finer grained versioning so that updates to different fields can be done conflict free if re-ordered. This would also lower the bandwidth costs of partial updates since the entire document would no longer be sent to all replicas and to other peer clusters.

One could potentially further minimize cross-DC traffic by introducing traffic aggregator nodes (one per cluster) that all udpates would flow through. This would likely only improve bandwidth utilization in low update environments. The improvements would come from fewer connections (and hence less connection overhead) and better compression (a block of many small updates would generally have a better compression ratio than the same updates compressed individually).

Many zookeeper clients in a peer cluster could generate significant amounts of traffic between data centers. There could be a designated listener to the remote cluster state that could disseminate this state to others in the local cluster rather than hitting ZK directly.

Also worth investigating is the use of a local zookeeper observer node that could service all local ZK reads for the remote ZK quorum.

The first phase of this design only deals with updates. Collection level operations such as adding a new shard, splitting a shard, and changing replication levels, must be performed by the client on every cluster as applicable.

The collections API (and other higher level APIs) could be made peer-aware such that these operations would also be forwarded to peer clusters, as well as including a queuing mechanism for the cases when a peer cluster is unreachable.

Forwarding of updates from one cloud to another should be done via standard SolrJ client. Any needed enhancements/modification should be done to a SolrJ client such that those enhancements may also be used in other contexts.

Solr Filter Caching

The filter caching features in Solr allow for precise control over how filter queries are handled in order to maximize performance. Solr has the ability to specify if a filter is cached, specify the order filters are evaluated, and specify post filtering.

Adding a filter expressed as a query to a Solr request is a easy… simply add an additional fq parameter for each filter query.

http://localhost:8983/solr/select?
q=cars
&fq=color:black
&fq=model:Lamborghini
&fq=year:[2014 TO *]

By default, Solr resolves all of the filters before the main query. Each filter query is looked up individually in Solr’s filterCache (which is pretty advanced itself, supporting concurrent lookups, different eviction policies such as LRU or LFU, and auto-warming). Caching each filter query separately accelerates Solr’s query throughput by greatly improving cache hit rates since many types of filters tend to be reused across different requests.

Update: starting with Heliosearch 0.07, there is support built directly into the standard query parser for creating a filter query that uses the filter cache.

The advanced filter control API adds the ability to *not* cache a filter. Some filters may see almost no reuse across different requests, and not caching them can lead to a smaller, more effective filterCache with a higher hit rate.

To tell Solr not to cache a filter, we use the same powerful local params DSL that adds metadata to query parameters and is used to specify different types of query syntaxes and query parsers. For a normal query that does not have any localParam metadata, simply prepend a local param of cache=false. For example:

&fq={!cache=false}year:[2014 TO *]

To add cache=false to a filter query that already had localParams, simply add it right in with the rest of the params. For example, if we want to use Solr’s native spatial abilities to restrict our matches to locations within 50 km of Stanford, our filter query would look like:

&fq={!geofilt sfield=location pt=37.42,-122.17 d=50}

It’s easy to modify this filter to tell Solr not to cache it by adding cache=false in with the rest of the local parameters:

&fq={!geofilt sfield=location pt=37.42,-122.17 d=50 cache=false}

When a filter isn’t generated up front and cached, it’s executed in parallel with the main query. First, the filter is asked about the first document id that it matches. The query is then asked about the first document that is equal to or greater than that document. The filter is then asked about the first document that is equal to or greater than that. The filter and the query play this game of leapfrog until they land on the same document and it’s declared a match, after which the document is collected and scored.

Advanced filtering adds even more fine grained control by introducing the notion of cost. If there are multiple non-cached filters in a response, filters with a lower cost will be checked before those with a higher cost.

&fq={!cache=false cost=10}year:[2014 TO *]
&fq={!geofilt cache=false cost=20}
&pt=48.53,-131.26
&sfield=dealer_location
&d=50

In the example above, the filter based on year has a lower cost and will thus always be checked before the spatial filter.

As an aside, notice how spatial queries will use global spatial request parameters if they are not specified locally. This can make it even easier to construct requests containing spatial functions.

Some filters are slow enough that you don’t even want to run them in parallel with the query and other filters, even if they are consulted last, since asking them “what is the next doc you match on or after this given doc” is so expensive. For these types of filters, you really want to only ask them “do you match this doc” only after the query and all other filters have been consulted. Solr has special support for this called “post filtering”.

Post filtering is triggered by filters that have a cost>=100 and have explicit support for it. If there are multiple post filters in a single request, they will be ordered by cost.

The frange qparser has post filter support and allows powerful queries specifying ranges over arbitrarily complex function queries.

For example, if we wanted to take the log of popularity, divide it by the square root of the distance, and filter out documents with a result less than 5, we could run this as a post filter using frange:

&fq={!frange l=5 cache=false cost=200}div(log(popularity),sqrt(geodist()))

Post filtering support for the spatial filter queries bbox and geofilt has been available since Solr 4.0 too. To execute our previous un-cached spatial filter as a post filter, simply modify its cost to be greater than 100:

&fq={!geofilt cache=false cost=150}
&pt=37.42,-122.17
&sfield=location
&d=50

If you have expensive custom logic you’d like to add as a post filter (say per-document custom security ACLs), you can implement your own QParserPlugin that returns Query objects that implement Solr’s PostFilter interface. You can set the default cost or hardcode a cost higher than 100 if you want to only support post filtering. Then, you can use your custom parser as you would any other builtin query type via fq={!myqueryparser arg1=x arg2=y} and Solr will handle the rest!

In conclusion, hopefully this gives more insight into just one of many factors working under the hood to make Solr so fast. To try out the absolute latest functionality, you can always get a nightly build of trunk. Feedback is always appreciated!

Off-Heap FieldCache Faceting and Sorting

Lucene has a segmented architecture - when a small amount of documents are added to an existing index, this will often just add an additional small segment to the index.

Caching data structures at the segment level (e.g. field values used for sorting) is often desirable so that when a new view of the index is opened, additional segment caches only need to be created for those new segments. To date, Heliosearch’s off-heap nCache has been all segment-level to enable good near real-time performance.

However some search operations require data structures across the entire index to operate efficiently, and faceting by string fields is one of those. Doing more work when a new view of the index is opened (such as computing global ords) saves work for every faceting request that will use that view.

The term “FieldCache insanity” refers to the same logical data being cached more than once in a different form, taking up twice as much memory as needed. Apache Solr currently suffers from FieldCache insanity when the same field is sometimes used to facet and sometimes used to sort. Faceting uses a top-level FieldCache entry and sorting uses a per-segment FieldCache entry.

Top-level string support has just been added to Heliosearch’s nCache to enable fast faceting and other operations that benefit from global ords.

  • Off-heap data lowers garbage collection pauses and GC overhead.
  • Sorting and faceting on the same field does not cause insanity!
  • Speeds up faceting
  • Slightly speeds up sorting
  • Can save memory (string values only appear once instead of being duplicated across segments)
  • enables future native-code optimizations (Unlike Java arrays, off-heap data can be transparently accessed from native code)

You can force Heliosearch to use a top-level FieldCache entry via the top() function. For example, instead of specifying

sort=myfield_s desc

use

sort=top(myfield_s) desc

NOTE: Although the top() function exists in Solr, the functionality was removed and it is currently a no-op.

Forcing a top-level FieldCache entry for sorting is not something one would normally explicitly need to do. If a top-level entry for a string field already exists, it will be used even if top() was not specified. Faceting on a single-valued string field will automatically use a top-level string cache, there is nothing you need to specify.

Heliosearch avoids insanity by using a slice/view of the top-level string cache if a per-segment string cache is requested. This avoids the overhead of data duplication.

Benchmark details:

  • Ubuntu Linux 13.10, Java 1.7, quad-core CPU
  • 10M document index
  • Documents consist of an ID field, and 6 different single-valued string fields with varying numbers of unique values ranging from 10 to 1 Million
  • Client had 4 request threads
  • Each individual client request uses a random field to make the test realistic and to avoid the JVM overspecializing the code for any given field
  • Solr versions: Apache Solr 4.8.1, Heliosearch/Solr snapshot (based on Solr 4.9)

  sorted_query_latency

  Although the performance increase going from a per-segment to a top-level cache may be small for sorting, it’s essentially free if that top-level cache is needed for something else like fast faceting. We also previously covered the performance difference between Solr and Heliosearch for string sorting.

UPDATE: native code faceting has been implemented and results in even better performance than shown below.

Using the same set of fields, we also tested the performance of faceted search on the single valued string fields (which now automatically use the new off-heap nCache when on Heliosearch). The result was a 23% increase in request throughput! The chart below shows faceted request latency broken out by percentiles.

  faceted_request_latency  

The chart below shows the memory usage of Solr and Heliosearch after running both a faceting and sorting test concurrently. The “Max Process Size” was observed via “top” during the entire test, and “Min JVM Heap Size” was obtained by waiting for the tests to finish, then attaching jconsole to the server and forcing garbage collections until the smallest in-use heap size was obtained.

  memory_for_facet_sort

 

We started developing off-heap data structures for Solr (via Heliosearch) with the goal of solving many people’s JVM garbage collection problems, and enabling future native code optimizations. The performance increases to both sorting and faceting that we’ve now seen are a very nice added bonus!

If you try out Heliosearch on your own project, drop by the user list and let us know how it went, or stop in at the dev list to help further development!

A History of Lucene and Solr

I’ve often seen mistaken descriptions of Solr as just “a http wrapper around Lucene”. Unfortunately that mischaracterization was never nipped in the bud early enough and has continued to be repeated in many places such as press articles (where it is picked up and repeated again). Of course people who have been involved with Lucene and Solr from the beginning know better!

The fact that there was so much core functionality in Solr that Lucene users wanted even led the projects to merge in 2010.

Here’s a partial history of some Solr milestones that include core search functionality (i.e. not related to just exposing Lucene via HTTP):

Functionality Implemented in Solr Available in Lucene
Numerics + range queries Jan 2006 Sept 2009 Lucene 2.9
Index Replication Jan 2006 July 2013 Lucene 4.4 Replication Module
Unique keys (overwriting) Jan 2006 ? 2007 IndexWriter.updateDocument
Many analysis filters, WordDelimiterFilter, Soundex, Regex/Pattern, HTML, kstem, trim, reverse wildcard, multi-word synonym, etc Jan 2006 - various Oct 2012 Lucene 4.0 all analysis filters moved from Solr to Lucene
Searcher concurrency control Jan 2006 Nov 2011, Lucene 3.5, SearcherManager
Faceted search Sep 2006 Sep 2011, Lucene 3.4, LUCENE-3079
Function queries Jan 2006 Jun 2007, Solr’s FunctionQuery was copied (not moved) into Lucene 2.2 but it stagnated, function queries were later moved from Solr to Lucene for version 4.0 (Oct 2012)
Distributed search Feb 2008 Jul 2011, Lucene 3.3, partial support via TopDocs.merge
Query-time Join April 2011 Jan 2012, LUCENE-3602
Grouping / Field Collapsing Aug 2010 (dev patches used by many in production much earlier however) May 2011 - Oct 2011, Grouping moved from Solr to Lucene LUCENE-1421, LUCENE-3483, etc.
Constant score queries, including prefix/range queries that don’t explode when too many terms are matched Jan 2006 May 2006, moved from Solr to Lucene LUCENE-383 etc.
Multi-valued field cache (UnInvertedField) Nov 2008 SOLR-475 Mar 2011, moved from Solr to Lucene LUCENE-3003
Distributed faceting Feb 2008 Jul 2013, Lucene 4.4, partial support via FacetResult.mergeHierarchies?
Auto-suggest Aug 2010, SOLR-1316 May 2011 Moved from Solr to Lucene, LUCENE-2995
field types Jan 2006 Oct 2012 Lucene 4.0 FieldType class
Configurable analysis component factories Jan 2006 July 2012, all analysis factories moved from Solr to Lucene, LUCENE-2510
User-oriented query parsers (dismax, edismax) Jan 2006, Nov 2009 SOLR-1553 Nov 2013 LUCENE-5336
Real-time Get Nov 2011 SOLR-2700 Jan 2013 LUCENE-4695
Filter Cache Jan 2006 Nov 2014 LUCENE-6077
Query Cache Jan 2006 Apr 2015, Lucene 5.1 LUCENE-6303

    Of course, I’ve only touched on some of the features that were in Solr first and later became available in Lucene. I’ve left out all of the features that Lucene still does not have (like optimistic locking, numeric statistics), and more server-ish features (many query parser types, in/out support for JSON, XML, CSV, etc.)

The reality is that both Lucene and Solr have long been innovating in the open source search space.

Parameter Substitution / Macro Expansion

Macro Expansion is a new Solr 5.1 feature that does parameter substitution across all request parameters.

The macro expansion is done at the same point in time that default parameters are applied (i.e. when the request reaches the correct solr request handler). This means that request handler defaults, appends, and invariants configured for the handler may reference request parameters, and vice-versa.

Here’s a very simple example of parameterizing a range query in lucene/solr syntax:

q=price:[ ${low} TO ${high} ]
&low=100
&high=200

One can also specify default values:

q=price:[ ${low:0} TO ${high:100} ]

Substitutions can themselves contain further macro expansions:

q=${price_query}
&price_query=${price_field}:[ ${low} TO ${high} ] AND inStock:true
&price_field=specialPrice
&low=50
&high=100

Even parameter names can have macro expansion applied. A simple example is faceting on a field to be specified via another parameter ff:

&facet.field=${ff}
&f.${ff}.facet.mincount=1
&f.${ff}.facet.limit=5
&ff=categoryField1

If you want to disable macro expansion, simply pass expandMacros=false as a request parameter.

We’ll be adding even more powerful macro expansion capabilities in the future, so drop by the solr-user mailing list to join the discussion!

Go to the Download page to try out the current functionality.

Solr 4.8 Features

Solr 4.8 has been released. Here’s an overview of how to use some of the new features. Also see Solr download links and upcoming features of the next Solr release.

The complexphrase query parser can produce phrase queries with embedded wildcards and boolean queries. It works via multiple passes, parsing a query and then re-parsing any phrase queries for additional markup. At query execution time, span queries are generated to implement the complex phrase logic.

The simplest example is a phrase query containing a prefix query:

q={!complexphrase}"apple ip*"

This will match text with both “apple ipod” and “apple ipad”. One can specify inOrder=false as a localParam to also match “ipod apple” and “ipad apple”.

q={!complexphrase inOrder=false}"apple ip*"

One can also specify a different default field to search with the df localParam:

q={!complexphrase df=name}"john* smith"

This will match both “john smith” and “johnathan smith” in the name field. Of course one could always specify the field directly in the query as well:

q={!complexphrase}name:"john* smith"

Phrase slop works to specify the proximity of the clauses. For example, the following would also match a name of “johnathan q smith”:

q={!complexphrase}name:"john* smith"~1

And of course we can throw in parens, OR clauses, and other complex logic as well:

q={!complexphrase}name:"(aaa OR (bbb* OR ccc)) ddd -eee (fff~1 OR ggg)" AND text:"nnn? (ooo OR ppp) -qqq www"~3

 

Previously, one had to use XML or binary format (or SolrJ) to index nested child documents (needed for block join functionality). Support has now been added for JSON:

curl http://localhost:8983/solr/update/json?softCommit=true -H 'Content-type:application/json' -d '
[
{
"id": "chapter1",
"title" : "Indexing Child Documents in JSON",
"content_type": "chapter",
"_childDocuments_": [
{
"id": "1-1",
"content_type": "page",
"text": "ho hum... this is page 1 of chapter 1"
},
{
"id": "1-2",
"content_type": "page",
"text": "more text... this is page 2 of chapter 1"
}
]
}
]
'

Now if we query on “ho hum”, we obviously get page 1 of chapter 1 back:

http://localhost:8983/solr/query?q="ho hum"
[...]
"response":{"numFound":1,"start":0,"docs":[
{
"id":"1-1",
"content_type":["page"]}]
}

But if we wanted to select chapters based on matches in pages, we could utilize a parent block join:

http://localhost:8983/solr/query?q={!parent which='content_type:chapter'}"ho hum"
[...]
"response":{"numFound":1,"start":0,"docs":[
{
"id":"chapter1",
"content_type":["chapter"]}]
}

A child block join can be used to restrict (or match) child pages based on matches in a chapter (parent). For example, the following request returns all pages for which the chapter title contains “Indexing”:

http://localhost:8983/solr/query?q={!child of=content_type:chapter}title:Indexing
[...]
"response":{"numFound":2,"start":0,"docs":[
{
"id":"1-1",
"content_type":["page"]},
{
"id":"1-2",
"content_type":["page"]}]
}

The query above would probably be more useful as a filter… for example, if we wanted to search for “hum” on all pages where the chapter had “Indexing” in the title:

http://localhost:8983/solr/query?q=hum&fq={!child of=content_type:chapter}title:Indexing
[...]
"response":{"numFound":1,"start":0,"docs":[
{
"id":"1-1",
"content_type":["page"]}]
}

Solr 5.3 and later has the ability to combine faceting and nested objects / block join.

 

The ExpandComponent can be used to expand parent/child relationships in Solr. Joel previously blogged about the Expand Component and gave an example of how it could be used to expand a block join.

 

This is more in the “configuration” category of features. SolrCloud has always allowed multiple collections to share configuration, and now that capability has been brought to Solr’s non-cloud mode.

Since collections can be created or destroyed, we obviously don’t want shared configuration for these collections to be under the collection itself. The default location for config sets is in the “configsets” directory under the solr home (the example solr server currently doesn’t have this directory by default).

Let’s create a configSet named “generic” and then create two new collections (single core) called “books” and “music”:

~/solr/example$ mkdir -p solr/configsets/generic/conf/
~/solr/example$ cp -r solr/collection1/conf/* solr/configsets/generic/conf/
~/solr/example$ curl 'http://localhost:8983/solr/admin/cores?action=CREATE&name=books&configSet=generic'
~/solr/example$ curl 'http://localhost:8983/solr/admin/cores?action=CREATE&name=music&configSet=generic'

Now you should be able to go to the admin console http://localhost:8983/solr and go to the “Core Selector” on the bottom left hand side to see the new cores/collections we just created.

Let’s inspect what was done from the command line:

~/solr/example$ ls -F solr
README.txt bin/ books/ collection1/ configsets/ music/ solr.xml zoo.cfg
~/solr/example$ ls -F solr/books
core.properties data/
~/solr/example$ cat solr/books/core.properties
#Written by CorePropertiesLocator
#Thu Apr 24 21:12:33 EDT 2014
name=books
configSet=generic

So we can see that the new cores created only contain a data directory and lack a “conf” directory of their own. The core.properties file points to the correct named configSet.

 

Stopwords and Synonyms may now be managed via a REST API! The new analysis filter types are ManagedStopFilterFactory and ManagedSynonymFilterFactory. The example schema.xml now contains a field type that uses these new analysis filters:

<!-- A text type for English text where stopwords and synonyms are managed using the REST API -->
<fieldType name="managed_en" class="solr.TextField" positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.ManagedStopFilterFactory" managed="english" />
<filter class="solr.ManagedSynonymFilterFactory" managed="english" />
</analyzer>
</fieldType>

To test this out, let’s also change the dynamic field *_en to use managed_en:

<dynamicField name="*_en" type="managed_en" indexed="true" stored="true" multiValued="true"/>

After starting the example server, we can retrieve the current english synonyms:

curl "http://localhost:8983/solr/collection1/schema/analysis/synonyms/english"
[...]
"managedMap":{
"gb":["gib",
"gigabyte"],
"happy":["glad",
"joyful"],
"tv":["television"]}}}

  Lets add a new synonym:

curl -XPUT "http://localhost:8983/solr/collection1/schema/analysis/synonyms/english" -H 'Content-type:application/json' --data-binary '{"mb":["MiB","megabyte"]}'

  Before these changes are visible to the actual search or indexing code in Solr, we need to reload the Solr core:

curl "http://localhost:8983/solr/admin/cores?action=RELOAD&core=collection1"

  And now we can do a query on a field that matches the dynamicField we set up and can see the results of the new synonym:

curl "http://localhost:8983/solr/query?q=foo_en:mb&debugQuery=true"
[...]
"debug":{
"rawquerystring":"foo_en:mb",
"querystring":"foo_en:mb",
"parsedquery":"(foo_en:megabyte foo_en:mib)/no_coord",
"parsedquery_toString":"foo_en:megabyte foo_en:mib",

  To delete the stopword we just added:

curl -XDELETE "http://localhost:8983/solr/collection1/schema/analysis/synonyms/english/mb"

To retrieve the list of stopwords:

curl "http://localhost:8983/solr/collection1/schema/analysis/stopwords/english"

To add a new stopword:

curl -XPUT "http://localhost:8983/solr/collection1/schema/analysis/stopwords/english" -H 'Content-type:application/json' --data-binary '["foo"]'

To delete the stopword we just added:

curl -XDELETE "http://localhost:8983/solr/collection1/schema/analysis/stopwords/english/foo"

 

There have been numerous SolrCloud changes, including:

  • A new List collections and cluster status API which clients can use to read collection and shard information instead of reading data directly from ZooKeeper.
  • Some long running SolrCloud commands (like shard splitting) may now be run in “async” mode to avoid client timeouts
  • A new ADDREPLICA command in the Collections API

Other changes include:

  • Solr 4.8 now requires Java7!
  • RegexReplaceProcessorFactory now supports pattern capture group substitution in the replacement string.
  • A DocExpirationUpdateProcessorFactory that can mark documents based on a TTL (time-to-live) and periodically delete expired documents

Heliosearch/Solr Off-Heap FieldCache Performance

Heliosearch’s off-heap FieldCache was previously introduced and benchmarked for integer fields. Support for all numeric field types as well as string fields has now been completed, and this post will focus on the performance of string fields.

A review of nCache (n is for “native”) features and goals:

  • nCache has Off-Heap Data-structures, just like the Off-Heap Filters to lower garbage collection pauses and GC overhead.
  • nCache is a managed cache, meaning you can do anything with it that you can do with other Solr caches, including configuring size and warming policies, and viewing cache statistics through the admin page.
  • nCache is NRT friendly. Field values are cached on per-segment basis, enabling rapid turn-around time for new index snapshots.
  • nCache is designed for maximium performance, even when the system is not experiencing garbage collection issues.
  • nCache uses no weak references like the Lucene FieldCache does.

 

The first benchmark involved sorting by string fields with different numbers of unique values. Queries were of the following form:

q={!cache=false}*:*
&sort=my_str_field1 desc

The test index consisted of 10M documents. 80% of the documents had a value for any given field being sorted on. The query was executed 50 timed per field, and the median latency for each field was recorded.

sort_10M_str_latency

  Next we tested the concurrent query throughput on the same 10M document index. Each query would sort on a random string field with a random sort order (asc or desc). 1000 queries were run for each throughput test, and each test was repeated 5 times (restarting the JVM before each) to generate an average throughput.

The hardware consisted of a 3GHz quad-core AMD processor running Ubuntu Linux 13.10. The latest 64 bit Oracle JVMs for Java7 and Java8 were used.

  The previous query sorting performance test was re-run on a 3.4GHz quad-core Intel processor running Windows 8. nCache shows an even greater performance advantage here (68% throughput increase using Java8). This probably had more to do with the different processor architecture (Intel vs AMD) than the different operating systems.

sorted_str_windows

  We also compared the process sizes via “top” and tracked the maximum size during tests (averaging across different test runs).

This new functionality is included in the latest Heliosearch release. Heliosearch is currently API compatible with Solr at the HTTP level, so it should be easy to try it out and see what types of performance increases you get. Let us know how it goes in the the heliosearch user forum, and join our the heliosearch dev forum if you want to contribute!

Solr 4.7 Features

Solr 4.7 has been released! Here’s a slightly more in-depth overview of some selected features.

Both single node, and distributed deep paging have been added to Solr! I previously created an example of how to use Solr’s deep paging, and Hoss has a great set of benchmarks showing the performance increases. Here’s one graph from that post showing the most basic case (sorting by score descending) and how performance varies with paging depth. Ignore the green “strawman” line… that was proof-of-concept code that was never committed.

In short, pass cursorMark=* on the first paging request and then you will be given back a nextCursorMark value which you should then pass as cursorMark for your next request.

 

SimpleQueryParser (registered via the name “simple”) is an alternative to edismax in that they both share the goal of handling user queries without throwing exceptions. Unlike edismax, this parser does not handle full “lucene” query syntax. The q.operator parameter controls what operators are available (by default all are). Example:

&defType=simple # type of the main query is "simple"
&q=solr -search # the user query string
&q.op=AND # all clauses mandatory (the default is OR)
&q.operators=WHITESPACE,NOT # enable the "-" operator (we need whitespace parsing too so the "-" will be seen as an operator)
&qf=title^2 text # query across the title and text field, giving a boost of 2 to the title field

The output of that example query (in lucene sytnax) would be:

+(text:solr^3.0 title:solr^2.0) +(-(text:search^3.0 title:search^2.0) *:*)

 

Composite ID routing allows one to partition the hash range such that related documents appear on the same part of the hash ring. This allows one to efficiently query over a set of related documents (say a users email messages) without querying the whole collection. The default compositeId router has been extended to accept tri-level routes so partitioning may be done at more than one level. For example, one use case would be to partition first by application id, then by user id, with the final part of the hash being the users document id.

All this simply works out of the box (no configuration needed!). Index documents with ID’s like the following: {"id" : "heliosearch!yonik!mydoc1", ... And then at query time you could specify a route key that restricts queries to nodes containing heliosearch documents: _route_=heliosearch! Or that restricts queries to nodes containing yonik’s heliosearch documents: _route_=heliosearch!yonik!

 

Migrate a set of documents to another collection

Section titled “Migrate a set of documents to another collection”

A new MIGRATE operation has been added to the Solr Collections API that allows one to move part of one collection to another collection based on _route_ (i.e. the ID prefix when using compositeId routing). This is actually a live migration! While the source documents are being copied to the target collection, any updates to those documents will also be forwarded to the target collection. For a short amount of time after the copy is complete, updates to the source documents will continue being forwarded to the target collection. It’s the clients responsibility after that point to send the documents to the correct collection.

Here’s a quick example of migration in action: First start up a single node in ZK mode:

java -Dbootstrap_confdir=./solr/collection1/conf -Dcollection.configName=myConf -DzkRun -DnumShards=1 -jar start.jar

Create two new collections:

curl "http://localhost:8983/solr/admin/collections?action=CREATE&name=c2&replicationFactor=2&maxShardsPerNode=100&numShards=1"
curl "http://localhost:8983/solr/admin/collections?action=CREATE&name=c3&replicationFactor=2&maxShardsPerNode=100&numShards=1"

Index some documents to collection “c2”

curl "http://localhost:8983/solr/c2/update?commit=true" -H 'Content-type:application/json' -d '[{"id":"a!doc1"}, {"id":"b!doc2"},{"id":"c!doc3"},{"id":"d!doc4"}]'

Now migrate all documents with a route key of a! from collection “c2” to “c3”

curl 'http://localhost:8983/solr/admin/collections?action=MIGRATE&collection=c2&split.key=a!&target.collection=c3'

The docs should now be migrated! To verify, call commit on the target collection to make the docs visible, and do a query.

curl "http://localhost:8983/solr/c3/update?softCommit=true"
curl "http://localhost:8983/solr/c3/query?q=*:*"

There are quite a few other new Solr features/improvements, including

  • For security minded folks, SSL support for SolrCloud
  • The ability to build Solr indexes with Hadoop MapReduce
  • Many more Suggester options
  • Updated geospatial support

nCache: Heliosearch/Solr Off-Heap FieldCache

Heliosearch has a new replacement for the Lucene FieldCache currently used by Solr for sorting, faceting, and function queries.   Introducing nCache (n is for “native”):

  • nCache has Off-Heap Data-structures, just like the Off-Heap Filters to lower garbage collection pauses and GC overhead.
  • nCache is a managed cache, meaning you can do anything with it that you can do with other Solr caches, including configuring size and warming policies, and viewing cache statistics through the admin page.
  • nCache is NRT friendly. Field values are cached on per-segment basis, enabling rapid turn-around time for new index snapshots.
  • nCache is designed for maximium performance, even when the system is not experiencing garbage collection issues.
  • nCache uses no weak references like the Lucene FieldCache does.

  UPDATE SINCE THIS POST: nCache now has support for all numerics and string fields.

Currently, only integer fields have been implemented for nCache, so this is what we tested.

The first test involved sorting by integer fields with different numbers of unique values. Queries were of the following form:

q={!cache=false}*:*
&sort=my_int_field1 desc

The test index consisted of 50M documents, and the query for a given field was executed 10 times consecutively, and the fastest time was retained.

int_sort_latency

  Next we tested the concurrent query throughput on the same 50M document index. The first set of queries consisted of sorting by a random integer field (the same set we used for the first test). The second set of queries consisted of using a function query to add two of the integer fields together and sort by the resulting score.

The function queries were of the following form:

q={!func cache=false}add(my_int_field1, my_int_field2)

int_query_throughput

  The first time one sorts on an indexed field, the FieldCache (or nCache) entry is built by un-inverting the field. With per-segment caches, only new segments will need un-inverting when the index changes (although a major merge can cause all segments to change).

The un-invert time for all of the integer fields for all 22 segments in the 50M document index was tested by repeating the test 3 times (stopping the server each time) and taking the lowest (fastest) result.

  There were no significant garbage collection pauses during these tests. Different query loads that produce more garbage should show an even greater throughput advantage for Heliosearch’s off-heap nCache.

nCache is like any other Solr Cache, so you can configure and manage it and get statistics via the admin page, or via JMX.

Some of the statistics available include: size - the memory used by the entry (in bytes) for the field (most of it will be off-heap memory) segments - the number of segments populated for the field carriedOver - the number of segments shared with the previous searcher / index snapshot   Here is an example of the admin statistics after running through some of the tutorial. http://localhost:8983/solr/#/collection1/plugins/cache?entry=nCache

Heliosearch uses nCache by default, just as it uses off-heap filters by default. Simply download the latest release and start using it! If you’re new to Heliosearch/Solr, you may want to start here.

Only integer fields have been implemented so far, but other field types will quickly follow.

Heliosearch/Solr Off-Heap Filters

Off-Heap Native Filters is the first feature we added to Heliosearch, a new open source project designed to bring Solr performance to the next level.

JVMs have never been good at dealing with large heaps. Large heaps mean lots of garbage collection work, and often means some pretty long stop-the-world GC pauses where nothing else can proceed. This can cause query/request timeouts, or even zookeeper session timeouts in SolrCloud mode.

Heliosearch/Solr has some pretty advanced filter caching, but it can take up a significant amount of memory, depending on the application. This is exactly the type of large, longer lived objects that can benefit by being moved off the JVM heap and explicitly managed. Off-heap memory is invisible to the garbage collector.

Heliosearch filters (Solr DocSet objects) are now allocated off-heap and reference counted so they can be freed as soon as they are no longer being used. The JVM GC no longer needs to waste time copying around these blocks of memory. This helps to both eliminate the long GC pauses as well as increase request throughput.

I expected that I’d have to try a lot of different things to re-create the stop-the-word GC pauses reported by others, but they happened on my first try! They weren’t as big as others reported, but my heap size is small as those things go. Bigger heaps are correlated with bigger GC pauses.

  • Ubuntu Linux server, 8GB RAM, 4 CPU Cores, Java 1.7 64 bit
  • Client: 8 threads, each doing a query of an id with a random filter (500 different filters)
  • filterCache: size=1000, large enough to hold all filters w/o evictions
  • Index: 3.8GB, 50M docs

  Apache Solr command line:

java -jar -Xmx4G start.jar

  Heliosearch/Solr command line:

java -jar start.jar

  I had to set the heap size to 4GB when running Apache Solr to avoid OOM exceptions. Since the maximum amount of RAM on the box was 8G, I wanted to leave the remaining memory for the OS to cache the index files (else things would get really slow).

Here are the graphs of the resulting GC activity for a run of 20,000 query requests. The grey bars represent time spent in a GC. The red line is the actual size of the heap, and the blue line represents the actual amount of the heap in use.

It was even easy to externally see the stop-the-world pauses on Solr while the test was running. Logging was enabled, so every request left a log message, causing the terminal to rapidly scroll. Whenever a major GC compaction hit, the terminal abruptly stopped scrolling. solr_gc

The Heliosearch GC graph completes sooner because less time is spent doing garbage collection. Notice the almost complete absence of stop-the-world full GC pauses, and greatly reduced other GC pauses. heliosearch_gc

This chart shows percentile query latencies of the second 10,000 queries in a 20,000 query run (just to ensure hotspot and the caches were all warmed up).

query_latency

query_throughput The Query Throughput graph illustrates just how much CPU time is spent in garbage collection that can be freed up with off-heap data structures. This is an extreme result of course. The throughput increase caused by off-heaping data structures would be more moderate if one is not experiencing frequent large garbage collections.

The maximum resident memory of the process (monitored externally via top) was measured over 5 runs.

Apache Solr Heliosearch
minimum run 3.8 GB 3.6 GB
maximum run 4.3 GB 3.7 GB

Heliosearch, with it’s off-heap filters, had a more stable memory profile and used less memory on average. This left more memory free for the operating system to cache index files, which is very important for good performance.

In this simple test, off-heap filters eliminated long GC pauses, made requests more predictable by reducing large outliers, and increased overall query throughput.

Try it yourself and give us feedback in the Heliosearch Forum!

Getting Started with Solr

Getting Started with Solr: a Simple Solr Tutorial

Section titled “Getting Started with Solr: a Simple Solr Tutorial”

Note: this tutorial is for Solr 4

Download Apache Solr 4.

You only need to download the single .ZIP or .TGZ file and extract it anywhere you like - no installation is required!!

$ cd example
$ java -jar start.jar

You’re now ready to start using Solr! To verify it’s up and running, you can point your browser at the admin page:

http://localhost:8983/solr/

Solr Admin

If something didn’t work, check if you have the proper prerequisites.

Now that Solr is running, we can add a document (also known as “indexing” a document):

$ curl http://localhost:8983/solr/update -H 'Content-type:application/json' -d '
[
{"id" : "book1",
"title" : "American Gods",
"author" : "Neil Gaiman"
}
]'

And then we can ask for it back:

$ curl http://localhost:8983/solr/get?id=book1
{
"doc": {
"id" : "book1",
"author": "Neil Gaiman",
"title" : "American Gods",
"_version_": 1410390803582287872
}
}

Of course for queries, you can always just use your browser and click on the link http://localhost:8983/solr/get?id=book1 or cut’n’paste the URL into your browser and modify the query directly in the address bar to try out different requests.

The author and title fields are pre-defined in the schema, but Solr can use convention over configuration for new fields if one does not wish to edit the schema. In this manner, Solr includes the essential benefits of schemaless - namely the ability to add new fields on the fly without having to pre-define them.

Let’s update book1 with cat, a category field, and two new fields that haven’t been defined in the schema, a publication year, and an ISBN. Via dynamic fields, a field name ending with _i tells Solr to treat the value as an integer, while a field name ending with _s is treated as a string.

$ curl http://localhost:8983/solr/update -H 'Content-type:application/json' -d '
[
{"id" : "book1",
"cat" : { "add" : "fantasy" },
"pubyear_i" : { "add" : 2001 },
"ISBN_s" : { "add" : "0-380-97365-0"}
}
]'

By using convention via dynamicFields, Solr avoids the pitfalls of trying to guess at the types of new fields while retaining the benefits of dynamically adding new fields as needed.

  Now let’s add a few more documents, this time in CSV (comma separated values) format:

$ curl http://localhost:8983/solr/update?commitWithin=5000 -H 'Content-type:text/csv' -d '
id,cat,pubyear_i,title,author,series_s,sequence_i
book2,fantasy,1996,A Game of Thrones,George R.R. Martin,A Song of Ice and Fire,1
book3,fantasy,1999,A Clash of Kings,George R.R. Martin,A Song of Ice and Fire,2
book4,sci-fi,1951,Foundation,Isaac Asimov,Foundation Series,1
book5,sci-fi,1952,Foundation and Empire,Isaac Asimov,Foundation Series,2
book6,sci-fi,1992,Snow Crash,Neal Stephenson,Snow Crash,
book7,sci-fi,1984,Neuromancer,William Gibson,Sprawl trilogy,1
book8,fantasy,1985,The Black Company,Glen Cook,The Black Company,1
book9,fantasy,1965,The Black Cauldron,Lloyd Alexander,The Chronicles of Prydain,2
'

We added the commitWithin=5000 parameter to indicate that we would like our updates to be visible within 5000 milliseconds (5 seconds). The Lucene library that Solr uses for full-text search works off of point-in-time snapshots that must be periodically updated in order for queries to see new changes.

Note that although we often use JSON in our examples, Solr is actually data format agnostic - you’re not artificially tied to any particular transfer-syntax or serialization format such as JSON or XML.

  Now let’s query our book collection! For example, we can find all books with “black” in the title field:

http://localhost:8983/solr/query?
q=title:black
fl=author,title

The fl parameter stands for “field list” and specifies what stored fields should be returned from documents matching the query. We should see a result like the following:

{"response":{"numFound":2,"start":0,"docs":[
{
"title":["The Black Company"],
"author":"Glen Cook"},
{
"title":["The Black Cauldron"],
"author":"Lloyd Alexander"}]
}}

 

Let’s try a more advanced query that combines many elements - limiting the number of books shown for any given series to 1 by grouping documents by series_s, sorting by publication year descending, and requesting facet counts for the book category:

http://localhost:8983/solr/query?
q=*:*
&fl=id,title,series_s,pubyear_i
&sort=pubyear_i desc
&group=true
&group.main=true
&group.field=series_s
&facet=true
&facet.field=cat

We can see how easy it is to construct and understand even a complex request by stepping through the parameters:

  • q=*:* the main query, *:* matches all documents
  • fl=id,title,series_s,pubyear_i field list - the list of fields we want to return for matching documents
  • sort=pubyear_i desc sorts the list of matching documents by pubyear_i in descending order
  • group=true turns on the grouping / field-collapsing feature
  • group.main=true put the grouped documents where the main query results normally appear instead of in the grouped section of the response.
  • group.field=series_s group together matching documents by the series_s field
  • facet=true turns on the faceting feature
  • facet.field=cat get facet counts for each value of the cat field. In this example, we have 5 “fantasy” books and 4 “sci-fi” books that match the query

Notice that by using simple parameters, as opposed to a compilcated hierarchial DSL, it’s very easy to add additional request parameters without worrying about matching up braces or how they nest within a request. For example, if you wanted to get facet counts by publication year, you could simply add facet.field=pubyear_i anywhere in the list of request parameters. Simple parameter-based requests are especially valuable during ad-hoc testing where it’s easy to add, remove, and edit request parameters right in the address bar of your browser! They also play nicer with HTML forms which can directly create Solr requests from the request parameters.

Welcome to the community! Now that you’ve discovered just how easy it is to get up and running, you should check out all of the other powerful features that Solr has to offer.

Remember to subscribe to the solr-user mailing list where you’ll meet a ton of helpful users and developers!

Subscribe: solr-user-subscribe

Advanced Filter Caching in Solr

The filter caching features in Solr allow for precise control over how filter queries are handled in order to maximize performance. Solr has the ability to specify if a filter is cached, specify the order filters are evaluated, and specify post filtering.

Adding a filter expressed as a query to a Solr request is a easy… simply add an additional fq parameter for each filter query.

http://localhost:8983/solr/demo/select?
q=cars
&fq=color:black
&fq=model:Lamborghini
&fq=year:[2014 TO *]

By default, Solr resolves all of the filters before the main query. Each filter query is looked up individually in Solr’s filterCache (which is pretty advanced itself, supporting concurrent lookups, different eviction policies such as LRU or LFU, and auto-warming). Caching each filter query separately accelerates Solr’s query throughput by greatly improving cache hit rates since many types of filters tend to be reused across different requests.

Update: starting with Solr 5.2, there is support built directly into the standard query parser for creating a filter query that uses the filter cache.

The advanced filter control API adds the ability to *not* cache a filter. Some filters may see almost no reuse across different requests, and not caching them can lead to a smaller, more effective filterCache with a higher hit rate.

To tell Solr not to cache a filter, we use the same powerful local params DSL that adds metadata to query parameters and is used to specify different types of query syntaxes and query parsers. For a normal query that does not have any localParam metadata, simply prepend a local param of cache=false. For example:

&fq={!cache=false}year:[2014 TO *]

To add cache=false to a filter query that already had localParams, simply add it right in with the rest of the params. For example, if we want to use Solr’s native spatial abilities to restrict our matches to locations within 50 km of Stanford, our filter query would look like:

&fq={!geofilt sfield=location pt=37.42,-122.17 d=50}

It’s easy to modify this filter to tell Solr not to cache it by adding cache=false in with the rest of the local parameters:

&fq={!geofilt sfield=location pt=37.42,-122.17 d=50 cache=false}

When a filter isn’t generated up front and cached, it’s executed in parallel with the main query. First, the filter is asked about the first document id that it matches. The query is then asked about the first document that is equal to or greater than that document. The filter is then asked about the first document that is equal to or greater than that. The filter and the query play this game of leapfrog until they land on the same document and it’s declared a match, after which the document is collected and scored.

Advanced filtering adds even more fine grained control by introducing the notion of cost. If there are multiple non-cached filters in a response, filters with a lower cost will be checked before those with a higher cost.

&fq={!cache=false cost=10}year:[2014 TO *]
&fq={!geofilt cache=false cost=20}
&pt=48.53,-131.26
&sfield=dealer_location
&d=50

In the example above, the filter based on year has a lower cost and will thus always be checked before the spatial filter.

As an aside, notice how spatial queries will use global spatial request parameters if they are not specified locally. This can make it even easier to construct requests containing spatial functions.

Some filters are slow enough that you don’t even want to run them in parallel with the query and other filters, even if they are consulted last, since asking them “what is the next doc you match on or after this given doc” is so expensive. For these types of filters, you really want to only ask them “do you match this doc” only after the query and all other filters have been consulted. Solr has special support for this called “post filtering”.

Post filtering is triggered by filters that have a cost>=100 and have explicit support for it. If there are multiple post filters in a single request, they will be ordered by cost.

The frange qparser has post filter support and allows powerful queries specifying ranges over arbitrarily complex function queries.

For example, if we wanted to take the log of popularity, divide it by the square root of the distance, and filter out documents with a result less than 5, we could run this as a post filter using frange:

&fq={!frange l=5 cache=false cost=200}div(log(popularity),sqrt(geodist()))

Post filtering support for the spatial filter queries bbox and geofilt has been available since Solr 4.0 too. To execute our previous un-cached spatial filter as a post filter, simply modify its cost to be greater than 100:

&fq={!geofilt cache=false cost=150}
&pt=37.42,-122.17
&sfield=location
&d=50

If you have expensive custom logic you’d like to add as a post filter (say per-document custom security ACLs), you can implement your own QParserPlugin that returns Query objects that implement Solr’s PostFilter interface. You can set the default cost or hardcode a cost higher than 100 if you want to only support post filtering. Then, you can use your custom parser as you would any other builtin query type via fq={!myqueryparser arg1=x arg2=y} and Solr will handle the rest!

In conclusion, hopefully this gives more insight into just one of many factors working under the hood to make Solr so fast. To try out the absolute latest functionality, you can always get a nightly build of trunk. Feedback is always appreciated!

MurmurHash3 for Java

I needed a really good hash function for the distributed indexing in SolrCloud. Since it is be used for partitioning documents, it needed to be really high quality (well distributed) since we don’t want uneven shards. It also needed to be cross-platform, so a client could calculate this hash value themselves if desired, to calculate which partition a given document belongs on.

MurmurHash3 is one of the top favorite new hash function these days, being both really fast and of high quality. Unfortunately it’s written in C++, and a quick google did not yield any suitable high quality port (this was back in 2011). So I took 15 minutes (it’s small!) to port the 32 bit version, since it should be faster than the other versions for small keys like document ids. It works in 32 bit chunks and produces a 32 bit hash - more than enough for partitioning documents by hash code.

It would be nice to prevent others from having to do the same thing. Since stuff like this is small enough, I simply put it under the public domain and uploaded to github. This way anyone can just copy the file or the function into their project and avoid extra dependencies and license hassles.

Here’s the code, copy away!

Fast forward to 2015, and we’re implementing hyperloglog based distributed cardinality count for the new Facet Analytics Module. That algorithm requires excellent 64 bit hashes.

The first step was to evaluate the Google Guava implementation of the 128 bit MurmurHash3 algorithm since Solr already uses the guava library. After a quick inspection, of the source code, I was disappointed. Their implementation is part of a larger hashing framework that introduces all sorts of inefficiencies.

  • The guava implementation does not match the reference C++ implementation for all seeds!
  • The implementation allocates *multiple* new objects for every hash
  • Even for hashing primitives (like int and long), the implementation creates a new byte buffer and copies in the value, then calls hash

It looks as if the Google implementation focused on hashing streams and large amounts of data and is completely inappropriate for hashing large numbers of small values.

New MurmurHash 128 bit Java implementation

Section titled “New MurmurHash 128 bit Java implementation”

I searched for another suitable implementation, but did not see one that had an appropriate license and did not do any object allocation. So I reinvented the wheel again and implemented a port myself, starting from the reference implementation.

  • Matches the reference MurmurHash3 implementation, for all seeds, so it can safely be used in multi-language scenarios where hashes must match.
  • Does not allocate any objects.
  • Public domain, so you can just copy the file into your project and not have to worry about extra licenses or extra JAR dependencies.

I also added in fmix32 and fmix64 from MurmurHash3 for quickly hashing integers and longs respectively.

If you need 32 bit hashes:

  • int - use MurmurHash3.fmix32(val)
  • long - use (int)MurmurHash3.fmix64(val)
  • float - use MurmurHash3.fmix32(Float.floatToRawIntBits(value))
  • double - use (int)MurmurHash3.fmix64(Double.doubleToRawLongBits(value))
  • bytes - use MurmurHash3.murmurhash3_x86_32

If you need 64 bit hashes:

  • int - use MurmurHash3.fmix64((long)val)
  • long - use MurmurHash3.fmix64(val)
  • float - use MurmurHash3.fmix64((long)Float.floatToRawIntBits(value))
  • double - use MurmurHash3.fmix64(Double.doubleToRawLongBits(value))
  • bytes - use MurmurHash3.murmurhash3_x64_128(value) and then just use one half (one long) of the 128bit result

This implementation is public domain, so just copy the code into your project!

Solr's Realtime Get

Solr took another step toward increasing it’s NoSQL datastore capabilities, with the addition of realtime get.

As readers probably know, Lucene/Solr search works off of point-in-time snapshots of the index. After changes have been made to the index, a commit (or a new Near Real Time softCommit) needs to be done before those changes are visible. Even with Solr’s new NRT (Near Real Time) capabilities, it’s probably not advisable to reopen the searcher more than once a second. However there are some use cases that require the absolute latest version of a document, as opposed to just a very recent version. This is where Solr’s new realtime get comes to the rescue, where the latest version of a document can be retrieved without reopening the searcher and risk disrupting other normal search traffic.

The realtime get handler is registered at the /get URL. As an example, a request like

http://localhost:8983/solr/get?id=SOLR1000&fl=id,name

returns a response like

{"doc":{"id":"SOLR1000","name":"Solr, the Enterprise Search Server"}}

Notice that the optional fl (field list) parameter works as normal, allowing you to select the fields you want returned.

There’s also a realtime get component that can be inserted into any request handler, including the standard request handler.

The realtime get feature uses transaction logging to keep track of uncommitted updates to the index.  When a get request for a document is received, this log is checked first and retrieved from there if found.  If it’s not found, then the latest opened searcher is used to retrieve the document.  Checking the log is super fast, and IO reads from the log are fully concurrent for maximum scalability.

Download a recent nightly build of Solr 4.0-dev and follow the Quick Start guide  on the Solr wiki.  Feedback on the solr-user mailing list is always appreciated!

Solr relevancy function queries

Lucene’s default ranking function uses factors such as tf, idf, and norm to help calculate relevancy scores. Solr has now exposed these factors as function queries.

  • docfreq(field,term) returns the number of documents that contain the term in the field.
  • termfreq(field,term) returns the number of times the term appears in the field for that document.
  • idf(field,term) returns the inverse document frequency for the given term, using the Similarity for the field.
  • tf(field,term) returns the term frequency factor for the given term, using the Similarity for the field.
  • norm(field) returns the “norm” stored in the index, the product of the index time boost and then length normalization factor.
  • maxdoc() returns the number of documents in the index, including those that are marked as deleted but have not yet been purged.
  • numdocs() returns the number of documents in the index, not including those that are marked as deleted but have not yet been purged.

We can use these new functions to develop and test custom ranking functions!  For example, if we wanted simple tf*idf for a given term, we could issue the following function query (if you have solr’s example server running with exampledocs indexed, just click on the following link):

http://localhost:8983/solr/select/?fl=score,id&defType=func&q=mul(tf(text,memory),idf(text,memory))

To avoid repeating the term we are using (text,memory) we can pull the field and term out into other query parameters:

http://localhost:8983/solr/select/?fl=score,id&defType=func&q=mul(tf($f,$t),idf($f,$t))&f=text&t=memory

Utilizing Solr’s new ability to sort by arbitrary function queries, we could now sort a query by the number of times a specific term appears in each document.  The following query searches for documents matching “DDR”, but then sorts by the number of times “memory” appears in the text field.

http://localhost:8983/solr/select/?fl=score,id&q=DDR&sort=termfreq(text,memory) desc

We could also utilize the “norm” function to sort by the longest field first.  This assumes there were no index time boosts and thus the norm is just the standard length normalization factor.

http://localhost:8983/solr/select/?fl=score,id&q=DDR&sort=norm(text) asc

Given Solr’s plethora of function queries (including the new spatial queries that return distance between points), the possibilities are almost endless.  To try this out,  you’ll need a recent nightly build of Solr 4.0-dev, or LucidWorks Enterprise, our commercial version of Solr.

Solr Result Grouping / Field Collapsing Improvements

I previously introduced Solr’s Result Grouping, also called Field Collapsing, that limits the number of documents shown for each “group”, normally defined as the unique values in a field or function query.

Since then, there have been a number of bug fixes, performance improvements, and feature enhancements. You’ll need a recent nightly build of Solr 4.0-dev to try it out.

One improvement is the ability to group by query via the group.query parameter. This functionality is very similar to facet.query, except that it retrieves the top documents that match the query, not just the count. This has many potential uses, including always getting the top documents for specific groups, or defining custom groups such has price ranges.

Another useful capability is the addition of the group.main parameter. Setting this to true causes the results of the first grouping command to be used as the main result list in a flattened response format that legacy clients will be able to handle.

For example, the grouped response format normally returns highly structured results under “grouped”. …&q=solr+memory&group=true&group.field=manu_exact

 

"grouped":{
"manu_exact":{
"matches":6,
"groups":[{
"groupValue":"Apache Software Foundation",
"doclist":{"numFound":1,"start":0,"docs":[
{
"id":"SOLR1000",
"name":"Solr, the Enterprise Search Server",
"manu":"Apache Software Foundation"}]
}},
{
"groupValue":"Corsair Microsystems Inc.",
"doclist":{"numFound":2,"start":0,"docs":[
{
"id":"VS1GB400C3",
"name":"CORSAIR ValueSelect 1GB 184-Pin DDR SDRAM Unbuffered DDR 400 (PC 3200) System Memory - Retail",
"manu":"Corsair Microsystems Inc."}]
}},
[...]

If we add group.main=true to the request, then we get back a much more familiar looking response (i.e. it looks like a normal non-grouped response): …&q=solr+memory&group=true&group.field=manu_exact&group.main=true

 

"response":{"numFound":6,"start":0,"docs":[
{
"id":"SOLR1000",
"name":"Solr, the Enterprise Search Server",
"manu":"Apache Software Foundation"},
{
"id":"VS1GB400C3",
"name":"CORSAIR ValueSelect 1GB 184-Pin DDR SDRAM Unbuffered DDR 400 (PC 3200) System Memory - Retail",
"manu":"Corsair Microsystems Inc."},

One can also use the group.format=simple parameter to select this simplified flattened response within the normal “grouped” section of the response.

Other recent enhancements include support for debugging explain, highlighting, faceting, and the ability to handle missing values in the grouping field by treating all documents without a value as being in the “null” group.

There have been a number of performance enhancements, including an improvement to the short circuiting logic… cutting off low ranking documents earlier in the process. This important optimization resulted in a speedup of about 9x for collapsing on certain fields!

Collapsing on string fields was further optimized with specialized code that worked on ord values instead of the string values. This doubled the performance yet again!

Please see the Solr Wiki for further documentation on all of result grouping’s capabilities and parameters.

Indexing JSON in Solr 3.1

Solr has been able to produce JSON results for a long time, by adding wt=json to any query. A new capability has recently been added to allow indexing in JSON, as well as issuing other update commands such as deletes and commits.

All of the functionality that was available through XML update commands can now be given in JSON. For example, you can index a document like so:

$ curl http://localhost:8983/solr/update/json -H 'Content-type:application/json' -d '
{
"add": {
"doc": {
"id" : "ISBN:978-0641723445",
"title" : "The Lightning Thief",
"author" : "Rick Riordan",
"series_t" : "Percy Jackson and the Olympians",
"cat" : ["book","hardcover"],
"genre_s" : "fantasy",
"pages_i" : 384,
"price" : 12.50,
"inStock" : true,
"popularity" : 10
}
}
}'

Of course, if you want the doc to be visible, you must do a commit. This could have been done by adding a commit=true parameter to the URL in the previous command, or we could have added a commit command within the JSON itself. This time we’ll issue a separate commit command.

curl "http://localhost:8983/solr/update/json?commit=true"

And now, we can query the Solr index and verify the document has been correctly added (requesting the results in JSON of course!) http://localhost:8983/solr/select?wt=json&indent=true&q=title:lightning

There’s more documentation on the Solr Wiki. To use this functionality, you’ll need to use a recent build of Solr 3.1 or greater.

Solr Result Grouping / Field Collapsing

Result Grouping, also called Field Collapsing, has been committed to Solr! This functionality limits the number of documents for each “group”, usually defined by the unique values in a field (just like field faceting).

You can think of it like faceted search, except instead of just getting a count, you get the top documents for that constraint or category. There are tons of potential use cases:

  • For web search, only show 1 or 2 results for a given website by collapsing on a site field.
  • For email search, only show 1 or 2 results for a given email thread
  • For e-commerce, show the top 3 products for each store category (i.e. “electronics”, “housewares”)
  • Hiding duplicate documents at query time.

In addition to being able to group by the values of a field, you can also group by the values of a function query. Given that geo search works as a function query, this also opens up possibilities for showing top query matches within 1 mile, between 1 and 2 miles, etc.

Just like faceting, we’ll be adding new functionality and making continual improvements. Result Grouping is documented on the Solr Wiki, and you will need a recent nightly build of Solr 4.0-dev to try it out (just make sure it’s dated after this post).

CSV output for Solr

Solr has been able to slurp in CSV for quite some time, and now I’ve finally got around to adding the ability to output query results in CSV also. The output format matches what the CSV loader can slurp.

Adding a simple wt=csv to a query request will cause the docs to be written in a CSV format that can be loaded into something like Excel.

http://localhost:8983/solr/select?q=ipod&fl=id,cat,name,popularity,price,score&wt=csv

id,cat,name,popularity,price,score
IW-02,"electronics,connector",iPod & iPod Mini USB 2.0 Cable,1,11.5,0.98867977
F8V7067-APL-KIT,"electronics,connector",Belkin Mobile Power Cord for iPod w/ Dock,1,19.95,0.6523595
MA147LL/A,"electronics,music",Apple 60 GB iPod with Video Playback Black,10,399.0,0.2446348

CSV formats tend to vary, so there are a number of parameters that allow you to customize the output. For example setting csv.escape= and csv.separator=%09 (a URL-encoded tab character) will use a tab separator and backslash escaping to match the default CSV format that MySQL uses.

http://localhost:8983/solr/select?q=ipod&fl=score,id&wt=csv&csv.escape=&csv.separator=%09

score id
0.98867977 IW-02
0.6523595 F8V7067-APL-KIT
0.2446348 MA147LL/A

The CSVResponseWriter is documented on the Solr Wiki, but you will need a recent nightly build (Solr 3.1-dev or Solr 4.0-dev) to try it out.

The Solr Bias

It’s perfectly rational for Solr’s competitors that use Lucene to argue against any changes that they see benefiting Solr. While perfectly rational and understandable on their part, it should not be allowed to have an impact on the actual development of Lucene and Solr, nor the merge designed to benefit both projects. Ongoing development of Lucene and Solr is about mutually beneficial improvement and promotion.

What may not be quite as immediately obvious is the fact that many a developer also harbors this bias. A developer who has custom code based on Lucene, may have had a coworker brainstorm or a manager ask “could we just use Solr?”. That developer may be 100% right and Solr may be the wrong solution, but the mere fact that they were forced to argue for their solution and against Solr can introduce a bias. Of course, even in those cases where Solr could be a fit, “Not Invented Here” syndrome can rear its ugly head.

I started off the Lucene/Solr merge with a more idealistic and care free attitude, not caring what things were called or where code was located, as long as we could improve both Lucene and Solr. I felt that if we could “get over the hump” and achieve a “one project, two downloads” philosophy among most of the committers, things would work out fine. That was naive. As long as the projects are still viewed as so separate by others, there will naturally exist a bias, a persistent force, acting to suppress Solr and even pushing on some of the committers. This bias, while sometimes rational and stems from self-interest, lacks technical merit and thus tends to manifest via other arguments such as the all powerful, always applicable, “users will be confused”.

The solution is to accept that the bias exists, and simply adopt frameworks and language to reduce it. For example, when factoring out a module, instead of “moving” it from Solr to Lucene, it should be a joint lucene/solr module. Simple changes in language like this will act to undercut the bias over time, as developers using the Java APIs will now be using both. This simple name change and way of thinking will no doubt cause much hand wringing and indirect attacks due to the existing bias, but developers interested in better products should simply push through that.

Ranges over Functions in Solr 1.4

Solr 1.4 contains a new feature that allows range queries or range filters over arbitrary functions.  It’s implemented as a standard Solr QParser plugin, and thus easily available for use any place that accepts the standard Solr Query Syntax by specifying the frange query type.  Here’s an example of a filter specifying the lower and upper bounds for a function:

fq={!frange l=0 u=2.2}log(sum(user_ranking,editor_ranking))

The other interesting use for frange is to trade off memory for speed when doing range queries on any type of single-valued field.  For example, one can use frange on a string field provided that there is only one value per field, and that numeric functions are avoided.

For example, here is a filter that only allows authors between martin and rowling, specified using a standard range query: fq=author_last_name:[martin TO rowling]

And the same filter using a function range query (frange): fq={!frange l=martin u=rowling}author_last_name

This can lead to significant performance improvements for range queries with many terms between the endpoints, at the cost of memory to hold the un-inverted form of the field in memory (i.e. a FieldCache entry - same as would be used for sorting). If the field in question is already being used for sorting or other function queries, there won’t be any additional memory overhead.

The following chart shows the results of a test of frange queries vs standard range queries on a string field with 200,000 unique values. For example, frange was 14 times faster when executing a range query / range filter that covered 20% of the terms in the field. For narrower ranges that matched less than 5% of the values, the traditional range query performed better.

Percent of terms covered Fastest implementation Speedup (how many times faster)
100% frange 43.32
20% frange 14.25
10% frange 8.07
5% frange 1.337
1% normal range query 3.59

Of course, Solr 1.4 also contains the new TrieRange functionality that will generally have the best time/space profile for range queries over numeric fields.

Filtered query performance increases for Solr 1.4

One of the many performance improvements in the upcoming Solr 1.4 release involves improved filtering performance. Solr 1.4 filters are both faster (anywhere from 30% to 80% faster to calculate intersections, depending on configuration), take less memory (40% smaller), and are more efficiently applied to the query during a search.

In previous Solr releases, filters were applied after the main query and thus had little impact on overall query performance. Filters are now checked in parallel with the query, resulting in greater speedups the fewer documents that match the filters.

Example: Adding a filter that matched 10% of a large index resulted in a 300% performance increase for a dismax query consisting of three words on a single field with proximity boost.

Related issues:

https://issues.apache.org/jira/browse/SOLR-1169

https://issues.apache.org/jira/browse/SOLR-1179

Solr scalability improvements

With CPU cores constantly increasing, there has been some major work done in Lucene/Solr to increase the scalability under multi-threaded load.

One bottleneck was synchronization around the checking of deleted docs in a Lucene IndexReader.  Since another thread could delete a document at any time, the IndexReader.isDeleted() call was synchronized.  It’s a very quick call, simply checking if a bit is set in a BitVector, but the problem was that it can be called millions of times in the process of satisfying a single query. The Read-only IndexReader feature allowed for the removal of this synchronization by prohibiting deletion.

The standard method for Lucene to read index files is via Java’s RandomAccessFile.  Reading a part of the file involves two calls, a seek() to position the file pointer followed by a read() to get the data.  For multiple threads to share the same RandomAccessFile instance, this obviously involves synchronization to avoid one thread changing the file pointer before another thread gets to read at the file position it set.   If the data to be read isn’t in the operating system cache, it’s even worse news… the synchronization causes all other reads to block while the data is retrieved from disk, even if some of those reads could have been quickly satisified.

The preferred solution would be to have a method on RandomAccessFile that accepted an offset to read from.  This could easily be implemented by the JVM via a pread() system call.  But since Sun has not provided this functionality, we need to use something else.  NIO’s FileChannel does have the type of method we are looking for:  FileChannel.read(ByteBuffer dst, long position)

Solr now uses the non-synchronizing NIO method of reading index files (via Lucene’s NIOFSDirectory) by default if you are on a non-Windows platform.  Windows systems default to the older method since it turns out to be faster than the new method - the reason being a long standing “bug” in Java that still synchronizes internally even when using FileChannel.read().

Solr’s standard LRU cache implementation use a synchronized LinkedHashMap.  A single cache could be checked hundreds or thousands of times during the course of a single request that involves faceting.  A non-blocking ConcurrentLRUCache was developed as an alternative implementation, and is now the default for Solr’s filter cache.  One user indicated that this has doubled their query throughput under ideal circumstances.

Solr 1.3 has read-only IndexReaders, but for the other scalability improvements, including the improved faceting, you’ll have to grab a nightly Solr build.

Solr Faceted Search Performance Improvements

Having performance issues with Solr’s faceted search and certain types of fields?  Help has arrived in the form of a new Solr faceting algorithm!  This new faceting implementation dramatically improves the performance of faceted search, making it suitable for a much wider range of applications.

The existing multivalued field faceting algorithm (where each document may have multiple values) steps over each term in the index for that field.  For each term, the set of documents that match that term is retrieved from the filterCache, and an intersection count is calculated with the set of documents that match the query.  This works well for fields with a limited number of terms (less than 1000), but not so great for fields with many terms.

The new method works by un-inverting the indexed field to be faceted, allowing quick lookup of the terms in the field for any given document.  It’s actually a hybrid approach - to save memory and increase speed, terms that appear in many documents (over 5%) are not un-inverted, instead the traditional set intersection logic is used to get the counts.

Results: up to 5000% increase in queries per second and up to 700% improvement in memory utilization.

More gory details and detailed benchmark results can be found at http://issues.apache.org/jira/browse/SOLR-475

Try it now with a Solr nightly/test development build dated 11/25/2008 or later.

lookup3ycs : a standard high performance string hash

I was surprised to discovered that there isn’t a good cross-platform hash function defined for strings. MD5, SHA, FVN, etc, all define hash functions over bytes, meaning that it’s under-specified for strings.

So I set out to create a standard 32 bit string hash that would be well defined for implementation in all languages, have very high performance, and have very good hash properties such as distribution. After evaluating all the options, I settled on using Bob Jenkins’ lookup3 as a base. It’s a well studied and very fast hash function, and the hashword variant can work with 32 bits at a time (perfect for hashing unicode code points). It’s also even faster on the latest JVMs which can translate pairs of shifts into native rotate instructions.

The only problem with using lookup3 hashword is that it includes a length in the initial value. This would suck some performance out since directly hashing a UTF8 or UTF16 string (Java) would require a pre-scan to get the actual number of unicode code points. The solution was to simply remove the length factor, which is equivalent to biasing initVal by -(numCodePoints*4). This slightly modified lookup3 I define as lookup3ycs.

So the definition of the cross-platform string hash lookup3ycs is:

Section titled “So the definition of the cross-platform string hash lookup3ycs is:”

The hash value of a character sequence (a string) is defined to be the hash of it’s unicode code points, according to lookup3 hashword, with the initval biased by -(length*4).

So by definition

lookup3ycs(k,offset,length,initval) == lookup3(k,offset,length,initval-(length*4))

AND

lookup3ycs(k,offset,length,initval+(length*

4)) == lookup3(k,offset,length,initval)

An obvious advantage of this relationship is that you can use lookup3 if you don’t have an implementation of lookup3ycs.

Here’s my optimized version for Java

Update: I’ve also included a 64 bit version called lookup3ycs64

Distributed Search for Solr

A new chapter in Solr scalability has been opened with the addition of distributed search!

http://wiki.apache.org/solr/DistributedSearch

Distributed Search splits an index into multiple shards, and queries across all the shards, combining the results and presenting a single merged response that looks like it came from a single server.

Solr’s current implementation uses SolrJ (the solr java client) to talk to other Solr servers via HTTP, in two main phases. The first phase collects matching document ids and scores, as well as doing any requested faceting. The second phase retrieves the stored fields for selected documents, does highlighting, and may include additional faceting requests to nail down exact facet counts.

Solr at Web 2.0 Expo Berlin

I’ll be giving a Solr presentation Nov 8th in Berlin, titled “Add Powerful Full Text Search to Your Web App with Solr”. Should be fun, just wish I had more free time while in Berlin…