Solr 5.4 Features

Aug 7, 2015

Yonik Seeley

Here’s an overview of some of the new features in Solr 5.4 Also see Solr Download Links and upcoming Features of the next Solr release.

filter() operator in lucene/solr query syntax

A filter query retrieves a set of documents matching a query from the Solr filter cache. This improves performance of additional queries that use the same filter clauses. All documents matching the query produce a score of 0 by default, but this can be changed by specifying a boost.

Filter Query Example:

description:HDTV OR filter(+promotion:tv +promotion_date:[NOW/DAY TO NOW/DAY+7DAY])

docValues fields now take less heap memory

Multi-valued fields with docValues (as well as binary docValues fields), previously had an on-heap index structure pointing to the on-disk (i.e. off-heap) values. This index has been moved off-heap, and directly read as needed from the index file.

Sparse encoding for docValues fields

Sparse docValues fields with less than 1% of documents containing a value in the field, are now internally encoded with a new SPARSE_COMPRESSED method to save storage space.

JSON Facet API “method” parameter

Terms/field faceting has a new parameter called “method” to give an execution hint while faceting on a field.

• method:uif - Stands for UninvertedField, a method of faceting indexed, multi-valued fields using top-level data structures that optimize for performance over NRT capabilities.
• method:dv - Stands for DocValues, a method of faceting indexed, multi-valued fields using per-segment data structures. This method mirrors faceting on real docValues fields but works by building on-heap docValues on the fly from the index when docValues aren’t available. This method is better for a quickly changing index.
• method:stream - This method creates each individual facet bucket (including any sub-facets) on-the-fly while streaming the response back to the requester. Currently only supports sorting by index order.

NOTE: currently, if a field is indexed with docValues, the dv method will be used even if method:uif is specified.

DocValue faceting performance

Faceting on DocValue fields and single-valued fields with the new Facet Module (JSON Facet API), has been optimized. When sorting by count, and when there are multiple hits expected per bucket, per-segment ords are collected before being mapped to global ords.

This optimization applies to the following field types:

• single-valued and multi-valued string fields with docValues
• indexed single-valued string fields
• indexed multi-valued string fields, when facet.method=dv is used

Here’s an example of the speedups obtained when faceting over 5M documents on different single-valued fields:

• Field with 10 unique values: +31%
• Field with 100 unique values: +29%
• Field with 1000 unique values: +59%
• Field with 10000 unique values: +88%
• Field with 1M unique values: +115%

Clusterstate migration tool

A collection API command MIGRATESTATEFORMAT that will migrate from an older-style shared clusterstate.json in Zookeeper, to per-collection cluster state files (state.json per collection).

Example:

  http://localhost:8983/solr/admin/collections?action=MIGRATESTATEFORMAT&collection=collection1

ConfigSets management API

A HTTP API to CREATE, DELETE, and LIST config sets (schema, solrconfig.xml, etc) in SolrCloud mode.

For example, the following command creates a new configset from an existing configset:

http://localhost:8983/solr/admin/configs?action=CREATE&name=booksConfig&baseConfigSet=genericTemplate

See ConfigSets API in the Solr reference guide for more details.

Force a shard leader

An expert-level FORCELEADER command has been added to help manually recover from a rare scenario where it looks like there are no replicas for a shard suitable to become the leader (i.e. when all of them are marked as recovering).

See Force Leader in the collections API section of the Solr reference guide.

More complex sorts for collapse post-filter

A new sort parameter was added to the collapse qparser to handle complex sorts.

fq={!collapse field=category sort='popularity desc, score desc'}

See Collapse and Expand Results in the Solr ref guide for more info on using the collapse post-filter.

Basic Auth support in SolrJ

SolrJ now has support for basic auth credentials. Those credentials need to be set for each request. Example:

  QueryRequest solrRequest = new QueryRequest(params);
  solrRequest.setBasicAuthCredentials(username, password);
  QueryResponse response = solrRequest.process(solrClient, "collection1");

Multi-Select Faceting

Facet & Analytics Performance