Skip to content

Blog

Solr 6.1 Features

Here’s an overview of some of the new features in Solr 6.1.

Download Solr 6.1 to try these features out and give us feedback! You can also check out upcoming features of the next Solr release.

A shortestPath Streaming Expression was added that implements a distributed breadth-first graph traversal to find the shortest paths in a directed directed graph.

Example:

shortestPath(collection,
from="john@company.com",
to="jane@company.com",
edge="from=to",
threads="6",
partitionSize="300",
fq="limiting query",
maxDepth="4")

gatherNodes Streaming Expression (coming soon)

Section titled “gatherNodes Streaming Expression (coming soon)”

The gatherNodes expression is a more general form of graph traversal than shortestPath, and can be used for more use cases.

Example:

gatherNodes(friends,
gatherNodes(friends,
search(articles, q=“body:(queryA)”, fl=“author”),
walk ="author->user”,
gather="friend"),
walk=“friend->user”,
gather="friend",
scatter=“roots, branches, leaves”)

ToleranteUpdateProcessorFactory will skip update commands that would otherwise cause subsequent updates in a batch to fail.

<updateRequestProcessorChain name="tolerant">
<processor class="solr.TolerantUpdateProcessorFactory"/>
<processor class="solr.DistributedUpdateProcessorFactory" />
<processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>

Passing update.chain=tolerant will use the processor chain defined above. One can also pass maxErrors=10 to limit the number of errors before aborting the complete update request. For example, if one is loading a large CSV file with millions of entries for the first time, it may be useful to abort early if every addition would fail due to a configuration error.

Filter creation for small cardinality sets (those that match few documents) now produces much less garbage. Up to 3x performance producing small sets (due to less GC overhead).

The HDFS block cache now skips caching “read once” scenarios such as index merges.

Solr 7 Features

Here’s an overview of some of the new features in Solr 7.0 Download Solr 7 to try these features out and give us feedback!

 

The now deprecated trie-based numeric fields use (and abuse) the full-text index to index parts of numbers to speed up range queries. The new Points-based numeric fields do not use the full-text index, but instead have a dedicated index structure designed specifically for multidimensional ranges over numbers: the BKD tree. The BKD tree index structure is smaller and faster for range queries, but is slightly slower for exact match queries.

Since Point fields do not currently support un-inversion (i.e. FieldCache), some search functionality requires that points have docValues enabled for fast per-document lookups. This includes sorting, function queries, {!graph}, and {!join} queries.

  Current template schemas define types like the following:

<fieldType name="pint" class="solr.IntPointField" docValues="true"/>
<fieldType name="pints" class="solr.IntPointField" docValues="true" multiValued="true"/>

Along with corresponding dynamic field types:

<dynamicField name="*_i" type="pint" indexed="true" stored="true"/>
<dynamicField name="*_is" type="pints" indexed="true" stored="true"/>

  Interested in how the underlying BKD data structure works? Here’s a great lesson from Robert Sedgewick, Princeton University on KD trees (BKD trees are just a variant of KD trees):

 

The JSON Facet API has a new parameter called refine that turns on two-phase refinement of partial facets during distributed faceting. This guarantees that the statistics (counts or other metrics) within returned facet buckets are accurate.

Partial facets are those facets that specify a limit and thus may not return all facet buckets from all shards in a distributed search. If refine is set to true on one of these partial facets, a second phase is used to “refine” the top buckets from the first phase, collecting information from other shards that did not yet contribute to those buckets. Without refinement, counts and statistics for the bucket can be incorrect. The second phase of faceting normally does not cause any additional HTTP requests since they are piggy-backed onto the normal second phase of distributed search that retrieves stored fields for the top document ids.

Example:

json.facet={
x : {
type : terms,
field : cat,
limit : 5,
refine : true
}
}

 

The min/max facet aggregations (facet functions) have been extended beyond numeric fields and functions to include single-valued string fields.

 

Streaming Expressions has many new functions:

  • movingAvg
  • arraySort
  • cumulative
  • anova
  • hist
  • array
  • sequence
  • finddelay
  • knn
  • describe
  • copyOfRange
  • sql
  • copyOf
  • distance
  • scale
  • rank
  • length
  • reverse

 

SolrCloud now has support for different replica types.

NRT stands for Near Real Time. This is the default and original replica type in SolrCloud. Updates flow from the leader to all replicas and are added to replica transaction logs as well as indexed. This is the only type of replica to support soft commits since TLOG and PULL replicas need hard commits to copy over new index segments.

Updates flow from the leader to all replicas and are added to replica transaction logs (tlogs) only. Replicas are kept up-to-date by pulling new index segment files from the leader. The transaction logs allow these replicas to recover and become a leader if necessary, as well as directly service real-time get requests.

Updates do not flow from the leader to replicas. Replicas are kept up-to-date by pulling new index segment files from the leader. This is similar to the original non-SolrCloud master-slave replication. Replicas of this type may not become leaders since updates would most likely be lost. Real-time get requests to a PULL replica are forwarded to the leader since these replicas lack transaction logs to find the most recent uncommitted updates.

 

  • The default response format has been changed from XML to indented JSON. Add wt=xml to the request obtain an XML response, and add indent=off if you wish to turn off indenting.
  • The new v2 API, exposed at /api/ is now the preferred API (esp for using the collections API), but /solr/ continues to be supported.
  • The alternate “Analytics Component” in the contrib modules was bumped to v2, with distributed support and a new JSON-based request syntax
  • Auto-scaling framework that allows Solr to place new replicas based on metrics such as free disk space.
  • The standard lucene/solr query parser now defaults to sow=false, meaning that for text fields, it does not split on whitespace before handing the text to the analyzer. This enabled multi-word synonyms to be matched by the analyzer.
  • When a collection is created without specifying a configset, the new ‘_default’ configset is now used. It is data-driven (schemaless), and indexes strings as analyzed text in addition to using a copyField to *_str field suitable for sorting or faceting.

See CHANGES for more detailed upgrade notes. Also see the official release notes on the Solr wiki. The Solr Reference Guide should contain other upgrade info.

Solr 6 Features

Here’s an overview of some of the new features in Solr 6.0.

Download Solr 6 to try these features out and give us feedback! You can also check out upcoming features of the next Solr release.

Parallel SQL queries across SolrCloud collections. The SQL engine is built on top of Solr’s Streaming API (Streaming Expressions), which provides support for parallel relational algebra and real-time map-reduce.

  • SQL statements are compiled to Streaming Expressions for parallel execution across SolrCloud worker nodes.
  • SolrCloud collections are abstracted as Relational Tables.
  • Full support for Lucene/Solr query syntax in the WHERE clause.
  • Many operations such as grouping/rollups can be automatically parallelized, utilizing the real-time MapReduce capabilities of the streaming expressions.
  • Grouping/rollup operations can be pushed down and leverage the JSON Facet API for increased performance.
  • Currently works in SolrCloud mode only… no standalone mode yet.
  • SQL functionality is currently experimental and incomplete (for example underlying streaming join functionality is still in the process of being integrated.)

Examples:

select category, count(*), sum(inventory), min(price), max(price), avg(outstanding) from collection1 where text='4k HDTV' group by category order by sum(inventory) asc limit 10
select id,category from collection1 where category = '(dvd OR bluray)' order by category desc limit 100
select fieldA, fieldB, count(*), sum(fieldC), avg(fieldY)
from collection1
where fieldC = 'term1 term2'
group by fieldA, fieldB
having sum(fieldC) > 1000
order by sum(fieldC) asc
limit 100

See the Parallel SQL documentation for more info.

There is a SQL Request Handler mapped to the /sql endpoint. Only SolrCloud collections can currently be searched with the SQL handler.

Example:

$ curl http://localhost:8983/solr/techproducts/sql -d "stmt=select id from techproducts"
{"result-set":{"docs":[
{"id":"EN7800GTX/2DHTV/256M"},
{"id":"100-435805"},
{"id":"UTF8TEST"},
{"id":"SOLR1000"},
{"id":"9885A004"}
]}}

Solr has a new JDBC driver that may be used to access the new SQL functionality. The Solr JDBC driver has been tested with DbVisualizer, Apache Zeppelin, and SQuirreL SQL so far.

The general form of the JDBC connection string is:

jdbc:solr://SOLR_ZK_CONNECTION_STRING?collection=COLLECTION_NAME

The JDBC driver is not yet documented in the Ref Guide, so see https://issues.apache.org/jira/browse/SOLR-8521 for more documentation in the meantime.

A number of distributed join operations have been added to streaming expressions:

  • innerJoin
  • leftOuterJoin
  • hashJoin
  • outerHashJoin

Example:

innerJoin(
search(collection1, q=*:*, fl="fieldA, fieldB, fieldC", ...),
search(collection2, q=*:*, fl="fieldA, fieldD, fieldE", ...),
on="fieldA=fieldA"
)

The rollup streaming expression groups tuples by common field values and emits the rollup value along with other specified metrics.

Example:

rollup(
search(collection1, qt="/export"
q="*:*",
fl="id,manu,price",
sort="manu asc"),
over="manu"),
count(*),
max(price)
)

The facet streaming expression is much like the rollup expression, but it pushes down the computation to the leaves using the JSON Facet API.

Example:

facet(
techproducts,
q="*:*",
buckets="manu",
bucketSorts="count(*) desc",
bucketSizeLimit=1000,
count(*),
sum(price),
max(popularity)
)

Many other Streaming Expressions were added for the Solr 6 release. At this point, they should all be documented in the Streaming Expressions section of the Solr Ref Guide.

   

A basic graph traversal query that follows nodes to edges, optionally filtering during traversal.

Example: Assume we have documents that represent people, and each document has a field called “parent_id” which lists the parents. This example query matches “Philip J. Fry” and all of his ancestors:

fq={!graph from=parent_id to=id}id:"Philip J. Fry"

The main argument to the graph query defines the root set, in this case id:"Philip J. Fry". The graph query then iteratively follows the parent_id field to documents with corresponding id fields (i.e. for each iteration, the values for parent_id in the current set are matched to id field of all documents). The basic graph query is equivalent to a repeated join query.

Graph query parameters:

  • from - The field used in the current set of documents to match to the to field in the set of destination documents.- to - The field used to find matches in the values obtained from the from fields of the starting set of documents.- traversalFilter - A filter query that is applied on each iteration.
  • returnRoot - Controls whether the root set of documents should be included. Defaults to “true”.
  • returnOnlyLeaf - If true, only returns leaf documents. Defaults to “false”.
  • maxDepth - The maximum number of iterations before graph traversal stops. Defaults to -1 (unlimited).

NOTE: This graph query only traverses edges in the same index (i.e. it does not traverse edges across different nodes/cores). Distributed graph traversal is being developed and will be in future versions of Solr 6.x

Default scoring now uses Okapi BM25 by default. You can enable the old tf-idf vector space similarity by using ClassicSimilarity.

Here is an example of how to use classic tf-idf similarity just for the “text” fieldType in the Solr schema:

<fieldType name="text" class="solr.TextField">
<analyzer class="org.apache.lucene.analysis.standard.StandardAnalyzer"/>
<similarity class="solr.ClassicSimilarityFactory"/>
</fieldType>

For BM25, one can tweak the scoring function on a per-fieldType basis. For example:

<fieldType name="text2" class="solr.TextField">
<analyzer class="org.apache.lucene.analysis.standard.StandardAnalyzer"/>
<similarity class="solr.BM25SimilarityFactory">
<float name="k1">1.2
<float name="b">0.75
</similarity>
</fieldType>

For those new to full-text search terminology, “similarity” produces a score for how similar a document is to a full-text query. When determining this score, document statistics as well as corpus statistics are used. Some scoring factors include:

  • The number of times a search term appears in the document field. More matches produces a higher score.
  • The size of the document field. Longer fields produce a lower score, with the idea being that for a given number of term matches, shorter is better (more specific match)
  • The average length of the field across the entire corpus (BM25 considers this, classic tf-idf does not)
  • How common the query terms are across the entire corpus. The idea being that rarer terms carry more information. For example, if I searched for “blue whale”, all else being equal, I’d probably want things about whales to score higher than things about blue.

Real-Time Get (normally handled at the /get URL) now handles filters (fl parameters) to restrict matching documents.

Example:

curl "http://localhost:8983/solr/demo/get?id=book1,book2,book3&fl=security:group1"

An experimental version of CDCR (Cross Data Center Replication) has been added that supports an active-passive configuration.

Updates are asynchronously sent from the active cluster leader to the passive cluster leader. Since updates are asynchronously relayed from the leader’s transaction logs, temporary connectivity issues between data centers can be tolerated with no interruption in service for the primary DC.

Here is the in-progress CDCR documentation. The documentation link is likely to change once CDCR moves out of it’s “experimental” phase.

Solr 5.5 Features

Here’s an overview of some of the new features in Solr 5.5 Also see Solr Download Links and upcoming Features of the next Solr release.

Return docValues fields like stored fields

Section titled “Return docValues fields like stored fields”

In previous Solr versions, returning the top N documents only retrieved field values from the row-store (i.e. fields where “stored” is true). Values will now also be retrieved from docValues (which are essentially column stored) for fields where “stored” is false. This avoids the need to duplicate the value in the row store if it’s already column stored.

A "facet-trace" section has been added to the debug info for JSON Facets. The exact format is subject to change, but the current information includes information about each facet command, including the processor used to execute the facet and the domain size. This information is included recursively included for each facet bucket for facets with sub-facets.

Example request:

$ curl http://localhost:8983/solr/techproducts/query -d 'q=*:*&rows=0&debug=true&
json.facet={
categories:{
type : terms,
field : cat
}
}

Example response debug info:

[...]
"facet-trace":{
"processor":"FacetQueryProcessor",
"elapse":0,
"query":null,
"domainSize":32,
"sub-facet":[{
"processor":"FacetFieldProcessorUIF",
"elapse":0,
"field":"cat",
"limit":10,
"numBuckets":16,
"domainSize":32}]},
[...]

facet.method=uif parameter causes traditional field faceting to delegate to the JSON Facet API with method=uif. This is roughly equivalent to what the Solr 4 default faceting method was for multi-valued fields. It is optimized for performance over static indexes rather than NRT (quickly changing indexes).

The compression mode for stored fields can now be specified via codecFactory in solrconfig.xml See the Codec Factory section in the Solr reference guide for more details.

Generic support was added for making collection APIs async. Async support was added for the following commands: delete/reload collection, create/delete alias, create/delete shard, delete replica, add/delete replica property, add/remove role, overseer status, balance shard unique, rebalance leaders, modify collection, migrate state format.

See Asynchronous Calls in the Solr ref guide.

There is a new experimental BlockJoinFacetComponent for calculating facets by a child.facet.field parameter with a {!parent} query. The component is not enabled by default. Note that this component is unrelated to the block join faceting support in the JSON Facet API.

The XML query parser, registered as “xmlparser” is a direct interface to Lucene’s XMLQueryParser (CoreParser). Personal recommendation: avoid the use of this query parser unless you have very unusual/specific needs.

Example:

curl http://localhost:8983/solr/techproducts/query -d 'debugQuery=true&
q={!xmlparser}
<BooleanQuery>
<Clause occurs="must"> <TermQuery fieldName="name">ipod</TermQuery> </Clause>
<Clause occurs="must"> <TermQuery fieldName="manu">apple</TermQuery> </Clause>
</BooleanQuery>
'

This parser does not do any text analysis on terms, so provided terms will need to match what is in the index exactly (i.e. you will need to do things like lowercasing and stemming yourself). Good backward compatibility is unlikely with this parser as it exposes more internal implementation details. Term queries on fields such as numeric fields, enum fields, and boolean fields will only work if you know the internal term representation in the index.

See XmlQParser in the Solr ref guide for more info.

A configset, or configuration set, is a set of config files for a Solr collection. For SolrCloud mode, an upconfig option has been added to the /bin/solr script to upload a configset to zookeeper. A matching downconfig option has been added to download a configset from zookeeper.

For examples and documentation, see bin/solr Zookeeper Operations in the Solr reference guide.

There is an internal CheckHdfsIndex class that can be run from the command line for HDFS indexes like CheckIndex can be run for normal indexes. Example:

java -cp "./server/solr-webapp/webapp/WEB-INF/lib/*:./server/lib/ext/*" -ea:org.apache.lucene... org.apache.solr.index.hdfs.CheckHdfsIndex /path/to/my/index/

  For reference, here is also the command to run CheckIndex on a local (non-HDFS) lucene index:

java -cp "./server/solr-webapp/webapp/WEB-INF/lib/*:./server/lib/ext/*" -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex ./example/techproducts/solr/techproducts/data/index

Which will result in the following output:

Opening index @ ./example/techproducts/solr/techproducts/data/index
Segments file=segments_2 numSegments=1 version=6.0.0 id=bv1fdquc5dh3nvcf4jxiwfow4 format= userData={commitTimeMSec=1456073865693}
1 of 1: name=_0 maxDoc=32
version=6.0.0
id=bv1fdquc5dh3nvcf4jxiwfow3
codec=Lucene60
compound=false
numFiles=13
size (MB)=0.026
diagnostics = {java.runtime.version=1.8.0_40-b25, java.vendor=Oracle Corporation, java.version=1.8.0_40, java.vm.version=25.40-b25, lucene.version=6.0.0, os=Mac OS X, os.arch=x86_64, os.version=10.11.2, source=flush, timestamp=1456073865745}
no deletions
test: open reader.........OK [took 0.059 sec]
test: check integrity.....OK [took 0.000 sec]
test: check live docs.....OK [took 0.000 sec]
test: field infos.........OK [25 fields] [took 0.000 sec]
test: field norms.........OK [5 fields] [took 0.001 sec]
test: terms, freq, prox...OK [1187 terms; 1813 terms/docs pairs; 1496 tokens] [took 0.025 sec]
test: stored fields.......OK [356 total field count; avg 11.1 fields per doc] [took 0.013 sec]
test: term vectors........OK [3 total term vector count; avg 1.0 term/freq vector fields per doc] [took 0.006 sec]
test: docvalues...........OK [0 docvalues fields; 0 BINARY; 0 NUMERIC; 0 SORTED; 0 SORTED_NUMERIC; 0 SORTED_SET] [took 0.000 sec]
No problems were detected with this index.
Took 0.246 sec total.

Multi-Select Faceting

NOTE: This uses syntax from the upcoming Solr 5.4 release. If you are using Solr 5.2 or 5.3, specify domain:{excludeTags:mytag} as excludeTags:mytag.

Multi-select faceting is a powerful faceting style that allows users to see and select multiple facet constraints (facet values) for certain facets. This example uses Solr’s JSON Facet API along with filter tagging and excluding to implement this style of faceting.

Let’s say we have 3 facets, Size, Color, and Brand. This is multi-select faceting because for the Color and Brand facets, we want the user to be able to select multiple constraints (values). The Size facet is single-select since we’ve decided that customers in general will only be interested in one size at a time.

Here is our super-fancy ASCII UI, after the user has searched for "running shorts":

=== Size === === Color === === Brand ===
[Small] (7) [ ] Red (2) [ ] Nike (7)
[Medium] (5) [ ] Blue (8) [ ] Adidas (5)
[Large] (6) [ ] Green (3) [ ] Reebok (4)
[ ] Black (5) [ ] Under Armour (2)
(Top matches sorted by popularity displayed here... use your imagination!)

Note that the Color and Brand facets have checkboxes to indicate which constraints have been selected. We’re starting off with no constraints. Below the facets is where we would display the top matching items, along with pretty pictures, prices, etc.

 

The user selects “Blue” so we add that as a filter and re-issue the request (we’re using Solr’s JSON Facet API):

&q="running shorts"
&fq=color:Blue
&json.facet={
sizes:{type:terms, field:size},
colors:{type:terms, field:color},
brands:{type:terms, field:brand}
}

We get back the response and update our UI from that data:

=== Size === === Color === === Brand ===
[Small] (3) [ ] Red (0) [ ] Nike (3)
[Medium] (2) [x] Blue (8) [ ] Adidas (2)
[Large] (3) [ ] Green (0) [ ] Reebok (2)
[ ] Black (0) [ ] Under Armour (1)
(Top Blue running shorts displayed here)

What’s right: The Size and Brand facets now reflect the fact that we’ve selected Blue, and that’s what we wanted. Our list of top matches also only includes Blue things, just as we wanted.

What’s wrong: Because we filtered out anything that wasn’t Blue, we get back 0 counts for other colors! But we still want the other color information so the customer can select additional colors.

 

When we compute the multi-select Color facet, we want to ignore any constraints (filters) on that facet so we will get back the correct counts for other colors. To accomplish this, we can tag filters and then selectively exclude filters (i.e. pretend they don’t exist) by tag when faceting.

The same thing applies to the multi-select Brand facet… we want any Brand selections to affect everything else (including all other facets), except for the Brand facet itself.

When the user selects Blue, we add that as a filter tagged with COLOR and re-issue the request:

&q="running shorts"
&fq={!tag=COLOR}color:Blue
&json.facet={
sizes:{type:terms, field:size},
colors:{type:terms, field:color, domain:{excludeTags:COLOR} },
brands:{type:terms, field:brand, domain:{excludeTags:BRAND} }
}

Now when we get back our response, it still includes the other Colors in the Color facet.

=== Size === === Color === === Brand ===
[Small] (3) [ ] Red (2) [ ] Nike (3)
[Medium] (2) [x] Blue (8) [ ] Adidas (2)
[Large] (3) [ ] Green (3) [ ] Reebok (2)
[ ] Black (5) [ ] Under Armour (1)
(Top Blue running shorts displayed here)

The domain is the set of documents that facets will be calculated over. In the JSON Facet API, the domain keyword/command is normally used to change the domain before the facets are calculated.

In our example above, we specified domain:{excludeTags:COLOR} for the colors facet. This will re-calculate the facet domain as if any filters tagged with COLOR were not applied.

 

Ok, now the user selects Black as well.

We naively add an additional filter, fq={!tag=COLOR}color:Black to the request, just as we would with traditional single-select faceting.

What’s wrong: Everything! Our request matches nothing and we get back all 0’s.

This is because filters are logically intersected. We searched for things that were Blue AND Black, and that will match nothing (assuming our items only have a single color). What we really want is Blue OR Black.

 

We need the logical OR, or union, of all the selected colors.

&q="running shorts"
&fq={!tag=COLOR}color:(Blue Black)
&json.facet={
sizes:{type:terms, field:size},
colors:{type:terms, field:color, domain:{excludeTags:COLOR} },
brands:{type:terms, field:brand, domain:{excludeTags:BRAND} }
}
=== Size === === Color === === Brand ===
[Small] (5) [ ] Red (2) [ ] Nike (5)
[Medium] (4) [x] Blue (8) [ ] Adidas (3)
[Large] (4) [ ] Green (3) [ ] Reebok (3)
[x] Black (5) [ ] Under Armour (2)
(Top Blue and Black running shorts displayed here)

Note that the counts on the other facets increased to reflect the larger domain (it includes both blue and black items).

Although our simple example just dealt with facet counts, multi-select faceting via excludeTags works with the broad range of features in the JSON Facet API. The domain change will apply to everything else under that facet, including Sub-facets and Facet Functions.

   

Tagging and excluding filters with excludeTags

Section titled “Tagging and excluding filters with excludeTags”

Solr filter queries (fq parameters) can be tagged with arbitrary strings using the localParams {!tag=mystring} syntax. Example: fq={!tag=COLOR}color:Blue

  • Multiple filters can be tagged with the same tag. Example: fq={!tag=foo}one_filter&fq={!tag=foo}another_filter
  • A single filter may be tagged with multiple tags. Example: fq={!tag=tag1,tag2,tag3}my_field:my_filter

During faceting, the facet domain may be changed to exclude filters that match certain tags via the excludeTags keyword. It’s as if the filter was never specified for that specific facet. This is useful for implementing multi-select faceting Example: colors:{type:terms, field:color, domain:{excludeTags:COLOR}}

  • excludeTags can be multi-valued comma-separated string. Example: excludeTags:"tag1,tag2"
  • excludeTags can be a JSON array of tags. Example: excludeTags:["tag1","tag2"]
  • One can exclude tags that are not used in the current request. This makes constructing requests simpler since you don’t need to worry about changing the faceting part of the request based on what filters have been applied.
  • For nested facets, excludeTags can appear at any level of the hierarchy. They do not currently “stack” though. If a parent facet has excludeTags:tag1 and a child facet wants to additionally exclude tag2 filters, then they must currently do so explicitly with excludeTags:“tag1,tag2”. Nested exclusions are experimental and subject to change.

Solr 5.4 Features

Here’s an overview of some of the new features in Solr 5.4 Also see Solr Download Links and upcoming Features of the next Solr release.

filter() operator in lucene/solr query syntax

Section titled “filter() operator in lucene/solr query syntax”

A filter query retrieves a set of documents matching a query from the Solr filter cache. This improves performance of additional queries that use the same filter clauses. All documents matching the query produce a score of 0 by default, but this can be changed by specifying a boost.

Filter Query Example:

description:HDTV OR filter(+promotion:tv +promotion_date:[NOW/DAY TO NOW/DAY+7DAY])

docValues fields now take less heap memory

Section titled “docValues fields now take less heap memory”

Multi-valued fields with docValues (as well as binary docValues fields), previously had an on-heap index structure pointing to the on-disk (i.e. off-heap) values. This index has been moved off-heap, and directly read as needed from the index file.

Sparse docValues fields with less than 1% of documents containing a value in the field, are now internally encoded with a new SPARSE_COMPRESSED method to save storage space.

Terms/field faceting has a new parameter called “method” to give an execution hint while faceting on a field.

  • method:uif - Stands for UninvertedField, a method of faceting indexed, multi-valued fields using top-level data structures that optimize for performance over NRT capabilities.
  • method:dv - Stands for DocValues, a method of faceting indexed, multi-valued fields using per-segment data structures. This method mirrors faceting on real docValues fields but works by building on-heap docValues on the fly from the index when docValues aren’t available. This method is better for a quickly changing index.
  • method:stream - This method creates each individual facet bucket (including any sub-facets) on-the-fly while streaming the response back to the requester. Currently only supports sorting by index order.

NOTE: currently, if a field is indexed with docValues, the dv method will be used even if method:uif is specified.

Faceting on DocValue fields and single-valued fields with the new Facet Module (JSON Facet API), has been optimized. When sorting by count, and when there are multiple hits expected per bucket, per-segment ords are collected before being mapped to global ords.

This optimization applies to the following field types:

  • single-valued and multi-valued string fields with docValues
  • indexed single-valued string fields
  • indexed multi-valued string fields, when facet.method=dv is used

Here’s an example of the speedups obtained when faceting over 5M documents on different single-valued fields:

  • Field with 10 unique values: +31%
  • Field with 100 unique values: +29%
  • Field with 1000 unique values: +59%
  • Field with 10000 unique values: +88%
  • Field with 1M unique values: +115%

A collection API command MIGRATESTATEFORMAT that will migrate from an older-style shared clusterstate.json in Zookeeper, to per-collection cluster state files (state.json per collection).

Example:

http://localhost:8983/solr/admin/collections?action=MIGRATESTATEFORMAT&collection=collection1

A HTTP API to CREATE, DELETE, and LIST config sets (schema, solrconfig.xml, etc) in SolrCloud mode.

For example, the following command creates a new configset from an existing configset:

http://localhost:8983/solr/admin/configs?action=CREATE&name=booksConfig&baseConfigSet=genericTemplate

See ConfigSets API in the Solr reference guide for more details.

An expert-level FORCELEADER command has been added to help manually recover from a rare scenario where it looks like there are no replicas for a shard suitable to become the leader (i.e. when all of them are marked as recovering).

See Force Leader in the collections API section of the Solr reference guide.

More complex sorts for collapse post-filter

Section titled “More complex sorts for collapse post-filter”

A new sort parameter was added to the collapse qparser to handle complex sorts.

fq={!collapse field=category sort='popularity desc, score desc'}

See Collapse and Expand Results in the Solr ref guide for more info on using the collapse post-filter.

SolrJ now has support for basic auth credentials. Those credentials need to be set for each request. Example:

QueryRequest solrRequest = new QueryRequest(params);
solrRequest.setBasicAuthCredentials(username, password);
QueryResponse response = solrRequest.process(solrClient, "collection1");

Facet & Analytics Performance

These benchmarks compare the performance of the new JSON Facet API with it’s “performance-first” architecture, and the existing (legacy) Solr Facets.

Test index details:

documents: 5M
index segments: 25
index size: 1.74GB
6 single valued string fields with 10, 100, 1000, 10000, 100000, 1000000 unique values respectively.
6 single valued integer fields as above.
6 multi-valued string fields with 1-5 values per field, with 10, 100, 1000, 10000, 100000, 1000000 unique values respectively.
6 multi-valued integer fields as above.
5% chance of any given field having no values for a particular document.

Test requests details:

Base test query and filters (the domain) matches 2,161,827 documents.
Single client thread (and both requests only use a single internal thread per request).
Single warm-up run per implementation that is discarded.
Multiple runs across all fields, with fastest time being taken for each field.

These benchmarks test faceting on one field and finding the average value in another field per facet bucket.

JSON Facet API command:

json.facet={
f:{
type : terms,
field : m100_5_ss,
facet : { mean : "avg(s10_s)" }
}
}

Legacy Facet command:

facet=true&
stats=true&
stats.field={!tag=stat1+mean=true}s10_s&
facet.pivot={!stats=stat1}m100_5_ss&
f.m100_5_ss.facet.limit=10

Only sorting by count was tested since legacy facets (pivot + stats component) do not support sorting buckets by anything else.

Count Distinct in Solr

A 100% accurate count of distinct values (count distinct) is not generally possible without actually observing all of the values together. However there are a number of ways to estimate the count.

The unique facet function is Solr’s fastest implementation to calculate the number of distinct values. It always provides exact counts on a single Solr node. For distributed search over multiple nodes, it provides exact counts when the number of values per node does not exceed 100 (by default).

When the number of unique values does exceed 100 in any given shard, the following algorithm is used:

  • It estimates the count by sending the top 100 results from each shard along with the total exact “unique” count for each shard.
  • totalSeen is the number of actual results we saw from all shards (i.e. not deduped yet).
  • uniqueSeen is the number of unique values we saw from all shards (i.e. deduped).
  • notSeen is the number of unique values from each shard that were not sent (because of the 100 cutoff).
  • factor = uniqueSeen / totalSeen (i.e. what fraction of values that we saw were unique)
  • estimate = uniqueSeen + ( notSeen * factor ) (i.e. we simply apply the factor to the number of values we didn’t see)

Example use:

$ curl http://localhost:8983/solr/techproducts/query -d '
q=*:*&
json.facet={
x : "unique(manu_exact)" // manu_exact is the manufacturer indexed as a single string
}'

For more facet functions, adding facet functions to each facet bucket, or sorting by facet function, see Solr Facet Functions

(New in Solr 5.2) The HyperLogLog algorithm was developed as an advanced statistical method of estimating the distinct number of values without using too much memory. It does add more calculation overhead however, and is thus slower than Solr’s “unique” facet function.

HyperLogLog requires very high quality hashes for accurate estimation. Solr uses a port of MurmurHash3 for Java to calculate these hashes.

The Facet Analytics Module and the older Stats component both have support for HyperLogLog as of Solr 5.2

A new “hll” facet function was added as an alternative to the existing faster (but less accurate for high cardinality) “unique” function.

To get the unique number of manufacturers using the HyperLogLog algorithm:

$ curl http://localhost:8983/solr/techproducts/query -d '
q=*:*&
json.facet={
x : "hll(manu_exact)" // manu_exact is the manufacturer indexed as a single string
}'

If we want the number of unique manufacturers per bucket of a facet:

$ curl http://localhost:8983/solr/techproducts/query -d '
q=*:*&
json.facet={
categories: {
type : terms,
field : cat,
facet : {
x : "hll(manu_exact)"
}
}
}'

And we get a response containing:

"facets":{
"count":32,
"categories":{
"buckets":[{
"val":"electronics",
"count":12,
"x":9},
{
"val":"currency",
"count":4,
"x":4},
[...]

To get the unique number of manufacturers per facet bucket using the Stats component:

$ curl http://localhost:8983/solr/techproducts/query -d '
q=*:*&
stats=true&
facet=true&
stats.field={!tag=stat1 cardinality=true}manu_exact&
facet.pivot={!stats=stat1}cat'

And we get a response containing:

"facet_counts":{
"facet_queries":{},
"facet_fields":{},
"facet_dates":{},
"facet_ranges":{},
"facet_intervals":{},
"facet_heatmaps":{},
"facet_pivot":{
"cat":[{
"field":"cat",
"value":"electronics",
"count":12,
"stats":{
"stats_fields":{
"manu_exact":{
"cardinality":9}}}},
{
"field":"cat",
"value":"currency",
"count":4,
"stats":{
"stats_fields":{
"manu_exact":{
"cardinality":4}}}},
[...]

Here is a performance comparison of the different implementations in various scenarios.

Test configuration:

Index:
documents: 5M
index segments: 25
index size: 1.74GB
6 single valued string fields with 10, 100, 1000, 10000, 100000, 1000000 unique values respectively.
6 single valued integer fields as above.
6 multi-valued string fields with 1-5 values per field, with 10, 100, 1000, 10000, 100000, 1000000 unique values respectively.
6 multi-valued integer fields as above.
5% chance of any given field having no values for a particular document.
Queries:
Base test query and filters (the domain) matches 2,161,827 documents.
Single client thread (and both requests only use a single internal thread per request).
Single warm-up run per implementation that is discarded.
Multiple runs across all fields, with fastest time being taken for each field.

Legacy Faceting command (stats component + pivot faceting):

facet=true&stats=true&stats.field={!tag=stat1+cardinality=true}s10_s&facet.pivot={!stats=stat1}m100_5_ss&f.m100_5_ss.facet.limit=10

JSON Faceting command (new Facet Module):

json.facet={
f:{
type : terms,
field : m100_5_ss,
facet : { stat1:"hll(s10_s)" }
}
}

This first test facets on the multi-valued string field m100_5_ss (it has up to 5 values per field, and 100 unique values in total). Then for the top 10 buckets, the cardinality of the single-valued string field (the “stat field”) is calculated. The chart below shows performance for different number of unique values in the stat field.

This next test reverses the 2 fields above, first faceting on the single valued string field s100_s (it has 100 unique values in total) and then calculating cardinality over multi-valued string fields with different numbers of unique values in the index.

The last test facets on the integer field s100_i and then calculates cardinality over another integer field with varying number of unique terms in the index.

Solr 5.3 Features

Here’s an overview of some of the new features in Solr 5.3 Also see Solr Download Links and upcoming Features of the next Solr release.

The JSON Facet API can now change the domain for facet commands, essentially doing a block join and moving from parents to children, or children to parents before calculating the facet data.

For example, if you indexed chapters with pages as nested child documents, then you could map from chapters to pages before faceting by adding the following parameter to the facet command:

domain : { blockChildren : "type:chapter" }

Or if you started with pages, you could map to chapters with

domain : { blockParent : "type:chapter" }

Note that in both cases, we provide the parent filter (how parent documents are defined) of “type:chapter” regardless of which direction we are mapping.

See this Nested Objects tutorial for complete examples of combining faceting and block join / nested documents.

 

Major improvements in performance of the new Facet Module / JSON Facet API. See the facet performance benchmarks for more details and benchmark results.  

Just like the JSON Facet API, pivot facets can how nest other facet types such as range and query facets.

Example:

&facet=true
&facet.range={!tag=r1}price
&f.price.facet.range.start=0
&f.price.facet.range.end=100
&f.price.facet.range.gap=10
&facet.query={!tag=q1}popularity:[8 TO 10]
&facet.pivot={!range=r1 query=q1}category

The equivalent in the JSON Facet API would be:

json.facet={
categories : {
type : terms,
field : category,
facet : {
r1 : {
type : range,
start : 0,
end : 100,
gap : 10
},
q1 : { query : "popularity:[8 TO 10]" }
}
}
}

The MoreLikeThis QParser mlt now supports all options provided by the MLT Handler. The query parser is much more versatile than the handler as it works in cloud mode as well as anywhere a normal query can be specified.

Example (on techproducts index):

q={!mlt qf=name mintf=1 mindf=1}SP2514N

More documentation on the mlt parser can be found in the Solr Ref Guide

The new SchemaRequest Java class in SolrJ can be used to make requests to the Schema API.

Also see the Solr Schema API itself in the ref guide.

Scoring mode for query-time join and block join

Section titled “Scoring mode for query-time join and block join”

Solr’s pseudo-join query parser has a new optional attribute score that can be used specify the scores produced on the resulting documents. It’s value can be min, max,avg,or total.

Query-time join example:

q={!join from=author_id to=id score=total}blog_text:awesome

Block join example:

q={!parent of=type:author score=total}blog_text:awesome

See Nested Objects in Solr for more information on nested documents and block join.

Lucene/Solr query syntax (i.e. Solr’s dialect of the lucene syntax) now supports nested C-style comments.

+cat:electronics /* this is a comment */ +name:HDTV

Smile is a binary data interchange format that is very close to Solr’s own “javabin” (encoded sizes are very close). Adding wt=smile to a request will cause the response to come back in this format.

A second parameter has been added to the field function to select the minimum or maximum value of a multi-valued field with docValues.

Example:

sort=field(my_dv_field,max) asc

In addition to many other improvements in the security framework, Solr now includes an AuthenticationPlugin implementing HTTP Basic Auth that stores credentials securely in ZooKeeper. This is a simple way to require a username and password for anyone accessing Solr’s admin screen or APIs.

See the Basic Authentication Plugin section of the Solr ref guide under the Securing Solr section.

JSON Facet API

Related Pages

Solr 5 has a completely re-written faceted search and analytics module with a structured JSON API to control the faceting and analytics commands. NOTE: Some examples use syntax only supported in later Solr 5 releases, or even Solr 6. Download a recent Solr release or snapshot to try them out.

The structured nature of nested sub-facets are more naturally expressed in a nested structure like JSON rather than the flat structure that normal query parameters provide.

Goals of the new Faceting Module:

  • First class JSON support
  • Easier programmatic construction of complex nested facet commands
  • Support a much more canonical response format that is easier for clients to parse
  • First class analytics support
  • Ability to sort facet buckets by any calculated metric
  • Support a cleaner way to do distributed faceting
  • Support better integration with other search features

Of course if you prefer to use Solr’s existing faceting capabilities, that’s fine too. You can even use both at once if you want!

UPDATE: The JSON Facet API is now part of the JSON Request API, so a complete request may be expressed in JSON.

Some of the ease-of-use enhancements over traditional Solr faceting come from the inherent nested structure of JSON. As an example, here is the faceting command for two different range facets using Solr’s flat legacy API:

&facet=true
&facet.range={!key=age_ranges}age
&f.age.facet.range.start=0
&f.age.facet.range.end=100
&f.age.facet.range.gap=10
&facet.range={!key=price_ranges}price
&f.price.facet.range.start=0
&f.price.facet.range.end=1000
&f.price.facet.range.gap=50

And here is the equivalent faceting command in the new JSON Faceting API:

{
age_ranges: {
type : range
field : age,
start : 0,
end : 100,
gap : 10
}
,
price_ranges: {
type : range
field : price,
start : 0,
end : 1000,
gap : 50
}
}

These aren’t even nested facets, but already one can see how much nicer the JSON API looks. With deeply nested sub-facets and statistics, the clarity of the inherently nested JSON API only grows.

A number of JSON extensions have been implemented to further increase the clarity and ease of constructing a JSON faceting command by hand. For example:

{ // this is a single-line comment, which can help add clarity to large JSON commands
/* traditional C-style comments are also supported */
x : "avg(price)" , // Simple strings can occur unquoted
y : 'unique(manu)' // Strings can also use single quotes (easier to embed in another String)
}

Nicely indented JSON is very easy to understand. If you get a large piece of non-indented JSON somehow, and are trying to make sense of it, you can cut and paste into one of the online validators: http://jsonlint.com http://jsonformatter.curiousconcept.com Both of these validators will indent your JSON, even when it contains extensions unsupported by them (such as comments or bare strings).

 

There are two types of facets, one that breaks up the domain into multiple buckets, and aggregations / facet functions that provide information about the set of documents belonging to each bucket.

Faceting can be nested! Any bucket produced by faceting can further be broken down into multiple buckets by a sub-facet.

Statistics are now fully integrated into faceting. Since we start off with a single facet bucket with a domain defined by the main query and filters, we can even ask for statistics for this top level bucket, before breaking up into further buckets via faceting. Example:

json.facet={
x : "avg(price)", // the average of the price field will appear under "x"
y : "unique(manufacturer)" // the number of unique manufacturers will appear under "y"
}

See facet functions for a complete list of the available aggregation functions.

The general form of the JSON facet commands are: <facet_name> : { <facet_type> : <facet_parameter(s)> } Example: top_authors : { terms : { field : authors, limit : 5 } }

After Solr 5.2, a flatter structure with a “type” field may also be used: <facet_name> : { "type" : <facet_type> , <other_facet_parameter(s)> } Example: top_authors : { type : terms, field : authors, limit : 5 }

The results will appear in the response under the facet name specified. Facet commands are specified using json.facet request parameters.

To test out different facet requests by hand, it’s easiest to use “curl” from the command line. Example:

$ curl http://localhost:8983/solr/query -d 'q=*:*&rows=0&
json.facet={
categories:{
type : terms,
field : cat,
sort : { x : desc},
facet:{
x : "avg(price)",
y : "sum(price)"
}
}
}
'

 

The terms facet, or field facet, produces buckets from the unique values of a field. The field needs to be indexed or have docValues.

The simplest form of the terms facet

{
top_genres : { terms : genre_field }
}

An expanded form allows for more parameters:

{
top_genres : {
type : terms,
field : genre_field,
limit : 3,
mincount : 2
}
}

Example response:

"top_genres":{
"buckets":[
{
"val":"Science Fiction",
"count":143},
{
"val":"Fantasy",
"count":122},
{
"val":"Biography",
"count":28}
]
}

Parameters:

  • field - The field name to facet over.

  • offset - Used for paging, this skips the first N buckets. Defaults to 0.

  • limit - Limits the number of buckets returned. Defaults to 10.

  • mincount - Only return buckets with a count of at least this number. Defaults to 1.

  • sort - Specifies how to sort the buckets produced. “count” specifies document count, “index” sorts by the index (natural) order of the bucket value. One can also sort by any facet function / statistic that occurs in the bucket. The default is “count desc”. This parameter may also be specified in JSON like sort:{count:desc}. The sort order may either be “asc” or “desc”

  • missing - A boolean that specifies if a special “missing” bucket should be returned that is defined by documents without a value in the field. Defaults to false.

  • numBuckets - A boolean. If true, adds “numBuckets” to the response, an integer representing the number of buckets for the facet (as opposed to the number of buckets returned). Defaults to false.

  • allBuckets - A boolean. If true, adds an “allBuckets” bucket to the response, representing the union of all of the buckets. For multi-valued fields, this is different than a bucket for all of the documents in the domain since a single document can belong to multiple buckets. Defaults to false.

  • prefix - Only produce buckets for terms starting with the specified prefix.

  • method - Provides an execution hint for how to facet the field.

    • method:uif - Stands for UninvertedField, a method of faceting indexed, multi-valued fields using top-level data structures that optimize for performance over NRT capabilities.
    • method:dv - Stands for DocValues, a method of faceting indexed, multi-valued fields using per-segment data structures. This method mirrors faceting on real docValues fields but works by building on-heap docValues on the fly from the index when docValues aren’t available. This method is better for a quickly changing index.
    • method:stream - This method creates each individual facet bucket (including any sub-facets) on-the-fly while streaming the response back to the requester. Currently only supports sorting by index order.

 

The query facet produces a single bucket that matches the specified query.

An example of the simplest form of the query facet

{
high_popularity : { query : "popularity:[8 TO 10]" }
}

An expanded form allows for more parameters (or sub-facets / facet functions):

{
high_popularity : {
type : query,
q : "popularity:[8 TO 10]",
facet : { average_price : "avg(price)" }
}
}

Example response:

"high_popularity" : {
"count" : 147,
"average_price" : 74.25
}

 

The range facet produces multiple range buckets over numeric fields or date fields.

Range facet example:

{
prices : {
type : range,
field : price,
start : 0,
end : 100,
gap : 20
}
}

Example response:

"prices":{
"buckets":[
{
"val":0.0, // the bucket value represents the start of each range. This bucket covers 0-20
"count":5},
{
"val":20.0,
"count":3},
{
"val":40.0,
"count":2},
{
"val":60.0,
"count":1},
{
"val":80.0,
"count":1}
]
}

To ease migration, these parameter names, values, and semantics were taken directly from the old-style (non JSON) Solr range faceting.

Parameters:

  • field - The numeric field or date field to produce range buckets from

  • mincount - Minimum document count for the bucket to be included in the response. Defaults to 0.

  • start - Lower bound of the ranges

  • end - Upper bound of the ranges

  • gap - Size of each range bucket produced

  • hardend - A boolean, which if true means that the last bucket will end at “end” even if it is less than “gap” wide. If false, the last bucket will be “gap” wide, which may extend past “end”.

  • other - This param indicates that in addition to the counts for each range constraint between facet.range.start and facet.range.end, counts should also be computed for…

  • "before" all records with field values lower then lower bound of the first range

  • "after" all records with field values greater then the upper bound of the last range

  • "between" all records with field values between the start and end bounds of all ranges

  • "none" compute none of this information

  • "all" shortcut for before, between, and after

  • include - By default, the ranges used to compute range faceting between facet.range.start and facet.range.end are inclusive of their lower bounds and exclusive of the upper bounds. The “before” range is exclusive and the “after” range is inclusive. This default, equivalent to lower below, will not result in double counting at the boundaries. This behavior can be modified by the facet.range.include param, which can be any combination of the following options…

  • "lower" all gap based ranges include their lower bound

  • "upper" all gap based ranges include their upper bound

  • "edge" the first and last gap ranges include their edge bounds (ie: lower for the first one, upper for the last one) even if the corresponding upper/lower option is not specified

  • "outer" the “before” and “after” ranges will be inclusive of their bounds, even if the first or last ranges already include those boundaries.

  • "all" shorthand for lower, upper, edge, outer

Parameters that all faceting methods have in common include