Solr 'n Stuff | Blog

Solr 7.1 Features

Sun, 03 Sep 2017 00:00:00 GMT

Here’s an overview of some of the new features in Solr 7.1:

JSON Query Syntax / JSON Query DSL

There is now a JSON mapping to Solr QParsers. Currently, one must use the JSON Request API to use this JSON syntax, but SOLR-11295 will provide a more general purpose entry point to JSON syntax in the future.

The general form of a query type in existing local params syntax is:

 {!query_type param1=val1 param2=val2}query_value

OR, specifying the main query value using the v parameter:

 {!query_type param1=val1 param2=val2 v=query_value}

In the new JSON syntax, the corresponding mappings would be

 {query_type:{param1:val1, param2:val2, v:query_value}}

OR using query (which is a synonym for v when mapping to local params)

 {query_type:{param1:val1, param2:val2, query:query_value}}

Here’s an example of a pseudo-join query in local-params syntax that queries for book_review1 and follows the “book_id” field of the result(s) to the matching “id” field:

curl http://localhost:8983/solr/techproducts/query -d 'q={!join from:book_id to:id}id:book_review1'

In JSON syntax (indented for better readability), this would be

curl http://localhost:8983/solr/techproducts/query -d '
{
  query:{
    join:{
      from : book_id,
      to : id,
      query : "id:book_review1"
    }
  }
}'

Note that the query parameter is specified in lucene syntax above, but it can be any query in local-params syntax or JSON syntax. Example:

curl http://localhost:8983/solr/techproducts/query -d '
{
  query:{
    join:{
      from : book_id,
      to : id,
      query : { field : {f:"id", v:"book_review1"} }  // invokes the "field" QParser
    }
  }
}'

Integral and date type support for min/max

Min and max aggregations in the JSON Facet API have been updated to include support for int, long, and date types. Previously, min and max of all of these field types returned double results.

Auto-scaling enhancements

Solr’s auto-scaling framework contains a number of new features and improvements, including

Ability to move replicas when nodes are added or removed to the cluster.
Solr’s autoAddReplicas feature now uses the autoscaling framework and works for all filesystems.
New API to control triggers and listeners (set-trigger, remove-trigger, suspend-trigger, set-listener, etc.)

As of this writing, the Solr 7.1 reference guide is not published yet. However you can check out the auto-scaling section of the very latest version of the unreleased reference guide for the master branch.

Polygons in Spatial RPT fields

Here’s an example field using the new Geo3D spatialContextFactory that supports polygons:

 <fieldType name="srptgeom_geo3d" class="solr.RptWithGeometrySpatialField"
             spatialContextFactory="Geo3D" planetModel="wgs84"/>

Note: “wgs84” is an ellipsoid coordinate system for the Earth used by GPS. “sphere” can also be used for a spherical model that has faster but less accurate distance calculations.

Streaming Expressions functions

Expanded support for statistical functions including various distributions, rank correlations, distances and more:

harmonicFit, polyfit, polyfitDerivative, betaDistribution, gammaDistribution, zipFDistribution, logNormalDistribution, weibullDistribution, monteCarlo, expMovingAverage, binomialCoefficient, factorial, movingMedian, primes, sumDifference, meanDifference, ebeAdd, ebeSubtract, ebeDivide, ebeMultiply, dotProduct, cosineSimilarity, Canberra, Chebyshev, Earth Movers and Manhattan Distance, Kendall’s Tau-b rank and Spearmans rank, discrete counting and probability, cumulativeProbability, normalDistribution, uniformDistribution, kolmogorovSmirnov, addAll

Solr 8 Features

Sun, 03 Sep 2017 00:00:00 GMT

Here’s an overview of some of the new features in the next major upcoming release: Solr 8.0 Download a Solr 8 nightly build to try these features out and give us feedback!

IN PROGRESS

We’re in the progress of releasing Solr 7, so there’s not much new here yet. Check back later!

Solr 6.6 Features

Fri, 02 Jun 2017 00:00:00 GMT

Here’s an overview of some of the new features in Solr 6.6, released on June 6, 2017.

Download Solr 6.6 to try these features out and give us feedback! You can also check out upcoming features of the next Solr release.

This is likely to be the last Solr 6.x release before Solr 7.

The release notes from the Apache Solr Wiki:

Fields and field types

Payload support with payload() value source and {!payload_score} and {!payload_check} query parsers
Solr support for SimpleTextCodec, via in solrconfig.xml (per-field specification in the schema is not possible)
Multi-field support to TermsComponent when requesting terms’ statistics
Support for PointFields in Grouping, CollapseQParser, and ExpandComponent.

New API

UPLOAD command (Config Set API) for uploading zipped configsets
MOVEREPLICA command (Collections API) for moving a replica across nodes
LISTALIASES command (Collections API) to return a list of all collection aliases
STATUS command (Core Admin API) to emit collection details of each core

Script

Basic authentication can be enabled/disabled using bin/solr|bin/solr.cmd
ls command to ZkCLI for listing only sub-directories

Faceting

Variance and Standard Deviation aggregators for the JSON Facet API
JSON Faceting now supports a query time ‘join’ domain change option

Streaming expressions and evaluators

CartesianProductStream, which turns a single tuple with a multi-valued field into N tuples, one for each value in the multi-valued field
stats and search Streaming Expressions should now work in non-SolrCloud mode
analyze Stream Evaluator to support streaming NLP
New Stream Evaluators: Basic math, Date/time, UUID, Correlation, regress, predict, covariance, convolution, normalize
New Streaming Expressions: shuffle, echo, eval, timeseries, let, get

Examples

Solr default/example uses WordDelimiterGraphFilterFactory and SynonymGraphFilterFactory
New DataImportHandler ‘atom’ example, replacing broken ‘rss’ example
Redone DataImportHandler ‘tika’ example, removing all unused and irrelevant definitions

Metrics

Expose cache statistics using metrics API
Improvements to metric reporters and API: support for “regex” parameter in /admin/metrics, “enabled” flag in reporter configurations, correct handling of “serviceUrl” in SolrJmxReporter, better handling of service clients for JMX, Ganglia and Graphite reporters

Upgrades/deprecations/removals

Deprecated LatLonType, GeoHashField, SpatialPointVectorFieldType, and SpatialTermQueryPrefixTreeFieldType. Instead, switch to LatLonPointSpatialField or SpatialRecursivePrefixTreeFieldType or RptWithGeometrySpatialField
Deprecated PostingsSolrHighlighter. Use UnifiedSolrHighlighter instead.

SolrJ

CloudSolrClient can now be initialized using the base URL of a Solr instance instead of ZooKeeper hosts
SolrJ: Added SolrParams.toLocalParamsString() and ClientUtils.encodeLocalParamVal

Others

New AtomicUpdateProcessor to convert normal update operations to atomic update operations
totalTermFreq support to TermsComponent
Hide keystore and truststore passwords from /admin/info/* outputs
Configurability for thread pool size to recoveryExecutor
Introducing sort=childfield(field) asc for searching by {!parent}

Optimizations

facet.heatmap is now significantly faster when the docset (base query) matches everything and there are no deleted docs. It is also faster when the docset matches a small fraction of the index or none
Reduced heap consumption for filter({!join … score=…})
JSON Facet API now uses hyper-log-log++ for determining the number of buckets when merging requests from a multi-shard distributed request
Better ZkStateWriter batching
Using cache for DistributedQueue in case of single-consumer

Solr 6.5 Features

Mon, 23 Jan 2017 00:00:00 GMT

Here’s an overview of some of the new features in Solr 6.5. Download Solr 6.5 to try these features out and give us feedback!

This feature list was adapted from the release notes on the wiki: https://wiki.apache.org/solr/ReleaseNote65

PointFields (fixed-width multi-dimensional numeric & binary types enabling fast range search) are now supported
In-place updates to numeric docValues fields (single valued, non-stored, non-indexed) supported using atomic update syntax
A new LatLonPointSpatialField that uses points or doc values for query
It is now possible to declare a field as “large” in order to bypass the document cache

Query

New sow=false request param (split-on-whitespace) for edismax & standard query parsers enables query-time multi-term synonyms
XML QueryParser (defType=xmlparser) now supports span queries

Highlighting

hl.maxAnalyzedChars now have consistent default across highlighters
UnifiedSolrHighlighter and PostingsSolrHighlighter now support CustomSeparatorBreakIterator

Streaming Expressions

Scoring formula is adjusted for the scoreNodes function
Calcite Planner now applies constant Reduction Rules to optimize plans
A new significantTerms Streaming Expression that is able to extract the significant terms in an index
StreamHandler is now able to use runtimeLib jars
Arithmetic operations are added to the SelectStream

Ease of use / Admin UI

Added modernized self-documenting /v2 API
The .system collection is now created on first request if it does not exist
Admin UI: Added shard deletion button

Metrics

Metrics API now supports non-numeric metrics (version, disk type, component state, system properties…)
The disk free and aggregated disk free metrics are now reported
The DirectUpdateHandler2 now implements MetricsProducer and exposes stats via the metrics api and configured reporters.

Optimizations

BlockCache is faster due to less failures when caching a new block
MMapDirectoryFactory now supports “preload” option to ask mapped pages to be loaded into physical memory on init

Misc Changes

Security: BasicAuthPlugin now supports standalone mode
Arbitrary java system properties can be passed to zkcli
SolrHttpClientBuilder can be configured via java system property
Javadocs and Changes.html are no longer included in the binary distribution, but are hosted online

For more detailed lists with pointers to JIRA issues, see the HTML version of CHANGES.txt

Facet Domains

Tue, 01 Nov 2016 00:00:00 GMT

What is the Facet Domain

The domain of a facet is the set of values (normally defined by a set of documents) that calculations will be done over. The root domain is the set of documents that match the base query and any filters.

Changing the Facet Domain

For any facet command, one can use the domain keyword to change the facet domain before facet computation.

blockParent

The existence of the blockParent parameter in the domain block will cause incoming child documents to be mapped to their parents. The value of the blockParent parameter is the parent filter that specifies the complete set of parent documents for this block join operation.

Nested Documents -> Faceting On Parents for an example.

blockChildren

The existence of the blockChildren parameter in the domain block will cause incoming parent documents to be mapped to their children. The value of the blockChildren parameter is the parent filter that specifies the complete set of parent documents for this block join operation.

Nested Documents -> Faceting On Children for an example.

excludeTags

The excludeTags parameter causes the domain to be re-calculated from the root all the way to the current point, but excluding any filters with the specified tags. This is normally used as part of multi-select faceting.

filter

The filter parameter is used to specify a filter or list of filters to be intersected with the incoming domain before faceting. These filters are applied after other domain transformations such as blockParent, blockChildren, or excludeTags. The same exact syntax for filter is supported in the JSON Request API for the top level document list (or will be after SOLR-9733).

Example:

json.facet = {
  categories : {
    type : terms,
    field : cat,
    domain : { filter:"user:yonik" }
  }
}

The “param” query type can be used to grab zero or more filters from a request parameter:

q=my query&
myfilt=type:review&
myfilt=rating:5&
json.facet = {
  categories : {
    type : terms,
    field : cat,
    domain : {
      filter:[
        "user:yonik",
        {param : myfilt} ]
    }
  }
}

Solr 6.4 Features

Sun, 30 Oct 2016 00:00:00 GMT

Here’s an overview of some of the new features in Solr 6.4.

Download Solr 6.4 to try these features out and give us feedback! You can also check out upcoming features of the next Solr release.

`filter` list for facet domain

Any JSON facet command (terms, range, query) can now filter the facet domain in a simpler manner, without resorting to nested query facets.

Example:

json.facet = {
  categories : {
    type : terms,
    field : cat,
    domain : { filter:"user:yonik" }
  }
}

The filters are applied after other domain change operations and are particularly useful when faceting on child documents. The filter attribute can be a single query or a list of multiple queries to intersect.

Using a param to refer to a filter by query parameter was added shortly after.

Learning to Rank

Learning to Rank (LTR) plugin for reranking results with machine learning models.

See the Lucene/Solr Revolution presentation Learning to Rank in Solr as well as the Solr Ref Guide.

snapshotcli.sh command line tool to manage snapshots

The full path of the script is ./solr/server/scripts/cloud-scripts/snapshotscli.sh

An example from the pull request:

// Start solr and initialize a sample collection
bin/solr start -c
bin/solr create_collection -c books
curl 'http://localhost:8983/solr/books/update?commit=true' -H 'Content-type:application/json' -d '
[ {"id" : "book1", "title" : "American Gods", "author" : "Neil Gaiman" } ]'

//Create and export a snapshot
./snapshotscli.sh --create snap-1 -c books -z localhost:9983
./snapshotscli.sh --list -c books -z localhost:9983
./snapshotscli.sh --describe snap-1 -c books -z localhost:9983
./snapshotscli.sh --export snap-1 -c books -z localhost:9983 -d /tmp
./snapshotscli.sh --delete snap-1 -c books -z localhost:9983

// Restore the backup and verify the doc count
curl 'http://localhost:8983/solr/admin/collections?action=restore&name=snap-1&location=/tmp&collection=books_restored'
curl 'http://localhost:8983/solr/books_restored/select?q=*:*'

More efficient filter queries

When parsing filter queries (including fq parameters) the standard solr query parser will avoid using BooleanQuery for term disjunctions on string and numeric fields, and will use TermsQuery instead.

This has a number of positive effects:

Avoids Lucene’s dreaded static maxBooleanClauses issue that causes “too many boolean clauses” exceptions
The resulting query should be smaller to cache
The resulting query should have higher performance

For example, the following filter will now be faster, and will no longer throw a “too many boolean clauses” exception:

fq=id:(myid1 myid2 myid3 myid4 ... myid2000)

`having` streaming expression

The having streaming expression provides functionality like the SQL HAVING clause, and will be used to implement that in the future. Aggregated buckets produced by rollup are filtered by a having clause.

The following streaming expression filters out buckets (authors) with less than 100,000 in sales across all of their books:

having(rollup(over=author,
              sum(sales),
              search(book_sales_collection,
                     q=*:*,
                     fl="author,sales",
                     sort="author asc")),
       gte(sum(sales), 100000))

Also see the a streaming expressions documentation in the ref guide.

Nested Objects in Solr

Wed, 26 Oct 2016 00:00:00 GMT

Nested Documents

Nested Documents (also called Nested Objects) provides the ability to “nest” some documents inside of other documents in a parent/child relationship.

Why Nested Documents

One reason for using nested documents is to prevent false matches. For example, we may have a T-Shirt with 2 SKUs, a Large Red, and a Medium Blue.

Say we tried to model this as a single document:

{
 product : "Awesome T-Shirt",
 color : [ "Red", "Blue" ],
 size : [ "L", "M" ]
}

Now if we search for color:RED AND size:M , it would incorrectly match our document! But if we represented the SKUs as two different documents, then there would be no incorrect match.

{
 color : "Red",
 size : "L",
}
{
 color : "Blue",
 size : "M",
}

Lucene Index Representation

Lucene has a flat object model and does not really support “nesting” of documents in the index. Lucene *does* support adding a list of documents atomically and contiguously (i.e. a virtual “block”), and this is the feature used by Solr to implement “nested objects”.

When you add a parent document with 3 children, these appear int the index contiguously as

child1, child2, child3, parent

There is no Lucene-level information that links parent and child, or distinguishes this parent/child block from the other documents in the index that come before or after. Successfully using parent/child relationships relies on more information being provided at query time.

Limitations

All children of a parent document must be indexed together with the parent document. One cannot update any document (parent or child) individually. The entire block needs to be re-indexed of any changes need to be made.

Schema Requirements

There are no schema requirements except that the _root_ field must exist (but that is there by default in all our schemas). Any document can have nested child documents.

Block Join

“Block Join” refers to the set of related query technologies to efficiently map from parents to children or vice versa at query time. The locality of children and parents can be used to both speed up query operations and lower memory requirements compared to other join methods.

Indexing Nested Documents

NOTE: This example currently requires Solr 5.3 or later.

First, bring up Solr and create a collection (if you have not done so already):

$ bin/solr start            # this starts solr
$ bin/solr create -c demo   # this creates a document collection called "demo"

Let’s remove any leftover docs from other examples:

curl http://localhost:8983/solr/demo/update?commitWithin=3000 -d '{delete:{query:"*:*"}}'

Now let’s add a book with some reviews as nested child documents (notice the _childDocuments_ element):

$ curl http://localhost:8983/solr/demo/update?commitWithin=3000 -d '
[
 {id : book1, type_s:book, title_t : "The Way of Kings", author_s : "Brandon Sanderson",
  cat_s:fantasy, pubyear_i:2010, publisher_s:Tor,
  _childDocuments_ : [
    { id: book1_c1, type_s:review, review_dt:"2015-01-03T14:30:00Z",
      stars_i:5, author_s:yonik,
      comment_t:"A great start to what looks like an epic series!"
    }
    ,
    { id: book1_c2, type_s:review, review_dt:"2014-03-15T12:00:00Z",
      stars_i:3, author_s:dan,
      comment_t:"This book was too long."
    }
  ]
 }
]'

Now we can see that these are really just indexed as 3 documents, all visible by default:

curl http://localhost:8983/solr/demo/query -d 'q=*:*&fl=id'

  "response":{"numFound":3,"start":0,"docs":[
      {
        "id":"book1_c1"},
      {
        "id":"book1_c2"},
      {
        "id":"book1"}]
  }

Now lets add an additional document with nested child documents for use with our query examples:

$ curl http://localhost:8983/solr/demo/update?commitWithin=3000 -d '
[
 {id : book2, type_s:book, title_t : "Snow Crash", author_s : "Neal Stephenson",
  cat_s:sci-fi, pubyear_i:1992, publisher_s:Bantam,
  _childDocuments_ : [
    { id: book2_c1, type_s:review, review_dt:"2015-01-03T14:30:00Z",
      stars_i:5, author_s:yonik,
      comment_t:"Ahead of its time... I wonder if it helped inspire The Matrix?"
    }
    ,
    { id: book2_c2, type_s:review, review_dt:"2015-04-10T9:00:00Z",
      stars_i:2, author_s:dan,
      comment_t:"A pizza boy for the Mafia franchise? Really?"
    }
    ,
    { id: book2_c3, type_s:review, review_dt:"2015-06-02T00:00:00Z",
      stars_i:4, author_s:mary,
      comment_t:"Neal is so creative and detailed! Loved the metaverse!"
    }
  ]
 }
]'

Block Join Query

TODO

Returning Child Documents

One can return child documents along with every returned parent document by using the [child] doc transformer (it’s added to the fl field list parameter ). The list of child documents will be included under the “_childDocuments_” field of each parent.

$ curl http://localhost:8983/solr/demo/query -d '
q=cat_s:(fantasy OR sci-fi)&
fl=id,[child parentFilter=type_s:book]'

  "response":{"numFound":2,"start":0,"docs":[
      {
        "id":"book1",
        "_childDocuments_":[
        {
          "id":"book1_c1",
          "type_s":"review",
          "review_dt":"2015-01-03T14:30:00Z",
          "stars_i":5,
          "author_s":"yonik",
          "comment_t":["A great start to what looks like an epic series!"]},
        {
          "id":"book1_c2",
          "type_s":"review",
[...]

Child Doc Transformer Parameters:

parentFilter - identifies all of the parents. See the section on The Parent Filter for more info.
childFilter - optional query to filter which child documents should be included.
limit - maximum number of child documents to return per parent (defaults to 10)

Also see the Solr ref guide entry on the [child] doc transformer.

Faceting

The JSON Facet API has support for switching the facet domain based on the nested document relationships.

Faceting on Parents

The main query gives us a document list of reviews by author_s:yonik If we want to facet on the book genre (cat_s field) then we need to switch the domain from the children (type_s:reviews) to the parents (type_s:books).

$ curl http://localhost:8983/solr/demo/query -d '
q=author_s:yonik&fl=id,comment_t&
json.facet={
  genres : {
    type: terms,
    field: cat_s,
    domain: { blockParent : "type_s:book" }
  }
}'

And we get a facet over the books which yonik reviewed:

  "response":{"numFound":2,"start":0,"docs":[
      {
        "id":"book1_c1",
        "comment_t":["A great start to what looks like an epic series!"]},
      {
        "id":"book2_c1",
        "comment_t":["Ahead of its time... I wonder if it helped inspire The Matrix?"]}]
  },
  "facets":{
    "count":2,
    "genres":{
      "buckets":[{
          "val":"fantasy",
          "count":1},
        {
          "val":"sci-fi",
          "count":1}]
  }}

Faceting on Children

Now lets say we’re displaying the top sci-fi and fantasy books, and we want to find out who reviews the most books out of our selection. Since our root implicit facet bucket (formed by the query and filters) consists of parent documents (books), we need to switch the facet domain to the children for the author facet.

$ curl http://localhost:8983/solr/demo/query -d '
q=cat_s:(sci-fi OR fantasy)&fl=id,title_t&
json.facet={
  top_reviewers : {
    type: terms,
    field: author_s,
    domain: { blockChildren : "type_s:book" }
  }
}'

Response:

  "response":{"numFound":2,"start":0,"docs":[
      {
        "id":"book1",
        "title_t":["The Way of Kings"]},
      {
        "id":"book2",
        "title_t":["Snow Crash"]}]
  },
  "facets":{
    "count":2,
    "top_reviewers":{
      "buckets":[{
          "val":"dan",
          "count":2},
        {
          "val":"yonik",
          "count":2},
        {
          "val":"mary",
          "count":1}]
  }}

Filtering Children

By default, blockChildren will match all children of every parent doc from the input domain. It’s often the case that only a subset of the children are desired. The easiest way to limit children is with the filter clause.

For example, if we wanted to find the same top reviewers as before, but only for 5 star reviews:

$ curl http://localhost:8983/solr/demo/query -d '
q=cat_s:(sci-fi OR fantasy)&fl=id,title_t&
json.facet={
  top_reviewers : {
    type: terms,
    field: author_s,
    domain: {
      blockChildren : "type_s:book",
      filter : "stars_i:5"
    }
  }
}'

The Parent Filter

Note that regardless of which direction we are mapping (parents to children or children to parents), or what documents we are operating on, we provide a parent filter to define the complete set of parents in the index. In these examples, the parent filter is "type_s:book".

Solr JSON Request API

Tue, 18 Oct 2016 00:00:00 GMT

JSON Request API Motivation

Although query parameters are often an easy method to create a Solr requests by hand, they have a number of drawbacks:

Inherently un-structured, requiring unsightly parameters like f.facet_name.facet.range.start=5
Inherently un-typed… everything is a string.
More difficult to decipher large requests.
Harder to programmatically create a request.
Impossible to validate. Because of the lack of structure, we don’t know the set of valid parameter and thus can’t do good error checking.

Solr already had a JSON API for faceting and analytics, and this new feature has extended that to the complete Solr request!

Simple Example

First let’s add a few excellent books from the fantasy genre (the “commitWithin=1000” will cause them to be visible to searches within 1000 milliseconds):

$ curl http://localhost:8983/solr/update?commitWithin=1000 -d '
[
 {"id":"book1", "author":"Brandon Sanderson", "title":"The Final Empire",
   "series_s":"Mistborn", "sequence_i":1, "genre_s":"fantasy"},
 {"id":"book2", "author":"Brandon Sanderson", "title":"The Well of Ascension",
   "series_s":"Mistborn", "sequence_i":2, "genre_s":"fantasy"},
 {"id":"book3", "author":"Brandon Sanderson", "title":"The Hero of Ages",
   "series_s":"Mistborn", "sequence_i":3, "genre_s":"fantasy"}
]'

Now we can search them with a JSON request rather than using query parameters:

$ curl http://localhost:8983/solr/query -d '
{
  query:"hero"
}'

RESPONSE:

{
  "responseHeader":{
    "status":0,
    "QTime":2,
    "params":{
      "json":"n{n  query:"hero"n}"}},
  "response":{"numFound":1,"start":0,"docs":[
      {
        "id":"book3",
        "author":"Brandon Sanderson",
        "author_s":"Brandon Sanderson",
        "title":["The Hero of Aages"],
        "series_s":"Mistborn",
        "sequence_i":3,
        "genre_s":"fantasy",
        "_version_":1486581355536973824
      }]
  }
}

A few things to note from our example:

JSON body is considered a parameter named “json” and echoed back with the other params (unless you disable it with echoParams=none). This will also cause it to be logged, which is normally important.
The JSON we send to Solr can include unquoted simple strings and can contain comments. See JSON Extensions.
We don’t need to pass the Content-Type for indexing or for querying when we’re using JSON since Solr is now smart enough to auto-detect it when Curl is the client.
HTTP GET requests are now allowed to have a request body (i.e. try using “curl -XGET” for the query)

Here’s a more complete example:

curl -XGET http://localhost:8983/solr/query -d '
{
  query : "*:*",
  filter : [
    "author:brandon",
    "genre_s:fantasy"
  ],
  offset : 0,
  limit : 5,
  fields : ["title","author"],  // we could also use the string form "title,author"
  sort : "sequence_i desc",

  facet : {  // the JSON Facet API is nicely integrated as well
    avg_price : "avg(price)",
    top_authors : {terms : author}
  }
}'

Passing JSON via Request Parameter

It may sometimes be more convenient to pass the JSON body as a request parameter rather than in the actual body of the HTTP request. Solr treats a json parameter the same as a JSON body.

$ curl http://localhost:8983/solr/query -d 'json={query:"hero"}&fq=author:brandon'

Smart merging of multiple JSON parameters

Multiple json parameters in a single request are merged before being interpreted.

Single-valued elements are overwritten by the last value.
Multi-valued elements likefields and filter are appended.
Parameters of the form json.<path>=<json_value> are merged in the appropriate place in the hierarchy. For example a json.facet parameter is the same as “facet” within the JSON body.
A JSON body, or straight json parameters are always parsed first, meaning that other request parameters come after, and overwrite single valued elements.

Smart merging gives the best of both worlds… the structure of JSON with the ability to selectively separate out / decompose parts of the request!

Simple Example

curl 'http://localhost:8983/solr/query?json.limit=5&json.filter="genre_s:fantasy"' -d '
{
  query : "hero",
  limit : 10,
  filter : "author:brandon"
}'

is equivalent to

curl http://localhost:8983/solr/query -d '
{
  query : "hero",
  limit : 5,   // this parameter was overwritten
  filter : [ "author:brandon" , "genre_s:fantasy" ]  // this parameter was appended to
}'

In fact, you don’t even need to start with a JSON body for smart merging to be very useful. Consider the following request composed entirely of request params:

curl http://localhost:8983/solr/query -d 'q=*:*&rows=1&
  json.facet.avg_price="avg(price)"&
  json.facet.top_authors={type:terms,field:author_s,limit:5}'

That is equivalent to having the following JSON body or json parameter:

{
  facet: {
    avg_price: "avg(price)",
    top_authors: {
      type: terms,
      field: author_s",
      limit: 5
    }
  }
}

Debugging

What to see what your merged JSON looks like? Just ask for debugging information (i.e. use the debug=true param), and it will come back under the "json" key along with the other debugging information.

Passing Parameters via JSON

We can also pass normal request parameters in the JSON body within the params block:

$ curl "http://localhost:8983/solr/query?fl=title,author"-d '
{
  params:{
    q:"title:hero",
    rows:1
  }
}
'

Which is equivalent to:

$ curl "http://localhost:8983/solr/query?fl=title,author&q=title:hero&rows=1"

Error Detection

Because we didn’t pollute the root body of the JSON request with the normal Solr request parameters (they are all contained in the params block), we now have the ability to validate requests and return an error for unknown JSON keys.

$ curl http://localhost:8983/solr/query -d '
{
  query : "hero",
  fulter : "author:brandon"  // oops, we misspelled "filter"
}'

And we get an error back containing the error string:

"Unknown top-level key in JSON request : fulter"

Parameter Substitution / Macro Expansion

Of course request templating via Parameter Substitution works fully with JSON request bodies or parameters as well.

Example:

$ curl "http://localhost:8983/solr/query?FIELD=text&TERM=hero&HOWMANY=10" -d '
{
  query:"${FIELD}:${TERM}",
  limit:${HOWMANY}
}'

Current Status

The JSON Request API is currently in it’s infancy - only a few query parameters are supported (although the JSON Facet API which is part of this is more mature). Other Solr features you may want access to (like highlighting) currently need to be controlled through the normal Solr request params (e.g. just ad hl=true to the normal request parameters, or in the params block of a JSON request.

Have ideas on what will make the API better? Want to help out with development? We’d love to hear from you on the solr-user mailing list!

Solr 6.2 Features

Thu, 15 Sep 2016 00:00:00 GMT

Here’s an overview of some of the new features in Solr 6.2.

Download Solr 6.2 to try these features out and give us feedback! You can also check out upcoming features of the next Solr release.

Boolean Comparison Function Queries

Function queries for comparisons of numeric arguments:

gt - greater than
gte - greater than or equal
lt - less than
lte - less than or equal
eq - equal to

These functions return a boolean value for any given document, which is normally used in conjunction with the if function which returns the second argument if the first argument is true, or the third argument otherwise.

Example: For a given document, if num_reviewers is less than 5, return the value of the editor_score field, otherwise return the user_score field.

 if( lt(num_reviewers,5), editor_score, user_score)

An easy way to experiment with function queries and see the results is to use the pseudo-field functionality to return specific function values along with stored fields.

Example: http://localhost:8983/solr/techproducts/query?q=\*:\*&fl=id,popularity,**gt(popularity,3)**

[...]
      {
        "id":"SP2514N",
        "popularity":6,
        "gt(popularity,3)":true},
      {
        "id":"6H500F0",
        "popularity":6,
        "gt(popularity,3)":true},
      {
        "id":"F8V7067-APL-KIT",
        "popularity":1,
        "gt(popularity,3)":false},
[...]

Feature Selection and Logistic Regression

The features and train streaming expressions are machine learning related operations used to implement a text classifier. The goal is to create a classifier (or model) based on a training set of documents that can be used to tell if a new document should be considered part of that set.

features - streaming expression that selects important features (important terms) from a set of documents called the training set (specified by a query).
train - trains a logistic regression model on a text field using the selected features and the training set.

Here’s an example of the streaming expression syntax taken from the JIRA:

train(collection1, q="*:*",
      features(collection1,
               q="*:*",
               field="body",
               outcome="out_i",
               positiveLabel=1,
               numTerms=100),
      field="body",
      outcome="out_i",
      maxIterations=100)

See the features streaming expression and the train streaming expression in the reference guide.

`topic` streaming expression (Beta)

The topic streaming expression implements a publish/subscribe messaging capabilities, where clients can subscribe to queries and receive deliveries of new documents that match those topic queries. When the first call is made, an id is specified to uniquely identify the topic. A checkpointCollection is also specified, and this is where topic progress is stored to keep track of which documents have already been returned.

Example syntax:

topic(checkpointCollection,
      collection1,
      id="yonik_topic1",
      q="title:(lucene solr)",
      fl="id, title, abstract, author")

A client can keep calling the same topic to receive new documents, or it can be wrapped in a daemon streaming expression to provide push functionality.

`scoreNodes` graph streaming expression

The scoreNodes streaming expression is used to add a relevancy score to a graph expression, similar to tf-idf for text. The tf factor is how many times a node appears in the graph traversal, and biases scores in favor of nodes that appear often. The idf factor biases scores in favor of nodes that are more rare in the index.

There is a good example of using this in the ref guide: Calculating Market Basket Co-occurrence, for calculating product recommendations based on your current shopping basket and what other shopping baskets look like.

collections API: `REPLACENODE` and `DELETENODE` commands

New actions have been added to the collections API:

REPLACENODE - moves all replicas from one node to another.
DELETENODE - deletes all replicas currently on a given node.

Examples: /admin/collections?action=RELACENODE&source=MY_OLD_NODE&target=MY_NEW_NODE /admin/collections?action=DELETENODE&node=MY_NODE_NAME

See the relevant sections in the Collections API for more details.

Kerberos security enhancements

Kerberos delegation token support was added to the authentication filter, especially useful for when distributed clients such as MapReduce do not have access to the user’s credentials.

See Kerberos Authentication Plugin in the ref guide.

Whole index replication for CDCR (Cross Data Center Replication)

CDCR can now fall back to whole-index replication to bring a replica up to date when transaction logs have not retained enough updates to do so.

“role” tag in replica placement rules

SolrCloud nodes in the cluster have tags that may be used when performing rule based replica placement. A “role” tag has been added that currently be used to prevent placement of new replicas on the overseer node by adding rule=role:!overseer during collection creation.

See Rule Based Replica Placement in the ref guide for more info.

Solr 6.3 Features

Tue, 23 Aug 2016 00:00:00 GMT

Here’s an overview of some of the new features in Solr 6.3. Download Solr 6.3 to try these features out and give us feedback! You can also check out upcoming features of the next Solr release.

New `dvhash` faceting method in the JSON Facet API

A new faceting method for the JSON Facet API has been added that utilizes hashing instead of direct indexing by ordinal. Currently, when hashing on a field with docValues enabled, or when specifying method=dv, there are accumulator arrays that are indexed by ordinal. When the cardinality of the domain is low, and the cardinality of the field is high, direct indexing of large arrays both wastes memory and takes longer to scan looking for the highest accumulated entries.

Example:

json.facet={
  top_authors : {
    type : terms,
    field : author,
    limit : 20,
    method : dvhash
  }
}

Since it may be difficult for a client to know when the facet domain will be large vs the field cardinality (esp if the domain is based off of a user query), future work is expected to implement automatically switching between faceting methods.

Optimizing, storing and deploying AI models with Streaming Expressions

The classify streaming expression classifies documents according to a model.

classify(<document_stream>,            // the stream to fetch documents that need to be classified
         <model_stream>,               // the stream to fetch the latest model
         field=<input_text_field>,     // the field on the document to classify
         fieldType=<analysis_fieldType> // (opt) fieldType to use for tokenization of the input field
)

For each input document, the classifier decorates the document with two (currently hardcoded) fields with the result:

score_d - the raw score
probability_d - the positive probability that the document belongs to the group

The update streaming expression can be used to store classifier models in a SolrCloud collection, and the topic streaming expression can be used with the classify expression to both deploy models as well as stream new data through the models.

The development JIRA SOLR-9258 has examples, and should be consulted until the Solr reference guide is updated.

“executor” Streaming Expression

The executor streaming expression wraps another stream containing streaming expressions. By default the expression will be contained in the expr_s field in each tuple of the wrapped stream. The executor has an internal thread pool so expressions can be executed in parallel on a single worker. This expression can further be wrapped in a parallel streaming expression to enable execution across a cluster of worker nodes.

Example syntax from the JIRA of a work queue… the “topic” expression retrieves expressions to execute (and keeps track of where it left off):

daemon(executor(threads=10, topic(storedExpressions, fl="expr_s", ...)))

“commit” Streaming Expression

The commit streaming expression wraps another stream (normally involving updates) and performs a commit at the end.

Example:

commit(targetCollection, update(targetCollection, search(sourceCollection, q="myquery", ...)))

“fetch” Streaming Expression

The fetch streaming expression wraps another expression and fetches additional fields from documents in batches. It’s very much like an inner join, with one side of the join being very small.

Example that queries books and adds publisher address to each result:

fetch(publisherCollection,
      search(booksCollection, q="*:*", fl="title,author,publisher", sort="publisher asc"),
      fl="publisherAddress",
      on="publisher=publisherId")

Lots More…

See the release notes on the Solr wiki and the CHANGES file for an in-depth list of changes.

Solr 6.1 Features

Sat, 09 Apr 2016 00:00:00 GMT

Here’s an overview of some of the new features in Solr 6.1.

Download Solr 6.1 to try these features out and give us feedback! You can also check out upcoming features of the next Solr release.

shortestPath Streaming Expression

A shortestPath Streaming Expression was added that implements a distributed breadth-first graph traversal to find the shortest paths in a directed directed graph.

Example:

shortestPath(collection,
                     from="john@company.com",
                     to="jane@company.com",
                     edge="from=to",
                     threads="6",
                     partitionSize="300",
                     fq="limiting query",
                     maxDepth="4")

gatherNodes Streaming Expression (coming soon)

The gatherNodes expression is a more general form of graph traversal than shortestPath, and can be used for more use cases.

Example:

 gatherNodes(friends,
             gatherNodes(friends,
                         search(articles, q=“body:(queryA)”, fl=“author”),
                         walk ="author->user”,
                         gather="friend"),
             walk=“friend->user”,
             gather="friend",
             scatter=“roots, branches, leaves”)

TolerantUpdateProcessorFactory

ToleranteUpdateProcessorFactory will skip update commands that would otherwise cause subsequent updates in a batch to fail.

  <updateRequestProcessorChain name="tolerant">
    <processor class="solr.TolerantUpdateProcessorFactory"/>
    <processor class="solr.DistributedUpdateProcessorFactory" />
    <processor class="solr.RunUpdateProcessorFactory" />
  </updateRequestProcessorChain>

Passing update.chain=tolerant will use the processor chain defined above. One can also pass maxErrors=10 to limit the number of errors before aborting the complete update request. For example, if one is loading a large CSV file with millions of entries for the first time, it may be useful to abort early if every addition would fail due to a configuration error.

Small filter optimization

Filter creation for small cardinality sets (those that match few documents) now produces much less garbage. Up to 3x performance producing small sets (due to less GC overhead).

HDFS optimizations

The HDFS block cache now skips caching “read once” scenarios such as index merges.

Solr 7 Features

Sat, 09 Apr 2016 00:00:00 GMT

Here’s an overview of some of the new features in Solr 7.0 Download Solr 7 to try these features out and give us feedback!

Point Numeric Fields

The now deprecated trie-based numeric fields use (and abuse) the full-text index to index parts of numbers to speed up range queries. The new Points-based numeric fields do not use the full-text index, but instead have a dedicated index structure designed specifically for multidimensional ranges over numbers: the BKD tree. The BKD tree index structure is smaller and faster for range queries, but is slightly slower for exact match queries.

Since Point fields do not currently support un-inversion (i.e. FieldCache), some search functionality requires that points have docValues enabled for fast per-document lookups. This includes sorting, function queries, {!graph}, and {!join} queries.

Current template schemas define types like the following:

    <fieldType name="pint"  class="solr.IntPointField" docValues="true"/>
    <fieldType name="pints" class="solr.IntPointField" docValues="true" multiValued="true"/>

Along with corresponding dynamic field types:

    <dynamicField name="*_i"  type="pint"    indexed="true"  stored="true"/>
    <dynamicField name="*_is" type="pints"   indexed="true"  stored="true"/>

Interested in how the underlying BKD data structure works? Here’s a great lesson from Robert Sedgewick, Princeton University on KD trees (BKD trees are just a variant of KD trees):

The JSON Facet API has a new parameter called refine that turns on two-phase refinement of partial facets during distributed faceting. This guarantees that the statistics (counts or other metrics) within returned facet buckets are accurate.

Partial facets are those facets that specify a limit and thus may not return all facet buckets from all shards in a distributed search. If refine is set to true on one of these partial facets, a second phase is used to “refine” the top buckets from the first phase, collecting information from other shards that did not yet contribute to those buckets. Without refinement, counts and statistics for the bucket can be incorrect. The second phase of faceting normally does not cause any additional HTTP requests since they are piggy-backed onto the normal second phase of distributed search that retrieves stored fields for the top document ids.

Example:

json.facet={
  x : {
    type : terms,
    field : cat,
    limit : 5,
    refine : true
  }
}

min/max Aggregations support Strings

The min/max facet aggregations (facet functions) have been extended beyond numeric fields and functions to include single-valued string fields.

New Streaming Evaluators

Streaming Expressions has many new functions:

movingAvg
arraySort
cumulative
anova
hist
array
sequence
finddelay
knn
describe
copyOfRange
sql
copyOf
distance
scale
rank
length
reverse

Replica Types

SolrCloud now has support for different replica types.

NRT

NRT stands for Near Real Time. This is the default and original replica type in SolrCloud. Updates flow from the leader to all replicas and are added to replica transaction logs as well as indexed. This is the only type of replica to support soft commits since TLOG and PULL replicas need hard commits to copy over new index segments.

TLOG

Updates flow from the leader to all replicas and are added to replica transaction logs (tlogs) only. Replicas are kept up-to-date by pulling new index segment files from the leader. The transaction logs allow these replicas to recover and become a leader if necessary, as well as directly service real-time get requests.

PULL

Updates do not flow from the leader to replicas. Replicas are kept up-to-date by pulling new index segment files from the leader. This is similar to the original non-SolrCloud master-slave replication. Replicas of this type may not become leaders since updates would most likely be lost. Real-time get requests to a PULL replica are forwarded to the leader since these replicas lack transaction logs to find the most recent uncommitted updates.

Other Changes

The default response format has been changed from XML to indented JSON. Add wt=xml to the request obtain an XML response, and add indent=off if you wish to turn off indenting.
The new v2 API, exposed at /api/ is now the preferred API (esp for using the collections API), but /solr/ continues to be supported.
The alternate “Analytics Component” in the contrib modules was bumped to v2, with distributed support and a new JSON-based request syntax
Auto-scaling framework that allows Solr to place new replicas based on metrics such as free disk space.
The standard lucene/solr query parser now defaults to sow=false, meaning that for text fields, it does not split on whitespace before handing the text to the analyzer. This enabled multi-word synonyms to be matched by the analyzer.
When a collection is created without specifying a configset, the new ‘_default’ configset is now used. It is data-driven (schemaless), and indexes strings as analyzed text in addition to using a copyField to *_str field suitable for sorting or faceting.

See CHANGES for more detailed upgrade notes. Also see the official release notes on the Solr wiki. The Solr Reference Guide should contain other upgrade info.

Solr 6 Features

Tue, 05 Apr 2016 00:00:00 GMT

Here’s an overview of some of the new features in Solr 6.0.

Download Solr 6 to try these features out and give us feedback! You can also check out upcoming features of the next Solr release.

Parallel SQL

Parallel SQL queries across SolrCloud collections. The SQL engine is built on top of Solr’s Streaming API (Streaming Expressions), which provides support for parallel relational algebra and real-time map-reduce.

SQL statements are compiled to Streaming Expressions for parallel execution across SolrCloud worker nodes.
SolrCloud collections are abstracted as Relational Tables.
Full support for Lucene/Solr query syntax in the WHERE clause.
Many operations such as grouping/rollups can be automatically parallelized, utilizing the real-time MapReduce capabilities of the streaming expressions.
Grouping/rollup operations can be pushed down and leverage the JSON Facet API for increased performance.
Currently works in SolrCloud mode only… no standalone mode yet.
SQL functionality is currently experimental and incomplete (for example underlying streaming join functionality is still in the process of being integrated.)

Examples:

select category, count(*), sum(inventory), min(price), max(price), avg(outstanding) from collection1 where text='4k HDTV' group by category order by sum(inventory) asc limit 10

select id,category from collection1 where category = '(dvd OR bluray)' order by category desc limit 100

select fieldA, fieldB, count(*), sum(fieldC), avg(fieldY)
from collection1
where fieldC = 'term1 term2'
group by fieldA, fieldB
having sum(fieldC) > 1000
order by sum(fieldC) asc
limit 100

See the Parallel SQL documentation for more info.

/sql

There is a SQL Request Handler mapped to the /sql endpoint. Only SolrCloud collections can currently be searched with the SQL handler.

Example:

$ curl http://localhost:8983/solr/techproducts/sql -d "stmt=select id from techproducts"

{"result-set":{"docs":[
{"id":"EN7800GTX/2DHTV/256M"},
{"id":"100-435805"},
{"id":"UTF8TEST"},
{"id":"SOLR1000"},
{"id":"9885A004"}
]}}

JDBC driver

Solr has a new JDBC driver that may be used to access the new SQL functionality. The Solr JDBC driver has been tested with DbVisualizer, Apache Zeppelin, and SQuirreL SQL so far.

The general form of the JDBC connection string is:

jdbc:solr://SOLR_ZK_CONNECTION_STRING?collection=COLLECTION_NAME

The JDBC driver is not yet documented in the Ref Guide, so see https://issues.apache.org/jira/browse/SOLR-8521 for more documentation in the meantime.

Streaming Expressions

Distributed Joins

A number of distributed join operations have been added to streaming expressions:

innerJoin
leftOuterJoin
hashJoin
outerHashJoin

Example:

innerJoin(
  search(collection1, q=*:*, fl="fieldA, fieldB, fieldC", ...),
  search(collection2, q=*:*, fl="fieldA, fieldD, fieldE", ...),
  on="fieldA=fieldA"
)

rollup Streaming Expression

The rollup streaming expression groups tuples by common field values and emits the rollup value along with other specified metrics.

Example:

rollup(
      search(collection1, qt="/export"
             q="*:*",
             fl="id,manu,price",
             sort="manu asc"),
       over="manu"),
       count(*),
       max(price)
)

The facet streaming expression is much like the rollup expression, but it pushes down the computation to the leaves using the JSON Facet API.

Example:

facet(
  techproducts,
  q="*:*",
  buckets="manu",
  bucketSorts="count(*) desc",
  bucketSizeLimit=1000,
  count(*),
  sum(price),
  max(popularity)
)

Streaming Expression Docs

Many other Streaming Expressions were added for the Solr 6 release. At this point, they should all be documented in the Streaming Expressions section of the Solr Ref Guide.

Graph traversal query

A basic graph traversal query that follows nodes to edges, optionally filtering during traversal.

Example: Assume we have documents that represent people, and each document has a field called “parent_id” which lists the parents. This example query matches “Philip J. Fry” and all of his ancestors:

fq={!graph from=parent_id to=id}id:"Philip J. Fry"

The main argument to the graph query defines the root set, in this case id:"Philip J. Fry". The graph query then iteratively follows the parent_id field to documents with corresponding id fields (i.e. for each iteration, the values for parent_id in the current set are matched to id field of all documents). The basic graph query is equivalent to a repeated join query.

Graph query parameters:

from - The field used in the current set of documents to match to the to field in the set of destination documents.- to - The field used to find matches in the values obtained from the from fields of the starting set of documents.- traversalFilter - A filter query that is applied on each iteration.
returnRoot - Controls whether the root set of documents should be included. Defaults to “true”.
returnOnlyLeaf - If true, only returns leaf documents. Defaults to “false”.
maxDepth - The maximum number of iterations before graph traversal stops. Defaults to -1 (unlimited).

NOTE: This graph query only traverses edges in the same index (i.e. it does not traverse edges across different nodes/cores). Distributed graph traversal is being developed and will be in future versions of Solr 6.x

BM25 scoring

Default scoring now uses Okapi BM25 by default. You can enable the old tf-idf vector space similarity by using ClassicSimilarity.

Here is an example of how to use classic tf-idf similarity just for the “text” fieldType in the Solr schema:

  <fieldType name="text" class="solr.TextField">
    <analyzer class="org.apache.lucene.analysis.standard.StandardAnalyzer"/>
    <similarity class="solr.ClassicSimilarityFactory"/>
  </fieldType>

For BM25, one can tweak the scoring function on a per-fieldType basis. For example:

    <fieldType name="text2" class="solr.TextField">
      <analyzer class="org.apache.lucene.analysis.standard.StandardAnalyzer"/>
      <similarity class="solr.BM25SimilarityFactory">
        <float name="k1">1.2
        <float name="b">0.75
      </similarity>
    </fieldType>

For those new to full-text search terminology, “similarity” produces a score for how similar a document is to a full-text query. When determining this score, document statistics as well as corpus statistics are used. Some scoring factors include:

The number of times a search term appears in the document field. More matches produces a higher score.
The size of the document field. Longer fields produce a lower score, with the idea being that for a given number of term matches, shorter is better (more specific match)
The average length of the field across the entire corpus (BM25 considers this, classic tf-idf does not)
How common the query terms are across the entire corpus. The idea being that rarer terms carry more information. For example, if I searched for “blue whale”, all else being equal, I’d probably want things about whales to score higher than things about blue.

Filters for Real-Time Get

Real-Time Get (normally handled at the /get URL) now handles filters (fl parameters) to restrict matching documents.

Example:

curl "http://localhost:8983/solr/demo/get?id=book1,book2,book3&fl=security:group1"

Cross Data Center Replication

An experimental version of CDCR (Cross Data Center Replication) has been added that supports an active-passive configuration.

Updates are asynchronously sent from the active cluster leader to the passive cluster leader. Since updates are asynchronously relayed from the leader’s transaction logs, temporary connectivity issues between data centers can be tolerated with no interruption in service for the primary DC.

Here is the in-progress CDCR documentation. The documentation link is likely to change once CDCR moves out of it’s “experimental” phase.

Solr 5.5 Features

Thu, 10 Dec 2015 00:00:00 GMT

Here’s an overview of some of the new features in Solr 5.5 Also see Solr Download Links and upcoming Features of the next Solr release.

Return docValues fields like stored fields

In previous Solr versions, returning the top N documents only retrieved field values from the row-store (i.e. fields where “stored” is true). Values will now also be retrieved from docValues (which are essentially column stored) for fields where “stored” is false. This avoids the need to duplicate the value in the row store if it’s already column stored.

A "facet-trace" section has been added to the debug info for JSON Facets. The exact format is subject to change, but the current information includes information about each facet command, including the processor used to execute the facet and the domain size. This information is included recursively included for each facet bucket for facets with sub-facets.

Example request:

$ curl http://localhost:8983/solr/techproducts/query -d 'q=*:*&rows=0&debug=true&
 json.facet={
   categories:{
     type : terms,
     field : cat
   }
 }

Example response debug info:

  [...]
  "facet-trace":{
      "processor":"FacetQueryProcessor",
      "elapse":0,
      "query":null,
      "domainSize":32,
      "sub-facet":[{
          "processor":"FacetFieldProcessorUIF",
          "elapse":0,
          "field":"cat",
          "limit":10,
          "numBuckets":16,
          "domainSize":32}]},
[...]

UnInvertedField faceting method restored

facet.method=uif parameter causes traditional field faceting to delegate to the JSON Facet API with method=uif. This is roughly equivalent to what the Solr 4 default faceting method was for multi-valued fields. It is optimized for performance over static indexes rather than NRT (quickly changing indexes).

Stored field compression mode

The compression mode for stored fields can now be specified via codecFactory in solrconfig.xml See the Codec Factory section in the Solr reference guide for more details.

Async Collection APIs

Generic support was added for making collection APIs async. Async support was added for the following commands: delete/reload collection, create/delete alias, create/delete shard, delete replica, add/delete replica property, add/remove role, overseer status, balance shard unique, rebalance leaders, modify collection, migrate state format.

See Asynchronous Calls in the Solr ref guide.

BlockJoinFacetComponent

There is a new experimental BlockJoinFacetComponent for calculating facets by a child.facet.field parameter with a {!parent} query. The component is not enabled by default. Note that this component is unrelated to the block join faceting support in the JSON Facet API.

XML Query Syntax

The XML query parser, registered as “xmlparser” is a direct interface to Lucene’s XMLQueryParser (CoreParser). Personal recommendation: avoid the use of this query parser unless you have very unusual/specific needs.

Example:

curl http://localhost:8983/solr/techproducts/query -d 'debugQuery=true&
q={!xmlparser}
<BooleanQuery>
  <Clause occurs="must"> <TermQuery fieldName="name">ipod</TermQuery> </Clause>
  <Clause occurs="must"> <TermQuery fieldName="manu">apple</TermQuery> </Clause>
</BooleanQuery>
'

This parser does not do any text analysis on terms, so provided terms will need to match what is in the index exactly (i.e. you will need to do things like lowercasing and stemming yourself). Good backward compatibility is unlikely with this parser as it exposes more internal implementation details. Term queries on fields such as numeric fields, enum fields, and boolean fields will only work if you know the internal term representation in the index.

See XmlQParser in the Solr ref guide for more info.

upconfig and downconfig

A configset, or configuration set, is a set of config files for a Solr collection. For SolrCloud mode, an upconfig option has been added to the /bin/solr script to upload a configset to zookeeper. A matching downconfig option has been added to download a configset from zookeeper.

For examples and documentation, see bin/solr Zookeeper Operations in the Solr reference guide.

CheckIndex for HDFS

There is an internal CheckHdfsIndex class that can be run from the command line for HDFS indexes like CheckIndex can be run for normal indexes. Example:

java -cp "./server/solr-webapp/webapp/WEB-INF/lib/*:./server/lib/ext/*" -ea:org.apache.lucene... org.apache.solr.index.hdfs.CheckHdfsIndex /path/to/my/index/

For reference, here is also the command to run CheckIndex on a local (non-HDFS) lucene index:

java -cp "./server/solr-webapp/webapp/WEB-INF/lib/*:./server/lib/ext/*" -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex ./example/techproducts/solr/techproducts/data/index

Which will result in the following output:

Opening index @ ./example/techproducts/solr/techproducts/data/index

Segments file=segments_2 numSegments=1 version=6.0.0 id=bv1fdquc5dh3nvcf4jxiwfow4 format= userData={commitTimeMSec=1456073865693}
  1 of 1: name=_0 maxDoc=32
    version=6.0.0
    id=bv1fdquc5dh3nvcf4jxiwfow3
    codec=Lucene60
    compound=false
    numFiles=13
    size (MB)=0.026
    diagnostics = {java.runtime.version=1.8.0_40-b25, java.vendor=Oracle Corporation, java.version=1.8.0_40, java.vm.version=25.40-b25, lucene.version=6.0.0, os=Mac OS X, os.arch=x86_64, os.version=10.11.2, source=flush, timestamp=1456073865745}
    no deletions
    test: open reader.........OK [took 0.059 sec]
    test: check integrity.....OK [took 0.000 sec]
    test: check live docs.....OK [took 0.000 sec]
    test: field infos.........OK [25 fields] [took 0.000 sec]
    test: field norms.........OK [5 fields] [took 0.001 sec]
    test: terms, freq, prox...OK [1187 terms; 1813 terms/docs pairs; 1496 tokens] [took 0.025 sec]
    test: stored fields.......OK [356 total field count; avg 11.1 fields per doc] [took 0.013 sec]
    test: term vectors........OK [3 total term vector count; avg 1.0 term/freq vector fields per doc] [took 0.006 sec]
    test: docvalues...........OK [0 docvalues fields; 0 BINARY; 0 NUMERIC; 0 SORTED; 0 SORTED_NUMERIC; 0 SORTED_SET] [took 0.000 sec]

No problems were detected with this index.

Took 0.246 sec total.

Multi-Select Faceting

Wed, 25 Nov 2015 00:00:00 GMT

NOTE: This uses syntax from the upcoming Solr 5.4 release. If you are using Solr 5.2 or 5.3, specify domain:{excludeTags:mytag} as excludeTags:mytag.

Multi-Select Faceting with Solr

Multi-select faceting is a powerful faceting style that allows users to see and select multiple facet constraints (facet values) for certain facets. This example uses Solr’s JSON Facet API along with filter tagging and excluding to implement this style of faceting.

Sports store example

Let’s say we have 3 facets, Size, Color, and Brand. This is multi-select faceting because for the Color and Brand facets, we want the user to be able to select multiple constraints (values). The Size facet is single-select since we’ve decided that customers in general will only be interested in one size at a time.

Here is our super-fancy ASCII UI, after the user has searched for "running shorts":

=== Size ===   === Color ===    === Brand ===
[Small] (7)    [ ] Red (2)      [ ] Nike (7)
[Medium] (5)   [ ] Blue (8)     [ ] Adidas (5)
[Large] (6)    [ ] Green (3)    [ ] Reebok (4)
               [ ] Black (5)    [ ] Under Armour (2)

(Top matches sorted by popularity displayed here... use your imagination!)

Note that the Color and Brand facets have checkboxes to indicate which constraints have been selected. We’re starting off with no constraints. Below the facets is where we would display the top matching items, along with pretty pictures, prices, etc.

1. User selects “Blue”

The Wrong Way

The user selects “Blue” so we add that as a filter and re-issue the request (we’re using Solr’s JSON Facet API):

&q="running shorts"
&fq=color:Blue
&json.facet={
      sizes:{type:terms, field:size},
      colors:{type:terms, field:color},
      brands:{type:terms, field:brand}
}

We get back the response and update our UI from that data:

=== Size ===   === Color ===    === Brand ===
[Small] (3)    [ ] Red (0)      [ ] Nike (3)
[Medium] (2)   [x] Blue (8)     [ ] Adidas (2)
[Large] (3)    [ ] Green (0)    [ ] Reebok (2)
               [ ] Black (0)    [ ] Under Armour (1)

(Top Blue running shorts displayed here)

What’s right: The Size and Brand facets now reflect the fact that we’ve selected Blue, and that’s what we wanted. Our list of top matches also only includes Blue things, just as we wanted.

What’s wrong: Because we filtered out anything that wasn’t Blue, we get back 0 counts for other colors! But we still want the other color information so the customer can select additional colors.

The Right Way

When we compute the multi-select Color facet, we want to ignore any constraints (filters) on that facet so we will get back the correct counts for other colors. To accomplish this, we can tag filters and then selectively exclude filters (i.e. pretend they don’t exist) by tag when faceting.

The same thing applies to the multi-select Brand facet… we want any Brand selections to affect everything else (including all other facets), except for the Brand facet itself.

When the user selects Blue, we add that as a filter tagged with COLOR and re-issue the request:

&q="running shorts"
&fq={!tag=COLOR}color:Blue
&json.facet={
      sizes:{type:terms, field:size},
      colors:{type:terms, field:color, domain:{excludeTags:COLOR} },
      brands:{type:terms, field:brand, domain:{excludeTags:BRAND} }
}

Now when we get back our response, it still includes the other Colors in the Color facet.

=== Size ===   === Color ===    === Brand ===
[Small] (3)    [ ] Red (2)      [ ] Nike (3)
[Medium] (2)   [x] Blue (8)     [ ] Adidas (2)
[Large] (3)    [ ] Green (3)    [ ] Reebok (2)
               [ ] Black (5)    [ ] Under Armour (1)

(Top Blue running shorts displayed here)

domain and excludeTags

The domain is the set of documents that facets will be calculated over. In the JSON Facet API, the domain keyword/command is normally used to change the domain before the facets are calculated.

In our example above, we specified domain:{excludeTags:COLOR} for the colors facet. This will re-calculate the facet domain as if any filters tagged with COLOR were not applied.

2. User selects “Black”

Ok, now the user selects Black as well.

The Wrong Way

We naively add an additional filter, fq={!tag=COLOR}color:Black to the request, just as we would with traditional single-select faceting.

What’s wrong: Everything! Our request matches nothing and we get back all 0’s.

This is because filters are logically intersected. We searched for things that were Blue AND Black, and that will match nothing (assuming our items only have a single color). What we really want is Blue OR Black.

The Right Way

We need the logical OR, or union, of all the selected colors.

&q="running shorts"
&fq={!tag=COLOR}color:(Blue Black)
&json.facet={
      sizes:{type:terms, field:size},
      colors:{type:terms, field:color, domain:{excludeTags:COLOR} },
      brands:{type:terms, field:brand, domain:{excludeTags:BRAND} }
}

=== Size ===   === Color ===    === Brand ===
[Small] (5)    [ ] Red (2)      [ ] Nike (5)
[Medium] (4)   [x] Blue (8)     [ ] Adidas (3)
[Large] (4)    [ ] Green (3)    [ ] Reebok (3)
               [x] Black (5)    [ ] Under Armour (2)

(Top Blue and Black running shorts displayed here)

Note that the counts on the other facets increased to reflect the larger domain (it includes both blue and black items).

Not just counts!

Although our simple example just dealt with facet counts, multi-select faceting via excludeTags works with the broad range of features in the JSON Facet API. The domain change will apply to everything else under that facet, including Sub-facets and Facet Functions.

Tagging and excluding filters with excludeTags

Solr filter queries (fq parameters) can be tagged with arbitrary strings using the localParams {!tag=mystring} syntax. Example: fq={!tag=COLOR}color:Blue

Multiple filters can be tagged with the same tag. Example: fq={!tag=foo}one_filter&fq={!tag=foo}another_filter
A single filter may be tagged with multiple tags. Example: fq={!tag=tag1,tag2,tag3}my_field:my_filter

During faceting, the facet domain may be changed to exclude filters that match certain tags via the excludeTags keyword. It’s as if the filter was never specified for that specific facet. This is useful for implementing multi-select faceting Example: colors:{type:terms, field:color, domain:{excludeTags:COLOR}}

excludeTags can be multi-valued comma-separated string. Example: excludeTags:"tag1,tag2"
excludeTags can be a JSON array of tags. Example: excludeTags:["tag1","tag2"]
One can exclude tags that are not used in the current request. This makes constructing requests simpler since you don’t need to worry about changing the faceting part of the request based on what filters have been applied.
For nested facets, excludeTags can appear at any level of the hierarchy. They do not currently “stack” though. If a parent facet has excludeTags:tag1 and a child facet wants to additionally exclude tag2 filters, then they must currently do so explicitly with excludeTags:“tag1,tag2”. Nested exclusions are experimental and subject to change.

Solr 5.4 Features

Fri, 07 Aug 2015 00:00:00 GMT

Here’s an overview of some of the new features in Solr 5.4 Also see Solr Download Links and upcoming Features of the next Solr release.

filter() operator in lucene/solr query syntax

A filter query retrieves a set of documents matching a query from the Solr filter cache. This improves performance of additional queries that use the same filter clauses. All documents matching the query produce a score of 0 by default, but this can be changed by specifying a boost.

Filter Query Example:

description:HDTV OR filter(+promotion:tv +promotion_date:[NOW/DAY TO NOW/DAY+7DAY])

docValues fields now take less heap memory

Multi-valued fields with docValues (as well as binary docValues fields), previously had an on-heap index structure pointing to the on-disk (i.e. off-heap) values. This index has been moved off-heap, and directly read as needed from the index file.

Sparse encoding for docValues fields

Sparse docValues fields with less than 1% of documents containing a value in the field, are now internally encoded with a new SPARSE_COMPRESSED method to save storage space.

Terms/field faceting has a new parameter called “method” to give an execution hint while faceting on a field.

method:uif - Stands for UninvertedField, a method of faceting indexed, multi-valued fields using top-level data structures that optimize for performance over NRT capabilities.
method:dv - Stands for DocValues, a method of faceting indexed, multi-valued fields using per-segment data structures. This method mirrors faceting on real docValues fields but works by building on-heap docValues on the fly from the index when docValues aren’t available. This method is better for a quickly changing index.
method:stream - This method creates each individual facet bucket (including any sub-facets) on-the-fly while streaming the response back to the requester. Currently only supports sorting by index order.

NOTE: currently, if a field is indexed with docValues, the dv method will be used even if method:uif is specified.

DocValue faceting performance

Faceting on DocValue fields and single-valued fields with the new Facet Module (JSON Facet API), has been optimized. When sorting by count, and when there are multiple hits expected per bucket, per-segment ords are collected before being mapped to global ords.

This optimization applies to the following field types:

single-valued and multi-valued string fields with docValues
indexed single-valued string fields
indexed multi-valued string fields, when facet.method=dv is used

Here’s an example of the speedups obtained when faceting over 5M documents on different single-valued fields:

Field with 10 unique values: +31%
Field with 100 unique values: +29%
Field with 1000 unique values: +59%
Field with 10000 unique values: +88%
Field with 1M unique values: +115%

Clusterstate migration tool

A collection API command MIGRATESTATEFORMAT that will migrate from an older-style shared clusterstate.json in Zookeeper, to per-collection cluster state files (state.json per collection).

Example:

  http://localhost:8983/solr/admin/collections?action=MIGRATESTATEFORMAT&collection=collection1

ConfigSets management API

A HTTP API to CREATE, DELETE, and LIST config sets (schema, solrconfig.xml, etc) in SolrCloud mode.

For example, the following command creates a new configset from an existing configset:

http://localhost:8983/solr/admin/configs?action=CREATE&name=booksConfig&baseConfigSet=genericTemplate

See ConfigSets API in the Solr reference guide for more details.

Force a shard leader

An expert-level FORCELEADER command has been added to help manually recover from a rare scenario where it looks like there are no replicas for a shard suitable to become the leader (i.e. when all of them are marked as recovering).

See Force Leader in the collections API section of the Solr reference guide.

Basic Auth support in SolrJ

SolrJ now has support for basic auth credentials. Those credentials need to be set for each request. Example:

  QueryRequest solrRequest = new QueryRequest(params);
  solrRequest.setBasicAuthCredentials(username, password);
  QueryResponse response = solrRequest.process(solrClient, "collection1");

Facet & Analytics Performance

Thu, 16 Jul 2015 00:00:00 GMT

These benchmarks compare the performance of the new JSON Facet API with it’s “performance-first” architecture, and the existing (legacy) Solr Facets.

Test index details:

  documents: 5M
  index segments: 25
  index size: 1.74GB
  6 single valued string fields with 10, 100, 1000, 10000, 100000, 1000000 unique values respectively.
  6 single valued integer fields as above.
  6 multi-valued string fields with 1-5 values per field, with 10, 100, 1000, 10000, 100000, 1000000 unique values respectively.
  6 multi-valued integer fields as above.
  5% chance of any given field having no values for a particular document.

Test requests details:

  Base test query and filters (the domain) matches 2,161,827 documents.
  Single client thread (and both requests only use a single internal thread per request).
  Single warm-up run per implementation that is discarded.
  Multiple runs across all fields, with fastest time being taken for each field.

Faceting + Average performance

These benchmarks test faceting on one field and finding the average value in another field per facet bucket.

JSON Facet API command:

json.facet={
  f:{
    type : terms,
    field : m100_5_ss,
    facet : { mean : "avg(s10_s)" }
  }
}

Legacy Facet command:

facet=true&
stats=true&
stats.field={!tag=stat1+mean=true}s10_s&
facet.pivot={!stats=stat1}m100_5_ss&
f.m100_5_ss.facet.limit=10

Only sorting by count was tested since legacy facets (pivot + stats component) do not support sorting buckets by anything else.

Count Distinct in Solr

Sun, 17 May 2015 00:00:00 GMT

A 100% accurate count of distinct values (count distinct) is not generally possible without actually observing all of the values together. However there are a number of ways to estimate the count.

The unique facet function is Solr’s fastest implementation to calculate the number of distinct values. It always provides exact counts on a single Solr node. For distributed search over multiple nodes, it provides exact counts when the number of values per node does not exceed 100 (by default).

When the number of unique values does exceed 100 in any given shard, the following algorithm is used:

It estimates the count by sending the top 100 results from each shard along with the total exact “unique” count for each shard.
totalSeen is the number of actual results we saw from all shards (i.e. not deduped yet).
uniqueSeen is the number of unique values we saw from all shards (i.e. deduped).
notSeen is the number of unique values from each shard that were not sent (because of the 100 cutoff).
factor = uniqueSeen / totalSeen (i.e. what fraction of values that we saw were unique)
estimate = uniqueSeen + ( notSeen * factor ) (i.e. we simply apply the factor to the number of values we didn’t see)

Example use:

$ curl http://localhost:8983/solr/techproducts/query -d '
q=*:*&
json.facet={
  x : "unique(manu_exact)"    // manu_exact is the manufacturer indexed as a single string
}'

For more facet functions, adding facet functions to each facet bucket, or sorting by facet function, see Solr Facet Functions

HyperLogLog

(New in Solr 5.2) The HyperLogLog algorithm was developed as an advanced statistical method of estimating the distinct number of values without using too much memory. It does add more calculation overhead however, and is thus slower than Solr’s “unique” facet function.

HyperLogLog requires very high quality hashes for accurate estimation. Solr uses a port of MurmurHash3 for Java to calculate these hashes.

The Facet Analytics Module and the older Stats component both have support for HyperLogLog as of Solr 5.2

A new “hll” facet function was added as an alternative to the existing faster (but less accurate for high cardinality) “unique” function.

To get the unique number of manufacturers using the HyperLogLog algorithm:

$ curl http://localhost:8983/solr/techproducts/query -d '
q=*:*&
json.facet={
  x : "hll(manu_exact)"    // manu_exact is the manufacturer indexed as a single string
}'

If we want the number of unique manufacturers per bucket of a facet:

$ curl http://localhost:8983/solr/techproducts/query -d '
q=*:*&
json.facet={
  categories: {
    type : terms,
    field : cat,
    facet : {
      x : "hll(manu_exact)"
    }
  }
}'

And we get a response containing:

  "facets":{
    "count":32,
    "categories":{
      "buckets":[{
          "val":"electronics",
          "count":12,
          "x":9},
        {
          "val":"currency",
          "count":4,
          "x":4},
    [...]

Stats Component HyperLogLog Example

To get the unique number of manufacturers per facet bucket using the Stats component:

$ curl http://localhost:8983/solr/techproducts/query -d '
q=*:*&
stats=true&
facet=true&
stats.field={!tag=stat1 cardinality=true}manu_exact&
facet.pivot={!stats=stat1}cat'

And we get a response containing:

  "facet_counts":{
    "facet_queries":{},
    "facet_fields":{},
    "facet_dates":{},
    "facet_ranges":{},
    "facet_intervals":{},
    "facet_heatmaps":{},
    "facet_pivot":{
      "cat":[{
          "field":"cat",
          "value":"electronics",
          "count":12,
          "stats":{
            "stats_fields":{
              "manu_exact":{
                "cardinality":9}}}},
        {
          "field":"cat",
          "value":"currency",
          "count":4,
          "stats":{
            "stats_fields":{
              "manu_exact":{
                "cardinality":4}}}},
   [...]

Cardinality Performance

Here is a performance comparison of the different implementations in various scenarios.

Test configuration:

Index:
  documents: 5M
  index segments: 25
  index size: 1.74GB
  6 single valued string fields with 10, 100, 1000, 10000, 100000, 1000000 unique values respectively.
  6 single valued integer fields as above.
  6 multi-valued string fields with 1-5 values per field, with 10, 100, 1000, 10000, 100000, 1000000 unique values respectively.
  6 multi-valued integer fields as above.
  5% chance of any given field having no values for a particular document.

Queries:
  Base test query and filters (the domain) matches 2,161,827 documents.
  Single client thread (and both requests only use a single internal thread per request).
  Single warm-up run per implementation that is discarded.
  Multiple runs across all fields, with fastest time being taken for each field.

Legacy Faceting command (stats component + pivot faceting):

facet=true&stats=true&stats.field={!tag=stat1+cardinality=true}s10_s&facet.pivot={!stats=stat1}m100_5_ss&f.m100_5_ss.facet.limit=10

JSON Faceting command (new Facet Module):

json.facet={
  f:{
    type : terms,
    field : m100_5_ss,
    facet : { stat1:"hll(s10_s)" }
  }
}

This first test facets on the multi-valued string field m100_5_ss (it has up to 5 values per field, and 100 unique values in total). Then for the top 10 buckets, the cardinality of the single-valued string field (the “stat field”) is calculated. The chart below shows performance for different number of unique values in the stat field.

This next test reverses the 2 fields above, first faceting on the single valued string field s100_s (it has 100 unique values in total) and then calculating cardinality over multi-valued string fields with different numbers of unique values in the index.

The last test facets on the integer field s100_i and then calculates cardinality over another integer field with varying number of unique terms in the index.

Solr 5.3 Features

Sun, 17 May 2015 00:00:00 GMT

Here’s an overview of some of the new features in Solr 5.3 Also see Solr Download Links and upcoming Features of the next Solr release.

Faceting Nested Documents

The JSON Facet API can now change the domain for facet commands, essentially doing a block join and moving from parents to children, or children to parents before calculating the facet data.

For example, if you indexed chapters with pages as nested child documents, then you could map from chapters to pages before faceting by adding the following parameter to the facet command:

   domain : { blockChildren : "type:chapter" }

Or if you started with pages, you could map to chapters with

   domain : { blockParent : "type:chapter" }

Note that in both cases, we provide the parent filter (how parent documents are defined) of “type:chapter” regardless of which direction we are mapping.

See this Nested Objects tutorial for complete examples of combining faceting and block join / nested documents.

Major improvements in performance of the new Facet Module / JSON Facet API. See the facet performance benchmarks for more details and benchmark results.

Just like the JSON Facet API, pivot facets can how nest other facet types such as range and query facets.

Example:

  &facet=true
  &facet.range={!tag=r1}price
  &f.price.facet.range.start=0
  &f.price.facet.range.end=100
  &f.price.facet.range.gap=10
  &facet.query={!tag=q1}popularity:[8 TO 10]
  &facet.pivot={!range=r1 query=q1}category

The equivalent in the JSON Facet API would be:

json.facet={
  categories : {
    type : terms,
    field : category,
    facet : {
      r1 : {
        type : range,
        start : 0,
        end : 100,
        gap : 10
      },
      q1 : { query : "popularity:[8 TO 10]" }
    }
  }
}

More Like This Query Parser options

The MoreLikeThis QParser mlt now supports all options provided by the MLT Handler. The query parser is much more versatile than the handler as it works in cloud mode as well as anywhere a normal query can be specified.

Example (on techproducts index):

q={!mlt qf=name mintf=1 mindf=1}SP2514N

More documentation on the mlt parser can be found in the Solr Ref Guide

Schema API support in SolrJ

The new SchemaRequest Java class in SolrJ can be used to make requests to the Schema API.

Also see the Solr Schema API itself in the ref guide.

Scoring mode for query-time join and block join

Solr’s pseudo-join query parser has a new optional attribute score that can be used specify the scores produced on the resulting documents. It’s value can be min, max,avg,or total.

Query-time join example:

q={!join from=author_id to=id score=total}blog_text:awesome

Block join example:

q={!parent of=type:author score=total}blog_text:awesome

See Nested Objects in Solr for more information on nested documents and block join.

Query Comments

Lucene/Solr query syntax (i.e. Solr’s dialect of the lucene syntax) now supports nested C-style comments.

+cat:electronics /* this is a comment */ +name:HDTV

Smile data format

Smile is a binary data interchange format that is very close to Solr’s own “javabin” (encoded sizes are very close). Adding wt=smile to a request will cause the response to come back in this format.

min/max for multi-valued docValues fields

A second parameter has been added to the field function to select the minimum or maximum value of a multi-valued field with docValues.

Example:

sort=field(my_dv_field,max) asc

HTTP Basic Authentication

In addition to many other improvements in the security framework, Solr now includes an AuthenticationPlugin implementing HTTP Basic Auth that stores credentials securely in ZooKeeper. This is a simple way to require a username and password for anyone accessing Solr’s admin screen or APIs.

See the Basic Authentication Plugin section of the Solr ref guide under the Securing Solr section.

JSON Facet API

Fri, 17 Apr 2015 00:00:00 GMT

Related Pages

Introduction

Solr 5 has a completely re-written faceted search and analytics module with a structured JSON API to control the faceting and analytics commands. NOTE: Some examples use syntax only supported in later Solr 5 releases, or even Solr 6. Download a recent Solr release or snapshot to try them out.

The structured nature of nested sub-facets are more naturally expressed in a nested structure like JSON rather than the flat structure that normal query parameters provide.

Goals of the new Faceting Module:

First class JSON support
Easier programmatic construction of complex nested facet commands
Support a much more canonical response format that is easier for clients to parse
First class analytics support
Ability to sort facet buckets by any calculated metric
Support a cleaner way to do distributed faceting
Support better integration with other search features

Of course if you prefer to use Solr’s existing faceting capabilities, that’s fine too. You can even use both at once if you want!

UPDATE: The JSON Facet API is now part of the JSON Request API, so a complete request may be expressed in JSON.

Ease of Use

Some of the ease-of-use enhancements over traditional Solr faceting come from the inherent nested structure of JSON. As an example, here is the faceting command for two different range facets using Solr’s flat legacy API:

&facet=true
&facet.range={!key=age_ranges}age
&f.age.facet.range.start=0
&f.age.facet.range.end=100
&f.age.facet.range.gap=10
&facet.range={!key=price_ranges}price
&f.price.facet.range.start=0
&f.price.facet.range.end=1000
&f.price.facet.range.gap=50

And here is the equivalent faceting command in the new JSON Faceting API:

{
  age_ranges: {
    type : range
    field : age,
    start : 0,
    end : 100,
    gap : 10
  }
  ,
  price_ranges: {
    type : range
    field : price,
    start : 0,
    end : 1000,
    gap : 50
  }
}

These aren’t even nested facets, but already one can see how much nicer the JSON API looks. With deeply nested sub-facets and statistics, the clarity of the inherently nested JSON API only grows.

JSON extensions

A number of JSON extensions have been implemented to further increase the clarity and ease of constructing a JSON faceting command by hand. For example:

{  // this is a single-line comment, which can help add clarity to large JSON commands
     /* traditional C-style comments are also supported */
  x : "avg(price)" , // Simple strings can occur unquoted
  y : 'unique(manu)' // Strings can also use single quotes (easier to embed in another String)
}

Debugging JSON

Nicely indented JSON is very easy to understand. If you get a large piece of non-indented JSON somehow, and are trying to make sense of it, you can cut and paste into one of the online validators: http://jsonlint.com http://jsonformatter.curiousconcept.com Both of these validators will indent your JSON, even when it contains extensions unsupported by them (such as comments or bare strings).

There are two types of facets, one that breaks up the domain into multiple buckets, and aggregations / facet functions that provide information about the set of documents belonging to each bucket.

Faceting can be nested! Any bucket produced by faceting can further be broken down into multiple buckets by a sub-facet.

Statistics are now fully integrated into faceting. Since we start off with a single facet bucket with a domain defined by the main query and filters, we can even ask for statistics for this top level bucket, before breaking up into further buckets via faceting. Example:

json.facet={
  x : "avg(price)",           // the average of the price field will appear under "x"
  y : "unique(manufacturer)"  // the number of unique manufacturers will appear under "y"
}

See facet functions for a complete list of the available aggregation functions.

The general form of the JSON facet commands are: <facet_name> : { <facet_type> : <facet_parameter(s)> } Example: top_authors : { terms : { field : authors, limit : 5 } }

After Solr 5.2, a flatter structure with a “type” field may also be used: <facet_name> : { "type" : <facet_type> , <other_facet_parameter(s)> } Example: top_authors : { type : terms, field : authors, limit : 5 }

The results will appear in the response under the facet name specified. Facet commands are specified using json.facet request parameters.

Test Using Curl

To test out different facet requests by hand, it’s easiest to use “curl” from the command line. Example:

$ curl http://localhost:8983/solr/query -d 'q=*:*&rows=0&
 json.facet={
   categories:{
     type : terms,
     field : cat,
     sort : { x : desc},
     facet:{
       x : "avg(price)",
       y : "sum(price)"
     }
   }
 }
'

The terms facet, or field facet, produces buckets from the unique values of a field. The field needs to be indexed or have docValues.

The simplest form of the terms facet

{
  top_genres : { terms : genre_field }
}

An expanded form allows for more parameters:

{
  top_genres : {
    type : terms,
    field : genre_field,
    limit : 3,
    mincount : 2
  }
}

Example response:

    "top_genres":{
      "buckets":[
        {
          "val":"Science Fiction",
          "count":143},
        {
          "val":"Fantasy",
          "count":122},
        {
          "val":"Biography",
          "count":28}
      ]
    }

Parameters:

field - The field name to facet over.
offset - Used for paging, this skips the first N buckets. Defaults to 0.
limit - Limits the number of buckets returned. Defaults to 10.
mincount - Only return buckets with a count of at least this number. Defaults to 1.
sort - Specifies how to sort the buckets produced. “count” specifies document count, “index” sorts by the index (natural) order of the bucket value. One can also sort by any facet function / statistic that occurs in the bucket. The default is “count desc”. This parameter may also be specified in JSON like sort:{count:desc}. The sort order may either be “asc” or “desc”
missing - A boolean that specifies if a special “missing” bucket should be returned that is defined by documents without a value in the field. Defaults to false.
numBuckets - A boolean. If true, adds “numBuckets” to the response, an integer representing the number of buckets for the facet (as opposed to the number of buckets returned). Defaults to false.
allBuckets - A boolean. If true, adds an “allBuckets” bucket to the response, representing the union of all of the buckets. For multi-valued fields, this is different than a bucket for all of the documents in the domain since a single document can belong to multiple buckets. Defaults to false.
prefix - Only produce buckets for terms starting with the specified prefix.
method - Provides an execution hint for how to facet the field.
- method:uif - Stands for UninvertedField, a method of faceting indexed, multi-valued fields using top-level data structures that optimize for performance over NRT capabilities.
- method:dv - Stands for DocValues, a method of faceting indexed, multi-valued fields using per-segment data structures. This method mirrors faceting on real docValues fields but works by building on-heap docValues on the fly from the index when docValues aren’t available. This method is better for a quickly changing index.
- method:stream - This method creates each individual facet bucket (including any sub-facets) on-the-fly while streaming the response back to the requester. Currently only supports sorting by index order.

The query facet produces a single bucket that matches the specified query.

An example of the simplest form of the query facet

{
  high_popularity : { query : "popularity:[8 TO 10]" }
}

An expanded form allows for more parameters (or sub-facets / facet functions):

{
  high_popularity : {
    type : query,
    q : "popularity:[8 TO 10]",
    facet : { average_price : "avg(price)" }
  }
}

Example response:

  "high_popularity" : {
    "count" : 147,
    "average_price" : 74.25
  }

The range facet produces multiple range buckets over numeric fields or date fields.

Range facet example:

{
  prices : {
    type : range,
    field : price,
    start : 0,
    end : 100,
    gap : 20
  }
}

Example response:

    "prices":{
      "buckets":[
        {
          "val":0.0,  // the bucket value represents the start of each range.  This bucket covers 0-20
          "count":5},
        {
          "val":20.0,
          "count":3},
        {
          "val":40.0,
          "count":2},
        {
          "val":60.0,
          "count":1},
        {
          "val":80.0,
          "count":1}
      ]
    }

To ease migration, these parameter names, values, and semantics were taken directly from the old-style (non JSON) Solr range faceting.

Parameters:

field - The numeric field or date field to produce range buckets from
mincount - Minimum document count for the bucket to be included in the response. Defaults to 0.
start - Lower bound of the ranges
end - Upper bound of the ranges
gap - Size of each range bucket produced
hardend - A boolean, which if true means that the last bucket will end at “end” even if it is less than “gap” wide. If false, the last bucket will be “gap” wide, which may extend past “end”.
other - This param indicates that in addition to the counts for each range constraint between facet.range.start and facet.range.end, counts should also be computed for…
"before" all records with field values lower then lower bound of the first range
"after" all records with field values greater then the upper bound of the last range
"between" all records with field values between the start and end bounds of all ranges
"none" compute none of this information
"all" shortcut for before, between, and after
include - By default, the ranges used to compute range faceting between facet.range.start and facet.range.end are inclusive of their lower bounds and exclusive of the upper bounds. The “before” range is exclusive and the “after” range is inclusive. This default, equivalent to lower below, will not result in double counting at the boundaries. This behavior can be modified by the facet.range.include param, which can be any combination of the following options…
"lower" all gap based ranges include their lower bound
"upper" all gap based ranges include their upper bound
"edge" the first and last gap ranges include their edge bounds (ie: lower for the first one, upper for the last one) even if the corresponding upper/lower option is not specified
"outer" the “before” and “after” ranges will be inclusive of their bounds, even if the first or last ranges already include those boundaries.
"all" shorthand for lower, upper, edge, outer

Common Parameters

Parameters that all faceting methods have in common include

domain - Facet domain transformations, to change the incoming domain of the facet command before faceting is executed. This is useful for multi-select faceting and nested document (block join) faceting. Additional filters to be applied to the domain can also be specified here.

Solr 'n Stuff | Blog

Solr 7.1 Features

JSON Query Syntax / JSON Query DSL

Integral and date type support for min/max

Auto-scaling enhancements

Polygons in Spatial RPT fields

Streaming Expressions functions

Solr 8 Features

IN PROGRESS

Solr 6.6 Features

Fields and field types

New API

Script

Faceting

Streaming expressions and evaluators

Examples

Metrics

Upgrades/deprecations/removals

SolrJ

Others

Optimizations

Solr 6.5 Features

Field Type related changes

Query

Highlighting

Streaming Expressions

Ease of use / Admin UI

Metrics

Optimizations

Misc Changes

Facet Domains

What is the Facet Domain

Changing the Facet Domain

blockParent

blockChildren

excludeTags

filter

Solr 6.4 Features

filter list for facet domain

Learning to Rank

snapshotcli.sh command line tool to manage snapshots

More efficient filter queries

having streaming expression

Nested Objects in Solr

Nested Documents

Why Nested Documents

Lucene Index Representation

Limitations

Schema Requirements

Block Join

Indexing Nested Documents

Block Join Query

Returning Child Documents

Faceting

Faceting on Parents

Faceting on Children

Filtering Children

The Parent Filter

Solr JSON Request API

JSON Request API Motivation

Simple Example

Passing JSON via Request Parameter

Smart merging of multiple JSON parameters

Simple Example

Facet Example

Debugging

Passing Parameters via JSON

Error Detection

Parameter Substitution / Macro Expansion

Current Status

Solr 6.2 Features

Boolean Comparison Function Queries

Feature Selection and Logistic Regression

topic streaming expression (Beta)

scoreNodes graph streaming expression

collections API: REPLACENODE and DELETENODE commands

Kerberos security enhancements

Whole index replication for CDCR (Cross Data Center Replication)

“role” tag in replica placement rules

Solr 6.3 Features

`filter` list for facet domain

`having` streaming expression

`topic` streaming expression (Beta)

`scoreNodes` graph streaming expression

collections API: `REPLACENODE` and `DELETENODE` commands

New `dvhash` faceting method in the JSON Facet API