Blog

Solr 7.1 Features

Sep 3, 2017

Yonik Seeley

Here’s an overview of some of the new features in Solr 7.1: Download Solr 7.1 or later to try these features out and give us feedback!

 

JSON Query Syntax / JSON Query DSL

There is now a JSON mapping to Solr QParsers. Currently, one must use the JSON Request API to use this JSON syntax, but SOLR-11295 will provide a more general purpose entry point to JSON syntax in the future.

The general form of a query type in existing local params syntax is:

 {!query_type param1=val1 param2=val2}query_value

OR, specifying the main query value using the v parameter:

 {!query_type param1=val1 param2=val2 v=query_value}

In the new JSON syntax, the corresponding mappings would be

 {query_type:{param1:val1, param2:val2, v:query_value}}

OR using query (which is a synonym for v when mapping to local params)

 {query_type:{param1:val1, param2:val2, query:query_value}}

  Here’s an example of a pseudo-join query in local-params syntax that queries for book_review1 and follows the “book_id” field of the result(s) to the matching “id” field:

curl http://localhost:8983/solr/techproducts/query -d 'q={!join from:book_id to:id}id:book_review1'

In JSON syntax (indented for better readability), this would be

curl http://localhost:8983/solr/techproducts/query -d '
{
  query:{
    join:{
      from : book_id,
      to : id,
      query : "id:book_review1"
    }
  }
}'

Note that the query parameter is specified in lucene syntax above, but it can be any query in local-params syntax or JSON syntax. Example:

curl http://localhost:8983/solr/techproducts/query -d '
{
  query:{
    join:{
      from : book_id,
      to : id,
      query : { field : {f:"id", v:"book_review1"} }  // invokes the "field" QParser
    }
  }
}'

 

Integral and date type support for min/max

Min and max aggregations in the JSON Facet API have been updated to include support for int, long, and date types. Previously, min and max of all of these field types returned double results.

 

Auto-scaling enhancements

Solr’s auto-scaling framework contains a number of new features and improvements, including

• Ability to move replicas when nodes are added or removed to the cluster.
• Solr’s autoAddReplicas feature now uses the autoscaling framework and works for all filesystems.
• New API to control triggers and listeners (set-trigger, remove-trigger, suspend-trigger, set-listener, etc.)

As of this writing, the Solr 7.1 reference guide is not published yet. However you can check out the auto-scaling section of the very latest version of the unreleased reference guide for the master branch.

 

Polygons in Spatial RPT fields

Here’s an example field using the new Geo3D spatialContextFactory that supports polygons:

 <fieldType name="srptgeom_geo3d" class="solr.RptWithGeometrySpatialField"
             spatialContextFactory="Geo3D" planetModel="wgs84"/>

Note: “wgs84” is an ellipsoid coordinate system for the Earth used by GPS. “sphere” can also be used for a spherical model that has faster but less accurate distance calculations.

 

Streaming Expressions functions

Expanded support for statistical functions including various distributions, rank correlations, distances and more:

harmonicFit, polyfit, polyfitDerivative, betaDistribution, gammaDistribution, zipFDistribution, logNormalDistribution, weibullDistribution, monteCarlo, expMovingAverage, binomialCoefficient, factorial, movingMedian, primes, sumDifference, meanDifference, ebeAdd, ebeSubtract, ebeDivide, ebeMultiply, dotProduct, cosineSimilarity, Canberra, Chebyshev, Earth Movers and Manhattan Distance, Kendall’s Tau-b rank and Spearmans rank, discrete counting and probability, cumulativeProbability, normalDistribution, uniformDistribution, kolmogorovSmirnov, addAll

Solr 8 Features

Sep 3, 2017

Yonik Seeley

Here’s an overview of some of the new features in the next major upcoming release: Solr 8.0 Download a Solr 8 nightly build to try these features out and give us feedback!

 

IN PROGRESS

We’re in the progress of releasing Solr 7, so there’s not much new here yet. Check back later!

Solr 6.6 Features

Jun 2, 2017

Yonik Seeley

Here’s an overview of some of the new features in Solr 6.6, released on June 6, 2017.

Download Solr 6.6 to try these features out and give us feedback! You can also check out upcoming features of the next Solr release.

This is likely to be the last Solr 6.x release before Solr 7.

  The release notes from the Apache Solr Wiki:

Fields and field types

• Payload support with payload() value source and {!payload_score} and {!payload_check} query parsers
• Solr support for SimpleTextCodec, via in solrconfig.xml (per-field specification in the schema is not possible)
• Multi-field support to TermsComponent when requesting terms’ statistics
• Support for PointFields in Grouping, CollapseQParser, and ExpandComponent.

New API

• UPLOAD command (Config Set API) for uploading zipped configsets
• MOVEREPLICA command (Collections API) for moving a replica across nodes
• LISTALIASES command (Collections API) to return a list of all collection aliases
• STATUS command (Core Admin API) to emit collection details of each core

Script

• Basic authentication can be enabled/disabled using bin/solr|bin/solr.cmd
• ls command to ZkCLI for listing only sub-directories

Faceting

• Variance and Standard Deviation aggregators for the JSON Facet API
• JSON Faceting now supports a query time ‘join’ domain change option

Streaming expressions and evaluators

• CartesianProductStream, which turns a single tuple with a multi-valued field into N tuples, one for each value in the multi-valued field
• stats and search Streaming Expressions should now work in non-SolrCloud mode
• analyze Stream Evaluator to support streaming NLP
• New Stream Evaluators: Basic math, Date/time, UUID, Correlation, regress, predict, covariance, convolution, normalize
• New Streaming Expressions: shuffle, echo, eval, timeseries, let, get

Examples

• Solr default/example uses WordDelimiterGraphFilterFactory and SynonymGraphFilterFactory
• New DataImportHandler ‘atom’ example, replacing broken ‘rss’ example
• Redone DataImportHandler ‘tika’ example, removing all unused and irrelevant definitions

Metrics

• Expose cache statistics using metrics API
• Improvements to metric reporters and API: support for “regex” parameter in /admin/metrics, “enabled” flag in reporter configurations, correct handling of “serviceUrl” in SolrJmxReporter, better handling of service clients for JMX, Ganglia and Graphite reporters

Upgrades/deprecations/removals

• Deprecated LatLonType, GeoHashField, SpatialPointVectorFieldType, and SpatialTermQueryPrefixTreeFieldType. Instead, switch to LatLonPointSpatialField or SpatialRecursivePrefixTreeFieldType or RptWithGeometrySpatialField
• Deprecated PostingsSolrHighlighter. Use UnifiedSolrHighlighter instead.

SolrJ

• CloudSolrClient can now be initialized using the base URL of a Solr instance instead of ZooKeeper hosts
• SolrJ: Added SolrParams.toLocalParamsString() and ClientUtils.encodeLocalParamVal

Others

• New AtomicUpdateProcessor to convert normal update operations to atomic update operations
• totalTermFreq support to TermsComponent
• Hide keystore and truststore passwords from /admin/info/* outputs
• Configurability for thread pool size to recoveryExecutor
• Introducing sort=childfield(field) asc for searching by {!parent}

Optimizations

• facet.heatmap is now significantly faster when the docset (base query) matches everything and there are no deleted docs. It is also faster when the docset matches a small fraction of the index or none
• Reduced heap consumption for filter({!join … score=…})
• JSON Facet API now uses hyper-log-log++ for determining the number of buckets when merging requests from a multi-shard distributed request
• Better ZkStateWriter batching
• Using cache for DistributedQueue in case of single-consumer

Solr 6.5 Features

Jan 23, 2017

Yonik Seeley

Here’s an overview of some of the new features in Solr 6.5. Download Solr 6.5 to try these features out and give us feedback!

This feature list was adapted from the release notes on the wiki: https://wiki.apache.org/solr/ReleaseNote65

Field Type related changes

• PointFields (fixed-width multi-dimensional numeric & binary types enabling fast range search) are now supported
• In-place updates to numeric docValues fields (single valued, non-stored, non-indexed) supported using atomic update syntax
• A new LatLonPointSpatialField that uses points or doc values for query
• It is now possible to declare a field as “large” in order to bypass the document cache

Query

• New sow=false request param (split-on-whitespace) for edismax & standard query parsers enables query-time multi-term synonyms
• XML QueryParser (defType=xmlparser) now supports span queries

Highlighting

• hl.maxAnalyzedChars now have consistent default across highlighters
• UnifiedSolrHighlighter and PostingsSolrHighlighter now support CustomSeparatorBreakIterator

Streaming Expressions

• Scoring formula is adjusted for the scoreNodes function
• Calcite Planner now applies constant Reduction Rules to optimize plans
• A new significantTerms Streaming Expression that is able to extract the significant terms in an index
• StreamHandler is now able to use runtimeLib jars
• Arithmetic operations are added to the SelectStream

Ease of use / Admin UI

• Added modernized self-documenting /v2 API
• The .system collection is now created on first request if it does not exist
• Admin UI: Added shard deletion button

Metrics

• Metrics API now supports non-numeric metrics (version, disk type, component state, system properties…)
• The disk free and aggregated disk free metrics are now reported
• The DirectUpdateHandler2 now implements MetricsProducer and exposes stats via the metrics api and configured reporters.

Optimizations

• BlockCache is faster due to less failures when caching a new block
• MMapDirectoryFactory now supports “preload” option to ask mapped pages to be loaded into physical memory on init

Misc Changes

• Security: BasicAuthPlugin now supports standalone mode
• Arbitrary java system properties can be passed to zkcli
• SolrHttpClientBuilder can be configured via java system property
• Javadocs and Changes.html are no longer included in the binary distribution, but are hosted online

For more detailed lists with pointers to JIRA issues, see the HTML version of CHANGES.txt

Facet Domains

Nov 1, 2016

Yonik Seeley

What is the Facet Domain

The domain of a facet is the set of values (normally defined by a set of documents) that calculations will be done over. The root domain is the set of documents that match the base query and any filters.

Changing the Facet Domain

For any facet command, one can use the domain keyword to change the facet domain before facet computation.

blockParent

The existence of the blockParent parameter in the domain block will cause incoming child documents to be mapped to their parents. The value of the blockParent parameter is the parent filter that specifies the complete set of parent documents for this block join operation.

Nested Documents -> Faceting On Parents for an example.

blockChildren

The existence of the blockChildren parameter in the domain block will cause incoming parent documents to be mapped to their children. The value of the blockChildren parameter is the parent filter that specifies the complete set of parent documents for this block join operation.

Nested Documents -> Faceting On Children for an example.

excludeTags

The excludeTags parameter causes the domain to be re-calculated from the root all the way to the current point, but excluding any filters with the specified tags. This is normally used as part of multi-select faceting.

filter

The filter parameter is used to specify a filter or list of filters to be intersected with the incoming domain before faceting. These filters are applied after other domain transformations such as blockParent, blockChildren, or excludeTags. The same exact syntax for filter is supported in the JSON Request API for the top level document list (or will be after SOLR-9733).

Example:

json.facet = {
  categories : {
    type : terms,
    field : cat,
    domain : { filter:"user:yonik" }
  }
}

The “param” query type can be used to grab zero or more filters from a request parameter:

q=my query&
myfilt=type:review&
myfilt=rating:5&
json.facet = {
  categories : {
    type : terms,
    field : cat,
    domain : {
      filter:[
        "user:yonik",
        {param : myfilt} ]
    }
  }
}

Solr 6.4 Features

Oct 30, 2016

Yonik Seeley

Here’s an overview of some of the new features in Solr 6.4.

Download Solr 6.4 to try these features out and give us feedback! You can also check out upcoming features of the next Solr release.

filter list for facet domain

Any JSON facet command (terms, range, query) can now filter the facet domain in a simpler manner, without resorting to nested query facets.

Example:

json.facet = {
  categories : {
    type : terms,
    field : cat,
    domain : { filter:"user:yonik" }
  }
}

The filters are applied after other domain change operations and are particularly useful when faceting on child documents. The filter attribute can be a single query or a list of multiple queries to intersect.

Using a param to refer to a filter by query parameter was added shortly after.

 

Learning to Rank

Learning to Rank (LTR) plugin for reranking results with machine learning models.

See the Lucene/Solr Revolution presentation Learning to Rank in Solr as well as the Solr Ref Guide.

 

snapshotcli.sh command line tool to manage snapshots

The full path of the script is ./solr/server/scripts/cloud-scripts/snapshotscli.sh

An example from the pull request:

// Start solr and initialize a sample collection
bin/solr start -c
bin/solr create_collection -c books
curl 'http://localhost:8983/solr/books/update?commit=true' -H 'Content-type:application/json' -d '
[ {"id" : "book1", "title" : "American Gods", "author" : "Neil Gaiman" } ]'

//Create and export a snapshot
./snapshotscli.sh --create snap-1 -c books -z localhost:9983
./snapshotscli.sh --list -c books -z localhost:9983
./snapshotscli.sh --describe snap-1 -c books -z localhost:9983
./snapshotscli.sh --export snap-1 -c books -z localhost:9983 -d /tmp
./snapshotscli.sh --delete snap-1 -c books -z localhost:9983

// Restore the backup and verify the doc count
curl 'http://localhost:8983/solr/admin/collections?action=restore&name=snap-1&location=/tmp&collection=books_restored'
curl 'http://localhost:8983/solr/books_restored/select?q=*:*'

 

More efficient filter queries

When parsing filter queries (including fq parameters) the standard solr query parser will avoid using BooleanQuery for term disjunctions on string and numeric fields, and will use TermsQuery instead.

This has a number of positive effects:

• Avoids Lucene’s dreaded static maxBooleanClauses issue that causes “too many boolean clauses” exceptions
• The resulting query should be smaller to cache
• The resulting query should have higher performance

For example, the following filter will now be faster, and will no longer throw a “too many boolean clauses” exception:

fq=id:(myid1 myid2 myid3 myid4 ... myid2000)

 

having streaming expression

The having streaming expression provides functionality like the SQL HAVING clause, and will be used to implement that in the future. Aggregated buckets produced by rollup are filtered by a having clause.

The following streaming expression filters out buckets (authors) with less than 100,000 in sales across all of their books:

having(rollup(over=author,
              sum(sales),
              search(book_sales_collection,
                     q=*:*,
                     fl="author,sales",
                     sort="author asc")),
       gte(sum(sales), 100000))

Also see the a streaming expressions documentation in the ref guide.

Nested Objects in Solr

Oct 26, 2016

Yonik Seeley

Nested Documents

Nested Documents (also called Nested Objects) provides the ability to “nest” some documents inside of other documents in a parent/child relationship.

Why Nested Documents

One reason for using nested documents is to prevent false matches. For example, we may have a T-Shirt with 2 SKUs, a Large Red, and a Medium Blue.

Say we tried to model this as a single document:

{
 product : "Awesome T-Shirt",
 color : [ "Red", "Blue" ],
 size : [ "L", "M" ]
}

Now if we search for color:RED AND size:M , it would incorrectly match our document! But if we represented the SKUs as two different documents, then there would be no incorrect match.

{
 color : "Red",
 size : "L",
}
{
 color : "Blue",
 size : "M",
}

Lucene Index Representation

Lucene has a flat object model and does not really support “nesting” of documents in the index. Lucene *does* support adding a list of documents atomically and contiguously (i.e. a virtual “block”), and this is the feature used by Solr to implement “nested objects”.

When you add a parent document with 3 children, these appear int the index contiguously as

child1, child2, child3, parent

There is no Lucene-level information that links parent and child, or distinguishes this parent/child block from the other documents in the index that come before or after. Successfully using parent/child relationships relies on more information being provided at query time.

Limitations

All children of a parent document must be indexed together with the parent document. One cannot update any document (parent or child) individually. The entire block needs to be re-indexed of any changes need to be made.

Schema Requirements

There are no schema requirements except that the _root_ field must exist (but that is there by default in all our schemas). Any document can have nested child documents.

Block Join

“Block Join” refers to the set of related query technologies to efficiently map from parents to children or vice versa at query time. The locality of children and parents can be used to both speed up query operations and lower memory requirements compared to other join methods.

Indexing Nested Documents

NOTE: This example currently requires Solr 5.3 or later.

First, bring up Solr and create a collection (if you have not done so already):

$ bin/solr start            # this starts solr
$ bin/solr create -c demo   # this creates a document collection called "demo"

Let’s remove any leftover docs from other examples:

curl http://localhost:8983/solr/demo/update?commitWithin=3000 -d '{delete:{query:"*:*"}}'

Now let’s add a book with some reviews as nested child documents (notice the _childDocuments_ element):

$ curl http://localhost:8983/solr/demo/update?commitWithin=3000 -d '
[
 {id : book1, type_s:book, title_t : "The Way of Kings", author_s : "Brandon Sanderson",
  cat_s:fantasy, pubyear_i:2010, publisher_s:Tor,
  _childDocuments_ : [
    { id: book1_c1, type_s:review, review_dt:"2015-01-03T14:30:00Z",
      stars_i:5, author_s:yonik,
      comment_t:"A great start to what looks like an epic series!"
    }
    ,
    { id: book1_c2, type_s:review, review_dt:"2014-03-15T12:00:00Z",
      stars_i:3, author_s:dan,
      comment_t:"This book was too long."
    }
  ]
 }
]'

Now we can see that these are really just indexed as 3 documents, all visible by default:

curl http://localhost:8983/solr/demo/query -d 'q=*:*&fl=id'

  "response":{"numFound":3,"start":0,"docs":[
      {
        "id":"book1_c1"},
      {
        "id":"book1_c2"},
      {
        "id":"book1"}]
  }

Now lets add an additional document with nested child documents for use with our query examples:

$ curl http://localhost:8983/solr/demo/update?commitWithin=3000 -d '
[
 {id : book2, type_s:book, title_t : "Snow Crash", author_s : "Neal Stephenson",
  cat_s:sci-fi, pubyear_i:1992, publisher_s:Bantam,
  _childDocuments_ : [
    { id: book2_c1, type_s:review, review_dt:"2015-01-03T14:30:00Z",
      stars_i:5, author_s:yonik,
      comment_t:"Ahead of its time... I wonder if it helped inspire The Matrix?"
    }
    ,
    { id: book2_c2, type_s:review, review_dt:"2015-04-10T9:00:00Z",
      stars_i:2, author_s:dan,
      comment_t:"A pizza boy for the Mafia franchise? Really?"
    }
    ,
    { id: book2_c3, type_s:review, review_dt:"2015-06-02T00:00:00Z",
      stars_i:4, author_s:mary,
      comment_t:"Neal is so creative and detailed! Loved the metaverse!"
    }
  ]
 }
]'

Block Join Query

TODO

Returning Child Documents

One can return child documents along with every returned parent document by using the [child] doc transformer (it’s added to the fl field list parameter ). The list of child documents will be included under the “_childDocuments_” field of each parent.

$ curl http://localhost:8983/solr/demo/query -d '
q=cat_s:(fantasy OR sci-fi)&
fl=id,[child parentFilter=type_s:book]'

  "response":{"numFound":2,"start":0,"docs":[
      {
        "id":"book1",
        "_childDocuments_":[
        {
          "id":"book1_c1",
          "type_s":"review",
          "review_dt":"2015-01-03T14:30:00Z",
          "stars_i":5,
          "author_s":"yonik",
          "comment_t":["A great start to what looks like an epic series!"]},
        {
          "id":"book1_c2",
          "type_s":"review",
[...]

Child Doc Transformer Parameters:

• parentFilter - identifies all of the parents. See the section on The Parent Filter for more info.
• childFilter - optional query to filter which child documents should be included.
• limit - maximum number of child documents to return per parent (defaults to 10)

Also see the Solr ref guide entry on the [child] doc transformer.

Faceting

The JSON Facet API has support for switching the facet domain based on the nested document relationships.

Faceting on Parents

The main query gives us a document list of reviews by author_s:yonik If we want to facet on the book genre (cat_s field) then we need to switch the domain from the children (type_s:reviews) to the parents (type_s:books).

$ curl http://localhost:8983/solr/demo/query -d '
q=author_s:yonik&fl=id,comment_t&
json.facet={
  genres : {
    type: terms,
    field: cat_s,
    domain: { blockParent : "type_s:book" }
  }
}'

And we get a facet over the books which yonik reviewed:

  "response":{"numFound":2,"start":0,"docs":[
      {
        "id":"book1_c1",
        "comment_t":["A great start to what looks like an epic series!"]},
      {
        "id":"book2_c1",
        "comment_t":["Ahead of its time... I wonder if it helped inspire The Matrix?"]}]
  },
  "facets":{
    "count":2,
    "genres":{
      "buckets":[{
          "val":"fantasy",
          "count":1},
        {
          "val":"sci-fi",
          "count":1}]
  }}

Faceting on Children

Now lets say we’re displaying the top sci-fi and fantasy books, and we want to find out who reviews the most books out of our selection. Since our root implicit facet bucket (formed by the query and filters) consists of parent documents (books), we need to switch the facet domain to the children for the author facet.

$ curl http://localhost:8983/solr/demo/query -d '
q=cat_s:(sci-fi OR fantasy)&fl=id,title_t&
json.facet={
  top_reviewers : {
    type: terms,
    field: author_s,
    domain: { blockChildren : "type_s:book" }
  }
}'

Response:

  "response":{"numFound":2,"start":0,"docs":[
      {
        "id":"book1",
        "title_t":["The Way of Kings"]},
      {
        "id":"book2",
        "title_t":["Snow Crash"]}]
  },
  "facets":{
    "count":2,
    "top_reviewers":{
      "buckets":[{
          "val":"dan",
          "count":2},
        {
          "val":"yonik",
          "count":2},
        {
          "val":"mary",
          "count":1}]
  }}

 

Filtering Children

By default, blockChildren will match all children of every parent doc from the input domain. It’s often the case that only a subset of the children are desired. The easiest way to limit children is with the filter clause.

For example, if we wanted to find the same top reviewers as before, but only for 5 star reviews:

$ curl http://localhost:8983/solr/demo/query -d '
q=cat_s:(sci-fi OR fantasy)&fl=id,title_t&
json.facet={
  top_reviewers : {
    type: terms,
    field: author_s,
    domain: {
      blockChildren : "type_s:book",
      filter : "stars_i:5"
    }
  }
}'

The Parent Filter

Note that regardless of which direction we are mapping (parents to children or children to parents), or what documents we are operating on, we provide a parent filter to define the complete set of parents in the index. In these examples, the parent filter is "type_s:book".

Solr JSON Request API

Oct 18, 2016

Yonik Seeley

JSON Request API Motivation

Although query parameters are often an easy method to create a Solr requests by hand, they have a number of drawbacks:

• Inherently un-structured, requiring unsightly parameters like f.facet_name.facet.range.start=5
• Inherently un-typed… everything is a string.
• More difficult to decipher large requests.
• Harder to programmatically create a request.
• Impossible to validate. Because of the lack of structure, we don’t know the set of valid parameter and thus can’t do good error checking.

Solr already had a JSON API for faceting and analytics, and this new feature has extended that to the complete Solr request!

Simple Example

First let’s add a few excellent books from the fantasy genre (the “commitWithin=1000” will cause them to be visible to searches within 1000 milliseconds):

$ curl http://localhost:8983/solr/update?commitWithin=1000 -d '
[
 {"id":"book1", "author":"Brandon Sanderson", "title":"The Final Empire",
   "series_s":"Mistborn", "sequence_i":1, "genre_s":"fantasy"},
 {"id":"book2", "author":"Brandon Sanderson", "title":"The Well of Ascension",
   "series_s":"Mistborn", "sequence_i":2, "genre_s":"fantasy"},
 {"id":"book3", "author":"Brandon Sanderson", "title":"The Hero of Ages",
   "series_s":"Mistborn", "sequence_i":3, "genre_s":"fantasy"}
]'

Now we can search them with a JSON request rather than using query parameters:

$ curl http://localhost:8983/solr/query -d '
{
  query:"hero"
}'

RESPONSE:

{
  "responseHeader":{
    "status":0,
    "QTime":2,
    "params":{
      "json":"n{n  query:"hero"n}"}},
  "response":{"numFound":1,"start":0,"docs":[
      {
        "id":"book3",
        "author":"Brandon Sanderson",
        "author_s":"Brandon Sanderson",
        "title":["The Hero of Aages"],
        "series_s":"Mistborn",
        "sequence_i":3,
        "genre_s":"fantasy",
        "_version_":1486581355536973824
      }]
  }
}

A few things to note from our example:

• JSON body is considered a parameter named “json” and echoed back with the other params (unless you disable it with echoParams=none). This will also cause it to be logged, which is normally important.
• The JSON we send to Solr can include unquoted simple strings and can contain comments. See JSON Extensions.
• We don’t need to pass the Content-Type for indexing or for querying when we’re using JSON since Solr is now smart enough to auto-detect it when Curl is the client.
• HTTP GET requests are now allowed to have a request body (i.e. try using “curl -XGET” for the query)

Here’s a more complete example:

curl -XGET http://localhost:8983/solr/query -d '
{
  query : "*:*",
  filter : [
    "author:brandon",
    "genre_s:fantasy"
  ],
  offset : 0,
  limit : 5,
  fields : ["title","author"],  // we could also use the string form "title,author"
  sort : "sequence_i desc",

  facet : {  // the JSON Facet API is nicely integrated as well
    avg_price : "avg(price)",
    top_authors : {terms : author}
  }
}'

Passing JSON via Request Parameter

It may sometimes be more convenient to pass the JSON body as a request parameter rather than in the actual body of the HTTP request. Solr treats a json parameter the same as a JSON body.

$ curl http://localhost:8983/solr/query -d 'json={query:"hero"}&fq=author:brandon'

Smart merging of multiple JSON parameters

Multiple json parameters in a single request are merged before being interpreted.

• Single-valued elements are overwritten by the last value.
• Multi-valued elements likefields and filter are appended.
• Parameters of the form json.<path>=<json_value> are merged in the appropriate place in the hierarchy. For example a json.facet parameter is the same as “facet” within the JSON body.
• A JSON body, or straight json parameters are always parsed first, meaning that other request parameters come after, and overwrite single valued elements.

Smart merging gives the best of both worlds… the structure of JSON with the ability to selectively separate out / decompose parts of the request!

Simple Example

curl 'http://localhost:8983/solr/query?json.limit=5&json.filter="genre_s:fantasy"' -d '
{
  query : "hero",
  limit : 10,
  filter : "author:brandon"
}'

is equivalent to

curl http://localhost:8983/solr/query -d '
{
  query : "hero",
  limit : 5,   // this parameter was overwritten
  filter : [ "author:brandon" , "genre_s:fantasy" ]  // this parameter was appended to
}'

Facet Example

In fact, you don’t even need to start with a JSON body for smart merging to be very useful. Consider the following request composed entirely of request params:

curl http://localhost:8983/solr/query -d 'q=*:*&rows=1&
  json.facet.avg_price="avg(price)"&
  json.facet.top_authors={type:terms,field:author_s,limit:5}'

That is equivalent to having the following JSON body or json parameter:

{
  facet: {
    avg_price: "avg(price)",
    top_authors: {
      type: terms,
      field: author_s",
      limit: 5
    }
  }
}

Debugging

What to see what your merged JSON looks like? Just ask for debugging information (i.e. use the debug=true param), and it will come back under the "json" key along with the other debugging information.

Passing Parameters via JSON

We can also pass normal request parameters in the JSON body within the params block:

$ curl "http://localhost:8983/solr/query?fl=title,author"-d '
{
  params:{
    q:"title:hero",
    rows:1
  }
}
'

Which is equivalent to:

$ curl "http://localhost:8983/solr/query?fl=title,author&q=title:hero&rows=1"

Error Detection

Because we didn’t pollute the root body of the JSON request with the normal Solr request parameters (they are all contained in the params block), we now have the ability to validate requests and return an error for unknown JSON keys.

$ curl http://localhost:8983/solr/query -d '
{
  query : "hero",
  fulter : "author:brandon"  // oops, we misspelled "filter"
}'

And we get an error back containing the error string:

"Unknown top-level key in JSON request : fulter"

Parameter Substitution / Macro Expansion

Of course request templating via Parameter Substitution works fully with JSON request bodies or parameters as well.

Example:

$ curl "http://localhost:8983/solr/query?FIELD=text&TERM=hero&HOWMANY=10" -d '
{
  query:"${FIELD}:${TERM}",
  limit:${HOWMANY}
}'

Current Status

The JSON Request API is currently in it’s infancy - only a few query parameters are supported (although the JSON Facet API which is part of this is more mature). Other Solr features you may want access to (like highlighting) currently need to be controlled through the normal Solr request params (e.g. just ad hl=true to the normal request parameters, or in the params block of a JSON request.

Have ideas on what will make the API better? Want to help out with development? We’d love to hear from you on the solr-user mailing list!

Solr 6.2 Features

Sep 15, 2016

Yonik Seeley

Here’s an overview of some of the new features in Solr 6.2.

Download Solr 6.2 to try these features out and give us feedback! You can also check out upcoming features of the next Solr release.

Boolean Comparison Function Queries

Function queries for comparisons of numeric arguments:

• gt - greater than
• gte - greater than or equal
• lt - less than
• lte - less than or equal
• eq - equal to

These functions return a boolean value for any given document, which is normally used in conjunction with the if function which returns the second argument if the first argument is true, or the third argument otherwise.

Example: For a given document, if num_reviewers is less than 5, return the value of the editor_score field, otherwise return the user_score field.

 if( lt(num_reviewers,5), editor_score, user_score)

An easy way to experiment with function queries and see the results is to use the pseudo-field functionality to return specific function values along with stored fields.

Example: http://localhost:8983/solr/techproducts/query?q=\*:\*&fl=id,popularity,**gt(popularity,3)**

[...]
      {
        "id":"SP2514N",
        "popularity":6,
        "gt(popularity,3)":true},
      {
        "id":"6H500F0",
        "popularity":6,
        "gt(popularity,3)":true},
      {
        "id":"F8V7067-APL-KIT",
        "popularity":1,
        "gt(popularity,3)":false},
[...]

 

Feature Selection and Logistic Regression

The features and train streaming expressions are machine learning related operations used to implement a text classifier. The goal is to create a classifier (or model) based on a training set of documents that can be used to tell if a new document should be considered part of that set.

• features - streaming expression that selects important features (important terms) from a set of documents called the training set (specified by a query).
• train - trains a logistic regression model on a text field using the selected features and the training set.

Here’s an example of the streaming expression syntax taken from the JIRA:

train(collection1, q="*:*",
      features(collection1,
               q="*:*",
               field="body",
               outcome="out_i",
               positiveLabel=1,
               numTerms=100),
      field="body",
      outcome="out_i",
      maxIterations=100)

See the features streaming expression and the train streaming expression in the reference guide.

 

topic streaming expression (Beta)

The topic streaming expression implements a publish/subscribe messaging capabilities, where clients can subscribe to queries and receive deliveries of new documents that match those topic queries. When the first call is made, an id is specified to uniquely identify the topic. A checkpointCollection is also specified, and this is where topic progress is stored to keep track of which documents have already been returned.

Example syntax:

topic(checkpointCollection,
      collection1,
      id="yonik_topic1",
      q="title:(lucene solr)",
      fl="id, title, abstract, author")

A client can keep calling the same topic to receive new documents, or it can be wrapped in a daemon streaming expression to provide push functionality.

 

scoreNodes graph streaming expression

The scoreNodes streaming expression is used to add a relevancy score to a graph expression, similar to tf-idf for text. The tf factor is how many times a node appears in the graph traversal, and biases scores in favor of nodes that appear often. The idf factor biases scores in favor of nodes that are more rare in the index.

There is a good example of using this in the ref guide: Calculating Market Basket Co-occurrence, for calculating product recommendations based on your current shopping basket and what other shopping baskets look like.

 

collections API: REPLACENODE and DELETENODE commands

New actions have been added to the collections API:

• REPLACENODE - moves all replicas from one node to another.
• DELETENODE - deletes all replicas currently on a given node.

Examples: /admin/collections?action=RELACENODE&source=MY_OLD_NODE&target=MY_NEW_NODE /admin/collections?action=DELETENODE&node=MY_NODE_NAME

See the relevant sections in the Collections API for more details.

 

Kerberos security enhancements

Kerberos delegation token support was added to the authentication filter, especially useful for when distributed clients such as MapReduce do not have access to the user’s credentials.

See Kerberos Authentication Plugin in the ref guide.

 

Whole index replication for CDCR (Cross Data Center Replication)

CDCR can now fall back to whole-index replication to bring a replica up to date when transaction logs have not retained enough updates to do so.  

“role” tag in replica placement rules

SolrCloud nodes in the cluster have tags that may be used when performing rule based replica placement. A “role” tag has been added that currently be used to prevent placement of new replicas on the overseer node by adding rule=role:!overseer during collection creation.

See Rule Based Replica Placement in the ref guide for more info.

Solr 6.3 Features

Aug 23, 2016

Yonik Seeley

Here’s an overview of some of the new features in Solr 6.3. Download Solr 6.3 to try these features out and give us feedback! You can also check out upcoming features of the next Solr release.

New dvhash faceting method in the JSON Facet API

A new faceting method for the JSON Facet API has been added that utilizes hashing instead of direct indexing by ordinal. Currently, when hashing on a field with docValues enabled, or when specifying method=dv, there are accumulator arrays that are indexed by ordinal. When the cardinality of the domain is low, and the cardinality of the field is high, direct indexing of large arrays both wastes memory and takes longer to scan looking for the highest accumulated entries.

Example:

json.facet={
  top_authors : {
    type : terms,
    field : author,
    limit : 20,
    method : dvhash
  }
}

Since it may be difficult for a client to know when the facet domain will be large vs the field cardinality (esp if the domain is based off of a user query), future work is expected to implement automatically switching between faceting methods.

Optimizing, storing and deploying AI models with Streaming Expressions

The classify streaming expression classifies documents according to a model.

classify(<document_stream>,            // the stream to fetch documents that need to be classified
         <model_stream>,               // the stream to fetch the latest model
         field=<input_text_field>,     // the field on the document to classify
         fieldType=<analysis_fieldType> // (opt) fieldType to use for tokenization of the input field
)

For each input document, the classifier decorates the document with two (currently hardcoded) fields with the result:

• score_d - the raw score
• probability_d - the positive probability that the document belongs to the group

The update streaming expression can be used to store classifier models in a SolrCloud collection, and the topic streaming expression can be used with the classify expression to both deploy models as well as stream new data through the models.

The development JIRA SOLR-9258 has examples, and should be consulted until the Solr reference guide is updated.

“executor” Streaming Expression

The executor streaming expression wraps another stream containing streaming expressions. By default the expression will be contained in the expr_s field in each tuple of the wrapped stream. The executor has an internal thread pool so expressions can be executed in parallel on a single worker. This expression can further be wrapped in a parallel streaming expression to enable execution across a cluster of worker nodes.

Example syntax from the JIRA of a work queue… the “topic” expression retrieves expressions to execute (and keeps track of where it left off):

daemon(executor(threads=10, topic(storedExpressions, fl="expr_s", ...)))

“commit” Streaming Expression

The commit streaming expression wraps another stream (normally involving updates) and performs a commit at the end.

Example:

commit(targetCollection, update(targetCollection, search(sourceCollection, q="myquery", ...)))

“fetch” Streaming Expression

The fetch streaming expression wraps another expression and fetches additional fields from documents in batches. It’s very much like an inner join, with one side of the join being very small.

Example that queries books and adds publisher address to each result:

fetch(publisherCollection,
      search(booksCollection, q="*:*", fl="title,author,publisher", sort="publisher asc"),
      fl="publisherAddress",
      on="publisher=publisherId")

Lots More…

See the release notes on the Solr wiki and the CHANGES file for an in-depth list of changes.

Older posts