Traditional faceted search (also called guided navigation) involves counting search results that belong to categories (also called facet constraints). The new facet functions in Solr extends normal faceting by allowing additional aggregations on document fields themselves. Combined with the new Sub-facet feature, this provides powerful new realtime analytics capabilities. Also see the page about the new JSON Facet API.
Faceting involves breaking up the domain into multiple buckets and providing information about each bucket. There are multiple aggregation functions / statistics that can be used:
Aggregation
Example
Effect
sum
sum(sales)
summation of numeric values
avg
avg(popularity)
average of numeric values
sumsq
sumsq(rent)
sum of squares
min
min(salary)
minimum value
max
max(mul(price,popularity))
maximum value
unique
unique(state)
number of unique values (count distinct)
hll
hll(state)
number of unique values using the HyperLogLog algorithm
percentile
percentile(salary,50,75,99,99.9)
calculates percentiles
stddev
stddev(salary)
calculates standard deviation (Solr6.6+)
variance
variance(salary)
calculates variance (Solr 6.6+)
Numeric aggregation functions such as avg can be on any numeric field, or on another function of multiple numeric fields.
See Count Distinct in Solr for more information on distributed cardinality estimation / calcDistinct.
The faceting domain starts with the set of documents that match the main query and filters. We can ask for statistics over this whole set of documents:
http://localhost:8983/solr/query?q=*:*&
json.facet={x:'avg(price)'}
And the response will contain a facets section:
[...]
"facets":{
"count":32,
"x":164.10218846797943
}
[...]
If we want to break up the domain into buckets and then calculate a function per bucket, we simply add a nested facet command to the facet parameters. For example (using curl this time):
$curl http://localhost:8983/solr/query -d 'q=*:*&
json.facet={
categories:{
type : terms, // terms facet creates a bucket for each indexed term (or value) in the field
field : cat,
facet:{
x : "avg(price)",
y : "sum(price)"
}
}
}
'
The response will contain the two stats we asked for in each category bucket.
The default sort for a field or terms facet is by bucket count descending. We can optionally sort ascending or descending by any facet function that appears in each bucket. For example, if we wanted to find the top buckets by average price, then we would add sort:"x desc" to the previous facet request:
Subfacets (also called Nested Facets) is a more generalized form of Solr’s current pivot faceting that allows adding additional facets for every bucket produced by a parent facet.
Subfacet advantages over pivot faceting:
Subfacets work with facet functions (statistics), enabling powerful real-time analytics
Can add a subfacet to any facet type (field, query, range)
A subfacet can be of any type (field/terms, query, range)
A given facet can have multiple subfacets
Just like top-level facets, each subfacet can have it’s own configuration (i.e. offset, limit, sort, stats)
Subfacets are part of the new Facet Module, and are naturally expressed in the JSON Facet API. Every facet command is actually a sub-facet since there is an implicit top-level facet bucket (the domain) defined by the documents matching the main query and filters. Simply add a facet section to the parameters of any existing facet command.
For example, a terms facet on the “genre” field looks like:
top_genres:{
type: terms,
field: genre,
limit: 5
}
Now if we wanted to add a subfacet to find the top 4 authors for each genre bucket:
Assume we want to do the following complex faceting request:
Facet on the “genre” field and find the top buckets
For ever “genre” bucket generated above, find the top 7 authors
For ever “genre” bucket, create a bucket of high popularity items (defined by popularity 8 - 10) and call it “highpop”
For ever “highpop” bucket generated above, find the top 5 publishers
In short, this request finds the top authors for each genre and finds the the top publishers for high popularity books in each genre. Using the JSON Facet API, the full request (using curl) would look like the following:
type: terms, // nested terms facet under the nested query facet
field: publisher,
limit: 5
}
}
}
}
}
}
'
An example response would look like the following:
[...]
"facets":{
"top_genres":{
"buckets":[{
"val":"Fantasy",
"count":5432,
"top_authors":{ // these are the top authors in the "Fantasy" genre
"buckets":[{
"val":"Mercedes Lackey",
"count":121},
{
"val":"Piers Anthony",
"count":98}]}},
"highpop":{ // bucket for books in the "Fantasy" genre with popularity between 8 and 10
"count":876
"publishers":{ // top publishers in this bucket (highpop fantasy)
"buckets":[{
"val":"Bantam Books",
"count":346},
{
"val":"Tor",
"count":217}]}},
{
"val":"Science Fiction", // the next genre bucket
"count":4188,
[...]
All the reporting and sorting was done using document count (i.e. number of books). If instead, we wanted to find top authors by total revenue (assuming we had a “sales” field), then we could simply change the author facet from the previous example as follows:
Facet functions and Subfacets are in Solr 5.1 and later, but the syntax used on this page requires Solr 5.3 or later. Download the latest release and give it a spin!
The new facet module has a native JSON Facet API, first-class support for statistics and analytics via facet functions (aggregations), and supports unlimited nesting of facets within other facets via sub-facets.
One can calculate statistics such as averages, number of unique values (distinct values), and percentiles over each facet bucket (groups of documents), and even sort facet buckets by any calculated metrics.
Parameter substitution is now done across the entire query request. It supports default values, multiple levels of indirection, and it even works within the body of a JSON request. This can also be viewed as a powerful form of request templates.
Example:
q=price:[ ${low} TO ${high} ]
&low=100
&high=200
Parameters can also be passed in the params block of a JSON request.
Syntax within the standard lucene/solr query parser for constant score queries quit the general form of ^=<constant_score>. Think of a query boost with ^ replaced with ^=. Example:
There is a new general purpose parallel computing framework for SolrCloud. The Streaming API is (currently) a Java API that can do streaming aggregations (like sum and average) and streaming transformations (like group-by and join).
The admin UI can show segment info such as size, number of docs, and number of deletions for each segment in the index. For the “demo” collection, simply point your browser at http://localhost:8983/solr/#/demo/segments Or click on the “Segments Info” link in the admin UI after you select the core/collection you are interested in.
Many additional configuration items can now be managed via the Config API. This includes managing named components such as requestHandler, queryParser, queryResponseWriter, valueSourceParser, transformer, and queryConverter.
Changes do not directly change solrconfig.xml, but instead are reflected in configoverlay.json which override settings in solrconfig.xml.
Upload config sets to zookeeper with CloudSolrClient
Named config sets (schema.xml, solrconfig,xml, etc) are referenced by name when creating new collections in SolrCloud. These config sets may now be uploaded and downloaded via SolrJ to and from the local filesystem. The following methods were added to CloudSolrClient:
There is a new API to add a jar to a collection’s classpath (as well as update and delete a jar). Components that depend on such a jar should have a new attribute called runtimeLib set to true since a separate classloader is used for these jars.
Caches using the LRUCache implementation can specify a new parameter maxRamMB that will evict based on RAM use rather than number of elements in the cache. Least recently used items are evicted until the RAM use is brought under the limit. RAM use calculations do not currently cover the cache keys, so using this for the query cache and caching large queries can still lead to greater memory use than expected.
Multi-select faceting is a powerful faceting style that allows users to see and select multiple facet constraints (facet values) for a facet. For example, one may want to select multiple price ranges or multiple colors they are interested in.
The new Facet Analytics Module / JSON Facet API now supports multi-select faceting via filter exclusions. A new excludeTags parameter will disregard any top-level filters with matching tags.
Both the older Stats component and the new Facet Analytics Module have added support for HyperLogLog based statistical cardinality estimate. For the JSON Facet API, a new hllfacet function was added as an alternative to the existing faster (but less accurate for high cardinality) unique function. Example:
json.facet={ numProducts : "hll(product_id)" }
See Solr Count Distinct functionality for examples that calculate the number of distinct values in a given field per facet bucket.
Add a new “facet.range.method” parameter to let users choose how to do range faceting between an implementation based on filters (previous algorithm, using “facet.range.method=filter”) or DocValues (“facet.range.method=dv”). Input parameters and output of both methods are the same.
If you have a field value that consists of well formed XML or JSON, you can return those raw values in the appropriate response writer. Example: ?fl=id,name,json_s:[json],xml_s:[xml]
This new SolrCloud feature allows the specification of rules which govern placement of replicas in the cluster. Rules are specified during collection creation and persisted in zookeeper.
The percentile aggregation function was just added to the new Solr Facet Module. This allows one to calculate one or more percentiles for each facet bucket (i.e. each group of documents produced by faceting), and even sort facet buckets by any given percentile.
The percentile aggregation even works with distributed search! The algorithm used is Ted Dunnings “t-digest”, which gives good approximations with relatively little memory consumption.
We can also sort by a percentile statistic. If you request more than one percentile value, the sort will be on the first value in the list requested. Let’s find the top states by 99.9th percentile salary:
JSON strings are normally encapsulated by double quotes. It’s often desirable to use single quotes if for example you are embedding some JSON in another double quoted string in a program.
Allowing trailing commas or extra commas can make it easier to produce JSON that doesn’t throw a parse exception. One use-case is templating JSON. Given the following template,
Large string values can optionally be handled in a streaming fashion a piece at a time. Noggit will only construct a single String object in memory if asked. This allows for stream processing with very little memory overhead.
{
"big_string" : "A very large string... pretend its's 1GB in size... we can process it and send it on without reading it all into memory at once!"
Noggit can also handle multiple JSON values streamed over a single connection and simply catenated together. Primitive values should of course be separated by whitespace to avoid ambiguity.
{first_object:10}
['another array object']['yet another object']
{more:objects}{another:object}
['who knows how many json values will be streamed by the writer...']
Noggit can parse huge JSON messages with minimal overhead.
A single byte of state needed per nested object or array. This is needed to keep track of the type of enclosing entity.
A user can optionally provide an input buffer for Noggit to use when parsing from a Reader, allowing re-use across different parsers and thus lower memory consumption and garbage collection activity.
Streaming values: very large values (such as strings) can be obtained in chunks, thus the whole value never needs to reside in memory at once.
Lucene/Solr trunk (the future 6.0 release) is now on Java8, while version 5.x is still on Java7. Linux and Windows allows one to install a JDK any place in the filesystem, and I use the convention of installing in /opt/jdk7 and /opt/jdk8. Things are a little more difficult on Mac OS-X however, as you can’t chose the install location. Luckily there is a command called java_home to show you where a JDK is installed.
Here’s a snippet from my .profile to help manage working with different java versions:
Terminal window
OS=`uname`
case"$OS"in
CYGWIN*)
OS=cygwin
OPT=c:/opt
;;
*)
OPT=/opt
;;
esac
set-java () {
exportJAVA_HOME="$*"
if [ $OS="cygwin" ]; then
exportPATH="`cygpath$JAVA_HOME/bin`:$PATH"
else
exportPATH="$JAVA_HOME/bin:$PATH"
fi
}
if [ $OS="Darwin" ]; then
JAVA7=`/usr/libexec/java_home-v1.7`
JAVA8=`/usr/libexec/java_home-v1.8`
else
JAVA7=$OPT/jdk7
JAVA8=$OPT/jdk8
fi
set-java$JAVA8
Now, if I switch from working on trunk to working on Lucene 5 or Solr 5, I can easily switch the default JDK for a single terminal via the set-java shell function.
Terminal window
/opt/heliosearch$java-version
javaversion"1.8.0_25"
Java(TM) SE Runtime Environment (build1.8.0_25-b17)
Solr 4.10 and Heliosearch .07 have added a terms query (or terms filter) to more efficiently match many terms in a single field. A large number of terms are often useful for things like access control lists or security filters. Previously, the only way to do this was a large boolean query with many clauses, which has unnecessary overhead when scoring is not needed.
Solr’s implementation uses Lucene’s TermFilter class, as does Elasticsearch’s terms filter.
The Heliosearch terms query implementation has some additional features:
prefix compression including off-heap construction
direct creation of off-heap filter for faster execution and less garbage production
For reference, specifying a filter query (fq) in the normal lucene syntax via a boolean query looks like the following (assumes default boolean operator of OR):
Performance of terms queries is shown relative to using a Boolean query in Solr. For example the last column in the first chart represents a 10 term filter that matches 10,000,000 documents (1 million per term). The request execution time is:
381,342 microseconds with a Solr Boolean Querty
122,119 microseconds with a Solr Terms Query
67,075 microseconds with a Heliosearch Terms Query
Benchmark details:
10M document index
64 bit Java 1.8.0_20 Oracle JDK
Windows 8 64 bit, quad-core Intel i5-3570K @ 3.4GHz
Request time was measured externally and includes the entire request time, including the time for the client to send the request and read the response.
The first performance tests were run multiple times and the amount of garbage produced was recorded.
The Heliosearch off-heap optimizations clearly pay dividends here, resulting in much less heap usage, less garbage production (which will mean less garbage collection work), and a smaller process size.
Native code faceting for Solr has just been added to Heliosearch, and benchmarks show an impressive 2x performance increase! This is faceting code written in C++ and statically compiled for maximum performance, and loaded into the JVM via JNI (Java Native Interface).
The different operating systems were run on different hardware (hence the large performance differences of the same code across the different platforms).
The gcc/g++ included with OS-X is actually clang/LLVM - clang is the C language front end and LLVM is the back end that produces executable code. At least for this initial native code, g++ 4.8.2 was about 5% faster than clang/LLVM 5.1, hence we’ll most likely use gcc/g++ by default. The easiest way to get gcc/g++ on your Mac is
$ brew install gcc48
After installation, gcc/g++ will continue pointing to the clang/LLVM versions, but there will be gcc-4.8 and g++-4.8 you can use in /usr/local/bin.
Besides the incredible over 2x faceting performance improvement, native code has other advantages as well:
Avoidance of Java hotspot bugs in compiling code. Compiling the code just once statically means it’s the same for every run, for everyone.
No variations from run-to-run due to how hotspot compiles the code (unexplained slowdowns).
No hotspot warm-up period, or time spent optimizing, or de-optimizing code.
It’s easy to take advantage of these performance improvements and new features since Heliosearch/Solr is currently a drop-in replacement (at the HTTP-API level) for Apache Solr. Download the latest release and try it out.
We’d love to hear how it’s working for you… drop by the user mailing list and let us know. Want to dabble in C/C++ code again? Drop by the dev mailing list to help out with development!
Solr needs a flexible cross-datacenter architecture that can handle both a variety of application needs as well as a variety of infrastructure resources.
Clusters will be configured to know about each other, most likely through keeping a cluster peer list in zookeeper. One essential piece of information will be the zookeeper quorum address for each cluster peer. Any node in one cluster can know the configuration of another cluster via a zookeeper client.
Update flow will go from the shard leader in one cluster to the shard leader in the peer clusters. This can be bi-directional, with updates flowing in both directions. Updates can be either synchronous or asynchronous, with per-update granularity.
Solr transaction logs are currently removed when no longer needed. They will be kept around (potentially much longer) to act as the source of data to be sent to peer clusters. Recovery can also be bi-directional with each peer cluster sending the other cluster missed updates.
The shard leader versions a document and then forwards it to replicas. Update re-orders are handled by the receiver by dropping updates that are detected to be older than the latest document version in the index. This works given that complete documents are always sent to replicas, even if it started as a partial update on the leader.
Solr version numbers are derived from a timestamp (the high bits are milliseconds and the low bits are incremented for each tie in the same millisecond to guarantee a monotonically increasing unique version number for any given leader).
If updates are accepted for the same document in two different clouds (implying two different leaders versioning the document), then having the correct last document “win” relies on clock synchronization between the two leaders. Updates to the same document at different data centers within the clock skew time risk being incorrectly ordered.
Solr only has versions at the document level. The current partial update implementation (because of other constraints) reads the current stored fields of the document, makes the requested update, and indexes the new resulting document. This creates a problem with accepting Solr atomic updates / partial updates to the same document in both data-centers.
Example:
DC1: writes document A, version=time1
DC2: receives document A (version=time1) update from DC1
DC1: updates A.street_address (Solr reads version time1, writes version time2)
DC2: updates A.phone_number (Solr reads version time1, writes version time3)
DC1: receives document A (version=time3) from DC2, writes it.
DC2: received document A (version=time2) from DC1, ignores it (older version)
Although both data-centers became “consistent”, the partial update of street_address was completely lost in the process.
To deal with potential update conflicts arising from updating the same document in different data centers, each document can have a primary cluster.
A routing enhancement can ensure that a document sent to the wrong cluster will be forwarded to the correct cluster.
Routing can take as input a request parameter, a document field, or the unique id field. The primary cluster could be determined by hash code (essentially random), or could be determined by a mapping specified in the cluster peer list. Changes to this mapping for fail-over would not happen automatically in Solr. If a data center becomes unreachable, the application/client layers have responsibility for deciding that a different cluster should become the primary for that set of documents.
Primary cluster routing will be optional. Many applications will naturally not trigger the type of undesirable update behavior described, or will have the ability to work around update limitations.
Implement true partial updates with vector clocks and/or finer grained versioning so that updates to different fields can be done conflict free if re-ordered. This would also lower the bandwidth costs of partial updates since the entire document would no longer be sent to all replicas and to other peer clusters.
One could potentially further minimize cross-DC traffic by introducing traffic aggregator nodes (one per cluster) that all udpates would flow through. This would likely only improve bandwidth utilization in low update environments. The improvements would come from fewer connections (and hence less connection overhead) and better compression (a block of many small updates would generally have a better compression ratio than the same updates compressed individually).
Many zookeeper clients in a peer cluster could generate significant amounts of traffic between data centers. There could be a designated listener to the remote cluster state that could disseminate this state to others in the local cluster rather than hitting ZK directly.
Also worth investigating is the use of a local zookeeper observer node that could service all local ZK reads for the remote ZK quorum.
The first phase of this design only deals with updates. Collection level operations such as adding a new shard, splitting a shard, and changing replication levels, must be performed by the client on every cluster as applicable.
The collections API (and other higher level APIs) could be made peer-aware such that these operations would also be forwarded to peer clusters, as well as including a queuing mechanism for the cases when a peer cluster is unreachable.
Forwarding of updates from one cloud to another should be done via standard SolrJ client. Any needed enhancements/modification should be done to a SolrJ client such that those enhancements may also be used in other contexts.
