Amgix Now is the newest member of the Amgix family, and it's available today. This post will explain what Amgix Now is and why we built it, we will then put it through some tests and see how it does on various datasets.

Jump to: Benchmarks, Results, or Takeaways

What is Amgix Now

Amgix Now is a high-performance hybrid search engine. Unlike Amgix, which is designed to be a distributed modular hybrid search system, Amgix Now is built as a single Rust binary that delivers Amgix search features with a focus on search relevance, high throughput and low latency.

docker run -d -v <path-to-data>:/data -p 8235:8235 amgixio/amgix-now:0

For GPU-accelerated hybrid search use amgixio/amgix-now:0-gpu image.

The same API as Amgix: Getting Started

Why We Built it

It started as an experiment: what would happen if we took Amgix, stripped out all the cluster communication and coordination layers and implemented it as a stand-alone hybrid search engine, in Rust?

WMTR and other built-in tokenizers were already implemented in Rust. At first, we ported basic API endpoints, collection and document processing logic, supporting database functionality and minimal internal coordination logic.

We ran some tests and were impressed by the initial performance indicators. We also realized that for many use cases, a simple reliable fast hybrid search engine is all that is needed, and if it can handle a decent amount of load, you may never need to upgrade to anything more advanced.

At this point, it became not "if" but "how soon" we can build it.

We ported the rest of the API endpoints, internal logic, hybrid search and embedding pieces.

After the first milestone of passing all the Amgix API integration tests was achieved, we focused on running tests, tweaking, tuning and optimizing internal architecture and performance.

The result is the Amgix Now we have today.

So in a way, it was never about "why", it kind of demanded itself into existence. We couldn't help ourselves.

Where Amgix Now Fits in Amgix Family

The table below explains some of the differences and trade-offs of using Amgix Now, Amgix One, and full Amgix cluster.

• * Amgix Now can be configured as a primary/fallback behind a proxy and a shared external Qdrant instance
• ** Multiple Amgix One instances can be deployed with external RabbitMQ and database instances, but it's awkward and better served by a full-scale Amgix cluster deployment

Some key points:

• the API is identical between Amgix Now and Amgix (with the exception of metrics and a couple of async queue endpoints that are not implemented)
• Qdrant is the default and only storage supported in Amgix Now
• Like Amgix, Amgix Now is not tightly coupled with a backend database. You can point Amgix Now at an external Qdrant instance.
• All document upsert and delete endpoints are synchronous (there is no async ingestion pipeline or queue)
• Documents are automatically deduplicated on upserts and internal locking preserves write atomicity (just like Amgix)

This makes for an interesting upgrade story. Because the API is identical and storage is identical (if you use Qdrant) you can switch from Amgix Now to Amgix One or full Amgix by simply mounting the same storage directory and starting a different version of Amgix. No application changes are required. You can go from Amgix Now to Amgix One and back again by simply stopping one container and starting another.

Benchmarks

Introduction

You may have noticed, if you read our blog posts and documentation pages, that we love benchmarks. Well, maybe not "love" (because they are a lot of work), but we do them a lot. Part of it is that the Amgix project is young and benchmarks are our way to tell the world that we are serious and we may be worth your consideration. The other part is that we like precision. When something is fast, we want to know how fast and when something is precise we want to know how precise. Benchmarks give us a tangible way to measure what we are building.

What are We Testing

The primary goal of these tests is for us to find out and share with you how Amgix Now performs on various datasets.

Modern search engines offer a staggering array of features and options, but no matter the level of sophistication of these solutions, there are two core functions that any user expects from a search system: relevance and speed.

Can you find what I'm looking for and can you find it quickly?

To that end, the tests below focus on these metrics by measuring relevance (nDCG@k), recall (Recall@k) and search query latencies (p50, p95). We also report some operational observations along the way.

Frame of Reference

The results below include the Amgix Now numbers side-by-side with the numbers for the three widely used and popular search engines: Typesense, Meilisearch, and Elasticsearch. These are mature projects/businesses with decades of software engineering expertise, countless features, established communities and dedicated users and customers. Listing Amgix Now side-by-side with these giants seems... well... immodest.

We didn't take the decision to include their numbers in this report lightly. For one, it opens us up to all kinds of criticism, fair or unfair. But it is also a ton of work to research and test everything on four different systems.

Nevertheless, we decided to go ahead and publish all the numbers. And here is why:

1. Publishing only Amgix Now numbers in isolation lacks context. Is 22ms on p50 on this dataset good or meh? Including the numbers from the other systems gives us a frame of reference. This is how some of the other search engines perform on the same task.

2. Even if you have no interest in Amgix, it may be useful for the engineering community to see the detailed metrics for the established search engines on the same hardware, on the same tests. Not to see who is better or worse. All systems have their strengths and weaknesses. Search relevance and speed are hardly the only metrics that one takes into account when selecting a search solution. But to be able to make more informed decisions when you need to. In fact, we would have loved it if the detailed benchmarks like this existed somewhere on the internet, so we can just cite them and not do all the work.

A Word About WMTR

Amgix Now includes all the same tokenizers that Amgix comes with: WMTR, Full Text, Trigrams and Whitespace. For these tests we are using WMTR as it is the default keyword tokenizer.

WMTR (Weighted Multilevel Token Representation) is Amgix's custom lexical tokenizer. It represents text through multiple lexical views at once: a surface-form view that stays closer to the original tokens, a language-aware normalized view built with Unicode word boundaries, stopword filtering, and stemming, and a character-level view that captures short local patterns inside the text. Those signals are then weighted together into a single sparse vector representation.

Because it requires no ML model, it's fast and lightweight, making it a great default for many use cases where you simply want high-quality lexical search. It is a good fit both for ordinary natural-language text and for noisier datasets (ERP, E-commerce, science, etc.) where short identifiers, mixed-format strings, and token structure still carry meaning.

BEIR Benchmark Context

The BEIR (Benchmarking IR) benchmark suite provides standardized datasets for evaluating information retrieval systems. In short, it's a framework that allows us to measure relevance of the results returned by a search system.

For context, we will include BM25 Multifield baseline scores from Pyserini's BEIR regression results with each BEIR dataset tested.

Test Setup

Expand collapsed sections for details:

Results

PC Parts Dataset

The PC Parts dataset is not a standard BEIR dataset. It's a list of 6,636 video card names (Asus TUF GAMING, Asus EAH6450 SILENT/DI/512MD3(LP), Gigabyte GV-N660OC-2GD, Gigabyte WINDFORCE). There are only 16 queries: short partial strings, typos, non alpha-numeric characters.

The data lacks linguistic richness that most tokenizers (model-based or not) are optimized for. However, this is representative of the data that exists in many domains (ERP, E-commerce, etc.), where you may have thousands of records with part numbers like Pin 12LP'-x03/5-XL and short or non-existent descriptions. Users searching these datasets often know the exact model/part number they want to find, so they search for unique bits of identifiers, special characters and all, to quickly find what they came for. Full-text tokenizers (for example) struggle with this, they typically strip out all special characters, drop single digits, etc. FT tokenizer may represent the above part number as pin 12lp x03 xl, which drops some or all differentiators, meaningful for this context, reducing recall and precision of searches.

Because the dataset is unique, not a Natural Language friendly data, we tested two different configs of search engines: Defaults (out-of-the-box experience) and Optimized (settings tuned for better relevance).

Defaults

{
    "vectors": [{"name": "keyword", "type": "wmtr", "index_fields": ["name", "content"]}]
}

{
    "query": "<QUERY_TEXT>",
    "limit": 100
}

{
    "name": "<COLLECTION_NAME>",
    "fields": [
        {
            "name": "name", 
            "type": "string"
        },
        {
            "name": "text",
            "type": "string"
        },
    ]
}

{
    "q": "<QUERY_TEXT>",
    "query_by": "name,text",
    "per_page": 100
}

{"uid": "<INDEX_NAME>", "primaryKey": "id"}

{
    "id": "<DOC_ID>",
    "name": "<TITLE>",
    "text": "<TEXT>"
}

{
    "q": "<QUERY_TEXT>",
    "limit": 100
}

{
    "settings": {
        "analysis": {
            "analyzer": {
                "id_std": {
                    "type": "custom",
                    "tokenizer": "standard",
                    "filter": ["lowercase", "stop", "english_stemmer"]
                }
            },
            "filter": {
                "english_stemmer": {"type": "stemmer", "language": "english"},
            }
        }
    },
    "mappings": {
        "properties": {
            "name": {
                "type": "text",
                "analyzer": "id_std"
            },
            "content": {
                "type": "text",
                "analyzer": "id_std"
            }
        }
    }
}

{
    "size": 100,
    "track_total_hits": false,
    "query": {
        "multi_match": {
            "query": "<QUERY_TEXT>",
            "fields": ["name", "content"],
            "type": "most_fields"
        }
    }
}

Optimized

{
    "vectors": [{"name": "keyword", "type": "wmtr", "index_fields": ["name", "content"]}]
}

{
    "query": "<QUERY_TEXT>",
    "limit": 100
}

{
    "name": "<COLLECTION_NAME>",
    "fields": [
        {
            "name": "name", 
            "type": "string",
            "infix": true
        },
        {
            "name": "text",
            "type": "string",
            "infix": true
        },
    ]
}

{
    "q": "<QUERY_TEXT>",
    "query_by": "name,text",
    "per_page": 100,
    "infix": "always",
    "num_typos": 3
}

{"uid": "<INDEX_NAME>", "primaryKey": "id"}

{
    "id": "<DOC_ID>",
    "name": "<TITLE>",
    "text": "<TEXT>"
}

{
    "q": "<QUERY_TEXT>",
    "limit": 100
}

curl -X PATCH 'http://localhost:7700/indexes/<INDEX_NAME>/settings/typo-tolerance' \
    -H 'Content-Type: application/json' \
    -d '{
        "minWordSizeForTypos": {
            "oneTypo": 3,
            "twoTypos": 3
        }
    }'

{
    "settings": {
        "analysis": {
            "analyzer": {
                "id_std": {
                    "type": "custom",
                    "tokenizer": "standard",
                    "filter": ["lowercase", "stop", "english_stemmer"]
                },
                "id_ngram3": {
                    "type": "custom",
                    "tokenizer": "keyword",
                    "filter": ["lowercase", "ngram_3"]
                }
            },
            "filter": {
                "english_stemmer": {"type": "stemmer", "language": "english"},
                "ngram_3": {"type": "ngram", "min_gram": 3, "max_gram": 3}
            },
        }
    },
    "mappings": {
        "properties": {
            "name": {
                "type": "text",
                "analyzer": "id_std",
                "fields": {
                    "ngram": {"type": "text", "analyzer": "id_ngram3"},
                }
            },
            "content": {
                "type": "text",
                "analyzer": "id_std",
                "fields": {
                    "ngram": {"type": "text", "analyzer": "id_ngram3"},
                }
            }
        }
    }
}

{
    "size": 100,
    "track_total_hits": false,
    "query": {
        "multi_match": {
            "query": "<QUERY_TEXT>",
            "fields": ["name^1", "name.ngram^1", "content^1", "content.ngram^1"],
            "type": "most_fields"
        }
    }
}

BEIR SciFact Dataset

The SciFact dataset contains 5,183 documents and 300 queries focused on scientific claims and evidence verification. This makes it a great benchmark for evaluating search performance on domain-specific scientific content.

{
    "vectors": [{"name": "keyword", "type": "wmtr", "index_fields": ["name", "content"]}]
}

{
    "query": "<QUERY_TEXT>",
    "limit": 100
}

{
    "query": "<QUERY_TEXT>",
    "limit": 100,
    "fusion_mode": "linear",
    "vector_weights": [
        {"vector_name": "keyword", "field": "name", "weight": 1.0},
        {"vector_name": "keyword", "field": "content", "weight": 2.0},
    ]
}

{
    "name": "<COLLECTION_NAME>",
    "fields": [
        {
            "name": "name", 
            "type": "string"
        },
        {
            "name": "text",
            "type": "string"
        },
    ]
}

{
    "q": "<QUERY_TEXT>",
    "query_by": "name,text",
    "per_page": 100,
    "num_typos": 0,
    "typo_tokens_threshold": 0
}

{
    "q": "<QUERY_TEXT>",
    "query_by": "name,text",
    "per_page": 100,
    "num_typos": 0,
    "typo_tokens_threshold": 0,
    "query_by_weights": "1,2"
}

{"uid": "<INDEX_NAME>", "primaryKey": "id"}

{
    "id": "<DOC_ID>",
    "name": "<TITLE>",
    "text": "<TEXT>"
}

{
    "q": "<QUERY_TEXT>",
    "limit": 100
}

{
    "settings": {
        "analysis": {
            "analyzer": {
                "id_std": {
                    "type": "custom",
                    "tokenizer": "standard",
                    "filter": ["lowercase", "stop", "english_stemmer"]
                }
            },
            "filter": {
                "english_stemmer": {"type": "stemmer", "language": "english"},
            }
        }
    },
    "mappings": {
        "properties": {
            "name": {
                "type": "text",
                "analyzer": "id_std"
            },
            "content": {
                "type": "text",
                "analyzer": "id_std"
            }
        }
    }
}

{
    "size": 100,
    "track_total_hits": false,
    "query": {
        "multi_match": {
            "query": "<QUERY_TEXT>",
            "fields": ["name", "content"],
            "type": "most_fields"
        }
    }
}

{
    "size": 100,
    "track_total_hits": false,
    "query": {
        "multi_match": {
            "query": "<QUERY_TEXT>",
            "fields": ["name^1", "content^2"],
            "type": "most_fields"
        }
    }
}

BEIR Quora Dataset

The Quora dataset contains 523K documents and 10,000 queries, and pairs duplicate questions from Quora. Documents are question text only.

{
    "vectors": [{"name": "keyword", "type": "wmtr", "index_fields": ["name", "content"]}]
}

{
    "query": "<QUERY_TEXT>",
    "limit": 100
}

{
    "name": "<COLLECTION_NAME>",
    "fields": [
        {
            "name": "name", 
            "type": "string"
        },
        {
            "name": "text",
            "type": "string"
        },
    ]
}

{
    "q": "<QUERY_TEXT>",
    "query_by": "name,text",
    "per_page": 100,
    "num_typos": 0,
    "typo_tokens_threshold": 0
}

{"uid": "<INDEX_NAME>", "primaryKey": "id"}

{
    "id": "<DOC_ID>",
    "name": "<TITLE>",
    "text": "<TEXT>"
}

{
    "q": "<QUERY_TEXT>",
    "limit": 100
}

{
    "settings": {
        "analysis": {
            "analyzer": {
                "id_std": {
                    "type": "custom",
                    "tokenizer": "standard",
                    "filter": ["lowercase", "stop", "english_stemmer"]
                }
            },
            "filter": {
                "english_stemmer": {"type": "stemmer", "language": "english"},
            }
        }
    },
    "mappings": {
        "properties": {
            "name": {
                "type": "text",
                "analyzer": "id_std"
            },
            "content": {
                "type": "text",
                "analyzer": "id_std"
            }
        }
    }
}

{
    "size": 100,
    "track_total_hits": false,
    "query": {
        "multi_match": {
            "query": "<QUERY_TEXT>",
            "fields": ["name", "content"],
            "type": "most_fields"
        }
    }
}

BEIR NQ Dataset

The Natural Questions (NQ) dataset contains 2.6M documents and 3,452 queries, and uses Wikipedia passages and factoid-style queries. This dataset is difficult for keyword searches, relevance scores are usually pretty low, but it gives us a chance to see how search engines perform on a large dataset.

{
    "vectors": [{"name": "keyword", "type": "wmtr", "index_fields": ["name", "content"]}]
}

{
    "query": "<QUERY_TEXT>",
    "limit": 100
}

{
    "query": "<QUERY_TEXT>",
    "limit": 100,
    "fusion_mode": "linear",
    "vector_weights": [
        {"vector_name": "keyword", "field": "name", "weight": 1.0},
        {"vector_name": "keyword", "field": "content", "weight": 2.0},
    ]
}

{
    "name": "<COLLECTION_NAME>",
    "fields": [
        {
            "name": "name", 
            "type": "string"
        },
        {
            "name": "text",
            "type": "string"
        },
    ]
}

{
    "q": "<QUERY_TEXT>",
    "query_by": "name,text",
    "per_page": 100,
    "num_typos": 0,
    "typo_tokens_threshold": 0
}

{
    "q": "<QUERY_TEXT>",
    "query_by": "name,text",
    "per_page": 100,
    "num_typos": 0,
    "typo_tokens_threshold": 0,
    "query_by_weights": "1,2"
}

{"uid": "<INDEX_NAME>", "primaryKey": "id"}

{
    "id": "<DOC_ID>",
    "name": "<TITLE>",
    "text": "<TEXT>"
}

{
    "q": "<QUERY_TEXT>",
    "limit": 100
}

{
    "settings": {
        "analysis": {
            "analyzer": {
                "id_std": {
                    "type": "custom",
                    "tokenizer": "standard",
                    "filter": ["lowercase", "stop", "english_stemmer"]
                }
            },
            "filter": {
                "english_stemmer": {"type": "stemmer", "language": "english"},
            }
        }
    },
    "mappings": {
        "properties": {
            "name": {
                "type": "text",
                "analyzer": "id_std"
            },
            "content": {
                "type": "text",
                "analyzer": "id_std"
            }
        }
    }
}

{
    "size": 100,
    "track_total_hits": false,
    "query": {
        "multi_match": {
            "query": "<QUERY_TEXT>",
            "fields": ["name", "content"],
            "type": "most_fields"
        }
    }
}

{
    "size": 100,
    "track_total_hits": false,
    "query": {
        "multi_match": {
            "query": "<QUERY_TEXT>",
            "fields": ["name^1", "content^2"],
            "type": "most_fields"
        }
    }
}

Operational Observations

Because NQ is such a large dataset, we thought it would be a good opportunity to observe and record some additional metrics, like upload times, memory utilization at various points, and disk space used. Expand this section to see the details of how we recorded the numbers:

Relevance and Latency

Takeaways

Relevance

The relevance numbers for Amgix Now are not new to us. It uses the same WMTR tokenizer as Amgix that we have extensively tested before in our early benchmarks, WMTR benchmarks, and the discussion about Linear fusion.

Amgix Now performed very well on PC Parts dataset and matched or closely approached the published BEIR BM25 baseline on the BEIR datasets.

Latencies

p50 Story

• On the small PC Parts with very few queries, Amgix Now stayed within the pack and probably within the normal measurement variance numbers. Seemingly, a little better on some runs (Optimized) and in the middle on others (Defaults). If you average the numbers between Defaults and Optimized results, Amgix Now was a tiny bit ahead of the pack with the average of 3.1ms.

• On the small SciFact and the medium sized Quora datasets with plenty of queries, Amgix Now slightly edged the pack. Again, given the measurement variance, we can say it was among the leaders.

• On the large NQ dataset, Amgix Now showed a middle of the pack performance.

p95 Story

Here, Amgix Now leads on all datasets, except for NQ, showing impressive stability of search latencies. In fact, if you look at the latency variance (p95 - p50) across the datasets you can see this clearly:

Overall

Amgix Now performed very competitively across the datasets on both relevance and latency. It's new and nowhere near as battle-tested as the other systems included in this report, but we believe the numbers show that it's well worth consideration when you are choosing a search engine for your next project.