Elasticsearch explained: how it stores JSON documents, analyses text into an inverted index, ranks results by relevance and still finds matches despite typos.
Elastic Search, explained by cats. Kitty has a lot of data. Logs. Metrics. Messages. Endless tuna orders. She tries to search using a database. Rows. Tables. Indexes fighting for their lives. Databases store data. they’re not built to search it.
So Kitty uses Elasticsearch. Elasticsearch is a distributed search and analytics engine. Designed to find things fast. Really fast. Data is stored as JSON documents. not rows and tables. Each document is analyzed, tokenized, and added to an index. Like a giant cat encyclopedia.
Now Kitty can search by: Word. Phrase. Partial match. Relevance. Even typos. It’s perfect for: Log analytics. Full-text search. Business and operational insights. That’s Elasticsearch. Kitty uses it when Kitty needs answers from mountains of data before her nap timer runs out
Elasticsearch, sometimes written Elastic Search, is a search engine you run alongside your database, built to find the few documents that matter among millions in milliseconds, even when the person searching misspells a word.
It sits behind a lot of site search boxes, log dashboards and security tools.
Why a database struggles with search
A relational database is very good at exact questions: the order with id 4182, every order placed yesterday, every customer in Leeds. Its indexes keep values sorted, so it can jump straight to a match.
Search questions are fuzzier. Someone types 'tuna flakes' and expects the order 'Tuna Deluxe, extra flakes', with the best match first. They type 'tunna' and still expect results. In SQL the obvious tool is LIKE '%tuna%', but a pattern that starts with a wildcard can't use an ordinary index, so the database reads every row. Even then it can't tell you which match is the best one.
Many databases now have full-text search of their own, and for a small app that is often enough. Elasticsearch starts from the other end: the way it stores data and the way it ranks results are both designed around finding things.
Documents, not rows
Elasticsearch stores data as JSON documents. A document is one thing you want to find: an order, a log line, a product. Documents of the same kind live in an index, which plays roughly the part of a table in a database.
Elastic Search, explained by cats. Kitty has a lot of data. Logs. Metrics. Messages. Endless tuna orders. She tries to search using a database. Rows. Tables. Indexes fighting for their lives. Databases store data. they’re not built to search it.
So Kitty uses Elasticsearch. Elasticsearch is a distributed search and analytics engine. Designed to find things fast. Really fast. Data is stored as JSON documents. not rows and tables. Each document is analyzed, tokenized, and added to an index. Like a giant cat encyclopedia.
Now Kitty can search by: Word. Phrase. Partial match. Relevance. Even typos. It’s perfect for: Log analytics. Full-text search. Business and operational insights. That’s Elasticsearch. Kitty uses it when Kitty needs answers from mountains of data before her nap timer runs out
Each field has a type, set in the index's mapping. Two string types matter most:
text is analysed for full-text search, as the next section explains.
keyword is stored exactly as written, for filters, sorting and counting.
If you don't define a mapping, Elasticsearch guesses one from the first document it sees. This is called dynamic mapping. A string normally becomes text with a keyword copy alongside it, but date detection can map a date-like string as a date instead.
What happens when a document is indexed
Before a text field is indexed for search, it goes through an analyser. The original value is normally retained separately in the document's _source. The standard analyser does two things:
It splits the text into words, called tokens, and drops most punctuation.
It lowercases every token.
So 'Tuna Deluxe, extra flakes' becomes tuna, deluxe, extra and flakes. Language analysers go further. The English one, for example, also reduces words to a common stem, so 'flakes' and 'flake' match each other.
Each token then goes into an inverted index: a map from every term to the documents that contain it, like the index at the back of a book. With three orders, 'Tuna Deluxe, extra flakes' (1), 'Tuna and salmon platter' (2) and 'Tuna flakes, small tin' (3), part of it looks like this:
Term
Documents
tuna
1, 2, 3
deluxe
1
flakes
1, 3
salmon
2
A search runs the query text through the same analyser, then looks up each term. Finding every document that contains 'flakes' is one lookup, not a read of every document, which is why search stays fast as the data grows.
Ways to search
Elasticsearch has a JSON query language with a query type for each kind of question:
Word: a match query finds documents that contain any of the search terms.
Phrase: match_phrase needs the terms next to each other, in order. The index stores each token's position, so it can check.
Partial match: prefix and match_phrase_prefix match the start of a word, which is how search-as-you-type boxes work.
Typos: fuzzy matching lets a term match a word a small number of edits away, where an edit is adding, removing, changing or swapping one character. With "fuzziness": "AUTO", terms of one or two characters must match exactly, terms of three to five may be one edit off, and longer terms two.
Here is an order stored, then searched for with two typos, using curl against a local node on the default port. Set ELASTIC_PASSWORD to the node's password and run these commands where http_ca.crt is available; new installations normally enable authentication and HTTPS:
The query is analysed into tunna and flake. Each is five letters long, so each may be one edit off: removing an 'n' turns tunna into tuna, and adding an 's' turns flake into flakes. The order comes back as a hit with a _score field, the number used to rank it.
How relevance is scored
Every hit gets a score, and results come back highest first. The default scoring formula is BM25, which weighs three things:
Term frequency: a field that mentions the term more often scores higher, with each extra mention adding less.
Rarity: a rare term counts for more than a common one. If every order mentions tuna, a match on 'deluxe' says far more about what the searcher wants.
Field length: a match in a short field counts for more than the same match buried in a long one.
You can tune this, for example by boosting matches in a product's name above matches in its description. A LIKE query has nothing like it: without an ORDER BY, its matches come back in no particular order.
Distributed by design
An index is split into shards, and each shard is a complete search index of its own, built on the Apache Lucene library. A cluster of machines, called nodes, shares the shards between them. Each primary shard can have replicas on other nodes, which serve searches too and take over if a node fails.
A search goes to one copy of every shard. Each shard finds its own best hits, and the node that received the request merges them into one ranked list. Adding nodes spreads the shards more thinly, so a cluster grows by adding machines rather than buying a bigger one.
Search is also near real-time. A new document becomes searchable after the next refresh, which by default happens about once a second, not the instant the write returns.
What it is used for
Full-text search: site search, product catalogues, documentation.
Log analytics: applications ship their logs into Elasticsearch and teams explore them in Kibana, Elastic's dashboard tool. Together with Logstash for collecting logs, this is the 'ELK stack'.
Security data: security tools collect events from servers and networks so analysts can search them and spot attacks.
Business and operational insights: aggregations count, average and group documents by any field, which is what dashboards are built on.
Recent versions can also store vectors and search by meaning rather than by matching words, the idea behind a vector database. OpenSearch, a fork AWS started in 2021 after Elastic changed Elasticsearch's licence, works the same way at the level this page covers.
When not to use it
Don't make it the only copy of important data. Elasticsearch has no transactions across documents. Most teams keep their main database as the source of truth and copy data into Elasticsearch, so the index can always be rebuilt.
Skip it for a small app whose database can already search well enough. A cluster costs memory, disk space and someone's time to run.
Avoid it for relational queries with joins. Documents are meant to stand alone, so data that needs joining is usually copied into each document instead.
Common mistakes
Leaving the mapping to chance: dynamic mapping guesses from the first document, and changing a field's type later means copying everything into a new index. Define the mapping for fields you care about.
Filtering on a text field: a term query for 'Delivered' on an analysed field finds nothing, because the index holds delivered. Filter on the keyword version.
Searching straight after writing: tests that index a document and search for it at once fail now and then, because the refresh hasn't happened yet. Ask the write to wait for it with ?refresh=wait_for.
Paging deep with from and size: by default a search refuses to go past the first 10,000 hits. Use search_after to walk through large result sets.
Creating too many small shards: every shard has a fixed cost in memory and bookkeeping, so thousands of tiny ones slow a cluster down.
Key takeaways
Elasticsearch is a distributed search and analytics engine, built to find things in large amounts of data fast.
It stores JSON documents and analyses text into tokens in an inverted index, so finding a word is one lookup.
It can search by word, phrase, partial match and relevance, and tolerate typos with fuzzy matching.
Keep your main database as the source of truth, and define mappings rather than letting Elasticsearch guess.