Elasticsearch analyzers determine the recall and precision of full-text search. This article focuses on built-in analyzers, the IK Chinese analyzer, and the pinyin analyzer to address three common pain points: poor Chinese tokenization, hard-to-maintain domain vocabulary, and weak pinyin search support. Keywords: Elasticsearch, IK Analyzer, pinyin analyzer.
Technical Specification Snapshot
| Parameter | Description |
|---|---|
| Core Technology | Elasticsearch 7.17.x |
| Primary Language | Java |
| Protocol / Ecosystem | HTTP, REST API |
| Source Format | Blog-based technical practice summary |
| GitHub Dependencies | analysis-ik, analysis-pinyin |
| Applicable Scenarios | Chinese search, pinyin retrieval, full-text indexing |
| Star Count | Not provided in the original source; refer to the plugin repositories for real-time data |
Elasticsearch analyzers are the deciding factor in full-text search quality.
An analyzer converts raw text into searchable terms. It typically consists of three parts: character filters, a tokenizer, and token filters. These components handle character preprocessing, tokenization, and secondary term processing.
In Elasticsearch, the tokenization strategy directly affects index granularity, query recall, and relevance ranking. English often works well with the default analyzer, but if you apply the default configuration to Chinese, Elasticsearch will usually split text into single characters, which creates noisy search results.
POST /_analyze
{
"analyzer": "standard",
"text": "我是中国人" // Chinese is usually split into single characters by the default analyzer
}This request verifies how the default analyzer tokenizes Chinese text.
Built-in analyzers work for basic scenarios, but their Chinese support is limited.
standard is better suited for general English text.
standard is the default analyzer. It tokenizes by word boundaries and lowercases terms. It works well for English search, but for Chinese it often splits text into single-character tokens such as “我”, “是”, “中”, “国”, and “人”, which makes it unsuitable for Chinese content search.
simple, keyword, and whitespace solve specific structured-text problems.
simple splits on non-letter characters and fits basic English text. keyword does not split text at all, so it is ideal for exact-match fields such as phone numbers, identity numbers, and status enums. whitespace splits only on spaces and works for pre-tokenized text.
POST /_analyze
{
"analyzer": "keyword",
"text": "I Love You" // Preserve the entire string as a single term
}This request demonstrates the typical exact-match behavior of the keyword analyzer.
stop, pattern, and fingerprint are more utility-oriented.
stop targets English stop-word filtering. pattern tokenizes with regular expressions, which is flexible but performance-sensitive. fingerprint normalizes text and generates a fingerprint token, which is useful for deduplication, clustering, and similar-content detection, but not as a primary analyzer for Chinese.
The IK analyzer is the mainstream choice for Chinese production environments.
The core value of IK is that it provides controllable Chinese tokenization. It has two commonly used modes: ik_max_word, which fits the indexing stage and maximizes recall, and ik_smart, which fits the query stage and favors more reasonable token granularity with less noise.
A common production pattern is to use ik_max_word for indexing and ik_smart for querying. This preserves more candidate terms while avoiding excessive query expansion.
elasticsearch-plugin install https://get.infini.cloud/elasticsearch/analysis-ik/7.17.26
chown -R elasticsearch:elasticsearch /data00/software/elasticsearch-7.17.26
systemctl restart elasticsearch.service # Restart to activate the pluginThese commands install the IK plugin online and activate it.
IK local dictionaries work well for low-frequency domain term updates.
When the default tokenization does not match your business semantics, you can extend IK with ext_dict local dictionaries. For example, if you treat “我是中国人” as a complete domain phrase, you can significantly improve recall and matching quality for that specific expression.
<entry key="ext_dict">ik_diy.dic</entry>This configuration line adds a custom dictionary file to the IK analyzer.
IK remote dictionaries are better suited for hot updates in production.
The key advantage of remote dictionaries is that you can update vocabulary without restarting Elasticsearch nodes. They fit high-change scenarios such as e-commerce product terms, trending entity names, and new terms on content platforms. The server only needs to provide UTF-8 text and return Last-Modified and ETag headers.
Compared with local dictionaries, remote dictionaries are better for centralized governance, multi-node synchronization, and automatic refresh. In large clusters, this is a critical capability for reducing operational overhead.
<entry key="remote_ext_dict">http://es-dict.example.com:81/ext_dict.txt</entry>This configuration line enables an IK remote extension dictionary for hot updates.
Nginx can serve as a lightweight remote dictionary service.
You can expose a plain-text dictionary file through Nginx and restrict access so that only Elasticsearch nodes can read it. During validation, check whether Content-Type, ETag, and Last-Modified are present. Otherwise, IK cannot determine whether the dictionary has changed.
The pinyin analyzer covers diverse Chinese search input patterns.
A pinyin analyzer converts Chinese characters into full pinyin, initials, or mixed terms. It is useful for product search, name search, in-site suggestions, and pinyin typo tolerance. When a user types shouji and finds “手机”, or types zs and finds “张三”, this capability is what makes it work.
Using a pinyin analyzer alone has limited value. The best practice is to perform Chinese tokenization first and then map the resulting terms into pinyin, so you can support both Chinese-character search and pinyin search.
POST /_analyze
{
"analyzer": "pinyin",
"text": "我爱中国" // Outputs pinyin terms such as wo, ai, zhong, and guo
}This request verifies the basic output of the pinyin analyzer.
Combining IK with pinyin provides a cost-effective Chinese search architecture.
A combined solution usually defines a custom analyzer: the tokenizer uses ik_max_word or ik_smart, and the filter uses pinyin. This allows the index to retain original Chinese terms while also generating derived terms such as full pinyin and initials, which supports multi-entry search.
PUT /pinyin_index
{
"settings": {
"analysis": {
"filter": {
"my_pinyin_filter": {
"type": "pinyin",
"keep_full_pinyin": true,
"keep_first_letter": true,
"keep_original": true // Preserve the original Chinese terms as well
}
},
"analyzer": {
"my_pinyin_analyzer": {
"type": "custom",
"tokenizer": "ik_max_word", // Perform Chinese tokenization first
"filter": ["my_pinyin_filter"] // Then convert to pinyin
}
}
}
}
}This configuration builds a custom index analysis chain that supports both Chinese-character and pinyin search.
Production configuration should balance recall, precision, and operational maintainability.
A recommended baseline is: use analyzer=ik_max_word and search_analyzer=ik_smart for Chinese body text; update new domain terms through remote dictionaries; and add a pinyin analyzer to high-frequency search fields such as names, brands, and product titles.
If a field requires exact filtering, such as phone numbers, identity numbers, or order numbers, do not use text + IK. Use keyword or a text + keyword multi-field mapping instead. An analyzer is not better just because it is more complex. The right choice is the one that matches the field semantics.
FAQ
1. Why should you avoid using standard directly for Chinese fields?
Because standard usually splits Chinese text into single characters, which breaks semantic meaning. Search recall may appear to increase, but precision will drop significantly.
2. How should ik_max_word and ik_smart divide responsibilities?
A common best practice is to use ik_max_word during indexing to improve recall, and ik_smart during querying to control noise. This combination is the most stable approach.
3. If domain vocabulary changes frequently, should you choose a local dictionary or a remote dictionary?
Choose a remote dictionary first. It supports hot updates, centralized management, and multi-node synchronization, which makes it clearly superior to local dictionary approaches that require node restarts.
AI Visual Insight: This animated image shows a WeChat sharing prompt on the blog page. It serves as a distribution-entry cue rather than a visualization of Elasticsearch tokenization, index structure, or search results, so it does not carry core technical details.
AI Readability Summary: This article systematically explains the Elasticsearch analyzer ecosystem, covering built-in analyzers such as standard, simple, keyword, and whitespace, as well as installation, dictionary extension, remote hot updates, and combined retrieval configuration for the IK analyzer and the pinyin analyzer. It helps you build a Chinese search solution that balances recall, precision, and operational efficiency.