{"id":637,"date":"2020-05-02T10:21:09","date_gmt":"2020-05-02T10:21:09","guid":{"rendered":"https:\/\/blog.samarthya.me\/wps\/?p=637"},"modified":"2020-05-02T10:21:13","modified_gmt":"2020-05-02T10:21:13","slug":"analyze-_analyze-api","status":"publish","type":"post","link":"https:\/\/blog.samarthya.me\/wps\/2020\/05\/02\/analyze-_analyze-api\/","title":{"rendered":"Analyze: _analyze api"},"content":{"rendered":"

Text is a key data in most of the information that is ingested in the elasticsearch<\/strong>, and whether it is type text<\/code> or keyword<\/code> it requires careful consideration to choose which analyzer to use.<\/p>\n

What is Analyzer?<\/p>\n

An analyzer<\/em> \u2009which can be either builtin or custom,\u2009is just a package which contains three lower-level building blocks<\/p>\n

    \n
  1. character filters<\/em>,<\/li>\n
  2. tokenizers<\/em>, and<\/li>\n
  3. token filters<\/em>.<\/li>\n<\/ol>\n

    Main documentation here<\/a><\/p>\n

    \n

    NOTE: The analysis settings are non dynamic and can\u2019t be updated while the index is open.\u00a0<\/p>\n<\/blockquote>\n

    GET _analyze
    {
    \"text\": \"My simple text!\",
    \"explain\": true
    }<\/pre>\n
    Look at the output to understand which analyzer has been used and how it processes.<\/p>\n
    {
    \"detail\" : {
    \"custom_analyzer\" : false,
    \"analyzer\" : {
    \"name\" : \"standard\",<\/strong>
    \"tokens\" : [
    {
    \"token\" : \"my\",
    \"start_offset\" : 0,
    \"end_offset\" : 2,
    \"type\" : \"<ALPHANUM>\",
    \"position\" : 0,
    \"bytes\" : \"[6d 79]\",
    \"positionLength\" : 1,
    \"termFrequency\" : 1
    },
    {
    \"token\" : \"simple\",
    \"start_offset\" : 3,
    \"end_offset\" : 9,
    \"type\" : \"<ALPHANUM>\",
    \"position\" : 1,
    \"bytes\" : \"[73 69 6d 70 6c 65]\",
    \"positionLength\" : 1,
    \"termFrequency\" : 1
    },
    {
    \"token\" : \"text\",
    \"start_offset\" : 10,
    \"end_offset\" : 14,
    \"type\" : \"<ALPHANUM>\",
    \"position\" : 2,
    \"bytes\" : \"[74 65 78 74]\",
    \"positionLength\" : 1,
    \"termFrequency\" : 1
    }
    ]
    }
    }
    }<\/pre>\n
    Look at the STANDARD analyzer being used and how it breaks the original text from<\/p>\n
    My simple text!<\/strong><\/span><\/pre>\n\n\n
    to<\/p>\n\n\n\n
    \"tokens<\/strong>\" : [\n{\n\"token\" : \"my\",\n\"start_offset\" : 0,\n\"end_offset\" : 2,\n\"type\" : \"\",\n\"position\" : 0,\n\"bytes\" : \"[6d 79]\",\n\"positionLength\" : 1,\n\"termFrequency\" : 1\n},\n{\n\"token\" : \"simple\",\n\"start_offset\" : 3,\n\"end_offset\" : 9,\n\"type\" : \"\",\n\"position\" : 1,\n\"bytes\" : \"[73 69 6d 70 6c 65]\",\n\"positionLength\" : 1,\n\"termFrequency\" : 1\n},\n{\n\"token\" : \"text\",\n\"start_offset\" : 10,\n\"end_offset\" : 14,\n\"type\" : \"\",\n\"position\" : 2,\n\"bytes\" : \"[74 65 78 74]\",\n\"positionLength\" : 1,\n\"termFrequency\" : 1\n}\n]<\/pre>\n\n\n\n
    Look at every space the input is broken and even the punctuation has been removed – text<\/code> instead of text!<\/code>.<\/p>\n\n\n\n
    Changing the analyzer to – Whitespace<\/h3>\n\n\n\n
    GET _analyze
    {
    \"text\": \"My simple text!\",
    \"explain\": true,
    \"analyzer\": \"whitespace\"
    }<\/pre>\n\n\n\n
    The output is something like<\/p>\n\n\n\n
    {\n\"detail\" : {\n\"custom_analyzer\" : false,\n\"analyzer\" : {\n\"name\" : \"whitespace<\/strong>\",\n\"tokens\" : [\n{\n\"token\" : \"My\",\n\"start_offset\" : 0,\n\"end_offset\" : 2,\n\"type\" : \"word\",\n\"position\" : 0,\n\"bytes\" : \"[4d 79]\",\n\"positionLength\" : 1,\n\"termFrequency\" : 1\n},\n{\n\"token\" : \"simple\",\n\"start_offset\" : 3,\n\"end_offset\" : 9,\n\"type\" : \"word\",\n\"position\" : 1,\n\"bytes\" : \"[73 69 6d 70 6c 65]\",\n\"positionLength\" : 1,\n\"termFrequency\" : 1\n},\n{\n\"token\" : \"text!\",\n\"start_offset\" : 10,\n\"end_offset\" : 15,\n\"type\" : \"word\",\n\"position\" : 2,\n\"bytes\" : \"[74 65 78 74 21]\",\n\"positionLength\" : 1,\n\"termFrequency\" : 1\n}\n]\n}\n}\n}<\/pre>\n\n\n\n
    Look at how the output is similar except the punctuation has not been removed and the the case has been retained.<\/p>\n\n\n\n
    My<\/code> instead of my<\/code> as in the previous example.<\/p>\n\n\n\n
    Other Analyzer: Stop, Keyword, English<\/h3>\n\n\n\n
    Stop<\/h4>\n\n\n\n
    GET _analyze
    {
    \"text\": \"My simple and sweet text!\",
    \"explain\": true,
    \"analyzer\": \"stop\"
    }<\/pre>\n\n\n\n
    The stop words will be removed (Look at no and below)<\/p>\n\n\n\n
    {\n\"detail\" : {\n\"custom_analyzer\" : false,\n\"analyzer\" : {\n\"name\" : \"stop\",\n\"tokens\" : [\n{\n\"token\" : \"my\",\n\"start_offset\" : 0,\n\"end_offset\" : 2,\n\"type\" : \"word\",\n\"position\" : 0,\n\"bytes\" : \"[6d 79]\",\n\"positionLength\" : 1,\n\"termFrequency\" : 1\n},\n{\n\"token\" : \"simple\",\n\"start_offset\" : 3,\n\"end_offset\" : 9,\n\"type\" : \"word\",\n\"position\" : 1,\n\"bytes\" : \"[73 69 6d 70 6c 65]\",\n\"positionLength\" : 1,\n\"termFrequency\" : 1\n},\n{\n\"token\" : \"sweet\",\n\"start_offset\" : 14,\n\"end_offset\" : 19,\n\"type\" : \"word\",\n\"position\" : 3,\n\"bytes\" : \"[73 77 65 65 74]\",\n\"positionLength\" : 1,\n\"termFrequency\" : 1\n},\n{\n\"token\" : \"text\",\n\"start_offset\" : 20,\n\"end_offset\" : 24,\n\"type\" : \"word\",\n\"position\" : 4,\n\"bytes\" : \"[74 65 78 74]\",\n\"positionLength\" : 1,\n\"termFrequency\" : 1\n}\n]\n}\n}\n}<\/pre>\n\n\n\n
    English: Analyzer<\/h4>\n\n\n\n
    GET _analyze\n{\n\"text\": \"My simple and sweet text! by working.\",\n\"explain\": true,\n\"analyzer\": \"english<\/strong>\"\n}<\/pre>\n\n\n\n
    Look at the output as under<\/p>\n\n\n\n
    {\n\"detail\" : {\n\"custom_analyzer\" : false,\n\"analyzer\" : {\n\"name\" : \"english\",\n\"tokens\" : [\n{\n\"token\" : \"my\",\n\"start_offset\" : 0,\n\"end_offset\" : 2,\n\"type\" : \"\",\n\"position\" : 0,\n\"bytes\" : \"[6d 79]\",\n\"keyword\" : false,\n\"positionLength\" : 1,\n\"termFrequency\" : 1\n},\n{\n\"token\" : \"simpl\",<\/span><\/strong>\n\"start_offset\" : 3,\n\"end_offset\" : 9,\n\"type\" : \"\",\n\"position\" : 1,\n\"bytes\" : \"[73 69 6d 70 6c]\",\n\"keyword\" : false,\n\"positionLength\" : 1,\n\"termFrequency\" : 1\n},\n{\n\"token\" : \"sweet\",\n\"start_offset\" : 14,\n\"end_offset\" : 19,\n\"type\" : \"\",\n\"position\" : 3,\n\"bytes\" : \"[73 77 65 65 74]\",\n\"keyword\" : false,\n\"positionLength\" : 1,\n\"termFrequency\" : 1\n},\n{\n\"token\" : \"text\",\n\"start_offset\" : 20,\n\"end_offset\" : 24,\n\"type\" : \"\",\n\"position\" : 4,\n\"bytes\" : \"[74 65 78 74]\",\n\"keyword\" : false,\n\"positionLength\" : 1,\n\"termFrequency\" : 1\n},\n{\n\"token\" : \"work\",<\/span><\/strong>\n\"start_offset\" : 29,\n\"end_offset\" : 36,\n\"type\" : \"\",\n\"position\" : 6,\n\"bytes\" : \"[77 6f 72 6b]\",\n\"keyword\" : false,\n\"positionLength\" : 1,\n\"termFrequency\" : 1\n}\n]\n}\n}\n}<\/pre>\n\n\n\n
    Keyword<\/h4>\n\n\n\n
    GET _analyze\n{\n\"text\": \"My simple and sweet text! by working.\",\n\"explain\": true,\n\"analyzer\": \"keyword\"\n}<\/pre>\n\n\n\n
    The output will change as under<\/p>\n\n\n\n
    {\n\"detail\" : {\n\"custom_analyzer\" : false,\n\"analyzer\" : {\n\"name\" : \"keyword\",\n\"tokens\" : [\n{\n\"token\" : \"My simple and sweet text! by working.\",\n\"start_offset\" : 0,\n\"end_offset\" : 37,\n\"type\" : \"word\",\n\"position\" : 0,\n\"bytes\" : \"[4d 79 20 73 69 6d 70 6c 65 20 61 6e 64 20 73 77 65 65 74 20 74 65 78 74 21 20 62 79 20 77 6f 72 6b 69 6e 67 2e]\"<\/span><\/strong>,\n\"positionLength\" : 1,\n\"termFrequency\" : 1\n}\n]\n}\n}\n}<\/pre>\n\n\n\n
    Tokenizer and filters<\/h2>\n\n\n\n
    How about playing with the some examples of tokenizer and filters.<\/p>\n\n\n\n
    GET _analyze\n{\n\"tokenizer\": \"standard\",\n\"filter\": [\"lowercase\"],\n\"text\": \"My simple and sweet exmple at work(ing)!\"\n}<\/pre>\n\n\n\n
    I am using the standard<\/code> as tokenizer and lowercase<\/code> as filter. Look at how it tokenizes the information.<\/p>\n\n\n\n
    {\n\"tokens\" : [\n{\n\"token\" : \"my\",\n\"start_offset\" : 0,\n\"end_offset\" : 2,\n\"type\" : \"\",\n\"position\" : 0\n},\n{\n\"token\" : \"simple\",\n\"start_offset\" : 3,\n\"end_offset\" : 9,\n\"type\" : \"\",\n\"position\" : 1\n},\n{\n\"token\" : \"and\",\n\"start_offset\" : 10,\n\"end_offset\" : 13,\n\"type\" : \"\",\n\"position\" : 2\n},\n{\n\"token\" : \"sweet\",\n\"start_offset\" : 14,\n\"end_offset\" : 19,\n\"type\" : \"\",\n\"position\" : 3\n},\n{\n\"token\" : \"exmple\",\n\"start_offset\" : 20,\n\"end_offset\" : 26,\n\"type\" : \"\",\n\"position\" : 4\n},\n{\n\"token\" : \"at\",\n\"start_offset\" : 27,\n\"end_offset\" : 29,\n\"type\" : \"\",\n\"position\" : 5\n},\n{\n\"token\" : \"work\",\n\"start_offset\" : 30,\n\"end_offset\" : 34,\n\"type\" : \"\",\n\"position\" : 6\n},\n{\n\"token\" : \"ing\",\n\"start_offset\" : 35,\n\"end_offset\" : 38,\n\"type\" : \"\",\n\"position\" : 7\n}\n]\n}<\/pre>\n\n\n\n
    adding snowball to the filter list now, and adding the missing a<\/code> in example<\/code> for the text.<\/p>\n\n\n\n
    {\n\"tokens\" : [\n{\n\"token\" : \"my\",\n\"start_offset\" : 0,\n\"end_offset\" : 2,\n\"type\" : \"\",\n\"position\" : 0\n},\n{\n\"token\" : \"simpl\",\n\"start_offset\" : 3,\n\"end_offset\" : 9,\n\"type\" : \"\",\n\"position\" : 1\n},\n{\n\"token\" : \"and\",\n\"start_offset\" : 10,\n\"end_offset\" : 13,\n\"type\" : \"\",\n\"position\" : 2\n},\n{\n\"token\" : \"sweet\",\n\"start_offset\" : 14,\n\"end_offset\" : 19,\n\"type\" : \"\",\n\"position\" : 3\n},\n{\n\"token\" : \"exampl\",\n\"start_offset\" : 20,\n\"end_offset\" : 27,\n\"type\" : \"\",\n\"position\" : 4\n},\n{\n\"token\" : \"at\",\n\"start_offset\" : 28,\n\"end_offset\" : 30,\n\"type\" : \"\",\n\"position\" : 5\n},\n{\n\"token\" : \"work\",\n\"start_offset\" : 31,\n\"end_offset\" : 35,\n\"type\" : \"\",\n\"position\" : 6\n},\n{\n\"token\" : \"ing\",\n\"start_offset\" : 36,\n\"end_offset\" : 39,\n\"type\" : \"\",\n\"position\" : 7\n}\n]\n}<\/pre>\n\n\n\n
    look how it removed the e<\/code> in the example other than other things.<\/p>\n\n\n\n
    Defining a new analyzer<\/h3>\n\n\n\n
    PUT myIndex\n{\n  \"settings\": {\n     \"analysis\": {\n       \"char_filter\": {\n        },\n       \"filter\": {\n        },\n       \"analyzer\": {\n           \"new_analyzer\": {}\n        }\n     }\n   }\n}<\/pre>\n\n\n\n
    <\/p>\n","protected":false},"excerpt":{"rendered":"
    Text is a key data in most of the information that is ingested in the elasticsearch, and whether it is type text or keyword it requires careful consideration to choose which analyzer to use. What is Analyzer? An analyzer \u2009which can be either builtin or custom,\u2009is just a package which contains three lower-level building blocks […]<\/p>\n","protected":false},"author":2,"featured_media":641,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"image","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[34],"tags":[66,53],"class_list":["post-637","post","type-post","status-publish","format-image","has-post-thumbnail","hentry","category-technical","tag-analyze","tag-elasticsearch","post_format-post-format-image"],"_links":{"self":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts\/637","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/comments?post=637"}],"version-history":[{"count":0,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts\/637\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/media\/641"}],"wp:attachment":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/media?parent=637"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/categories?post=637"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/tags?post=637"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}