{"id":442,"date":"2020-02-24T11:35:34","date_gmt":"2020-02-24T11:35:34","guid":{"rendered":"https:\/\/www.samarthya.me\/wps\/?p=442"},"modified":"2020-02-24T12:25:40","modified_gmt":"2020-02-24T12:25:40","slug":"elasticsearch-v7-6-2","status":"publish","type":"post","link":"https:\/\/blog.samarthya.me\/wps\/2020\/02\/24\/elasticsearch-v7-6-2\/","title":{"rendered":"Elasticsearch v7.6"},"content":{"rendered":"\n<p>In the previous blog we searched for a match_all, and sorted it by age. In this we will improve upon the query we are writing and look for other options.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">GET profile\/_search \n  {\n    \"query\": {\n      \"match\": {\n        \"title\": \"Mr. Ms.\"\n      }\n    },\n    \"sort\": [\n      {\n        \"age\": \"asc\"\n      }\n    ],\n    \"size\": 3\n  }<\/pre>\n\n\n\n<p>The command executed above, uses <code>match<\/code> which allows to search for specific terms within a field <code>title<\/code> and the terms we are looking for are <code>Mr.<\/code> or <code>Mrs.<\/code><\/p>\n\n\n\n<div class=\"wp-block-group\"><div class=\"wp-block-group__inner-container is-layout-flow wp-block-group-is-layout-flow\">\n<div class=\"wp-block-group\"><div class=\"wp-block-group__inner-container is-layout-flow wp-block-group-is-layout-flow\">\n<p>The response as received is as under<\/p>\n<\/div><\/div>\n\n\n\n<pre class=\"wp-block-preformatted\">{\n   \"took\" : 0,\n   \"timed_out\" : false,\n   \"_shards\" : {\n     \"total\" : 1,\n     \"successful\" : 1,\n     \"skipped\" : 0,\n     \"failed\" : 0\n   },\n   \"hits\" : {\n     \"total\" : {\n       \"value\" : 10,\n       \"relation\" : \"eq\"\n     },\n     \"max_score\" : null,\n     \"hits\" : [\n       {\n         \"_index\" : \"profile\",\n         \"_type\" : \"_doc\",\n         \"_id\" : \"4\",\n         \"_score\" : null,\n         \"_source\" : {\n           \"name\" : \"Deepa G\",\n           \"age\" : 22,\n           \"title\" : \"Ms.\",\n           \"role\" : \"QA\",\n           \"org\" : \"Security\"\n         },\n         \"sort\" : [\n           22\n         ]\n       },\n       {\n         \"_index\" : \"profile\",\n         \"_type\" : \"_doc\",\n         \"_id\" : \"6\",\n         \"_score\" : null,\n         \"_source\" : {\n           \"name\" : \"Smdie G\",\n           \"age\" : 24,\n           \"title\" : \"Mr.\",\n           \"role\" : \"Program management\",\n           \"org\" : \"Security\"\n         },\n         \"sort\" : [\n           24\n         ]\n       },\n       {\n         \"_index\" : \"profile\",\n         \"_type\" : \"_doc\",\n         \"_id\" : \"7\",\n         \"_score\" : null,\n         \"_source\" : {\n           \"name\" : \"Amdie G\",\n           \"age\" : 24,\n           \"title\" : \"Mr.\",\n           \"role\" : \"Program management\",\n           \"org\" : \"Security\"\n         },\n         \"sort\" : [\n           24\n         ]\n       }\n     ]\n   }\n }<\/pre>\n<\/div><\/div>\n\n\n\n<div class=\"wp-block-group\"><div class=\"wp-block-group__inner-container is-layout-flow wp-block-group-is-layout-flow\">\n<h3 class=\"wp-block-heading\">match_phrase<\/h3>\n\n\n\n<pre class=\"wp-block-preformatted\">GET profile\/_search \n  {\n    \"query\": {\n      \"match_phrase\": {\n        \"name\": \"G\"\n      }\n    },\n    \"sort\": [\n      {\n        \"age\": \"asc\"\n      }\n    ],\n    \"size\": 2\n  }<\/pre>\n\n\n\n<p>In the example above I am trying to look for a phrase <code>G<\/code> rather than individual term.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">{\n   \"took\" : 0,\n   \"timed_out\" : false,\n   \"_shards\" : {\n     \"total\" : 1,\n     \"successful\" : 1,\n     \"skipped\" : 0,\n     \"failed\" : 0\n   },\n   \"hits\" : {\n     \"total\" : {\n       \"value\" : 8,\n       \"relation\" : \"eq\"\n     },\n     \"max_score\" : null,\n     \"hits\" : [\n       {\n         \"_index\" : \"profile\",\n         \"_type\" : \"_doc\",\n         \"_id\" : \"4\",\n         \"_score\" : null,\n         \"_source\" : {\n           \"name\" : \"Deepa G\",\n           \"age\" : 22,\n           \"title\" : \"Ms.\",\n           \"role\" : \"QA\",\n           \"org\" : \"Security\"\n         },\n         \"sort\" : [\n           22\n         ]\n       },\n       {\n         \"_index\" : \"profile\",\n         \"_type\" : \"_doc\",\n         \"_id\" : \"5\",\n         \"_score\" : null,\n         \"_source\" : {\n           \"name\" : \"Reepa G\",\n           \"age\" : 24,\n           \"title\" : \"Mrs.\",\n           \"role\" : \"QA\",\n           \"org\" : \"Security\"\n         },\n         \"sort\" : [\n           24\n         ]\n       }\n     ]\n   }\n }<\/pre>\n\n\n\n<p>I have limited the size to 2 and hence even though it matched more than 2 documents, the result size was limited to 2.<\/p>\n<\/div><\/div>\n\n\n\n<h2 class=\"wp-block-heading\">Complex Query<\/h2>\n\n\n\n<p>Creating a complex query is equally intuitive.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Bool &#8211; Query<\/h3>\n\n\n\n<div class=\"wp-block-group\"><div class=\"wp-block-group__inner-container is-layout-flow wp-block-group-is-layout-flow\">\n<p>Let&#8217;s look for all the users who are between the age group of <code>&gt;= 30 &amp;&amp; &lt;= 50<\/code>.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">GET profile\/_search \n  {\n    \"query\": {\n      \"bool\": {\n        \"must\": [\n          { \"range\": {\n            \"age\": {\n              \"gte\": 30,\n              \"lte\": 50\n            }\n          }}\n        ]\n      }\n    },\n    \"sort\": [\n      {\n        \"age\": \"asc\"\n      }\n    ],\n    \"size\": 2\n  }<\/pre>\n\n\n\n<p>The response as expected is<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">{\n   \"took\" : 3,\n   \"timed_out\" : false,\n   \"_shards\" : {\n     \"total\" : 1,\n     \"successful\" : 1,\n     \"skipped\" : 0,\n     \"failed\" : 0\n   },\n   \"hits\" : {\n     \"total\" : {\n       \"value\" : 3,\n       \"relation\" : \"eq\"\n     },\n     \"max_score\" : null,\n     \"hits\" : [\n       {\n         \"_index\" : \"profile\",\n         \"_type\" : \"_doc\",\n         \"_id\" : \"11\",\n         \"_score\" : null,\n         \"_source\" : {\n           \"name\" : \"Veronica G\",\n           \"age\" : 37,\n           \"title\" : \"Ms.\",\n           \"role\" : \"Engineering\",\n           \"org\" : \"Security\"\n         },\n         \"sort\" : [\n           37\n         ]\n       },\n       {\n         \"_index\" : \"profile\",\n         \"_type\" : \"_doc\",\n         \"_id\" : \"10\",\n         \"_score\" : null,\n         \"_source\" : {\n           \"name\" : \"Pranav G\",\n           \"age\" : 47,\n           \"title\" : \"Mr.\",\n           \"role\" : \"Engineering\",\n           \"org\" : \"Security\"\n         },\n         \"sort\" : [\n           47\n         ]\n       }\n     ]\n   }\n }<\/pre>\n\n\n\n<p>A simple modifications to exclude <code>Veronica<\/code> is as under.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-style-default is-layout-flow wp-block-quote-is-layout-flow\"><p>must_not clause is more of a filter.<\/p><\/blockquote>\n\n\n\n<pre class=\"wp-block-preformatted\">GET profile\/_search \n  {\n    \"query\": {\n      \"bool\": {\n        \"must\": [\n          { \"range\": {\n            \"age\": {\n              \"gte\": 30,\n              \"lte\": 50\n            }\n          }}\n        ],\n        \"must_not\": [\n          {\n            \"match\": {\n              \"name\": \"Veronica G\"\n            }\n          }\n        ]\n      }\n    },\n    \"sort\": [\n      {\n        \"age\": \"asc\"\n      }\n    ],\n    \"size\": 2\n  }<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Aggregating<\/h2>\n\n\n\n<p>Let&#8217;s show aggregation<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">GET profile\/_search \n  {\n    \"aggs\": {\n      \"Group-By-Age\": {\n        \"terms\": {\n          \"field\": \"title.keyword\"\n        }\n      }\n    }, \n    \"size\": 0\n  }<\/pre>\n\n\n\n<p>and the results are<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">{\n   \"took\" : 1,\n   \"timed_out\" : false,\n   \"_shards\" : {\n     \"total\" : 1,\n     \"successful\" : 1,\n     \"skipped\" : 0,\n     \"failed\" : 0\n   },\n   \"hits\" : {\n     \"total\" : {\n       \"value\" : 11,\n       \"relation\" : \"eq\"\n     },\n     \"max_score\" : null,\n     \"hits\" : [ ]\n   },\n   \"aggregations\" : {\n     \"Group-By-Age\" : {\n       \"doc_count_error_upper_bound\" : 0,\n       \"sum_other_doc_count\" : 0,\n       \"buckets\" : [\n         {\n           \"key\" : \"Mr.\",\n           \"doc_count\" : 8\n         },\n         {\n           \"key\" : \"Ms.\",\n           \"doc_count\" : 2\n         },\n         {\n           \"key\" : \"Mrs.\",\n           \"doc_count\" : 1\n         }\n       ]\n     }\n   }\n }<\/pre>\n\n\n\n<p>What are these buckets?<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">\"<strong>buckets<\/strong>\" : []<\/pre>\n\n\n\n<p>In our example we are aggregating for <code>title.keyword<\/code> and the <code>key<\/code> shows the unique values found and the <code>doc_count<\/code> is the total matches.<\/p>\n\n\n\n<p>So for our response we have 8 <code>Mr.<\/code> , 2 <code>Ms.<\/code> and 1 `Mrs.`<\/p>\n\n\n\n<p>If the <code>size<\/code> is not zero in the query all the responses that matched will also be returned in the <code>hits[]<\/code><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Complexity increased<\/h3>\n\n\n\n<p>A little modifications to further aggregate and group by role.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">GET profile\/_search \n  {\n    \"aggs\": {\n      \"Group-By-Age\": {\n        \"terms\": {\n          \"field\": \"title.keyword\"\n        },\n        \"aggs\": {\n          \"Group-By-Role\": {\n            \"terms\": {\n              \"field\": \"role.keyword\"\n            }\n          }\n        }\n      }\n    },\n    \"size\": 0\n  }<\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\">{\n   \"took\" : 1,\n   \"timed_out\" : false,\n   \"_shards\" : {\n     \"total\" : 1,\n     \"successful\" : 1,\n     \"skipped\" : 0,\n     \"failed\" : 0\n   },\n   \"hits\" : {\n     \"total\" : {\n       \"value\" : 11,\n       \"relation\" : \"eq\"\n     },\n     \"max_score\" : null,\n     \"hits\" : [ ]\n   },\n   \"aggregations\" : {\n     \"Group-By-Age\" : {\n       \"doc_count_error_upper_bound\" : 0,\n       \"sum_other_doc_count\" : 0,\n       \"buckets\" : [\n         {\n           \"key\" : \"Mr.\",\n           \"doc_count\" : 8,\n           \"Group-By-Role\" : {\n             \"doc_count_error_upper_bound\" : 0,\n             \"sum_other_doc_count\" : 0,\n             \"buckets\" : [\n               {\n                 \"key\" : \"Program management\",\n                 \"doc_count\" : 2\n               },\n               {\n                 \"key\" : \"Engineering\",\n                 \"doc_count\" : 1\n               },\n               {\n                 \"key\" : \"Engineering management\",\n                 \"doc_count\" : 1\n               },\n               {\n                 \"key\" : \"Lead\",\n                 \"doc_count\" : 1\n               },\n               {\n                 \"key\" : \"Lead Engr\",\n                 \"doc_count\" : 1\n               },\n               {\n                 \"key\" : \"Manager\",\n                 \"doc_count\" : 1\n               },\n               {\n                 \"key\" : \"Product management\",\n                 \"doc_count\" : 1\n               }\n             ]\n           }\n         },\n         {\n           \"key\" : \"Ms.\",\n           \"doc_count\" : 2,\n           \"Group-By-Role\" : {\n             \"doc_count_error_upper_bound\" : 0,\n             \"sum_other_doc_count\" : 0,\n             \"buckets\" : [\n               {\n                 \"key\" : \"Engineering\",\n                 \"doc_count\" : 1\n               },\n               {\n                 \"key\" : \"QA\",\n                 \"doc_count\" : 1\n               }\n             ]\n           }\n         },\n         {\n           \"key\" : \"Mrs.\",\n           \"doc_count\" : 1,\n           \"Group-By-Role\" : {\n             \"doc_count_error_upper_bound\" : 0,\n             \"sum_other_doc_count\" : 0,\n             \"buckets\" : [\n               {\n                 \"key\" : \"QA\",\n                 \"doc_count\" : 1\n               }\n             ]\n           }\n         }\n       ]\n     }\n   }\n }<\/pre>\n\n\n\n<h3 class=\"has-text-align-center wp-block-heading\">&#8212; THE &#8211; END &#8212; <\/h3>\n<\/div><\/div>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>In the previous blog we searched for a match_all, and sorted it by age. In this we will improve upon the query we are writing and look for other options. GET profile\/_search { &#8220;query&#8221;: { &#8220;match&#8221;: { &#8220;title&#8221;: &#8220;Mr. Ms.&#8221; } }, &#8220;sort&#8221;: [ { &#8220;age&#8221;: &#8220;asc&#8221; } ], &#8220;size&#8221;: 3 } The command executed [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":444,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[34],"tags":[53,54],"class_list":["post-442","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technical","tag-elasticsearch","tag-v7-6"],"_links":{"self":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts\/442","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/comments?post=442"}],"version-history":[{"count":0,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts\/442\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/media\/444"}],"wp:attachment":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/media?parent=442"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/categories?post=442"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/tags?post=442"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}