Many of the services available within the AWS ecosystem are actually managed services. They are added to the ecosystem to make managing even the most complex cloud infrastructure easy; you can generally focus on deploying services and apps or focus on developing your business rather than worrying about infrastructure all the time.
Amazon Elasticsearch Service is a good example of a managed service that can be very handy. As the name suggests, Amazon ES is designed to make utilizing Elasticsearch easy. As a managed service, Amazon ES takes care of most of the heavy lifting, including deploying and scaling Elasticsearch clusters.
One of the most exciting updates about Amazon ES is its support for Virtual Private Clouds. Setting up the Elasticsearch service domain is now something you can also do within a VPC, with all internal communications routed accordingly. There is no need to leave AWS VPCs exposed to external nodes. The approach also mitigates certain security risks.
The user interface is kept simple despite the complexity of this too. You can create and configure an Amazon ES domain directly from the ES console inside your AWS Management dashboard. Elasticsearch is designed primarily for structured data, so the next step to take is uploading data to your ES instances. This, too, is an easy task to complete, especially now that the platform supports wider data migration options.
Setting up an Amazon ES instance can also be done via an API. To complete the set, you get support for availability zones, subnets, and other parameters designed to help manage the network around your ES instances carefully. The dashboard for ES even includes information about cluster health and general performance.
Depending on how your architecture is configured, setting up the Elasticsearch service domain using your VPC and a VPN is a great solution to crucial issues, mainly the fact that you want your VPC to remain private and inaccessible from the outside world. At the same time, you maintain the option to open nodes to external users or services.
However, you cannot do both with Amazon ES and VPC. You have to set up the ES domain to be accessible one way or the other. The same is true when the ES domains are configured to work with VPNs and tunnels to your cloud ecosystem.
That’s actually the beauty of Amazon ES. Even when it is opened for external access, not everyone can use that service domain freely. Other measures such as IAM and the security policies you adopt will still monitor and filter traffic. This means that external users that don’t show sufficient credentials will still be blocked from accessing your VPC.
The simplicity of Amazon ES isn’t its only big advantage. Configuring ES is now easy thanks to templates and other bundles being made available on AWS CloudFormation. Data ingest is also made more efficient with the help of scalable (and highly available) computing power.
For instance, you can use Amazon Kinesis Data Firehose for data ingression. Since Kinesis Data Firehose supports automatic scaling and throughput monitoring out of the box, data ingress can be done without having to manually scale up (and down) your instances.
We also have native support for tools like Logtash. There are add-ons and plugins that make Elasticsearch very flexible as a platform. In fact, the Amazon ES eliminates many of the headaches faced by developers and those who want to treat infrastructure as a code.
Amazon ES is a handy tool to have. The sooner you start managing your endpoints and adjusting your policies to match best practices, the more tools will you have in your arsenal.
Follow the below guidelines to set up your Elasticsearch Service Domain using VPC with a VPN.
To set-up under multi availability zones, we need to select “Enable zone awareness.”
Once we enable this option, the ES domain will work automatically with a minimum of 2 nodes.
When you create an ES domain, a role is automatically created by AWS to connect your EC2 servers to the new domain.
curl 'ElasticSearch endpoint/_cat/health?v'
If the connection is a success then move onto the next step.
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.3-x86_64.rpmsudo rpm -vi filebeat-5.6.3-x86_64.rpm
Now check the status:
[root@2xhib ec2-user]# cat /etc/filebeat/filebeat.yml
You should see the following output on your screen.
Also, add log paths.
# all global-based paths that should be crawled and fetched.
Ibexlabs is an experienced DevOps & Managed Services provider and an AWS consulting partner. Our AWS Certified DevOps consultancy team evaluates your infrastructure and make recommendations based on your individual business or personal requirements. Contact us today and set up a free consultation to discuss a custom-built solution tailored just for you.
Hunt, R. (2017). Amazon Elasticsearch Service now supports VPC | Amazon Web Services. Retrieved from https://aws.amazon.com/blogs/aws/amazon-elasticsearch-service-now-supports-vpc/