Set up an Amazon Elastic Search Service Domain using a VPC with VPN
Many of the services available within the AWS ecosystem are actually managed services. They are added to the ecosystem to make managing even the most complex cloud infrastructure easy; you can generally focus on deploying services and apps or focus on developing your business rather than worrying about infrastructure all the time.
Amazon Elasticsearch Service is a good example of a managed service that can be very handy. As the name suggests, Amazon ES is designed to make utilizing Elasticsearch easy. As a managed service, Amazon ES takes care of most of the heavy lifting, including deploying and scaling Elasticsearch clusters.
A Growing Tool
One of the most exciting updates about Amazon ES is its support for Virtual Private Clouds. Setting up the Elasticsearch service domain is now something you can also do within a VPC, with all internal communications routed accordingly. There is no need to leave AWS VPCs exposed to external nodes. The approach also mitigates certain security risks.
The user interface is kept simple despite the complexity of this too. You can create and configure an Amazon ES domain directly from the ES console inside your AWS Management dashboard. Elasticsearch is designed primarily for structured data, so the next step to take is uploading data to your ES instances. This, too, is an easy task to complete, especially now that the platform supports wider data migration options.
Setting up an Amazon ES instance can also be done via an API. To complete the set, you get support for availability zones, subnets, and other parameters designed to help manage the network around your ES instances carefully. The dashboard for ES even includes information about cluster health and general performance.
Amazon Elasticsearch with VPC and VPN
Depending on how your architecture is configured, setting up the Elasticsearch service domain using your VPC and a VPN is a great solution to crucial issues, mainly the fact that you want your VPC to remain private and inaccessible from the outside world. At the same time, you maintain the option to open nodes to external users or services.
However, you cannot do both with Amazon ES and VPC. You have to set up the ES domain to be accessible one way or the other. The same is true when the ES domains are configured to work with VPNs and tunnels to your cloud ecosystem.
That’s actually the beauty of Amazon ES. Even when it is opened for external access, not everyone can use that service domain freely. Other measures such as IAM and the security policies you adopt will still monitor and filter traffic. This means that external users that don’t show sufficient credentials will still be blocked from accessing your VPC.
More Benefits to Enjoy
The simplicity of Amazon ES isn’t its only big advantage. Configuring ES is now easy thanks to templates and other bundles being made available on AWS CloudFormation. Data ingest is also made more efficient with the help of scalable (and highly available) computing power.
For instance, you can use Amazon Kinesis Data Firehose for data ingression. Since Kinesis Data Firehose supports automatic scaling and throughput monitoring out of the box, data ingress can be done without having to manually scale up (and down) your instances.
We also have native support for tools like Logtash. There are add-ons and plugins that make Elasticsearch very flexible as a platform. In fact, the Amazon ES eliminates many of the headaches faced by developers and those who want to treat infrastructure as a code.
Amazon ES is a handy tool to have. The sooner you start managing your endpoints and adjusting your policies to match best practices, the more tools will you have in your arsenal.
Follow the below guidelines to set up your Elasticsearch Service Domain using VPC with a VPN.
Setting Up Your Amazon Elasticsearch Service Domain using VPC with a VPNSteps:
- Creating an Elasticsearch domain in AWS. Use the steps outlined here to connect to your Amazon Elasticsearch Service domains from within an Amazon VPC. (Hunt, 2017)
To set-up under multi availability zones, we need to select “Enable zone awareness.”
Once we enable this option, the ES domain will work automatically with a minimum of 2 nodes.
When you create an ES domain, a role is automatically created by AWS to connect your EC2 servers to the new domain.
- In addition, you need to create separate security groups for your ES domain in AWS and add all ‘Server Security Groups’ (which we need to transfer log files to the domain) with only allowed port 443 including in the VPN IP.
- To test the connection from your server, run the command:
curl ‘ElasticSearch endpoint/_cat/health?v’
If the connection is a success then move onto the next step.
- Install a log forwarder in all client servers. To achieve this, run the command:
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.3-x86_64.rpmsudo rpm -vi filebeat-5.6.3-x86_64.rpm
Now check the status:
/etc/init.d/filebeat status
- Next, we need to configure filebeat with our ES domain.
[root@2xhib ec2-user]# cat /etc/filebeat/filebeat.yml
You should see the following output on your screen.
Also, add log paths.
# all global-based paths that should be crawled and fetched.
Paths:
– /home/webvent/logs/*log
– /home/webvent/logs/*.log.1
– /var/log/httpd/*.log
- This next step involves starting the Filebeat service. For the most simple Filebeat configuration, you can outline a single input through a single path. Filebeat supports various outputs, but usually, you’ll either send events directly to Elasticsearch (which is what we will do), or to Logstash for additional processing.
- Now to create an index in Kibana. Follow the steps outlined in the image below.
- Finally, you’ll see the logs in your ElasticSearch console.
Ibexlabs is an experienced DevOps & Managed Services provider and an AWS consulting partner. Our AWS Certified DevOps consultancy team evaluates your infrastructure and make recommendations based on your individual business or personal requirements. Contact us today and set up a free consultation to discuss a custom-built solution tailored just for you.
References
Hunt, R. (2017). Amazon Elasticsearch Service now supports VPC | Amazon Web Services. Retrieved from https://aws.amazon.com/blogs/aws/amazon-elasticsearch-service-now-supports-vpc/