Many businesses need to share files internally or with third parties. It gets very critical to decide on the workflows to make this data sharing secure and scalable. To address this challenge, AWS Transfer Family provides data sharing capabilities with protocol options. It easily integrates with the existing identity providers and provides other network access controls. These features make the whole process of data transfer simple and business-centric. A business process may need recurring data sharing and, in that case, AWS Transfer Family can help in creating an architecture that is easy to use, reliable, and scalable as per the business needs.
Ibexlabs deals with customer problems by providing unique cloud-based solutions. In this blog, we focus on AWS Transfer Family and its utility in maintaining steady and secure data transfer with the help of a case wherein we helped a client to maintain data sharing with ease and efficiency.
AWS Transfer Family is an easy and secure file-sharing service that can transfer data into or out of the AWS storage services. These AWS storage services include:
To transfer the data, AWS Transfer Family uses three protocols:
These protocols make AWS Transfer Family a secure service for data transfer. While transferring the data, the transfer workflow is an important aspect. AWS Transfer Family moves files seamlessly and modernizes the workflow not in days but in hours by utilizing the existing authentication systems.
The compliance certifications that come with AWS Transfer Family make it a reliable solution for the data transfer and data encryption, VPC, and FIPS endpoints add to its capabilities. It can support thousands of concurrent users for transferring the data with a variety of options to scale. While storing the data in Amazon S3, and Amazon EFS, AWS Transfer Family also allows different controls over privacy and access of the files that further add insights into business dynamics.
Clients from different industries approach Ibexlabs for their technical needs. We are going to look at a case where a sales and marketing team is trying to promote solar power installations. To execute this, each installation allots IDs related to the projects or the dealers who are handling the project. Now, there comes a challenge when there are multiple dealers who are working on multiple projects.
These projects can be in different phases of installation. Since the solar power installation goes through various phases, it becomes tough to process this data in an organized way. In this particular case, customers were sending the data in the form of CSV (Comma Separated Values) files. It became cumbersome for the client to handle the bulk of CSV files together and process it for each solar power installation project.
There are various features that come with AWS Transfer Family such as optimized file-sharing that go beyond the server management limitations as per the business needs. The capability to share data in real-time on different scales helps in managing the file transfer workloads. Connecting your data infrastructure to AWS also allows you to use big data analytics and innovate by using artificial intelligence (AI) and machine learning (ML).
Ibexlabs implemented AWS Transfer Family to move the data from the external partner to the client’s S3 bucket. As shown in the architecture, Ibexlabs used AWS Lambda functions to process the data. AWS Lambda allows to map and associate the data to the respective project and the dealer. Lambda functions are capable of responding to execution requests at any scale, it may include a dozen events per day to hundreds of thousands per second.
Pic- Data Transfer Architecture with AWS Transfer Family
The client needed to update their database in real-time with a solution that could keep up with the pace. Also, it was necessary to integrate seamlessly into the serverless architecture for data transfer.
As we can see in the diagram, AWS Transfer Family uses SFTP (Safe File Transfer Protocol) to transfer the data. The architecture diagram also shows how AWS Transfer Family and AWS Serverless Services can be used together for a target workflow. Let’s look at the workflow,
To automate the above workflow Ibexlabs uses AWS Serverless Application Model (SAM).
For big data and analytics organizations, it is important to provide value-added data. It calls for providing secure access to the data. But as per the business needs, the scale of this data sharing may change from time to time. As we saw in the above case, Amazon Transfer Family equipped with multiple protocols provides access to the data in Amazon S3 and Amazon EFS.
The access control mechanism and the folder structure that come with Amazon transfer Family can create a workflow to decide selective access to the data. So, it becomes easier to address the scale of data sharing as per the business needs. The high availability of Amazon Transfer Family with the real-time scaling capabilities make it a holistic solution for data sharing without compromising security and compliance.
Sita Lohith Peravali is a DevOps Engineer with Ibexlabs. He has been working closely with the team to implement AWS Architecture, Security best practices for multiple clients, and develop solutions in the Ibex Azure Catalog.
