Create a REST API as an Amazon S3 Proxy in API Gateway Automation
Amazon Web Services is a leading cloud provider which provides us with plenty of Paas, and Iaas, and services that we can use to build and deploy our applications. we gonna build and Deploy a REST API with API Gateway which acts as a proxy to S3 and can be used to perform Read/Write on S3 without any intermediate service, here we gonna use an open-source framework Serverless for deploying our API.
API Gateway
Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale.
S3
Amazon Simple Storage Service (Amazon S3) is an object storage service offering scalability, data availability, security, and performance.
Serverless
Serverless is a framework that can be used to build applications on AWS, this will encapsulate things and provide us with a simple structure to create something in AWS. We can Code less and Build more with Serverless.
Install Serverless
Install serverless by running the below command
npm i -g serverless
Check the version of serverless
serverless -v
Setup IAM Role for API Gateway to invoke AWS S3
To invoke S3 from API Gateway, it must have the required IAM permissions, for that we need to create an IAM Role and attach it to API Gateway.
Best Practice: Write IAM policies as strictly as possible to avoid any security issues.
IAM Policy Statement:
{ "Version": "2012-10-17", "Statement": [ { "Action": "s3:PutObject", "Resource": "arn:aws:s3:::my-bucket/*", "Effect": "Allow" } ] }
The above policy has one statement `s3:PutObject`, for resource `<S3 ARN>` which allows Put Object requests on S3 Bucket.
Trusted Policy: It is a policy in which we define the principals that we allow to assume the role, so here only the API Gateway service can assume this role based on the below policy.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "apigateway.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }
Note: We’ll be creating the above role using the serverless script given below.
Create REST API and Its Resources
To create a REST API, we need to do the following:
- Create REST API in API Gateway.
- Create Resources in created REST API.
- Configure S3 Integration.
We can automate all the above things using the below script.
Serverless config
Write the below content serverless.yml
service: api-gateway-to-s3-rest-api useDotenv: true provider: name: aws stage: ${opt:stage, 'dev'} region: ap-south-1 lambdaHashingVersion: 20201221 logs: restApi: true level: INFO deploymentBucket: blockPublicAccess: true name: ${self:custom.config.CODE_DEPLOYMENT_BUCKET} maxPreviousDeploymentArtifacts: 3 custom: config: CODE_DEPLOYMENT_BUCKET: ${env:CODE_DEPLOYMENT_BUCKET} S3_BUCKET: ${env:S3_BUCKET} resources: Resources: IAMRoleForAPIGateway: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: apigateway.amazonaws.com Action: sts:AssumeRole Description: Role for ${self:service} Path: / Policies: - PolicyName: ${self:service}-role-policy PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Action: s3:PutObject Resource: arn:aws:s3:::${self:custom.config.S3_BUCKET}/* RoleName: ${self:service}-${self:provider.stage}-Role RestAPI: Type: AWS::ApiGateway::RestApi Properties: Description: API to write request_body to S3 EndpointConfiguration: Types: - REGIONAL Name: ${self:service}-${self:provider.stage} RestAPIResource: Type: AWS::ApiGateway::Resource Properties: ParentId: !GetAtt RestAPI.RootResourceId PathPart: api RestApiId: !Ref RestAPI RestAPIObjectPathResource: Type: AWS::ApiGateway::Resource Properties: ParentId: !Ref RestAPIResource PathPart: '{object_path+}' RestApiId: !Ref RestAPI RestAPIPutMethod: Type: AWS::ApiGateway::Method Properties: HttpMethod: PUT AuthorizationType: NONE Integration: Type: AWS IntegrationHttpMethod: PUT Uri: arn:aws:apigateway:${self:provider.region} :s3:path/${stageVariables.bucket_name}/{object_path} Credentials: !GetAtt IAMRoleForAPIGateway.Arn RequestParameters: "integration.request.path.object_path" : "method.request.path.object_path" IntegrationResponses: - StatusCode: 200 - StatusCode: 400 RequestParameters: "method.request.path.object_path": true OperationName: write-data-to-s3 ResourceId: !Ref RestAPIObjectPathResource RestApiId: !Ref RestAPI MethodResponses: - StatusCode: 200 ResponseModels: application/json: "Empty" - StatusCode: 400 ResponseModels: application/json: "Error" RestAPIDeployment: Type: AWS::ApiGateway::Deployment Properties: RestApiId: !Ref RestAPI DependsOn: - RestAPIPutMethod RestAPIStage: Type: AWS::ApiGateway::Stage Properties: DeploymentId: !Ref RestAPIDeployment Description: ${self:provider.stage} stage RestApiId: !Ref RestAPI StageName: ${self:provider.stage} Variables: bucket_name : ${self:custom.config.S3_BUCKET}
Provider
In the provider, we configure cloud provider details like cloud provider name, region, logs config, etc.
Custom
In this section (not a default option, we can replace custom with any name) we gonna maintain any configuration variables required for this script.
Resources
resources to create, here in this case we’re creating the REST API along with associated resources, methods, and its integration.
Here we’ll not be creating a new bucket through this script, create an s3 bucket if not exists already.
Environment variables
Here in this script, we’ve below env variables:
- CODE_DEPLOYMENT_BUCKET: Cloudformation script generated based on our serverless script and application code (if any) will be stored in this bucket and it will then be used for deployment.
- S3_BUCKET: S3 Bucket on which we gonna perform an API request.
Deploy Our REST API
To deploy our REST API, you can run the below command.
serverless deploy --stage alpha
API Gateway Resources
After running the above command successfully you can check API Gateway, you’ll find REST API with GET and PUT methods configured with S3 as a proxy.
Here object_path+ (suffix +) indicates that we can pass as many folders in the place of object_path /dir_1/dir_2/…./dir_n/myfile.txt, without + we can only pass one string /myfile.txt.
Test our API
We can invoke the API through any browser client to test out the API which will read/write a file in the S3.
PUT Method
In the below image I’ve invoked an API with the PUT method, containing a request body Hello Beginner the new file will be created in our bucket with the object path dillip/file.txt in the below case.
Note: As we’ve set NONE for method authorization, no need to pass any authorization headers or data.
GET Method
In the below image, I’ve invoked an API with the GET method, retrieving the file in our bucket with the object path dillip/file.txt we’ve created in the above PUT request.
Other Ways
There is an alternate way to do the above thing, i.e API Gateway REST API with Lambda integration, but we’ve to choose the approach based on our required outcome.
API Gateway -> AWS Lambda -> AWS S3 (Lambda as Intermediate), this approach is suitable if you wanna perform some complex logic upon objects before making a request to S3.
Conclusion
In this article, we’ve learned a simple way to build REST API which exposes GET and PUT methods to retrieve or manipulate the objects in the AWS S3, here AWS API Gateway acts as a proxy to AWS S3 Bucket.