Data Structure
Networking
RDBMS
Operating System
Java
MS Excel
iOS
HTML
CSS
Android
Python
C Programming
C++
C#
MongoDB
MySQL
Javascript
PHP
Physics
Chemistry
Biology
Mathematics
English
Economics
Psychology
Social Studies
Fashion Studies
Legal Studies
- Selected Reading
- UPSC IAS Exams Notes
- Developer's Best Practices
- Questions and Answers
- Effective Resume Writing
- HR Interview Questions
- Computer Glossary
- Who is Who
How to use Boto3 library in Python to upload an object in S3 using AWS Resource?
Problem Statement − Use Boto3 library in Python to upload an object into S3. For example, how to upload test.zip into Bucket_1 of S3.
Approach/Algorithm to solve this problem
Step 1 − Import boto3 and botocore exceptions to handle exceptions.
Step 2 − From pathlib, import PurePosixPath to retrive filename from path
Step 3 − s3_path and filepath are the two parameters in function upload_object_into_s3
Step 4 − Validate the s3_path is passed in AWS format as s3://bucket_name/key and filepath as local path C://users/filename
Step 5 − Create an AWS session using boto3 library.
Step 6 − Create an AWS resource for S3.
Step 7 − Split the S3 path and perform operations to separate the root bucket name and key path
Step 8 − Get the file name for complete filepath and add into S3 key path.
Step 9 − Now use the function upload_fileobj to upload the local file into S3.
Step 10 − Use the function wait_until_exists to wait until operation is finished.
Step 11 − Handle the exception based on response code to validate whether file is uploaded or not.
Step 12 − Handle the generic exception if something went wrong while uploading the file
Example
Use the following code to upload a file into AWS S3 −
import boto3
from botocore.exceptions import ClientError
from pathlib import PurePosixPath
def upload_object_into_s3(s3_path, filepath):
if 's3://' in filepath:
print('SourcePath is not a valid path.' + filepath)
raise Exception('SourcePath is not a valid path.')
elif s3_path.find('s3://') == -1:
print('DestinationPath is not a s3 path.' + s3_path)
raise Exception('DestinationPath is not a valid path.')
session = boto3.session.Session()
s3_resource = session.resource('s3')
tokens = s3_path.split('/')
target_key = ""
if len(tokens) > 3:
for tokn in range(3, len(tokens)):
if tokn == 3:
target_key += tokens[tokn]
else:
target_key += "/" + tokens[tokn]
target_bucket_name = tokens[2]
file_name = PurePosixPath(filepath).name
if target_key != '':
target_key.strip()
key_path = target_key + "/" + file_name
else:
key_path = file_name
print(("key_path: " + key_path, 'target_bucket: ' + target_bucket_name))
try:
# uploading Entity from local path
with open(filepath, "rb") as file:
s3_resource.meta.client.upload_fileobj(file, target_bucket_name, key_path)
try:
s3_resource.Object(target_bucket_name, key_path).wait_until_exists()
file.close()
except ClientError as error:
error_code = int(error.response['Error']['Code'])
if error_code == 412 or error_code == 304:
print("Object didn't Upload Successfully ", target_bucket_name)
raise error
return "Object Uploaded Successfully"
except Exception as error:
print("Error in upload object function of s3 helper: " + error.__str__())
raise error
print(upload_object_into_s3('s3://Bucket_1/testfolder', 'c://test.zip'))Output
key_path:/testfolder/test.zip, target_bucket: Bucket_1 Object Uploaded Successfully
680 Views
