There is quite a bit more to Amazon S3 than storing public files. In this article, we look into how we can manage private files. To do so, we learn how to set up a proper private Amazon S3 bucket and how to upload and access files. We use streams and generate presigned URLs with an expiration time.
You can find the code from this series in this repository.
Setting up Amazon S3
The first thing to do is to create a new bucket.
This time, we intend to restrict access to the files we upload. Every time we want our users to be able to access a file, they will need to do it through our API.
The IAM user that we’ve created in the previous part of this series has access to all our buckets. Therefore, all we need to do to start using it is to add the name of the bucket to our environment variables.
1# ...
2AWS_PRIVATE_BUCKET_NAME=nestjs-series-private-bucket1An error has occurred. Please try again later.Managing files through the API
Once we have the above set up, we can start uploading files to our private bucket. When doing so, we want to save them in a similar way wheKolejny rok n dealing with public files. This time we won’t save the URL of the file, though.
Let’s allow our users to manage some files. To do that, let’s create the entity of a private file. It needs to contain the id of the user.
1import { Column, Entity, ManyToOne, PrimaryGeneratedColumn } from 'typeorm';
2import User from '../users/user.entity';
3
4@Entity()
5class PrivateFile {
6 @PrimaryGeneratedColumn()
7 public id: number;
8
9 @Column()
10 public key: string;
11
12 @ManyToOne(() => User, (owner: User) => owner.files)
13 public owner: User;
14}
15
16export default PrivateFile;Now we need to add information about the other side of the relationship.
1import { Entity, OneToMany } from 'typeorm';
2import PrivateFile from '../privateFIles/privateFile.entity';
3
4@Entity()
5class User {
6 // ...
7
8 @OneToMany(
9 () => PrivateFile,
10 (file: PrivateFile) => file.owner
11 )
12 public files: PrivateFile[];
13}
14
15export default User;If you want to know more about defining relationships with Postgres and TypeORM, check out API with NestJS #7. Creating relationships with Postgres and TypeORM
We need to save the key so that we can access or delete our private files. Let’s create a separate service to manage them.
1import { Injectable } from '@nestjs/common';
2import { InjectRepository } from '@nestjs/typeorm';
3import { Repository } from 'typeorm';
4import { S3 } from 'aws-sdk';
5import { ConfigService } from '@nestjs/config';
6import { v4 as uuid } from 'uuid';
7import PrivateFile from './privateFile.entity';
8
9@Injectable()
10export class PrivateFilesService {
11 constructor(
12 @InjectRepository(PrivateFile)
13 private privateFilesRepository: Repository<PrivateFile>,
14 private readonly configService: ConfigService
15 ) {}
16
17 async uploadPrivateFile(dataBuffer: Buffer, ownerId: number, filename: string) {
18 const s3 = new S3();
19 const uploadResult = await s3.upload({
20 Bucket: this.configService.get('AWS_PRIVATE_BUCKET_NAME'),
21 Body: dataBuffer,
22 Key: `${uuid()}-${filename}`
23 })
24 .promise();
25
26 const newFile = this.privateFilesRepository.create({
27 key: uploadResult.Key,
28 owner: {
29 id: ownerId
30 }
31 });
32 await this.privateFilesRepository.save(newFile);
33 return newFile;
34 }
35}1import { Module } from '@nestjs/common';
2import { TypeOrmModule } from '@nestjs/typeorm';
3import { PrivateFilesService } from './privateFiles.service';
4import { ConfigModule } from '@nestjs/config';
5import PrivateFile from './privateFile.entity';
6
7@Module({
8 imports: [
9 TypeOrmModule.forFeature([PrivateFile]),
10 ConfigModule,
11 ],
12 providers: [PrivateFilesService],
13 exports: [PrivateFilesService]
14})
15export class PrivateFilesModule {}Once that’s done, we can use all of the above to upload private files for our users.
1import { Injectable } from '@nestjs/common';
2import { InjectRepository } from '@nestjs/typeorm';
3import { Repository } from 'typeorm';
4import User from './user.entity';
5import { PrivateFilesService } from '../privateFIles/privateFiles.service';
6
7@Injectable()
8export class UsersService {
9 constructor(
10 @InjectRepository(User)
11 private usersRepository: Repository<User>,
12 private readonly privateFilesService: PrivateFilesService
13 ) {}
14
15 // ...
16
17 async addPrivateFile(userId: number, imageBuffer: Buffer, filename: string) {
18 return this.privateFilesService.uploadPrivateFile(imageBuffer, userId, filename);
19 }
20}1import { UsersService } from './users.service';
2import { Controller, Post, Req, UploadedFile, UseGuards, UseInterceptors } from '@nestjs/common';
3import JwtAuthenticationGuard from '../authentication/jwt-authentication.guard';
4import RequestWithUser from '../authentication/requestWithUser.interface';
5import { FileInterceptor } from '@nestjs/platform-express';
6import { Express } from 'express';
7
8@Controller('users')
9export class UsersController {
10 constructor(
11 private readonly usersService: UsersService,
12 ) {}
13
14 // ...
15
16 @Post('files')
17 @UseGuards(JwtAuthenticationGuard)
18 @UseInterceptors(FileInterceptor('file'))
19 async addPrivateFile(@Req() request: RequestWithUser, @UploadedFile() file: Express.Multer.File) {
20 return this.usersService.addPrivateFile(request.user.id, file.buffer, file.originalname);
21 }
22}After doing all of the above, our users can start uploading private files.
You can also implement deleting the files in a very similar way as in the previous part of this series. It might be worth checking if the currently logged in user is an owner of the file, though.
Accessing private files
Since the files we upload above are private, we can’t access them by simply entering a URL. Trying to do so will result in getting an error.
There is more than one way to approach this issue. Let’s start with the most straightforward one.
Fetching the file from Amazon S3 as a stream
The first solution to the above issue is to send the file through our API. The most fitting way to do that is to pipe a readable stream that we can get from the AWS SDK to our response. Thanks to working directly with streams, we don’t have to download the file into the memory in our server.
If you want to know more about piping streams in Node check out Node.js TypeScript #5. Writable streams, pipes, and the process streams
The first thing to do is to get a readable stream of data from our Amazon S3 bucket.
1import { Injectable } from '@nestjs/common';
2import { InjectRepository } from '@nestjs/typeorm';
3import { Repository } from 'typeorm';
4import { S3 } from 'aws-sdk';
5import { ConfigService } from '@nestjs/config';
6import PrivateFile from './privateFile.entity';
7import { NotFoundException } from '@nestjs/common';
8
9@Injectable()
10export class PrivateFilesService {
11 constructor(
12 @InjectRepository(PrivateFile)
13 private privateFilesRepository: Repository<PrivateFile>,
14 private readonly configService: ConfigService
15 ) {}
16
17 // ...
18
19 public async getPrivateFile(fileId: number) {
20 const s3 = new S3();
21
22 const fileInfo = await this.privateFilesRepository.findOne({ id: fileId }, { relations: ['owner'] });
23 if (fileInfo) {
24 const stream = await s3.getObject({
25 Bucket: this.configService.get('AWS_PRIVATE_BUCKET_NAME'),
26 Key: fileInfo.key
27 })
28 .createReadStream();
29 return {
30 stream,
31 info: fileInfo,
32 }
33 }
34 throw new NotFoundException();
35 }
36}Now we need to make sure if the users should be able to download the file.
1import { Injectable, UnauthorizedException } from '@nestjs/common';
2import { InjectRepository } from '@nestjs/typeorm';
3import { Repository } from 'typeorm';
4import User from './user.entity';
5import { FilesService } from '../files/files.service';
6import { PrivateFilesService } from '../privateFIles/privateFiles.service';
7
8@Injectable()
9export class UsersService {
10 constructor(
11 @InjectRepository(User)
12 private usersRepository: Repository<User>,
13 private readonly filesService: FilesService,
14 private readonly privateFilesService: PrivateFilesService
15 ) {}
16
17 // ...
18
19 async getPrivateFile(userId: number, fileId: number) {
20 const file = await this.privateFilesService.getPrivateFile(fileId);
21 if (file.info.owner.id === userId) {
22 return file;
23 }
24 throw new UnauthorizedException();
25 }
26}The interesting thing happens in the controller. Since we are working with streams directly, we need to access the Response object that NestJS uses under the hood.
1import { UsersService } from './users.service';
2import {
3 Controller,
4 Get,
5 Param,
6 Req,
7 Res,
8 UseGuards,
9} from '@nestjs/common';
10import JwtAuthenticationGuard from '../authentication/jwt-authentication.guard';
11import RequestWithUser from '../authentication/requestWithUser.interface';
12import { Response } from 'express';
13import FindOneParams from '../utils/findOneParams';
14
15@Controller('users')
16export class UsersController {
17 constructor(
18 private readonly usersService: UsersService,
19 ) {}
20
21 // ...
22
23 @Get('files/:id')
24 @UseGuards(JwtAuthenticationGuard)
25 async getPrivateFile(
26 @Req() request: RequestWithUser,
27 @Param() { id }: FindOneParams,
28 @Res() res: Response
29 ) {
30 const file = await this.usersService.getPrivateFile(request.user.id, Number(id));
31 file.stream.pipe(res)
32 }
33}Above, we use the FindOneParams DTO for the purpose of validation. If you want to know more, check out API with NestJS #4. Error handling and data validation
Generating presigned URLs
Responding with the data of the file is not the only solution to provide our users with their files. Another way is generating presigned URLs that allow access for a specific expiration time.
We can generate URLs for different actions. To create one for getting a resource, we need to use the getObject operation name:
1import { Injectable } from '@nestjs/common';
2import { InjectRepository } from '@nestjs/typeorm';
3import { Repository } from 'typeorm';
4import { S3 } from 'aws-sdk';
5import { ConfigService } from '@nestjs/config';
6import PrivateFile from './privateFile.entity';
7
8@Injectable()
9export class PrivateFilesService {
10 constructor(
11 @InjectRepository(PrivateFile)
12 private privateFilesRepository: Repository<PrivateFile>,
13 private readonly configService: ConfigService
14 ) {}
15
16 // ...
17
18 public async generatePresignedUrl(key: string) {
19 const s3 = new S3();
20
21 return s3.getSignedUrlPromise('getObject', {
22 Bucket: this.configService.get('AWS_PRIVATE_BUCKET_NAME'),
23 Key: key
24 })
25 }
26}The default expiration time of a presigned URL is 15 minutes. We could change it by adding an Expires parameter.
Let’s provide the user with an array of all of the uploaded files.
1import { Injectable, NotFoundException } from '@nestjs/common';
2import { InjectRepository } from '@nestjs/typeorm';
3import { Repository } from 'typeorm';
4import User from './user.entity';
5import { FilesService } from '../files/files.service';
6import { PrivateFilesService } from '../privateFIles/privateFiles.service';
7
8@Injectable()
9export class UsersService {
10 constructor(
11 @InjectRepository(User)
12 private usersRepository: Repository<User>,
13 private readonly filesService: FilesService,
14 private readonly privateFilesService: PrivateFilesService
15 ) {}
16
17 // ...
18
19 async getAllPrivateFiles(userId: number) {
20 const userWithFiles = await this.usersRepository.findOne(
21 { id: userId },
22 { relations: ['files'] }
23 );
24 if (userWithFiles) {
25 return Promise.all(
26 userWithFiles.files.map(async (file) => {
27 const url = await this.privateFilesService.generatePresignedUrl(file.key);
28 return {
29 ...file,
30 url
31 }
32 })
33 )
34 }
35 throw new NotFoundException('User with this id does not exist');
36 }
37}1import { UsersService } from './users.service';
2import {
3 Controller,
4 Get,
5 Req,
6 UseGuards,
7} from '@nestjs/common';
8import JwtAuthenticationGuard from '../authentication/jwt-authentication.guard';
9import RequestWithUser from '../authentication/requestWithUser.interface';
10
11@Controller('users')
12export class UsersController {
13 constructor(
14 private readonly usersService: UsersService,
15 ) {}
16
17 // ...
18
19 @Get('files')
20 @UseGuards(JwtAuthenticationGuard)
21 async getAllPrivateFiles(@Req() request: RequestWithUser) {
22 return this.usersService.getAllPrivateFiles(request.user.id);
23 }
24}Now, the user can access all of the files in a very straightforward way.
Summary
In this article, we’ve broadened our knowledge about Amazon S3. This time, we’ve learned how to manage private files. It included fetching them with the use of streams and generating presigned URLs. All of the above knowledge covers a variety of cases. Therefore, it allows us to manage files with Amazon S3 properly.