Microservice for user management exemplifying part of the ML development architecture, implemented with Systems Manager Parameter Store, Api-Gateway, Serverless-Framework, Lambda, NodeJs, Sequelize, Mysql, Amazon RDS, Unit Test with Jest, among others. AWS services are tested locally. The project code and its documentation (less technical doc) have been developed in English.
See
- 1.0) Project description.
- 1.1) Project execution.
- 1.2) Project setup from scratch
- 1.3) Technologies.
1.0) Description π
See
- The Microservice is designed under the MVC architecture. This architecture consists of and is divided into the model layer (definition of the user table), the service layer (the connection and transactions to the db with sequelize) and the controller layer (the implemented lambdas).
- Each lambda performs the token authentication check, those that wait for a body type event check these fields and all the logic to be performed is abstracted from it to decouple functionalities together with low coupling.
- Endpoints that allow the return of more than one object according to the applied search logic are handled with pagination if required. Default pagination is applied.
- The image of the AWS architecture used describes the operating flow of the microservice in a general way. Any request to the microservice starts from a client (Postman, server, etc.).
Step 0
: This request is received by the api-gateway and will only be validated if the correct x-api-key is found within the headers of said request.Steps 1A, 1B, etc
: All these steps correspond to an endpoint with its specific resource. For ex. for getAllUsers (1A) it is http://localhost:4000/dev/users/list ....check those endpoints in endpoints section. Each lambda performs x-api-key and token checking.Steps 2
: The lambdas perform the validations of the corresponding ssm with the System Manager Paramater Store... they validate token, connection values with the db, etc.Steps 3
: The lambdas perform the necessary transactions and operations with the db (Mysql).Clarifications
: This operation is emulated within the same network and in a local environment with the corresponding serverless plugins.
1.1) Project execution π
See
- Once a work environment has been created through some IDE, we clone the project
git clone https://github.com/andresWeitzel/Microservice_Mercadolibre_Users_AWS
- We position ourselves on the project
cd 'projectName'
- We install the latest LTS version of Nodejs(v18)
- We install the Serverless Framework globally if we have not already done so
npm install -g serverless
- We verify the version of Serverless installed
sls -v
- We install all the necessary packages
npm i
- The ssm and env variables used in the project are maintained to simplify the project configuration process. It is recommended to add the corresponding files (serverless_ssm.yml and .env) to the .gitignore.
- The start script configured in the project's package.json is responsible for launching
- The serverless-offline plugin
- The remark-lint plugin for .md files (only --output is applied for check and autoformat without terminating the process and being able to execute the serverless script)
- The test is for using jest
"scripts": {
"serverless-offline": "sls offline start",
"start": "npm run format-md && npm run serverless-offline",
"start:dev": "nodemon -e js,ts,yml,json --exec \"sls offline start\"",
"format-prettier": "prettier --write \"{src,test}/**/*.{js,ts}\"",
"check": "remark . --quiet --frail",
"format-remark": "remark . --quiet --frail --output",
"format-md": "remark . --output",
"test": "jest --verbose",
"test:watch": "jest --watch --verbose",
"test:cov": "jest --coverage --verbose"
},
- We run the app from terminal.
npm start
- If a message appears indicating that port 4000 is already in use, we can terminate all dependent processes and run the app again
npx kill-port 4000
npm start
1.2) Project setup from scratch π
Ver
- We create a work environment through some ide, after creating a folder we position ourselves on it
cd 'projectName'
- We install the latest LTS version of Nodejs(v18)
- We install the Serverless Framework globally if we have not already done so
npm install -g serverless
- We verify the version of Serverless installed
sls -v
- We initialize a serverles template
serverless create --template aws-nodejs
- We initialize an npm project
npm init -y
- We install serverless offline
npm i serverless-offline --save-dev
- We install serverless ssm
npm i serverless-offline-ssm --save-dev
plugins:
- serverless-offline-ssm
- serverless-offline
- We will configure a standard markdown file format for the project via remark-lint
npm install remark-cli remark-preset-lint-consistent remark-preset-lint-recommended remark-lint-list-item-indent --save-dev
npm install remark-lint-emphasis-marker remark-lint-strong-marker --save-dev
npm install remark-lint-table-cell-padding --save-dev
- Then we add the configuration for the scripts from the package.json
"scripts": {
"check": "remark . --quiet --frail",
"format": "remark . --quiet --frail --output",
},
- In my case, I want an autoformat to be applied for each execution, we execute the scripts together (only the --output is applied for check and autoformat without terminating the process and being able to execute the serverless script)
"scripts": {
"check": "remark . --quiet --frail",
"format": "remark . --quiet --frail --output",
"format-md": "remark . --output",
"serverless-offline": "sls offline start",
"start": "npm run format-md && npm run serverless-offline"
},
- Then we add the remark configs, at the end, in the package.json
"remarkConfig": {
"settings": {
"emphasis": "*",
"strong": "*"
},
"plugins": [
"remark-preset-lint-consistent",
"remark-preset-lint-recommended",
"remark-lint",
"remark-lint-table-cell-padding",
[
"remark-lint-list-item-indent",
"tab size"
],
[
"remark-lint-emphasis-marker",
"*"
],
[
"remark-lint-strong-marker",
"*"
]
]
}
-
For more information about it, visit the official page
-
The ssm variables used in the project are maintained to simplify the project configuration process. It is recommended to add the corresponding file (serverless_ssm.yml) to the .gitignore.
-
The following script (start), configured in the project's package.json, is responsible for executing
- The serverless-offline plugin
- The remark-lint plugin for .md files
- Others
"scripts": {
"serverless-offline": "sls offline start",
"start": "npm run format-md && npm run serverless-offline",
"start:dev": "nodemon -e js,ts,yml,json --exec \"sls offline start\"",
"format-prettier": "prettier --write \"{src,test}/**/*.{js,ts}\"",
"check": "remark . --quiet --frail",
"format-remark": "remark . --quiet --frail --output",
"format-md": "remark . --output",
"test": "jest --verbose",
"test:watch": "jest --watch --verbose",
"test:cov": "jest --coverage --verbose"
},
- We run the app from terminal.
npm start
- If a message appears indicating that port 4000 is already in use, we can terminate all dependent processes and run the app again
npx kill-port 4000
npm start
1.3) Technologies π
See
| Technologies | Version | Purpose |
| ------------- | ------------- | ------------- |
| SDK | 4.3.2 | Automatic Module Injection for Lambdas |
| Serverless Framework Core v3 | 3.23.0 | Core Services AWS |
| Systems Manager Parameter Store (SSM) | 3.0 | Management of Environment Variables |
| Jest | 29.7 | Framework para pruebas unitarias, integraciΓ³n, etc. |
| Amazon Api Gateway | 2.0 | API Manager, Authentication, Control and Processing |
| NodeJS | 14.18.1 | js library |
| Sequelize | ^6.11.0 | ORM |
| Mysql | 10.1 | SGDB |
| XAMPP | 3.2.2 | Server package |
| VSC | 1.72.2 | IDE |
| Postman | 10.11 | http client |
| CMD | 10 | SΓmbolo del Sistema para linea de comandos |
| Git | 2.29.1 | Version control |
| Otros | Otros |
Plugin |
---|
Serverless Plugin |
serverless-offline |
serverless-offline-ssm |
| ExtensiΓ³n |
| ------------- |
| Prettier - Code formatter |
| YAML - Autoformatter .yml |
| Error Lens - for errors and indent |
| Tabnine - IA Code |
| Otros - Otros |
2.0) Endpoints and resources π
See
- http://localhost:4000/dev/v1/test
- http://localhost:4000/dev/v1/db-connection
- http://localhost:4000/dev/v1/users/list
- http://localhost:4000/dev/v1/users/id/{user-id}
- http://localhost:4000/dev/v1/users/country-id/{country-id}
- http://localhost:4000/dev/v1/users/email/{email}
- http://localhost:4000/dev/v1/users/first-name/{first-name}
- http://localhost:4000/dev/v1/users/identification-number/{ident-number}
- http://localhost:4000/dev/v1/users/identification-type/{ident-type}
- http://localhost:4000/dev/v1/users/last-name/{last-name}
- http://localhost:4000/dev/v1/users/nickname/{nickname}
- http://localhost:4000/dev/v1/users/creation-date/{creation-date}
- http://localhost:4000/dev/v1/users/update-date/{update-date}
All endpoints are optional paginated except /test, /db-connection and /id/{user-id}}
- {required-value}
- Default pagination: ?page=0&limit=5
- Optional pagination: ?page={nro}&limit={nro}
2.1) Examples π
See
| Variable | Initial value | Current value |
| ------------- | ------------- | ------------- |
| base_url | http://localhost:4000/dev/ | http://localhost:4000/dev/ |
| x-api-key | f98d8cd98h73s204e3456998ecl9427j | f98d8cd98h73s204e3456998ecl9427j |
| bearer_token | Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c | Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c |
curl --location 'http://localhost:4000/dev/v1/db-connection' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c' \
--header 'Content-Type: application/json' \
--header 'x-api-key: f98d8cd98h73s204e3456998ecl9427j' \
--data ''
{
"message": "Connection has been established successfully."
}
{
"message": "Bad request, check missing or malformed headers"
}
{
"message": "Bad request, could not get the paginated list of users."
}
{
"message": "Not authenticated, check x_api_key and Authorization"
}
{
"message": "Error in connection lambda. Caused by Error: throw a new error to check for the exception caught by lambda"
}
curl --location 'http://localhost:4000/dev/v1/users/list?page=0&limit=2' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c' \
--header 'Content-Type: application/json' \
--header 'x-api-key: f98d8cd98h73s204e3456998ecl9427j' \
--data ''
{
"message": [
{
"id": 3,
"nickname": "HECTOR SS G",
"first_name": "Hector",
"last_name": "Gomez",
"email": "hectorGomez78@gmail.com",
"identification_type": "DNI",
"identification_number": "2172265827",
"country_id": "AR",
"creation_date": "2023-03-20 21:02:33",
"update_date": "2023-03-20 21:02:33"
},
{
"id": 4,
"nickname": "GABRIELA JIMENEZ",
"first_name": "Gabriela",
"last_name": "Jimenez",
"email": "gabriela.consultas@hotmail.com",
"identification_type": "DNI",
"identification_number": "410871223",
"country_id": "AR",
"creation_date": "2023-03-20 21:02:33",
"update_date": "2023-03-20 21:02:33"
}
]
}
{
"message": "Bad request, check missing or malformed headers"
}
{
"message": "Bad request, could not get the paginated list of users."
}
{
"message": "Not authenticated, check x_api_key and Authorization"
}
{
"message": "ECONNREFUSED. An error has occurred with the connection or query to the database. Verify that it is active or available"
}
{
"message": "ERROR. An error has occurred in the process operations and queries with the database Caused by SequelizeConnectionRefusedError: connect ECONNREFUSED 127.0.0.1:3306."
}
{
"message": "Error in getAll lambda. Caused by Error: throw a new error to check for the exception caught by lambda"
}
curl --location 'http://localhost:4000/dev/v1/users/id/4' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c' \
--header 'Content-Type: application/json' \
--header 'x-api-key: f98d8cd98h73s204e3456998ecl9427j'
{
"message": {
"id": 4,
"nickname": "GABRIELA JIMENEZ",
"first_name": "Gabriela",
"last_name": "Jimenez",
"email": "gabriela.consultas@hotmail.com",
"identification_type": "DNI",
"identification_number": "410871223",
"country_id": "AR",
"creation_date": "2023-03-20 21:02:33",
"update_date": "2023-03-20 21:02:33"
}
}
{
"message": "Bad request, check missing or malformed headers"
}
{
"message": "Bad request, could not fetch user based on id."
}
{
"message": "Bad request, the id passed as a parameter is not valid."
}
{
"message": "Not authenticated, check x_api_key and Authorization"
}
{
"message": "ECONNREFUSED. An error has occurred with the connection or query to the database. Verify that it is active or available"
}
{
"message": "ERROR. An error has occurred in the process operations and queries with the database Caused by SequelizeConnectionRefusedError: connect ECONNREFUSED 127.0.0.1:3306."
}
{
"message": "Error in getById lambda. Caused by Error: throw a new error to check for the exception caught by lambda"
}
curl --location 'http://localhost:4000/dev/v1/users/country-id/AR?page=0&limit=3' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c' \
--header 'Content-Type: application/json' \
--header 'x-api-key: f98d8cd98h73s204e3456998ecl9427j' \
--data ''
{
"message": [
{
"id": 3,
"nickname": "HECTOR SS G",
"first_name": "Hector",
"last_name": "Gomez",
"email": "hectorGomez78@gmail.com",
"identification_type": "DNI",
"identification_number": "2172265827",
"country_id": "AR",
"creation_date": "2023-03-20 21:02:33",
"update_date": "2023-03-20 21:02:33"
},
{
"id": 4,
"nickname": "GABRIELA JIMENEZ",
"first_name": "Gabriela",
"last_name": "Jimenez",
"email": "gabriela.consultas@hotmail.com",
"identification_type": "DNI",
"identification_number": "410871223",
"country_id": "AR",
"creation_date": "2023-03-20 21:02:33",
"update_date": "2023-03-20 21:02:33"
},
{
"id": 5,
"nickname": "GUSTA G K",
"first_name": "Gustavo",
"last_name": "Gomez",
"email": "gustavo_andaluz@gmail.com",
"identification_type": "PASAPORTE",
"identification_number": "748000221",
"country_id": "AR",
"creation_date": "2023-03-20 21:02:33",
"update_date": "2023-03-20 21:02:33"
}
]
}
{
"message": "Bad request, check missing or malformed headers"
}
{
"message": "Bad request, could not get paginated list of users according to country id. Try again."
}
{
"message": "Bad request, the country id passed as a parameter is not valid."
}
{
"message": "Not authenticated, check x_api_key and Authorization"
}
{
"message": "ECONNREFUSED. An error has occurred with the connection or query to the database. Verify that it is active or available"
}
{
"message": "ERROR. An error has occurred in the process operations and queries with the database Caused by SequelizeConnectionRefusedError: connect ECONNREFUSED 127.0.0.1:3306."
}
{
"message": "Error in getLikeCountryId lambda. Caused by Error: throw a new error to check for the exception caught by lambda"
}
- OTHER GET OPERATIONS (SEE POSTMAN COLLECTION)
curl --location 'http://localhost:4000/dev/v1/users/add-user/' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c' \
--header 'Content-Type: application/json' \
--header 'x-api-key: f98d8cd98h73s204e3456998ecl9427j' \
--data-raw ' {
"nickname": "VALE18BNX",
"first_name": "Valeria",
"last_name": "Castro",
"email": "vale_18_nnbs@gmail.com",
"identification_type": "DNI",
"identification_number": "3987261233",
"country_id": "AR12"
}'
{
"message": {
"id": null,
"nickname": "VALE18BNX",
"first_name": "Valeria",
"last_name": "Castro",
"email": "vale_18_nnbs@gmail.com",
"identification_type": "DNI",
"identification_number": "3987261233",
"country_id": "AR12",
"creation_date": "2023-06-28T16:46:31.000Z",
"update_date": "2023-06-28T16:46:31.000Z"
}
}
{
"message": "Bad request, check missing or malformed headers"
}
{
"message": "Bad request, check request attributes. Missing or incorrect. CHECK: nickname, first_name and last_name (required|string|minLength:4|maxLength:50), email (required|string|minLength:10|maxLength:100), identification_type and identification_number (required|string|minLength:6|maxLength:20), country_id (required|string|minLength:2|maxLength:5)"
}
{
"message": "Bad request, could not add user.CHECK: The first_name next together the last_name should be uniques. The identification_type next together the identification_number should be uniques."
}
{
"message": "Not authenticated, check x_api_key and Authorization"
}
{
"message": "ECONNREFUSED. An error has occurred with the connection or query to the database. CHECK: The first_name next together the last_name should be uniques. The identification_type next together the identification_number should be uniques."
}
{
"message": "ERROR. An error has occurred in the process operations and queries with the database Caused by SequelizeConnectionRefusedError: connect ECONNREFUSED 127.0.0.1:3306."
}
{
"message": "Error in addUser lambda. Caused by Error: throw a new error to check for the exception caught by lambda"
}
curl --location --request PUT 'http://localhost:4000/dev/v1/users/update-user/26' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c' \
--header 'Content-Type: application/json' \
--header 'x-api-key: f98d8cd98h73s204e3456998ecl9427j' \
--data-raw ' {
"nickname": "VALE18BNX EDITED",
"first_name": "Valeria EDITED",
"last_name": "Castro",
"email": "vale_18_nnbs@gmail.com",
"identification_type": "DNI",
"identification_number": "3987261233",
"country_id": "AR12",
"creation_date": "2023-06-28 16:46:31",
"update_date": "2023-06-28 16:46:31"
}'
{
"message": {
"id": 26,
"nickname": "VALE18BNX EDITED",
"first_name": "Valeria EDITED",
"last_name": "Castro",
"email": "vale_18_nnbs@gmail.com",
"identification_type": "DNI",
"identification_number": "3987261233",
"country_id": "AR12",
"creation_date": "2023-06-28 19:46:31",
"update_date": "2023-06-28 16:53:17"
}
}
{
"message": "Bad request, check missing or malformed headers"
}
{
"message": "Bad request, check request attributes and object to update"
}
{
"message": "Bad request, could not add user.CHECK: The first_name next together the last_name should be uniques. The identification_type next together the identification_number should be uniques."
}
{
"message": "Not authenticated, check x_api_key and Authorization"
}
{
"message": "ECONNREFUSED. An error has occurred with the connection or query to the database. CHECK: The first_name next together the last_name should be uniques. The identification_type next together the identification_number should be uniques."
}
{
"message": "ERROR. An error has occurred in the process operations and queries with the database Caused by SequelizeConnectionRefusedError: connect ECONNREFUSED 127.0.0.1:3306."
}
{
"message": "Error in updateUser lambda. Caused by Error: throw a new error to check for the exception caught by lambda"
}
curl --location --request DELETE 'http://localhost:4000/dev/v1/users/delete-user/26' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c' \
--header 'Content-Type: application/json' \
--header 'x-api-key: f98d8cd98h73s204e3456998ecl9427j' \
--data ''
{
"message": "User has been deleted successfully."
}
{
"message": "Bad request, check missing or malformed headers"
}
{
"message": "Bad request, a non-existent user cannot be deleted. Operation not allowed"
}
{
"message": "Not authenticated, check x_api_key and Authorization"
}
{
"message": "ECONNREFUSED. An error has occurred with the connection or query to the database. CHECK: The first_name next together the last_name should be uniques. The identification_type next together the identification_number should be uniques."
}
{
"message": "ERROR. An error has occurred in the process operations and queries with the database Caused by SequelizeConnectionRefusedError: connect ECONNREFUSED 127.0.0.1:3306."
}
{
"message": "Error in deleteUser lambda. Caused by Error: throw a new error to check for the exception caught by lambda"
}