MongoDB

Overview

MongoDB is a database which stores data in a “collection” in a user-friendly manner. MongoDB is a collection of actual documents instead of storing the data in the relational data formats that are generally in the form of rows and columns. It stores the data in the BSON format i.e. in the form of field value pairs.

To get a brief overview of MongoDB, you can go through the official MongoDB documentation.

Setting up MongoDB

To establish a connection between Nected and MongoDB for seamless data interaction, you need to configure a mongodb connector. This section outlines the step-by-step process for setting up the connector effectively.

1. Add Integration

As you move to the Integration Page of Nected.io, over there you need to select MongoDB from the list of available database connectors.

2. Configure Settings

As soon as you select the MongoDB Integration, You need to enter the following environment details then after you can test your connection in staging :-

• Environment Type - There are two available options, first is staging (which is used for testing and quality assurance, it is an environment where developers can test their connectors and make new changes in the environment if required. The second one is the Production environment where the fully-tested and assured software is made available to users.

• Connector Name - Enter the name of the connector of your choice, but it should be unique and should not contain any white spaces.

• Protocol - You need to choose one of the protocol among the given two protocols i.e. MongoDb and [MongoDb+Srv](https://www.mongodb.com/basics/mongodb-connection-string#:~:text=What is a MongoDB SRV,shorter and easier to use.). To get the exact difference between them you can read MongoDB Vs MongoDB+Srv.

• Host (Database IP address) - Enter the Host IP address of the database, as mentioned in the database specifications.

• Port Number - Enter the Port number on which connection can communicate with the database.

• Username - Enter the Username of the user who is authorised to connect to the database and have read/write privileges on the database.

• Password (encrypted) - Enter the password for the authorised user to connect to the database. Passwords are encrypted to ensure maximum security of your database.

• Database - Enter the name of the MongoDB database from which Nected can access the data and push data to it.

• Expand to see the IP addresses to add to your allow-list – Nected service IP address (43.205.43.45) added to the allow-list on your database server.

3. Test connection

Test the connection to your database after entering all the required information. Testing the connections is mandatory as it makes sure that the provided information is accurate and the connection is established successfully without any issues.

As soon as you test your connections successfully, there will be a message on you screen showing a message “Tested Successfully”, it indicates a successful connection establishment

4. Publish in Staging

After a successful connection to the database, publish the connector in staging to set up Rules in a Staging environment. In this example, the database type is staging, and it will provide the Publish in Staging option. For a production database, the option will be Publish in Production.

As soon as you publish your data in staging, your database will move in the staging environment and a “staging” tag gets associated with you database.

5. Connector Status

After entering all the MongoDB database credentials and publishing the connector, it's important to check the connector status. For a "Staging" environment, the status will be "staging," and for a "Production" environment, it will be "Production." Monitoring the status ensures that the connector is active and properly integrated into the selected environment.

Querying MongoDB Database via DataSet

Querying mongodb database refers to a process in which we filter the data in a manner such that at last we are remaining with the required data by using specific action i.e. querying

A Step by step guide to perform querying on mongo Database via Dataset :

Adding a new Dataset:

Before you can query the Mongo database, you need to add a new dataset to your Nected environment. Follow these steps:

1. Navigate to Datasets: Click on "Datasets" in the left navigation panel to access the Datasets page.

2. Create a Dataset: Click the "+ Create Dataset" button on the Datasets page. Choose the database connector associated with the dataset you wish to create. Note that you can create multiple datasets for one connector.

3. Dataset Information: Complete the dataset information form, including details such as the dataset name, dataset type (staging or production), the source for dataset parameters.

4. Finally, write the specific database query to retrieve data from your connected database. For example, to connect the complete posts attribute of database within your dataset, write

db.posts.find( { } ); // this will find all the entries under the posts section of the database.

Breakdown of the above query:

1 “db”: indicates the context of the current database (in the mongo shell). 2. “posts”: is the name of the attribute which is specified under your database you want to query. 3. ”find( {} )”: is the query that retrieves all the information specified the "posts".

Note: As soon as you perform query on the dataset, the output result will be limited to top 50 matching records only, as it is more than sufficient for testing a database connection.

Performing different operation using Mongo queries:

1. Querying Database:

find(): Using this, one can easily retrieve the required information from a set of data that matches the specified criteria. For instance to search a person in the database whose “id” is 12, you will write:

db.posts.find({"_id": 12});

2. Aggregation:

aggregate(): Using this, one can easily perform data aggregation and transformation operations. For instance, to calculate the total deposit amount for a specific day:

db.deposits.aggregate([
  {
    "$match": {
      "date": ISODate("2023-10-27") // Replace the date with your specific date
    }
  },
  {
    "$group": {
      "_id": null,
      "totalDeposits": { "$sum": "$amount" } // amount is getting aggre
    }
  }
])

1. Filtering:

filtering(): Using this, one can easily filter out the required data from the given dataset. For instance we are filtering to find all transactions above a certain amount (say 1000), you can use:

db.transactions.find({ "amount": { "$gt": <1000> } })  // You need to put your own threshold amount

1. Sorting:

Sorting(): Using this, one can easily sort the selected data using a particular specified condition. For instance we are sorting the transactions by amount in descending order:

db.transactions.find().sort({ "amount": -1 })

To perform the sorting in the ascending order:

db.transactions.find().sort({"amount": 1})

Supported & Non-Supported MongoDB queires:

In Nected, MongoDB queries offer a robust platform for data management. Supported queries for creating a dataset encompass a wide range of find, sort, projection, aggregation, and even extended JSON queries. However, write queries such as insert, update, and delete are not supported when creating a dataset. For actions, Nected supports insert and update queries but does not support delete queries. This combination of supported and non-supported MongoDB queries ensures efficient data handling within the platform.

Supported Queries for Creating a Dataset

Supported Queries for Creating a Dataset in MongoDB: Nected enables a wide array of queries, including find, sort, projection, aggregation, and extended JSON queries, for efficient dataset creation in MongoDB.

Query Type	Description	Example
Find Queries	Retrieve documents based on specific criteria, with filtering and sorting options.	db.customers.find({ "country": "USA" });
Sort Queries	Organize query results with sorting based on specific fields.	db.sales.find().sort({ "orderDate": -1 });
Projection Queries	Specify fields to include or exclude in query results for optimized data retrieval.	db.users.find({}, { "name": 1, "email": 1, "_id": 0 });
Aggregate Queries	Utilize MongoDB's aggregation pipelines for complex data transformations and analysis.	db.sales.aggregate([{ $group: { "_id": "$customerName", "totalSales": { "$sum": "$amount" } } }]);
Extended JSON Queries	Perform advanced queries using extended JSON syntax for enhanced data filtering and manipulation.	db.products.find({ "price": { "$gt": 50 } });

Non-Supported Queries for Creating a Dataset:

Non-Supported Queries for Creating a Dataset in MongoDB: Delete queries and other write operations are not supported when creating a dataset in Nected for MongoDB.

Query Type	Description	Example
Write Queries	Insert, update, or delete documents are not supported when creating a dataset.	db.products.insert({ "name": "New Product", "price": 50 });
Aggregation Pipeline Updates	Modifying documents using aggregation pipelines during dataset creation is not supported.	db.customers.updateMany({}, [{ $set: { newField: "value" } }]);
Index Creation	Creating or managing indexes on collections is not supported during dataset creation.	db.products.createIndex({ name: 1 });
Extended JSON Updates	Using extended JSON for document updates is not supported during dataset creation.	db.sales.update({ "totalAmount": { $gt: 100 } }, { $set: { "status": "Approved" } });

Trigger MongoDB operations as Rule Actions:

In Nected, rule actions provide a way to automate Mongo operations when specific conditions are met. While datasets are typically used for read-only queries, rule actions allow you to modify the dataset and trigger specific operations within your MongoDB database. Below is a short presentation of how you can perform it using rule action.

Now we will explore some of the Mongo queries using which we can be used as rule actions to write or edit data.

1. Inserting New Financial Transactions:

Depending on the need of the user, ‘insertOne()’ can be used to used to insert one single transaction whereas to insert multiple transactions, one can use ‘insertMany()’.

For instance, we are inserting a single transaction in a collection named “transactions” specifying a 1000 rupees payment is received.

db.transactions.insertOne({
  "description": "Payment received",
  "amount": 1000.00,
  "date": ISODate("2023-10-27T08:00:00Z"),
  // Other transaction fields
})

To insert multiple transactions at a same time we will use insertMany(). For instance we are inserting a couple of transactions in a collection named “transactions” in which the first transaction is specifying a received payment of 1000 rupees while another one is specifying an expense of 500 rupees. You can add more transactions objects similarly as according to the need of the user.

db.transactions.insertMany([
  {
    "description": "Payment received",
    "amount": 1000.00,
    "date": ISODate("2023-10-27T08:00:00Z"),
    // Other transaction fields
  },
  {
    "description": "Expense payment",
    "amount": -500.00,
    "date": ISODate("2023-10-27T14:30:00Z"),
    // Other transaction fields
  },
  // Add more transaction objects as needed
])

1. Updating Transaction Status:

Depending on the need of the user, ‘updateOne()’ can be used to used to update one single transaction whereas to update multiple transactions, one can use ‘updateMany()’.

For instance, we are updating a single transaction specifying a100 rupees payment is received

db.transactions.updateOne(
  { "_id": ObjectId("transaction_id_to_update") }, // Replace with the actual transaction _id
  {
    "$set": {
      "status": "new_status" // Replace with the new status you want to set
    }
  }
)

To update multiple transactions at a same time we will use updateMany(). For instance we are updating a category specified as groceries under this category we are updating all the pending status to completed as:

db.transactions.updateMany(
  { "category": "Groceries", "status": "Pending" },
  {
    "$set": {
      "status": "Completed"
    }
  }
)

1. Deleting Transactions:

Depending on the need of the user, ‘deleteOne()’ can be used to delete one single transaction whereas to delete multiple transactions, one can use ‘deleteMany()’.

To delete a single transaction from a collection named “transactions” we need to specify the transaction id of that particular transaction within the query as:

db.transactions.deleteOne({ "_id": ObjectId("transaction_id_to_delete") })

To delete multiple transaction we need to specify the exact category in which all that transactions are placed we want to delete.

db.transactions.deleteMany({ "category": "Obsolete" })

As soon as the query gets executed, it deletes all transactions with a "category" defined as "Obsolete.”

These mongo queries demonstrate how you can use rule actions to perform data modification operations. Rule actions offer flexibility in automating data updates, ensuring data accuracy, and enhancing the functionality of your database.

Supported Queries for Writing an Action:

Supported Queries for Writing an Action in MongoDB: Nected supports insert and update queries for actions in MongoDB, allowing for data modification.

Query Type	Description	Example
Insert Queries	Add new documents to a collection as part of an action.	db.orders.insertOne({ "order_id": 789, "customer_id": 123, "order_date": new Date("2023-11-15") });
Update Queries	Modify existing documents in a collection as part of an action.	db.customers.updateOne({ "customer_id": 123 }, { "$set": { "email": "[e-mail]" } });

Non-Supported Queries for Writing an Action:

Non-Supported Queries for Writing an Action in MongoDB: Delete queries are not supported as part of an action in Nected for MongoDB.

Query Type	Description	Example
Delete Queries	Removing documents from a collection using delete queries is not supported as part of an action.	db.products.deleteOne({ "price": { $lt: 10 } });
Aggregation Pipeline Updates	Modifying documents using aggregation pipelines as part of an action is not supported.	db.customers.updateMany({}, [{ "$set": { "newField": "value" } }]);
Index Creation	Creating or managing indexes on collections is not supported as part of an action.	db.products.createIndex({ "name": 1 });
Extended JSON Updates	Using extended JSON for document updates is not supported as part of an action.	db.sales.update({ "totalAmount": { $gt: 100 } }, { "$set": { "status": "Approved" } });

Troubleshooting

Error can be from two ends:

1. Client side error

2. Server side error

Client Side error

1. HTTP Error:

• Error code 502:

Cause: Misconfigurations, network interruptions, gateway or proxy server problems, and unavailable web servers are the most common causes of "502 Bad Gateway" errors.

Solution: Refreshing the website, looking for proxy server or CDN difficulties, making sure the web server is available, reviewing proxy server setup, looking at local firewall settings, and thinking about DNS or server load issues are some ways to fix it.

• Error code 503:

Cause: A "503 Service Unavailable" error appears when a web server is momentarily unable to process the request; this is frequently the result of software problems, maintenance, or server overload.

Solution: To restore service, wait for the server to become available, monitor for maintenance alerts, maximize server resources, and fix any software or configuration issues.

• Error code 504:

Cause: A "504 Gateway Timeout" error occurs when a proxy server or gateway does not receive a timely response from the upstream server. This is frequently the result of a server that is overloaded or performing slowly.

Solution: Resolve the problem by either reducing server load, checking for network difficulties, extending proxy server timeout settings, or optimizing the server for better response times.

2. IP whitelisting error:

• Access Denied: This error message denotes an attempt to access a system or resource by an IP address that is not on the whitelist.

• Unauthorized Access: This error message suggests that the IP address attempting to access the resource or service is not permitted.

• Firewall error: Errors related to unauthorized IP addresses may be referred to as security errors when IP whitelisting is primarily utilized for security reasons.

3. Database doesn’t exist:

Usually, when you try to access or operate on a database that the system or application cannot locate, you get a "database does not exist" error. Depending on the database management system or program you are using, this error message may differ somewhat. Here are a few typical error messages along with possible explanations:

Server Side error

1. Checksum mismatch

At a particular time, single user is allowed to perform update operation and as soon as he finished then after another user can access the updated set and can perform update as according to his need.

Solution: Until a user is signed in with that account and performing update operation, other users should not access the same resource for performing the update operation.

2. Unique constant combination

The combination of entity name and workspace ID must be unique for every new entity created, every time a user will try to access the service with the already created user name it will produce an error.

Solution: The user name of every new entity must be unique in order to overcome the internal database clashes.

3. Internal Server error

whenever the rule execution server in the backend service is not responding there comes an error called as internal server error.

Solution: To restore service, wait for the server to become available, monitor for maintenance alerts.

4. The connector is being used in database

If the user will try to delete the database connections which are already in use, it faces an error showing “the connector is being used in the database, deleting it will break the execution, hence it cannot be deleted”.

Solution: Never try to delete the database connectors which are already in use.

5. Database not showing in create datasets

Connection with the database should be tested and published, if the connection is not published, user cannot access it in create dataset section section for further manipulation.

Solution: Always publish your connection just after you perform a successful testing.