603 better oauth datasources

[NaderRNA]

Ability to connect to datasources (airbyte) using OAuth login page

Uses passport.js to authenticate the OAuth, gets the refreshToken (which is the only thing airbyte needs to establish a connection to a source with OAuth).

Aims to keep current datasource flow as unchanged as possible.

We only have client ID and client Secrets for the following which will be the only ones supported:
(crossed out oauth providers are implemented)

• Hubspot
• Salesforce (forcedotcom)
• Airtable
• Google Ads (not sure if we have an approved clientId and secret)
• Xero
• Slack

[github-actions]

File	Coverage
All files	96%	✅
src/lib/utils/validationutils.ts	95%	✅

Minimum allowed coverage is 80%

Generated by 🐒 cobertura-action against b4f9ea2

[github-advanced-security]

ESLint found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

To fix the problem, we need to introduce rate limiting to the routes that use the unauthedMiddlewareChain. The best way to do this is by using the express-rate-limit package, which allows us to easily set up rate limiting middleware. We will configure a rate limiter to allow a maximum of 100 requests per 15 minutes and apply it to the relevant routes.

1. Install the express-rate-limit package.
2. Import the express-rate-limit package in the webapp/src/router.ts file.
3. Configure the rate limiter with appropriate settings.
4. Apply the rate limiter to the routes that use the unauthedMiddlewareChain.

[NaderRNA]

Current status of the code:

1. Successfully authenticates the user with specifically HubSpot currently.
2. Passes the refreshToken back through the callback function (OAuth controller)
3. Callback redirects back to datasource/add page with information required to set up datasource
4. CreateDatasourceForm makes a POST with the sourceConfig set to the required config for the given oauth connector
5. That then creates the TestDatasource object using the OAuth credentials
6. Continue datasource creation as standard using the existing flow

The next few commits will likely be refactoring the implementation to do the following in order:

1. pass the datasource name and datasource description through the flow
2. Provide the user with better feedback indicating what's happening (adding a new loader for OAuth to say that authentication was successful and Agent Cloud is testing the datasource)
3. Make the entire flow more compatible for other oauth providers, not just hubspot

To fix the problem, we need to introduce rate limiting to the routes that use the useJWT middleware. The best way to achieve this is by using the express-rate-limit package, which allows us to easily set up and apply rate limiting to specific routes or the entire application.

1. Install the express-rate-limit package.
2. Import the express-rate-limit package in the webapp/src/router.ts file.
3. Set up a rate limiter with appropriate configuration (e.g., maximum of 100 requests per 15 minutes).
4. Apply the rate limiter to the routes that use the useJWT middleware.

To fix the problem, we need to introduce rate limiting to the route handlers that perform authorization. The best way to do this is by using the express-rate-limit package, which allows us to easily set up rate limiting middleware. We will configure the rate limiter to allow a maximum of 100 requests per 15 minutes and apply it to the relevant routes.

1. Install the express-rate-limit package.
2. Import the express-rate-limit package in the webapp/src/router.ts file.
3. Set up the rate limiter with the desired configuration.
4. Apply the rate limiter to the relevant routes.

To fix the problem, we need to introduce rate limiting to the routes that use the useJWT middleware. The best way to do this is by using the express-rate-limit package, which allows us to easily set up rate limiting for our Express routes. We will create a rate limiter and apply it to the relevant routes to ensure that they are protected against excessive requests.

1. Install the express-rate-limit package if it is not already installed.
2. Import the express-rate-limit package in the webapp/src/router.ts file.
3. Create a rate limiter with appropriate settings (e.g., maximum of 100 requests per 15 minutes).
4. Apply the rate limiter to the routes that use the useJWT middleware.

[NaderRNA]

Somewhere along the line here a bug was introduced that replaced the file upload/drop area with the connectors dropdown

[ragyabraham]

@NaderRNA what's happening with this PR?? where are we with this??

[NaderRNA]

Currently the branch supports custom OAuth implementation for any OAuth provider where we have a Client ID and Client Secret for the oauth. Currently HubSpot is completley functional using OAuth but there are some issues with out Airtable, Salesforce and Xero app credentials so I'm just trying to iron those out with Andrew.
If you'd like a version that can go to prod alongside BYO VectorDBs I can modify this code so it won't break prod and will work for HubSpot.
But currently if you know the OAuth provider endpoint and required scopes you can create any custom OAuth implementation using this implementation of the code, nothing is hardcoded so introduction of new OAuth implementation is a lot easier for both account creation and datasource auth.

[NaderRNA]

reverting back because Airtable was creating more problems as I was implementing it. Using custom strategies on passport seems to be broken with our current versions of passport.js. To use airtable or anything not supported by passport.js, a custom implementation is required that skips passport implementation and is completely end to end handled by the webapp

[ragyabraham]

@iandjx where are we at with this.... this would be a really good addition