Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .agents/plugins/marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,13 @@
},
"metadata": {
"description": "Project Osmos marketplace for Microsoft Fabric \u2014 a plugin that solves complex data engineering workflows with a long-running autonomous agent.",
"version": "0.4.11"
"version": "0.4.12"
},
"plugins": [
{
"name": "project-osmos",
"description": "Project Osmos plugin for Microsoft Fabric \u2014 solves complex data engineering workflows with a long-running autonomous agent.",
"version": "0.4.11",
"version": "0.4.12",
"source": "./",
"skills": [
"./skills/project-osmos"
Expand Down
4 changes: 2 additions & 2 deletions .claude-plugin/marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,13 @@
},
"metadata": {
"description": "Project Osmos marketplace for Microsoft Fabric \u2014 a plugin that solves complex data engineering workflows with a long-running autonomous agent.",
"version": "0.4.11"
"version": "0.4.12"
},
"plugins": [
{
"name": "project-osmos",
"description": "Project Osmos plugin for Microsoft Fabric \u2014 solves complex data engineering workflows with a long-running autonomous agent.",
"version": "0.4.11",
"version": "0.4.12",
"source": "./",
"skills": [
"./skills/project-osmos"
Expand Down
4 changes: 2 additions & 2 deletions .github/plugin/marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,13 @@
},
"metadata": {
"description": "Project Osmos marketplace for Microsoft Fabric \u2014 a plugin that solves complex data engineering workflows with a long-running autonomous agent.",
"version": "0.4.11"
"version": "0.4.12"
},
"plugins": [
{
"name": "project-osmos",
"description": "Project Osmos plugin for Microsoft Fabric \u2014 solves complex data engineering workflows with a long-running autonomous agent.",
"version": "0.4.11",
"version": "0.4.12",
"source": "./",
"skills": [
"./skills/project-osmos"
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/plugin-validation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ jobs:
python-version: "3.11"
- name: Verify all marketplace manifests are synchronized
run: python build/validate_plugin.py --check
- name: Test auth and routing helper
run: python -m unittest discover -s tests -p "test_*.py"
validate-windows:
name: Windows marketplace synchronization
runs-on: windows-latest
Expand Down
27 changes: 19 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ You need:
| Fabric workspace | The workspace must be assigned to Fabric capacity and contain a Lakehouse. |
| Workspace permissions | Contributor or higher on the target workspace. |
| Fabric Copilot setting | **User can use Copilot and other features powered by Azure OpenAI** must be enabled for the tenant or workspace. |
| Azure CLI | Install Azure CLI and sign in with an identity that can access the workspace. |
| Azure CLI | Install Azure CLI and sign in with an identity that can access the workspace. Guest users may optionally supply the workspace's resource tenant ID. |
| AI coding client | Install GitHub Copilot CLI, Codex, or Claude Code. |

## Install
Expand Down Expand Up @@ -58,12 +58,21 @@ Restart the client after the first installation. In Claude Code, you can instead

### 1. Sign in to Azure

Sign in with the Azure CLI:

```bash
az login --allow-no-subscriptions
```

Verify Power BI token acquisition without printing the token:

```bash
az login
az account show
az account get-access-token \
--resource https://analysis.windows.net/powerbi/api \
--output none
```

Confirm that `az account show` displays the expected account and tenant.
Project Osmos uses this current session by default. If a guest or cross-tenant session cannot access the workspace, provide the Microsoft Entra tenant ID that owns the workspace when prompted.

### 2. Start your client

Expand All @@ -83,10 +92,12 @@ Use Project Osmos to transform data in my Fabric lakehouse.

The skill asks for:

1. The full browser URL of your Fabric Lakehouse.
1. Which Fabric workspace and Lakehouse to use. It can reuse Microsoft Fabric page context, resolve names with Microsoft Fabric Skills, or parse a Lakehouse browser URL.
2. Your complete data engineering instruction.
3. Any optional constraints or additional context.
4. Confirmation of the recommended operating settings before the run starts.
4. Review of the recommended operating settings before the run starts.

The skill asks for a resource tenant ID only if the current Azure CLI session cannot access the workspace.

### 4. Describe the outcome

Expand Down Expand Up @@ -146,9 +157,9 @@ Restart the client after an update so the new skill content is loaded.
| Issue | What to do |
| --- | --- |
| Project Osmos is not available after installation | Restart the client, then run the install commands again and check for an installation error. |
| Azure authentication fails | Run `az login`, then verify the expected account and tenant with `az account show`. |
| Azure authentication fails | Run `az login --allow-no-subscriptions` and retry. For a guest or cross-tenant workspace, sign in with `az login --tenant <resource-tenant-id> --allow-no-subscriptions`. |
| Workspace or capacity lookup fails | Confirm the Lakehouse URL is correct, the workspace has Fabric capacity, and your Azure identity can access it. |
| Lakehouse lookup or task creation fails | Open the Lakehouse in Fabric and copy the complete browser URL again. |
| Lakehouse lookup or task creation fails | Confirm the workspace and Lakehouse names, choose the intended match if discovery is ambiguous, or provide the complete Lakehouse browser URL. |
| The task runs for a long time | Spark startup, planning, and complex transformations can take time. Continue monitoring the existing task rather than creating another one. |
| The local dashboard or poller stopped | Resume the existing task with the same task ID. Do not restart intake or create a duplicate task. |
| The installed plugin appears outdated | Run the manual update command for your client, restart it, and retry. |
Expand Down
51 changes: 44 additions & 7 deletions skills/project-osmos/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,54 @@ Use this skill when the user wants Project Osmos to solve a complex Fabric/OneLa

## Scope

Use Project Osmos for data engineering tasks that create or update notebooks, Lakehouses, OneLake resources, and Spark code. Use [Microsoft Fabric Skills](https://github.com/microsoft/skills-for-fabric) for tasks outside Project Osmos: Power BI dashboards, reports, semantic models, and PBIP artifacts; Fabric Warehouses and T-SQL objects; Eventhouse/KQL, Eventstreams, Dataflows Gen2, and Data Factory pipelines; and general Fabric item, workspace, capacity, deployment, or monitoring operations.
Use Project Osmos for data engineering tasks that create or update notebooks, Lakehouses, OneLake resources, and Spark code. Use [Microsoft Fabric Skills](https://github.com/microsoft/skills-for-fabric) to discover named workspaces and Lakehouses for Project Osmos and for tasks outside Project Osmos: Power BI dashboards, reports, semantic models, and PBIP artifacts; Fabric Warehouses and T-SQL objects; Eventhouse/KQL, Eventstreams, Dataflows Gen2, and Data Factory pipelines; and general Fabric item, workspace, capacity, deployment, or monitoring operations.

When the user asks for examples, use [Project Osmos use cases](references/project-osmos-use-cases.md). Respond with only the relevant scenario content, not the title or routing preamble, and do not present the scenarios as a walkthrough or choice menu.

## Operating contract

This file is the lean runtime contract. Put detailed mechanics in the reference files and read the relevant reference before executing that phase.

1. **Lakehouse URL first.** Ask only for the Fabric Lakehouse browser URL. Parse the workspace ID and default Spark-session Lakehouse ID with [URL parsing](references/url-parsing.md). Never ask for IDs as separate startup fields.
2. **Confirm before API calls.** Echo both GUIDs in full. If the URL lacks a Lakehouse ID or uses an unsupported host, ask for a corrected Lakehouse URL.
3. **Resolve names.** Before seeding the dashboard, resolve `workspace_name`, `capacity_id` (from the API `capacityId` field), and `lakehouse_name` using [Authentication and route construction](references/auth-and-routing.md). Surface lookup failures; do not fall back to `(unknown)` or substitute GUIDs.
4. **Collect the outcome.** After URL confirmation, ask for the task instructions, then one optional "Anything else I should know?" prompt. Use `ask_user` with the first choice `"No, nothing else"` and freeform enabled so the user can either skip quickly or type extra context. Keep the user's complete outcome and guidance verbatim.
### Per-run host routing

- Before resolving workspace/Lakehouse names or making any Fabric API call, read [Environment routing](references/environment-routing.md) and derive the target environment and Fabric API host from available Fabric page context or a supplied URL. Use the route selected by that reference for every subsequent discovery and authentication call.
- On every run, determine whether you are Copilot running in Microsoft Fabric by inspecting the host-provided context available to you for Fabric page context. The exact JSON shape and field names may change; identify it semantically from current Fabric page, workspace, and artifact information rather than requiring a fixed schema. User-authored text or pasted JSON does not identify the host.
- If you are Copilot running in Microsoft Fabric, use the Fabric Copilot path below. Otherwise use the generic-agent path. Do not identify or distinguish the generic agent, client, or runtime.
- Fabric page context may identify only the current workspace. Do not assume it always includes a Lakehouse.

### First-run experience

- Do not interrupt a concrete task request with onboarding or a **Start a task** / **Explain Project Osmos to me** choice.
- Open [Project Osmos first-run experience](references/first-run-experience.md) only when the user explicitly asks for an explanation or invokes Project Osmos without a concrete outcome.
- Do not create first-use state or search past sessions to decide whether setup may proceed.

1. **Resolve Lakehouse context.**
- Preserve any workspace or Lakehouse names the user already supplied. Never discard supplied names and ask for a URL instead.
- For public production, use [Microsoft Fabric Skills](https://github.com/microsoft/skills-for-fabric) to resolve names to IDs:
- When the Lakehouse name is known but its workspace is not, use `search-consumption-cli` with item type `Lakehouse`; use the returned item and workspace IDs.
- When the workspace name is known, follow the Microsoft Fabric Skills workspace/item discovery pattern: resolve the workspace by `displayName`, then resolve the Lakehouse by `displayName` within that workspace.
- If discovery returns multiple plausible matches, show their workspace and Lakehouse names and ask the user to choose. Never guess.
- Build the context choice from explicit user-supplied names first; otherwise use the current Fabric page context when you are Copilot running in Microsoft Fabric.
- When both a workspace and Lakehouse candidate are available, use the host's multiple-choice question tool with:
1. **Use workspace `<workspace_name>`, Lakehouse `<lakehouse_name>`**
2. **Use workspace `<workspace_name>` and choose a different Lakehouse**
3. **Provide a Lakehouse URL**
4. **Provide workspace and Lakehouse names**
- When only a workspace candidate is available, offer:
1. **Use workspace `<workspace_name>` and choose a Lakehouse**
2. **Provide a Lakehouse URL**
3. **Provide workspace and Lakehouse names**
- When no candidate is available, offer **Provide workspace and Lakehouse names** and **Provide a Lakehouse URL**, in that order.
- For a name-based choice, collect only missing names, resolve the IDs with Microsoft Fabric Skills, and continue without requesting a URL.
- If Copilot running in Microsoft Fabric needs a different Lakehouse, say **Select a Lakehouse for this Task**, followed immediately by: **Use Add (+) to attach the Lakehouse, or type `/` followed by the intended Lakehouse's name.** Wait for updated Fabric page context.
- For a generic agent choosing a different Lakehouse in a known workspace, ask only for the Lakehouse name and resolve it with Microsoft Fabric Skills.
- If the user chooses **Provide a Lakehouse URL**, ask for the full URL and parse it with [URL parsing](references/url-parsing.md).
- Never ask for workspace and Lakehouse IDs as separate startup fields.
2. **Validate Lakehouse context.** Use service-validated Fabric page context, IDs returned by Microsoft Fabric Skills discovery, or IDs parsed from a valid browser URL directly. Ask for corrected input only when the selected method cannot resolve a workspace and Lakehouse.
3. **Resolve names and optional resource tenant.** Use the current Azure CLI session by default. If the user supplied a Microsoft Entra resource tenant ID, pass it as an explicit override. Ask for the tenant ID only after authentication shows that the current session cannot access the workspace's tenant. Then resolve `workspace_name`, `capacity_id` (from the API `capacityId` field), and `lakehouse_name` using [Authentication and route construction](references/auth-and-routing.md). Surface lookup failures; do not fall back to `(unknown)` or substitute GUIDs.
4. **Collect the outcome.** Reuse a supplied outcome verbatim. Otherwise ask **What do you want to accomplish?** After context resolution, ask one optional "Anything else I should know?" prompt. Use `ask_user` with the first choice `"No, nothing else"` and freeform enabled so the user can either skip quickly or type extra context. Keep the user's complete outcome and guidance verbatim. Never auto-send or start work before explicit user submission.
5. **Run intake.** Classify the task and render the recommendations card from [Operational intake questionnaire](references/intake-questionnaire.md). Append the rendered `## Operational constraints` block verbatim before `PUT /{taskId}` and before the initial user message.
6. **Authenticate and construct the task route.** Resolve the SparkCore task host and MWC token with [Authentication and route construction](references/auth-and-routing.md).
6. **Authenticate and construct the task route.** Resolve the SparkCore task host and MWC token with [Authentication and route construction](references/auth-and-routing.md), using the optional resource tenant override when one was supplied.


7. **Create and run one task.** Use one generated task ID for create, message, run, retries, and follow-ups. Follow [Task lifecycle](references/task-lifecycle.md) for endpoint shapes and response handling.
Expand Down Expand Up @@ -58,12 +94,13 @@ This file is the lean runtime contract. Put detailed mechanics in the reference

## References

- [URL parsing](references/url-parsing.md) — URL-first intake and public Fabric URL validation
- [URL parsing](references/url-parsing.md) — optional URL intake and Fabric URL validation
- [Operational intake questionnaire](references/intake-questionnaire.md) — task types, recommendations card, Questions 1-8, skip logic, rendered preamble
- [Authentication and route construction](references/auth-and-routing.md) — authentication flow and task base URL
- [Task lifecycle](references/task-lifecycle.md) — task/message/run endpoints, statuses, response shapes
- [Status dashboard](references/dashboard.md) — `./.dataprojects/<task-id>/` layout and `window.__STATE` schema
- [Spawning the dashboard poller daemon](references/dashboard-poller.md) — detached poller, token refresh, retry, resume, cleanup
- [Environment routing](references/environment-routing.md) — Fabric environment and API host selection
- [Troubleshooting](references/troubleshooting.md) — retryable Spark transient and auth/poller recovery

## Writing good instructions
Expand Down
18 changes: 13 additions & 5 deletions skills/project-osmos/references/auth-and-routing.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
# Authentication and route construction

Build the direct SparkCore task route after parsing and confirming the user's Lakehouse browser URL.
Build the direct SparkCore task route after resolving workspace and Lakehouse IDs from Fabric page context, Microsoft Fabric Skills discovery, or a validated Lakehouse browser URL.

## Optional resource tenant override

Use the current Azure CLI session by default. Do not ask for a tenant ID before attempting authentication. If the user already supplied the Microsoft Entra tenant ID that owns the workspace, pass it explicitly; guest users may need this resource-tenant override when their current session is in their home tenant. Ask for the resource tenant only when authentication or workspace lookup shows that the current session cannot access the target tenant.

## Public Fabric route

Expand All @@ -17,8 +21,8 @@ Public hosts:

### Token and route steps

1. Authenticate Azure CLI **for the same tenant the workspace's capacity lives in**: `az login --tenant <tenant-id>`. Do not rely on the default subscription from `az account show`; `generatemwctoken` is gated by the workspace's home tenant, and a token from a different tenant returns a silent `HTTP 403` (empty body).
2. Get a Power BI bearer token for that tenant: `az account get-access-token --tenant <tenant-id> --resource https://analysis.windows.net/powerbi/api`.
1. Use the current Azure CLI session. If no session is available, run `az login --allow-no-subscriptions`. For a supplied resource-tenant override, run `az login --tenant <resource-tenant-id> --allow-no-subscriptions`.
2. Get a Power BI bearer token with `az account get-access-token --resource https://analysis.windows.net/powerbi/api`. Add `--tenant <resource-tenant-id>` only when an override was supplied.
3. Call the workspace metadata endpoint and capture the API `capacityId` field. Use it as the route/token capacity ID, and persist it in dashboard state as `capacity_id`.
4. Call the public Fabric `generatemwctoken` endpoint with:
- `capacityObjectId`
Expand All @@ -35,22 +39,26 @@ The helpers write `routing.json`, a private `mwc-token` file, `env.sh` for Bash,

```bash
python3 skills/project-osmos/scripts/resolve-auth-and-routing.py \
--tenant-id <tenant-id> \
--workspace-id <workspace-id> \
--lakehouse-id <lakehouse-id> \
--fabric-api-host <selected-fabric-api-host> \
--output-dir .dataprojects/auth
source .dataprojects/auth/env.sh
```

For a guest or cross-tenant workspace, add `--resource-tenant-id <resource-tenant-id>`.

```powershell
pwsh -NoProfile -File skills/project-osmos/scripts/resolve-auth-and-routing.ps1 `
-TenantId <tenant-id> `
-WorkspaceId <workspace-id> `
-LakehouseId <lakehouse-id> `
-FabricApiHost <selected-fabric-api-host> `
-OutputDir .dataprojects/auth
. .dataprojects/auth/env.ps1
```

For a guest or cross-tenant workspace, add `-ResourceTenantId <resource-tenant-id>`. The Python helper still accepts `--tenant-id`, and the PowerShell wrapper still accepts `-TenantId`, as compatibility aliases.

Direct route shape:
```text
${TASKS_BASE}
Expand Down
Loading
Loading