scanpipe as stand-alone library

[JonoYang]

In the purldb, I want to be able to match resources/directories in a codebase to see if they are part of an indexed package. I am getting the codebase from a scancode.io scan where fingerprints have been computed for the directories and resources.

Currently, I do this by creating a separate django project with scanpipe in it and import all the scancode.io settings (https://github.com/aboutcode-org/purldb/tree/main/matchcode_project), and run it in the same docker compose file as purldb. This setup is untenable and I want to make it work better. I am wondering if it's reasonable to have scanpipe as a django app that I can add to a django project, and create a codebase and run pipelines without having to instantiate the entirety of scancode.io.

I want (from my purldb django app) to be able to create a scanpipe project (instantiate codebase, resources, packages, dependencies) from this codebase scan and run a pipeline that checks fingerprints against the purldb using the django orm.

The aboutcode.pipelines library provides the framework for the pipelines and execution, but does not have the other stuff for dealing with codebases, resources, and packages.

Alternatively, is there a way to run scancode.io in such a way that I can:

• Add purldb to DATABASES in scancode.io settings
• Query the purldb using the django orm from scancode.io without scancode.io trying to manage the model or having to declare purldb as an app in scancode.io

[tdruez]

Scancode.io is already available as a wheel, so we should mostly have everything we need to use it as a library within another Django app.

The main issue in the current state is the setting management that would need some adjustments for this app library setup.
For now, we can from scancodeio.settings import * but I'm expecting some conflicts with this approach.

POC:

Setup a new Django app

uv init app --python 3.14
cd app
uv add django scancodeio
uv run django-admin startproject config .
source .venv/bin/activate

Edit config/settings.py

Import ScanCode.io settings in the imports

from scancodeio.settings import *

Add scanpipe in INSTALLED_APPS

INSTALLED_APPS = [
    ...
    'scanpipe',
]

Run migrations to create DB tables

./manage.py migrate

Manage projects from Python

./manage.py shell

from scanpipe.management.commands import create_project

project = create_project(
    name="projet1",
    pipelines=["scan_single_package"],
    input_urls=["https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz"],
    execute=True,
)

# project.discoveredpackages.all()
# etc...

Manage projects from CLI

./manage.py create-project projet1 \
  --pipeline scan_single_package \
  --input-url https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz \
  --execute

[JonoYang]

Thanks, this looks like how I am using scio in purldb. My issue with using scio in purldb like this is by calling from scancodeio.settings import *, we need to create /etc/scancodeio/.env with the secret key. In my purldb depoyment, I have both /etc/scancodeio/.env and /etc/purldb/.env . I would like to just use /etc/purldb/.env as the primary environment file.

[tdruez]

@JonoYang I've started to implement the changes regarding the settings in #2187
Could you give it a try and let me know if it needs some adjustments?

---

New instructions:

Edit settings.py

Add scanpipe in INSTALLED_APPS:

INSTALLED_APPS = [
    ...
    "scanpipe",
]

Settings can be provided via environment variables using the SCANCODEIO_ prefix (e.g. SCANCODEIO_WORKSPACE_LOCATION).

Optionally override any scanpipe default via the SCANPIPE dict:

SCANPIPE = {
    "WORKSPACE_LOCATION": "/path/to/workspace",
    "PROCESSES": 2,
}

All settings have built-in defaults, the SCANPIPE dict is only needed for values that differ from those defaults.