MystiGuardian is a Discord bot focused on apprenticeship opportunities scraping and management. The project scrapes apprenticeship listings from multiple sources and posts them to Discord channels.
- Apprenticeships (not jobs) - The primary focus is on degree apprenticeships and similar opportunities
- Use
Apprenticeshipinterface/class names, notJob - Use
ApprenticeshipSpreadsheetManagernotJobSpreadsheetManager - Use
HigherinApprenticeshipfor Higher In apprenticeships - Use
FindAnApprenticeshipfor GOV.UK apprenticeships
- DiscordBot - Main Discord bot module with JDA integration
- ApprenticeshipScraper - Scraping and Google Sheets management
- OAuth - OAuth authentication and web service
- Annotations - Custom annotations for the project
-
io.github.yusufsdiscordbot.mystiguardianApprenticeshipScraper.java- Facade for web scraping (delegates to specialized scrapers)
-
io.github.yusufsdiscordbot.mystiguardian.apprenticeshipApprenticeship.java- Interface for apprenticeship objectsApprenticeshipSource.java- Enum for apprenticeship sources (RATE_MY_APPRENTICESHIP, GOV_UK)HigherinApprenticeship.java- Implementation for Higher In apprenticeshipsFindAnApprenticeship.java- Implementation for GOV.UK apprenticeships
-
io.github.yusufsdiscordbot.mystiguardian.scraperHigherinScraper.java- Specialized scraper for Higher In (Rate My Apprenticeship)FindAnApprenticeshipScraper.java- Specialized scraper for GOV.UK Find an Apprenticeship
-
io.github.yusufsdiscordbot.mystiguardian.categoriesHigherinCategories.java- Configuration class with 83 Higher In category slugsGovUkRoutes.java- Configuration class with 15 GOV.UK route mappings
-
io.github.yusufsdiscordbot.mystiguardian.managerApprenticeshipSpreadsheetManager.java- Google Sheets integration and Discord posting
-
io.github.yusufsdiscordbot.mystiguardian.configDAConfig.java- Configuration for Digital Apprenticeship postingApprenticeshipCategoryGroup.java- Category grouping for role pinging
-
io.github.yusufsdiscordbot.mystiguardian- Main bot configuration and initialization
- Event system for new apprenticeship announcements
- Discord commands (admin, moderation, miscellaneous)
-
io.github.yusufsdiscordbot.mystiguardian.event- Event dispatcher and handlers
NewDAEvent- Triggered when new apprenticeships are found
- Java 21+ with virtual threads support
- Gradle with Kotlin DSL for build configuration
- Gradle Version Catalog (
gradle/libs.versions.toml) for centralized dependency management - JDA (Java Discord API) for Discord integration
- Google Sheets API for spreadsheet management
- Lombok for boilerplate reduction
- Always use the version catalog - Never hardcode versions in build.gradle.kts
- Access dependencies via
libs.dependency.name(e.g.,libs.jda,libs.lombok) - Use bundles for related dependencies (e.g.,
libs.bundles.logging,libs.bundles.database) - Update versions in
gradle/libs.versions.tomlto maintain consistency across all modules - See
docs/VERSION_MANAGEMENT.mdfor complete guide
- Use Lombok annotations (@Getter, @Setter, @Slf4j, etc.)
- Use records for immutable data classes where appropriate
- Use var for local variable type inference when type is obvious
- Use SLF4J for logging (
logger.info(),logger.error(), etc.) - Follow Google Java Style Guide (enforced by Spotless)
- Always log errors with context using SLF4J
- Use try-catch blocks for I/O operations
- Handle rate limits and API failures gracefully
- PostgreSQL with JOOQ for type-safe queries
- Flyway for database migrations (SQL-based schema management)
- Database configuration in
DataSourceConfig - Use
MystiGuardianDatabasefor database operations - All schema changes via SQL migration files in
src/main/resources/db/migration/
- Flyway manages all database schema changes
- Migration files located in
DiscordBot/src/main/resources/db/migration/ - Naming convention:
V{VERSION}__{Description}.sql(e.g.,V1__Initial_schema.sql) - Migrations run automatically on bot startup
- Never modify executed migration files - always create new ones
- Current migrations:
V1__Initial_schema.sql- All base tablesV2__Add_stored_files_table.sql- File management system
Available Flyway tasks (configure database credentials in gradle.properties or via -P flags):
# Run migrations
./gradlew :DiscordBot:flywayMigrate -PdataSourceUrl=jdbc:postgresql://localhost:5432/db -PdataSourceUser=user -PdataSourcePassword=pass
# Check migration status
./gradlew :DiscordBot:flywayInfo
# Validate migrations
./gradlew :DiscordBot:flywayValidate
# Clean database (DANGER: deletes all data)
./gradlew :DiscordBot:flywayClean
# Repair migration history
./gradlew :DiscordBot:flywayRepair-
Higher In (
RATE_MY_APPRENTICESHIP)- Scrapes from
higherin.com - Supports multiple tech, business, engineering categories
- Returns
HigherinApprenticeshipobjects
- Scrapes from
-
GOV.UK Find an Apprenticeship (
GOV_UK)- Scrapes from
findapprenticeship.service.gov.uk - Returns
FindAnApprenticeshipobjects
- Scrapes from
- Apprenticeships are stored in Google Sheets
- Sheet structure: ID, Title, Company, Location, Categories, Salary, Opening Date, Closing Date, URL, Source
- New apprenticeships are detected by comparing with existing sheet data
- Duplicate detection uses apprenticeship ID
- New apprenticeships are posted to configured Discord channels
- Embeds show apprenticeship details with category tags
- Role pinging based on category mappings in
config.json - Batch posting (10 embeds per message) with rate limit handling
token- Discord bot tokenprefix- Command prefix (default:/)ownerId- Bot owner Discord IDdiscordAuth- OAuth configurationda(Digital Apprenticeships) - Scraping configurationchannelIds- Discord channel IDs to post tospreadsheetId- Google Sheets IDcategoryGroups- Category to role mappings
rolesToPing- List of role IDs to ping for new apprenticeships
- Unit tests in
src/test/java - Use JUnit 5
- Mock external APIs where appropriate
- Create a new class implementing
Apprenticeshipinterface inapprenticeshippackage - Add source enum to
ApprenticeshipSourceenum - Create a new scraper class in
scraperpackage (e.g.,NewSourceScraper.java) - Update
ApprenticeshipScraperfacade to delegate to the new scraper - Update sheet format in
ApprenticeshipSpreadsheetManager.convertApprenticeshipsToRows()
- Create class implementing
ISlashCommand - Add
@SlashEventBusannotation - Implement
onSlashCommandInteractionEvent()method - Command auto-registers on bot startup
- Create new file:
V{NEXT_VERSION}__{Description}.sqlinsrc/main/resources/db/migration/ - Write SQL DDL statements (CREATE, ALTER, etc.)
- Test locally - migration runs automatically on bot startup
- Never modify executed migrations - always create new ones
- After migration runs, regenerate JOOQ classes:
./gradlew :DiscordBot:generateJooq
- Higher In categories: Update
HigherinCategoriesclass incategoriespackage- Categories organized by sector (Technology, Finance, Business, etc.)
- Add/remove categories from the appropriate sector constant
- Categories are URL slugs used in Higher In search paths
- GOV.UK routes: Update
GovUkRoutesclass incategoriespackage- Routes map category names to their official GOV.UK route IDs
- Use
Map.entry("Category Name", routeId)format
- Always use "apprenticeship" terminology in code, comments, and logs
- The project focuses on degree apprenticeships (Level 6+)
- Respect rate limits when scraping (500ms delay between requests)
- Use virtual threads for concurrent operations when available (Java 21+)
- The bot runs as a long-lived service with scheduled tasks for scraping
Update CHANGELOG.md for any significant changes:
- New features or commands
- Bug fixes
- Breaking changes
- Dependency updates (major versions)
- Performance improvements
- Configuration changes
Follow Keep a Changelog format:
## [X.X.X] - DD/MM/YYYY
### Added
- New features
### Changed
- Changes in existing functionality
### Deprecated
- Soon-to-be removed features
### Removed
- Removed features
### Fixed
- Bug fixes
### Security
- Security fixesFollow Semantic Versioning:
- MAJOR (X.0.0) - Breaking changes
- MINOR (0.X.0) - New features (backwards compatible)
- PATCH (0.0.X) - Bug fixes (backwards compatible)
Update version in build.gradle.kts:
version = "X.X.X"- Document changes as you make them
- Be descriptive but concise
- Group related changes together
- Include issue/PR numbers if applicable
- Keep entries in reverse chronological order (newest first)