OWLLESS: Open-Weights Local LLM Extractor for Social Networks Studies

📖 Table of Contents

Table of Contents

1. ➤ About The Project
2. ➤ Overview
3. ➤ Project Files Description
4. ➤ Quick start
5. ➤ Scenario: Full framework usage
6. ➤ Scenario: Partial framework usage
7. ➤ Scenario: Add new variables
8. ➤ Scenario: New research domain
9. ➤ Scenario: Switch LLM
10. ➤ Scenario: HTML Format of Publications
11. ➤ Credits

📝 About The Project

This project facilitates the semi-automated extraction and analysis of information from scientific social network research papers. It processes individual studies through a systematic framework - extracting key information, standardizing data formats, and generating a comprehensive overview table for cross-study comparisons. While primarily focused on social network literature, the framework is designed to be extensible to other research domains. The framework was created to work with open-weight LLMs and on local infrastructure.

Disclaimer: As demonstrated in the validation study presented in paper, this tool is designed for a human-in-the-loop approach. It is intended to support researchers in extracting relevant information but does not replace the need for expert review. Fully automated extraction is not guaranteed to be error-free; therefore, human verification of the output is still essential.

☁️ Overview

This project systematically extracts and analyzes critical variables from social network research papers to create a comprehensive meta-analysis database. Building upon the foundational work of Neal (2024) and Neal (2025), we replicate their structured approach while significantly expanding the scope of extracted variables to capture a more complete picture of social network methodologies and findings.

For a complete list of all extracted variables and their definitions, see the Variable Schema.

💾 Project files

• requirements.txt - Required packages for the framework
• metadata_extraction_pipeline.py - Main data extraction framework
• my_utils.py - Helper functions for the framework
• e.g. /yaml_files/04_net_unify_roster.yaml - Examplar yaml file with the prompt for the base reasoning of roster variables (see below). All yaml files are in the folder yaml_files.

💾 Quick start

Follow these steps to install all requirements to use the framework:

git clone https://github.com/yourusername/yourrepo.git
cd yourrepo

# Install Python version (recommended: use pyenv)
pyenv install 3.11.4  # or your specified version
pyenv local 3.11.4

# Create virtual environment and install dependencies
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
pip install --upgrade pip
pip install -r requirements.txt

System Dependencies

The framework requires the following system-level tools that are not installable via pip:

Poppler (required for partition_pdf with hi_res strategy)

Poppler provides the PDF rendering utilities (pdfinfo, pdftoppm) used by pdf2image under the hood.

• Linux (Debian/Ubuntu):

  sudo apt install poppler-utils

• macOS:

  brew install poppler

• Windows:
  1. Download the latest release from poppler-windows
  2. Extract the archive (e.g., to C:\poppler)
  3. Add the bin folder to your PATH:

     $env:PATH = "C:\poppler\poppler-24.08.0\Library\bin;" + $env:PATH

     Or add it permanently via System Environment Variables.

Tesseract OCR (required for hi_res PDF extraction strategy)

• Linux: sudo apt install tesseract-ocr

• macOS: brew install tesseract

• Windows: Download the installer from UB Mannheim.

  After installation, add it to your PATH:

  $env:PATH = "C:\Program Files\Tesseract-OCR;" + $env:PATH

Windows-specific notes

When installing on Windows, note the following package adjustments:

• Use python-magic-bin instead of python-magic (the latter requires libmagic which is not natively available on Windows)
• pyxnat does not build on Windows and should be removed from requirements
• pdfplumber may need to be installed with --no-deps if its pinned pdfminer.six version conflicts with unstructured. Install separately:

  pip install pdfminer.six==20240706
  pip install --no-deps pdfplumber==0.11.5

Linux / macOS notes

• libmagic is required by python-magic:
  • Linux: sudo apt install libmagic1 (Debian/Ubuntu) or sudo yum install file-libs (RHEL/CentOS)
  • macOS: brew install libmagic
• Ghostscript is required by camelot-py for PDF table extraction:
  • Linux: sudo apt install ghostscript
  • macOS: brew install ghostscript
• tkinter is needed by some visualization dependencies. It usually comes with Python, but on some Linux distros it must be installed separately:
  • Debian/Ubuntu: sudo apt install python3-tk
  • Fedora/RHEL: sudo dnf install python3-tkinter

🔸 Scenario: Full framework usage

This project implements a comprehensive LLM-based extraction framework for systematically collecting and processing metadata from social network research papers. The complete methodology is described in detail in LINK TO PAPER. The framework supports the semi-automated extraction of study design characteristics, sample demographics, and social network data collection protocols from scientific publications, transforming unstructured text into a structured, analyzable dataframe.

🚀 Framework Execution

When you run the code as provided, it will extract all variables documented in the tables above and output them in a structured format. The framework processes PDF documents through multiple extraction stages, utilizing large language models to identify and extract relevant information based on predefined schemas for each data domain (design, participants, roster, limit, extreme, and network_details).

⚠️ Important Implementation Notes

• Outline: The code is structured with markdown sections and it is recommended to leverage the outline features to display the different sections e.g. in VSCode.
• File Paths: The code was originally written using Windows path conventions. If you are running on macOS or Linux, please adjust the file paths accordingly (e.g., change backslashes \ to forward slashes /).
• Time Delay: The time delay between processing steps is optional and depends on your specific requirements. In our implementation, we customized the delay to allow our framework to run overnight without exhausting hardware resources.
• Processing Options: You can choose to process papers individually (one paper end-to-end) or in batches as we did (5 papers per cell or processing stage, waiting for each batch to complete before starting the next).
• Supplementary Materials: Supplementary materials are processed as separate papers and must be provided in PDF format. The framework treats each supplementary file as an independent document during extraction. After extraction is complete, you can merge the supplementary data with the main paper data using the combination code provided in the repository with examples.
• Custom Merging: If you prefer a different merging strategy than the default approach, you can modify the merge logic in the combination scripts to suit your specific needs.
• JSON Formatting Issues: Occasionally, the LLM output may have JSON formatting errors. If extraction fails, navigate to the output file and verify that it begins with ```json and ends with ```. You can manually correct these delimiters if needed to ensure proper parsing.

Extracted Variables

The following variables were extracted from the LLM framework and generated into a usable dataframe.

Category	Variable	Explanation
Study Design	Sample Size (N)	Total number of participants included in the analysis
	Geographical Location	Geographic location where data was collected
	Data Collection Time Period	Time period during which data was collected
	Sample Language(s)	Language(s) spoken by participants
	Dataset Citation/Reference	Citation or reference for the dataset if secondary data was used
	Open Data	Indicator of whether data is openly available (Yes/No)
	Supplementary Material Exists	Indicator of whether supplementary materials are available (Yes/No)
	Supplementary Material Link/DOI	Direct link or DOI to access supplementary materials
	Online/Offline Data Collection	Mode of data collection (online or offline)
	Data Collection Summary	Summary description of data collection procedures
	Design Type	Study design classification (Longitudinal/Panel or Cross-sectional)
Sample Characteristics	Wave	Timeframe or wave of data collection
	Cohort	(Sub-)Group of the sample
	Network Sample Size	Number of participants in the network sample
	Response Rate (%)	Percentage of invited participants who completed the study
	Missing Rate (%)	Percentage of missing data
	Min Age (Years)	Minimum age of participants
	Max Age (Years)	Maximum age of participants
	Mean Age (Years)	Average age of participants
	Age Standard Deviation (SD)	Standard deviation of participant ages
	Gender Distribution	Gender distribution (Absolute counts, % Female, % Male)
Social Network Data Collection and Metrics	Network Scope and Boundaries	Definition of social network boundaries (e.g., workplace, classroom, online community)
	Relationship Type - Nature of Ties	Nature of the collected ties (e.g., friendship, advice, conflict)
	Relationship Type - Positions/Roles	Positions and roles participants were asked about (e.g., lawyer, doctor, teacher)
	Data Collection Method	Method used for network data collection (e.g., Free Recall, Roster Method, Position Generator)
	Name Generator/Elicitation Instrument	Detailed description of the name generator or elicitation instrument used
	Best/Closest Relationships Indicator	Whether the study focused on best/closest relationships (Yes/No)
	Sampling Strategy	Type of network sampling used (Complete Network, Ego-Network, Snowball Sampling)
	Maximum Nominations Allowed	Maximum number of nominations participants could make
	Minimum Nominations Required	Minimum number of nominations required (if applicable)
	Soft or Hard Boundary	Type of boundary for nomination limit (soft or hard)
	Cues Used Indicator	Whether cues were used during nominations (Yes/No)
	Cues Specification	Description of specific cues used (if applicable)
	Usage of Full Names	Whether participants were required to use full names (Yes/No)
	Technical Support for Auto-complete	Whether technical support for auto-complete or suggestions was provided (Yes/No)
	Network Directionality	Whether the network is directed or undirected
	Tie Inclusion Criteria	Criteria for including ties (e.g., Only Reciprocal Ties Used, All Ties Used)
	Key Network Descriptors	Description of important aspects of the data used for analysis
	Network Size	Total number of participants in the network
	Network Density	Proportion of actual ties to possible ties in the network
	Reciprocity (%)	Percentage of reciprocated ties in the network
	Average Degree	Average number of ties per node
	Degree Standard Deviation (SD)	Standard deviation of degree distribution
	Average Outdegree	Average number of outgoing ties per node
	Outdegree Standard Deviation (SD)	Standard deviation of outdegree distribution
	Average Indegree	Average number of incoming ties per node
	Indegree Standard Deviation (SD)	Standard deviation of indegree distribution
	Average Path Length	Average shortest path length between all pairs of nodes
	Clustering Coefficient	Measure of the degree to which nodes cluster together
	Jaccard Index	Measure of similarity between network structures

Name conventions in the code

As the data is extracted during the base reasoning as a JSON, we expect if values don't exist it either uses "NA" or null for numerical values. In rare cases it can happen that the LLM skips the variable as it does not find the information it looks for. It happens usually only when a lot of information is extracted in one iteration.

Document Category	Document Variable	Code Section/Domain	Code Variable Name	Variable Values
Study Design	Sample Size (N)	design	design_inital_participants	e.g. 401
	Geographical Location	design	design_country	e.g. Norway
	Data Collection Time Period	design	timeframe_names	e.g. May 2018, May 2019
	Design Type	design	design_timeframe	longitudinal or cross-sectional
	Sample Language(s)	design	design_language	e.g. English
	Dataset Citation/Reference	design	design_originality_citation	Citation or NA
	Open Data	design	design_open_access	True or false
	Open Data	design	design_open_access_url	URL or NA
	Supplementary Material Exists	design	design_supplementary_materials	True or false
	Supplementary Material Link/DOI	design	design_supplementary_materials_url	URL or NA
	Online/Offline Data Collection	design	design_datacollection_online	Online or offline
	Data Collection Summary	design	design_datacollection	e.g. The data collection occured via an online platform in May 2018 and May 2019.
Sample Characteristics	Wave	participants	wave	e.g. t1
	Cohort	participants	cohort	e.g. all or class1
	Network Sample Size	participants	participants_sample_size	e.g. 41
	Response Rate (%)	participants	design_response_rate	e.g. 90%
	Missing Rate (%)	participants	design_missing_rate	e.g. 10%
	Min Age (Years)	participants	participants_age_min	e.g. 14
	Max Age (Years)	participants	participants_age_max	e.g. 18
	Mean Age (Years)	participants	participants_age_average	e.g. 15.53
	Age Standard Deviation (SD)	participants	participants_age_sd	e.g. NA
	Gender Distribution	participants	participants_gender_female_absolute	e.g. null (NA)
	Gender Distribution	participants	participants_gender_female_percent	e.g. 47%
	Gender Distribution	participants	participants_gender_male_absolute	e.g. null (NA)
	Gender Distribution	participants	participants_gender_male_percent	e.g. 53%
Social Network Data Collection and Metrics	Network Scope and Boundaries	limit	network_boundary	e.g. classmates
	Relationship Type - Nature of Ties	roster	roster_relationships	e.g. classmates
	Relationship Type - Positions/Roles	roster	position_relations	e.g. NA
	Data Collection Method	roster	network_generation_method	e.g. roster
	Name Generator / Elicitation Instrument	roster / limit / extreme / network_details	namegen_question	e.g. Who are your best friends?
	Best/Closest Relationships Indicator	extreme	relationship_measured_extreme	True or False
	Sampling Strategy	extreme	design_network	e.g. ego network
	Minimum Nominations Required	limit	min_nominations	e.g. null (NA)
	Maximum Nominations Allowed	limit	max_nominations	e.g. 10
	Soft or Hard Boundary	limit	limit_nominations_softhard	e.g. soft
	Cues Used Indicator	roster	namegen_cue	e.g. no
	Cues Specification	roster	namegen_cue_wording	e.g. NA
	Usage of Full Names	roster	namegen_fullnames	e.g. nicknames
	Technical Support for Auto-complete	roster	namegen_tech_help	e.g. no
	Network Directionality	extreme	asymmetric_ties	True or False
	Tie Inclusion Criteria	extreme	only_reciprocity	True or False
	Network Size	network_details	network_size	e.g. 39
	Network Density	network_details	network_density	e.g. null (NA)
	Reciprocity (%)	network_details	network_reciprocity	e.g. 0.89
	Average Degree	network_details	network_degree_mean	e.g. null (NA)
	Degree Standard Deviation (SD)	network_details	network_outdegree_sd	e.g. null (NA)
	Average Outdegree	network_details	network_degree_mean	e.g. 5.45
	Outdegree Standard Deviation (SD)	network_details	network_outdegree_sd	e.g. 3.2
	Average Indegree	network_details	network_indegree_mean	e.g. null (NA)
	Indegree Standard Deviation (SD)	network_details	network_indegree_sd	e.g. null (NA)
	Average Path Length	network_details	network_average_path_length	e.g. 3.2
	Clustering Coefficient	network_details	network_clustering	e.g. null (NA)
	Jaccard Index	network_details	network_jaccard_coefficient	e.g. 0.2

The data is extracted in different stages which need to be combined afterwards. These are the merging phases:

Merge Step	Data Sections Being Merged	Merge Keys	Description
Step 1: Network Elicitation Data	roster + limit	namegen_question	Merge roster-based variables with nomination limit constraints using the name generator question as the key
	roster + extreme	namegen_question	Merge roster-based variables with extreme/boundary condition variables using the name generator question as the key
Step 2: Complete Network Data	network_details + roster (with limit and extreme)	namegen_question	Merge network metrics and descriptive statistics with the combined roster data (which already includes limit and extreme variables) using the name generator question as the key
Step 3: Full Dataset	participants + all network data (roster + limit + extreme + network_details)	namegen_question wave cohort (optional: sample_size)	Merge participant characteristics with all network data using multiple keys: name generator question, wave, and cohort. Sample size can optionally be used as an additional merge key for validation
Final Output	design + complete merged data	study identifier or paper ID	Add study design metadata to the complete merged dataset, typically using a study identifier or paper ID as the key

📊 Output Structure

The framework generates structured JSON files for each data domain, which are subsequently merged according to the merge framework described above. The final output is a comprehensive dataframe containing all extracted variables, ready for human review, modification, and further processing to enable meta-analysis and statistical modeling of social network research methodologies.

🔸 Scenario: Partial framework usage

📊 Components

While the full framework is designed to work as an integrated system, you can run individual components separately based on your needs. Below are the available options for independent execution.

🔹 Design extracts study design information from papers. This component always runs independently with no dependencies on other components. It can be executed at any time without prerequisite steps.

🔹 Roster extracts relationship information from papers. This component can run standalone and outputs relationship data for downstream processing.

🔹 Limit extracts study boundary and limitation information. This component can run standalone. However, if you are running it without Roster, you must remove the extra input dependencies from Roster in the YAML configuration file and python code.

🔹 Extreme extracts information of extreme relationships. This component can run standalone. Similar to Limit, if you are running it without Roster, you must remove the extra input dependencies from Roster in the YAML configuration file.

🔹 Network Details extracts network statistics and metrics from the papers. This component has dependencies on both Roster and the Name Generator outputs. It is highly recommended to maintain this dependency structure as it provides essential contextual information for accurate network analysis.

🔹 Participants extracts participant-level information from the papers. The Name Generator by Roster must be provided as a required input. However, context sharing with Network Details is optional. If you do not need Network Details context, you can modify the participant code to remove these dependencies. This allows for more flexible participant extraction when network statistics are not required for your analysis.

🔸 Scenario: Add new variables

If you want to extract additional variables within the social network domain:

1. Identify the appropriate extraction step: Review the Variable Schema to understand which variables are extracted at each framework stage and determine the most suitable location for your new variable.

2. Evaluate variable scope: Determine whether the information applies to the entire paper or requires contextual information:

   • Paper-level variables: Information that is consistent throughout the study (e.g., overall sample size, primary data collection method) can be added to the "Design" step
   • Context-dependent variables: Information that varies by cohort, analysis, or subgroup requires careful evaluation based on your domain knowledge to determine the appropriate extraction context

3. Create the extraction specification: Once you've identified the right step and context, write a concise description following the format shown in the .yaml file for your chosen extraction step. Be specific about:

   • What information to extract
   • Where it typically appears in papers
   • Any contextual dependencies
   • Expected format or units

🔸 Scenario: New research domain

If you want to apply this framework to a domain beyond social network science:

1. Map your domain's key variables:

   • Identify what information you need to extract
   • Determine how different pieces of information connect to each other
   • Understand the relationships and dependencies between variables

2. Assess reporting patterns:

   • Is the information reported once or multiple times throughout papers?
   • Does it require contextual information to interpret correctly?
   • Are there standard sections where this information appears?

3. Evaluate section coverage:

   • The current framework focuses on page 1, methods, results sections and supplementary materials
   • If your target information appears elsewhere (e.g., discussion), you'll need to expand the page numbers and sections processed

4. Define clear extraction rules:

   • Use "average" instead of "mean" when describing statistical measures
   • Provide concrete examples for each variable
   • Be precise and specific in your descriptions
   • Avoid ambiguous terminology

5. Establish a validation dataset:

   • Create a "golden dataset" of test papers with known expected outputs
   • Document what you expect to extract from each test paper
   • Define quality thresholds for when extraction performance is "good enough"
   • Note that results may vary depending on which extraction model you use, making this validation dataset critical for consistent evaluation

Best practices

• Start with a small set of core variables and expand iteratively
• Test extraction on diverse paper formats and writing styles
• Document edge cases and how they should be handled
• Regularly validate against your golden dataset as you make changes

🔸 Switch LLM

This project leverages open-source models in a two-stage approach:

• Pixtral: Extracts text and tables from paper pages
• Llama 4: Performs reasoning and formats output as JSON

Option A: Use the Same Models (Self-Hosted)

Deploy the default models on your own hardware:

Hardware Requirements: We used two NVIDIA RTX 6000 Ada Generation GPUs for LLM hosting (1 model at a time, 1 request at once)

Model Links:

• Pixtral: Hugging Face link
• Llama 4: Hugging Face link

Option B: Use Alternative Open-Source Models

Extraction Model Alternatives

If you want to replace Pixtral:

• Newer versions: Directly swap to updated Pixtral versions and test performance
• Different models: Experiment with alternatives that offer:
  • Ability to process more images simultaneously
  • Ability to extract and process tables
  • Larger context windows (note: this may lead to accuracy degradation)
  • Native PDF support capabilities

Reasoning Model Alternatives

If you want to replace Llama 4:

• Evaluate whether the new model supports advanced features like chain-of-validation
• Test if newer capabilities improve extraction accuracy
• Verify JSON output formatting remains consistent

Option C: Use Proprietary Models

Proprietary models (e.g., GPT-4, Claude) may offer direct PDF processing capabilities.

Important Considerations:

1. Publisher restrictions: Check the publisher's terms of service, as most forbid using their content with proprietary LLM models
2. Direct PDF processing: If available, evaluate:
   • How well the reasoning performs using the prompts described earlier
   • Table recognition accuracy
   • Information extraction quality
3. Extended capabilities: With larger context windows and more powerful models, you can:
   • Extract more variables and context in a single pass
   • Process longer papers more efficiently
   • Decrease the processing time
   • Experiment with more complex extraction patterns

Testing Requirements

Critical: In all scenarios, thorough testing is essential because:

• LLMs are non-deterministic
• Performance cannot be predicted in advance
• Results vary between models and even between runs
• Your golden dataset validation is crucial for quality assurance

Always benchmark new models against your established test papers for using the framework for all your publications of interest.

🔸 HTML Format of Publications

Although the framework was primarily optimized for PDF ingestion, its modular architecture allows for adaptation to HTML-based publications. By leveraging HTML’s semantic tags, the Extraction Phase can be largely bypassed or replaced by a simplified parsing layer. This shift eliminates the need for complex layout analysis and OCR, allowing the pipeline to proceed directly to Summarization, Standardization, and Postprocessing. However, full compatibility requires specific adjustments to the data ingestion scripts to account for varied HTML structures across different publishers.

Credits

Developer:

Zoran Kovacevic

Anja Shevchyk

Jennifer W. Neal