Survey-based auto mode, parallel processing, and config/README updates

README.md

@@ -6,7 +6,7 @@ AWARE Narrator is a comprehensive Python toolkit that processes sensor data from
 ## Manual Setup Requirements
 
 ### Google Maps Geocoding Update (Only if using Google Maps API)
-If you plan to use Google Maps API for reverse geocoding (requires a valid API key in `config.yaml`), you may need to manually update the geocoding module due to recent changes in the Google Maps Services Python library:
+If you plan to use Google Maps API for reverse geocoding (requires `USE_GOOGLE_MAP: true` and a valid API key file referenced by `GOOGLE_MAP_KEY` in `config.yaml`), you may need to manually update the geocoding module due to recent changes in the Google Maps Services Python library:
 
 1. Check for updates on the GitHub repository: https://github.com/googlemaps/google-maps-services-python.git
 2. Find your local geocoding.py file path (typically in your mamba/conda environment)
@@ -27,7 +27,7 @@ cp geocoding.py $(python -c "import googlemaps; import os; print(os.path.dirname
 
 This manual update ensures compatibility with the address descriptor feature in geocoding.py
 
-**Note:** This setup is only required if you plan to use Google Maps API for reverse geocoding. If you leave the `GOOGLE_MAP_KEY` empty in your `config.yaml`, the toolkit will work without this update.
+**Note:** This setup is only required if you plan to use Google Maps API for reverse geocoding. If you set `USE_GOOGLE_MAP: false` in your `config.yaml`, the toolkit will work without this update.
 
 
 ## Installation
@@ -88,7 +88,7 @@ This project supports **Mamba** and **Conda** for managing and installing depend
 ## Project Structure
 The project includes several Python scripts:
 
-- **`aware_narrator.py`** - Main script that processes sensor data and generates narratives
+- **`aware_narrator.py`** - Main script that processes sensor data and generates narratives (manual mode with a fixed time range, or auto mode driven by per-survey timestamps; supports parallel participant processing via `num_workers`)
 - **`extract_sessions.py`** - Extracts session boundaries from screen on/off events
 - **`json2jsonl.py`** - Utility to convert JSON files to JSONL format
 - **`split_description.py`** - Splits output narratives by sensor type
@@ -103,21 +103,63 @@ The scripts require a YAML configuration file with the following structure:
 ```yaml
 # Configuration for Aware Narrator
 # Two modes: manual or auto
-# Manual mode: Uses P_IDs, START_TIME, and END_TIME for all participants
-# Auto mode: Uses pid_timerange_file for participant-specific time ranges
+# Manual mode: Uses P_IDs, START_TIME, and END_TIME, output_file
+# Auto mode: Uses survey_time_file, time_ranges, output_dir
 
 MODE: "manual"  # Options: "manual" or "auto"
 
-# Manual mode configuration (used when MODE: "manual")
+
+# START of manual mode configuration (used when MODE: "manual")
+
 P_IDs:
   - SS001
 
 START_TIME: "2025-05-25 06:00:00"
 END_TIME: "2025-06-04 23:59:00"
+output_file: "description/{P_ID}_{START_TIME}_{END_TIME}.txt"
+
+# END of manual mode configuration
+
+
+# START of auto mode configuration (used when MODE: "auto")
+
+# CSV File containing survey-specific answered timestamp, one pid might have multiple rows
+# Main columns: pid,survey_id,survey_time_unix
+survey_time_file: "resources/survey_time.csv"
+
+# Direction for time range processing:
+#   - "backward": survey_time is the END point, process data BEFORE survey (default, for weekly surveys)
+#                 e.g., 7d range = [survey_time - 7d, survey_time]
+#   - "forward":  survey_time is the START point, process data AFTER that time (for cumulative from baseline)
+#                 e.g., 7d range = [survey_time, survey_time + 7d]
+direction: "backward"  # Options: "backward" or "forward"
+
+# Alignment for the reference timestamp:
+#   - true:  Align to midnight (00:00:00) of the survey timestamp's day
+#   - false: Use the exact survey timestamp as-is (default)
+align_to_midnight: false
+
+# Skip records from the survey date itself:
+#   - true:  Exclude all data from the calendar day of the survey
+#   - false: Include survey day data (default)
+skip_survey_day: false
+
+# Time ranges to save descriptions
+# Option 1: Specify start and end to auto-generate all ranges in between (same unit required)
+#   Order follows start→end (ascending if start < end, descending if start > end)
+time_range_start: 7d
+time_range_end: 1d
+#
+# Option 2: Explicit list (used if time_range_start/time_range_end are not set)
+# time_ranges:
+#   - 7d
+#   - 3d
+#   - 1d
+
+output_dir: "description/{P_ID}"
+
+# END of auto mode configuration
 
-# Auto mode configuration (used when MODE: "auto")
-# File containing participant-specific time ranges
-pid_timerange_file: "resources/pid_timerange_60m.json"
 
 # Replace by your own mapping csv. 
 # Must contain device_id and pid columns. 
@@ -131,15 +173,17 @@ pid_to_deviceid_map: "resources/pid_deviceid_mapping.csv" # generated by running
 timezone: "Australia/Melbourne" # replace by actual timezone
 
 input_directory: "participant_data/{P_ID}"
+package_to_app_map: "resources/app_package_pairs.jsonl" # required; generated by the screentext preprocess pipeline
 session_data_file: "step1_data/{P_ID}/sessions.jsonl" # required for applicaion and keyboard analysis; generated by running extract_sessions.py with screen.jsonl for the corresponding P_ID
 cleaned_screentext_file: "step1_data/{P_ID}/clean_input.jsonl" # required for screen text description; generated by using screen text preprocessing pipeline
 
 
 reverse_geocoding_output_dir: "locations_query_results/{P_ID}"
-output_file: "description/{P_ID}_output.txt"
 daily_output_dir: "daily_description/{P_ID}"
 
 
+num_workers: 1 # Number of parallel workers for processing participants (1 = sequential, >1 = parallel via ProcessPoolExecutor)
+
 sensor_integration_time_window: 60 # minutes
 gate_time_window: 5 # minutes; required for wifi and bluetooth scan data integration.
 
@@ -159,7 +203,8 @@ sensors:
   - "locations"
 
 DISCARD_SYSTEM_UI: true # Applied to 'applications_foreground', 'applications_notifications', 'installations' based on system_ui_apps
-GOOGLE_MAP_KEY: "" # Used for Google API Reverse geocoding
+USE_GOOGLE_MAP: false # Set to true to enable Google Maps reverse geocoding
+GOOGLE_MAP_KEY: "GOOGLE_MAP_API_KEY.txt" # Path to a text file containing your Google API key (only read when USE_GOOGLE_MAP is true)
 eps: 0.000047  # DBSCAN clustering parameter: radians (0.000047 radians × 6371000 m ≈ 300m)
 min_samples: 3  # DBSCAN clustering parameter: mininum number of points to form a cluster
 
@@ -203,10 +248,25 @@ system_ui_apps:
 
 ### Key Configuration Parameters:
 
-#### Data Source Parameters:
-- **`pid_to_deviceid_map`**: CSV file mapping participant IDs to device IDs
+#### Mode Selection:
+- **`MODE`**: `"manual"` (fixed time range for all participants) or `"auto"` (per-participant, driven by survey timestamps)
+
+#### Manual Mode Parameters (used when `MODE: "manual"`):
 - **`P_IDs`**: List of Participant IDs to process
 - **`START_TIME` / `END_TIME`**: Time range for data processing (YYYY-MM-DD HH:MM:SS format)
+- **`output_file`**: Path for the main narrative output (supports `{P_ID}`, `{START_TIME}`, `{END_TIME}` placeholders)
+
+#### Auto Mode Parameters (used when `MODE: "auto"`):
+- **`survey_time_file`**: CSV file with `pid`, `survey_id`, `survey_time_unix` columns; each row generates one set of narratives anchored to that survey's timestamp
+- **`direction`**: `"backward"` (survey_time is the end of the range, e.g. for weekly recall surveys) or `"forward"` (survey_time is the start, e.g. for cumulative tracking from a baseline)
+- **`align_to_midnight`**: If `true`, snaps the survey timestamp to midnight of that day before computing ranges
+- **`skip_survey_day`**: If `true`, excludes data from the survey's own calendar day
+- **`time_range_start` / `time_range_end`**: Auto-generates all time ranges between these two values (same unit, e.g. `7d` to `1d`); alternatively use an explicit **`time_ranges`** list (e.g. `["7d", "3d", "1d"]`). Only the smallest range is kept when multiple ranges produce identical window counts.
+- **`output_dir`**: Base directory for narrative output, organized as `output_dir/{P_ID}/{time_range}/` (supports `{P_ID}` placeholder)
+
+#### Data Source Parameters:
+- **`pid_to_deviceid_map`**: CSV file mapping participant IDs to device IDs
+- **`package_to_app_map`**: JSONL file mapping app package names to human-readable app names (required; generated by the screentext preprocessing pipeline)
 - **`timezone`**: Timezone for timestamp conversion (e.g., "Australia/Melbourne")
 - **`input_directory`**: Path to participant data folder (supports {P_ID} placeholder)
 - **`session_data_file`**: Path to sessions.jsonl file for application and keyboard analysis
@@ -215,14 +275,15 @@ system_ui_apps:
 - **`sensor_integration_time_window`**: Time window in minutes for sensor data integration
 - **`gate_time_window`**: Time window in minutes for WiFi and Bluetooth scan integration
 - **`sensors`**: List of sensors to include in analysis
+- **`num_workers`**: Number of participants to process in parallel (1 = sequential; >1 uses a `ProcessPoolExecutor`)
 
 #### Output Parameters:
-- **`output_file`**: Path for the main narrative output (supports {P_ID} placeholder)
 - **`daily_output_dir`**: Directory for daily output files (supports {P_ID} placeholder)
 - **`DISCARD_SYSTEM_UI`**: Whether to filter out system UI applications
 
 #### Location Clustering Parameters:
-- **`GOOGLE_MAP_KEY`**: (Optional) API key for Google Maps reverse geocoding
+- **`USE_GOOGLE_MAP`**: Whether to enable Google Maps reverse geocoding
+- **`GOOGLE_MAP_KEY`**: Path to a text file containing your Google API key (only read when `USE_GOOGLE_MAP` is `true`)
 - **`eps`**: DBSCAN epsilon parameter (distance threshold in radians, ~300m = 0.000047)
 - **`min_samples`**: DBSCAN minimum samples to form a cluster
 - **`night_time_start` / `night_time_end`**: Hours defining nighttime for home location identification
@@ -355,6 +416,11 @@ python aware_narrator.py
 
 This processes all sensor data according to the configuration and generates comprehensive narratives.
 
+- **Manual mode** (`MODE: "manual"`): Processes each participant in `P_IDs` over the fixed `START_TIME`–`END_TIME` range and writes a single narrative to `output_file`.
+- **Auto mode** (`MODE: "auto"`): Processes each row of `survey_time_file`, generating one narrative per participant per survey per time range (see `time_ranges`/`time_range_start`/`time_range_end`), written under `output_dir`.
+- Set `num_workers` > 1 in `config.yaml` to process multiple participants in parallel.
+- Detailed logs are written to `logs/processing.log` and `logs/{P_ID}/processing_{P_ID}.log`; console output is limited to high-level progress and errors. A `processing_summary.txt` is also written to the output directory at the end of a run.
+
 ### 7. Split Description Script
 
 Split output narratives by sensor type:
@@ -415,11 +481,13 @@ resources/
 
 The toolkit generates several types of output:
 
-- **Main narrative file**: Comprehensive narrative saved to `output_file` path
-- **Daily narratives**: Separate daily files in `daily_output_dir`
+- **Manual mode narrative**: Comprehensive narrative and matching JSON (`{P_ID}_manual.json`) saved alongside `output_file`
+- **Auto mode narratives**: One narrative and JSON file per participant/survey/time range, under `output_dir/{P_ID}/{time_range}/{P_ID}_{survey_id}_{time_range}_output.txt` (and `.json`)
+- **Daily narratives**: Separate daily `.txt`/`.json` files in `daily_output_dir`
+- **Processing summary**: `processing_summary.txt` written to the output directory, and detailed logs in `logs/`
 - **Session data**: Screen usage sessions in `step1_data/{P_ID}/sessions.jsonl`
 - **Split narratives**: Sensor-specific files in `description_split/{P_ID}/`
-- **Reverse geocoding**: Location data with address information (if Google Maps API is enabled)
+- **Reverse geocoding**: Location data with address information (if `USE_GOOGLE_MAP` is enabled)
 - **Clustering analysis**: Location clusters with home detection
 
 ## Sensor Data Analysis
@@ -441,11 +509,13 @@ The toolkit analyzes the following sensor types:
 
 - **JSONDecodeError**: Ensure `config.yaml` is properly formatted YAML (no JSON-style comments)
 - **ModuleNotFoundError**: Install dependencies using `mamba env create --file environment.yml` or `conda env create --file environment.yml`
-- **Google Maps API errors**: Ensure a valid API key is provided in `GOOGLE_MAP_KEY` (or leave empty if not using Google Maps API)
+- **Google Maps API errors**: Ensure `USE_GOOGLE_MAP` is `true` and `GOOGLE_MAP_KEY` points to a text file containing a valid API key (set `USE_GOOGLE_MAP: false` if not using Google Maps API)
+- **App package mapping errors**: The script exits at startup if `package_to_app_map` is missing or empty — generate it via the screentext preprocessing pipeline first
 - **File not found errors**: Check that participant data follows the expected directory structure
 - **Session data missing**: Run `extract_sessions.py` first to generate session boundaries
 - **Empty sensor files**: Verify JSONL files contain valid JSON objects, one per line
 - **Geocoding issues**: If you encounter geocoding errors and are using Google Maps API, follow the manual setup instructions above to update the geocoding.py file
+- **Debugging a run**: Check `logs/processing.log` (and `logs/{P_ID}/processing_{P_ID}.log`) for detailed per-participant logs — the console only shows high-level summaries and errors
 
 ## License
 This project is for research purposes. Contact the developers for usage permissions.