.. _core_modules: Core Modules ============ Graphical User Interface (GUI) - `gui/` --------------------------------------- The GUI is built using PyQt6 and provides a user-friendly way to interact with the ADAS. * :doc:`gui/main_window.py <../../gui/main_window>`: * The main application window that houses all other UI pages using a ``QStackedWidget``. * Manages navigation between different views (mode selection, configuration, ADAS view, logs, Q&A). * Connects signals from the ADAS backend (e.g., new frames, status updates) to GUI elements. * :doc:`gui/mode_selection.py <../../gui/mode_selection>`: * The initial page allowing the user to choose between "Real Driving Mode," "Carla Simulation Mode," or the "Q&A (Traffic Code Chatbot)." * :doc:`gui/real_config.py <../../gui/real_config>` & :doc:`gui/carla_config.py <../../gui/carla_config>`: * Configuration pages for the selected mode. * **Real Mode:** Configure left/right camera sources and camera intrinsic/extrinsic parameters (focal length, principal point, baseline). Saves to ``config/real_config.yaml``. * **CARLA Mode:** Configure CARLA host, port, and parameters for two virtual cameras (image size, FOV, location). Saves to ``config/carla_config.yaml``. * :doc:`gui/adas_view.py <../../gui/adas_view>`: * Displays the processed video feed from the ADAS system. * Shows visual alerts (e.g., "Collision Warning," "Lane Departure") overlaid on the video. * Handles keyboard input for controlling the ego vehicle in CARLA mode (W, A, S, D). * :doc:`gui/logs_view.py <../../gui/logs_view>`: * Displays status messages and logs generated by the ADAS backend. * :doc:`gui/qa_page.py <../../gui/qa_page>`: * Provides an interface for the Moroccan Traffic Code Q&A chatbot. Users can type questions and receive answers. * :doc:`gui/theme.py <../../gui/theme>`: * Defines the dark theme stylesheet (``DARK_THEME``) used across the application for a consistent look and feel. .. figure:: /_images/mode_selection.png :alt: GUI Mode Selection Page GUI Mode Selection Page .. figure:: /_images/config_window.png :alt: GUI Config View Page GUI Config View Page ADAS Backend - `main.py` ------------------------ The ``ADAS`` class in :doc:`main.py <../../main>` is the heart of the backend system. * **``ADAS`` Class:** * Manages the overall state of the ADAS (running, mode, configuration). * Initializes the ``FrameProcessor`` for perception tasks. * Handles communication with the GUI through PyQt signals (``update_frame``, ``status_update``, ``play_alert``). * Controls the main processing loop. * **Processing Loop (``process_loop``):** * Runs in a separate thread to prevent freezing the GUI. * Continuously reads frames from the selected video source. * Passes frames to the ``FrameProcessor``. * Emits processed frames and alerts to the GUI. * If in CARLA mode, applies control commands (throttle, brake, steer) to the ego vehicle. * **Video Source Initialization:** * ``init_video_source()``: Calls mode-specific initialization. * ``init_real_mode()``: Initializes ``cv2.VideoCapture`` for left and right physical cameras based on paths/IDs from ``real_config.yaml``. * ``init_carla_mode()``: * Connects to the CARLA server using ``CarlaClient``. * Spawns an ego vehicle. * Spawns additional vehicles for traffic simulation. * Sets up stereo cameras on the ego vehicle based on ``carla_config.yaml``. * Listens to camera sensors to receive image data. * **Frame Reading (``read_frame``, ``read_real_frame``, ``read_carla_frame``):** * Abstracts frame acquisition based on the current mode. * **CARLA Integration:** * Manages the ``CarlaClient`` instance. * Spawns and destroys CARLA actors (vehicle, sensors). * Applies ``carla.VehicleControl`` based on user input or internal logic. Frame Processor - `src/process_fame.py` --------------------------------------- The ``FrameProcessor`` class in :doc:`src/process_fame.py <../../src/process_fame>` orchestrates the different perception modules. * **``FrameProcessor`` Class:** * Loads configuration from ``config/config.yaml`` which specifies model paths and parameters for perception modules. * Initializes: * ``ObjectDetector`` * ``UltrafastLaneDetector`` * ``ObjectTracker`` * (Commented out) ``CREStereo`` for disparity estimation. * (Commented out) ``TrafficSignDetector``. * **``process_frame(left_img, right_img)`` Method:** * Takes left and right input images. * Performs object detection on the left image. * Performs lane detection. * Estimates disparity ``CREStereo``. * Updates the ``ObjectTracker`` with new detections. * Attaches 3D coordinates to tracks if disparity is available. * Generates a visualization of the detections/tracks on the output image. * Returns the processed image and any alerts. * **Helper Functions:** * ``detections_from_ultralytics()``: Converts YOLO detection results into a format suitable for the ``ObjectTracker``. * ``visualize_tracks()``: Draws bounding boxes, track IDs, and (if available) 3D distances on an image. * ``attach_xyz_to_tracks()``: Calculates and attaches 3D world coordinates (X, Y, Z) to tracks using a disparity map and the camera's Q matrix. Perception Modules - `src/perception/` -------------------------------------- These modules perform specific visual perception tasks. * :doc:`src/perception/object_detection.py <../../src/perception/object_detection>`: * ``ObjectDetector`` class. * Uses a YOLO model for detecting objects. * ``detect()``: Takes an image and returns detection results. * ``visualize()``: Draws detections on an image. * :doc:`src/perception/lane_detection.py <../../src/perception/lane_detection>`: * ``UltrafastLaneDetector`` class. * Implements a lightweight lane detection model. * ``detect_lanes()``: Gets lane points. * ``draw_lanes()``: Visualizes detected lanes. * :doc:`src/perception/disparity_estimation.py <../../src/perception/disparity_estimation>`: * ``CREStereo`` class. * Designed for estimating disparity maps from stereo camera pairs. * ``estimate_disp()``: Returns a disparity map. .. figure:: /_images/object_detection_example.png :alt: Object Detection Example Object Detection Example (Placeholder - replace with your actual image) .. image:: /_images/lane_detection_example.png :alt: Lane Detection Example Lane Detection Example (Placeholder - replace with your actual image) .. image:: /_images/disparity_map_example.png :alt: Disparity Map Example Disparity Map Example (Placeholder - replace with your actual image) Object Tracker - `src/tracker.py` --------------------------------- Provides functionality to track objects over multiple frames. See :doc:`src/tracker.py <../../src/tracker>`. * **``Track`` Class:** Represents a single tracked object. * **``ObjectTracker`` Class:** Implements a simple SORT-like algorithm. * ``update(detections)``: Associates new detections with existing tracks . CARLA Integration - `carla_module/` ----------------------------------- Facilitates interaction with the CARLA simulator. * :doc:`carla_module/carla_client.py <../../carla_module/carla_client>`: * ``CarlaCameraConfig`` class: Holds configuration for CARLA cameras. * ``CarlaClient`` class: Manages connection and spawning of cameras. NLP Chatbot - `NLP/` -------------------- Implements the Q&A functionality. See :doc:`NLP/adas_chatbot.py <../../NLP/adas_chatbot>`. .. figure:: /_images/nlp.png :alt: GUI Mode Selection Page * **``NLTKDownloader``, ``DataPreprocessor``, ``NeuralNet``, ``ModelTrainer``, ``Chatbot`` classes.** * Uses ``NLP/data/moroccan_traffic_code.json`` for intent data. * Training is automatic if model files (``NLP/models/processed_data_pytorch.pkl``, ``NLP/models/qa_nn_model.pth``) are missing. Calibration - `src/calibration.py` ---------------------------------- Intended for stereo camera calibration. See :doc:`src/calibration.py <../../src/calibration>`. * **Purpose:** Determine intrinsic and extrinsic camera parameters. * ``calibrate_stereo_cameras()``, ``save_calibration()``, ``load_calibration()``. * *Note: This module is more of a standalone utility.* Configuration Files - `config/` ------------------------------- YAML files for managing configurations. * ``config.yaml``: Main perception model configurations. * ``carla_config.yaml``: CARLA mode settings. * ``real_config.yaml``: Real driving mode settings. * ``kitti.yaml``: Dataset configuration for YOLO training.