⚠️ WORK IN PROGRESS (WIP) ⚠️ This repository is currently under active development. All code, data, and structures are subject to continuous changes.
WalkGuide: AI-Powered Navigation
Integrated Mobile Application Project Flutter Mobile Frontend × Spring Boot Backend × OOAD
Group Members
| Name | NIM | Responsibility |
|---|---|---|
| Bambang Herlambang | 5803024019 | - |
| Jap Robertus | 5803024004 | - |
| Evan William | 5803024001 | Backend Engineer (Spring Boot API & Flutter) |
System Architecture · Tech Stack · Implementations · API Endpoints · Design Patterns · Results · Weekly Progress
Overview — WalkGuide System
Core Objective: How can we build an ultra-low latency, accessible navigation system for visually impaired users while providing real-time oversight for their guardians?
This project implements a dual-interface mobile application. The system relies on On-Device AI (TFLite / YOLOv8n) to eliminate network latency during obstacle detection, paired with a robust Spring Boot backend for secure authentication, guardian-user pairing, real-time location tracking via WebSocket, and push notifications via Firebase FCM.
Deployment: Backend is deployed on a university server at 202.46.28.160. The Flutter APK uses a dynamic server URL — no hardcoded addresses — allowing multi-device testing without rebuilding.
System Architecture
The study follows a strict three-pillar enterprise structure:
Pillar 1 — OOAD (Object-Oriented Analysis & Design): Comprehensive modeling using Use Case, Class, Sequence, and ERD diagrams. The codebase strictly implements ≥ 7 GoF Design Patterns (Builder, Singleton, Facade, Repository/Proxy, Observer, Strategy, Chain of Responsibility).
Pillar 2 — Flutter (Mobile Frontend): Implements Clean Architecture (Domain, Data, Presentation layers) with BLoC for state management. Uses Dio with interceptors for secure HTTP communication. Server URL is dynamically configured via SharedPreferences on first launch.
Pillar 3 — Spring Boot (Backend API): A layered architecture (Controller → Service → Repository) powered by Java 21. Features JWT-based Role-Based Access Control (RBAC), standardized ApiResponse envelopes, WebSocket (STOMP) for real-time data, and Firebase Admin SDK for push notifications.
Tech Stack
Backend (Spring Boot)
| Library / Tool | Version | Purpose |
|---|---|---|
| Java | 21 | Primary language |
| Spring Boot | 3.3.x | Main framework |
| Spring Security | (bundled) | Auth + RBAC |
| Spring Data JPA | (bundled) | ORM |
| Spring WebSocket (STOMP) | (bundled) | Real-time location & notification push |
| PostgreSQL Driver | (bundled) | DB connection — university server 202.46.28.160 |
| Flyway | 10.x | Database schema migration |
| JJWT | 0.11.5 | JWT access + refresh token |
| Firebase Admin SDK | 9.x | FCM push notifications |
| Agora RESTful API | - | Generate Agora RTC token for VoIP |
| Springdoc OpenAPI | 2.3.0 | Swagger UI documentation |
| Lombok | latest | Boilerplate reduction |
| JUnit 5 + Mockito | (bundled) | Unit testing |
| MockMvc + Testcontainers | 1.19.x | Integration testing with real PostgreSQL |
| JaCoCo | 0.8.x | Code coverage (target ≥ 70%) |
Flutter (Mobile)
| Package | Version | Purpose |
|---|---|---|
| flutter_bloc | 8.x | State management (sole pattern) |
| go_router | 14.x | Navigation + role-based route guards |
| dio | 5.x | HTTP client with interceptors |
| shared_preferences | 2.x | Persist dynamic server URL |
| flutter_secure_storage | 10.x | Secure JWT token storage |
| drift | 2.x | SQLite ORM for offline cache |
| tflite_flutter | 0.10.x | Run YOLOv8n on-device |
| camera | 0.10.x | Camera feed for YOLO inference |
| flutter_tts | 4.x | Text-to-Speech (ID + EN) |
| speech_to_text | 6.x | Always-listening voice commands |
| agora_rtc_engine | 6.x | VoIP call Guardian ↔ User |
| firebase_messaging | 14.x | FCM push notification receiver |
| flutter_map | 6.x | OpenStreetMap (free, no API key) |
| geolocator | 11.x | Real-time GPS |
| stomp_dart_client | 2.x | WebSocket STOMP for live tracking |
| get_it | 7.x | Service locator / dependency injection |
| dartz | 0.10.x | Either<Failure, Data> typed error handling |
| vibration | 1.x | Haptic feedback on obstacle detection |
External Services
| Service | Purpose | Cost |
|---|---|---|
| Firebase (FCM) | Push notifications | Free |
| Agora RTC | In-app VoIP audio call | 10,000 min/month free |
| University Server PostgreSQL | DB at 202.46.28.160:2002 |
Free (managed by lecturer) |
| OpenStreetMap + OSRM | Map tiles + turn-by-turn routing | Free |
| YOLOv8n (Ultralytics) | Obstacle detection model (.tflite) | Free (open source) |
Implementations
Design A — User Mode (Visually Impaired)
An accessibility-first interface that opens directly to an active camera view.
- On-Device AI: YOLOv8n obstacle detection via TFLite runs locally — zero network latency. Detects 80 COCO object classes, reports direction (LEFT/CENTER/RIGHT) and estimated distance.
- Voice Commands: 14 configurable voice commands (e.g., "Start Walkguide", "Call Guardian", "Send SOS", "Where Am I"). Always-listening
SpeechToTextwith auto-restart. - Hardware Mapping: Physical volume buttons mapped to critical actions (Vol Up = Call Guardian, Vol Down = Start WalkGuide by default).
- TTS Feedback: Queued and immediate Text-to-Speech for obstacle alerts, navigation, and screen announcements. Supports Bahasa Indonesia and English.
- SOS System: One-command emergency alert triggers high-priority FCM to Guardian with GPS coordinates.
Design B — Guardian Mode (Admin/Caregiver)
A command center dashboard for oversight and remote configuration.
- Real-time Live Map: Tracks User's GPS location via WebSocket STOMP subscription, rendered on OpenStreetMap with
flutter_map. Includes geofence circle overlay. - Remote AI Configuration: Adjusts YOLO confidence threshold, alert distance thresholds, max inference FPS, and enabled object labels.
- Voice Command Management: Edits trigger phrases and enables/disables any of the 14 voice commands remotely.
- Notifications: Sends text messages or voice notes (recorded audio) directly to the User's device.
- Geofence: Sets a geographic boundary — backend notifies Guardian via FCM when User exits.
Database Schema
Database runs on PostgreSQL at 202.46.28.160:2002. Schema is managed exclusively by Flyway migrations. spring.jpa.hibernate.ddl-auto=validate.
| Migration | Table | Status |
|---|---|---|
| V1 | users |
✅ Exists |
| V2 | Seed data | ✅ Exists |
| V3 | Guardian-User link | ✅ Exists |
| V4 | Add unique_user_id column |
🔄 Needs creation |
| V5 | pairing_relations |
🔄 Needs creation |
| V6 | activity_logs |
🔄 Needs creation |
| V7 | obstacle_logs |
🔄 Needs creation |
| V8 | location_history |
🔄 Needs creation |
| V9 | guardian_notifications |
🔄 Needs creation |
| V10 | sos_events |
🔄 Needs creation |
| V11 | user_settings |
🔄 Needs creation |
| V12 | ai_configs |
🔄 Needs creation |
| V13 | voice_command_configs |
🔄 Needs creation |
| V14 | hardware_shortcuts |
🔄 Needs creation |
| V15 | geofence_configs |
🔄 Needs creation |
| V16 | refresh_tokens |
🔄 Needs creation |
API Endpoints
26 REST endpoints across 5 groups (exam minimum: 10).
Auth — /api/v1/auth
| Method | Path | Auth | Description |
|---|---|---|---|
| GET | /ping |
None | Health check / server connection test |
| POST | /register |
None | Register Guardian or User |
| POST | /login |
None | Login, returns access + refresh token |
| POST | /refresh |
None | Refresh access token |
| POST | /logout |
✅ | Logout, invalidate refresh token |
| PUT | /fcm-token |
✅ | Update FCM device token |
Pairing — /api/v1/shared/pairing
| Method | Path | Auth | Description |
|---|---|---|---|
| POST | /invite |
GUARDIAN | Invite User by 12-char unique ID |
| POST | /respond |
USER | Accept or reject pairing invite |
| DELETE | /unpair |
✅ | Dissolve active pairing |
| GET | /status |
✅ | Get current pairing status |
Guardian — /api/v1/guardian
| Method | Path | Description |
|---|---|---|
| GET | /dashboard |
Combined home data (user status, recent logs, SOS count) |
| GET | /user-status |
Full status of paired User |
| GET | /user-location |
Last known GPS location |
| GET | /location-history |
Paginated location history |
| GET | /activity-logs |
Paginated activity logs of paired User |
| GET | /obstacle-logs |
Paginated obstacle detection logs |
| POST | /notifications/send |
Send text or voice note to User |
| GET | /sos-events |
Paginated SOS events |
| PUT | /sos/{id}/acknowledge |
Acknowledge SOS alert |
| GET/PUT | /ai-config |
Get or update AI configuration |
| GET/PUT | /voice-commands |
Get or update voice command configs |
| GET/PUT | /shortcuts |
Get or update hardware shortcut configs |
| GET/PUT | /geofence |
Get or update geofence config |
User — /api/v1/user
| Method | Path | Description |
|---|---|---|
| GET | /profile |
User profile (id, email, displayName, uniqueUserId) |
| GET/PUT | /settings |
Get or update TTS/haptic settings |
| GET | /voice-commands |
Get all voice commands (read-only) |
| GET/PUT | /shortcuts |
Get or capture hardware shortcut assignments |
| GET | /ai-config |
Get AI config (read-only) |
| POST | /location |
Send GPS update |
| POST | /obstacle |
Log detected obstacle |
| POST | /sos |
Trigger SOS alert |
| GET | /activity-logs |
Paginated own activity log |
| GET | /notifications |
Paginated notifications from Guardian |
| GET | /notifications/unread-count |
Unread notification count |
| PUT | /notifications/mark-all-read |
Mark all as read |
| PUT | /notifications/{id}/read |
Mark one as read |
| POST | /walkguide/start |
Log WalkGuide session start |
| POST | /walkguide/stop |
Log WalkGuide session stop |
Shared Call — /api/v1/shared/call
| Method | Path | Description |
|---|---|---|
| POST | /token |
Generate Agora RTC token |
| POST | /notify |
Send "Incoming Call" FCM to other party |
Design Patterns
7 GoF Design Patterns implemented (exam minimum: 4, minimum 1 per category).
| # | Category | Pattern | Location |
|---|---|---|---|
| 1 | Creational | Builder | User.java (@Builder), FcmService message construction |
| 2 | Creational | Singleton | Flutter: TtsService, YoloDetector, WebSocketService, AgoraService via GetIt |
| 3 | Structural | Facade | Flutter: VoiceCommandHandler; Backend: GuardianDashboardService |
| 4 | Structural | Repository (Proxy) | All *_repository_impl.dart — proxy between domain and data sources |
| 5 | Behavioral | Observer | BLoC pattern (BLoC = Subject, Widgets = Observers); WebSocket callbacks |
| 6 | Behavioral | Strategy | ObstacleAnalyzer direction strategy; ObstacleAlertStrategyService (backend) |
| 7 | Behavioral | Chain of Responsibility | Spring Security filter chain; Dio interceptor chain |
Metrics
| Metric | Instrument | Target |
|---|---|---|
| API Throughput | Apache JMeter / k6 | ≥ 100 req/s under load |
| API p95 Latency | Apache JMeter / k6 | < 500ms |
| UI Frame Rate | Flutter DevTools | ≥ 90% frames < 16ms |
| Code Coverage | JaCoCo | ≥ 70% (Service & Controller) |
| APK Size | flutter build apk --analyze-size |
< 50MB |
Repository Structure
🚨 DISCLAIMER: This repository is in active development. The directory tree below reflects the intended final architecture. Flutter implementation is currently pending — backend (Spring Boot) is the active development phase.
/
├── walkguide-backend/ # [Spring Boot Engine — ACTIVE]
│ ├── src/main/java/com/walkguide/
│ │ ├── config/ # SecurityConfig, WebSocketConfig, OpenApiConfig, FcmConfig, DataSeeder
│ │ ├── enums/ # UserRole, PairingStatus, ActivityLogType, VoiceCommandKey,
│ │ │ # HardwareShortcutKey, NotificationType, SosStatus
│ │ ├── entity/ # User, PairingRelation, ActivityLog, ObstacleLog,
│ │ │ # LocationHistory, GuardianNotification, SosEvent,
│ │ │ # UserSettings, AiConfig, VoiceCommandConfig,
│ │ │ # HardwareShortcut, GeofenceConfig, RefreshToken
│ │ ├── repository/ # JPA repositories for all entities
│ │ ├── dto/
│ │ │ ├── request/ # RegisterRequest, LoginRequest, InviteUserRequest,
│ │ │ │ # LocationUpdateRequest, ObstacleLogRequest,
│ │ │ │ # SendNotificationRequest, SosRequest,
│ │ │ │ # AiConfigUpdateRequest, VoiceCommandUpdateRequest,
│ │ │ │ # HardwareShortcutUpdateRequest, GeofenceConfigRequest,
│ │ │ │ # UserSettingsUpdateRequest, and more
│ │ │ └── response/ # ApiResponse (standard wrapper), AuthDataResponse,
│ │ │ # UserProfileResponse, PairingStatusResponse,
│ │ │ # ActivityLogResponse, ObstacleLogResponse,
│ │ │ # LocationResponse, NotificationResponse,
│ │ │ # SosEventResponse, AgoraTokenResponse,
│ │ │ # AiConfigResponse, VoiceCommandResponse,
│ │ │ # HardwareShortcutResponse, GeofenceResponse
│ │ ├── service/ # AuthService, PairingService, ActivityLogService,
│ │ │ # LocationService, ObstacleLogService,
│ │ │ # NotificationService, SosService,
│ │ │ # AiConfigService, VoiceCommandService,
│ │ │ # HardwareShortcutService, GeofenceService,
│ │ │ # UserSettingsService, FcmService,
│ │ │ # AgoraTokenService, GuardianDashboardService
│ │ ├── controller/ # AuthController, PairingController,
│ │ │ # GuardianController, UserController, CallController
│ │ ├── security/ # JwtUtil, JwtAuthFilter, CustomUserDetailsService
│ │ ├── websocket/ # LocationBroadcaster (STOMP broadcaster)
│ │ └── exception/ # GlobalExceptionHandler, ResourceNotFoundException,
│ │ # UnauthorizedException, PairingException
│ ├── src/main/resources/
│ │ ├── application.properties # DB config (PostgreSQL 202.46.28.160:2002), Flyway, JWT
│ │ ├── firebase/
│ │ │ └── google-services-admin.json # FCM service account key (gitignored!)
│ │ └── db/migration/
│ │ ├── V1__create_users_table.sql ✅ Exists
│ │ ├── V2__seed_users.sql ✅ Exists
│ │ ├── V3__link_guardian_user.sql ✅ Exists
│ │ ├── V4__add_unique_user_id.sql 🔄 Needs creation
│ │ ├── V5__create_pairing_relations.sql 🔄 Needs creation
│ │ ├── V6__create_activity_logs.sql 🔄 Needs creation
│ │ ├── V7__create_obstacle_logs.sql 🔄 Needs creation
│ │ ├── V8__create_location_history.sql 🔄 Needs creation
│ │ ├── V9__create_notifications.sql 🔄 Needs creation
│ │ ├── V10__create_sos_events.sql 🔄 Needs creation
│ │ ├── V11__create_user_settings.sql 🔄 Needs creation
│ │ ├── V12__create_ai_configs.sql 🔄 Needs creation
│ │ ├── V13__create_voice_commands.sql 🔄 Needs creation
│ │ ├── V14__create_hardware_shortcuts.sql 🔄 Needs creation
│ │ ├── V15__create_geofence_configs.sql 🔄 Needs creation
│ │ └── V16__create_refresh_tokens.sql 🔄 Needs creation
│ └── src/test/java/com/walkguide/
│ ├── service/ # AuthServiceTest, PairingServiceTest,
│ │ # NotificationServiceTest, SosServiceTest,
│ │ # LocationServiceTest, AiConfigServiceTest
│ └── controller/ # AuthControllerTest, GuardianControllerTest,
│ # UserControllerTest (MockMvc + Testcontainers)
│
├── walkguide-mobile/ # [Flutter Frontend — PENDING]
│ ├── lib/
│ │ ├── main.dart # Firebase init, GetIt setup, camera init
│ │ ├── app/
│ │ │ ├── app.dart # MultiBlocProvider + MaterialApp.router
│ │ │ ├── router.dart # GoRouter: /server-connect → /splash → role-based routes
│ │ │ └── injection_container.dart # GetIt singletons & factories
│ │ ├── core/
│ │ │ ├── constants/ # app_constants.dart (dynamic BASE_URL via SharedPreferences),
│ │ │ │ # route_constants.dart, voice_command_keys.dart
│ │ │ ├── errors/ # failures.dart, exceptions.dart
│ │ │ ├── network/ # api_client.dart (Dio), auth_interceptor.dart,
│ │ │ │ # error_interceptor.dart, network_info.dart
│ │ │ ├── storage/ # secure_storage.dart (JWT), local_database.dart (Drift)
│ │ │ ├── services/ # tts_service.dart, stt_service.dart,
│ │ │ │ # voice_command_handler.dart,
│ │ │ │ # hardware_shortcut_listener.dart,
│ │ │ │ # fcm_service.dart, haptic_service.dart,
│ │ │ │ # websocket_service.dart, agora_service.dart
│ │ │ ├── ai/ # model_loader.dart, yolo_detector.dart,
│ │ │ │ # obstacle_analyzer.dart
│ │ │ ├── theme/ # app_theme.dart, app_colors.dart
│ │ │ └── utils/ # permission_manager.dart, location_service.dart
│ │ ├── features/
│ │ │ ├── server_connect/ # ServerConnectScreen — FIRST SCREEN on fresh install
│ │ │ │ # BLoC: TestConnection, SaveAndContinue, ChangeServer
│ │ │ ├── auth/ # SplashScreen, LoginScreen, RegisterScreen
│ │ │ │ # Clean Arch: domain / data / presentation
│ │ │ ├── pairing/ # UserPairingScreen (show unique ID, accept invite)
│ │ │ │ # GuardianPairingScreen (invite by 12-char ID)
│ │ │ ├── walk_guide/ # WalkGuideScreen (full-screen camera + YOLO overlay)
│ │ │ │ # WalkGuideBloc: Start → camera stream → detect → TTS
│ │ │ ├── sos/ # SosScreen (large SOS button)
│ │ │ ├── activity_log/ # ActivityLogScreen (paginated, filterable)
│ │ │ ├── notifications/ # NotificationScreen + TTS read-all feature
│ │ │ ├── navigation_mode/ # NavigationModeScreen (OSM map + OSRM routing)
│ │ │ ├── settings/ # UserSettingsScreen (TTS, pairing, account)
│ │ │ ├── call/ # CallScreen + IncomingCallScreen (Agora VoIP)
│ │ │ ├── manual/ # ManualScreen (all voice commands & shortcuts)
│ │ │ └── guardian_dashboard/ # GuardianDashboardScreen, GuardianMapScreen,
│ │ │ # GuardianActivityLogScreen, GuardianSendNotifScreen,
│ │ │ # GuardianAiConfigScreen, GuardianVoiceCmdScreen,
│ │ │ # GuardianShortcutScreen, GuardianGeofenceScreen
│ │ └── shared/widgets/ # AppBottomNav, VoiceCommandListener,
│ │ # TtsAwareScaffold, LoadingOverlay, ErrorWidget
│ ├── assets/
│ │ ├── models/
│ │ │ ├── yolov8n.tflite # YOLOv8n converted to TFLite (~6MB)
│ │ │ └── labels.txt # 80 COCO object labels
│ │ └── images/
│ └── pubspec.yaml
│
├── ooad-docs/ # [Design Artifacts]
│ ├── diagrams/ # PlantUML / draw.io exports
│ └── Traceability_Matrix.md
│
└── README.md
Feature Flows (High-Level)
Flow 1 — Register & Pairing
Guardian registers → User registers (gets a 12-char uniqueUserId) → Guardian enters User's ID → backend sends FCM invite → User accepts in app → backend seeds 14 default voice commands, 5 hardware shortcuts, and AI config for the pair.
Flow 2 — WalkGuide Active Detection
User says "Start Walkguide" (or presses Volume Down) → camera starts → frames are throttled to max inference FPS → YOLOv8n detects obstacles → direction and distance are analyzed → TTS announces "Caution! Person ahead center. Very close. Please stop." → haptic feedback triggers → obstacle logged to backend → GPS location broadcast via WebSocket to Guardian's live map every 5 seconds.
Flow 3 — VoIP Call
User says "Call Guardian" → Agora token fetched from backend → FCM "Incoming Call" sent to Guardian → Guardian accepts in IncomingCallScreen → both join same Agora RTC channel → audio call connects.
Flow 4 — Guardian Sends Notification → User Hears via TTS
Guardian sends text/voice note → backend saves notification + sends FCM → User receives in foreground via WebSocket → flutter_local_notifications shown + TTS announced → User says "Read All My Notifications" → TTS reads each message → marked as read.
Flow 5 — SOS Alert
User says "Send SOS" (or hardware shortcut) → GPS captured → backend saves SOS event → high-priority FCM sent to Guardian → Guardian map opens to User's location → Guardian taps Acknowledge → FCM sent back to User → TTS "Guardian is on the way."
Flow 6 — Geofence
Guardian sets center + radius on map → User's location updates checked against Haversine distance → on exit: FCM to Guardian + GEOFENCE_EXIT activity log.
Quick Start (WIP)
# 1. Clone the repository
git clone https://github.com/YourGroup/walkguide-final-exam.git
# 2. Run Backend (Spring Boot)
cd walkguide-backend
./mvnw spring-boot:run
# Flyway auto-migrates V1–V16 on first start
# Swagger UI: http://202.46.28.160:8080/swagger-ui.html
# Health check: GET http://202.46.28.160:8080/api/v1/auth/ping
# 3. Run Mobile (Flutter) - Connect to local or university backend
cd ../walkguide-mobile
flutter pub get
flutter run
# On first launch: enter server URL in ServerConnectScreen
# Example: http://202.46.28.160:8080
Results
⏳ Work In Progress: Results are populated as benchmarking phases are completed.
| Metric | Baseline | Final Optimized | Status |
|---|---|---|---|
| Cold Start Time | Pending | Pending | ⏳ |
| Memory Leak (10 Navs) | Pending | Pending | ⏳ |
| API Error Rate | Pending | Pending | ⏳ |
| YOLO Inference Latency (ms) | Pending | Pending | ⏳ |
| API p95 Latency | Pending | Pending | ⏳ |
| JaCoCo Coverage | Pending | Pending | ⏳ |
Weekly Progress
| Week | Target | Status |
|---|---|---|
| 1 | Topic proposal, Use Case definitions, Repo setup | ✅ Done |
| 2–3 | OOAD diagrams, OpenAPI YAML drafted | 🔄 In Progress |
| 4 | Spring Boot: Auth, Pairing, Entity, Migration V4–V16 | 🔄 In Progress |
| 5 | Spring Boot: Location, SOS, Notification, WS, FCM, Agora | ⏳ Pending |
| 6 | Spring Boot: Unit + Integration tests, JaCoCo ≥ 70% | ⏳ Pending |
| 7 | Flutter: ServerConnect, Auth, WalkGuide + YOLO pipeline | ⏳ Pending |
| 8 | Flutter: Guardian dashboard, Call, SOS, Notifications | ⏳ Pending |
| 9 | Feature freeze, integration testing, benchmark on device | ⏳ Pending |
| 10 | Final benchmarks, Report writing, Demo Video | ⏳ Pending |
Core Objectives
O₁ (Performance): The dual-platform architecture must maintain a 60fps UI frame rate while handling concurrent backend requests without exceeding 500ms latency. YOLO inference runs on a separate isolate to avoid blocking the UI thread.
O₂ (Accessibility): The User mode must require zero visual interaction post-login, relying entirely on haptics, voice commands, and physical button mapping. All screen transitions are announced via TTS.
O₃ (Traceability): Every major feature implementation must be directly traceable back to the pre-development OOAD artifacts and GoF design patterns.
O₄ (Configurability): All AI sensitivity settings, voice command phrases, and hardware shortcuts are remotely configurable by the Guardian without requiring an app update.
License
Distributed under the MIT License.
Final Exam: Integrated Mobile Application Project
Flutter × Spring Boot × Object-Oriented Analysis and Design