feat: integrate YOLO runtime, update screens, and sync docs
This commit is contained in:
parent
aa3dc4560e
commit
6ca89ca896
@ -1,29 +1,20 @@
|
||||
# WalkGuide Traceability Audit
|
||||
|
||||
## Use Case To Code
|
||||
Dokumen audit utama Phase 1C sudah dibuat di:
|
||||
|
||||
| Use Case | Flutter Entry | Backend Entry | Status |
|
||||
|---|---|---|---|
|
||||
| Register/Login | `features/auth/*` | `AuthController`, `AuthService` | Implemented |
|
||||
| Pair Guardian/User | `features/pairing/pairing_screens.dart` | `PairingController`, `PairingService` | Implemented |
|
||||
| Start/Stop WalkGuide | `features/walk_guide/walk_guide_screen.dart` | `POST /user/walkguide/start`, `POST /user/walkguide/stop` | Implemented |
|
||||
| Obstacle Detection | `core/ai/*`, `walk_guide_screen.dart` | `POST /user/obstacle` | Partially implemented; real `.tflite` model file still required |
|
||||
| Location Tracking | `LocationReporterService`, navigation/walk screens | `LocationService`, `LocationBroadcaster` | Implemented |
|
||||
| SOS | `features/sos/sos_screen.dart` | `SosService`, `GuardianController` | Implemented |
|
||||
| Notifications | `features/notifications/notification_screen.dart` | `NotificationService`, `FcmService` | Partially implemented; backend FCM is log-only without credentials |
|
||||
| Guardian Dashboard | `guardian_dashboard_screen.dart` | `GuardianDashboardService` | Implemented |
|
||||
| Call | `features/call/call_screen.dart`, `CallService` | `CallController`, `AgoraTokenService` | Partially implemented; Agora App ID/credential required for live call |
|
||||
| Geofence | guardian dashboard screens | `GeofenceService` | Implemented |
|
||||
`ooad-docs/design-audit/Phase1C_Design_Traceability_Audit_WalkGuide.md`
|
||||
|
||||
## Design Deviations
|
||||
Gunakan file tersebut sebagai sumber utama untuk bagian laporan:
|
||||
|
||||
- Flutter uses a pragmatic mix of BLoC/Cubit, ChangeNotifier, and StatefulWidget. This deviates from strict full-BLoC architecture but keeps feature screens small for the exam demo.
|
||||
- Offline storage uses `SharedPreferences` queue instead of a full Drift database for cached entities. The dependency exists, but production-grade SQLite cache is not fully wired.
|
||||
- Backend FCM service is log-only until Firebase Admin credentials are provided.
|
||||
- Agora call flow is implemented at API and service level, but live RTC depends on a real Agora App ID.
|
||||
- OOAD - Design Traceability Audit
|
||||
- Class-to-code mapping
|
||||
- Design pattern audit
|
||||
- Sequence diagram trace
|
||||
- Design deviation log
|
||||
- Audit summary and sign-off
|
||||
|
||||
## Remaining Evidence
|
||||
Catatan penting:
|
||||
|
||||
- Export rendered OOAD diagrams from the PlantUML files in this folder.
|
||||
- Run backend tests and archive JaCoCo/Surefire reports.
|
||||
- Run Flutter benchmarks on a physical Android device and add screenshots/results.
|
||||
- Real YOLO/TFLite berjalan di Android/native.
|
||||
- Chrome/Web memakai fallback demo karena `tflite_flutter` bergantung pada `dart:ffi`, yang tidak tersedia di Web.
|
||||
- Backend test, Flutter test, Web build, dan Android debug APK build sudah lulus pada audit terakhir.
|
||||
|
||||
@ -0,0 +1,586 @@
|
||||
# Phase 1C - Design Traceability Audit
|
||||
|
||||
## WalkGuide - Post-Development OOAD Audit
|
||||
|
||||
**Project:** WalkGuide AI
|
||||
**Domain:** AI-assisted navigation and guardian monitoring for visually impaired users
|
||||
**Stack:** Flutter, Spring Boot, PostgreSQL, WebSocket STOMP, on-device YOLO/TFLite
|
||||
**Audit Basis:** `FULL_FLOW_ARCHITECTURE.md`, `ketentuan.md`, `Phase1C_Design_Traceability_Audit_Guide.pdf`, and current source code
|
||||
**Audit Status:** Conditional Pass - functionally integrated, with documented design deviations
|
||||
|
||||
---
|
||||
|
||||
## 0. Audit Method
|
||||
|
||||
This audit follows the Phase 1C guide:
|
||||
|
||||
1. Map designed classes to actual source files.
|
||||
2. Verify GoF design patterns with exact file and line references.
|
||||
3. Trace sequence diagram flows into real method call chains.
|
||||
4. Document all design deviations with rationale and impact.
|
||||
5. Summarize evidence, risks, and required follow-up actions.
|
||||
|
||||
Reproducibility commands used:
|
||||
|
||||
```powershell
|
||||
rg --files walkguide-backend\demo\src\main\java walkguide-mobile\walkguide_app\lib
|
||||
rg -n "class |interface |enum |@Entity|@RestController|@Service|@Repository|extends Cubit|BlocProvider|registerLazySingleton" walkguide-backend\demo\src\main\java walkguide-mobile\walkguide_app\lib
|
||||
rg -n "@(Get|Post|Put|Delete)Mapping|@RequestMapping" walkguide-backend\demo\src\main\java\com\walkguide\controller
|
||||
```
|
||||
|
||||
Validation commands executed after the latest implementation:
|
||||
|
||||
```powershell
|
||||
flutter analyze
|
||||
flutter test
|
||||
flutter build web --no-pub
|
||||
flutter build apk --debug --no-pub
|
||||
.\mvnw.cmd test
|
||||
```
|
||||
|
||||
Results:
|
||||
|
||||
| Area | Command | Result |
|
||||
|---|---|---|
|
||||
| Flutter static analysis | `flutter analyze` | Passed, no issues |
|
||||
| Flutter automated tests | `flutter test` | Passed, 291 tests |
|
||||
| Flutter Web build | `flutter build web --no-pub` | Passed |
|
||||
| Flutter Android build | `flutter build apk --debug --no-pub` | Passed |
|
||||
| Spring Boot tests | `.\mvnw.cmd test` | BUILD SUCCESS, 331 tests, 0 failures, 0 errors, 56 skipped |
|
||||
|
||||
---
|
||||
|
||||
## 1. Class Traceability Matrix
|
||||
|
||||
Status legend:
|
||||
|
||||
| Status | Meaning |
|
||||
|---|---|
|
||||
| OK | Designed class exists in code and matches intent |
|
||||
| RENAMED/MERGED | Design intent exists but class name or boundary changed |
|
||||
| ADDED | Implementation class added beyond original design |
|
||||
| PARTIAL | Exists but with reduced scope or platform limitation |
|
||||
| MISSING | Designed class is not implemented |
|
||||
|
||||
### 1.1 Backend Domain Entities and Database Tables
|
||||
|
||||
| Design Class / Table | Status | Code File | Evidence | Notes |
|
||||
|---|---|---|---|---|
|
||||
| `User` / `users` | OK | `walkguide-backend/demo/src/main/java/com/walkguide/entity/User.java` | `User.java:13`, `V1__create_users_table.sql:4` | Uses `role` string for `ROLE_GUARDIAN` and `ROLE_USER`; `uniqueUserId` is only for user role. |
|
||||
| `PairingRelation` / `pairing_relations` | OK | `entity/PairingRelation.java` | `PairingRelation.java:14`, `V5__create_pairing_relations.sql:5` | Represents guardian-user pairing and active/pending/rejected state. |
|
||||
| `ActivityLog` / `activity_logs` | OK | `entity/ActivityLog.java` | `ActivityLog.java:14`, `V6__create_activity_logs.sql:2` | Stores system/user activity history. |
|
||||
| `ObstacleLog` / `obstacle_logs` | OK | `entity/ObstacleLog.java` | `ObstacleLog.java:13`, `V7__create_obstacle_logs.sql:2` | Stores YOLO label, confidence, direction, distance, and location. |
|
||||
| `LocationHistory` / `location_history` | OK | `entity/LocationHistory.java` | `LocationHistory.java:13`, `V8__create_location_history.sql:2` | Supports location timeline and guardian map. |
|
||||
| `GuardianNotification` / `guardian_notifications` | OK | `entity/GuardianNotification.java` | `GuardianNotification.java:14`, `V9__create_guardian_notifications.sql:2` | Stores text/voice-note notifications. |
|
||||
| `SosEvent` / `sos_events` | OK | `entity/SosEvent.java` | `SosEvent.java:14`, `V10__create_sos_events.sql:2` | State flow: `TRIGGERED -> ACKNOWLEDGED -> RESOLVED`. |
|
||||
| `UserSettings` / `user_settings` | OK | `entity/UserSettings.java` | `UserSettings.java:13`, `V11__create_user_settings.sql:2` | TTS, haptic, and warning preferences. |
|
||||
| `AiConfig` / `ai_configs` | OK | `entity/AiConfig.java` | `AiConfig.java:13`, `V12__create_ai_configs.sql:2` | Guardian-configurable AI threshold and labels. |
|
||||
| `VoiceCommandConfig` / `voice_command_configs` | OK | `entity/VoiceCommandConfig.java` | `VoiceCommandConfig.java:14`, `V13__create_voice_command_configs.sql:2` | Stores configurable voice commands. |
|
||||
| `HardwareShortcut` / `hardware_shortcuts` | OK | `entity/HardwareShortcut.java` | `HardwareShortcut.java:14`, `V14__create_hardware_shortcuts.sql:2` | Stores key/button shortcuts. |
|
||||
| `GeofenceConfig` / `geofence_configs` | OK | `entity/GeofenceConfig.java` | `GeofenceConfig.java:13`, `V15__create_geofence_configs.sql:2` | Stores guardian geofence radius and center. |
|
||||
| `RefreshToken` / `refresh_tokens` | OK | `entity/RefreshToken.java` | `RefreshToken.java:13`, `V16__create_refresh_tokens.sql:2` | Supports access-token refresh and logout invalidation. |
|
||||
| `UserRole` enum | RENAMED/MERGED | `entity/User.java` | `User.java:26` | Architecture described `UserRole.java`, but implementation stores role as `String`. See DEV-002. |
|
||||
|
||||
### 1.2 Backend Repositories
|
||||
|
||||
| Design Repository | Status | Code File | Evidence | Notes |
|
||||
|---|---|---|---|---|
|
||||
| `UserRepository` | OK | `repository/UserRepository.java` | `UserRepository.java:9` | Extends `JpaRepository<User, Long>`. |
|
||||
| `PairingRelationRepository` | OK | `repository/PairingRelationRepository.java` | `PairingRelationRepository.java:10` | Pairing lookup by guardian/user/status. |
|
||||
| `ActivityLogRepository` | OK | `repository/ActivityLogRepository.java` | `ActivityLogRepository.java:13` | Paged log queries. |
|
||||
| `ObstacleLogRepository` | OK | `repository/ObstacleLogRepository.java` | `ObstacleLogRepository.java:8` | Paged obstacle history. |
|
||||
| `LocationHistoryRepository` | OK | `repository/LocationHistoryRepository.java` | `LocationHistoryRepository.java:9` | Last location and history queries. |
|
||||
| `GuardianNotificationRepository` | OK | `repository/GuardianNotificationRepository.java` | `GuardianNotificationRepository.java:9` | Unread count and notification list. |
|
||||
| `SosEventRepository` | OK | `repository/SosEventRepository.java` | `SosEventRepository.java:8` | User/guardian SOS history. |
|
||||
| `UserSettingsRepository` | OK | `repository/UserSettingsRepository.java` | `UserSettingsRepository.java:7` | One settings record per user. |
|
||||
| `AiConfigRepository` | OK | `repository/AiConfigRepository.java` | `AiConfigRepository.java:7` | One AI config per user. |
|
||||
| `VoiceCommandConfigRepository` | OK | `repository/VoiceCommandConfigRepository.java` | `VoiceCommandConfigRepository.java:9` | Voice command lookup. |
|
||||
| `HardwareShortcutRepository` | OK | `repository/HardwareShortcutRepository.java` | `HardwareShortcutRepository.java:7` | Shortcut lookup. |
|
||||
| `GeofenceConfigRepository` | OK | `repository/GeofenceConfigRepository.java` | `GeofenceConfigRepository.java:7` | Geofence lookup. |
|
||||
| `RefreshTokenRepository` | OK | `repository/RefreshTokenRepository.java` | `RefreshTokenRepository.java:7` | Refresh token lookup/deletion. |
|
||||
|
||||
### 1.3 Backend Services
|
||||
|
||||
| Design Service | Status | Code File | Evidence | Notes |
|
||||
|---|---|---|---|---|
|
||||
| `AuthService` | OK | `service/AuthService.java` | `AuthService.java:31`, `register:63`, `login:116`, `refreshToken:131` | Handles auth, token refresh, logout, FCM token update. |
|
||||
| `PairingService` | OK | `service/PairingService.java` | `PairingService.java:19`, `inviteUser:30`, `respondToPairing:71` | Handles invite, respond, unpair, and status. |
|
||||
| `ActivityLogService` | OK | `service/ActivityLogService.java` | `ActivityLogService.java:18` | Creates and retrieves activity logs. |
|
||||
| `LocationService` | OK | `service/LocationService.java` | `LocationService.java:24`, `updateLocation:36`, `haversineMeters:115` | Persists GPS updates and checks geofence distance. |
|
||||
| `ObstacleLogService` | OK | `service/ObstacleLogService.java` | `ObstacleLogService.java:18`, `saveObstacle:24` | Persists obstacle detection events. |
|
||||
| `NotificationService` | OK | `service/NotificationService.java` | `NotificationService.java:26`, `sendNotification:36` | Sends and reads guardian notifications. |
|
||||
| `SosService` | OK | `service/SosService.java` | `SosService.java:26`, `triggerSos:38`, `acknowledgeSos:87` | Persists SOS, sends FCM/log notification, broadcasts WebSocket. |
|
||||
| `AiConfigService` | OK | `service/AiConfigService.java` | `AiConfigService.java:17`, `getConfig:23`, `updateConfigByGuardian:29` | Guardian changes AI config for paired user. |
|
||||
| `VoiceCommandService` | OK | `service/VoiceCommandService.java` | `VoiceCommandService.java:20` | Reads and updates voice commands. |
|
||||
| `HardwareShortcutService` | OK | `service/HardwareShortcutService.java` | `HardwareShortcutService.java:15` | Reads and updates hardware shortcuts. |
|
||||
| `GeofenceService` | OK | `service/GeofenceService.java` | `GeofenceService.java:24` | Reads and updates geofence settings. |
|
||||
| `UserSettingsService` | OK | `service/UserSettingsService.java` | `UserSettingsService.java:12` | Reads and updates user accessibility settings. |
|
||||
| `FcmService` | PARTIAL | `service/FcmService.java` | `FcmService.java:17`, `sendToToken:19`, `sendHighPriority:46` | Currently log-only until Firebase Admin credentials are configured. See DEV-004. |
|
||||
| `AgoraTokenService` | PARTIAL | `service/AgoraTokenService.java` | `AgoraTokenService.java:34` | Token API exists; live token depends on Agora app id/certificate. |
|
||||
| `GuardianDashboardService` | OK | `service/GuardianDashboardService.java` | `GuardianDashboardService.java:13`, `getDashboard:22` | Facade for guardian dashboard aggregate data. |
|
||||
| `MockDataService` | ADDED | `service/MockDataService.java` | `MockDataService.java` | Supports tests/demo data; not part of original domain design. |
|
||||
| `ObstacleAlertStrategyService` | MISSING | - | `rg "ObstacleAlertStrategyService"` returns none | Architecture mentioned backend alert strategy, but current alert strategy is implemented on Flutter side. See DEV-005. |
|
||||
|
||||
### 1.4 Backend Controllers and API Entry Points
|
||||
|
||||
| Design Controller | Status | Code File | Evidence | Notes |
|
||||
|---|---|---|---|---|
|
||||
| `AuthController` | OK | `controller/AuthController.java` | `@RequestMapping:14`, `login:32`, `register:26` | `/api/v1/auth` routes. |
|
||||
| `PairingController` | OK | `controller/PairingController.java` | `@RequestMapping:13`, `invite:19`, `respond:28` | `/api/v1/shared/pairing` routes. |
|
||||
| `UserController` | OK | `controller/UserController.java` | `@RequestMapping:19`, `logObstacle:101`, `triggerSos:109` | `/api/v1/user` routes. |
|
||||
| `GuardianController` | OK | `controller/GuardianController.java` | `@RequestMapping:16`, `dashboard:32`, `sendNotif:77` | `/api/v1/guardian` routes. |
|
||||
| `CallController` | OK | `controller/CallController.java` | `@RequestMapping:38`, `token:55`, `notify:75`, `end:127` | `/api/v1/shared/call` routes. |
|
||||
| `GET /guardian/user-status` | RENAMED/MERGED | `GuardianController.dashboard()` | `GuardianController.java:32` | The explicit architecture endpoint is not present; status data is covered by dashboard and pairing/status. See DEV-006. |
|
||||
|
||||
### 1.5 Backend Security and Infrastructure
|
||||
|
||||
| Design Class | Status | Code File | Evidence | Notes |
|
||||
|---|---|---|---|---|
|
||||
| `SecurityConfig` | OK | `config/SecurityConfig.java` | `SecurityConfig.java:23`, `filterChain:33` | RBAC routes for auth, guardian, user, shared. |
|
||||
| `JwtAuthFilter` | OK | `security/JwtAuthFilter.java` | `JwtAuthFilter.java:19`, `doFilterInternal:24` | Chain of Responsibility filter before controller access. |
|
||||
| `JwtUtil` | OK | `security/JwtUtil.java` | `JwtUtil.java:19` | Token generation and validation. |
|
||||
| `SecurityHelper` | ADDED | `security/SecurityHelper.java` | `SecurityHelper.java:11` | Helper to get current authenticated user id. |
|
||||
| `WebSocketConfig` | OK | `config/WebSocketConfig.java` | `WebSocketConfig.java:25` | STOMP broker setup. |
|
||||
| `LocationBroadcaster` | OK | `websocket/LocationBroadcaster.java` | `LocationBroadcaster.java:25`, `broadcastLocation:37`, `broadcastSos:51` | Observer-style real-time broadcast. |
|
||||
| `OpenApiConfig` | OK | `config/OpenApiConfig.java` | `OpenApiConfig.java:12` | Swagger/OpenAPI config. |
|
||||
| `GlobalExceptionHandler` | OK | `exception/GlobalExceptionHandler.java` | `GlobalExceptionHandler.java:11` | Central error handling. |
|
||||
| `DataSeeder` | OK | `config/DataSeeder.java` | `DataSeeder.java:12` | Seeds default data if needed. |
|
||||
| `CustomUserDetailsService` | MISSING | - | `rg "CustomUserDetailsService"` returns none | JWT auth stores user id in authentication credentials directly. No separate `UserDetailsService` was required. |
|
||||
|
||||
### 1.6 Flutter Core, AI, and Services
|
||||
|
||||
| Design Class | Status | Code File | Evidence | Notes |
|
||||
|---|---|---|---|---|
|
||||
| `ApiClient` | OK | `lib/core/network/api_client.dart` | `ApiClient:5`, `init:11`, interceptors `22-26` | Dynamic base URL and Dio interceptor chain. |
|
||||
| `_AuthInterceptor` | OK | `lib/core/network/api_client.dart` | `_AuthInterceptor:37`, `onRequest:45`, `onError:54` | Injects JWT and refreshes token on 401. |
|
||||
| `_ErrorInterceptor` | OK | `lib/core/network/api_client.dart` | `_ErrorInterceptor:95` | Error pass-through/normalization boundary. |
|
||||
| `SecureStorage` | OK | `lib/core/storage/secure_storage.dart` | `SecureStorage:5` | Stores token, role, user id, display name. |
|
||||
| `TtsService` | OK | `lib/core/services/tts_service.dart` | `TtsService:3` | Text-to-speech service. |
|
||||
| `SttService` | OK | `lib/core/services/stt_service.dart` | `SttService:5` | Speech-to-text service. |
|
||||
| `HapticService` | OK | `lib/core/services/haptic_service.dart` | `HapticService:3` | Haptic feedback wrapper. |
|
||||
| `VoiceCommandHandler` | OK | `lib/core/services/voice_command_handler.dart` | `VoiceCommandHandler:23`, `_processText:56`, `_handleCommand:67` | Facade for STT result matching and TTS commands. |
|
||||
| `WebSocketService` | OK | `lib/core/services/websocket_service.dart` | `WebSocketService:19`, `subscribeLocation:98`, `subscribeSos:127`, `subscribeNotification:152` | Observer subscriptions for live events. |
|
||||
| `FcmService` | PARTIAL | `lib/core/services/fcm_service.dart` | `FcmService:7`, `put /auth/fcm-token:37` | Client token registration exists; live push depends on backend Firebase credentials. |
|
||||
| `LocationReporterService` | OK | `lib/core/services/location_reporter_service.dart` | `LocationReporterService:10`, `_sendOnce:31` | Sends periodic GPS updates; uses offline queue fallback. |
|
||||
| `OfflineQueueService` | ADDED | `lib/core/services/offline_queue_service.dart` | `OfflineQueueService:35` | Pragmatic SharedPreferences queue; not full Drift implementation. |
|
||||
| `CallService` | OK | `lib/core/services/call_service.dart` | `CallService:7`, `/shared/call/notify:36` | Backend call notify integration. |
|
||||
| `ObstacleAnalyzer` | OK | `lib/core/ai/obstacle_analyzer.dart` | `ObstacleAnalyzer:46`, `analyzeDirection:50`, `estimateDistance:57` | Strategy-like obstacle interpretation logic. |
|
||||
| `YoloDetector` | OK | `lib/core/ai/yolo_detector.dart` | `YoloDetector:11`, `init:27`, `detect:55`, `_decodeDetections:243` | Runs camera input through YOLO runtime on native targets. |
|
||||
| `YoloRuntime` | ADDED | `lib/core/ai/yolo_runtime.dart` | `export:1` | Conditional export for native TFLite vs Web stub. |
|
||||
| `YoloRuntimeNative` | ADDED | `lib/core/ai/yolo_runtime_native.dart` | `YoloRuntime:7`, `Interpreter.fromAsset:27` | Native TFLite runtime. |
|
||||
| `YoloRuntimeStub` | ADDED | `lib/core/ai/yolo_runtime_stub.dart` | `YoloRuntime:5` | Web-safe fallback; avoids `dart:ffi` compile error. See DEV-001. |
|
||||
|
||||
### 1.7 Flutter Feature Screens and State Management
|
||||
|
||||
| Design Element | Status | Code File | Evidence | Notes |
|
||||
|---|---|---|---|---|
|
||||
| `WalkGuideApp` | OK | `lib/app/app.dart` | `WalkGuideApp:8`, `BlocProvider:15` | Root app provides `AppCubit`. |
|
||||
| `AppCubit` | OK | `lib/app/app_cubit.dart` | `AppCubit:19`, `emit:23` | BLoC/Cubit state management is present. |
|
||||
| `Router` | OK | `lib/app/router.dart` | `router.dart` | GoRouter route definitions. |
|
||||
| `LoginScreen` | OK | `features/auth/login_screen.dart` | `_login:62`, `POST /auth/login:69` | Auth UI flow. |
|
||||
| `RegisterScreen` | OK | `features/auth/register_screen.dart` | `_register:47`, `POST /auth/register:56` | Registration UI flow. |
|
||||
| `UserPairingScreen` | OK | `features/pairing/pairing_screens.dart` | `UserPairingScreen:22`, respond:250 | User pairing response UI. |
|
||||
| `GuardianPairingScreen` | OK | `features/pairing/pairing_screens.dart` | `GuardianPairingScreen:90`, invite:110 | Guardian invite UI. |
|
||||
| `WalkGuideScreen` | OK | `features/walk_guide/walk_guide_screen.dart` | `WalkGuideScreen:24`, stream:87, detect:121, log:148 | Live camera + YOLO integration. |
|
||||
| `AiBenchmarkScreen` | OK | `features/walk_guide/walk_guide_screen.dart` | `AiBenchmarkScreen:244`, `detectSynthetic:299` | AI benchmark/demo path. |
|
||||
| `SosScreen` | OK | `features/sos/sos_screen.dart` | `_sendSos:179`, `POST /user/sos:183` | User SOS flow. |
|
||||
| `NotificationScreen` | OK | `features/notifications/notification_screen.dart` | `_load:34`, `GET /user/notifications:41` | User notification flow. |
|
||||
| `GuardianDashboardScreen` | OK | `features/home/presentation/guardian_dashboard_screen.dart` | `_loadAll:85`, `GET /guardian/dashboard:181` | Guardian dashboard. |
|
||||
| `GuardianAiConfigScreen` | OK | `features/guardian_dashboard/guardian_ai_config_screen.dart` | `_load:42`, `_save:86` | Guardian AI config UI. |
|
||||
| `ActivityLogScreen` | OK | `features/activity_log/activity_log_screen.dart` | `_load:42`, `GET /user/activity-logs:49` | User activity logs. |
|
||||
| `NavigationModeScreen` | OK | `features/navigation_mode/navigation_mode_screen.dart` | `POST /user/location:95` | Navigation/location mode. |
|
||||
| Strict feature-first Clean Architecture | PARTIAL | `features/auth/*`, several direct screen files | `auth/data`, `auth/domain`, `auth/presentation`; direct calls in feature screens | Auth follows domain/data split. Several features are pragmatic screen-first with direct API calls. See DEV-003. |
|
||||
| BLoC-only state management | PARTIAL | `app/app_cubit.dart`, multiple `StatefulWidget` screens | `AppCubit:19`, many screens use local `setState` | Cubit is used globally; feature state is mostly local `StatefulWidget`. See DEV-003. |
|
||||
|
||||
---
|
||||
|
||||
## 2. Design Pattern Audit
|
||||
|
||||
### 2.1 Builder Pattern - Creational
|
||||
|
||||
| Audit Field | Evidence |
|
||||
|---|---|
|
||||
| Design Intent | Create complex entity/DTO objects without long constructors. |
|
||||
| Implementation | Lombok `@Builder` on entities and response DTOs. |
|
||||
| Files | `entity/User.java:10`, `entity/AiConfig.java:10`, `dto/response/AuthDataResponse.java:6`, `dto/response/DashboardResponse.java:11`, `dto/response/SosEventResponse.java:15`. |
|
||||
| Client Usage | Services return `DashboardResponse.builder()...build()` in `GuardianDashboardService.java:27` and `:44`; SOS builds `SosEvent.builder()` in `SosService.java:39`. |
|
||||
| Verdict | OK. Builder is correctly used across backend entity/response construction. |
|
||||
|
||||
### 2.2 Singleton / Service Locator - Creational
|
||||
|
||||
| Audit Field | Evidence |
|
||||
|---|---|
|
||||
| Design Intent | Resource-heavy app services should have one shared lifecycle. |
|
||||
| Implementation | GetIt `registerLazySingleton` in Flutter DI. |
|
||||
| Files | `lib/app/injection_container.dart:21-35`. |
|
||||
| Registered Singletons | `SecureStorage`, `ApiClient`, `TtsService`, `SttService`, `HapticService`, `ObstacleAnalyzer`, `YoloDetector`, `FcmService`, `WebSocketService`, `LocationReporterService`, `CallService`, `VoiceCommandHandler`. |
|
||||
| Verdict | OK. Services are resolved from DI instead of constructed repeatedly in screens. |
|
||||
|
||||
### 2.3 Facade Pattern - Structural
|
||||
|
||||
| Audit Field | Evidence |
|
||||
|---|---|
|
||||
| Design Intent | Hide multiple lower-level service calls behind a simpler feature-facing API. |
|
||||
| Backend Implementation | `GuardianDashboardService.getDashboard()` aggregates pairing, location, activity logs, SOS count, and notification count. |
|
||||
| Backend Evidence | `GuardianDashboardService.java:13`, `getDashboard:22`, location call at `:37`, activity call at `:38`, counts at `:42-43`. |
|
||||
| Flutter Implementation | `VoiceCommandHandler` hides STT result matching and TTS-only commands behind `loadCommands`, `loadDefaultCommands`, and callback dispatch. |
|
||||
| Flutter Evidence | `voice_command_handler.dart:23`, `_processText:56`, `_handleCommand:67`. |
|
||||
| Verdict | OK. Both implementations reduce caller complexity. |
|
||||
|
||||
### 2.4 Repository Pattern - Structural
|
||||
|
||||
| Audit Field | Evidence |
|
||||
|---|---|
|
||||
| Design Intent | Service layer depends on persistence contracts, not raw SQL. |
|
||||
| Backend Implementation | Spring Data `JpaRepository` interfaces. |
|
||||
| Files | `UserRepository.java:9`, `PairingRelationRepository.java:10`, `ActivityLogRepository.java:13`, `ObstacleLogRepository.java:8`, and other repositories under `repository/`. |
|
||||
| Flutter Implementation | Auth feature has `AuthRepository` and `AuthRepositoryImpl`. |
|
||||
| Flutter Evidence | `features/auth/domain/auth_repository.dart:5`, `features/auth/data/auth_repository_impl.dart:8`. |
|
||||
| Verdict | OK for backend and Auth feature. PARTIAL for other Flutter features that still call `ApiClient` directly from screens. See DEV-003. |
|
||||
|
||||
### 2.5 Observer Pattern - Behavioral
|
||||
|
||||
| Audit Field | Evidence |
|
||||
|---|---|
|
||||
| Design Intent | Subscribers receive state/event updates without polling. |
|
||||
| Flutter Cubit | `AppCubit` emits state; `WalkGuideApp` provides it via `BlocProvider`. |
|
||||
| Flutter Evidence | `app_cubit.dart:19`, `emit:23`, `app.dart:15`. |
|
||||
| WebSocket Subject | Backend `LocationBroadcaster` pushes to STOMP destinations. |
|
||||
| Backend Evidence | `LocationBroadcaster.java:37` for `/topic/location/{userId}`, `:51` for `/queue/sos/{guardianId}`, `:65` for `/queue/notif/{userId}`. |
|
||||
| Flutter Observers | `WebSocketService.subscribeLocation`, `subscribeSos`, `subscribeNotification`. |
|
||||
| Flutter Evidence | `websocket_service.dart:98`, `:127`, `:152`. |
|
||||
| Verdict | OK. Observer behavior exists in both UI state and real-time messaging. |
|
||||
|
||||
### 2.6 Strategy Pattern - Behavioral
|
||||
|
||||
| Audit Field | Evidence |
|
||||
|---|---|
|
||||
| Design Intent | Obstacle interpretation should be isolated from camera/model runtime. |
|
||||
| Implementation | `ObstacleAnalyzer` encapsulates direction, distance, priority, and confidence filtering rules. |
|
||||
| Files | `obstacle_analyzer.dart:46`, `analyzeDirection:50`, `estimateDistance:57`, `prioritize:75`, `filterByConfidence:98`. |
|
||||
| Client Usage | `YoloDetector` delegates post-processing interpretation to analyzer at `yolo_detector.dart:66-67` and `:485-486`. |
|
||||
| Verdict | PARTIAL. Strategy-like separation exists in Flutter, but the architecture's backend `ObstacleAlertStrategyService` is not implemented. See DEV-005. |
|
||||
|
||||
### 2.7 Chain of Responsibility - Behavioral
|
||||
|
||||
| Audit Field | Evidence |
|
||||
|---|---|
|
||||
| Design Intent | Requests pass through ordered handlers for authentication, refresh, and error handling. |
|
||||
| Flutter Implementation | Dio interceptors are registered in order: `_AuthInterceptor`, `_ErrorInterceptor`, `LogInterceptor`. |
|
||||
| Flutter Evidence | `api_client.dart:22-26`, `_AuthInterceptor:37`, `onRequest:45`, `onError:54`, `_ErrorInterceptor:95`. |
|
||||
| Backend Implementation | Spring Security filter chain inserts `JwtAuthFilter` before `UsernamePasswordAuthenticationFilter`. |
|
||||
| Backend Evidence | `SecurityConfig.java:33`, `.addFilterBefore(...):53`, `JwtAuthFilter.java:24`. |
|
||||
| Verdict | OK. Request processing is ordered and composable. |
|
||||
|
||||
---
|
||||
|
||||
## 3. Sequence Diagram Trace
|
||||
|
||||
### 3.1 UC-01 Login and Route to Role Dashboard
|
||||
|
||||
Designed flow: User -> Login UI -> Flutter API client -> AuthController -> AuthService -> UserRepository/JwtUtil -> Flutter storage/router.
|
||||
|
||||
| # | Diagram Step | Actual Code Trace | Status |
|
||||
|---|---|---|---|
|
||||
| 1 | User submits credentials | `LoginScreen._login()` at `features/auth/login_screen.dart:62` | OK |
|
||||
| 2 | Flutter calls backend login | `dio.post('/auth/login')` at `features/auth/login_screen.dart:69` | OK |
|
||||
| 3 | Backend receives request | `AuthController.login()` at `AuthController.java:32` | OK |
|
||||
| 4 | Controller delegates service | `authService.login(req)` at `AuthController.java:35` | OK |
|
||||
| 5 | Service validates user/password | `AuthService.login()` at `AuthService.java:116` | OK |
|
||||
| 6 | Service returns token response | `AuthDataResponse.builder()` in `AuthService.java` | OK |
|
||||
| 7 | Flutter stores tokens and routes | `_saveAuthAndRoute()` at `features/auth/login_screen.dart:195` | OK |
|
||||
| 8 | App state set | `AppCubit.loadSession()` / `emit()` at `app_cubit.dart:23` | OK |
|
||||
|
||||
### 3.2 UC-02 Guardian Invites User for Pairing
|
||||
|
||||
Designed flow: Guardian -> Pairing UI -> PairingController -> PairingService -> repositories -> FCM/log notification -> response.
|
||||
|
||||
| # | Diagram Step | Actual Code Trace | Status |
|
||||
|---|---|---|---|
|
||||
| 1 | Guardian enters unique user id | `GuardianPairingScreen` at `pairing_screens.dart:90` | OK |
|
||||
| 2 | Flutter sends invite | `dio.post('/shared/pairing/invite')` at `pairing_screens.dart:110` | OK |
|
||||
| 3 | Backend receives invite | `PairingController.invite()` at `PairingController.java:19` | OK |
|
||||
| 4 | Service validates and creates relation | `PairingService.inviteUser()` at `PairingService.java:30` | OK |
|
||||
| 5 | Service sends notification | `fcmService.sendToToken(...)` at `PairingService.java:59` | PARTIAL - FCM is log-only until credentials are configured |
|
||||
| 6 | User accepts/rejects | `dio.post('/shared/pairing/respond')` at `pairing_screens.dart:250` -> `PairingController.respond()` at `PairingController.java:28` | OK |
|
||||
| 7 | Service updates status | `PairingService.respondToPairing()` at `PairingService.java:71` | OK |
|
||||
|
||||
### 3.3 UC-03 Start WalkGuide and Log Obstacle
|
||||
|
||||
Designed flow: User -> WalkGuide UI -> Camera stream -> YOLO detector -> ObstacleAnalyzer -> TTS/Haptic -> API -> UserController -> ObstacleLogService -> database.
|
||||
|
||||
| # | Diagram Step | Actual Code Trace | Status |
|
||||
|---|---|---|---|
|
||||
| 1 | User starts WalkGuide | `_toggle()` at `walk_guide_screen.dart:51` | OK |
|
||||
| 2 | Camera image stream starts | `controller.startImageStream(_onCameraImage)` at `walk_guide_screen.dart:87` | OK |
|
||||
| 3 | Frame callback triggered | `_onCameraImage()` at `walk_guide_screen.dart:109` | OK |
|
||||
| 4 | Run YOLO detection | `sl<YoloDetector>().detect(image)` at `walk_guide_screen.dart:121` | OK on native/mobile |
|
||||
| 5 | Native TFLite runtime loads model | `Interpreter.fromAsset()` at `yolo_runtime_native.dart:27` | OK on native/mobile |
|
||||
| 6 | Decode model output | `_decodeDetections()` at `yolo_detector.dart:243` | OK |
|
||||
| 7 | Analyze direction/distance | `analyzeDirection()` and `estimateDistance()` at `obstacle_analyzer.dart:50` and `:57` | OK |
|
||||
| 8 | Log obstacle to backend | `dio.post('/user/obstacle')` at `walk_guide_screen.dart:148` | OK |
|
||||
| 9 | Backend receives obstacle | `UserController.logObstacle()` at `UserController.java:101` | OK |
|
||||
| 10 | Backend persists obstacle | `ObstacleLogService.saveObstacle()` at `ObstacleLogService.java:24` | OK |
|
||||
| 11 | TTS alert | `TtsService.speakImmediate()` at `walk_guide_screen.dart:158` | OK |
|
||||
|
||||
Important platform note: on Chrome/Web, `YoloRuntimeStub` is selected and the app falls back to a demo detection. This is why the Web demo can show `person CENTER Close` repeatedly. Real YOLO runs only on Android/native. See DEV-001.
|
||||
|
||||
### 3.4 UC-04 User Sends SOS and Guardian Receives Alert
|
||||
|
||||
Designed flow: User -> SOS UI -> UserController -> SosService -> SosEventRepository -> FCM + WebSocket -> Guardian dashboard.
|
||||
|
||||
| # | Diagram Step | Actual Code Trace | Status |
|
||||
|---|---|---|---|
|
||||
| 1 | User presses SOS | `SosScreen._sendSos()` at `sos_screen.dart:179` | OK |
|
||||
| 2 | Flutter posts SOS | `dio.post('/user/sos')` at `sos_screen.dart:183` | OK |
|
||||
| 3 | Backend receives request | `UserController.triggerSos()` at `UserController.java:109` | OK |
|
||||
| 4 | Service creates SOS event | `SosService.triggerSos()` at `SosService.java:38` | OK |
|
||||
| 5 | SOS persisted | `sosEventRepository.save(sos)` in `SosService.java:46` | OK |
|
||||
| 6 | Activity log created | `activityLogService.createLog(...)` in `SosService.java:52` | OK |
|
||||
| 7 | Guardian notified by FCM | `fcmService.sendHighPriority(...)` in `SosService.java:67` | PARTIAL - log-only FCM |
|
||||
| 8 | Guardian notified by WebSocket | `locationBroadcaster.broadcastSos(...)` in `SosService.java:79` | OK |
|
||||
| 9 | Guardian acknowledges | `GuardianController.acknowledgeSos()` at `GuardianController.java:94` -> `SosService.acknowledgeSos()` at `SosService.java:87` | OK |
|
||||
|
||||
### 3.5 UC-05 Guardian Updates AI Configuration
|
||||
|
||||
Designed flow: Guardian -> AI Config UI -> GuardianController -> AiConfigService -> repository -> FCM/WebSocket notification -> User app reads latest config.
|
||||
|
||||
| # | Diagram Step | Actual Code Trace | Status |
|
||||
|---|---|---|---|
|
||||
| 1 | Guardian opens config | `GuardianAiConfigScreen._load()` at `guardian_ai_config_screen.dart:42` | OK |
|
||||
| 2 | Flutter fetches config | `GET /guardian/ai-config` at `guardian_ai_config_screen.dart:59` | OK |
|
||||
| 3 | Guardian saves config | `_save()` at `guardian_ai_config_screen.dart:86` | OK |
|
||||
| 4 | Flutter sends update | `PUT /guardian/ai-config` at `guardian_ai_config_screen.dart:89` | OK |
|
||||
| 5 | Backend receives update | `GuardianController.updateAiConfig()` at `GuardianController.java:109` | OK |
|
||||
| 6 | Service validates pairing and saves | `AiConfigService.updateConfigByGuardian()` at `AiConfigService.java:29` | OK |
|
||||
| 7 | User notified | `fcmService.sendToToken(...)` at `AiConfigService.java:48` | PARTIAL - log-only FCM |
|
||||
|
||||
---
|
||||
|
||||
## 4. Design Deviation Log
|
||||
|
||||
### DEV-001 - YOLO Native Runtime vs Web Fallback
|
||||
|
||||
| Field | Content |
|
||||
|---|---|
|
||||
| Type | Platform / AI runtime |
|
||||
| Design Element | In-device YOLOv8n object detection |
|
||||
| Design Intent | Camera frame is processed by YOLO/TFLite and shows real detected obstacle. |
|
||||
| Actual Implementation | Native/mobile uses `YoloRuntimeNative` and `tflite_flutter`; Web uses `YoloRuntimeStub` through conditional export at `yolo_runtime.dart:1`. |
|
||||
| Rationale | `tflite_flutter` depends on `dart:ffi`, which cannot compile for Chrome/Web. Conditional runtime keeps Web demo buildable while preserving Android YOLO. |
|
||||
| Impact | Chrome/Web may repeatedly show fallback `person CENTER Close (1-2m)`. Real AI must be demonstrated on Android/emulator/physical device. |
|
||||
| Re-alignment Required? | Yes. Add platform note to architecture and presentation script. |
|
||||
|
||||
### DEV-002 - `UserRole` Enum Not Implemented
|
||||
|
||||
| Field | Content |
|
||||
|---|---|
|
||||
| Type | Class / type safety |
|
||||
| Design Element | `UserRole.java` enum in architecture document |
|
||||
| Design Intent | Use enum values `ROLE_GUARDIAN` and `ROLE_USER`. |
|
||||
| Actual Implementation | `User.role` is a `String` at `User.java:26`; services compare string literals such as `ROLE_USER` and `ROLE_GUARDIAN`. |
|
||||
| Rationale | Simpler compatibility with existing JWT role strings and seeded data. |
|
||||
| Impact | Lower type safety; role typo would be caught at runtime rather than compile time. |
|
||||
| Re-alignment Required? | Preferred. Either add `UserRole` enum or update class diagram to reflect `String role`. |
|
||||
|
||||
### DEV-003 - Flutter Clean Architecture Applied Partially
|
||||
|
||||
| Field | Content |
|
||||
|---|---|
|
||||
| Type | Architecture / layering |
|
||||
| Design Element | Strict feature-first Clean Architecture with `domain/data/application/presentation` for every feature |
|
||||
| Design Intent | All features have clean repository/use-case/presentation boundaries and use BLoC consistently. |
|
||||
| Actual Implementation | Auth feature has domain/data split (`AuthRepository`, `AuthRepositoryImpl`, `AuthRemoteDataSource`), but several features call `ApiClient` directly from `StatefulWidget` screens. `features/screens.dart` also keeps a monolithic compatibility screen set. |
|
||||
| Rationale | Deadline-driven integration prioritized complete working flows and demo readiness. |
|
||||
| Impact | Some UI classes contain API orchestration and `try/catch`; this should be disclosed and improved if strict grading focuses on Flutter architecture. |
|
||||
| Re-alignment Required? | Yes for ideal final refactor. For presentation, document as pragmatic deviation. |
|
||||
|
||||
### DEV-004 - FCM Backend Is Log-Only
|
||||
|
||||
| Field | Content |
|
||||
|---|---|
|
||||
| Type | External service integration |
|
||||
| Design Element | Firebase Admin SDK sends real background push notification. |
|
||||
| Design Intent | Backend sends FCM push for SOS, incoming call, pairing, settings updates. |
|
||||
| Actual Implementation | `FcmService.sendToToken()` logs payload and contains commented Firebase implementation at `FcmService.java:19-43`; `sendHighPriority()` delegates at `:46-48`. |
|
||||
| Rationale | Firebase Admin credentials are not committed and may not be available for local demo. |
|
||||
| Impact | Foreground WebSocket and UI flows work; background push requires credentials before live demo. |
|
||||
| Re-alignment Required? | Yes if lecturer asks for real push notification evidence. |
|
||||
|
||||
### DEV-005 - Backend `ObstacleAlertStrategyService` Missing
|
||||
|
||||
| Field | Content |
|
||||
|---|---|
|
||||
| Type | Pattern / service boundary |
|
||||
| Design Element | Backend Strategy pattern for obstacle alert modes |
|
||||
| Design Intent | Backend strategy chooses TTS/haptic alert behavior based on AI config. |
|
||||
| Actual Implementation | Alert behavior is handled on-device. `ObstacleAnalyzer` performs direction/distance/priority logic, and `WalkGuideScreen` triggers TTS at `walk_guide_screen.dart:158`. No backend `ObstacleAlertStrategyService` exists. |
|
||||
| Rationale | Obstacle warning must happen locally and fast, even offline. Backend only stores obstacle logs and settings. |
|
||||
| Impact | Pattern is implemented in Flutter, not backend. Architecture diagram should move this responsibility to mobile. |
|
||||
| Re-alignment Required? | Yes. Update design pattern documentation to show Flutter-side strategy. |
|
||||
|
||||
### DEV-006 - `/guardian/user-status` Endpoint Merged into Dashboard/Status APIs
|
||||
|
||||
| Field | Content |
|
||||
|---|---|
|
||||
| Type | Interface / API contract |
|
||||
| Design Element | `GET /api/v1/guardian/user-status` from architecture endpoint list |
|
||||
| Design Intent | Guardian fetches paired user status from a dedicated endpoint. |
|
||||
| Actual Implementation | `GuardianController` provides `/dashboard`, `/user-location`, `/location-history`, `/activity-logs`, `/obstacle-logs`, and pairing status via `/shared/pairing/status`. No direct `/guardian/user-status` route exists. |
|
||||
| Rationale | Dashboard endpoint already aggregates paired user data, reducing duplicate endpoint surface. |
|
||||
| Impact | Functional data is available, but OpenAPI/architecture endpoint list should be updated. |
|
||||
| Re-alignment Required? | Yes. Update endpoint table or add thin alias endpoint. |
|
||||
|
||||
### DEV-007 - Local Database/Offline Design Simplified
|
||||
|
||||
| Field | Content |
|
||||
|---|---|
|
||||
| Type | Persistence / offline-first |
|
||||
| Design Element | Drift/SQLite ORM cache for offline-first synchronization |
|
||||
| Design Intent | Persist offline records locally and sync when online. |
|
||||
| Actual Implementation | `OfflineQueueService` exists and uses a pragmatic queue, while many features still call API directly. |
|
||||
| Rationale | Narrowed implementation to support reliable demo paths first. |
|
||||
| Impact | Offline-first is partial, not full production-grade local cache. |
|
||||
| Re-alignment Required? | Yes if offline-first is claimed as a complete advanced feature. |
|
||||
|
||||
### DEV-008 - Development Database Credentials Present as Defaults
|
||||
|
||||
| Field | Content |
|
||||
|---|---|
|
||||
| Type | Configuration / security |
|
||||
| Design Element | No hardcoded secrets; environment-separated configs |
|
||||
| Design Intent | Use environment variables for DB credentials. |
|
||||
| Actual Implementation | `application-dev.yml` uses env placeholders with fallback values for the university PostgreSQL database. |
|
||||
| Rationale | Internal exam database credentials were provided for local/demo usage and kept as fallbacks. |
|
||||
| Impact | Acceptable for private/internal submission, but risky for public repository. |
|
||||
| Re-alignment Required? | Before public upload, remove fallback password and rely on `DB_URL`, `DB_USERNAME`, `DB_PASSWORD`. |
|
||||
|
||||
---
|
||||
|
||||
## 5. Cross-Pillar Consistency
|
||||
|
||||
### 5.1 OOAD Class Diagram vs Backend Entities
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| Core domain entities exist as Java classes | OK |
|
||||
| Core domain entities exist as Flyway tables | OK |
|
||||
| Pairing relationship represented in code and DB | OK |
|
||||
| SOS state represented in enum and service flow | OK |
|
||||
| User role modeled | PARTIAL - implemented as string instead of enum |
|
||||
|
||||
### 5.2 ERD vs Database Migrations
|
||||
|
||||
| Table Group | Migration Evidence | Result |
|
||||
|---|---|---|
|
||||
| Auth/User | `V1__create_users_table.sql`, `V16__create_refresh_tokens.sql` | OK |
|
||||
| Pairing | `V5__create_pairing_relations.sql` | OK |
|
||||
| WalkGuide logs | `V6__create_activity_logs.sql`, `V7__create_obstacle_logs.sql`, `V8__create_location_history.sql` | OK |
|
||||
| Notifications/SOS | `V9__create_guardian_notifications.sql`, `V10__create_sos_events.sql` | OK |
|
||||
| Settings/configuration | `V11__create_user_settings.sql`, `V12__create_ai_configs.sql`, `V13__create_voice_command_configs.sql`, `V14__create_hardware_shortcuts.sql`, `V15__create_geofence_configs.sql` | OK |
|
||||
|
||||
### 5.3 Architecture vs Flutter Implementation
|
||||
|
||||
| Requirement | Evidence | Result |
|
||||
|---|---|---|
|
||||
| BLoC/Cubit exists | `AppCubit` and `BlocProvider` | OK |
|
||||
| BLoC-only across all features | Multiple local `StatefulWidget` states | PARTIAL |
|
||||
| GoRouter routing | `lib/app/router.dart` | OK |
|
||||
| Dynamic server URL | `AppConstants.getServerUrl()` and `ApiClient.init()` | OK |
|
||||
| JWT auth and refresh | `_AuthInterceptor.onError()` refresh path | OK |
|
||||
| YOLO model asset | `assets/models/yolov8n.tflite` included in APK; APK contains `assets/flutter_assets/assets/models/yolov8n.tflite` | OK |
|
||||
| Web build compatibility | `YoloRuntimeStub` conditional import | OK |
|
||||
| Real YOLO on Chrome | Not supported by `dart:ffi` | NOT APPLICABLE - native only |
|
||||
|
||||
### 5.4 API Contract vs Controllers
|
||||
|
||||
| API Area | Controller Evidence | Result |
|
||||
|---|---|---|
|
||||
| Auth | `AuthController.java:14-52` | OK |
|
||||
| Pairing | `PairingController.java:13-44` | OK |
|
||||
| User APIs | `UserController.java:19-174` | OK |
|
||||
| Guardian APIs | `GuardianController.java:16-162` | OK, except `/guardian/user-status` merged |
|
||||
| Shared Call APIs | `CallController.java:38-129` | OK |
|
||||
| Response envelope | `ApiResponse.java:5` used by controllers | OK |
|
||||
| OpenAPI file | `src/main/resources/openapi.yaml` | OK |
|
||||
|
||||
### 5.5 Design Pattern UML vs Code Alignment
|
||||
|
||||
This section maps each submitted PlantUML pattern diagram to the current codebase. It exists because Phase 1B requires every design pattern to be documented with UML and Phase 1C requires each pattern claim to be traceable to source code.
|
||||
|
||||
| # | Pattern | UML File | UML Intent | Code Evidence | Alignment | Audit Notes |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 1 | Builder | `ooad-docs/01_Builder_Pattern.puml` | `User.builder()`, `AuthDataResponse.builder()`, and Firebase `Message.builder()` build complex objects. | `User.java:10`, `AiConfig.java:10`, `AuthDataResponse.java:6`, `DashboardResponse.java:11`, `SosService.java:39`, `GuardianDashboardService.java:27` | PARTIAL | Lombok builders are implemented. Firebase `Message.builder()` is still commented/log-only in `FcmService.java:31-39`; `Role` is implemented as `String`, not enum. |
|
||||
| 2 | Singleton | `ooad-docs/02_Singleton_Pattern.puml` | GetIt manages single app-wide instances for TTS, STT, YOLO, WebSocket, Agora/call services. | `injection_container.dart:21-35`, `YoloDetector.dart:11`, `TtsService.dart:3`, `WebSocketService.dart:19`, `CallService.dart:7` | PARTIAL | Runtime behavior is singleton/service-locator through GetIt. UML shows static `_instance` fields and `registerSingleton`, but code uses `registerLazySingleton` and no static singleton fields. |
|
||||
| 3 | Facade | `ooad-docs/03_Facade_Pattern.puml` | `VoiceCommandHandler` and `GuardianDashboardService` hide subsystem complexity. | `voice_command_handler.dart:23`, `_processText:56`, `_handleCommand:67`, `GuardianDashboardService.java:13`, `getDashboard:22` | PARTIAL | Backend facade aligns strongly. Flutter facade exists, but UML includes `GoRouter`, `WalkGuideBloc`, `SosBloc`, and `NotificationBloc`; current code uses STT/TTS plus callback dispatch instead. |
|
||||
| 4 | Repository / Proxy | `ooad-docs/04_Repository_Proxy_Pattern.puml` | Repository interfaces proxy remote/local data sources and hide online/offline selection. | Backend: `UserRepository.java:9`, `ObstacleLogRepository.java:8`; Flutter Auth: `auth_repository.dart:5`, `auth_repository_impl.dart:8`; Offline queue: `offline_queue_service.dart:35` | PARTIAL | Backend repository pattern is complete. Flutter repository/proxy exists for Auth only; diagrammed `WalkGuideRepositoryImpl`, `ActivityLogRepositoryImpl`, Drift local data sources are not implemented as shown. |
|
||||
| 5 | Observer | `ooad-docs/05_Observer_Pattern.puml` | BLoC stream and WebSocket subscribers receive updates from subjects. | `app_cubit.dart:19`, `app.dart:15`, `LocationBroadcaster.java:37`, `:51`, `:65`, `websocket_service.dart:98`, `:127`, `:152` | PARTIAL | Observer behavior is present via Cubit and STOMP WebSocket. UML shows `WalkGuideBloc`, `BlocBuilder`, and `BlocListener`; current WalkGuide UI mostly uses `StatefulWidget` local state. |
|
||||
| 6 | Strategy | `ooad-docs/06_Strategy_Pattern.puml` | `ObstacleAlertStrategy` has concrete alert strategies selected by AI config. | `obstacle_analyzer.dart:46`, `analyzeDirection:50`, `estimateDistance:57`, `prioritize:75`, `walk_guide_screen.dart:158` | PARTIAL | Strategy-like separation exists in `ObstacleAnalyzer`, but the UML's `ObstacleAlertStrategy`, `TtsOnlyStrategy`, `TtsWithHapticStrategy`, and `HapticOnlyStrategy` classes are not present. Logged as DEV-005. |
|
||||
| 7 | Chain of Responsibility | `ooad-docs/07_ChainOfResponsibility_Pattern.puml` | HTTP requests pass through Flutter Dio interceptors and backend Spring Security filters. | `api_client.dart:22-26`, `_AuthInterceptor:37`, `_ErrorInterceptor:95`, `SecurityConfig.java:53`, `JwtAuthFilter.java:24` | PARTIAL | Chain exists. UML says `ErrorInterceptor` maps HTTP errors to typed `Failure` and `JwtAuthFilter` uses `UserDetailsService`; current code passes errors through and uses `JwtUtil` directly. |
|
||||
|
||||
Alignment summary:
|
||||
|
||||
| Alignment Type | Count | Patterns |
|
||||
|---|---:|---|
|
||||
| Fully aligned with implementation intent | 0 | None are exact 1:1 because the UML still contains planned classes/method names that changed during implementation. |
|
||||
| Partially aligned and defensible | 7 | Builder, Singleton, Facade, Repository/Proxy, Observer, Strategy, Chain of Responsibility. |
|
||||
| Completely missing | 0 | Every pattern has at least one implementation counterpart, but several diagrams require update before final report export. |
|
||||
|
||||
Required UML updates:
|
||||
|
||||
| Action ID | UML File | Required Update |
|
||||
|---|---|---|
|
||||
| UML-ACT-01 | `01_Builder_Pattern.puml` | Replace `Role` enum with `String role` or implement `UserRole`; mark Firebase `Message.builder()` as planned/log-only unless FCM credentials are configured. |
|
||||
| UML-ACT-02 | `02_Singleton_Pattern.puml` | Replace static `_instance` fields with GetIt `registerLazySingleton` lifecycle. |
|
||||
| UML-ACT-03 | `03_Facade_Pattern.puml` | Update `VoiceCommandHandler` dependencies to STT, TTS, command list, and callback; remove non-existing BLoC dependencies. |
|
||||
| UML-ACT-04 | `04_Repository_Proxy_Pattern.puml` | Either implement WalkGuide/Activity repositories or mark the diagram as target architecture and show current backend repository + Auth repository only. |
|
||||
| UML-ACT-05 | `05_Observer_Pattern.puml` | Replace `WalkGuideBloc` with `AppCubit` plus `WebSocketService` subscriptions, or implement the missing WalkGuide BLoC. |
|
||||
| UML-ACT-06 | `06_Strategy_Pattern.puml` | Replace concrete alert strategy classes with current `ObstacleAnalyzer` strategy-like methods, or implement the missing strategy classes. |
|
||||
| UML-ACT-07 | `07_ChainOfResponsibility_Pattern.puml` | Update `JwtAuthFilter` dependency from `UserDetailsService` to `JwtUtil`; update `ErrorInterceptor` behavior to pass-through unless typed failure mapping is implemented. |
|
||||
|
||||
---
|
||||
|
||||
## 6. Audit Summary Dashboard
|
||||
|
||||
| Metric | Count / Status | Notes |
|
||||
|---|---|---|
|
||||
| Backend core entity classes audited | 13 | All implemented |
|
||||
| Backend repository interfaces audited | 13 | All implemented |
|
||||
| Backend service classes audited | 16 actual + 1 missing design service | `ObstacleAlertStrategyService` missing by design shift |
|
||||
| Backend controllers audited | 5 | All implemented |
|
||||
| Flutter core service/AI classes audited | 17 | Includes conditional YOLO runtime files |
|
||||
| Flutter feature screen groups audited | 10+ | All major user/guardian flows present |
|
||||
| GoF patterns audited | 7 | Builder, Singleton, Facade, Repository, Observer, Strategy, Chain of Responsibility |
|
||||
| Patterns fully implemented | 5 | Builder, Singleton, Facade, Observer, Chain of Responsibility |
|
||||
| Patterns partially implemented | 2 | Repository in Flutter partial; Strategy moved to Flutter/mobile |
|
||||
| Design pattern UML files aligned to code | 7 partial | All 7 UML diagrams have implementation counterparts, but each needs a small update for exact final-code alignment |
|
||||
| Sequence flows traced | 5 | Login, Pairing, WalkGuide obstacle, SOS, AI Config |
|
||||
| Documented deviations | 8 | All have rationale and action |
|
||||
| Overall verdict | CONDITIONAL PASS | Working system; update diagrams/report for documented deviations |
|
||||
|
||||
### Overall Audit Verdict
|
||||
|
||||
The WalkGuide codebase is functionally ready for demonstration. The backend follows a layered Spring Boot architecture with PostgreSQL/Flyway, JWT/RBAC, controllers, services, repositories, OpenAPI, and test coverage. The Flutter app has working authentication, pairing, WalkGuide camera/YOLO native detection, SOS, notifications, guardian dashboard, map/location, settings, and AI benchmark screens.
|
||||
|
||||
The main OOAD risks are not functional blockers, but documentation and architecture-alignment issues:
|
||||
|
||||
1. Chrome/Web cannot run real TFLite YOLO and therefore uses fallback demo detection.
|
||||
2. Flutter Clean Architecture and BLoC are partial outside the auth/app shell layer.
|
||||
3. FCM backend is log-only until Firebase Admin credentials are configured.
|
||||
4. A few design elements were merged or moved, especially `UserRole`, `/guardian/user-status`, and backend alert strategy.
|
||||
|
||||
### Required Actions Before Final Submission
|
||||
|
||||
| ID | Action | Owner | Due |
|
||||
|---|---|---|---|
|
||||
| ACT-01 | Update presentation script: real YOLO must be shown on Android, not Chrome. | Flutter Engineer | Before demo |
|
||||
| ACT-02 | Update class diagram: `UserRole` is either added as enum or shown as `String role`. | OOAD Lead | Before report export |
|
||||
| ACT-03 | Update sequence diagram UC-03 to include platform branch: native YOLO vs Web fallback. | OOAD Lead | Before report export |
|
||||
| ACT-04 | Update API endpoint table: `/guardian/user-status` is merged into dashboard/status APIs, or add alias endpoint. | Backend Engineer | Before final freeze |
|
||||
| ACT-05 | Mark FCM as log-only unless Firebase Admin credentials are configured. | Backend Engineer | Before demo |
|
||||
| ACT-06 | Avoid claiming full offline-first/Drift unless local cache implementation is expanded. | Report Owner | Before submission |
|
||||
| ACT-07 | Archive or replace stale audit docs that mention classes not present in current source. | OOAD Lead | Before submission |
|
||||
|
||||
---
|
||||
|
||||
This document is produced as the OOAD Phase 1C traceability audit deliverable and should be stored with the original design artifacts under `ooad-docs/design-audit/`.
|
||||
@ -193,12 +193,12 @@
|
||||
{\fontsize{16}{20}\selectfont AI-Powered Navigation for the Visually Impaired}
|
||||
|
||||
\vspace{2cm}
|
||||
\begin{tcolorbox}[colback=white!15, colframe=white!40, arc=8pt, width=12cm]
|
||||
\centering\color{white}
|
||||
\begin{tcolorbox}[colback=white, colframe=primaryblue, arc=8pt, width=12cm]
|
||||
\centering\color{black}
|
||||
\begin{tabular}{ll}
|
||||
\textbf{Kelompok} & 08 \\[4pt]
|
||||
\textbf{Anggota 1} & Evan \\
|
||||
\textbf{Anggota 2} & Japson \\
|
||||
\textbf{Anggota 2} & Jap \\
|
||||
\textbf{Anggota 3} & Bambang \\[4pt]
|
||||
\textbf{Mata Kuliah} & Object-Oriented Analysis and Design \\
|
||||
\textbf{Tanggal} & 19 Mei 2026 \\
|
||||
@ -1309,29 +1309,6 @@ Each segment displays:
|
||||
|
||||
The existing \code{GET /guardian/location-history} endpoint (paginated, ordered by \code{created\_at DESC}) provides the raw data. The timeline grouping logic runs client-side in the Guardian Flutter app.
|
||||
|
||||
% ═══════════════════════════════════════════════════════════════════════════
|
||||
\chapter{Team Contribution}
|
||||
% ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Team Member Responsibilities}
|
||||
\label{tab:team}
|
||||
\begin{tabularx}{\textwidth}{L{2.5cm} L{4cm} L{7cm}}
|
||||
\toprule
|
||||
\textbf{Member} & \textbf{Primary Role} & \textbf{Contributions} \\
|
||||
\midrule
|
||||
Evan & Backend Engineer & Spring Boot architecture, all 14 service classes, Flyway migrations V4--V16, JWT security, WebSocket broadcaster, k6 load tests, JaCoCo coverage, OpenAPI YAML \\
|
||||
\midrule
|
||||
Japson & Flutter Engineer & Flutter Clean Architecture setup, GoRouter configuration, WalkGuide YOLO pipeline, TTS/STT services, voice command handler, Guardian dashboard screens, Agora call integration \\
|
||||
\midrule
|
||||
Bambang & OOAD Lead & All 6 OOAD diagram types (PlantUML), design pattern documentation, traceability audit, report authoring, Flutter unit/widget tests, integration test flows \\
|
||||
\bottomrule
|
||||
\end{tabularx}
|
||||
\end{table}
|
||||
|
||||
All three members contributed commits to both the Flutter and Spring Boot repositories. OOAD artifacts were submitted before development commenced (Week 2--3 checkpoint). Contribution percentages are cross-referenced with GitHub commit history in the repository README.
|
||||
|
||||
% ═══════════════════════════════════════════════════════════════════════════
|
||||
\chapter{Conclusion}
|
||||
% ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
@ -1,6 +1,8 @@
|
||||
import 'package:go_router/go_router.dart';
|
||||
|
||||
import '../core/constants/app_constants.dart';
|
||||
import '../features/activity_log/activity_log_screen.dart' as activity;
|
||||
import '../features/notifications/notification_screen.dart' as notifications;
|
||||
import '../features/screens.dart';
|
||||
import '../shared/widgets/app_shells.dart';
|
||||
|
||||
@ -38,10 +40,10 @@ final GoRouter appRouter = GoRouter(
|
||||
GoRoute(path: '/user/sos', builder: (_, __) => const SosScreen()),
|
||||
GoRoute(
|
||||
path: '/user/activity',
|
||||
builder: (_, __) => const ActivityLogScreen()),
|
||||
builder: (_, __) => const activity.ActivityLogScreen()),
|
||||
GoRoute(
|
||||
path: '/user/notifications',
|
||||
builder: (_, __) => const NotificationScreen()),
|
||||
builder: (_, __) => const notifications.NotificationScreen()),
|
||||
GoRoute(
|
||||
path: '/user/navigation',
|
||||
builder: (_, __) => const NavigationModeScreen()),
|
||||
|
||||
@ -1,19 +1,32 @@
|
||||
import 'dart:math' as math;
|
||||
|
||||
import 'package:camera/camera.dart';
|
||||
import 'package:flutter/foundation.dart';
|
||||
import 'package:flutter/services.dart';
|
||||
|
||||
import '../constants/app_constants.dart';
|
||||
import 'obstacle_analyzer.dart';
|
||||
import 'yolo_runtime.dart';
|
||||
|
||||
class YoloDetector {
|
||||
final ObstacleAnalyzer _analyzer;
|
||||
List<String> _labels = const [];
|
||||
YoloRuntime? _runtime;
|
||||
List<int> _inputShape = const [];
|
||||
List<int> _outputShape = const [];
|
||||
YoloTensorType? _inputType;
|
||||
YoloTensorType? _outputType;
|
||||
bool _ready = false;
|
||||
String? _lastError;
|
||||
|
||||
YoloDetector(this._analyzer);
|
||||
|
||||
bool get isReady => _ready;
|
||||
bool get isReady => _ready && _runtime != null;
|
||||
String? get lastError => _lastError;
|
||||
|
||||
Future<void> init() async {
|
||||
dispose();
|
||||
_lastError = null;
|
||||
try {
|
||||
_labels = (await rootBundle.loadString(AppConstants.yoloLabelsPath))
|
||||
.split('\n')
|
||||
@ -24,24 +37,489 @@ class YoloDetector {
|
||||
_ready = false;
|
||||
return;
|
||||
}
|
||||
await rootBundle.load(await AppConstants.getSelectedYoloModelPath());
|
||||
_runtime = await YoloRuntime.load(
|
||||
await AppConstants.getSelectedYoloModelPath(),
|
||||
);
|
||||
_inputShape = _runtime!.inputShape;
|
||||
_outputShape = _runtime!.outputShape;
|
||||
_inputType = _runtime!.inputType;
|
||||
_outputType = _runtime!.outputType;
|
||||
_ready = true;
|
||||
} catch (e) {
|
||||
_lastError = e.toString();
|
||||
debugPrint('YOLO fallback mode: $e');
|
||||
_ready = false;
|
||||
}
|
||||
}
|
||||
|
||||
Future<DetectionResult?> detectFallback() async {
|
||||
if (_ready) {
|
||||
// Full tensor pre/post-processing belongs here once yolov8n.tflite is present.
|
||||
// The app keeps a deterministic fallback so demo flows remain testable.
|
||||
Future<DetectionResult?> detect(
|
||||
CameraImage image, {
|
||||
double confidenceThreshold = 0.45,
|
||||
}) async {
|
||||
if (!isReady) return detectFallback();
|
||||
try {
|
||||
final input = _buildCameraInput(image);
|
||||
final output = Uint8List(_runtime!.outputByteLength);
|
||||
_runtime!.run(input, output);
|
||||
final detections = _decodeDetections(output);
|
||||
final filtered =
|
||||
_analyzer.filterByConfidence(detections, confidenceThreshold);
|
||||
return _analyzer.prioritize(filtered);
|
||||
} catch (e) {
|
||||
_lastError = e.toString();
|
||||
debugPrint('YOLO inference fallback: $e');
|
||||
return detectFallback();
|
||||
}
|
||||
}
|
||||
|
||||
Future<DetectionResult?> detectSynthetic({
|
||||
double confidenceThreshold = 0.25,
|
||||
}) async {
|
||||
if (!isReady) return detectFallback();
|
||||
try {
|
||||
final input = _buildSyntheticInput();
|
||||
final output = Uint8List(_runtime!.outputByteLength);
|
||||
_runtime!.run(input, output);
|
||||
final detections = _decodeDetections(output);
|
||||
final filtered =
|
||||
_analyzer.filterByConfidence(detections, confidenceThreshold);
|
||||
return _analyzer.prioritize(filtered) ?? detectFallback();
|
||||
} catch (e) {
|
||||
_lastError = e.toString();
|
||||
debugPrint('YOLO synthetic fallback: $e');
|
||||
return detectFallback();
|
||||
}
|
||||
}
|
||||
|
||||
Future<DetectionResult?> detectFallback() async {
|
||||
final label = _labels.isNotEmpty ? _labels.first : 'person';
|
||||
return _analyzer.analyzeFallback(label: label);
|
||||
}
|
||||
|
||||
void dispose() {
|
||||
_runtime?.close();
|
||||
_runtime = null;
|
||||
_inputShape = const [];
|
||||
_outputShape = const [];
|
||||
_inputType = null;
|
||||
_outputType = null;
|
||||
_ready = false;
|
||||
}
|
||||
|
||||
Uint8List _buildCameraInput(CameraImage image) {
|
||||
final inputInfo = _inputInfo();
|
||||
final bytes = Uint8List(inputInfo.byteLength);
|
||||
final data = ByteData.view(bytes.buffer);
|
||||
|
||||
for (var y = 0; y < inputInfo.height; y++) {
|
||||
final srcY = (y * image.height / inputInfo.height).floor();
|
||||
for (var x = 0; x < inputInfo.width; x++) {
|
||||
final srcX = (x * image.width / inputInfo.width).floor();
|
||||
final rgb = _rgbAt(image, srcX, srcY);
|
||||
_writeInputValue(data, bytes, inputInfo, x, y, 0, (rgb >> 16) & 0xFF);
|
||||
_writeInputValue(data, bytes, inputInfo, x, y, 1, (rgb >> 8) & 0xFF);
|
||||
_writeInputValue(data, bytes, inputInfo, x, y, 2, rgb & 0xFF);
|
||||
}
|
||||
}
|
||||
|
||||
return bytes;
|
||||
}
|
||||
|
||||
Uint8List _buildSyntheticInput() {
|
||||
final inputInfo = _inputInfo();
|
||||
final bytes = Uint8List(inputInfo.byteLength);
|
||||
final data = ByteData.view(bytes.buffer);
|
||||
|
||||
for (var y = 0; y < inputInfo.height; y++) {
|
||||
for (var x = 0; x < inputInfo.width; x++) {
|
||||
final shade = ((x / inputInfo.width) * 64 + 96).round();
|
||||
_writeInputValue(data, bytes, inputInfo, x, y, 0, shade);
|
||||
_writeInputValue(data, bytes, inputInfo, x, y, 1, shade);
|
||||
_writeInputValue(data, bytes, inputInfo, x, y, 2, shade);
|
||||
}
|
||||
}
|
||||
|
||||
return bytes;
|
||||
}
|
||||
|
||||
_InputInfo _inputInfo() {
|
||||
if (_inputShape.length != 4) {
|
||||
throw StateError('Unsupported YOLO input shape: $_inputShape');
|
||||
}
|
||||
final type = _inputType;
|
||||
if (type == null) throw StateError('YOLO input type is unavailable');
|
||||
|
||||
final channelsLast = _inputShape.last == 3;
|
||||
final channelsFirst = _inputShape[1] == 3;
|
||||
if (!channelsLast && !channelsFirst) {
|
||||
throw StateError('Unsupported YOLO channel layout: $_inputShape');
|
||||
}
|
||||
if (type != YoloTensorType.float32 &&
|
||||
type != YoloTensorType.uint8 &&
|
||||
type != YoloTensorType.int8) {
|
||||
throw StateError('Unsupported YOLO input type: $type');
|
||||
}
|
||||
|
||||
final height = channelsLast ? _inputShape[1] : _inputShape[2];
|
||||
final width = channelsLast ? _inputShape[2] : _inputShape[3];
|
||||
final bytesPerValue = type == YoloTensorType.float32 ? 4 : 1;
|
||||
return _InputInfo(
|
||||
width: width,
|
||||
height: height,
|
||||
channelsLast: channelsLast,
|
||||
type: type,
|
||||
bytesPerValue: bytesPerValue,
|
||||
);
|
||||
}
|
||||
|
||||
void _writeInputValue(
|
||||
ByteData data,
|
||||
Uint8List bytes,
|
||||
_InputInfo info,
|
||||
int x,
|
||||
int y,
|
||||
int channel,
|
||||
int value,
|
||||
) {
|
||||
final elementIndex = info.channelsLast
|
||||
? ((y * info.width + x) * 3 + channel)
|
||||
: (channel * info.width * info.height + y * info.width + x);
|
||||
final byteOffset = elementIndex * info.bytesPerValue;
|
||||
|
||||
if (info.type == YoloTensorType.float32) {
|
||||
data.setFloat32(byteOffset, value / 255.0, Endian.little);
|
||||
} else if (info.type == YoloTensorType.uint8) {
|
||||
bytes[byteOffset] = value;
|
||||
} else if (info.type == YoloTensorType.int8) {
|
||||
data.setInt8(byteOffset, value - 128);
|
||||
}
|
||||
}
|
||||
|
||||
int _rgbAt(CameraImage image, int x, int y) {
|
||||
final group = image.format.group;
|
||||
if (group == ImageFormatGroup.bgra8888) {
|
||||
final plane = image.planes.first;
|
||||
final index = y * plane.bytesPerRow + x * 4;
|
||||
if (index + 2 >= plane.bytes.length) return 0;
|
||||
final b = plane.bytes[index];
|
||||
final g = plane.bytes[index + 1];
|
||||
final r = plane.bytes[index + 2];
|
||||
return (r << 16) | (g << 8) | b;
|
||||
}
|
||||
|
||||
if (image.planes.length < 3) {
|
||||
final plane = image.planes.first;
|
||||
final index = math.min(y * plane.bytesPerRow + x, plane.bytes.length - 1);
|
||||
final luma = plane.bytes[index];
|
||||
return (luma << 16) | (luma << 8) | luma;
|
||||
}
|
||||
|
||||
final yPlane = image.planes[0];
|
||||
final uPlane = image.planes[1];
|
||||
final vPlane = image.planes[2];
|
||||
|
||||
final yIndex =
|
||||
math.min(y * yPlane.bytesPerRow + x, yPlane.bytes.length - 1);
|
||||
final uvX = x ~/ 2;
|
||||
final uvY = y ~/ 2;
|
||||
final uvPixelStride = uPlane.bytesPerPixel ?? 1;
|
||||
final uvIndex = math.min(
|
||||
uvY * uPlane.bytesPerRow + uvX * uvPixelStride,
|
||||
math.min(uPlane.bytes.length, vPlane.bytes.length) - 1,
|
||||
);
|
||||
|
||||
final yy = yPlane.bytes[yIndex].toDouble();
|
||||
final uu = uPlane.bytes[uvIndex].toDouble() - 128.0;
|
||||
final vv = vPlane.bytes[uvIndex].toDouble() - 128.0;
|
||||
|
||||
final r = _clampByte(yy + 1.402 * vv);
|
||||
final g = _clampByte(yy - 0.344136 * uu - 0.714136 * vv);
|
||||
final b = _clampByte(yy + 1.772 * uu);
|
||||
return (r << 16) | (g << 8) | b;
|
||||
}
|
||||
|
||||
int _clampByte(double value) => value.round().clamp(0, 255);
|
||||
|
||||
List<DetectionResult> _decodeDetections(Uint8List outputBuffer) {
|
||||
final values = _readOutput(outputBuffer);
|
||||
if (values.isEmpty || _outputShape.isEmpty) return const [];
|
||||
|
||||
final dims = _outputShape.where((d) => d > 0).toList();
|
||||
final effectiveShape =
|
||||
dims.isNotEmpty && dims.first == 1 ? dims.sublist(1) : dims;
|
||||
|
||||
if (effectiveShape.length == 2) {
|
||||
final a = effectiveShape[0];
|
||||
final b = effectiveShape[1];
|
||||
if (a >= 5 && b > a) {
|
||||
return _decodeFeatureFirst(values, a, b);
|
||||
}
|
||||
if (b >= 5) {
|
||||
return _decodeRowFirst(values, a, b);
|
||||
}
|
||||
}
|
||||
|
||||
if (effectiveShape.length == 3 && effectiveShape[0] == 1) {
|
||||
final a = effectiveShape[1];
|
||||
final b = effectiveShape[2];
|
||||
if (a >= 5 && b > a) {
|
||||
return _decodeFeatureFirst(values, a, b);
|
||||
}
|
||||
if (b >= 5) {
|
||||
return _decodeRowFirst(values, a, b);
|
||||
}
|
||||
}
|
||||
|
||||
return const [];
|
||||
}
|
||||
|
||||
List<double> _readOutput(Uint8List bytes) {
|
||||
final type = _outputType;
|
||||
if (type == null) return const [];
|
||||
final data = ByteData.view(bytes.buffer);
|
||||
|
||||
if (type == YoloTensorType.float32) {
|
||||
return [
|
||||
for (var i = 0; i < bytes.length; i += 4)
|
||||
data.getFloat32(i, Endian.little),
|
||||
];
|
||||
}
|
||||
if (type == YoloTensorType.uint8) {
|
||||
return [for (final byte in bytes) byte / 255.0];
|
||||
}
|
||||
if (type == YoloTensorType.int8) {
|
||||
return [for (var i = 0; i < bytes.length; i++) data.getInt8(i) / 127.0];
|
||||
}
|
||||
if (type == YoloTensorType.float16) {
|
||||
// Most exported YOLOv8 TFLite files use float32 output. Float16 is rare;
|
||||
// fallback keeps the app usable if an unsupported variant is selected.
|
||||
throw StateError('Float16 YOLO output is not supported yet');
|
||||
}
|
||||
throw StateError('Unsupported YOLO output type: $type');
|
||||
}
|
||||
|
||||
List<DetectionResult> _decodeFeatureFirst(
|
||||
List<double> values,
|
||||
int featureCount,
|
||||
int boxCount,
|
||||
) {
|
||||
final detections = <DetectionResult>[];
|
||||
final classStart = _classStart(featureCount);
|
||||
for (var i = 0; i < boxCount; i++) {
|
||||
final cx = values[i];
|
||||
final cy = values[boxCount + i];
|
||||
final w = values[2 * boxCount + i];
|
||||
final h = values[3 * boxCount + i];
|
||||
final scored = _bestClassFeatureFirst(values, featureCount, boxCount,
|
||||
classStart: classStart, boxIndex: i);
|
||||
if (scored == null) continue;
|
||||
detections.add(_fromCenterBox(
|
||||
label: scored.label,
|
||||
confidence: scored.score,
|
||||
cx: cx,
|
||||
cy: cy,
|
||||
width: w,
|
||||
height: h,
|
||||
));
|
||||
}
|
||||
return detections;
|
||||
}
|
||||
|
||||
List<DetectionResult> _decodeRowFirst(
|
||||
List<double> values,
|
||||
int rowCount,
|
||||
int featureCount,
|
||||
) {
|
||||
final detections = <DetectionResult>[];
|
||||
if (featureCount == 6 || featureCount == 7) {
|
||||
for (var row = 0; row < rowCount; row++) {
|
||||
final offset = row * featureCount;
|
||||
final score = values[offset + 4].clamp(0.0, 1.0);
|
||||
final classIndex = values[offset + 5].round();
|
||||
detections.add(_fromCornerBox(
|
||||
label: _labelFor(classIndex),
|
||||
confidence: score,
|
||||
left: values[offset],
|
||||
top: values[offset + 1],
|
||||
right: values[offset + 2],
|
||||
bottom: values[offset + 3],
|
||||
));
|
||||
}
|
||||
return detections;
|
||||
}
|
||||
|
||||
final classStart = _classStart(featureCount);
|
||||
for (var row = 0; row < rowCount; row++) {
|
||||
final offset = row * featureCount;
|
||||
final scored =
|
||||
_bestClassRow(values, offset, featureCount, classStart: classStart);
|
||||
if (scored == null) continue;
|
||||
detections.add(_fromCenterBox(
|
||||
label: scored.label,
|
||||
confidence: scored.score,
|
||||
cx: values[offset],
|
||||
cy: values[offset + 1],
|
||||
width: values[offset + 2],
|
||||
height: values[offset + 3],
|
||||
));
|
||||
}
|
||||
return detections;
|
||||
}
|
||||
|
||||
int _classStart(int featureCount) {
|
||||
if (featureCount == 85) return 5; // YOLOv5 COCO: xywh + obj + 80 cls
|
||||
if (_labels.isNotEmpty && featureCount == _labels.length + 5) return 5;
|
||||
return 4;
|
||||
}
|
||||
|
||||
_ScoredLabel? _bestClassFeatureFirst(
|
||||
List<double> values,
|
||||
int featureCount,
|
||||
int boxCount, {
|
||||
required int classStart,
|
||||
required int boxIndex,
|
||||
}) {
|
||||
final objectness = classStart == 5 ? values[4 * boxCount + boxIndex] : 1.0;
|
||||
var bestScore = 0.0;
|
||||
var bestClass = 0;
|
||||
for (var feature = classStart; feature < featureCount; feature++) {
|
||||
final score = objectness * values[feature * boxCount + boxIndex];
|
||||
if (score > bestScore) {
|
||||
bestScore = score;
|
||||
bestClass = feature - classStart;
|
||||
}
|
||||
}
|
||||
if (bestScore <= 0) return null;
|
||||
return _ScoredLabel(
|
||||
_labelFor(bestClass, classCount: featureCount - classStart),
|
||||
bestScore.clamp(0.0, 1.0),
|
||||
);
|
||||
}
|
||||
|
||||
_ScoredLabel? _bestClassRow(
|
||||
List<double> values,
|
||||
int offset,
|
||||
int featureCount, {
|
||||
required int classStart,
|
||||
}) {
|
||||
final objectness = classStart == 5 ? values[offset + 4] : 1.0;
|
||||
var bestScore = 0.0;
|
||||
var bestClass = 0;
|
||||
for (var feature = classStart; feature < featureCount; feature++) {
|
||||
final score = objectness * values[offset + feature];
|
||||
if (score > bestScore) {
|
||||
bestScore = score;
|
||||
bestClass = feature - classStart;
|
||||
}
|
||||
}
|
||||
if (bestScore <= 0) return null;
|
||||
return _ScoredLabel(
|
||||
_labelFor(bestClass, classCount: featureCount - classStart),
|
||||
bestScore.clamp(0.0, 1.0),
|
||||
);
|
||||
}
|
||||
|
||||
String _labelFor(int classIndex, {int? classCount}) {
|
||||
if (classCount != null && classCount > _labels.length) {
|
||||
return _cocoObstacleLabels[classIndex] ?? 'object';
|
||||
}
|
||||
if (classIndex >= 0 && classIndex < _labels.length) {
|
||||
return _labels[classIndex];
|
||||
}
|
||||
return 'object';
|
||||
}
|
||||
|
||||
DetectionResult _fromCenterBox({
|
||||
required String label,
|
||||
required double confidence,
|
||||
required double cx,
|
||||
required double cy,
|
||||
required double width,
|
||||
required double height,
|
||||
}) {
|
||||
return _fromCornerBox(
|
||||
label: label,
|
||||
confidence: confidence,
|
||||
left: cx - width / 2,
|
||||
top: cy - height / 2,
|
||||
right: cx + width / 2,
|
||||
bottom: cy + height / 2,
|
||||
);
|
||||
}
|
||||
|
||||
DetectionResult _fromCornerBox({
|
||||
required String label,
|
||||
required double confidence,
|
||||
required double left,
|
||||
required double top,
|
||||
required double right,
|
||||
required double bottom,
|
||||
}) {
|
||||
final inputInfo = _inputInfo();
|
||||
final normalized =
|
||||
[left, top, right, bottom].every((value) => value >= 0 && value <= 1.5);
|
||||
|
||||
final scaledLeft = normalized
|
||||
? left * ObstacleAnalyzer.frameWidth
|
||||
: left / inputInfo.width * ObstacleAnalyzer.frameWidth;
|
||||
final scaledRight = normalized
|
||||
? right * ObstacleAnalyzer.frameWidth
|
||||
: right / inputInfo.width * ObstacleAnalyzer.frameWidth;
|
||||
final scaledTop = normalized
|
||||
? top * ObstacleAnalyzer.frameHeight
|
||||
: top / inputInfo.height * ObstacleAnalyzer.frameHeight;
|
||||
final scaledBottom = normalized
|
||||
? bottom * ObstacleAnalyzer.frameHeight
|
||||
: bottom / inputInfo.height * ObstacleAnalyzer.frameHeight;
|
||||
|
||||
final box = BoundingBox(
|
||||
left: scaledLeft.clamp(0.0, ObstacleAnalyzer.frameWidth),
|
||||
top: scaledTop.clamp(0.0, ObstacleAnalyzer.frameHeight),
|
||||
right: scaledRight.clamp(0.0, ObstacleAnalyzer.frameWidth),
|
||||
bottom: scaledBottom.clamp(0.0, ObstacleAnalyzer.frameHeight),
|
||||
);
|
||||
|
||||
return DetectionResult(
|
||||
label: label,
|
||||
confidence: confidence,
|
||||
direction: _analyzer.analyzeDirection(box),
|
||||
estimatedDistance: _analyzer.estimateDistance(box),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const Map<int, String> _cocoObstacleLabels = {
|
||||
0: 'person',
|
||||
1: 'bicycle',
|
||||
2: 'car',
|
||||
3: 'motorcycle',
|
||||
5: 'bus',
|
||||
7: 'truck',
|
||||
13: 'bench',
|
||||
56: 'chair',
|
||||
};
|
||||
|
||||
class _InputInfo {
|
||||
final int width;
|
||||
final int height;
|
||||
final bool channelsLast;
|
||||
final YoloTensorType type;
|
||||
final int bytesPerValue;
|
||||
|
||||
const _InputInfo({
|
||||
required this.width,
|
||||
required this.height,
|
||||
required this.channelsLast,
|
||||
required this.type,
|
||||
required this.bytesPerValue,
|
||||
});
|
||||
|
||||
int get byteLength => width * height * 3 * bytesPerValue;
|
||||
}
|
||||
|
||||
class _ScoredLabel {
|
||||
final String label;
|
||||
final double score;
|
||||
|
||||
const _ScoredLabel(this.label, this.score);
|
||||
}
|
||||
|
||||
@ -0,0 +1 @@
|
||||
export 'yolo_runtime_stub.dart' if (dart.library.io) 'yolo_runtime_native.dart';
|
||||
@ -0,0 +1,57 @@
|
||||
import 'dart:typed_data';
|
||||
|
||||
import 'package:tflite_flutter/tflite_flutter.dart';
|
||||
|
||||
enum YoloTensorType { float32, uint8, int8, float16, unsupported }
|
||||
|
||||
class YoloRuntime {
|
||||
final Interpreter _interpreter;
|
||||
final List<int> inputShape;
|
||||
final List<int> outputShape;
|
||||
final YoloTensorType inputType;
|
||||
final YoloTensorType outputType;
|
||||
final int outputByteLength;
|
||||
|
||||
YoloRuntime._({
|
||||
required Interpreter interpreter,
|
||||
required this.inputShape,
|
||||
required this.outputShape,
|
||||
required this.inputType,
|
||||
required this.outputType,
|
||||
required this.outputByteLength,
|
||||
}) : _interpreter = interpreter;
|
||||
|
||||
static Future<YoloRuntime> load(String assetPath) async {
|
||||
final options = InterpreterOptions()..threads = 2;
|
||||
final interpreter =
|
||||
await Interpreter.fromAsset(assetPath, options: options);
|
||||
final input = interpreter.getInputTensor(0);
|
||||
final output = interpreter.getOutputTensor(0);
|
||||
return YoloRuntime._(
|
||||
interpreter: interpreter,
|
||||
inputShape: input.shape,
|
||||
outputShape: output.shape,
|
||||
inputType: _mapTensorType(input.type),
|
||||
outputType: _mapTensorType(output.type),
|
||||
outputByteLength: output.numBytes(),
|
||||
);
|
||||
}
|
||||
|
||||
void run(Uint8List input, Uint8List output) {
|
||||
_interpreter.run(input, output);
|
||||
}
|
||||
|
||||
void close() {
|
||||
_interpreter.close();
|
||||
}
|
||||
}
|
||||
|
||||
YoloTensorType _mapTensorType(TensorType type) {
|
||||
return switch (type) {
|
||||
TensorType.float32 => YoloTensorType.float32,
|
||||
TensorType.uint8 => YoloTensorType.uint8,
|
||||
TensorType.int8 => YoloTensorType.int8,
|
||||
TensorType.float16 => YoloTensorType.float16,
|
||||
_ => YoloTensorType.unsupported,
|
||||
};
|
||||
}
|
||||
@ -0,0 +1,23 @@
|
||||
import 'dart:typed_data';
|
||||
|
||||
enum YoloTensorType { float32, uint8, int8, float16, unsupported }
|
||||
|
||||
class YoloRuntime {
|
||||
const YoloRuntime();
|
||||
|
||||
static Future<YoloRuntime> load(String assetPath) async {
|
||||
throw UnsupportedError('YOLO TFLite runtime is only available on mobile.');
|
||||
}
|
||||
|
||||
List<int> get inputShape => const [];
|
||||
List<int> get outputShape => const [];
|
||||
YoloTensorType get inputType => YoloTensorType.unsupported;
|
||||
YoloTensorType get outputType => YoloTensorType.unsupported;
|
||||
int get outputByteLength => 0;
|
||||
|
||||
void run(Uint8List input, Uint8List output) {
|
||||
throw UnsupportedError('YOLO TFLite runtime is only available on mobile.');
|
||||
}
|
||||
|
||||
void close() {}
|
||||
}
|
||||
@ -48,8 +48,8 @@ class _ActivityLogScreenState extends State<ActivityLogScreen> {
|
||||
final res = await _api
|
||||
.get('/user/activity-logs')
|
||||
.timeout(const Duration(seconds: 10));
|
||||
final list = (res.data['data'] as List?) ?? [];
|
||||
final items = list.map((e) => _LogItem.fromJson(e)).toList();
|
||||
final list = _extractList(res.data);
|
||||
final items = list.map(_LogItem.fromJson).toList();
|
||||
setState(() {
|
||||
_items = items;
|
||||
_applyFilter(_selectedFilter);
|
||||
@ -66,6 +66,16 @@ class _ActivityLogScreenState extends State<ActivityLogScreen> {
|
||||
}
|
||||
}
|
||||
|
||||
List<Map<String, dynamic>> _extractList(dynamic responseBody) {
|
||||
final data = responseBody is Map ? responseBody['data'] : null;
|
||||
final rawList = data is Map ? data['content'] : data;
|
||||
if (rawList is! List) return const [];
|
||||
return rawList
|
||||
.whereType<Map>()
|
||||
.map((item) => Map<String, dynamic>.from(item))
|
||||
.toList();
|
||||
}
|
||||
|
||||
void _applyFilter(String filter) {
|
||||
_selectedFilter = filter;
|
||||
if (filter == 'ALL') {
|
||||
|
||||
@ -40,7 +40,7 @@ class _NotificationScreenState extends State<NotificationScreen> {
|
||||
final res = await _api
|
||||
.get('/user/notifications')
|
||||
.timeout(const Duration(seconds: 10));
|
||||
final list = (res.data['data'] as List?) ?? [];
|
||||
final list = _extractList(res.data);
|
||||
setState(() {
|
||||
_items = list.map((e) => _NotifItem.fromJson(e)).toList();
|
||||
});
|
||||
@ -56,10 +56,20 @@ class _NotificationScreenState extends State<NotificationScreen> {
|
||||
}
|
||||
}
|
||||
|
||||
List<Map<String, dynamic>> _extractList(dynamic responseBody) {
|
||||
final data = responseBody is Map ? responseBody['data'] : null;
|
||||
final rawList = data is Map ? data['content'] : data;
|
||||
if (rawList is! List) return const [];
|
||||
return rawList
|
||||
.whereType<Map>()
|
||||
.map((item) => Map<String, dynamic>.from(item))
|
||||
.toList();
|
||||
}
|
||||
|
||||
Future<void> _markRead(int id) async {
|
||||
try {
|
||||
await _api
|
||||
.patch('/user/notifications/$id/read')
|
||||
.put('/user/notifications/$id/read')
|
||||
.timeout(const Duration(seconds: 6));
|
||||
setState(() {
|
||||
final idx = _items.indexWhere((n) => n.id == id);
|
||||
@ -72,7 +82,7 @@ class _NotificationScreenState extends State<NotificationScreen> {
|
||||
setState(() => _markingAll = true);
|
||||
try {
|
||||
await _api
|
||||
.patch('/user/notifications/read-all')
|
||||
.put('/user/notifications/mark-all-read')
|
||||
.timeout(const Duration(seconds: 8));
|
||||
setState(() {
|
||||
_items = _items.map((n) => n.copyWith(isRead: true)).toList();
|
||||
|
||||
@ -356,9 +356,16 @@ class _WalkGuideScreenState extends State<WalkGuideScreen> {
|
||||
String _status = 'Ready';
|
||||
CameraController? _camera;
|
||||
DetectionResult? _lastDetection;
|
||||
bool _processingFrame = false;
|
||||
DateTime _lastInferenceAt = DateTime.fromMillisecondsSinceEpoch(0);
|
||||
DateTime _lastAlertAt = DateTime.fromMillisecondsSinceEpoch(0);
|
||||
|
||||
@override
|
||||
void dispose() {
|
||||
final camera = _camera;
|
||||
if (camera != null && camera.value.isStreamingImages) {
|
||||
unawaited(camera.stopImageStream().catchError((Object _) {}));
|
||||
}
|
||||
_camera?.dispose();
|
||||
sl<LocationReporterService>().stop();
|
||||
super.dispose();
|
||||
@ -370,8 +377,7 @@ class _WalkGuideScreenState extends State<WalkGuideScreen> {
|
||||
await _startCamera();
|
||||
await sl<LocationReporterService>().start(walkGuideActive: true);
|
||||
} else {
|
||||
await _camera?.dispose();
|
||||
_camera = null;
|
||||
await _stopCamera();
|
||||
await sl<LocationReporterService>().stop();
|
||||
}
|
||||
setState(() {
|
||||
@ -388,35 +394,89 @@ class _WalkGuideScreenState extends State<WalkGuideScreen> {
|
||||
final cameras = await availableCameras();
|
||||
if (cameras.isEmpty) return;
|
||||
final controller = CameraController(
|
||||
cameras.first, ResolutionPreset.medium,
|
||||
enableAudio: false);
|
||||
cameras.first,
|
||||
ResolutionPreset.medium,
|
||||
enableAudio: false,
|
||||
imageFormatGroup: ImageFormatGroup.yuv420,
|
||||
);
|
||||
await controller.initialize();
|
||||
if (!mounted) {
|
||||
await controller.dispose();
|
||||
return;
|
||||
}
|
||||
try {
|
||||
await controller.startImageStream(_onCameraImage);
|
||||
} catch (_) {
|
||||
// Preview still works; manual Demo Detect remains available.
|
||||
}
|
||||
setState(() => _camera = controller);
|
||||
} catch (_) {
|
||||
setState(() => _status = 'Camera unavailable. Demo mode active.');
|
||||
}
|
||||
}
|
||||
|
||||
Future<void> _stopCamera() async {
|
||||
final camera = _camera;
|
||||
_camera = null;
|
||||
if (camera == null) return;
|
||||
try {
|
||||
if (camera.value.isStreamingImages) {
|
||||
await camera.stopImageStream();
|
||||
}
|
||||
} catch (_) {}
|
||||
await camera.dispose();
|
||||
}
|
||||
|
||||
void _onCameraImage(CameraImage image) {
|
||||
if (!_active || _processingFrame) return;
|
||||
final now = DateTime.now();
|
||||
if (now.difference(_lastInferenceAt) < const Duration(milliseconds: 900)) {
|
||||
return;
|
||||
}
|
||||
_lastInferenceAt = now;
|
||||
_processingFrame = true;
|
||||
unawaited(_runYolo(image).whenComplete(() => _processingFrame = false));
|
||||
}
|
||||
|
||||
Future<void> _runYolo(CameraImage image) async {
|
||||
final detection = await sl<YoloDetector>().detect(image);
|
||||
if (detection == null || !mounted) return;
|
||||
await _handleDetection(detection);
|
||||
}
|
||||
|
||||
Future<void> _simulateObstacle() async {
|
||||
final detection = await sl<YoloDetector>().detectFallback();
|
||||
if (detection == null) return;
|
||||
await _handleDetection(detection, forceAlert: true);
|
||||
}
|
||||
|
||||
Future<void> _handleDetection(
|
||||
DetectionResult detection, {
|
||||
bool forceAlert = false,
|
||||
}) async {
|
||||
_lastDetection = detection;
|
||||
await _api.post('/user/obstacle', data: {
|
||||
'label': detection.label,
|
||||
'confidence': detection.confidence,
|
||||
'direction': detection.directionName,
|
||||
'estimatedDist': detection.estimatedDistance,
|
||||
'lat': null,
|
||||
'lng': null,
|
||||
});
|
||||
await sl<HapticService>().obstacleClose();
|
||||
await sl<TtsService>().speakImmediate(detection.spokenId);
|
||||
setState(() => _status =
|
||||
'Obstacle: ${detection.label} ${detection.directionName} ${detection.estimatedDistance}');
|
||||
|
||||
final now = DateTime.now();
|
||||
if (!forceAlert &&
|
||||
now.difference(_lastAlertAt) < const Duration(seconds: 4)) {
|
||||
return;
|
||||
}
|
||||
_lastAlertAt = now;
|
||||
|
||||
try {
|
||||
await _api.post('/user/obstacle', data: {
|
||||
'label': detection.label,
|
||||
'confidence': detection.confidence,
|
||||
'direction': detection.directionName,
|
||||
'estimatedDist': detection.estimatedDistance,
|
||||
'lat': null,
|
||||
'lng': null,
|
||||
});
|
||||
} catch (_) {}
|
||||
await sl<HapticService>().obstacleClose();
|
||||
await sl<TtsService>().speakImmediate(detection.spokenId);
|
||||
}
|
||||
|
||||
@override
|
||||
@ -1148,12 +1208,8 @@ class _AiBenchmarkScreenState extends State<AiBenchmarkScreen> {
|
||||
String label = 'person';
|
||||
String direction = 'CENTER';
|
||||
String distance = 'Demo';
|
||||
var modelLoaded = false;
|
||||
try {
|
||||
await rootBundle.load(_selectedModel).timeout(const Duration(seconds: 3));
|
||||
modelLoaded = true;
|
||||
} catch (_) {}
|
||||
final detection = await sl<YoloDetector>().detectFallback();
|
||||
final modelLoaded = sl<YoloDetector>().isReady;
|
||||
final detection = await sl<YoloDetector>().detectSynthetic();
|
||||
if (detection != null) {
|
||||
label = detection.label;
|
||||
direction = detection.directionName;
|
||||
|
||||
@ -33,9 +33,16 @@ class _WalkGuideScreenState extends State<WalkGuideScreen> {
|
||||
String _status = 'Ready';
|
||||
CameraController? _camera;
|
||||
DetectionResult? _lastDetection;
|
||||
bool _processingFrame = false;
|
||||
DateTime _lastInferenceAt = DateTime.fromMillisecondsSinceEpoch(0);
|
||||
DateTime _lastAlertAt = DateTime.fromMillisecondsSinceEpoch(0);
|
||||
|
||||
@override
|
||||
void dispose() {
|
||||
final camera = _camera;
|
||||
if (camera != null && camera.value.isStreamingImages) {
|
||||
unawaited(camera.stopImageStream().catchError((Object _) {}));
|
||||
}
|
||||
_camera?.dispose();
|
||||
sl<LocationReporterService>().stop();
|
||||
super.dispose();
|
||||
@ -47,8 +54,7 @@ class _WalkGuideScreenState extends State<WalkGuideScreen> {
|
||||
await _startCamera();
|
||||
await sl<LocationReporterService>().start(walkGuideActive: true);
|
||||
} else {
|
||||
await _camera?.dispose();
|
||||
_camera = null;
|
||||
await _stopCamera();
|
||||
await sl<LocationReporterService>().stop();
|
||||
}
|
||||
setState(() {
|
||||
@ -67,35 +73,89 @@ class _WalkGuideScreenState extends State<WalkGuideScreen> {
|
||||
final cameras = await availableCameras();
|
||||
if (cameras.isEmpty) return;
|
||||
final controller = CameraController(
|
||||
cameras.first, ResolutionPreset.medium,
|
||||
enableAudio: false);
|
||||
cameras.first,
|
||||
ResolutionPreset.medium,
|
||||
enableAudio: false,
|
||||
imageFormatGroup: ImageFormatGroup.yuv420,
|
||||
);
|
||||
await controller.initialize();
|
||||
if (!mounted) {
|
||||
await controller.dispose();
|
||||
return;
|
||||
}
|
||||
try {
|
||||
await controller.startImageStream(_onCameraImage);
|
||||
} catch (_) {
|
||||
// Preview still works; manual Demo Detect remains available.
|
||||
}
|
||||
setState(() => _camera = controller);
|
||||
} catch (_) {
|
||||
setState(() => _status = 'Camera unavailable. Demo mode active.');
|
||||
}
|
||||
}
|
||||
|
||||
Future<void> _stopCamera() async {
|
||||
final camera = _camera;
|
||||
_camera = null;
|
||||
if (camera == null) return;
|
||||
try {
|
||||
if (camera.value.isStreamingImages) {
|
||||
await camera.stopImageStream();
|
||||
}
|
||||
} catch (_) {}
|
||||
await camera.dispose();
|
||||
}
|
||||
|
||||
void _onCameraImage(CameraImage image) {
|
||||
if (!_active || _processingFrame) return;
|
||||
final now = DateTime.now();
|
||||
if (now.difference(_lastInferenceAt) < const Duration(milliseconds: 900)) {
|
||||
return;
|
||||
}
|
||||
_lastInferenceAt = now;
|
||||
_processingFrame = true;
|
||||
unawaited(_runYolo(image).whenComplete(() => _processingFrame = false));
|
||||
}
|
||||
|
||||
Future<void> _runYolo(CameraImage image) async {
|
||||
final detection = await sl<YoloDetector>().detect(image);
|
||||
if (detection == null || !mounted) return;
|
||||
await _handleDetection(detection);
|
||||
}
|
||||
|
||||
Future<void> _simulateObstacle() async {
|
||||
final detection = await sl<YoloDetector>().detectFallback();
|
||||
if (detection == null) return;
|
||||
await _handleDetection(detection, forceAlert: true);
|
||||
}
|
||||
|
||||
Future<void> _handleDetection(
|
||||
DetectionResult detection, {
|
||||
bool forceAlert = false,
|
||||
}) async {
|
||||
_lastDetection = detection;
|
||||
await sl<ApiClient>().dio.post('/user/obstacle', data: {
|
||||
'label': detection.label,
|
||||
'confidence': detection.confidence,
|
||||
'direction': detection.directionName,
|
||||
'estimatedDist': detection.estimatedDistance,
|
||||
'lat': null,
|
||||
'lng': null,
|
||||
});
|
||||
await sl<HapticService>().obstacleClose();
|
||||
await sl<TtsService>().speakImmediate(detection.spokenId);
|
||||
setState(() => _status =
|
||||
'Obstacle: ${detection.label} ${detection.directionName} ${detection.estimatedDistance}');
|
||||
|
||||
final now = DateTime.now();
|
||||
if (!forceAlert &&
|
||||
now.difference(_lastAlertAt) < const Duration(seconds: 4)) {
|
||||
return;
|
||||
}
|
||||
_lastAlertAt = now;
|
||||
|
||||
try {
|
||||
await sl<ApiClient>().dio.post('/user/obstacle', data: {
|
||||
'label': detection.label,
|
||||
'confidence': detection.confidence,
|
||||
'direction': detection.directionName,
|
||||
'estimatedDist': detection.estimatedDistance,
|
||||
'lat': null,
|
||||
'lng': null,
|
||||
});
|
||||
} catch (_) {}
|
||||
await sl<HapticService>().obstacleClose();
|
||||
await sl<TtsService>().speakImmediate(detection.spokenId);
|
||||
}
|
||||
|
||||
@override
|
||||
@ -235,12 +295,8 @@ class _AiBenchmarkScreenState extends State<AiBenchmarkScreen> {
|
||||
String label = 'person';
|
||||
String direction = 'CENTER';
|
||||
String distance = 'Demo';
|
||||
var modelLoaded = false;
|
||||
try {
|
||||
await rootBundle.load(_selectedModel).timeout(const Duration(seconds: 3));
|
||||
modelLoaded = true;
|
||||
} catch (_) {}
|
||||
final detection = await sl<YoloDetector>().detectFallback();
|
||||
final modelLoaded = sl<YoloDetector>().isReady;
|
||||
final detection = await sl<YoloDetector>().detectSynthetic();
|
||||
if (detection != null) {
|
||||
label = detection.label;
|
||||
direction = detection.directionName;
|
||||
@ -507,9 +563,7 @@ class _EmptyPanel extends StatelessWidget {
|
||||
final String title;
|
||||
final String message;
|
||||
const _EmptyPanel(
|
||||
{required this.icon,
|
||||
required this.title,
|
||||
required this.message});
|
||||
{required this.icon, required this.title, required this.message});
|
||||
|
||||
@override
|
||||
Widget build(BuildContext context) {
|
||||
|
||||
@ -1616,10 +1616,10 @@ packages:
|
||||
dependency: "direct main"
|
||||
description:
|
||||
name: tflite_flutter
|
||||
sha256: ffb8651fdb116ab0131d6dc47ff73883e0f634ad1ab12bb2852eef1bbeab4a6a
|
||||
sha256: "0bba9040d8decda0960d7abf8eabf32243bf092bc7d0084e8e19681866b0bdbe"
|
||||
url: "https://pub.dev"
|
||||
source: hosted
|
||||
version: "0.10.4"
|
||||
version: "0.12.1"
|
||||
timezone:
|
||||
dependency: transitive
|
||||
description:
|
||||
|
||||
@ -34,7 +34,7 @@ dependencies:
|
||||
|
||||
# Camera & AI
|
||||
camera: ^0.11.0+2
|
||||
tflite_flutter: ^0.10.4
|
||||
tflite_flutter: 0.12.1
|
||||
image: ^4.2.0
|
||||
|
||||
# Audio & TTS
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user