Google Sign-In Required

Use your company Google account to access the BetterFleet private content.

Back to private home

BetterFleet Support Private
Skip to content
BetterFleet Dev Wiki
Depot Sim Testing Requirements
Initializing search
    bf-dev
    • Home
    • Process
    • Products
    • Reference
    • Decisions
    • Work
    • Operations
    bf-dev
    • Home
      • Process Handbook
      • BetterFleet Workflow Map
      • Product Development System
      • Product Engineering Workflow
        • Process Workflows
        • Work Intake and Weekly Planning
        • Product Engineering Workflow in Linear
        • Product Engineering Delivery
        • Agent Guidance
        • Workflow
        • Skills
        • Skill Sources
        • Process Guides
        • GitLab Feature Flags
        • In-App Docs Authoring
        • Release Notes
        • Process Templates
        • Release Plan: <title>
      • Process Publishing
      • Product overview
        • General Reference
          • Core Domain Training
          • System Topology
          • Two-Axis Ontology Model
          • Ontology Primer
          • Worked Example
          • Evidence, Ownership, and Lineage
          • Energy Management
          • Standards and Protocol Map
          • Charging, Roaming, and Commercial Model
          • Charge Planning and Operations
          • Cross-Cutting Domains
          • Domain Coverage Matrix
        • BetterFleet Product Ontology
        • Core Operations Data Ontology
        • BetterFleet R&D Plan
        • Index
        • Architecture
        • Manage Product Capabilities
        • Manage Data and State
        • Manage Service Interaction Flows
        • Manage Reference
        • Manage Internal Application Diagrams
          • Manage Authorization And Permissions
          • bf-manage-core Auth and Authorization Model
          • Manage Authorization and Permissions
          • bf-manage-web Auth and Permission Model
          • Manage Service Catalog
          • bf-depot-sim
          • bf-digital-twin (Manage Role)
          • bf-fleet-health
          • bf-manage-connect
          • bf-manage-core
          • bf-manage-incidents
          • bf-manage-roaming
          • bf-manage-sitepwrmon
          • bf-manage-web
          • bf-schedule-creator (Manage Role)
          • bf-support-microsite
          • bf-telematics
        • Index
        • Architecture
        • Plan Reference
        • Plan Internal Application Diagrams
        • Plan Migration and Flags
        • Plan Simulation Request Lifecycle
          • Plan Service Catalog
          • bf-bnl-schedule-analysis-compute
          • bf-bnl-settings
          • bf-bnl-ui
          • bf-digital-twin (Plan Role)
          • bf-route-modelling
          • bf-schedule-creator (Plan Role)
      • Where to Ask Product Questions
      • Reference
        • Platform Reference
        • Platform Architecture
        • Script Runtime Model
        • Compose Profiles and Modes
        • Repository Map
        • Monolithic Git Transition FAQ
        • Monolithic Git Sizing
        • CI and Release Integration
        • Shared Reference
        • Shared Infrastructure Architecture
        • Secrets and Env Strategy
        • Vendors and Local Dependencies
        • System Reference
        • Cloud Data Dependencies
        • Ports and URLs
        • Service Matrix
          • API Docs
          • OCPI API Docs
          • OCPP API Docs
          • OSCP API Docs
          • VDV API Docs
          • Yard State API Docs
        • System Design
        • System Design: BBA Microgrid Controller Generic Packet Translation
        • System Design: Depot Simulation
        • System Design: IoT Sensor Packet
        • System Design: Microgrid Energy Orchestration
          • System Design: OCPP Profile 3 And ISO 15118 PKI
          • Architecture: BetterFleet OCPP Profile 3 and ISO 15118 PKI
          • Specification: BetterFleet OCPP Profile 3 and ISO 15118 Certificate Lifecycle Management
          • System Design: On-Prem Control
          • Challenge
          • Specification: BetterFleet On-Prem Continuity Control
          • System Design: OSCP
          • OSCP Protocol Documentation
          • Depot Sim Testing Requirements
            • Purpose
            • Scope
            • Key Outcomes
            • Reference for Manual Testing Approach
              • BetterFleet Initiated Handshake Timeout
              • Capacity Provider Initiated Handshake Timeout
              • BetterFleet Stops Sending Heartbeats
              • Capacity Provider Stops Sending Heartbeats
            • Future Goals
            • Current Testing Expectations
              • Timeouts
                • BetterFleet Initiated Handshake
                • Capacity Provider Initiated Handshake
                • BetterFleet Sending Heartbeats
              • Offline Handling
                • Capacity Provider Stops Sending Heartbeats
                • Capacity Provider Stops Accepting Heartbeats
              • Fallback and Gap Handling
                • Offline Detection With Matching Fallback Forecast
                • Forecast Gap With Matching Fallback Block
                • Forecast Gap Without Matching Fallback Block
                • Recovery From Fallback Mode
              • Intentional Disconnect Handling
                • Disconnect While OSCP Forecast Is Active
                • Disconnect While Fallback Mode Is Active
                • Network Outage is Resolved While Still Connected
                • Network Outage is Resolved After Offline Detection
              • Error Cases
                • BetterFleet Sends Heartbeat While Capacity Provider is NOT in Online State
                • Capacity Provider Sends Heartbeat While BetterFleet is NOT in Online State
                • BetterFleet Sends Handshake/Acknowledge While Capacity Provider is NOT in Offline State
                • Capacity Provider Sends Handshake/Acknowledge While BetterFleet is NOT in Offline State
          • System Design: OSCP Flexibility Provider Domain
      • Decisions
        • Architecture Decision Records
        • 0001 - Record architecture decisions
        • 0002 - Cognito for Authentication and Authorisation
        • 0003 - AWS Amplify for Authentication
        • 0004 - DynamoDB for default database
        • 0005 - Data Persistence
        • 0006 - Trunk-Based Development
        • 0007 - Generalised principle for automation
        • 0008 - Naming Repositories, Services, and URLs
        • 0009 - Use Timezone Aware DateTimes and UTC
        • 0010 - Use semantic release
        • 0011 - Centralized feature flag repository
        • 0012 - Use Named Exports in Storybook
        • 0013 - RESTful TITLE GraphQL
        • 0014 - Service Granularity
        • 0015 - Async/co-routine exception handling pattern
        • 0016 - Logging & log levels
        • 0017 - Instantiated Models
        • 0018 - Repository Pattern for Database Access
        • 0019 - Use of Design Tokens in TypeScript React Application
        • 0020 - API backwards compatibility and versioning
        • 0021 - Alembic Migration strategy
        • 0022 - Consistent react-hook-form usage
        • 0023 - Domain Event-Driven Architecture
        • 0024 - Domain Event Bus Tech Stack
        • 0025 - No enum types in DB table columns
        • 0026 - In-Memory Ormar Stores for Repository testing
        • 0027 - Storing Tab State in Query and Local Storage
        • 0028 - Adopt OpenTelemetry Semantic Conventions for Structured Logging
        • 0029 - Adopt RFC 9457 for HTTP Error Responses
        • 0030 - Use GitLab registry and Terraform state for ECS services
        • 0031 - Adopt DDD, Hexagonal Architecture, and CQRS for Python Domain Services
      • Work
        • Active Work
          • Work: Bba Microgrid Controller
          • Implementation Specification: BBA Microgrid Controller
          • BBA Microgrid Controller Deliverables (Stories)
          • Work: BFDev Monolithic Git
          • Challenge
          • Specification: BFDev Monolithic Git v2
          • BFDev Monolithic Git v2 Stories
          • Work: Complex Circuit Load Balancing
          • Implementation Specification: Complex Circuit Load Balancing
          • Complex Circuit Load Balancing Deliverables (Stories)
            • COR-10 and COR-11 Consolidation Review
          • Work: Dispatch Reliability and Reconciliation
          • Challenge
          • Specification: Dispatch Reliability and Reconciliation
          • Dispatch Reliability and Reconciliation (Unit User Stories)
            • Dispatch populated vehicle cards grey surface snapshot
            • Dispatch Visual Review
          • Work: Enable Scheduled Managed Charger Access
          • Challenge: Enable Scheduled Managed Charger Access
          • Specification Exploration Dossier: Enable Scheduled Managed Charger Access
          • Specification Review: Enable Scheduled Managed Charger Access
          • Specification: Enable Scheduled Managed Charger Access
          • Work: Guided Cut-Off and Release Orchestration
          • Specification: Guided Cut-Off and Release Orchestration
          • Guided Cut-Off and Release Orchestration (Unit User Stories)
          • Work: Production Deployment Validation
          • Challenge
          • Work: Scheduled Report Parity
          • Specification: Scheduled Report Parity
          • Work: Telematics
          • Telematics EventBridge Path
          • Telematics Ingress Architecture
          • Specification: Telematics Migration into bf-manage-core with 5-Minute Freshness and Health Visibility
          • Telematics Core Migration MVP (Implementation-Time BDD)
          • Work: Vector Derms
          • Implementation Specification: Vector DERMS
          • Vector DERMS Deliverables (Stories)
          • Work: Visiting Vehicle Charging Visibility
          • Specification: Visiting Vehicle Charging Visibility
          • Visiting Vehicle Charging Visibility (Unit User Stories)
          • Work: Workspace Owned Stripe Roaming
          • Specification: Workspace-Owned Stripe Credentials for Roaming Payments
        • Backlog Work
          • Work: Microgrid
          • Microgrid Backlog Stories
          • Work: Mobile Ops Companion
          • Challenge
          • Specification: Mobile Operations Companion v1
          • Mobile Operations Companion Deliverables (Stories)
          • Work: Oscp
          • OSCP Backlog Stories
        • Archived Work
          • Work: Code Canonical Orchestration
          • Challenge
          • Specification: Product Engineering Workflow
          • Product Engineering Workflow Deliverables (Unit User Stories)
          • Work: Release Notes Automation
          • Release Plan: Release Notes Automation
          • Release Notes Automation Backlog Stories
      • Operations
      • Onboarding Runbook
        • Operations Runbooks
        • Production Hotfix Release
        • Staging Hotfix Release
        • Manage Staging Release Validation
        • Terraform Plan Dry Runs
        • Operations Tooling
        • Code Indexing
        • Operations Evidence
        • Database Restoration Test Report
      • Daily Operations Runbook
      • Testing Guide
      • Troubleshooting
    • Purpose
    • Scope
    • Key Outcomes
    • Reference for Manual Testing Approach
      • BetterFleet Initiated Handshake Timeout
      • Capacity Provider Initiated Handshake Timeout
      • BetterFleet Stops Sending Heartbeats
      • Capacity Provider Stops Sending Heartbeats
    • Future Goals
    • Current Testing Expectations
      • Timeouts
        • BetterFleet Initiated Handshake
        • Capacity Provider Initiated Handshake
        • BetterFleet Sending Heartbeats
      • Offline Handling
        • Capacity Provider Stops Sending Heartbeats
        • Capacity Provider Stops Accepting Heartbeats
      • Fallback and Gap Handling
        • Offline Detection With Matching Fallback Forecast
        • Forecast Gap With Matching Fallback Block
        • Forecast Gap Without Matching Fallback Block
        • Recovery From Fallback Mode
      • Intentional Disconnect Handling
        • Disconnect While OSCP Forecast Is Active
        • Disconnect While Fallback Mode Is Active
        • Network Outage is Resolved While Still Connected
        • Network Outage is Resolved After Offline Detection
      • Error Cases
        • BetterFleet Sends Heartbeat While Capacity Provider is NOT in Online State
        • Capacity Provider Sends Heartbeat While BetterFleet is NOT in Online State
        • BetterFleet Sends Handshake/Acknowledge While Capacity Provider is NOT in Offline State
        • Capacity Provider Sends Handshake/Acknowledge While BetterFleet is NOT in Offline State
    1. Home
    2. Reference
    3. System design
    4. Oscp
    Shared Technical

    Depot Sim Testing Requirements¶

    Purpose¶

    This document is essentially a context dump for an AI to understand the current testing methodology, and expected testing surfaces for a Capacity Provider simulator to be natively provided by Depot Sim. A simple simulated capacity provider server exists currently, however it is rudimentary and must be replaced with an appropriate, feature-full alternative in Depot Sim.

    The document outlines all of the potential failures that need to be easily testable in an end-to-end environment. For example, in order to validate that the OSCP connection is appropriately culled when an offline condition is detected, the prospective simulator must be able to replicate that situation (manual process just kills the simple sim server which results in heartbeats no longer being sent).

    Scope¶

    Focused on the current and next confirmed Vector OSCP testing surfaces:

    • handshakes and acknowledgement
    • heartbeats and offline detection
    • fallback-mode activation and recovery
    • forecast gaps with and without matching fallback coverage
    • intentional disconnect handover back to the non-OSCP path

    Key Outcomes¶

    • All Behavioural outcome: * should be actionable.
    • All Failure mode: * can be triggered through the sim.
    • All Failure risk: * can be tested for and expected behaviours validated.

    Reference for Manual Testing Approach¶

    This is a reference to how manual testing was conducted prior to the creation of the proper Depot Sim Capacity Provider.

    This is while using the single-file simulator.

    Due to Event Bridge not being able to communicate directly with the dev server, those events were triggered using Postman and the appropriate event_id.

    BetterFleet Initiated Handshake Timeout¶

    • Use the UI to connect FP to CP
    • Option 1
      • Kill CP process
      • Wait for timeout handling to reset the connection
    • Option 2
      • Let CP return 204
      • Kill CP process
      • Wait for timeout handling to reset the connection
    • Verify that connection state is reset to offline

    Capacity Provider Initiated Handshake Timeout¶

    • Use Postman to send Handshake to BetterFleet
    • BetterFleet returns 204
    • Kill CP process
    • Wait for BetterFleet's outbound HandshakeAcknowledge request to fail or time out
    • Verify that connection state is reset to offline

    BetterFleet Stops Sending Heartbeats¶

    • Establish the connection first
    • Stop BetterFleet from sending heartbeats (cease using Postman)
    • Wait for CP offline detection to occur
    • Verify that connection state is reset to offline

    Capacity Provider Stops Sending Heartbeats¶

    • Establish the connection first
    • Stop CP from sending heartbeats to BetterFleet (kill CP process)
    • Wait for BetterFleet offline detection to occur
    • Verify that connection state is reset to offline

    Future Goals¶

    • Trigger specific OSCP messages such as UpdateGroupCapacityForecast
    • Send malformed payloads for those messages
      • Could be similar to the interoperability testing sys admin page.
    • Exercise fallback and gap-policy scenarios from the Depot Sim UI without external scripts or Postman-only flows.

    Current Testing Expectations¶

    Timeouts¶

    Behavioural outcome: sim needs to be able to imitate timeouts.

    BetterFleet Initiated Handshake¶

    sequenceDiagram
    participant CP as Capacity Provider
    participant BF as BetterFleet

    BF->>CP: Send Handshake
    CP-->>BF: Respond 204
    Note over CP,BF: Failure mode: we may never receive this

    CP-->>BF: Send HandshakeAcknowledge
    Note over CP,BF: Failure mode: we may never receive this

    Capacity Provider Initiated Handshake¶

    sequenceDiagram
    participant CP as Capacity Provider
    participant BF as BetterFleet

    CP->>BF: Send Handshake
    BF->>CP: Respond 204
    BF->>CP: Send HandshakeAcknowledge
    CP-->>BF: Respond 204
    Note over CP,BF: Failure mode: we may never receive this

    BetterFleet Sending Heartbeats¶

    sequenceDiagram
    participant CP as Capacity Provider
    participant BF as BetterFleet

    loop
        BF->>CP: Send Heartbeat
        CP-->>BF: Respond 204
        Note over CP,BF: Failure mode: we may never receive this
    end

    Offline Handling¶

    Behavioural outcome: sim needs to be able to be turned offline (i.e., stop sending heartbeats).

    Note: when determining offline detection, it is strictly based on the expiry time in the heartbeats, but for simplicity, these charts assume that offline detection occurs immediately.

    Capacity Provider Stops Sending Heartbeats¶

    sequenceDiagram
    participant CP as Capacity Provider
    participant BF as BetterFleet

    CP->>BF: Send Heartbeat
    BF-->>CP: Respond 204
    Note over CP: Server dies

    Note over CP,BF: Failure risk: connection should now be offline

    Capacity Provider Stops Accepting Heartbeats¶

    sequenceDiagram
    participant CP as Capacity Provider
    participant BF as BetterFleet

    Note over CP,BF: Both services are considered online
    Note over CP,BF: Network outage
    BF->>CP: Send Heartbeat
    Note over BF: Heartbeat times out
    Note over CP,BF: Failure risk: connection should now be offline

    Fallback and Gap Handling¶

    Behavioural outcome: sim needs to be able to drive BetterFleet into yellow fallback mode and allow the resulting constraint path to be validated end-to-end.

    Offline Detection With Matching Fallback Forecast¶

    • Establish the connection first.
    • Send a forecast set that includes active primary coverage plus valid fallback coverage for the same managed scope.
    • Stop CP heartbeats or otherwise force BetterFleet offline detection.
    • Verify that BetterFleet enters yellow fallback mode and continues operating against the fallback-derived constraint.
    • Verify that notification and incident context is created for the fallback transition.

    Forecast Gap With Matching Fallback Block¶

    • Send a forecast set where the primary forecast has no active block for now, but a matching fallback block exists.
    • Verify that BetterFleet enters yellow fallback mode.
    • Verify that the effective constraint uses the fallback-derived value for the uncovered period.

    Forecast Gap Without Matching Fallback Block¶

    • Send a forecast set where neither the primary forecast nor fallback forecast covers now.
    • Verify that BetterFleet enters yellow fallback mode.
    • Verify that BetterFleet applies the configured gap policy. The current expected selectable option is the existing circuit safe default or non-OSCP path.
    • Verify that notification and incident context is created for the gap-driven fallback transition.

    Recovery From Fallback Mode¶

    • Restore normal online state and send valid primary forecast coverage for now.
    • Verify that BetterFleet exits fallback mode deterministically and returns to the restored normal path.

    Intentional Disconnect Handling¶

    Behavioural outcome: BetterFleet can hand control back to the operator immediately without waiting for OSCP-derived constraints to expire naturally.

    Disconnect While OSCP Forecast Is Active¶

    • Establish the connection and send an active forecast.
    • Use Manage UI to disconnect the OSCP connection.
    • Verify that BetterFleet clears active OSCP-derived forecast and fallback envelopes immediately.
    • Verify that the managed scope returns to the non-OSCP path and the operator sees explicit handover messaging.

    Disconnect While Fallback Mode Is Active¶

    • Establish the connection, send fallback-capable forecast coverage, and force offline detection so BetterFleet enters yellow fallback mode.
    • Use Manage UI to disconnect the OSCP connection.
    • Verify that fallback mode exits as part of the disconnect flow and control returns to the non-OSCP path.

    Network Outage is Resolved While Still Connected¶

    sequenceDiagram
    participant CP as Capacity Provider
    participant BF as BetterFleet

    Note over CP,BF: Both services are considered online
    Note over CP,BF: Network outage
    BF->>CP: Send Heartbeat
    Note over BF: Heartbeat fails
    Note over CP,BF: Network outage resolved
    BF->>CP: Send Heartbeat
    CP-->>BF: Respond 204
    Note over CP,BF: Failure risk: connection should remain online

    Network Outage is Resolved After Offline Detection¶

    sequenceDiagram
    participant CP as Capacity Provider
    participant BF as BetterFleet

    Note over CP,BF: Both services are considered online
    Note over CP,BF: Network outage
    BF->>CP: Send Heartbeat
    Note over BF: Heartbeat fails
    Note over CP,BF: Offline detection occurs
    Note over CP,BF: Network outage is resolved

    Note over CP,BF: Failure risk: connection should now be offline

    Error Cases¶

    Behavioural outcome: sim state needs to be manually editable to create connection mismatches.

    Note: this is necessary to imitate conditions such as the CP thinking the connection has died, while BetterFleet continues to send heartbeats. Same goes for setting the CP to think it's in the connected state, and then receiving a handshake from BetterFleet. BetterFleet needs to handle these 403 cases gracefully.

    BetterFleet Sends Heartbeat While Capacity Provider is NOT in Online State¶

    sequenceDiagram
    participant CP as Capacity Provider
    participant BF as BetterFleet

    Note over CP: Offline
    Note over BF: Online

    BF->>CP: Send Heartbeat
    CP-->>BF: Respond 403
    Note over CP,BF: Failure risk: connection should now be offline

    Capacity Provider Sends Heartbeat While BetterFleet is NOT in Online State¶

    sequenceDiagram
    participant CP as Capacity Provider
    participant BF as BetterFleet

    Note over CP: Online
    Note over BF: Offline

    CP->>BF: Send Heartbeat
    BF-->>CP: Respond 403
    Note over CP,BF: Failure risk: connection should remain offline

    BetterFleet Sends Handshake/Acknowledge While Capacity Provider is NOT in Offline State¶

    sequenceDiagram
    participant CP as Capacity Provider
    participant BF as BetterFleet

    Note over CP: Online
    Note over BF: Offline

    BF->>CP: Send Handshake
    CP-->>BF: Respond 403
    Note over CP,BF: Failure risk: should not proceed with connection

    Capacity Provider Sends Handshake/Acknowledge While BetterFleet is NOT in Offline State¶

    sequenceDiagram
    participant CP as Capacity Provider
    participant BF as BetterFleet

    Note over CP: Offline
    Note over BF: Online

    CP->>BF: Send Handshake
    BF-->>CP: Respond 403
    Note over CP,BF: Failure risk: should now become offline
    Made with Material for MkDocs
    BFDev Docs Assistant
    New conversation?
    Ask one focused question at a time, this helps the assistant provide accurate answers about what's been implemented in BetterFleet.