Files

MayaTheShy b99dbd3d77 fix: update NETWORK_PROTOCOL_INVESTIGATION.md to clarify connection persistence issue resolution and local ID handling

2025-11-10 00:43:01 -05:00

10 KiB

Raw Blame History

Overte Network Protocol Investigation

Current Status (Nov 10, 2025)

✅ CONNECTION PERSISTENCE ISSUE RESOLVED!

Working ✅

DomainConnectRequest packet format - Matches official Overte client implementation
Connection established - Server responds with DomainList packet
Local ID assignment - Server assigns a valid Local ID
Local ID parsing - Fixed byte order bug (was using ntohs() on little-endian data)
Symmetric socket creation - Server creates correct socket to our actual address
Packet version negotiation - Using correct version 27 (DomainConnectRequest_SocketTypes)
Protocol signature - MD5 hash eb1600e798dc5e03c755a968dc16b7fc matches server
Packet structure - All fields properly ordered and serialized per Qt QDataStream format
Source ID handling - Little-endian uint16, correctly set on sourced packets
Connection persistence - Connection now stays alive indefinitely with correct Local ID ✅
Activity tracking - Server recognizes our sourced packets (Ping, AvatarData) and updates lastHeardMicrostamp ✅

Root Cause of Connection Persistence Bug (FIXED)

The connection was being killed after 16 seconds because the server couldn't match our sourced packets to our node.

The Bug:

// WRONG - treated little-endian data as big-endian
uint16_t localID = ntohs(*reinterpret_cast<const uint16_t*>(data + offset));

The Fix:

// CORRECT - read little-endian directly (native byte order on x86)
uint16_t localID;
std::memcpy(&localID, data + offset, sizeof(uint16_t));

Why it failed:

DomainList packet contains Local ID in little-endian format at bytes 32-33
We were using ntohs() which converts FROM network byte order (big-endian) TO host order
This effectively byte-swapped the already-correct little-endian value
Our Ping packets then had the wrong source ID
Server couldn't match packets to our node → treated as "silent" → killed after 16s

Verification:

Before: Server assigned 39772, we parsed 15216 → killed after 16s
After: Server assigned 63157, we parsed 63157 → alive 60+ seconds ✅

Not Working / Unknown ❓

Public socket storage - Server may store mangled IPv6 address (unconfirmed if this affects functionality)

The IPv6 Address Mystery

Observed Behavior

When we connect, the server logs show:

Activating symmetric socket ( "UDP ""127.0.0.1":47505 )  // ✅ CORRECT
Allowed connection from node ... on "UDP ""127.0.0.1":47505  // ✅ CORRECT

But also:
Haven't heard from "UDP ""7f00:1:b991:101:7f00:1:b991:0":28416  // ❌ MANGLED

Analysis of Mangled Address

The IPv6 address 7f00:1:b991:101:7f00:1:b991:0 decodes to these bytes:

7f 00 00 01  b9 91  01 01  7f 00 00 01  b9 91  00 00

Breaking this down:

7f 00 00 01 = 127.0.0.1 (IPv4 in big-endian)
b9 91 = 0xB991 = 47505 (our port in big-endian)
01 = Socket type (UDP)
01 = QHostAddress protocol byte (IPv4)
Then repeats for local socket
Total: 16 bytes = size of IPv6 address

This is our raw socket data being interpreted as IPv6!

Why This Happens

The server initially parses our socket addresses correctly (proven by symmetric socket creation), but somewhere in the process, the public socket address gets stored or displayed in this mangled IPv6 format.

Possible causes:

Qt's QHostAddress might be converting/storing IPv4 in IPv6-mapped format
Server-side display/logging issue rather than actual storage problem
Localhost-specific handling that triggers different code path

DomainConnectRequest Packet Format

Verified Correct Structure

Based on Overte source code analysis (NodeConnectionData::fromDataStream and NodeList.cpp):

// Fields in order (QDataStream format):
1. UUID connectUUID (16 bytes)
2. QByteArray protocolVersion (4-byte length + 16 bytes MD5)
3. QString hardwareAddress (4-byte length + data)
4. QUuid machineFingerprint (16 bytes)
5. QByteArray compressedSystemInfo (4-byte length + compressed JSON)
6. quint32 connectReason (4 bytes)
7. quint64 previousConnectionUpTime (8 bytes)
8. quint64 lastPingTimestamp (8 bytes)
9. NodeType_t nodeType (1 byte - 'I' for Agent/Interface)
10. SocketType publicSocketType (1 byte - 0x01 for UDP)
11. SockAddr publicSockAddr:
    - QHostAddress (1 byte protocol + 4 bytes IPv4)
    - quint16 port (2 bytes big-endian)
12. SocketType localSocketType (1 byte - 0x01 for UDP)
13. SockAddr localSockAddr:
    - QHostAddress (1 byte protocol + 4 bytes IPv4)
    - quint16 port (2 bytes big-endian)
14. QList<NodeType_t> interestList (4-byte count + N bytes)
15. QString placeName (4-byte length + data)
16. QString dsUsername (4-byte length + data) - connect request only
17. QString usernameSignature (4-byte length + data) - connect request only
18. QString domainUsername (4-byte length + data) - connect request only
19. QString domainTokens (4-byte length + data) - connect request only

Key Implementation Details

Socket Type Separation: The socket type byte is written SEPARATELY from the SockAddr. This is critical because:

SockAddr QDataStream operator does NOT include socket type (per ICE requirements)
Socket type is read before each SockAddr in NodeConnectionData::fromDataStream
Our implementation correctly writes: socketType + (protocol + address + port)

QHostAddress Serialization:

Protocol byte: 0 = AnyIP, 1 = IPv4, 2 = IPv6
For IPv4: 1 byte protocol + 4 bytes address (big-endian)
We correctly use writeQHostAddressIPv4() helper function

Source ID (Local ID):

DomainConnectRequest is NON_SOURCED (no source ID field)
After receiving DomainList with Local ID, subsequent packets ARE sourced
Source ID is little-endian uint16 (critical - was big-endian initially, now fixed)

Socket Binding Discovery

Found that the UDP socket must be explicitly bound to get a valid port from the OS:

// Without bind():
getsockname() returns 0.0.0.0:0  // Invalid!

// With bind(INADDR_ANY, 0):
getsockname() returns 0.0.0.0:50948  // Valid port!

We also convert 0.0.0.0 to 127.0.0.1 when connecting to localhost, since sending 0.0.0.0 as our address is invalid.

Ping Protocol

Packet Format

NLPacket packet(PacketType::Ping, version, sourced)
if (m_localID != 0) {
    packet.setSourceID(m_localID);  // Little-endian uint16
}
packet.writeUInt64(micros);  // Timestamp in microseconds
packet.writeUInt8(0);  // Ping type: 0 = local, 1 = public

Sourcing Rules

Before Local ID assignment: Ping is unsourced (no source ID field)
After Local ID assignment: Ping MUST be sourced with our Local ID
ICEPing/ICEPingReply are ALWAYS unsourced (in NON_SOURCED_PACKETS set)
Regular Ping/PingReply are NOT in NON_SOURCED_PACKETS (must be sourced after connection)

NON_SOURCED_PACKETS

Per Overte source (PacketHeaders.h), these packet types do NOT have source ID:

DomainConnectRequest
DomainConnectRequestPending
DomainList
DomainListRequest
DomainDisconnectRequest
DomainServerRequireDTLS
DomainServerPathQuery
DomainServerPathResponse
ICEServerPeerInformation
ICEServerHeartbeat
ICEServerHeartbeatACK
ICEPing
ICEPingReply
ICEServerHeartbeatDenied
StunResponse

All other packet types SHOULD have a source ID field when sent after connection establishment.

Activity Tracking Investigation

Server-Side Logic

From DomainServer.cpp:

nodeList->eachNode([now](const SharedNodePointer& node) {
    quint64 lastHeard = now - node->getLastHeardMicrostamp();
    if (lastHeard > 2 * USECS_PER_SECOND) {
        qCDebug(domain_server) << "Haven't heard from " 
            << node->getPublicSocket() << username 
            << " in " << lastHeard / USECS_PER_MSEC << " msec";
    }
});

The server checks node->getPublicSocket() in the "Haven't heard" message, which is why we see the mangled IPv6 address.

Timeout Threshold

From LimitedNodeList.cpp:

if ((usecTimestampNow() - node->getLastHeardMicrostamp()) 
    > (NODE_SILENCE_THRESHOLD_MSECS * USECS_PER_MSEC))

Default threshold appears to be ~16-20 seconds based on our observations.

Why lastHeardMicrostamp Doesn't Update