Porting Tracy to the Wii Writeup
Platform Define
For the Wii.cmake toolchain file we get __wii__ and HW_RVL (for revolution) defined for the Wii. For the Gamecube file we get __gamecube__ and HW_DOL (for dolphin)
Big Endian
Tracy has no support for big endian devices like the Wii so I had to implement support myself. This meant I had to gain an understanding of how Tracy sends and receives data in order to come up with the best solution for endian conversions.
Ideally we can have all data sent across the network in big endian, so our Wii doesn’t have to do any conversions at runtime.
The Initial Handshake
The handshake is the first exchange of information that occurs when connecting to a client. The server (the PC running the profiler GUI) will send something known as a shibboleth which is just an identification string known at compile-time and the protocol version which is a 32-bit unsigned integer.
If shibboleth or protocol version doesn’t match on the client, the connection will be rejected. Otherwise the client will send a WelcomeMessage which contains a bunch of data about the client like cpuid, sampling period, timer multiplier, program name, and pid.
struct WelcomeMessage
{
double timerMul;
int64_t initBegin;
int64_t initEnd;
uint64_t resolution;
uint64_t epoch;
uint64_t exectime;
uint64_t pid;
int64_t samplingPeriod;
uint8_t flags;
uint8_t cpuArch;
char cpuManufacturer[12];
uint32_t cpuId;
char programName[WelcomeMessageProgramNameSize];
char hostInfo[WelcomeMessageHostInfoSize];
}So we need a clever way to byteswap the protocol version and the elements within WelcomeMessage if we want to send them across the network in big endian. I imagine we’re going to have to do this A LOT. So we must make it as easy as possible to convert structs and primitives.
Byteorder conversion functions
Let’s first make a new compile-time definition: TRACY_BIGENDIAN. When defined, all data sent through the network must be in big endian byteorder. If not defined, then all data will be sent in little endian byteorder. So theoretically we could have TRACY_BIGENDIAN not defined and big endian devices would know to convert to little endian.
Also, if the system’s native byteorder is the same as the network’s byteorder, then all the endian conversion code should be no-ops. Switching byteorder from little endian to big endian and vice versa is the exact same operation which is handy.
We can start with a function that will return the byteorder we are using for the network.
constexpr std::endian network_byteorder()
{
#if TRACY_BIGENDIAN
return std::endian::big;
#else
return std::endian::little;
#endif
}We can use this to check at compile-time if the host’s byteorder is the same as the network’s:
if constexpr( std::endian::native == network_byteorder() )Big shoutout to C++20 for adding std::endian!
Let’s start by handling our integer types:
template<typename T>
requires std::is_integral_v<T>
constexpr T convert_endian( T value ) noexcept
{
if constexpr( std::endian::native == network_byteorder() || sizeof( T ) == 1 )
{
return value;
}
else if constexpr( sizeof( T ) == 2 )
{
return static_cast<T>( __builtin_bswap16( static_cast<uint16_t>( value ) ) );
}
else if constexpr( sizeof( T ) == 4 )
{
return static_cast<T>( __builtin_bswap32( static_cast<uint32_t>( value ) ) );
}
else if constexpr( sizeof( T ) == 8 )
{
return static_cast<T>( __builtin_bswap64( static_cast<uint64_t>( value ) ) );
}
}There are various enums throughout the codebase, so we can handle those now too:
template<typename T>
requires std::is_enum_v<T>
constexpr T convert_endian( T value ) noexcept
{
return static_cast<T>( convert_endian( static_cast<std::underlying_type_t<T>>( value ) ) );
}Next up is floats and doubles. Here I’ve made the assumption that whatever CPU is running this conversion code is IEE-754 compliant. This means that binary 32-bit and binary 64-bit formats don’t have padding bits. So I can treat a floating point type’s bytes the same as an integer type’s when swapping.
Instead of doing a memcpy into a uint type or a reinterpret_cast, we can use C++20′s std::bit_cast!
template<typename T>
requires std::is_floating_point_v<T>
constexpr T convert_endian( T value ) noexcept
{
if constexpr( std::endian::native == network_byteorder() )
{
return value;
}
else if constexpr( sizeof( T ) == sizeof( float ) )
{
static_assert( sizeof( float ) == sizeof( uint32_t ), "float must be 4 bytes" );
uint32_t tmp = std::bit_cast<uint32_t>( value );
tmp = convert_endian( tmp );
return std::bit_cast<float>( tmp );
}
else if constexpr( sizeof( T ) == sizeof( double ) )
{
static_assert( sizeof( double ) == sizeof( uint64_t ), "double must be 8 bytes" );
uint64_t tmp = std::bit_cast<uint64_t>( value );
tmp = convert_endian( tmp );
return std::bit_cast<double>( tmp );
}
}Lastly, we need a cunning way to convert structs. There’s no generic one size fits all solution here for structs that aren’t trivial and have some padding. so structs will have to implement a method called convert_endian() which will byteswap every element of the struct.
template<typename T>
concept StructWithConvertEndianMethod = requires( T value ) {{ value.convert_endian()}; };
template<StructWithConvertEndianMethod T>
constexpr void convert_endian( T& value )
{
if constexpr( std::endian::native == network_byteorder() )
{
return;
}
else
{
value.convert_endian();
}
}So that WelcomeMessage struct mentioned earlier will have a convert_endian() method like so:
void convert_endian()
{
timerMul = ::convert_endian( timerMul );
initBegin = ::convert_endian( initBegin );
initEnd = ::convert_endian( initEnd );
resolution = ::convert_endian( resolution );
epoch = ::convert_endian( epoch );
exectime = ::convert_endian( exectime );
pid = ::convert_endian( pid );
samplingPeriod = ::convert_endian( samplingPeriod );
flags = ::convert_endian( flags );
cpuArch = ::convert_endian( cpuArch );
cpuId = ::convert_endian( cpuId );
}Now we have all the tools needed to effortlessly byteswap all the things! We don’t even have to think about checking host byteorder since that’s figured out for us too at compile-time and convert_endian() will vanish to a no-op if needed!
Converting the byteorder of the protocol version and welcome message were no problem, and we’ve successfully gotten past the handshake!
Sending and Receiving Queue Items
You ready for the struct to end all structs? In some ways this could be seen as a blessing though. After the handshake tracy sends everything else as compressed lz4 and this is done in one place generalized with this QueueItem struct.
So we can just create one big fat convert_endian() method for QueueItem to do the endian conversion before sending it to be compressed and sent.
struct QueueItem
{
QueueHeader hdr;
union
{
QueueThreadContext threadCtx;
QueueZoneBegin zoneBegin;
QueueZoneBeginLean zoneBeginLean;
QueueZoneBeginThread zoneBeginThread;
QueueZoneEnd zoneEnd;
QueueZoneEndThread zoneEndThread;
QueueZoneValidation zoneValidation;
QueueZoneValidationThread zoneValidationThread;
QueueZoneColor zoneColor;
QueueZoneColorThread zoneColorThread;
QueueZoneValue zoneValue;
QueueZoneValueThread zoneValueThread;
QueueStringTransfer stringTransfer;
QueueFrameMark frameMark;
QueueFrameVsync frameVsync;
QueueFrameImage frameImage;
QueueFrameImageFat frameImageFat;
QueueSourceLocation srcloc;
QueueZoneTextFat zoneTextFat;
QueueZoneTextFatThread zoneTextFatThread;
QueueLockAnnounce lockAnnounce;
QueueLockTerminate lockTerminate;
QueueLockWait lockWait;
QueueLockObtain lockObtain;
QueueLockRelease lockRelease;
QueueLockReleaseShared lockReleaseShared;
QueueLockMark lockMark;
QueueLockName lockName;
QueueLockNameFat lockNameFat;
QueuePlotDataInt plotDataInt;
QueuePlotDataFloat plotDataFloat;
QueuePlotDataDouble plotDataDouble;
QueueMessage message;
QueueMessageColor messageColor;
QueueMessageLiteral messageLiteral;
QueueMessageLiteralThread messageLiteralThread;
QueueMessageColorLiteral messageColorLiteral;
QueueMessageColorLiteralThread messageColorLiteralThread;
QueueMessageFat messageFat;
QueueMessageFatThread messageFatThread;
QueueMessageColorFat messageColorFat;
QueueMessageColorFatThread messageColorFatThread;
QueueGpuNewContext gpuNewContext;
QueueGpuZoneBegin gpuZoneBegin;
QueueGpuZoneBeginLean gpuZoneBeginLean;
QueueGpuZoneEnd gpuZoneEnd;
QueueGpuTime gpuTime;
QueueGpuCalibration gpuCalibration;
QueueGpuTimeSync gpuTimeSync;
QueueGpuContextName gpuContextName;
QueueGpuContextNameFat gpuContextNameFat;
QueueGpuAnnotationName gpuAnnotationName;
QueueGpuAnnotationNameFat gpuAnnotationNameFat;
QueueMemAlloc memAlloc;
QueueMemFree memFree;
QueueMemDiscard memDiscard;
QueueMemNamePayload memName;
QueueThreadGroupHint threadGroupHint;
QueueCallstackFat callstackFat;
QueueCallstackFatThread callstackFatThread;
QueueCallstackAllocFat callstackAllocFat;
QueueCallstackAllocFatThread callstackAllocFatThread;
QueueCallstackSample callstackSample;
QueueCallstackSampleFat callstackSampleFat;
QueueCallstackFrameSize callstackFrameSize;
QueueCallstackFrameSizeFat callstackFrameSizeFat;
QueueCallstackFrame callstackFrame;
QueueSymbolInformation symbolInformation;
QueueSymbolInformationFat symbolInformationFat;
QueueCrashReport crashReport;
QueueCrashReportThread crashReportThread;
QueueSysTime sysTime;
QueueSysPower sysPower;
QueueContextSwitch contextSwitch;
QueueThreadWakeup threadWakeup;
QueueTidToPid tidToPid;
QueueHwSample hwSample;
QueuePlotConfig plotConfig;
QueueParamSetup paramSetup;
QueueCpuTopology cpuTopology;
QueueExternalNameMetadata externalNameMetadata;
QueueSymbolCodeMetadata symbolCodeMetadata;
QueueSourceCodeMetadata sourceCodeMetadata;
QueueSourceCodeNotAvailable sourceCodeNotAvailable;
QueueFiberEnter fiberEnter;
QueueFiberLeave fiberLeave;
QueueGpuZoneAnnotation zoneAnnotation;
};
};Only gotcha was this variant which stores its size value in a non standard 6 byte manner
struct QueueMemAlloc
{
int64_t time;
uint32_t thread;
uint64_t ptr;
char size[6];
};since this only happens once I just explicitly converted it in place instead of bothering with a new convert_endian implementation.
And it only grows larger as more features are added to Tracy. So we write a python script which parses the C++ header and generates the convert_endian() method. That way, we can just run the script whenever we upgrade Tracy and the new gigantic method will be regenerated.
This approach worked well and we were able to see profiling data from the Wii on Tracy’s UI! As for performance, our Wii was able to hit a consistent 62 FPS with a few zones.
Threading
libogc has a very straightforward threading implementation under ogc/lwp.h functions are prefixed with LWP_ We can create, join, get self, sleep, signal, set priority and more.
class Thread
{
public:
Thread( void ( *func )( void* ptr ), void* ptr )
: m_func( func )
, m_ptr( ptr )
{
LWP_CreateThread( &m_thread, Launch, this, nullptr, 0, 64 );
}
~Thread()
{
LWP_JoinThread( m_thread, nullptr );
}
[[nodiscard]] lwp_t Handle() const { return m_thread; }
private:
static void* Launch( void* ptr )
{
( (Thread*)ptr )->m_func( ( (Thread*)ptr )->m_ptr );
return nullptr;
}
void ( *m_func )( void* ptr );
void* m_ptr;
lwp_t m_thread;
};The implementation is pretty much identical to Tracy’s pthread implementation:
class Thread
{
public:
Thread( void(*func)( void* ptr ), void* ptr )
: m_func( func )
, m_ptr( ptr )
{
pthread_create( &m_thread, nullptr, Launch, this );
}
~Thread()
{
pthread_join( m_thread, nullptr );
}
pthread_t Handle() const { return m_thread; }
private:
static void* Launch( void* ptr ) { ((Thread*)ptr)->m_func( ((Thread*)ptr)->m_ptr ); return nullptr; }
void(*m_func)( void* ptr );
void* m_ptr;
pthread_t m_thread;
};Also had to do a similar sort of thing for TracyMutex
class TracyMutex
{
mutex_t m_mutex;
public:
TracyMutex() { LWP_MutexInit( &m_mutex, false ); }
~TracyMutex() { LWP_MutexDestroy( m_mutex ); }
void lock() { LWP_MutexLock( m_mutex ); }
void unlock() { LWP_MutexUnlock( m_mutex ); }
bool try_lock() { return LWP_MutexTryLock( m_mutex ) == 0; }
};Networking
The Wii appears to have equivalent functions for most of the POSIX socket API used by Tracy. However the addrinfo struct doesn’t exist, and so neither does getaddrinfo() and freeaddrinfo(). What even is addrinfo? Here’s the description for getaddrinfo() in the man-pages:
> Given node and service, which identify an Internet host and a service, getaddrinfo() returns one or more addrinfo structures, each of which contains an Internet address that can be specified in a call to bind(2) or connect(2). The getaddrinfo() function combines the functionality provided by the gethostbyname(3) and getservbyname(3) functions into a single interface, but unlike the latter functions, getaddrinfo() is reentrant and allows programs to eliminate IPv4-versus-IPv6 dependencies.
So basically it’s a protocol-agnostic way of representing an address. The Wii does have a net_gethostbyname() function so maybe we just need to learn more about that. Maybe it’ll do everything we need without the additional features of addrinfo.
> The gethostbyname(), gethostbyaddr(), herror(), and hstrerror() functions are obsolete. Applications should use getaddrinfo(3), getnameinfo(3), and gai_strerror(3) instead.
So it’s obsolete. Wonder why? My guess is probably because it’s pre-IPv6?
> The gethostbyname() function returns a structure of type hostent for the given host name. Here name is either a hostname or an IPv4 address in standard dot notation.
Yup this is IPv4 only. And that makes sense because the Wii doesn’t support IPv6, so there’s no need solve IPv6 problems on it.
UDP Broadcast
The Wii’s socket API would not allow me to open up a UDP socket. Although it’s a nice-to-have and not a requirement, it prevents me from making the Wii advertise itself to tracy servers.
Timing
Libogc has a very convenient macro called ticks_to_nanosecs which converts the Wii’s CPU ticks to nanoseconds for us! We can get CPU ticks with gettime()
#define ticks_to_nanosecs(ticks) ((((u64)(ticks)*8000)/(u64)(TB_TIMER_CLOCK/125)))64-bit Atomics
Tracy makes use of 64-bit atomics, and the Wii toolchain doesn’t have an implementation for 64-bit atomics in the standard library. Fortunately we are super lucky that the Wii only has a single core CPU. This means that the only way to achieve concurrency is to use an interrupt handler. So we can simply emulate atomic operations by disabling interrupts temporarily.
uint64_t __atomic_load_8( const volatile void* ptr, int )
{
uint32_t level = IRQ_Disable();
uint64_t val = *(const volatile uint64_t*)ptr;
IRQ_Restore( level );
return val;
}
void __atomic_store_8( volatile void* ptr, uint64_t val, int )
{
uint32_t level = IRQ_Disable();
*(volatile uint64_t*)ptr = val;
IRQ_Restore( level );
}
uint64_t __atomic_fetch_add_8( volatile void* ptr, uint64_t val, int )
{
uint32_t level = IRQ_Disable();
uint64_t old = *(volatile uint64_t*)ptr;
*(volatile uint64_t*)ptr = old + val;
IRQ_Restore( level );
return old;
}The compiler only asked for these three implementations. So reading the value, assigning the value, and adding to the value.
Memory Reductions
Here’s a snippet from the profiler constructor. The Wii has 88MB of total system memory and these queues are too large, causing the Wii program to crash. Fortunately I haven’t noticed any issues severely downsizing them!
, m_serialQueue( 1024*1024 )
, m_serialDequeue( 1024*1024 )
#ifndef TRACY_NO_FRAME_IMAGE
, m_fiQueue( 16 )
, m_fiDequeue( 16 )
#endif
, m_symbolQueue( 8*1024 )
, m_frameCount( 0 )
, m_isConnected( false )
#ifdef TRACY_ON_DEMAND
, m_connectionId( 0 )
, m_symbolsBusy( false )
, m_deferredQueue( 64*1024 )
#endif