Troubleshooting HomeKit: what tools to use and how (?)

I've seen lots of posts and comments about wifi and HomeKit issues, instability of this, instability of that. But not much in the way of actually hunting down root cause. I have seen even less on how to go about troubleshooting HomeKit. So is it a wifi issue? Or is it HomeKit?

For a guy who used to network analysis for a living, I've been curious about how to go about the HomeKit part. So, I did just a little bit of research to appease my curiosity, and figured I'd post and see if anyone wants to group source from there. Sorry, this might go over the head of some non-networking people, but might eventually help drive more concrete suggestions for everyone, if this goes anywhere, that is.

Looks like there are two main higher layer protocols in play: Bonjour (mdns) and HAP (HomeKit Accessory Protocol).

https://developer.apple.com/bonjour/

https://developer.apple.com/support/homekit-accessory-protocol/

HAP utilizes Bonjour for discovery. So let's start with Bonjour. It turns out that MacOS includes a CLI command to query mdns - dns-sd (DNS Service Discovery). Personally, I'd recommend downloading "Discovery - DNS-SD Browser" from the Mac and/or iOS App Stores so you don't have to deal with dropping to a shell.

HomeKit uses the service type of '_hap._tcp' on the 'local.' domain, so you can query for HomeKit devices as such:

$ dns-sd -Z _hap._tcp local.

For HomeKit hubs, look for the service type of _homekit._tcp:

$ dns-sd -Z _homekit._tcp local.

If using that "Discovery" app, just click/tap into that section and the info is right there. Sweet!

If you don't see your problematic device there, that's a problem. It could come down to basic network connectivity (wifi, etc) or an mdns issue. Not really sure how to troubleshoot an mdns issue, though. Thoughts?

If your problematic device does show up in the results, then you at least have something to work with. The bonjour info includes the hostname, IP address(es), MAC address(es) and port(s) for the service. Presumably, a device would drop of the results pretty quickly if it drops off the network (immediately? cached? Does only the device itself respond to mdns queries or are there servers/proxies that will also respond?).

Next step would presumably be to make sure you can establish a connection to that service. Using 'nc' or 'telnet' (if you have it), verify you can reach the hostname:port being advertised. If you can, then your issue doesn't seem to be basic networking or Bonjour, so you probably don't want to go replacing your wifi router just yet. But if you can't reach it, make sure it's pingable (you may need to verify pingability when it's working, as some devices don't ever respond to ping).

What else do people have in their bag of tricks? Hopefully without having to grab a packet trace, because not a lot of people have that skill.

What would you do next to dig into an mdns issue, or where would you start on troubleshooting HAP if mdns all checks out fine?