Calculating a radar position in "slanted" polar coordinates

First, apologies if this is not the right place for me to post this (please redirect me if so). This is based on a real problem I encountered today, and I'm wondering if my solution is actually correct.

The problem is as follows. I have some aircraft positions, but they're expressed in "slanted" polar coordinates, whereas what I want is just the latitude/longitude of the aircraft's ground position (that is, the projection of the aircraft directly onto the Earth).

Now I already have a function that can take a latitude/longitude position, a bearing, and a spherical distance and compute the latitude/longitude of the target ground position. But for these positions I have slightly different data, specifically:

• The latitude/longitude of the radar facility that observed the aircraft.
• The direct line of sight Euclidean distance from the facility to the aircraft.
• The altitude of the aircraft.

Let's call the radar facility's location F, the line of sight distance d, the altitude y, the aircraft's projected ground position G, the aircraft's in-air position A, the center of the earth O, and the radius of the earth r. Then I want the spherical distance d' from F to G given F, d, and y. This problem reduces to finding the radian measure of the angle FOG.

Now my first attempt was based on an incorrect assumption. Specifically, I thought that angle FGA was a right angle (where the segment FG is intended to be a straight line that would pass through the Earth, i.e., it's not a spherical segment). If it were, then the length of FG could be found with the Pythagorean theorem and the angle FOG could then be found from the law of cosines.

This produced the following (incorrect) result:

d' = r * acos( (2r^2 - d^2 + y^2) / (2r^2) ).

Despite this being incorrect, I did find that it seemed to have mostly all the properties that I expected the real solution to have.

Anyway, I then tried to find the correct solution without my false assumption. I didn't really have any good ideas on how to approach this, so I wrote everything in the complex plane, using Euler's formula, and put together a bit of a monster equation to solve. I was not able to solve this by hand, but Wolfram Alpha gave me the following result:

d' = r * acos( (2r^2 + 2ry - d^2 + y^2) / (2r^2 + 2ry) )

This is shockingly close to the result above even though the second approach was dramatically more messy and I couldn't even do it by hand.

I'm hoping someone has some insight on whether this is actually correct. I can post the details (excluding whatever magical manipulations Wolfram Alpha did) of the second approach if desired.