The issues with AI and its use on that Roswell Final Verdict Show
The way the show uses this system is just eyeballing inducing. The system appears to take in different modalities and put it together and claims a high accuracy.
First the issues with assessing quality of such a system which is extremely non-trivial and doesn't even have a solution for all the possible issues currently
A) If you are going to asses the quality you would need to get a reasonable size pool of people of all ages, ethnicities, genders, and spoken dialects. This would be super inconvenient to do so it was likely not done. You would then need to evaluate the system against each group. A simple measure that say might be useful to regular people would be for a 50/50 lie/honest sample of people of similar demographics what the systems measure lie/honest sample rate. If the model was transparent it would be showing that to the viewers along with the prediction and ideally some measure of error. Without this you wouldn't know if the system because the data it saw thinks all native Americans over 40 always lie.
B) Not only do you have to get samples for all demographics and evaluate against that you need to get samples across all the source variants . By sources it is meant you need to get samples from video recordings from those old film cameras from the 1950's all the way to an expensive RED camera to a cell phone. You need to do the equivalent for the audio channel from a cheap microphone to a Shure SM57 to an iPhones mic.
C) Not only that but you need to randomly completely drop the video feed if at prediction time you are going to do that and then measure again. Any system trained on audio+video is going to get a drop in performance if you drop the video channel.
If the above doesn't make you take their system with a grain of salt. You have the fact that you will then use the system on people on a hot button issue like UFOs which may or not be represented by the data so it won't be like the evaluation above anyways.
To make matters worse for the Roswell documentary they are going to completely sample only lower quality signal since its from the 1950s.
Posted Jul 11, 2021, 12:28 AM UTC
They didn't develop an AI for their show, they more likely teamed up with people already doing research in that field.
Don’t know why you got downvoted but I think that’s what happened. They partnered with a university I think.