Property Finder corrected a biased AI evaluation during its migration to Amazon Bedrock
Property Finder found that 96.8 % of its annotations used for evaluation were taken from predictions by the existing model. According to AWS, new expert annotations and prompt revisions enabled more accurate analysis of real estate photos with costs that were 86 % lower.
When migrating its analysis of real estate photos to Amazon Bedrock, Property Finder found that its evaluation favored the existing model. A total of 96.8 % of the annotations used for evaluation came directly from its predictions without independent verification by experts. Alternatives were therefore assessed on their agreement with the existing model, rather than their actual accuracy. The system processes tens of thousands of photos daily and evaluates 11 attributes for each, such as room type, watermarks or image quality.
The team created new expert annotations for 356 images from the United Arab Emirates. The first 256 were used for tuning, while the final comparison used all 356, including the images used during development. The evaluation now allowed multiple labels for a single photo, for example when a shot showed several rooms. The main metric was Micro F1; the supplementary Macro F1 was intended to flag weaker results for less common classes.
Working with the AWS Generative AI Innovation Center team, the company iterated on prompt versions v0 through v27. It compared predictions in JSON format with expert annotations and used error analysis to prepare further prompt versions; later, it used automated optimization. According to AWS, the solution achieved higher accuracy with costs that were 86 % lower. This is the result of this particular case study; for rare classes, the dataset may also have been too small to detect smaller differences. Details are available in the source article.
Why it matters
An evaluation dataset taken from the outputs of the existing model can prevent the selection of a more suitable replacement. The case demonstrates a concrete process for separating accuracy checks from mere agreement with the original system and linking error analysis to prompt revisions.
Two audiences, two different impacts
What this means
For individuals
A developer comparing image models needs to verify the origin of the reference annotations. The resulting ranking may reflect the behavior of the original model if its predictions serve as the correct answers.
For a business
At Property Finder, correcting the evaluation made it possible to reassess replacing the system that checks photos before listings are published. According to AWS, the resulting solution combined higher accuracy with costs that were 86 % lower, in daily operations involving tens of thousands of images.
ProcessesCheck the original
Event sources
only one source so far · 1 publisher, 0 independent. We count feeds from the same owner only once.