The Architecture of Opportunity: An AI-Powered Approach for Large-Scale Discovery and Recommendation

Abstract

Research organisations often struggle to plan proactively because signals about opportunities are dispersed across heterogeneous and largely unstructured web sources (e.g., researcher profiles, grant portals, and public announcements). Traditional tools such as SWOT support reflection, but they rarely operationalise these signals into measurable factors that enable continuous, data-driven opportunity discovery and recommendation. This thesis develops an end-to-end framework that transforms web-derived unstructured information into high-quality stakeholder opportunity datasets and recommendation outputs. The thesis is organised into three stages. Stage 1 (Data Collection and Data Quality) defines a unified stakeholder and opportunity data model and develops an extraction pipeline that integrates prompt engineering and few-shot GenAI extraction with traditional ML and rule-based validation to populate feature values from unstructured sources. It further improves dataset reliability through data cleaning, normalisation, and ML-based imputation of missing stakeholder and opportunity features. Stage 2 (Recommendation for a Known Opportunity) investigates the setting where the opportunity is given (e.g., a specific grant). It designs and evaluates models that rank and recommend the most relevant stakeholders using evidence derived from both structured fields and unstructured text, leveraging pre-trained language models to represent and retrieve matching signals. Stage 3 (Recommendation for an Unknown Opportunity) addresses the discovery of opportunities that are not provided in advance. It develops and validates methods to identify previously unseen opportunities from external signals and recommend them to suitable stakeholders. Overall, the thesis contributes a practical pipeline from web data to opportunity matching and opportunity discovery for research and educational institutions.