What Slows Down FMware Development? An Empirical Study of Developer Challenges and Resolution Times

Abstract

Foundation Models (FMs), such as GPT-4, have revolutionized software engineering by enabling the development of FMware — applications and infrastructures built around these powerful models. Despite their transformative potential, FMware solutions face significant challenges in their development, deployment, and maintenance, particularly across cloudbased and on-premise platforms; this is because many of the goals, processes, tools, and technical assets of FMware development are different from those of traditional software systems. This study presents an empirical investigation of the current FMware ecosystem, focusing on three key questions: (1) what topics are most prevalent in issue discussions of FMware systems, (2) what specific challenges are commonly faced by FMware developers, and (3) what kinds of issues in FMware development have the longest resolution times? Our analysis uses data extracted from both GitHub repositories of FMware systems as well as systems hosted on popular FMware platforms such as HuggingFace, GPTStore, Ora, and Poe. Our findings reveal a strong emphasis on education, content creation, and business strategy, alongside critical technical challenges such as memory errors, dependency management, and tokenizer configurations. We further identify bug reports and core functionality issues as the most common problem types on GitHub, and show that topics concerning code review, similarity search, and prompt template design require the longest time to resolve. By uncovering insights into developer practices and pain points, this research highlights opportunities for improving FMware development tools, workflows, and community support. These insights contribute to a deeper understanding of the current FMware landscape and provide actionable recommendations for practitioners and researchers.