NeuralOS: Towards Simulating Operating Systems via Neural Generative Models

Abstract

We introduce NeuralOS, a neural framework that simulates graphical user inter- faces (GUIs) of operating systems by directly predicting screen frames in response to user in- puts such as mouse movements, clicks, and keyboard events. NeuralOS combines a recur- rent neural network (RNN), which tracks computer state, with a diffusion-based neural renderer that generates screen images. The model is trained on a large-scale dataset of Ubuntu XFCE recordings, which include both randomly generated interactions and real- istic interactions produced by AI agents. Experiments show that NeuralOS successfully renders realistic GUI sequences, accurately captures mouse interactions, and reliably pre- dicts state transitions like application launches. Although modeling fine-grained keyboard interactions precisely remains challenging, NeuralOS offers a step toward creating fully adaptive, generative neural interfaces for future human-computer interaction systems.