Structural grammars, combined with form-finding, enable the generation of a wide variety of lightweight structures from a small set of rules, while simultaneously ensuring their static equilibrium. Reinforcement learning presents a unique opportunity to leverage such rule-based systems by learning optimal sequences of rule applications, resulting in novel and efficient structures. Bleker et al. established the theoretical framework for integrating the Combinatorial Equilibrium Modelling (CEM) form-finding technique with Deep Reinforcement Learning and Graph Neural Networks (GNNs), by defining a 2D rule-based action space and representing the resulting pin-jointed structures as graphs. Building on this and extending the action space to 3D, we implement and train a GNN-powered Deep Q-learning agent that, given a set of loads and support conditions, learns to sequentially design structures by selecting optimal actions from the CEM grammar ruleset. The primary challenge lies in enabling the agent to generalize to a wide range of design problems that extend beyond its training experience. We address this by focussing our investigation on two key directions – First, we structure and develop a curriculum learning algorithm that lets the agent builds experience progressively and tackle increasingly complex design challenges, with improved sample efficiency and stability. Second, we design and refine the GNN architecture and the underlying state representations to better capture the topological and physical relationships within structures, thereby enabling the agent to recognize similarities across otherwise unrelated scenarios. Through a combination of case studies and ablation studies, we evaluate the agent’s design capabilities and analyse the impact of individual architectural and algorithmic choices. We envision this framework to be a powerful tool for early-stage structural design, wherein an intelligent agent capable of form-finding based on a grammar-based ruleset can meaningfully support and collaborate with architects and engineers.