Agricultural productivity is subject to various stressors, including abiotic and biotic threats, many of which are exacerbated by climate change. Identifying key yield determinants will, therefore, provide the ability to predict yield and enhance the ability to manage within-field variability. Hence, the collection and integration of multi-layer heterogeneous large datasets and advanced analytics are needed to generate insights about the key constraints to yield potential for optimizing (e.g., water and nitrogen) at the field level or in each management zones (MZ). Thus, we aim to investigate the environmental, biological, and management factors that determine tree-level yield variability of almond orchards. We aim to achieve this by integrating multiple factors that are known to affect yield, including manageable primary resources (i.e., irrigation and fertilizer application), using machine learning algorithms and spatial statistics. The study was conducted in two almond orchards in Israel and the USA. We spatially evaluate the effects of abiotic and biotic stress-causing factors on the yield gap and monitor the crops using standard sampling (i.e., soil and leaves), UAVs (i.e., thermal, LiDAR, multispectral) with high spatial resolution and high revisit time. These by considering dynamic in-season MZs based on detecting abiotic and biotic stress-causing factors (i.e., water status, N-pool, N-use-efficacy, light interception, and vegetative growth), management of sensors or sampling, and remote sensing data. These were transformed into a tree-based data-driven spatial decision support system for irrigation and fertilization, including determining MZs and building prescription irrigation and fertilization maps to optimize differentiated yield.