In order to meet customer expectations for immediate delivery, retailers offering on-demand food delivery services attempt to strike a balance between maintaining efficient inventory replenishment and drawing appropriate delivery scopes. Recognizing the differentiated distribution characteristics of online and offline demand, this research captures the variation of these demands over time by different functions. Given the dynamics of online and offline demand, the study further constructs a Markov decision framework by considering the real-time in-store inventory as the state and the profit margin as the reward, and then applies the PPO algorithm for hierarchical training on SKU-level delivery scope and inventory replenishment decisions. Through numerical experiments, this study finds that, compared to the baseline strategy that only optimizes the replenishment decision, incorporating delivery scope brings about a significant improvement in cumulative profit under the same number of training iterations. By applying deep reinforcement learning algorithms to the joint optimization of delivery scope and inventory replenishment decisions, the study aims to provide intelligent decision support for retailers offering on-demand food delivery services.