In business, we often need to make decisions with quantitative and qualitative information. For example, setting shelf presentation levels for hundreds of products: we can’t run every possible test to find the “perfect” configuration, and we also can’t ignore the intuition and experience of human judgment along the way. The real challenge is converging quickly toward the most promising options.
Traditionally, this kind of exploration relies on mathematical techniques that propose possible solutions and refine them over time. But these methods can either be too simple — missing better options — or so complex that they become difficult to understand also slow to run.
Now, imagine treating Large Language Models (LLMs) not just as chat tools, but as powerful probability systems to enable human guided probabilistic processes that propose the next best moves by estimating which options are most likely to succeed, based on context, past patterns, and expert inputs. By pairing these probability-driven proposals with traditional methods, we can explore smarter, converge faster, keep humans in the driver’s seat, and avoid getting lost in irrelevant possibilities.
Using LLMs to Enhance Probability-Based Solutions: The Case of MCMC
To illustrate how LLMs can enhance decision-making, we focus on Markov Chain Monte Carlo (MCMC), a powerful and widely used method for exploring complex problems where direct calculation isn’t feasible. MCMC is a natural choice because it fundamentally relies on proposing new options and deciding whether to accept them, which aligns perfectly with the idea of using smarter, context-aware proposals.
While MCMC traditionally uses simple, fixed proposal patterns like small random steps, these can struggle in complex or constrained scenarios. More advanced variants exist but often come with high computational costs and are harder to customize for specific business contexts.
This sets the stage for LLMs to step in. Unlike fixed mathematical formulas, LLMs generate proposals by learning from rich data patterns and adapting dynamically to the entire history of exploration. This allows them to suggest smarter, more relevant moves that better reflect real-world complexities. Importantly, LLMs can also incorporate human judgment through prompts or fine-tuning, combining machine efficiency with expert insight. This hybrid approach has the potential to accelerate decision-making, improve accuracy, and better align solutions with business goals — especially in fast-moving environments like retail.
Why This Matters for Business
In many real-world situations, especially when things are constantly changing, it’s hard to predict how different factors will behave. This makes it challenging to explore the best solutions effectively.
For instance, take the widely used agentic system called Mixture-of-Experts (MoE),, where several specialized “agents” work together, each given a weight that shows how important it is for the current task. These weights are usually unknown, change depending on the situation, and can follow complex patterns that are hard to predict.
Figuring out these weights is key for the system to perform well. But traditional methods often struggle because:
- The problem’s complexity makes simple exploration slow or ineffective.
- It’s tough to manually design solutions that fit business rules or expert insights.
- The situation can change quickly, shifting the underlying patterns.
Using Large Language Models (LLMs) as adaptive proposal generators offers a promising alternative. LLMs can learn from past data patterns and suggest weight options that naturally adapt to changing contexts and constraints.
Because LLMs understand language, experts can also guide them through prompt engineer, injecting human judgment without the need for complex math. This makes the process more flexible, transparent, and reliable — enabling better business decisions in fast-changing, complex environments.
Toy Example: Metropolis-Hastings with Random Walk
To understand how Large Language Models can improve proposal generation, let’s start with a basic example.
The Metropolis-Hastings (MH) algorithm is a popular method used to generate samples from complex probability distributions — especially when sampling directly is difficult. Think of MH as a random walk through the space of possible solutions. At each step, it proposes a small move (like a step in a random direction) and decides whether to accept this new position based on how well it fits the goal (i.e. target distribution). Over many steps, this process produces samples that represent the overall shape of the solution distribution we want to explore.
This simple random walk approach works, but it can be slow or get stuck in tricky areas when the problem is complex.
Metropolis-Hastings with Random Walk — Pseudo Code
1. Set current_position = starting_point
2. Calculate current_score = probability_score(current_position)
3. Repeat for N steps:
a. Propose a new_position by taking a random step from current_position
b. Calculate new_score = probability_score(new_position)
c. Calculate acceptance_probability:
acceptance_probability = min(1, new_score / current_score)
d. Draw a random number between 0 and 1
If random_number < acceptance_probability:
Accept the move:
current_position = new_position
current_score = new_score
Else:
Stay at the same position
e. Record current_position in the list of samples
4. Return the list of all recorded samples
- We start at a position and score it.
- At each step, we propose a nearby move, evaluate it, and decide whether to accept it based on how much better (or worse) it is compared to the current position.
- Over time, the chain of accepted positions gives us samples that follow the target distribution.
Limitations of Random Walk Proposals
This “random walk” approach works well when the landscape is smooth and not too complex. However, when the distribution has multiple peaks, strong constraints, or hidden structure, simple proposals can be inefficient, which is where we can use LLMs proposal as an adaptive, and more aligned with real-world constraints.
Using an LLM to Replace Random Walk
Instead of using Gaussian noise to propose the next step (like in a standard MH random walk), we can use a LLMs to generate proposals. Here’s the idea:
- We feed the LLM the history of accepted states so far.
- We prompt it to suggest the next candidate number, allowing both increases and decreases from the current state.
- Since we design the LLM’s proposal to be symmetric (meaning it’s equally likely to suggest moving forward or backward between two states), the Metropolis-Hastings acceptance ratio stays simple: the forward and reverse proposal probabilities cancel out.
LLM Proposal in Metropolis-Hastings — Pseudo Code
1. Set current_position = starting_point
2. Calculate current_score = probability_score(current_position)
3. Repeat for N steps:
a. Create an LLM prompt containing:
- The recent sequence of accepted positions
- The current_position
- Instructions: "Propose the next candidate number based on the
sequence of accepted positions, allowing both upward and downward
moves by adding or subtracting a random variation
in the range of {variation}."
b. LLM outputs a proposed new_position
c. Calculate new_score = probability_score(new_position)
d. Calculate acceptance_probability:
acceptance_probability = min(1, new_score / current_score)
(For symmetric proposals, no adjustment needed)
e. Draw a random number between 0 and 1
If random_number < acceptance_probability:
Accept the move:
current_position = new_position
current_score = new_score
Else:
Stay at the same position
f. Record current_position in the list of samples
4. Return the list of all recorded samples
Benefits of LLM Proposals
Because the LLM bases its suggestions on the full sequence of previous states, it can naturally jump between different modes in the distribution, respect constraints, and incorporate human guidance. This adaptive ability helps overcome many limitations of traditional random walk proposals.
Visualizing Sampling: Two-Component Gaussian Mixture
To illustrate the difference, consider a mixture of two Gaussian components with distinct means and variances. The figure below compares samples generated by MH with LLM proposals versus classic random walk. The LLM-guided samples cluster tightly around the true peaks, showing better alignment with the target distribution, while the random walk samples are more spread out and less focused.
The left plot shows samples generated using a Metropolis-Hastings (MH) algorithm with a proposal distribution guided by a large language model (LLM). The samples here cluster tightly around the two true modes of the mixture, reflecting the LLM’s ability to propose values consistent with the underlying distribution’s structure. In contrast, the right plot presents samples from a classic MH random walk proposal, which exhibits more spread and less concentration around the true component means. This difference highlights the advantage of using a more informed proposal, like the LLM, in effectively capturing the key features of the target mixture distribution and producing samples with higher density near the modes.
Applying Context-Aware LLM Proposal Sampling in Mixture-of-Experts
Now let’s look at an example by applying the LLM proposal approach in a retail inventory domain.
In retail inventory intelligence, a Mixture-of-Experts (MoE) system may consist of multiple specialized agents — each expert excels in a different capability such as demand forecasting, supplier lead times, or markdown optimization. The challenge is not simply to sample static weights for these experts, but to assign context-aware weights that reflect the nature of each user’s inquiry.
With LLM-driven proposal sampling, the proposal step itself can reason about multiple factors before suggesting a new weight configuration for the MoE system:
- Intent of the inquiry — Is the user asking to find factual information, interpret existing data, perform analytics, or conduct root cause analysis?
- Complexity of the inquiry — Is it a single straightforward question or a compound, multi-layered request requiring coordinated expert input?
- Domain specificity — Does the question pertain to general operations, a specific category (e.g., perishables), or a specific market/region?
- Data context — What data is available and relevant (e.g., historical sales, supplier performance, weather impact)? Are there constraints such as data freshness, granularity, or reliability?
By evaluating these factors, the LLM can generate weight proposals that are not only probabilistically reasonable, but operationally optimal for the given retail scenario.
In this example, the MoE system includes five expert agents:
- Demand Forecaster — Predicts sales volumes based on historical patterns and external factors.
- Supply Chain Optimizer — Evaluates lead times, vendor reliability, and replenishment schedules.
- Markdown Strategist — Identifies pricing adjustments to clear stock efficiently while preserving margins.
- Category Analyst — Specializes in product category trends and customer preferences.
- Operational Exception Detector — Identifies anomalies such as stockouts, delivery delays, or mis-scanned items.
Example LLM Proposal Prompt
You are optimizing the weights of a Mixture-of-Experts (MoE) system for retail inventory analysis.
The 5 experts are:
1. Demand Forecaster
2. Supply Chain Optimizer
3. Markdown Strategist
4. Category Analyst
5. Operational Exception Detector
Current accepted weight vector: [w1, w2, w3, w4, w5]
Recent accepted weight vectors: [[...], [...], [...], [...], [...]]
New user inquiry:
"Why are we experiencing frequent stockouts for organic produce in the Northeast region, and how should we adjust our replenishment strategy?"
Analyze and reason about:
1. The intent of the inquiry (information retrieval, interpretation, analytics, root cause analysis).
2. The complexity (single or multi-part question).
3. The domain specificity (e.g., category, region).
4. The data context (available data, constraints).
Then propose a new weight vector [w1_new, w2_new, w3_new, w4_new, w5_new] that:
- Sums to 1.
- Assigns higher weight to experts most relevant to the inquiry context.
- Reflects small symmetric variation (±0.1) from current weights unless a large contextual shift is warranted.
- Incorporates both historical accepted weights and reasoning from the current inquiry.
Output only the proposed weight vector as a JSON array.