Guide 03 / Comparison and software evaluation

Test the alternatives

A useful comparison explains what changed, what stayed fixed and why the preferred option remains credible when assumptions move.

Build a repeatable baseline

Begin with a baseline the team can reproduce, using a controlled input set and a documented software version. Record aircraft configuration, seat and monument definitions, commercial assumptions and comfort criteria. If the baseline cannot be regenerated, later gains are difficult to distinguish from hidden changes in data or settings.

Define each alternative as a deliberate intervention. You might investigate a different cabin split, a revised service concept or another seat family. Change one group of variables at a time when the purpose is to understand cause and effect. Combined concepts can follow, but preserve a record of how their individual changes interact.

Use neutral scenario names. “Premium improvement” assumes the conclusion before it is tested; a name tied to the actual configuration is easier to audit. Also preserve rejected candidates and the reasons for rejection. They reveal binding constraints and prevent the team from repeatedly revisiting an option that failed for a known reason.

Ask what the AI actually does

“AI-powered” can describe very different functions. A tool might generate arrangements, predict a comfort proxy, search a design space or produce a written explanation. These functions have different evidence needs. Ask the supplier to identify which outputs come from machine learning, deterministic rules, numerical optimisation or manual input.

Constraint programming and mathematical optimisation can be valuable without a learned model. Conversely, a learned surrogate may accelerate a calculation while introducing approximation error. Neither approach is inherently more trustworthy because of its label. Evaluate fitness for the intended task, traceability and the consequences of an incorrect recommendation.

A language model can help structure requirements or summarise an input register, but it can also invent a dimension, omit an exception or state a requirement too confidently. Do not use fluent narrative as proof that a layout has been checked. Require the tool to separate retrieved sources, calculated results and generated commentary.

Assumption note: ask for a failed demonstration

Give the supplier a deliberately conflicting input set. A useful tool should identify infeasibility or unresolved conditions, not silently relax a hard rule to produce a pleasing arrangement. Ask how the conflict is logged and communicated to the user.

Keep objectives inspectable

Cabin planning is multi-objective. Capacity, passenger space, service practicality, mass, installation effort and commercial performance may pull in different directions. Define metrics separately before considering any weighted score. A single ranking can obscure a significant loss in one area behind smaller gains elsewhere.

A Pareto comparison identifies options for which improving one stated objective requires worsening another. It does not select the airline’s preferred compromise. That decision depends on priorities, uncertainty and responsibilities that belong to people. Show the trade-off surface where practical, then record why a particular compromise was chosen.

If a score is used, disclose units, normalisation and weights. Test whether reasonable alternative weights change the ranking. An option that wins only under a narrow weighting scheme should be described as preference-sensitive. The same applies to financial assumptions: a revenue estimate is conditional on demand, pricing and operational effects, not a property of the seat map alone.

Evaluation template — no aircraft performance values implied
DimensionHold or discloseAsk the tool to show
GeometryIdentical reference configurationChanged envelopes and conflicts
Passenger comfortSame population and task assumptionsGroup-specific results and limitations
Commercial modelDemand and cost assumptionsSensitivity, not just a point estimate
Search processSettings, seed where applicable, stopping ruleFeasibility status and reproducibility

Test robustness and practical fit

Vary uncertain assumptions across defensible ranges. Useful candidates for sensitivity checks include baggage load, passenger mix, service timings and estimated equipment mass. Distinguish sensitivity analysis from validation: showing that a result is stable within the model does not establish that the model represents the real cabin accurately.

For stochastic methods, retain random seeds where available and repeat runs to understand variability. Ask what the stopping rule means and whether the reported solution is proven optimal, locally improved or simply the best found within a time budget. “Optimised” is not a sufficiently precise result description.

Software also has to fit the planning workflow. Test export formats, units, revision history, permissions and the ability to retrieve the exact input set behind a recommendation. Review data handling before uploading proprietary aircraft drawings or passenger information. A polished demonstration on clean sample data is not evidence that the system will manage your configuration control.

End the comparison with a short decision record: chosen concept, alternatives considered, controlling trade-offs, sensitive assumptions and unresolved checks. Carry this record into engineering review. The purpose of a planning tool is not to remove judgement; it is to make judgement better informed and easier to challenge.

Decision gate: the preferred option can be reproduced, its ranking can be explained and its remaining uncertainties have owners.