Summary
LocalProperty.translate is all-or-nothing: if any column of a GroupingProperty (or SortingProperty) cannot be translated through a projection, the entire property is discarded. Routine column pruning above an aggregation therefore erases the fact that the data is still grouped by the aggregation's grouping keys, and downstream consumers see a side with no local properties at all.
The information that is lost is not "grouped by the surviving subset" — that would be unsound — but "grouped by a superset of the surviving columns", which is sound and is exactly what several consumers need.
Where
presto-spi/src/main/java/com/facebook/presto/spi/GroupingProperty.java:
@Override
public <T> Optional<LocalProperty<T>> translate(Function<E, Optional<T>> translator)
{
Set<Optional<T>> translated = columns.stream().map(translator).collect(toCollection(LinkedHashSet::new));
if (translated.stream().allMatch(Optional::isPresent)) {
...
return Optional.of(new GroupingProperty<>(columns));
}
return Optional.empty(); // <-- any untranslatable column drops the whole property
}SortingProperty.translate behaves the same way. PropertyDerivations.visitProject translates local properties across the projection's assignments, so a projection that drops one grouping column makes the whole grouping vanish.
Reproduction
SELECT r.regionkey, b.total
FROM region r
JOIN (SELECT regionkey, nationkey, sum(nationkey) AS total
FROM nation
GROUP BY regionkey, nationkey) b
ON r.regionkey = b.regionkeyb.nationkey is not selected, so column pruning removes it above the aggregation. The aggregation produces G(regionkey, nationkey), the projection above it cannot translate nationkey, and the join's build side ends up reporting no local properties — even though the rows are still grouped by (regionkey, nationkey).
Selecting b.nationkey as well keeps the property, so the derived property depends on whether a column happens to be referenced downstream rather than on how the data is actually organized.
Why it matters
Consumers that ask "is this input grouped on a superset of X?" cannot distinguish "not grouped" from "grouped on something I can no longer name". Two examples:
- deciding whether a join input can produce more than one row per join key (i.e. whether the join fans out), which is true precisely when the input is grouped on a strict superset of the join keys;
- any property-driven decision that would otherwise avoid a redundant exchange or aggregation, which silently stops applying after ordinary column pruning.
Suggested direction
Rather than discarding the property, represent the partially translatable case explicitly — for example a GroupingProperty carrying the translated columns plus a flag indicating at least one further, no-longer-nameable grouping column. isGroupedOn(X) would then answer false for such a property (sound, unchanged behaviour for existing callers), while a new predicate could answer "grouped on a superset of X". Returning just the translatable subset would be incorrect, since it would claim a coarser grouping than actually holds.
Happy to put up a PR if this direction seems reasonable.
Summary
LocalProperty.translateis all-or-nothing: if any column of aGroupingProperty(orSortingProperty) cannot be translated through a projection, the entire property is discarded. Routine column pruning above an aggregation therefore erases the fact that the data is still grouped by the aggregation's grouping keys, and downstream consumers see a side with no local properties at all.The information that is lost is not "grouped by the surviving subset" — that would be unsound — but "grouped by a superset of the surviving columns", which is sound and is exactly what several consumers need.
Where
presto-spi/src/main/java/com/facebook/presto/spi/GroupingProperty.java:SortingProperty.translatebehaves the same way.PropertyDerivations.visitProjecttranslates local properties across the projection's assignments, so a projection that drops one grouping column makes the whole grouping vanish.Reproduction
b.nationkeyis not selected, so column pruning removes it above the aggregation. The aggregation producesG(regionkey, nationkey), the projection above it cannot translatenationkey, and the join's build side ends up reporting no local properties — even though the rows are still grouped by(regionkey, nationkey).Selecting
b.nationkeyas well keeps the property, so the derived property depends on whether a column happens to be referenced downstream rather than on how the data is actually organized.Why it matters
Consumers that ask "is this input grouped on a superset of X?" cannot distinguish "not grouped" from "grouped on something I can no longer name". Two examples:
Suggested direction
Rather than discarding the property, represent the partially translatable case explicitly — for example a
GroupingPropertycarrying the translated columns plus a flag indicating at least one further, no-longer-nameable grouping column.isGroupedOn(X)would then answerfalsefor such a property (sound, unchanged behaviour for existing callers), while a new predicate could answer "grouped on a superset of X". Returning just the translatable subset would be incorrect, since it would claim a coarser grouping than actually holds.Happy to put up a PR if this direction seems reasonable.