Resources on node tags not working

[ClementFranger]

Problem :

Hello, I have tried to add a specific resource for my node tagged 'training', but this doesn't work. Here is my sagemaker.yaml :

And here is my data_science pipeline :

Kedro-Sagemaker instead put the default ml.t3.medium for my nodes tagged 'training' instead of ml.g4dn.xlarge

Solution :

This is due to this particular piece of code : KedroSageMakerGenerator._get_resources_for_node, particularly this code :

        node_resources = next(
            (
                self.config.aws.resources.get(n)
                for n in chain([node.name], iter(node.tags))
            ),
            None,
        )

This returns the next resources available which is always going to be the node one, even if it None. The correct logic should be something like :

        node_resources = next(
            (
                self.config.aws.resources.get(n)
                for n in chain([node.name], iter(node.tags)) **if self.config.aws.resources.get(n) is not None**
            ),
            None,
        )

Error :

You can test with this little script :

from itertools import chain

resources = {'__default__': dict(instance_type='ml.t3.medium'), 'training': dict(instance_type='ml.g4dn.xlarge')}

node_resources = next(
    (
        resources.get(n)
        for n in chain(['split_data_node'], iter(['training']))
    ),
    None,
)

print(node_resources)

This will return None but should return dict(instance_type='ml.g4dn.xlarge')