[terraform/aws] Create gp3 storage class, add release notes

[the-glu]

#1606 changed the default storage class for AWS clusters, but the gp3 storage class is not available by default in EKS clusters.

This PR adds a mechanism to terraform to create it. It can be opted out of, should someone with an existing cluster have created it by other means, e.g. manually.

Release notes have also been updated with information about the potential migration step needed.

This has been tested with a fresh cluster and a PVC that has been successfully provisioned.

[mickmis]

This has been tested with a fresh cluster and a PVC that has been successfully provisioned.

Have you tested the behavior on an existing cluster?

[the-glu]

Have you tested the behavior on an existing cluster?

Not explicitly, but as terraform maintain the state, I think it should be fine even if applied later. Especially as cluster from users that already have a configuration / copied example configuration will stay on gp2.

I tried to delete / recreate that part on my test cluster and it worked.

[mickmis]

Have you tested the behavior on an existing cluster?

Not explicitly, but as terraform maintain the state, I think it should be fine even if applied later. Especially as cluster from users that already have a configuration / copied example configuration will stay on gp2.

I tried to delete / recreate that part on my test cluster and it worked.

@barroco does that sound OK to you? I'm not confident enough on those tech to be sure or not. However for sure we should avoid deployment issues when users are upgrading.

[barroco]

Have you tested the behavior on an existing cluster?

Not explicitly, but as terraform maintain the state, I think it should be fine even if applied later. Especially as cluster from users that already have a configuration / copied example configuration will stay on gp2.
I tried to delete / recreate that part on my test cluster and it worked.

@barroco does that sound OK to you? I'm not confident enough on those tech to be sure or not. However for sure we should avoid deployment issues when users are upgrading.

Since the change is only creating resources, it would only fail if resources with same names were added manually previously. So I agree with @the-glu. If a user had created those resources, they can use the flag as documented in the release note.

Though, migrating from gp2 to gp3 is out of scope of this change and is left to users to organize. Guidance may be provided but I would expect an issue to be opened specifically to request it.

[barroco]

Can you please confirm you tested it again ?

[the-glu]

Can you please confirm you tested it again ?

Yes, tested with a new cluster and default config and works as expected: