Remove ExtraMetadata in favor of SparseFields by Copilot · Pull Request #14001 · GeoNode/geonode

Copilot · 2026-02-27T10:16:58Z

ExtraMetadata was a stopgap for storing custom per-resource metadata. With the metadata JSON schema system and SparseField now available, ExtraMetadata is superseded and can be removed.

Migration

A single migration (0098) handles both data and schema changes:

Data: Each ExtraMetadata row is migrated to a SparseField entry with name=extra_metadata_<pk> and value=json.dumps(metadata_dict). Entries whose serialized JSON exceeds the 1024-char SparseField.value limit are skipped with a warning.
Schema: Removes the metadata M2M field from ResourceBase and drops the ExtraMetadata model.

Code removal

base/models.py: Removed ExtraMetadata model and ResourceBase.metadata M2M field.
base/api/serializers.py: Removed ExtraMetadataSerializer and metadata deferred field from ResourceBaseSerializer.
base/api/views.py: Removed extra_metadata REST action (GET/PUT/POST/DELETE on /{pk}/extra_metadata/).
base/utils.py: Removed validate_extra_metadata() and its schema/settings imports.
resource/utils.py / resource/manager.py: Removed extra_metadata parameter from update_resource() and ResourceManager.update().
api/resourcebase_api.py: Removed metadata__* query param filtering from the legacy Tastypie API.
settings.py: Removed DEFAULT_EXTRA_METADATA_SCHEMA, CUSTOM_METADATA_SCHEMA, EXTRA_METADATA_SCHEMA, and from schema import Optional.

Tests

Removed TestExtraMetadataBaseApi and all test_extra_metadata_included_with_param / test_metadata_filters tests that covered the now-deleted functionality.

Note for external API consumers: The /extra_metadata/ endpoint and the metadata field in resource API responses are gone. Use the sparse fields API instead.

Original prompt

This section details on the original issue you should resolve

<issue_title>Refact extraMetadata</issue_title>
<issue_description>The extraMetadata field was introduced in order to store custom metadata without modifying the model.
After the introduction of the new metadata json schema and the possibility to access the sparse metadata fields, the extraMetadata field becomes outdated.

External clients (if any) using the API for handling the extraMetadata should be updated to use the new sparse field API.

So, at least, we need:

migration to move extraMetadata content into sparse fields

extraMetadatata field removal + related migration

IF we want to maintain back compatibility, we need to make sure that read and write accesses for the extraMetadata still work:

refact the views that handle extraMetadata field as input, and store the inner properties in SparseFields

refact the serializers outputting the extraMetadata, making them read the sparsefields instead.

@giohappy These last two points needs discussion to find out if they are really needed.</issue_description>

Comments on the Issue (you are @copilot in this section)

Fixes Refact extraMetadata #14000

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

cla-bot · 2026-02-27T10:17:01Z

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the users @Copilot on file. In order for us to review and merge your code, please contact the project maintainers to get yourself added.

…ata model Co-authored-by: etj <717359+etj@users.noreply.github.com>

cla-bot · 2026-02-27T10:32:18Z

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the users @Copilot on file. In order for us to review and merge your code, please contact the project maintainers to get yourself added.

etj · 2026-03-19T18:06:15Z

@copilot format with back, checks are failing because of it

Co-authored-by: etj <717359+etj@users.noreply.github.com>

cla-bot · 2026-03-19T18:09:08Z

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the users @Copilot on file. In order for us to review and merge your code, please contact the project maintainers to get yourself added.

Copilot · 2026-03-19T18:09:11Z

@copilot format with back, checks are failing because of it

Fixed in cfc914c. The flake8 errors were:

QuerySet imported but unused in serializers.py (was only used by the removed ExtraMetadataSerializer)
IsOwnerOrAdmin imported but unused in views.py (was only used by the removed extra_metadata action)
Trailing blank line at end of models.py
urljoin imported but unused in geoapps/api/tests.py (was only used by the removed test method)

etj · 2026-04-13T07:55:16Z

@copilot code is faling with "django.core.exceptions.ImproperlyConfigured: Field name metadata is not valid for model Dataset in geonode.layers.api.serializers.DatasetSerializer." Fix it.

Agent-Logs-Url: https://github.com/GeoNode/geonode/sessions/d5187544-0996-458a-88b7-66628254cbf4 Co-authored-by: etj <717359+etj@users.noreply.github.com>

cla-bot · 2026-04-13T07:56:49Z

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the users @Copilot on file. In order for us to review and merge your code, please contact the project maintainers to get yourself added.

Copilot · 2026-04-13T07:56:52Z

@copilot code is faling with "django.core.exceptions.ImproperlyConfigured: Field name metadata is not valid for model Dataset in geonode.layers.api.serializers.DatasetSerializer." Fix it.

Fixed in c1a9e81. DatasetSerializer.Meta.fields still listed "metadata" explicitly — removed it since the field no longer exists on the model.

cla-bot · 2026-04-13T08:03:46Z

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the users @Copilot on file. In order for us to review and merge your code, please contact the project maintainers to get yourself added.

etj · 2026-04-13T08:55:45Z

@copilot create another pull request which implements the back compatibility adapters described in the original issue. The API adapters are temporary (so they should be marked as deprecated), and they shall be pretty well isolated so that it will be easy to remove them after the deprecation period.

Copilot · 2026-04-13T08:55:56Z

@etj I've opened a new pull request, #14133, to work on those changes. Once the pull request is ready, I'll request review from you.

etj · 2026-05-12T16:53:38Z

@copilot black check is failing

Run black --check geonode
would reformat /home/runner/work/geonode/geonode/geonode/base/api/deprecated_extra_metadata.py
would reformat /home/runner/work/geonode/geonode/geonode/base/api/views.py
would reformat /home/runner/work/geonode/geonode/geonode/base/api/tests.py

Oh no! 💥 💔 💥
3 files would be reformatted, 630 files would be left unchanged.
Error: Process completed with exit code 1.

…ts.py Agent-Logs-Url: https://github.com/GeoNode/geonode/sessions/8a0d75b5-2c96-4456-9d56-6bef8adfdd7f Co-authored-by: etj <717359+etj@users.noreply.github.com>

cla-bot · 2026-05-12T16:54:52Z

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the users @Copilot on file. In order for us to review and merge your code, please contact the project maintainers to get yourself added.

Copilot · 2026-05-12T16:54:55Z

@copilot black check is failing
Run black --check geonode

...

Fixed in 3183588. Applied black formatting to all three reported files.

Copilot

Pull request overview

Removes the legacy ExtraMetadata model/M2M field and related API surface, migrating stored data into metadata.SparseField and cleaning up associated update paths and tests. The PR also introduces a deprecated compatibility layer that re-exposes the legacy /extra_metadata/ endpoint and metadata serializer field using SparseField.

Changes:

Add a data+schema migration to move ExtraMetadata rows into SparseField and drop the ExtraMetadata model / ResourceBase.metadata field.
Remove legacy ExtraMetadata usage across resource update codepaths, serializers/views, and Tastypie filtering.
Replace removed behavior with a deprecated adapter (DeprecatedExtraMetadataMixin / DeprecatedExtraMetadataField) plus new compatibility tests.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
geonode/settings.py	Reworks (and partially retains) legacy ExtraMetadata-related settings for backward compatibility.
geonode/resource/utils.py	Removes `ExtraMetadata` import and `extra_metadata` handling from `update_resource()`.
geonode/resource/manager.py	Removes `extra_metadata` parameter plumbing into `update_resource()`.
geonode/maps/api/tests.py	Removes tests asserting legacy `include[]=metadata` behavior.
geonode/layers/api/tests.py	Removes tests asserting legacy `include[]=metadata` behavior.
geonode/layers/api/serializers.py	Drops `metadata` from dataset serializer fields.
geonode/geoapps/api/tests.py	Removes tests asserting legacy `include[]=metadata` behavior.
geonode/documents/api/tests.py	Removes tests asserting legacy `include[]=metadata` behavior.
geonode/base/utils.py	Removes `validate_extra_metadata()` and related imports.
geonode/base/models.py	Removes `ResourceBase.metadata` M2M and deletes `ExtraMetadata` model.
geonode/base/migrations/0098_migrate_extrametadata_to_sparsefields.py	Adds migration to move ExtraMetadata data into SparseFields and then drop the old schema.
geonode/base/api/views.py	Removes legacy `extra_metadata` action and mixes in the deprecated compatibility adapter.
geonode/base/api/tests.py	Removes old ExtraMetadata API tests and adds tests for the deprecated compatibility adapter.
geonode/base/api/serializers.py	Replaces old `ExtraMetadataSerializer` field with a deprecated computed field backed by SparseField.
geonode/base/api/deprecated_extra_metadata.py	Implements deprecated compatibility: legacy `/extra_metadata/` action + `metadata` field backed by SparseField.
geonode/api/tests.py	Removes Tastypie tests for metadata filtering.
geonode/api/resourcebase_api.py	Removes Tastypie `metadata__*` filter plumbing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@@ -2196,6 +2188,7 @@ def get_geonode_catalogue_service():
    },
    **CUSTOM_METADATA_SCHEMA,
 }



+"""
+Kept for backward compatibility with geonode_mapstore_client and other packages
+that may still reference this setting. ExtraMetadata has been removed in favour
+of SparseFields; this empty dict prevents AttributeError on startup.
+"""
+EXTRA_METADATA_SCHEMA = {}
+


+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("base", "0097_alter_link_asset"),


+        warnings.warn(DEPRECATION_REASON, DeprecationWarning, stacklevel=2)
+        logger.warning(DEPRECATION_REASON)


+            "[DEPRECATED] Get/Update/Delete/Add extra metadata for a resource. " "Use the sparse fields API instead."
+        ),
+        deprecated=True,
+    )
+    @action(
+        detail=True,
+        methods=["get", "put", "delete", "post"],


etj · 2026-05-12T17:47:09Z

@copilot on master drf-spectacular has been removed. Pls remove related dependencies on the code as well, since tests are failing.

Agent-Logs-Url: https://github.com/GeoNode/geonode/sessions/5fc50f73-190e-46f5-b7fd-2d885ed21c7c Co-authored-by: etj <717359+etj@users.noreply.github.com>

cla-bot · 2026-05-12T17:48:40Z

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the users @Copilot on file. In order for us to review and merge your code, please contact the project maintainers to get yourself added.

Copilot · 2026-05-12T17:48:43Z

@copilot on master drf-spectacular has been removed. Pls remove related dependencies on the code as well, since tests are failing.

Fixed in 2f245ad. Removed the from drf_spectacular.utils import extend_schema import and the @extend_schema decorator from deprecated_extra_metadata.py — that was the only place drf_spectacular was referenced in our changes.

etj · 2026-05-13T12:16:23Z

@copilot fix

formatting with black
migrations:

  CommandError: Conflicting migrations detected; multiple leaf nodes in the migration graph: (0098_migrate_extrametadata_to_sparsefields, 0099_resourcebase_auth_config in base).

…cebase_auth_config Agent-Logs-Url: https://github.com/GeoNode/geonode/sessions/cb857a4d-d999-4228-ad98-fe4648e898aa Co-authored-by: etj <717359+etj@users.noreply.github.com>

cla-bot · 2026-05-13T12:17:53Z

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the users @Copilot on file. In order for us to review and merge your code, please contact the project maintainers to get yourself added.

Copilot · 2026-05-13T12:17:56Z

@copilot fix

formatting with black

migrations:
...

Fixed in fecb49e. Renamed 0098_migrate_extrametadata_to_sparsefields.py → 0100_migrate_extrametadata_to_sparsefields.py and updated its dependency from 0097_alter_link_asset to 0099_resourcebase_auth_config, resolving the leaf-node conflict. Black formatting was already clean after the previous commit.

etj · 2026-05-13T13:44:21Z

@copilot error in black formatting

would reformat /home/runner/work/geonode/geonode/geonode/base/api/tests.py

Oh no! 💥 💔 💥
1 file would be reformatted, 632 files would be left unchanged.
Error: Process completed with exit code 1.

Agent-Logs-Url: https://github.com/GeoNode/geonode/sessions/a710ea51-7e89-45d3-a6d4-74229bd12a5c Co-authored-by: etj <717359+etj@users.noreply.github.com>

cla-bot · 2026-05-13T13:46:02Z

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the users @Copilot on file. In order for us to review and merge your code, please contact the project maintainers to get yourself added.

Copilot · 2026-05-13T13:46:07Z

@copilot error in black formatting

would reformat /home/runner/work/geonode/geonode/geonode/base/api/tests.py

...

Fixed in 3c15e10. The locally installed black was a newer version (26.x) that agreed with the file, but CI uses black==24.4.2 (per pyproject.toml). Reformatted with the correct version.

codecov · 2026-05-13T14:26:55Z

Codecov Report

❌ Patch coverage is 71.48594% with 71 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.63%. Comparing base (a9ac0a2) to head (3c15e10).
⚠️ Report is 17 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #14001       +/-   ##
===========================================
+ Coverage   43.23%   74.63%   +31.40%     
===========================================
  Files         960      969        +9     
  Lines       58824    59229      +405     
  Branches     8018     8066       +48     
===========================================
+ Hits        25432    44208    +18776     
+ Misses      32113    13204    -18909     
- Partials     1279     1817      +538

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 5 comments.

@@ -2196,6 +2188,7 @@ def get_geonode_catalogue_service():
    },
    **CUSTOM_METADATA_SCHEMA,
 }


@@ -2196,6 +2188,7 @@ def get_geonode_catalogue_service():
    },
    **CUSTOM_METADATA_SCHEMA,
 }
+# ---- End deprecated ExtraMetadata settings ----

 """
 List of modules that implement custom metadata storers that will be called when the metadata of a resource is saved
@@ -2204,6 +2197,13 @@ def get_geonode_catalogue_service():
    # 'geonode.resource.regions_storer.spatial_predicate_region_assignor',
 ]

+"""
+Kept for backward compatibility with geonode_mapstore_client and other packages
+that may still reference this setting. ExtraMetadata has been removed in favour
+of SparseFields; this empty dict prevents AttributeError on startup.
+"""
+EXTRA_METADATA_SCHEMA = {}
+


    links = DynamicRelationField(LinksSerializer, source="id", read_only=True)

    # Deferred fields
-    metadata = ComplexDynamicRelationField(ExtraMetadataSerializer, many=True, deferred=True)
+    # Deprecated: use the sparse fields API instead of ``metadata``.
+    metadata = DeprecatedExtraMetadataField(deferred=True, read_only=True)
    data = DataBlobField(DataBlobSerializer, source="id", deferred=True, required=False)


+    @action(
+        detail=True,
+        methods=["get", "put", "delete", "post"],
+        url_path=r"extra_metadata",
+        url_name="extra-metadata",
+    )
+    def extra_metadata(self, request, pk, *args, **kwargs):
+        """Deprecated endpoint – delegates to SparseField storage."""
+        warnings.warn(DEPRECATION_REASON, DeprecationWarning, stacklevel=2)
+        logger.warning(DEPRECATION_REASON)
+
+        resource = ResourceBase.objects.filter(pk=pk).first()
+        if resource is None:
+            return Response({"detail": "Not found."}, status=404)
+
+        if request.method == "GET":
+            return self._extra_metadata_get(request, resource)
+        elif request.method == "POST":
+            return self._extra_metadata_post(request, resource)
+        elif request.method == "PUT":
+            return self._extra_metadata_put(request, resource)
+        elif request.method == "DELETE":
+            return self._extra_metadata_delete(request, resource)
+


+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("base", "0099_resourcebase_auth_config"),
+        ("metadata", "0001_initial"),
+    ]
+
+    operations = [
+        migrations.RunPython(migrate_extrametadata_to_sparsefields, migrations.RunPython.noop),
+        migrations.RemoveField(


Initial plan

1355a3f

Copilot AI assigned Copilot and etj Feb 27, 2026

Copilot started work on behalf of etj February 27, 2026 10:17 View session

Refactor ExtraMetadata: migrate to SparseFields and remove ExtraMetad…

1866162

…ata model Co-authored-by: etj <717359+etj@users.noreply.github.com>

giohappy marked this pull request as ready for review February 27, 2026 10:32

giohappy requested a review from etj February 27, 2026 10:32

Copilot AI changed the title ~~[WIP] Refactor extraMetadata field and migrate to sparse fields~~ Remove ExtraMetadata in favor of SparseFields Feb 27, 2026

Copilot finished work on behalf of etj February 27, 2026 10:34

Copilot started work on behalf of etj March 19, 2026 18:06 View session

Fix flake8 errors: remove unused imports and trailing blank line

cfc914c

Co-authored-by: etj <717359+etj@users.noreply.github.com>

Copilot finished work on behalf of etj March 19, 2026 18:10

Copilot started work on behalf of etj April 13, 2026 07:55 View session

Remove 'metadata' field from DatasetSerializer.Meta.fields

c1a9e81

Agent-Logs-Url: https://github.com/GeoNode/geonode/sessions/d5187544-0996-458a-88b7-66628254cbf4 Co-authored-by: etj <717359+etj@users.noreply.github.com>

Copilot finished work on behalf of etj April 13, 2026 07:57

Merge branch 'master' into copilot/refactor-extra-metadata-field

3eadbc8

Copilot AI mentioned this pull request Apr 13, 2026

Add deprecated backward-compatible ExtraMetadata API adapters #14133

Merged

giohappy marked this pull request as draft April 13, 2026 08:56

etj mentioned this pull request Apr 13, 2026

Replace extraMetadata with sparse fields GeoNode/geonode-mapstore-client#2490

Open

Copilot started reviewing on behalf of etj May 12, 2026 16:51 View session

Copilot started work on behalf of etj May 12, 2026 16:53 View session

Apply black formatting to deprecated_extra_metadata.py, views.py, tes…

3183588

…ts.py Agent-Logs-Url: https://github.com/GeoNode/geonode/sessions/8a0d75b5-2c96-4456-9d56-6bef8adfdd7f Co-authored-by: etj <717359+etj@users.noreply.github.com>

Copilot finished work on behalf of etj May 12, 2026 16:55

Copilot AI reviewed May 12, 2026

View reviewed changes

Copilot started work on behalf of etj May 12, 2026 17:47 View session

Remove drf_spectacular import from deprecated_extra_metadata.py

2f245ad

Agent-Logs-Url: https://github.com/GeoNode/geonode/sessions/5fc50f73-190e-46f5-b7fd-2d885ed21c7c Co-authored-by: etj <717359+etj@users.noreply.github.com>

Copilot finished work on behalf of etj May 12, 2026 17:49

Copilot started work on behalf of etj May 13, 2026 12:16 View session

Renumber migration to 0100 to fix leaf node conflict with 0099_resour…

fecb49e

…cebase_auth_config Agent-Logs-Url: https://github.com/GeoNode/geonode/sessions/cb857a4d-d999-4228-ad98-fe4648e898aa Co-authored-by: etj <717359+etj@users.noreply.github.com>

Copilot finished work on behalf of etj May 13, 2026 12:18

Copilot started work on behalf of etj May 13, 2026 13:44 View session

Apply black==24.4.2 formatting to base/api/tests.py

3c15e10

Agent-Logs-Url: https://github.com/GeoNode/geonode/sessions/a710ea51-7e89-45d3-a6d4-74229bd12a5c Co-authored-by: etj <717359+etj@users.noreply.github.com>

Copilot finished work on behalf of etj May 13, 2026 13:47

etj requested a review from Copilot May 13, 2026 14:53

Copilot started reviewing on behalf of etj May 13, 2026 14:53 View session

Copilot AI reviewed May 13, 2026

View reviewed changes

		warnings.warn(DEPRECATION_REASON, DeprecationWarning, stacklevel=2)
		logger.warning(DEPRECATION_REASON)

Conversation

Copilot AI commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Migration

Code removal

Tests

Comments on the Issue (you are @copilot in this section)

Uh oh!

cla-bot Bot commented Feb 27, 2026

Uh oh!

cla-bot Bot commented Feb 27, 2026

Uh oh!

etj commented Mar 19, 2026

Uh oh!

cla-bot Bot commented Mar 19, 2026

Uh oh!

Copilot AI commented Mar 19, 2026

Uh oh!

etj commented Apr 13, 2026

Uh oh!

cla-bot Bot commented Apr 13, 2026

Uh oh!

Copilot AI commented Apr 13, 2026

Uh oh!

cla-bot Bot commented Apr 13, 2026

Uh oh!

etj commented Apr 13, 2026

Uh oh!

Copilot AI commented Apr 13, 2026

Uh oh!

etj commented May 12, 2026

Uh oh!

cla-bot Bot commented May 12, 2026

Uh oh!

Copilot AI commented May 12, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

etj commented May 12, 2026

Uh oh!

cla-bot Bot commented May 12, 2026

Uh oh!

Copilot AI commented May 12, 2026

Uh oh!

etj commented May 13, 2026

Uh oh!

cla-bot Bot commented May 13, 2026

Uh oh!

Copilot AI commented May 13, 2026

Uh oh!

etj commented May 13, 2026

Uh oh!

cla-bot Bot commented May 13, 2026

Uh oh!

Copilot AI commented May 13, 2026

Uh oh!

codecov Bot commented May 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Feb 27, 2026 •

edited

Loading

codecov Bot commented May 13, 2026 •

edited

Loading