Skip to content

Conversation

@singhpk234
Copy link
Contributor

@singhpk234 singhpk234 commented Dec 30, 2025

About the change

This change attempts to avoid object store call to recreate the TableMetadata from the object store in case we are making a serialized table copy of the table, this could be seen as an optimization where rather than going to the obj store we keep the serialized copy of the table metadata and then just store it and send it via wire for example (spark driver sending to executor) or can also be looked as a feature where if the catalog has redacted some sensitive info, all the serializable table respects it and reads that rather than going to the object store (which it may not have access to) for example:

Presently its gauraded only for Remotely planned tables

I understand that spec wording the following:

Per REST spec the metadata location is nullable in certain cases

metadata-location:
type: string
description: May be null if the table is staged as part of a transaction
nullable: true

Addditionally we can send metadata location as null to force this scenario :

The spec says the following but that is SHOULD and not MUST :

The table metadata JSON is returned in the `metadata` field. The corresponding file location of table metadata should be returned in the `metadata-location` field, unless the metadata is not yet committed. For example, a create transaction may return metadata that is staged but not committed.

just wanted to float this to folks on what they think of this feature.

@github-actions github-actions bot added the core label Dec 30, 2025
@sfc-gh-prsingh sfc-gh-prsingh force-pushed the rest-catalog-serializable-table-optimization branch from 8baa35a to 87130b3 Compare December 30, 2025 01:36
@singhpk234 singhpk234 marked this pull request as ready for review January 1, 2026 01:54
@github-actions github-actions bot added the spark label Jan 5, 2026
@sfc-gh-prsingh sfc-gh-prsingh force-pushed the rest-catalog-serializable-table-optimization branch from c712137 to ac3e173 Compare January 5, 2026 04:43
@sfc-gh-prsingh sfc-gh-prsingh force-pushed the rest-catalog-serializable-table-optimization branch from ac3e173 to 79b5352 Compare January 5, 2026 04:51
@sfc-gh-prsingh sfc-gh-prsingh force-pushed the rest-catalog-serializable-table-optimization branch from 79b5352 to 676dd48 Compare January 5, 2026 04:52
}

public static class MetadataLogEntry {
public static class MetadataLogEntry implements Serializable {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should fix this irrespective, IMHO

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants