pref: fix N+1 issues for dref3 endpoint

[thenav56]

Addresses

• N+1 queries in dref3 endpoint

Changes

• Add prefetch for M2M and FK fields
• Merge N event retrieves to single
• Replace dynamic Ephemeral numeric id with <table-name>-<table-id>

Before

After

[thenav56]

Hey @szabozoltan69, @TKartist,

Instead of adding a separate table, I noticed there were still some N+1 issues in the query. After fixing them, the query is now about 4.5× faster.

Will this be enough for the current performance improvements?

[thenav56]

I tested using the normal endpoint /api/v2/dref3/ only and the response where same as before.

Could you also check if the other iterations still work as before?

I used this steps to compare responses

For easy data compare, we need to update the existing code to include this in BaseDref3Serializer

• Replace existing id attribute with

  id = serializers.SerializerMethodField()

• Add new function to generate id

  def get_id(self, obj):
    return f"{type(obj).__name__}-{obj.id}"

Generate dataset from develop

curl http://localhost:8000/api/v2/dref3/ > /tmp/before.json
# other iterations

Generate dataset from pref/dref3

curl http://localhost:8000/api/v2/dref3/ > /tmp/after.json
# other iterations

Compare using dyff

#!/bin/bash

PAIRS=(
  "/tmp/before.json:/tmp/after.json"
  "/tmp/before-iteration1.json:/tmp/after-iteration1.json"
  "/tmp/before-iteration2.json:/tmp/after-iteration2.json"
)

SORTED_DIR=$(mktemp -d -t dref3.XXXXXX)
echo "Creating temp directory to store sorted files: $SORTED_DIR"

mkdir -p "$SORTED_DIR"

for pair in "${PAIRS[@]}"; do
  IFS=":" read -r BEFORE AFTER <<< "$pair"

  echo "Comparing $BEFORE <-> $AFTER"
  BEFORE_SORTED="$SORTED_DIR/$(basename "$BEFORE" .json).json"
  AFTER_SORTED="$SORTED_DIR/$(basename "$AFTER" .json).json"

  jq 'sort_by(.id)' "$BEFORE" > "$BEFORE_SORTED"
  jq 'sort_by(.id)' "$AFTER" > "$AFTER_SORTED"

  echo "=== Comparing $BEFORE vs $AFTER ==="
  dyff between "$BEFORE_SORTED" "$AFTER_SORTED"
  echo
done

[szabozoltan69]

Will this be enough for the current performance improvements?

I suppose yes!

[szabozoltan69]

I tested using the normal endpoint /api/v2/dref3/ only and the response where same as before.

I also tested, via the CSV output, and (not mentioning the ID, which is brilliant, permanent) the results are the same on my local env also.
So I am glad to use this!!!

[szabozoltan69]

Before merging maybe we could sync with @ypyelab and @TKartist – is it a good idea to fully drop the "simply (ephemeral) numeric id", or could we use it also, putting the "permanent id" into another column? Because ordering by the recent ID does not lead to a readable format in CSV.

This new "permanent id" is like this:
id
Dref-6
DrefOperationalUpdate-4
Dref-10
DrefOperationalUpdate-22
DrefFinalReport-7
Dref-11
DrefOperationalUpdate-23
DrefFinalReport-11
Dref-12
DrefFinalReport-20

[TKartist]

It looks good.

I believe it would be a good idea to replace the current temp id we have now with the new permanent id considering there are people using the dataset and introducing a new field may cause confusion.

Thank you Navin.