feat: add utility function (and/or data) for URL datasets if necessary

[tianjianjiang]

A Light Discussion about Dataset Choices for URL (at least)

Besides a small subset of (m)C4, I prefer finding intersections among metadata (URL at least), promptsource, and evaluation WGs.

• TyDi QA (primary task) is probably the only common dataset

For either one of two WGs excluding us metadata here,

• From evaluation
  • https://www.kaggle.com/rmisra/news-category-dataset
  • GEM from eval WG, specifically
    • MLSum
    • WikiLingua
• From promptsource
  • app_reviews: although not really URL/URI but basically namespace and date
  • CC-News: virtually a subset of C4
  • Probably some more

[tianjianjiang]

ping @timoschick @SaulLu @cccntu for the light discussion.

[timoschick]

Hi @tianjianjiang , sorry for the late reply. If I understood you correctly, the above is a list of datasets that you think might be suitable for evaluating a model trained with URLs?

I haven't worked with TyDi QA before, but does it come with URLs in any form that we might use? https://www.kaggle.com/rmisra/news-category-dataset seems like an interesting dataset for URLs (although it seems like a very easy task at first sight). It might also make sense to look into the CTRL paper once again (although if I remember correctly, they don't really have any quantiative evaluation). Maybe we can discuss all datasets with the entire group in our next group meeting?

[tianjianjiang]

Hi @timoschick,
My apologies too for not being responsive for almost 2 weeks.

the above is a list of datasets that you think might be suitable for evaluating a model trained with URLs?

Yes~

I haven't worked with TyDi QA before, but does it come with URLs in any form that we might use?

Only the primary task actually but probably good enough.

https://www.kaggle.com/rmisra/news-category-dataset seems like an interesting dataset for URLs (although it seems like a very easy task at first sight).

I think so, too.
One of my plans is to try to submit it to HuggingFace datasets.

It might also make sense to look into the CTRL paper once again (although if I remember correctly, they don't really have any quantiative evaluation). Maybe we can discuss all datasets with the entire group in our next group meeting?

You're absolutely right, CTRL didn't provide any basic stats about URL-related performance.
I would love to have a quick discussion.

In the meantime, I'm trying to catch up with this PR draft for really running experiments with Colab: #24