feat(fetcher): implement SearchBasedFetcher for ScienceDirect using scopus search API #14551

Siva-Sai22 · 2025-12-08T10:34:33Z

Implemented web search capability for ScienceDirect by integrating the Scopus Search API, since the ScienceDirect Search API requires institutional access that most users don't have. Added a new ScopusQueryTransformer to convert JabRef search queries to Scopus query syntax, and wrote the Scopus class to implement PagedSearchBasedParserFetcher interface, enabling paginated search results with proper JSON response parsing, this allows users to search academic literature via the "Web search" feature in JabRef, which was previously not available for ScienceDirect.
Modified the ScienceDirect to use the api key of scopus.

Steps to test

Go to settings -> preferences -> websearch.
Add the Scopus API key and test it.
Then search using the Scopus (Example query: author = Einstein)

Mandatory checks

I own the copyright of the code submitted and I license it under the MIT license
I manually tested my changes in running JabRef (always required)
I added JUnit tests for changes (if applicable)
I added screenshots in the PR description (if change is visible to the user)
I described the change in CHANGELOG.md in a way that is understandable for the average user (if change is visible to the user)
I checked the user documentation: Is the information available and up to date? If not, I created an issue at https://github.com/JabRef/user-documentation/issues or, even better, I submitted a pull request updating file(s) in https://github.com/JabRef/user-documentation/tree/main/en.

github-actions · 2025-12-08T10:34:45Z

Hey @Siva-Sai22! 👋

Thank you for contributing to JabRef!

We have automated checks in place, based on which you will soon get feedback if any of them are failing.

After all automated checks pass, a maintainer will also review your contribution. Once that happens, you can go through their comments in the "Files changed" tab and act on them, or reply to the conversation if you have further inputs.

Please re-check our AI Usage Policy to ensure that your pull request is in line with it. It also contains links to our contribution guide in case of any other doubts related to our contribution workflow.

koppor

According to https://dev.elsevier.com/technical_documentation.html Scopus is not ScienceDirect.

I see two options:

To avoid confusion, ScienceDirect should be renamed to Elsevier. But then, we have a "big ball of mud", because the methods in that class are not coupled, but only semnatically correlated
Add new functionality to Scopus - this should make maintainence easier. This fetcher should be added to the list of fetchers in the "Web Search" prefernces

Please do option 2 to have the fetcher class small.

API key should renamed from "Scopus" to "Esevier", since it covers both.

Future work: Split up API key configuration and fetcher enablement.

koppor · 2025-12-08T19:46:47Z

jablib/src/main/java/org/jabref/logic/importer/fetcher/ScienceDirect.java

+            String response = new BufferedReader(new InputStreamReader(inputStream))
+                    .lines()
+                    .collect(Collectors.joining(OS.NEWLINE));
+
+            JSONObject jsonObject = new JSONObject(response);


There must be a method to convert an inputstream to a JSON object directly. But maybe, not mit dem JSONObject.

** JSONObject response = JsonReader.toJsonObject(inputStream);
**

koppor · 2025-12-08T21:00:21Z

Another view - from

Siva-Sai22 · 2025-12-09T11:20:04Z

API key should renamed from "Scopus" to "Esevier", since it covers both.

@koppor
But the ScienceDirect is accessing the api key using the FETCHER_NAME (ScienceDirect). If we rename it to Esevier we have to change the fetcher name also
importerPreferences.getApiKey(getName())

I think it would be better to take a scopus api key also.

koppor · 2025-12-11T09:28:18Z

API key should renamed from "Scopus" to "Esevier", since it covers both.

@koppor But the ScienceDirect is accessing the api key using the FETCHER_NAME (ScienceDirect). If we rename it to Esevier we have to change the fetcher name also importerPreferences.getApiKey(getName())

I think it would be better to take a scopus api key also.

I searched for the code you gave

It seems that EACH FETCHER uses that thing.

Since the new fetcher is a SearchBasedFetcher displayed in the preferences, the old fetcher should use the key of the other fetcher. With that, both fetchers will share the same key and it can be configured.

Update That means, my idea with "Elsevier" was wrong and a concrete fetcher name is used. -- Please double check with the preferences code.

Siva-Sai22 · 2025-12-11T18:44:59Z

@koppor I have checked the preferences code
The api key is stored in a keyring and retrieved using the same when called using the concrete fetcher name
So I think we can't have the same api key name for the two different fetchers.

I have changes in which we allow the user to input the api key for scopus (same as the scienceDirect one) and can display the search results.

If you are okay with this approach I will update the PR.

github-actions · 2025-12-17T18:15:34Z

Your pull request conflicts with the target branch.

Please merge with your code. For a step-by-step guide to resolve merge conflicts, see https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/addressing-merge-conflicts/resolving-a-merge-conflict-using-the-command-line.

koppor · 2025-12-17T23:39:30Z

@Siva-Sai22 It is difficult for me to follow. Please provide a link (you can search for files on GitHub by typing T). Provide a deep link to the line number. - It is also OK to provide a class name and line number. Answers might then even take longer, since we mostly do JabRef during the day in short breaks without an IDE opened.

koppor · 2025-12-17T23:41:27Z

I have changes in which we allow the user to input the api key for scopus (same as the scienceDirect one) and can display the search results.

No, this is not OK, because it seems that only one fetcher is displayed and not the other one.

To move on, you extract your new SearchBasedFetcher out and revert the original class to their own state. Then we maybe see easier how the key handling works.

Siva-Sai22 · 2025-12-18T07:39:02Z

@koppor

This is how api key is accesed in ScienceDirect (or any other fetcher)

jabref/jablib/src/main/java/org/jabref/logic/importer/fetcher/ScienceDirect.java

Line 140 in 75b867f

.header("X-ELS-APIKey", importerPreferences.getApiKey(getName()).orElse(""))
This is how the api keys are set (using the fetcher names)

jabref/jablib/src/main/java/org/jabref/logic/preferences/JabRefCliPreferences.java

Line 2315 in 75b867f

private Set<FetcherApiKey> getFetcherKeys() {

So the keys are fetched using the fetcher name.
The method you discussed above:

API key should renamed from "Scopus" to "Esevier", since it covers both.

This cannot be done because the api key is fetched using fetcher name.

As suggested by you the scienceDirect should be kept as it is (and not make it a search based fetcher) and instead make a new searchbasedfetcher known as Scopus which uses the Esevier's api for searching.

The scienceDirect and this scopus one will use the same api key. But inorder to use these the user has to put in the same api key twice (one for using scienceDirect which is fulltextFetcher (not modified) and one for the new Scopus which is searchBasedFetcher)

These are the changes : main...Siva-Sai22:jabref:test

koppor · 2025-12-19T12:53:38Z

@Siva-Sai22 I seee that the class ScienceDirect itself calls the api key. One can replace getName() by Elsevier.

Siva-Sai22 · 2025-12-20T10:30:40Z

@koppor For the tests I thought that if they covered the cases correctly then it is enough. So generated it through ai by explaining it the cases and focused more on the actual implementation. Sorry for that, next time onwards will write the tests also myself.

…pdate related tests and docs - Use Scopus API key for ScienceDirect fetcher and remove separate ScienceDirect API key handling - Update environment variable and build configuration to use ScopusApiKey - Adjust tests to reflect unified API key usage - Remove redundant or obsolete test cases in Scopus and query transformer tests - Update documentation and changelog to clarify Scopus/ScienceDirect API key requirements

…pdate related tests and docs - Use Scopus API key for ScienceDirect fetcher and remove separate ScienceDirect API key handling - Update environment variable and build configuration to use ScopusApiKey - Adjust tests to reflect unified API key usage - Remove redundant or obsolete test cases in Scopus and query transformer tests - Update documentation and changelog to clarify Scopus/ScienceDirect API key requirements # Conflicts: # jablib/src/main/java/org/jabref/logic/importer/fetcher/ScienceDirect.java

jablib/src/main/java/org/jabref/logic/importer/fetcher/ScienceDirect.java

jablib/src/main/java/org/jabref/logic/importer/fetcher/Scopus.java

koppor · 2025-12-23T21:19:10Z

jablib/src/main/java/org/jabref/logic/importer/fetcher/Scopus.java

+ * Note: ScienceDirect Search API requires institutional access, so we use Scopus API instead
+ * which is available to all registered developers.


Sure? I think, it is different functionalit!!

Actually I tried using the same api key from elsevier to the scienceDirect api. It always returned authorization error.

{"status":{"statusCode":"AUTHORIZATION_ERROR","statusText":"The requestor is not authorized to access the requested view or fields of the resource"}}}

https://www.elsevier.support/dataasaservice/answer/why-do-i-receive-error-responses-saying-some-api-resources-are-unauthorized-or-that-i-have-insufficient-privileges

…Direct.java Co-authored-by: Oliver Kopp <[email protected]>

Removed unnecessary comments about the Scopus Search API.

koppor

Not sure if you have the energy to have a good java doc and fix other small nitpicks.

It is OK to say: no time - then someone of the team will try to finish.

koppor · 2025-12-27T23:23:02Z

jablib/src/main/java/org/jabref/logic/importer/fetcher/transformers/ScopusQueryTransformer.java

+/**
+ * Query transformer for the Scopus Search API.
+ * <p>
+ * Scopus uses specific field codes for advanced search:
+ * - TITLE-ABS-KEY-AUTH() for searching in title, abstract, author and keywords
+ * - AUTH() for author search
+ * - SRCTITLE() for journal/source title search
+ * - PUBYEAR for publication year
+ * - DOI() for DOI search
+ * <p>
+ * Scopus supports AND, OR, AND NOT as boolean operators.
+ * <p>
+ * See: https://dev.elsevier.com/sc_search_tips.html
+ */


Convert to markdown - or use proper HTML - the item list is not correctly displayed as is.

koppor · 2025-12-27T23:24:01Z

jablib/src/main/java/org/jabref/logic/importer/fetcher/Scopus.java

+    @Nullable
+    private BibEntry parseScopusEntry(JSONObject jsonEntry) {


In JabRef, we use Optionals.

CHANGELOG.md

Co-authored-by: Oliver Kopp <[email protected]>

…afety

Siva-Sai22 · 2025-12-28T14:04:40Z

@koppor updated the pr with required changes. Please let me know if any other changes are required.

github-actions bot added the first contrib label Dec 8, 2025

Siva-Sai22 changed the title ~~Fix/#14421~~ feat(fetcher): implement SearchBasedFetcher for ScienceDirect using scopus search API Dec 8, 2025

github-actions bot added status: changes-required Pull requests that are not yet complete and removed status: changes-required Pull requests that are not yet complete labels Dec 8, 2025

koppor requested changes Dec 8, 2025

View reviewed changes

github-actions bot added the status: changes-required Pull requests that are not yet complete label Dec 8, 2025

Siedlerchr previously approved these changes Dec 8, 2025

View reviewed changes

Siedlerchr added status: ready-for-review Pull Requests that are ready to be reviewed by the maintainers and removed status: changes-required Pull requests that are not yet complete labels Dec 8, 2025

koppor added the status: devcall label Dec 8, 2025

koppor reviewed Dec 8, 2025

View reviewed changes

koppor added status: changes-required Pull requests that are not yet complete and removed status: devcall status: ready-for-review Pull Requests that are ready to be reviewed by the maintainers labels Dec 8, 2025

calixtus added the component: fetcher label Dec 12, 2025

koppor added status: waiting-for-feedback The submitter or other users need to provide more information about the issue and removed status: changes-required Pull requests that are not yet complete labels Dec 18, 2025

github-actions bot added the status: changes-required Pull requests that are not yet complete label Dec 19, 2025

koppor added 📌 Pinned and removed status: waiting-for-feedback The submitter or other users need to provide more information about the issue labels Dec 19, 2025

Siva-Sai22 and others added 6 commits December 20, 2025 16:02

test(scopus): remove unused BibEntry field from test class

9c85e5f

Merge branch 'main' into fix/JabRef#14421

ebd189e

docs(fetcher): fix alignment in API key table for Elsevier entry

44eb1be

test(scopus): remove unused BibEntry initialization in setup

5cf3bd7

github-actions bot added status: changes-required Pull requests that are not yet complete and removed status: changes-required Pull requests that are not yet complete labels Dec 20, 2025

koppor requested changes Dec 23, 2025

View reviewed changes

koppor mentioned this pull request Dec 24, 2025

Obtain fulltext from sciencedirect #12161

Open

2 tasks

Siva-Sai22 and others added 7 commits December 26, 2025 17:32

Update jablib/src/main/java/org/jabref/logic/importer/fetcher/Science…

20e308f

…Direct.java Co-authored-by: Oliver Kopp <[email protected]>

Clean up comments in Scopus.java

fb32e83

Removed unnecessary comments about the Scopus Search API.

Merge branch 'main' into fix/JabRef#14421

981f55b

Fix markdown error

54d5354

Fix Jbang errors

ff97a58

Merge branch 'JabRef:main' into fix/JabRef#14421

c9b30c8

Restore submodule to upstream version

c595af7

github-actions bot removed the status: changes-required Pull requests that are not yet complete label Dec 26, 2025

koppor requested changes Dec 27, 2025

View reviewed changes

github-actions bot added the status: changes-required Pull requests that are not yet complete label Dec 27, 2025

Siva-Sai22 and others added 3 commits December 28, 2025 19:25

Update CHANGELOG.md

0cd0e98

Co-authored-by: Oliver Kopp <[email protected]>

Merge branch 'JabRef:main' into fix/JabRef#14421

d1be03b

refactor(scopus): use Optional for parseScopusEntry to improve null s…

22fe601

…afety

github-actions bot removed the status: changes-required Pull requests that are not yet complete label Dec 28, 2025

github-actions bot added the status: changes-required Pull requests that are not yet complete label Dec 29, 2025

Merge branch 'main' into fix/JabRef#14421

73411ac

github-actions bot added status: changes-required Pull requests that are not yet complete and removed status: changes-required Pull requests that are not yet complete labels Dec 31, 2025

		* Note: ScienceDirect Search API requires institutional access, so we use Scopus API instead
		* which is available to all registered developers.

		@Nullable
		private BibEntry parseScopusEntry(JSONObject jsonEntry) {

Uh oh!

feat(fetcher): implement SearchBasedFetcher for ScienceDirect using scopus search API #14551

Are you sure you want to change the base?

feat(fetcher): implement SearchBasedFetcher for ScienceDirect using scopus search API #14551

Uh oh!

Conversation

Siva-Sai22 commented Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps to test

Mandatory checks

Uh oh!

github-actions bot commented Dec 8, 2025

Uh oh!

koppor left a comment

Choose a reason for hiding this comment

Uh oh!

koppor Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

Siedlerchr Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

koppor commented Dec 8, 2025

Uh oh!

Siva-Sai22 commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

koppor commented Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Siva-Sai22 commented Dec 11, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

koppor commented Dec 17, 2025

Uh oh!

koppor commented Dec 17, 2025

Uh oh!

Siva-Sai22 commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

koppor commented Dec 19, 2025

Uh oh!

Siva-Sai22 commented Dec 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

koppor Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

Siva-Sai22 Dec 26, 2025

Choose a reason for hiding this comment

Uh oh!

koppor left a comment

Choose a reason for hiding this comment

Uh oh!

koppor Dec 27, 2025

Choose a reason for hiding this comment

Uh oh!

koppor Dec 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Siva-Sai22 commented Dec 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Siva-Sai22 commented Dec 8, 2025 •

edited

Loading

Siva-Sai22 commented Dec 9, 2025 •

edited

Loading

koppor commented Dec 11, 2025 •

edited

Loading

Siva-Sai22 commented Dec 18, 2025 •

edited

Loading

Siva-Sai22 commented Dec 20, 2025 •

edited

Loading