Ett 1788 solr fields by liseli · Pull Request #248 · hathitrust/babel

liseli · 2026-08-21T17:05:38Z

This PR is about the Jira ticket

The main goal was to split Solr 6/Solr 9 config files because Solr 9's schema removed the fields ctrlnum, fulgeographic, fullgenre, and hlb3, documentation here

The solution was to add ls/lib/Config/solr9/ as a copy of ls/lib/Config/ and remove the fields from there. The folder ls/lib/Config/ is untouched and is the default configuration to start Babel accessing to Solr 6.

A new ls_config_dir config key has been defined. By default, LSS_CONFIG_DIR=ls/lib/Config and lets ls/cgi/ls and LS::FacetConfig::get_rel_weights_from_yaml pick the right directory instead of hardcoding ls/lib/Config. To connect to Solr9, the Compose profile LSS_CONFIG_DIR was defined in apache.yml/perl.yml.

I've updated the README.md file with the commands to run against Solr 9, adding LSS_CONFIG_DIR.

I've tested these changes on both Solr 6 and Solr 9, and I get results. I'll listed the commands below, so you can reproduce them locally.

Testing Babel + Solr 6

docker compose --profile frontend build
docker compose --profile backend build

docker compose --profile frontend up
docker compose --profile backend up

Testing Babel + Solr 9

docker compose --profile frontend build
LSS_SOLR_HOST=solr-lss-dev-solr9 LSS_SOLR_CORE_B=core-x LSS_CONFIG_DIR=ls/lib/Config/solr9 docker compose --profile backend --profile solr9 build

docker compose --profile frontend up
LSS_SOLR_HOST=solr-lss-dev-solr9 LSS_SOLR_CORE_B=core-x LSS_CONFIG_DIR=ls/lib/Config/solr9 docker compose --profile backend --profile solr9 up

In Solr 9, you should index the sample data on the server. The process for indexing data is explained here. I've tested it, and it works, but you will need to clone the lss_solr_configs repository to your machine and follow the steps to run the script indexing_data.sh.

If in the service solr-lss-dev-solr9 (docker logs solr-lss-dev-solr9) you see the error Could not create a new core in /var/solr/data/core-x_shard1_replica_n1 as another core is already defined there. This happens because the persist volumes babel_solr1_data and babel_zoo1_data exist from a previous run, and the create_collection call collided with the leftover directory. There is a Jira ticket to fix this problem

Temporary solution: --> I'll create a task to fix this issue.

docker compose --profile solr9 rm -f solr-lss-dev-solr-solr9
docker volume rm babel_solr1_data
LSS_SOLR_HOST=solr-lss-dev-solr9 LSS_SOLR_CORE_B=core-x LSS_CONFIG_DIR=ls/lib/Config/solr9 docker compose --profile backend --profile solr9 up

Following up on this ticket, it's a good time to start thinking about testing. Let's consider how we can verify that the existing LS tests cover both Solr 6 and Solr 9, and whether we might need to add any extra unit tests. Of course, this will be part of a different task and epic. Please feel free to add any additional tickets that came up during your review of this PR and that you think we should work on.

I've created some tests with Claude support on ls/t/LS/Query/Facets.t

… parameterization)

…ends on it)

…params

…_config_dir

aelkiss

Queries are working locally for me with solr 9; I haven't tested locally with solr 6 yet.

I also still want to look some at the question of tests here.

I have some questions about some of the duplicated files, but the bigger question is when is the right time to try to disable interleave (A vs B) for solr 9 -- do we try to do that now, or do we wait until later? On the one hand, it may require fewer changes to keep it with interleave enabled than if we try to disable it. On the other, there are already some some workarounds here to make it work despite not having two different cores; this PR adds some additional workarounds, and I don't think we want to continue those workarounds indefinitely.

aelkiss · 2026-08-25T13:41:27Z

+
+side_by_side =0
+use_B_query= 0
+use_interleave=1


Does it make sense to try to turn off interleave entirely for solr 9, since we don't have a separate core-y set up there?

If use_interleave=1 works to disable interleave for solr9, that is something we should check

aelkiss · 2026-08-25T14:28:41Z

+interleaver_class = LS::Interleaver::Balanced
+#interleaver_class = LS::Interleaver::AA
+
+# Note that code does not read A_core_name to get default core name.  The default core name and paths to solr are configured in mdp-lib/Config/uber.conf.


Is it worth trying to remove the A vs B stuff for solr 9 at this point, or does that introduce additional complications?

Currently, it introduces some extra complications because we need to make sure each change works well with both Solr 6 and Solr 9. I believe it's much better to make one change at a time, step by step, to keep things manageable and clear.

That makes sense to me.

aelkiss · 2026-08-25T14:32:48Z

As far as I can tell, this isn't used; we should remove.

Yes, that file could be removed.

aelkiss · 2026-08-25T14:35:18Z

As far as I can tell this isn't used any more either; @carylwyatt could confirm. We could probably remove all instances of js_css_filelist.txt in the babel repo as well as mdp-misc/scripts/make_common and mdp-misc/scripts/make_concat. (Not on this PR, but we should at least avoid duplicating js_css_filelist.txt under solr9)

Let's create a task to remove these specific files

aelkiss · 2026-08-25T14:37:38Z

Do we need additional copies of the files under solr9 when they are the same as in the base Config directory (i.e. will they be found?). I think this is the case at least for langformat.yaml and bindings.pl.

If we do need them under solr9, does it make sense to make a symlink into the parent Config directory for them, rather than copying?

We might consider setting up a symlink; however, this method creates dependencies between Solr 6 and Solr 9 config files, so it’s important to keep track of these details in our documentation. Duplicating the files can also make the transition smoother when we’re ready to retire Solr 6, since it allows us to safely delete the folder when the time comes.

Note that Solr9/facetconfig.pl is still reading the original copy (edit: and Eugene has already pointed that out in a comment):

my $lang_format_file = $ENV{SDRROOT} . '/ls/lib/Config/langformat.yaml';

If we don't intend to use the Solr9 copy then that's argument for not having it. I would expect that data to be invariant no matter which Solr we are using.

eumalin · 2026-08-25T15:45:55Z

Config is duplicated across two trees (ls/lib/Config and .../solr9) instead of one source with overrides. It may be causing a drift, such as sdrnum missing from the solr9 copy. It's worth thinking about consolidating them, it's not a priority/blocker, just a thought.

eumalin · 2026-08-25T15:36:26Z

+    - [author2, 50]
+    - [issn, 1]
+    - [isbn, 1]
+    - [oclc, 1]


Looks like [sdrnum, 1] is missing here, should it be added back?

eumalin · 2026-08-25T15:39:50Z

    healthcheck:
      <<: *healthcheck-defaults
-      test: [ "CMD-SHELL", "curl -s -u admin:solrRocks 'http://solr-lss-dev-solr9:8983/solr/admin/collections?action=CLUSTERSTATUS&collection=core-x' | grep -q '\"state\":\"active\"'" ]
+      test: [ "CMD-SHELL", "curl -s -u admin:solrRocks 'http://solr-lss-dev-solr9:8983/admin/collections?action=CLUSTERSTATUS&collection=core-x' | grep -q '\"state\":\"active\"'" ]


Is the url correct? It changed from /solr/admin/collections to /admin/collections

Yes, this seems to be an error, /admin/collections just gives a 404 and the health check fails.

eumalin · 2026-08-25T15:41:25Z

+#my $rel_weights_file_A = $ENV{SDRROOT} . '/ls/lib/Config/dismax_A.yaml';
+#my $rel_weights_file_B = $ENV{SDRROOT} . '/ls/lib/Config/dismax_B.yaml';
+
+my $lang_format_file = $ENV{SDRROOT} . '/ls/lib/Config/langformat.yaml';


Should this be using the new ls_config_dir variable?

eumalin · 2026-08-25T15:42:12Z

 $C->set_object('MdpConfig', $config);

 # additional configuration for click_logging and A/B testing
 my $AB_test_config_filename = $ENV{SDRROOT} . '/ls/lib/Config/AB_test_config';


Should this be using the new ls_config_dir variable?

Yes, we can use the ls_config_dir here.

aelkiss · 2026-08-26T18:10:14Z

    volumes:
      - solr1_data:/var/solr
-    command: [ "sh", "-c", "solr-foreground & sleep 150 && export SOLR_AUTHENTICATION_OPTS=-Dbasicauth=admin:solrRocks && solr create_collection -d /opt/solr/core-x -c core-x -shards 1 -replicationFactor 1 -p 8983 && wait" ]
+    command: [ "sh", "-c", "solr-foreground & sleep 150 && export SOLR_AUTHENTICATION_OPTS=-Dbasicauth=admin:solrRocks && solr create_collection -d /opt/solr/core-x -c core-x -shards 1 --replicationFactor 1 -p 8983 && wait" ]


Not sure where we are tracking the issue of the collection already existing, but I would recommend instead doing something like solr-foreground && create-collection-if-needed where create-collection-if-needed is a bash script that:

waits until solr is up and responsive (polling perhaps every 5-10 seconds)

checking if the core-x collection already exists

if not, creating it

liseli · 2026-08-26T18:19:18Z

Queries are working locally for me with solr 9; I haven't tested locally with solr 6 yet.

I also still want to look some at the question of tests here.

I have some questions about some of the duplicated files, but the bigger question is when is the right time to try to disable interleave (A vs B) for solr 9 -- do we try to do that now, or do we wait until later? On the one hand, it may require fewer changes to keep it with interleave enabled than if we try to disable it. On the other, there are already some some workarounds here to make it work despite not having two different cores; this PR adds some additional workarounds, and I don't think we want to continue those workarounds indefinitely.

Let's try disabling the interleave test for Solr 9 later; it should be an optimization, probably in the user stories related to measuring relevance results. Right now, the focus is ensuring we can run queries on Solr 9 and Solr 6 in the production environment. I'll create a task to disable interleave. In this PR, I kept the interleave but pointed to only core-x for Solr 9, so we do not have to create core-y.

liseli · 2026-08-26T18:32:38Z

Config is duplicated across two trees (ls/lib/Config and .../solr9) instead of one source with overrides. It may be causing a drift, such as sdrnum missing from the solr9 copy. It's worth thinking about consolidating them, it's not a priority/blocker, just a thought.

I have duplicated the Config folders because some fields used in Solr 6 have been removed from Solr 9.

moseshll · 2026-08-27T19:35:19Z

  };
 };

+subtest 'solr9 facetconfig has incompatible fields removed' => sub {


Good to have these tests. They don't really belong here though. Really this is testing a component of FacetConfig.pm (which I will rant about a little bit later) so I would create a file ls/t/LS/FacetConfig.t and populate it with

use strict; use warnings; use Test::More; <this subtest> done_testing();

Then restore this file to its original state.

moseshll · 2026-08-27T20:35:57Z

-    my $rel_weights_A = get_rel_weights_from_yaml($A_yaml_file_name);
-    my $rel_weights_B = get_rel_weights_from_yaml($B_yaml_file_name);
-    
+    my $rel_weights_A = get_rel_weights_from_yaml($C, $A_yaml_file_name);


This module would be relatively easy to get fully under test, especially since this PR introduces tests that are at least adjacent. This module really isn't about much other than being glue. The eval on facetconfig.pl is awful. I would suggest a follow-up task to get everything here fully tested as a prelude to refactoring.

liseli added 10 commits August 19, 2026 19:54

ls: fix perl.yml Solr dev config (docker-local.conf mount + core-name…

d1a33af

… parameterization)

parameterize LSS_SOLR core name for Solr 9 core-x-only routing

aa879f3

ls: resolve uber.conf mbooks_solr_engines TODO (slip/index/sync-i dep…

2dc5856

…ends on it)

ls: remove dead Operation::Login entry from bindings.pl %g_operation_…

bbf37a6

…params

Fixing Perl syntax error

687d141

document LSS_CONFIG_DIR for running against Solr 9

04a27af

split Solr 6/Solr 9 facet config into ls/lib/Config/solr9/ and add ls…

222c985

…_config_dir

add ls_config_dir

79e5d69

Apache PassEnv fix

8bdaee3

-replicationFactor is deprecated, so replace by --replicationFactor

8c86386

liseli requested review from aelkiss, eumalin and moseshll August 21, 2026 17:30

aelkiss reviewed Aug 25, 2026

View reviewed changes

eumalin reviewed Aug 25, 2026

View reviewed changes

aelkiss reviewed Aug 26, 2026

View reviewed changes

moseshll reviewed Aug 27, 2026

View reviewed changes

Uh oh!

Conversation

liseli commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aelkiss left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

moseshll Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eumalin commented Aug 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liseli commented Aug 26, 2026

Uh oh!

liseli commented Aug 26, 2026

Uh oh!

moseshll Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

moseshll Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

liseli commented Aug 21, 2026 •

edited

Loading

moseshll Aug 27, 2026 •

edited

Loading

moseshll Aug 27, 2026 •

edited

Loading

moseshll Aug 27, 2026 •

edited

Loading