Keyword Ingest module failure

[RTGholson]

Going through the online training for Autopsy (4.140) and have run into a bit of a snag. The keyword search ingest module is “running home to mama” whenever I attempt to run it in conjunction with the lab exercise.

The keyword list has been created and whenever I run the ingest module, it throws the following error(s):

Here are the contents of the log file that was generated:

2020-02-02 07:39:24.438 org.sleuthkit.autopsy.keywordsearch.Server isRunning
INFO: Solr server is running
2020-02-02 07:39:24.443 org.sleuthkit.autopsy.casemodule.Case openAppServiceCaseResources
SEVERE: Solr Keyword Search Service failed to open case resources for case1
java.util.concurrent.ExecutionException: org.sleuthkit.autopsy.appservices.AutopsyService$AutopsyServiceException: Failed to open or create core for C:\Users\RTGho\Desktop\Shortcuts\Autopsy_Lab\case1
java.util.concurrent.FutureTask.report(FutureTask.java:122)
java.util.concurrent.FutureTask.get(FutureTask.java:192)
org.sleuthkit.autopsy.casemodule.Case.openAppServiceCaseResources(Case.java:2403)
org.sleuthkit.autopsy.casemodule.Case.open(Case.java:1979)
org.sleuthkit.autopsy.casemodule.Case.lambda$doOpenCaseAction$6(Case.java:1847)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
org.sleuthkit.autopsy.appservices.AutopsyService$AutopsyServiceException: Failed to open or create core for C:\Users\RTGho\Desktop\Shortcuts\Autopsy_Lab\case1
java.util.concurrent.FutureTask.report(FutureTask.java:122)
java.util.concurrent.FutureTask.get(FutureTask.java:192)
org.sleuthkit.autopsy.casemodule.Case.openAppServiceCaseResources(Case.java:2403)
org.sleuthkit.autopsy.casemodule.Case.open(Case.java:1979)
org.sleuthkit.autopsy.casemodule.Case.lambda$doOpenCaseAction$6(Case.java:1847)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
org.sleuthkit.autopsy.keywordsearch.KeywordSearchModuleException: Could not create or open index
java.util.concurrent.FutureTask.report(FutureTask.java:122)
java.util.concurrent.FutureTask.get(FutureTask.java:192)
org.sleuthkit.autopsy.casemodule.Case.openAppServiceCaseResources(Case.java:2403)
org.sleuthkit.autopsy.casemodule.Case.open(Case.java:1979)
org.sleuthkit.autopsy.casemodule.Case.lambda$doOpenCaseAction$6(Case.java:1847)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
org.sleuthkit.autopsy.keywordsearch.KeywordSearchModuleException: Index directory could not be created or is missing
java.util.concurrent.FutureTask.report(FutureTask.java:122)
java.util.concurrent.FutureTask.get(FutureTask.java:192)
org.sleuthkit.autopsy.casemodule.Case.openAppServiceCaseResources(Case.java:2403)
org.sleuthkit.autopsy.casemodule.Case.open(Case.java:1979)
org.sleuthkit.autopsy.casemodule.Case.lambda$doOpenCaseAction$6(Case.java:1847)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
2020-02-02 07:39:24.449 org.sleuthkit.autopsy.imagegallery.datamodel.DrawableDB setPragmas
INFO: sqlite-jdbc version 3.25.2 loaded in native mode
2020-02-02 07:39:24.464 org.sleuthkit.autopsy.casemodule.Case openAsCurrentCase
INFO: Opened case1 (case1_20200131_090630) in C:\Users\RTGho\Desktop\Shortcuts\Autopsy_Lab\case1 as the current case
2020-02-02 07:39:24.473 org.sleuthkit.autopsy.ingest.IngestMonitor$MonitorTimerAction logMonitoredRootDirectory
INFO: Monitoring disk space of C:
2020-02-02 07:42:40.096 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Correlation Engine, version = 4.14.0
2020-02-02 07:42:40.096 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Data Source Integrity, version = 4.14.0
2020-02-02 07:42:40.096 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Embedded File Extractor, version = 4.14.0
2020-02-02 07:42:40.096 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Encryption Detection, version = 4.14.0
2020-02-02 07:42:40.096 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Exif Parser, version = 4.14.0
2020-02-02 07:42:40.096 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Extension Mismatch Detector, version = 4.14.0
2020-02-02 07:42:40.096 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = File Type Identification, version = 4.14.0
2020-02-02 07:42:40.096 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Hash Lookup, version = 4.14.0
2020-02-02 07:42:40.096 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Interesting Files Identifier, version = 4.14.0
2020-02-02 07:42:40.096 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = PhotoRec Carver, version = 7.0
2020-02-02 07:42:40.096 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Plaso, version = 4.14.0
2020-02-02 07:42:40.096 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Virtual Machine Extractor, version = 4.14.0
2020-02-02 07:42:40.097 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Keyword Search, version = 4.14.0
2020-02-02 07:42:40.097 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Email Parser, version = 4.14.0
2020-02-02 07:42:40.097 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Recent Activity, version = 4.14.0
2020-02-02 07:42:40.355 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader getIngestModuleFactories
INFO: Found ingest module factory: name = Android Analyzer, version = 4.14.0
2020-02-02 07:42:40.456 org.sleuthkit.autopsy.keywordsearch.KeywordSearchGlobalSearchSettingsPanel customizeComponents
WARNING: Could not get number of indexed files/chunks
2020-02-02 07:42:44.984 org.sleuthkit.autopsy.ingest.IngestManager startIngestJob
INFO: Starting ingest job 1
2020-02-02 07:42:44.985 org.sleuthkit.autopsy.keywordsearch.KeywordSearchIngestModule shutDown
INFO: Keyword search ingest module instance 2 shutting down
2020-02-02 07:42:44.985 org.sleuthkit.autopsy.ingest.IngestManager startIngestJob
SEVERE: Error starting Keyword Search ingest module for job 1
org.sleuthkit.autopsy.ingest.IngestModule$IngestModuleException: The index could not be opened or does not exist.
org.sleuthkit.autopsy.keywordsearch.KeywordSearchIngestModule.startUp(KeywordSearchIngestModule.java:215)
org.sleuthkit.autopsy.ingest.FileIngestPipeline$PipelineModule.startUp(FileIngestPipeline.java:225)
org.sleuthkit.autopsy.ingest.FileIngestPipeline.startUp(FileIngestPipeline.java:106)
org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startUpIngestPipelines(DataSourceIngestJob.java:460)
org.sleuthkit.autopsy.ingest.DataSourceIngestJob.start(DataSourceIngestJob.java:420)
org.sleuthkit.autopsy.ingest.IngestJob.start(IngestJob.java:158)
org.sleuthkit.autopsy.ingest.IngestManager.startIngestJob(IngestManager.java:407)
org.sleuthkit.autopsy.ingest.IngestManager.access$600(IngestManager.java:111)
org.sleuthkit.autopsy.ingest.IngestManager$StartIngestJobTask.call(IngestManager.java:895)
org.sleuthkit.autopsy.ingest.IngestManager$StartIngestJobTask.call(IngestManager.java:858)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
2020-02-02 07:42:44.985 org.sleuthkit.autopsy.ingest.IngestManager startIngestJob
SEVERE: Ingest job 1 could not be started
2020-02-02 07:45:21.081 org.sleuthkit.autopsy.keywordsearch.KeywordSearchGlobalSearchSettingsPanel customizeComponents
WARNING: Could not get number of indexed files/chunks
2020-02-02 07:45:52.636 org.sleuthkit.autopsy.texttranslation.translators.GoogleTranslator loadTranslator
INFO: No credentials file has been provided for Google Translator
2020-02-02 07:45:52.637 org.sleuthkit.autopsy.texttranslation.translators.GoogleTranslator loadTranslator
WARNING: Credentials were not successfully made, no translations will be available from the GoogleTranslator

Any suggestions and/or thoughts are greatly appreciated. I also posted a Discussion board comment in the training for that module and will update it with the solution if we can determine what the issue is.

Well, I think I see the issue :

org.sleuthkit.autopsy.keywordsearch.KeywordSearchModuleException: Could not create or open index
java.util.concurrent.FutureTask.report(FutureTask.java:122)

Just not sure how to rectify it.

Thanks!

[downey]

What happens if you use a location other than “C:\Users\RTGho\Desktop\Shortcuts” for your case output?

[RTGholson]

Hello downey:

I gave that a whirl. Relocated the case to “my documents” and tried opening it in Autopsy again and that yielded the same results and error messages. Rebooted the system (for grins and giggles) and tried it after the reboot with the same results unfortunately.

Thanks for the suggestion though!

[downey]

Do you see any log messages it you visit http://localhost:23232/solr in a browser when Autopsy is running?
Click on “Logging” in the Solr web UI.

[RTGholson]

@downey

http://localhost:23232/solr

Oh yea…lots. Here’s a screenshot:

[downey]

Unfortunately those messages are normal and don’t shed any new light on the issue. Where it says “Creating new index…” do you see a folder named “index” in the location where it claims to be creating it?
Another troubleshooting step would be to go to the “Core Selector” dropdown in the Solr web UI to see if you can successfully access the Solr index for your case.

[RTGholson]

Doh! Sorry about that.

Ok,

Yep there is an index folder in that location with 94 small files in it:

When I select the case in the “Core Selector” dropdown, everything below it is grayed out. I can select those items and see associated information on the right until I get to the “files” folder. It just says “loading” and never yields any information.

I’m going to load Autopsy up on another machine in the office tomorrow and see if I get the same results. I’m guessing it’s something germane to this particular machine, otherwise we would see other people with the same issue.

Really appreciate your time and suggestions though.

I’ll post my results with a new machine tomorrow and see if I encounter the same issue.

[RTGholson]

Well,

The issue did not repeat itself on a different machine, so I’m not sure what went wrong with the first installation.

Did a clean install on another windows box and the keyword search worked as suspected. I’m going to try to isolate the problem on the other machine and see if I can figure out why.

It’ll drive me nuts until I do…

[RTGholson]

Solved:

I told you it would drive me nuts…

After some back and forth the over the last week, Downey and I finally identified the culprit. The core properties for the Solr instance that was spinning up with my lab case were wrong. When I started the training lab, I had originally placed the case folder for the lab exercises on my desktop. As part of our troubleshooting, I moved that directory to my “Documents” While poking around the core admin properties section this morning, I noted that the core properties paths for my lab case were as follows:

The instanceDir was correct, but the dataDir wasn’t. I had to go in and modify the attributes of the “core.properties” file for my case to correct that error.

The "core.properties file was located in the “case1_20200131_090630_20200131_090632” sub-directory of my F: drive.

Once I commented out the original code, inserted the correct path and restarted Autopsy - Boom goes the dynamite. Working as expected now.

Once I restarted Autopsy, and checked the core admin properties for Solr again, all was as it should be:

If you run into a communications issue between Autopsy and Solr while attempting to run a keyword search do the following: Start Autopsy, load your case and make sure you can A) connect to Solr via http://localhost:23232/solr and B) that the core properties under the "Core Admin: section on the left side of the Solr control panel are correct. If they aren’t, you might have to correct that in the core.properties file noted above.

Cheers,

Ryan

[downey]

Following up on this…I’ve had some time to experiment and can reproduce the error dialogs observed in the first message in this thread. Here’s what I did to reproduce:

1. Start Autopsy, create a new case and add a data source.
2. Kill the Autopsy process tree (which will also kill the Solr process) through Task Manager.
3. Move the case output folder to a different location on disk.
4. Restart Autopsy and attempt to open the case. You should see the error dialog(s).

Explanation:

• Solr creates a core.properties file when a case is created.This is typically located in %APPDATA%\autopsy\solr but this can change depending on where Autopsy has been installed to.
• When Autopsy/Solr close cleanly, Solr renames “core.properties” to “core.properties.unloaded”.
• When Autopsy/Solr are killed or crash Solr does not get to rename the core.properties file and the next time it starts it attempts to read the contents of the core.properties file.
• If the case output folder has been moved someplace else the core.properties file will have incorrect information and you will experience the issue described in this thread.

An alternative to editing the core.properties file in this situation is simply to delete it. Solr will create a new one when Autopsy attempts to open the case