Suggest Data Uploads

This section describes the the steps to follow when uploading suggest data to Fredhopper.

Overview

Once the blacklist.csv and whitelist.csv files are finalised and they contain the latest terms, they will need to be uploaded to Fredhopper via the REST API. These files need to be uploaded before the Suggest service is re-indexed otherwise the included and excluded terms will not be considered by the system. We recommend completing a re-index of the Suggest service on your production environment at least once every 24hrs and this should take place after FAS has been re-indexed with a full data update. This will ensure that the Suggest service provides the most relevant suggestions including the latest custom exceptions. The two files must be included in an archive file named 'data.zip' and an accompanying md5 checksum must also be generated. When the archive file has been uploaded successfully to the data repository, a trigger should be sent to instruct the system to load the new data into the relevant Suggest instance.

It is important to note that the Suggest service will remain fully operational during the re-index process because redundancy is built into the solution.

File Name: data.zip

Example URL: https://my.eu1.fredhopperservices.com/suggest

Steps

1. Create ZIP archive

Create a zip file named "data.zip" which contains the blacklist and whitelist files.

zip data.zip *.csv

2. Create MD5 checksum

Generate an md5 checksum value for the data.zip or incremental-data.zip file:

md5sum data.zip > data.zip.md5

In this example, the generated checksum value = 731b0140b43a830255cc2dd28c4923fd.

3. Upload ZIP archive to Fredhopper

Upload the data.zip file to the Fredhopper Managed Services environment that is given to you by your Technical Consultant using the 'suggest' service interface.

Please note to include the checksum value in the request, as per the example below which is directed to the live1 instance.

curl -D - -k -u user:password -X PUT -H "Content-Type: application/zip" --data-binary @data.zip https://my.eu1.fredhopperservices.com/suggest:live1/data/input/data.zip?checksum=731b0140b43a830255cc2dd28c4923fd

Once the file has been uploaded, the system will send a simple HTTP response back with some important information contained within the header and body sections.

HTTP/1.1 100 Continue
  
HTTP/1.1 201 Created
Date: Fri, 23 Feb 2018 15:15:54 GMT
Server: Jetty(9.3.6.v20151106)
Location: http://my.eu1.fredhopperservices.com/suggest:live1/data/input/2018-02-23_15-15-53/
Content-Type: text/plain
Content-Length: 28
Via: 1.1 my.eu1.fredhopperservices.com
Vary: Accept-Encoding
Connection: close
  
data-id=2018-02-23_15-15-53

You should capture the 'data-id' section in the HTTP response body that you receive from the API as this will be used in subsequent steps. The 'data-id' value is unique to every request.

4. Trigger for data to be loaded

Thus far, we have only uploaded the data to the Fredhopper Managed Services environment. Now, we shall initiate a trigger instructing Fredhopper Suggest to re-index with the latest data and you must use the 'data-id' value that you captured in the previous step. Please see the example below:

curl -D - -k -u user:password -X PUT -H "Content-Type: text/plain" --data-binary "data-id=2018-02-23_15-15-53" https://my.eu1.fredhopperservices.com/suggest:live1/trigger/generate

The HTTP response header that you receive back from our system at this stage contains a new 'Location' value, which we can use to monitor the status of the re-index:

HTTP/1.1 201 Created
Date: Fri, 23 Feb 2018 15:15:54 GMT
Server: Jetty(9.3.6.v20151106)
Location: http://my.eu1.fredhopperservices.com/suggest:live1/trigger/generate/2018-02-23_15-41-44
Content-Length: 0
Via: 1.1 my.eu1.fredhopperservices.com
Vary: Accept-Encoding
Connection: close

5. Monitor the status of the re-index

The status value for the re-index job can be checked by sending the following command using the 'Location' value that you captured in the previous step:

curl -D - -k -u user:password -X GET https://my.eu1.fredhopperservices.com/suggest:live1/trigger/generate/2018-02-23_15-41-44/status

Possible status codes returned are:

Status	Description
Unknown	No known state yet: trigger has not yet been picked up
Scheduled	Trigger has been picked up, and will start execution soon
Running	Triggered job is running currently
Delayed	Triggered job is ready to run, but delayed (eg: due to insufficient capacity)
Success	Triggered job has finished successfully
Failure	Triggered job has failed

When deciding on the frequency to poll the progress of the generate processes from your-side, we recommend having a minimum interval of 60 secs in-between each check that you make.

There are no data quality reports available for Fredhopper Suggest.

Process Diagram