Custom Data Uploads

This page describes the the steps to follow when uploading a custom data feed to Fredhopper.

Overview

If you are implementing a custom feed for additional data enrichment you will need to send these to the upload service.

The following section details the standard data input format that is expected by the Upload service. It also explains how these files should be transmitted to Fredhopper's Managed Services environment and for the data to be triggered on the instance.

Once the additional data has been successfully uploaded and processed, it will be integrated with the next product feed update (full or incremental). The same custom data will be maintained for all subsequent load-data triggers until a new version of the custom data is uploaded using this process.

File Name: data.zip

Example URL: https://my.eu1.fredhopperservices.com/upload

Upload Steps

These are the steps to follow when uploading data to Fredhopper, this is the same for Item data, suggest data, and custom uploads.

The file names and URL's to upload too will differ and are covered in the examples.

1. Create a ZIP archive

Create a data.zip file containing the data you need to upload to Fredhopper.

2. Create MD5 checksum

Generate an md5 checksum value zip file.

Example Request

md5sum data.zip > data.zip.md5

This checksum we'll use to validate the upload.

3. Upload ZIP archive to Fredhopper

Upload the zip file to the Fredhopper Managed Services environment that is given to you by your Technical Consultant using the 'fas' service interface.

Please note to include the checksum value in the request, as per the example below which is directed to the live1 instance.

Example Request

curl -D - -u "[cloud_username]":"[cloud_password]" -X PUT -H "Content-Type: application/zip" --data-binary @data.zip https://my.eu1.fredhopperservices.com/upload:live1/data/input/data.zip\?checksum=19afc8c99ec7ea27f0dba8e1c4fcff1f

Once the file has been uploaded, the system will send a simple HTTP response back with some important information contained within the header and body sections:

Example Response

HTTP/1.1 100 Continue
 
HTTP/1.1 201 Created
Date: Fri, 21 Feb 2020 15:31:55 GMT
Server: Jetty(9.4.22.v20191022)
Location: https://my.eu1.fredhopperservices.com/upload:live1/data/input/2020-02-21_15-31-55
Content-Type: text/plain
Content-Length: 28
Connection: close
 
data-id=2020-02-21_15-31-55

You should capture the 'data-id' section in the HTTP response body that you receive from the API as this will be used in subsequent steps. The 'data-id' value is unique to every request.

4. Trigger for data to be loaded

Thus far, we have only uploaded the data to the Fredhopper Managed Services environment. Now, we shall initiate a trigger instructing Fredhopper to re-index with the new data and you must use the 'data-id' value that you captured in the previous step.

Example Request

curl -D - -u "[cloud_username]":"[cloud_password]" -X PUT -H "Content-Type: text/plain" --data-binary "data-id=2020-02-21_15-31-55" https://my.eu1.fredhopperservices.com/upload:live1/trigger/finalize

The HTTP response header that you receive back from our system at this stage contains a new 'Location' value, which we can use to monitor the status of the re-index:

Example Response

HTTP/1.1 201 Created
Date: Fri, 21 Feb 2020 15:47:17 GMT
Server: Jetty(9.4.22.v20191022)
Location: https://my.eu1.fredhopperservices.com/upload:live1/trigger/finalize/2020-02-21_15-47-17
Content-Length: 0
Connection: close

5. Monitor the status of the re-index

The status value for the re-index job can be checked by sending the following command using the 'Location' value that you captured in the previous step:

Example Request

curl -s -D - -u "[cloud_username]":"[cloud_password]" https://my.eu1.fredhopperservices.com/upload:live1/trigger/finalize/2020-02-21_15-47-17/status

Example Response

HTTP/1.1 200 OK
Date: Fri, 21 Feb 2020 15:49:09 GMT
Server: Jetty(9.4.22.v20191022)
Content-Type: text/plain
Content-Length: 57
Connection: close
 
RUNNING
Started execution at Fri Feb 21 15:47:17 UTC 2020

When complete you should get a success response.

Example Response

HTTP/1.1 200 OK
Date: Fri, 21 Feb 2020 15:51:15 GMT
Server: Jetty(9.4.22.v20191022)
Content-Type: text/plain
Content-Length: 8
Connection: close
 
SUCCESS

Possible status codes returned are:

Status	Description
Unknown	No known state yet: trigger has not yet been picked up
Scheduled	Trigger has been picked up, and will start execution soon
Running	Triggered job is running currently
Delayed	Triggered job is ready to run, but delayed (eg: due to insufficient capacity)
Success	Triggered job has finished successfully
Failure	Triggered job has failed

When deciding on the frequency to poll the progress of the load-data processes from your-side, we recommend having a minimum interval of 60 secs in-between each check that you make.