Full and Incremental Feed Uploads
This page describes the the steps to follow when uploading a full or incremental item data feed to Fredhopper.
Full Feeds
Once the format of the product files is complete, they will need to be uploaded to Fredhopper via the REST API. We recommend completing a full product data update on your production environment using this method at least once every 24hrs on an automated schedule. The files must be included in an archive file named 'data.zip' and an accompanying md5 checksum must also be generated. When the archive file has been uploaded successfully to the data repository, a trigger should be sent to instruct the system to load the new data into the relevant FAS instance. If a full data update is sent and triggered when an incremental data update is already running, the load-data job will be added to a queue and the system will process it when the other job has completed.
It is important to note that, when the Indexer node is completing a re-index of the product data, it will not be possible to access the Merchandising Studio or the pre-published query endpoint as these services are hosted on here. Access is restored as soon as the job has completed and the job starts to re-index the next node in the solution. The published query endpoint will never become inaccessible during this process because redundancy is built into the solution.
File Name: data.zip
Example URL: https://my.eu1.fredhopperservices.com/fas
Incremental Feeds
Once the format of the incremental product files is complete, they will need to be uploaded to Fredhopper via the REST API. A typical incremental update frequency would be every 15-60 mins. The files must be included in a 'data-incremental.zip' archive file and an md5 checksum must also be generated. When the archive file has been uploaded successfully to the data repository, a trigger should be sent to instruct the system to load the new data into the relevant FAS instance. If an incremental update is sent and triggered when a full data update is already running, the update job will be added to a queue and the system will process it when the other job has completed.
It is important to note that all services remain online and accessible during the incremental update re-index process.
File Name: incremental-data.zip
Example URL: https://my.eu1.fredhopperservices.com/fas
Upload Steps
These are the steps to follow when uploading data to Fredhopper, this process is the same for full and incremental feeds.
The file names and URL's to upload too will differ and are covered in the examples.
1. Create a ZIP archive
Create a zip file containing the data you need to upload to Fredhopper.
Full feeds should be named data.zip and incremental feeds named data-incremental.zip
The example above is for CSV files, for JSON files this is the same process, just specify the correct file type.
2. Create MD5 checksum
Generate an md5 checksum value zip file.
This checksum we'll use to validate the upload.
3. Upload ZIP archive to Fredhopper
Upload the zip file to the Fredhopper Managed Services environment that is given to you by your Technical Consultant using the 'fas' service interface.
Please note to include the checksum value in the request, as per the example below which is directed to the test1 instance.
Once the file has been uploaded, the system will send a simple HTTP response back with some important information contained within the header and body sections:
There is no different in the response between full and incremental feeds.
You should capture the 'data-id' section in the HTTP response body that you receive from the API as this will be used in subsequent steps. The 'data-id' value is unique to every request.
4. Trigger for data to be loaded
Thus far, we have only uploaded the data to the Fredhopper Managed Services environment. Now, we shall initiate a trigger instructing Fredhopper to re-index with the new data and you must use the 'data-id' value that you captured in the previous step.
The HTTP response header that you receive back from our system at this stage contains a new 'Location' value, which we can use to monitor the status of the re-index.
5. Monitor the status of the re-index
The status value for the re-index job can be checked by sending the following command using the 'Location' value that you captured in the previous step.
Possible status codes returned are:
Unknown
No known state yet: trigger has not yet been picked up
Scheduled
Trigger has been picked up, and will start execution soon
Running
Triggered job is running currently
Delayed
Triggered job is ready to run, but delayed (eg: due to insufficient capacity)
Success
Triggered job has finished successfully
Failure
Triggered job has failed
When deciding on the frequency to poll the progress of the load-data processes from your-side, we recommend having a minimum interval of 60 secs in-between each check that you make.
6. Check the Data Quality Report
Fredhopper compiles a data quality summary report for each full load and incremental data job, which may be viewed once the process has completed successfully. We recommend that these reports are reviewed regularly over time, e.g. monthly or quarterly, to ensure that the product data continues to be processed fully by Fredhopper and there are no errors.
For each check that is listed in the summary report, a more detailed report exists within a .gz Gnu zip file to provide the cause of any issues and these can be accessed from the 'data-quality' sub-folder. Using the above example, the detailed reports can be accessed from the following location:
Process Diagram
Last updated
