Incremental Feed CSV Data Format

This page describes the data format to follow when implementing a CSV integration with Fredhopper.

Files Required

The table below lists all the files that can be expected as part of the incremental update:

Filename	Description
categories.csv	This file should be the exact same format as that provided in the most recent full re-index data file upload: it should contain the complete category tree rather than just the categories which are related to the catalogue items being updated as part of the incremental update.
products.csv	This file should only contain references to the product ids which are to be updated as part of the incremental update.
variants.csv	Similar in logic to products.csv, this file should only contain references to any variants which are to be updated.
custom_attributes_meta.csv	This file should contain attribute definitions for all the attributes in the catalogue, whether they feature within the incremental update or not.
custom_attributes_values.csv	This file should (ideally only) contain the product level attributes which are to be updated as part of the incremental update (optional).
custom_variant_attributes.csv	This file should (ideally only) contain the variant level attributes which are to be updated as part of the incremental update (optional).

If your data model is such that there is no concept of variants, then the two variant CSV files can be omitted. Whilst these files have a .csv extension, the data columns are in fact separated by double tab characters rather than a comma.

Required Files

For incremental updates, only the necessary files are required. This is depends on the update scenario and on what is updated. Only variants, products or both and what type of attribute are updated. Please see Fast Incremental below for more information regarding the fast incremental file requirements.

• categories.csv - Ideally the same file as used in the most recent full data feed.

• custom_attributes_meta.csv - Minimally all relevant attributes, but is recommended to use the version from the most recent full data feed.

• products.csv - Only the updated products

• custom_attributes_values.csv - Recommended to export only the new attribute values. This depends on the operation type. When using 'replace' all new values need to be provided.

• variants.csv (optional) - Only necessary when variant data is updated. Only relevant variant id's need to be listed.

• custom_variant_attributes.csv (optional) - Only necessary when variant data is updated. Recommended to export only the new attribute values. This depends on the operation type. When using 'replace' all new values need to be provided.

All files always need to have at least one value line (apart from the header line). Optional files should not be included to the data batch when there are no values for those specific files.

If a file is created and sent but not fitting these criteria, the data-load trigger will fail. This is a fail-safe mechanism.

Operation Type

The full re-index process always expects that the entire product catalogue be provided within the data files provided. This differs in comparison to the incremental update process as here we may wish to only update an attribute value, add a new product or delete a variant from the catalogue entirely. The way that we inform Fredhopper which action to perform in an incremental update is defined by the 'operation_type' attribute within the data. The locale for this attribute does not matter and if specified, it will be disregarded. For example, if a variant with two locales must be removed it is sufficient to include only one delete operation type in custom_variant_attributes.csv (see in the sample data below how this is done).

Please be aware that you must not specify an attribute_value_id for 'operation_type' entries.

The 'operation_type' attribute is defined within custom_attributes_values.csv and (if required) custom_variant_attributes.csv for each product / variant respectively and can be one of four values below. If no 'operation_type' attribute is configured for an item, then the 'add' operation will be applied by default.

Operation Type	Description
update	An existing product or variant item needs to have some of its data modified. By default, if the given item does not exist within the FAS index, no changes will be completed. This default behaviour can be overridden in the system.xml settings so that the item is processed as an add type of update instead. This operation_type cannot be used to nullify an existing attribute value or remove an attribute completely for an item - the replace operation must be used in this case.
replace	An existing product or variant item (and all its attributes) should be removed completely from the FAS index and will be replaced with the new data provided within the incremental update.
add	A new product or variant item has been defined within the incremental update and the attribute values need be loaded for the first time. By default, if the item exists already within the index, FAS will apply the replace logic to the existing item in this case. This default behaviour can be overridden in the system.xml so that the existing item does not get replaced and a warning will be displayed in the logs for the affected item(s) instead.
delete	An existing product or variant item needs to be removed completely from the FAS index.

*An item can be either a product or variant.

FAS will execute all operations on the specified item so, if both a product and its variants need updating, all of those updates must be provided explicitly. The only exception to this rule is that, deleting a product that has variants attached, will remove both the product and all variants automatically without the need for each variant to be deleted explicitly.

Fast Incremental Updates

The speed that it takes to process and load an incremental update is already much faster than it takes a full data upload to complete. However, it is possible to have a faster, more expedited update process when sending an incremental update batch that contains only numerical and/or basic text values. In this situation, some of the standard data processing tasks and verification checks that are built into the process can be skipped because they are not necessary. This method should be used whenever possible to provide near real-time updates of numerical and/or text values, for example to apply stock and/or price updates. For this to work, the operation_type can only be 'update' and the input files will only contain a subset of the product/variant attributes that are to be updated.

The data API will automatically complete a fast incremental update when all of the following conditions have been met:

• The custom_attributes_meta.csv file is present in the input data-incremental.zip file

• The custom_attributes_values.csv and/or the custom_variant_attributes.csv file(s) is(are) present in data-incremental.zip file

• The operation_type values for all changes are set to 'update' (using any other value will result in a 'normal' incremental update)

• The data type of all the attribute values to be updated are either int, float or text (using any other data type will result in a 'normal' incremental update)

Incremental Product Data - Input Format

The data model depicted in the examples below represents a single universe, multi locale deployment which has two categories (womens & shoes) that contain two products and three variants in total. The update scenario that we would like to achieve is as follows:

• p1 is updated with a new colour attribute value (blue)

• p2 is added as a new item to the product catalogue

• v1 is removed from the product catalogue

• v2 is updated with a new price value (30.00)

categories.csv

category_id	parent_category_id	locale	name
catalog01	catalog01	en_US	Shop
catalog01	catalog01	de_DE	Geschaeft
women	catalog01	en_US	Women
women	catalog01	de_DE	Damen
shoes	catalog01	en_US	Shoes
shoes	catalog01	de_DE	Schuhe

products.csv

product_id	locale	category_ids
p1	de_DE	shoes
p1	en_US	shoes
p2	de_DE	shoes
p2	en_US	shoes

custom_attributes_meta.csv

attribute_id	type	locale	name
stock	int		Stock
color	set	en_US	Color
color	set	de_DE	Kleur
price	float		Price

custom_attributes_values.csv

product_id	locale	attribute_id	attribute_value_id	attribute_value
p1	en_US	color	blue	Blue
p1	de_DE	color	blue	Blauw
p1	en_US	operation_type		update
p2	en_US	color	red	Red
p2	de_DE	color	red	Rot
p2		stock		10
p2	en_US	operation_type		add

variants.csv

variant_id	product_id	locale
v1	p1	en_US
v1	p1	de_DE
v2	p1	en_US
v2	p1	de_DE
v3	p2	en_US
v3	p2	de_DE

custom_variants_attributes.csv

variant_id	locale	attribute_id	attribute_value_id	attribute_value
v1		operation_type		delete
v2		price		30.00
v2	en_US	operation_type		update