Importation en lot de vos sites via un fichier CSV avec l'API Woosmap

6 min - Author: Marine

Sommaire

L'une des premières choses qu'un nouvel utilisateur Woosmap fait avant d'intégrer l'application Woosmap sur son site est d'importer ses données de localisation via la Woosmap Data Management API. Ce guide propose un exemple Python pour faciliter la gestion de vos points de vente à partir d'un fichier CSV.

Traitement de votre fichier CSV

Exemple de fichier CSV

Les données utilisées pour illustrer ce guide sont un extrait CSV de Data Gouv France répertoriant l'ensemble des musées en France. Voici un aperçu du fichier museum-france-data.csv :

Ce fichier CSV contient des valeurs de géométrie nulles (points non géocodés) et des colonnes non pertinentes. Un nettoyage est nécessaire. csvkit est une suite d'outils en ligne de commande pour convertir et manipuler des fichiers CSV. En particulier, csvgrep et csvcut sont très utiles pour extraire les données souhaitées. Vous pouvez également utiliser une interface graphique comme Apple Numbers, Microsoft Excel ou Google Sheets, mais csvkit offre un traitement plus efficace.

Conservez uniquement les données géocodées (où le champ latitude n'est pas vide).

csvgrep -d ";" -c latitude -r "^(?=\s*\S).*$" france-museum-.csv |

Extrayez ensuite les colonnes souhaitées.

csvcut -c "NOM DU MUSEE",ADR,CP,VILLE,SITWEB,latitude,longitude > france_museum_geocoded.csv

CSV Python Module

Because Woosmap Data Management API only accepts JSON as input, the next step is to transform each row of your file to Woosmap elements. The following python script is based on the native csv module. It makes it easier to deal with csv formatted file, especially when working with data exported from spreadsheets and databases into text files. We preferred to iterate over the data rows using csv.DictReader instead of classic reader mainly because it takes advantage of the header row with the column names and gives you the opportunity to grab it cell value by its name.

Dialect

There is no well-defined standard for comma-separated value files, there are plenty of ways one CSV file can differ from another, yet contains exactly the same data. Many tools, which can import or export tabular data allow the user to indicate the field delimiter, quote character, line terminator, and other characteristics of the file. All these parameters are grouped together conveniently into a dialect object.

When creating a csv.DictReader object, the programmer can specify a subclass of the Dialect class as the dialect parameter. In our CSV file the fields are separated by commas and some values are double quoted.

import csv
from _csv import QUOTE_ALL
from csv import Dialect
class data_gov_dialect(Dialect):
delimiter =','
quotechar = '"'
doublequote = True
skipinitialspace = False
lineterminator = '\n'
quoting = QUOTE_ALL
csv.register_dialect('dg', data_gov_dialect)

Work with Woosmap API

Data Structure

The one step further is to transform each row element to a location object that follows the required structured format. The mandatory fields are a storeId (as String), a name and a location geometry object (a couple of Latitude and Longitude). Below is the method used to transform each row of our CSV to a Woosmap location element:

def get_geometry(store):
return {
'lat': store['latitude'],
'lng': store['longitude']
}
def get_contact(store):
return {
'website': store['SITWEB']
}
def get_address(store):
return {
'lines': [store['ADR']],
'city': store['VILLE'],
'zipcode': store['CP'],
}
def datagov2woosmap(store, id):
geometry = get_geometry(store)
address = get_address(store)
contact = get_contact(store)
return {
'storeId': id,
'name': store['NOM DU MUSEE'],
'address': address,
'contact': contact,
'location': geometry
}

Il est maintenant possible d'ouvrir le fichier et de parcourir les données pour construire un tableau de tous les éléments Woosmap avant de les envoyer à l'API via POST :

with open(endpoint_csv, 'r') as f: #open the file
reader = csv.DictReader(f, dialect="dg") #read it as a dict of dict
batch = []
for location in reader: #loop over this dictreader
try:
woosmap_location = datagov2woosmap(location) #build the woosmap element
batch.append(woosmap_location) #and append it to an array
except InvalidGeometry:
pass
import(batch) #finally import all your data

Import using HTTP POST Method

The Woosmap Data API is a RESTful API with endpoint https://api.woosmap.com/stores. It takes a mandatory parameter : your private key. Therefore the url you call should look like this: htps://api.woosmap.com/?private_key=YOUR_PRIVATE_KEY. The API provides 4 HTTP methods for interacting with resources but in our case we are interested in creating new resources so we’ll use the POST method.

import requests
private_key = ''  #your woosmap_api_private_key here
endpoint_api = 'https://api.woosmap.com/stores'
def import(batch):
session = requests.Session()
response = session.post(endpoint_api,
params={'private_key': private_key},
json={'stores': batch})

Rolling batch import

If you work with more than 1000 locations, we recommend you to set a rolling import every X elements. To do this, define a numerical constant in your code (100 for example) and execute your POST request each time this counter value is reached.

batch_size = 100
batch = []
for location in data: #loop over this array
try:
woosmap_location = datagov2woosmap(location) #build the woosmap element
if len(batch) == batch_size: 
import(batch) #import when you batch array contains 100 elements
batch = []
else:
batch.append(woosmap_location)
except InvalidGeometry:
pass
if batch:
import(batch) #finally import the remaining elements

Comme vous pouvez le voir dans l'exemple hébergé sur GitHub, le code permet de mettre à jour vos points de vente via le verbe HTTP PUT et de supprimer l'ensemble de la ressource via DELETE avant de la recréer. L'identifiant unique de vos données étant la base de la méthode de mise à jour, veillez à conserver le même identifiant.

Sources

Woosmap API documentation and the official Woosmap website
Python Sample batchimport on GitHub
CSV Manipulation with csvkit utilities