Social Media is the best Sensor Network of your City.

People, by using Social Networks, have become one of the most interesting and reliable sensor networks of any City.

Smart City technology in the last few years has mainly concentrated in the issues of networking, security, sensors and sharing open data. However, a very rich source of information about the city has developed in parallel (for free!) and has not yet been integrated in a meaningful way into the Smart City paradigm: people’s opinions, measured in real-time through Social Networks. In this paper we develop the idea that this can be considered as a form of rich sensor network, sending us in real-time a sample of people’s opinions.

Over the last four years we have developed this approach, working with several Spanish city councils, including the cities of Barcelona and Tarragona. One of the main lessons that we have learned is that considerable technological effort is necessary to develop a stack of processes and algorithms using Big Data approaches (with algorithms for Machine Learning, Natural Language Processing and Search), and furthermore that this technological effort is necessary but not sufficient: a high level of human intervention is necessary also before meaningful data can be derived from such opinion networks.

Technological Challenges

Emotions, needs, desires, fears, are being pushed to the Internet in a continuous stream. However, harnessing this information is not easy, and technology it still in its infancy.

A full stack of technology is needed to capture, filter, process, analyse and serve social media data. It needs to be shaped into a coherent set of summaries and indicators, which in turn enable a set of APIs and applications, from real time alert systems to periodic reporting.

Considerable technological effort is necessary at every level of the analytics stack that is required:

• Capture: support automatic access and querying to the myriad of Social Network sites and APIs.
• Filtering: Disambiguate the name of the city and of its main points of interest (e.g. there are more than 20 Barcelonas in the world). Disambiguate city conversation from its soccer team and from mentions in people’s addresses and job listings, which combined make up to 70% of the volume of captured mentions. Detection BOTs and other artificial methods to generate “false” conversations.
• Categorisation: Separate conversation about tourism, health, culture, security and so on (e.g. we use over 50 different topics at three different levels to analyse conversation in the city of Barcelona)
• Summarisation: Understanding which are the main trends for a given topic is crucial, we cannot descent to the level of the individual clipping since there are literally millions per month.
• Key Performance Indicators: Besides summarisation, we needed to derive numerical indicators.
• Big (Natural Language) Data: Almost all the processes mentioned above require sifting through millions of data points on demand, while receiving thousands of new data points per second. Worse of all, the main data points are free textual mentions which are unstructured.
• Visualisation: Providing powerful and intuitive dashboards to read summaries and indicators and drill down to specific mentions when necessary.

Beyond the technological problems, for this data to gain acceptance it is very important to:

• educate decision makers and citizens in general about the nature of this new “sensor” and the analytics that can be derived from them
• establish clear rules to the data collection process, its limits, transparency, limitations, etc.
• establish standards and procedures for capturing and sharing analytics denied from social media data.
• establish defensive systems capable

Opportunities

Social Network data, once properly filtered and analysed, can be exploited by cities in numerous ways. We will cite some of the areas in which we have worked in the last four years with the city council of Barcelona:

• City Tourism : We can analyse demographics of tourist, analyse comments made by tourist of different nationalities about different areas of interest (PoI) in a city. We can rank PoIs by their impact in Social Medias, track the evolution of less known PoIs which are being promoted by the City Council, etc.
• City Security and Crisis Response : Using statistical models, we can trigger alerts in real time based on unexpected Social Network activity. Depending on the topic of this alert we can channel it directly to the corresponding department (e.g. health vs. security vs. maintenance).
• Polling : sampling people’s concerns and interest before designing questions for a poll make polls more relevant and cost-effective
• Branding : we can analyse how large events (trade show, sports event, etc.) are connected to city brand.
• Benchmarking: we can draw comparative analysis of the importance (for the population) of issues in one city with respect to a second similar city.

---

Article written for presentation at the 2016 Smart City Expo.

Image from https://www.flickr.com/photos/55049135@N00/19038712376