DSI and DELIS

The DELIS (Dynamically Evolving Large-scale Information Systems) European project deals with methods, techniques, tools, and prototypical implementations in order to cope with challenges imposed by the size and dynamics of today's and especially future information systems. Special emphasis is on exploiting synergies between the different groups that are part of the project. The fundamental features of the way networks and systems are studied within DELIS stem from the focus on how individual (user) node behavior impacts the network as a whole, and the focus on whether it is feasible and how to attain a useful networking infrastructure, given individual, self-centered behavior. Thus, the key technical challenges in DELIS aim to create a self-organizing, self-repairing, self-monitoring network infrastructure, given largely autonomous, perhaps egoistic and selfish peers, coexisting with altruists. The DELIS subproject structure exemplifies this.

The DSI (Computer Science Department) of the Università di Milano is part of the Subproject 1 (Monitoring, Visualizing, and Analyzing Large Dynamically Evolving Information Systems) aimed, in particular, at the analysis of the structure of large dynamic networks, specifically of the web graph: given a portion of the web, its web graph is a directed graph whose nodes are the web pages, and with an arc from node A to node B iff there is a hypertextual link to page B contained in the text of page A. Web graphs are of uttermost importance in various fields such as, for example, in the way search engines organize their results, in the search for virtual communities, in the analysis of web spam etc.

With the aim of studying real-world web graph and their evolution in time, we started collecting in a systematic way large portions of the Web; this is the starting point to allow other partners, for example, to assess their statements and models about the temporal evolution of the statistical and topological properties of the web graph and the possible impact of different crawling strategies.

How/when/where the snapshots were collected

Together with our DELIS partners, we decided to take snapshots at a monthly rate focussing on the .uk domain: the choice of the domain was the most obvious, given the European nature of the project and in consideration of the linguistic and social centrality of the UK within Europe; the frequency chosen is the largest possible that does not raise issues of unpoliteness. The first snapshot was collected in May 2006. All snapshots were collected at the DSI, using hardware that was partly funded by the DELIS project.

All the data sets were collected using UbiCrawler, a scalable, fault-tolerant and fully distributed web crawler developed by the Laboratory for Web Algorithmics (LAW) at the DSI, and the corresponding web graphs are made available in this page in the WebGraph compressed format --- please, read the tutorial to learn how to use such format.

How to get the data

Presently the data is available only to the DELIS project members; six months after the end of the project, all the data will be made publicily available.

If you partecipate to the DELIS project and need to access the data, please join the LAW-DELIS mailing-list.

The snapshots

Full-texts

Graphs

Overlap (host count)

Overlap (static URLs)