Crisis Alert is an experiment is building a structured, explorable dataset from a mostly unstructured source. In this case - lots and lots of parliamentary debates.
It uses Natural Language Processing (specifically spacy) to analyse every sentence said in Parliament since 1803 to identify past crises. This involves looking for specific sentence structures that identify an "Oil Crisis" or a "Crisis of Confidence".
Once these were identified some functionally equivalent crisis were grouped. For example: "Housing Crisis", "Crisis in Housing" and "Recent Housing Crisis" are all effectively "Housing Crisis".
The idea is to create something that lets you explore UK political history (what we thought were the bad things going on at the time) - and see it presented in a context that helps you understand what else was (and is) going on.
Given the timeline of when a crisis appeared and dropped away, this can be compared to other crises to see if they share a similar pattern. If they do - they might well be different ways of talking about the same problem.
This uses the Pearson correlation coefficient to establish how related two crisis are - and above a certain threshold decides it is worth including.
This is a useful way of making the data more explorable - and drawing connections between entries without having to explicitly state them. They can be statistically discovered.
For an example of a success - it correctly works out the 'Financial Crisis' was related to the 'Banking Crisis'.
However, it's not universally successful. In some cases it draws connections between short crises happening at the same time. For instance, it tries to link the 'Falkland Crisis' to the 'Railway Crisis' - as these are both mostly an issue of 1982.
For historical data (1803 onwards), the Hansard archive was downloaded from http://hansard-archive.parliament.uk/.
For more recent data (post 1930s), I used the data mySociety have reprocessed and made available at http://parser.theyworkforyou.com/hansard.html
For data derived from mySociety - I can link back to the relevant debate in theyworkforyou.com.
For older references I attempt to link to the relevant section of Hansard available online through millbanksystems - however as the linking convention is a little variable these don't always link correctly.