De-duplicating the BBC goodfood ingredients

– Hello everybody.

Thank you very muchagain, joining us today.

I am Lju Lazarevic and like Adam, I'm also part of thedevelopment relations team

and I'm going to be talking to you about some approaches that I've used for dealing with duplicates.

So, some time ago, I was thinking about interestingdata sets to play with.

And I came across the BBC goodfood website so this is, for those of youwho are not based in the U.

this originally, whichwas created by the BBC, and there was a magazineseries and all sorts with it, is a very large repository of recipes and there's something like200, 000 recipes on there, it's huge.

And they're all freely available and you get many differentrecipes in different categories so it could be by meal, by nutritional requirements and so forth.

So they have a huge number of a huge number of recipes and I was quite keen to thinkabout, can I get at that data? So I don't necessarily wantto take the recipes and things but I was quite keen to see, there's some cool things thatyou can do with that data around ingredients so things such as, can you group together similar recipes? Is there some interestingthings you can do around recommendations and so forth.

There is a very rich dataset with many possibilities.

So, I was exploring thisdata, it's a big one, and I was looking at thesource code of the first pages and I spotted this veryhandy, Jason-esque looking nugget of data and I thought brilliant, I can take this data and this data has gotthings such as the offer, it has the ingredients, it'll have informationaround say, cook time, any keywords and so forth.

So it's a really nice, rich bit of information that's easy to pull out and extract.

This is something that me andmy colleague, Mark Needham, did a while back wherewe created some stuff to pull this data out.

And we eventually pulled itinto this data model here.

So there's loads and loads of data, but we kept it quitesimple and the idea is that we've got recipes, so recipe'sat the center of our universe of our data model, and here we've attached some information so we've got things around youknow, diet type of the recipe what collection did it come from? Attaching the number ofkeywords associated with it.

He wrote it and so forth, and of course, the one thatI'm really interested in is what ingredients area part of that recipe? And the purposes of whatwe're going to be doing in this session, or what we're looking at, is, as you'll see in a coupleof examples we'll show, there are a lot of duplicateingredients in the, in the data set and there'sgonna be many good reasons why we want to remove those duplicates.

So for example, maybe we want to do some kind of work, as being able to categorizetypes of ingredients, and maybe we want to get ahierarchy of different tomatoes, maybe what we want to try and do is, by de-duplicating our ingredients, we can start to spot duplicate recipes.

Or on a similar vein as well, based on the kind of ingredients there, maybe we can start to understand what kind of recipes would be a better recommendationfor somebody because it contains the same coreingredients and so forth, so there will be a huge number of reasons why we want to de-duplicatethose ingredients.

And when I talk aboutduplicates, there are many.

So just as an example, if we ran a query to find allof the ingredients in our data that contain the word, so basically, name is the name, it's the property thatcontains the ingredient name and we look for everythingthat contains almonds, we get a lot.

So as you can see, I seewe got almond and almonds.

So we've got plurals going on here.

We've got things like almond flakes, flaked almonds, flaked almond, almonds, ground almond, ground, so we've got a lot of a lot of duplication going on.

We've even got the one example here where we'vegot whole blanched almonds and we've got blanched almonds.

So we've got a lot ofdifferent things going on here.

We've got a situation aroundtenses being involved, we've got plurals involved, we've got an additional word in as well.

So we've got lots of things which come together whichmay construe duplicate data.

But it can't really callit duplicate data just based on my observation.

It's going to be duplicate data because I will have inadvertently, either deliberately or inadvertently, I'll have put together aset of conditions in my mind as to what woulddetermine, in this example, these ingredients to be duplicates.

And this is going to bedriven by business rules.

And this is always going to be based on what questions do we want to answer? So, I'll just give you a quick example.

So in the previous slides, I said that almond andalmonds were duplicates.

But this is working in the example where what I'm keen to do is to simplify down the ingredients and I'm saying in this example, plurals of words are duplicates but for example, if youwere looking at stock so if you had some kindof stock or stock keeping, it could well be then almond on its own and almonds could be twoseparate distinct entities and you wouldn't wantto group them together.

So you really have to think about what business rulesare going to be driving the decisions you've made to say whether or not twoentities are the same.

So in this example, the businessrules that I've come up with when thinking about how I'm going to de-duplicate theingredients, the data here, there are gonna be things around plurals.

So what I'm saying is, any plurals I want to get rid of because those are probably goingto link to the same entity.

Same with tenses.

So for example, we had almond flakes so that's in the presenttense and flaked almonds.

So we've got the D at the end of flakes, so that's in the past tenseso I want to get rid of those.

And I want to get rid of any stock words, so for example somebody might say ginger comma garlic, maybethey had ginger and garlic, so I want to get rid ofall the words like, 'and' or 'the' and so forth.

And the reason I wantto get rid of those is, when I'm doing, I'm doing thisas part of the data cleaning because I don't see those asuseful words in this context.

Again, it will depend onthe kinds of questions you're looking to answer in your context.

I also want to try and identifythings that sound similar.

So this list is kind of a mix of, the data cleaning rules I'm applying and these are both going to apply and I'm actually goingthrough and sorting the data which we're going to do shortly but it's also looking at what other things I'm going to do as well and what structures of thegraph I'm interested in.

So any things that sound similar, I want to group together andsay they might be the same.

I want to tokenize words and the reason why I want to do this, is I'm trying to removethe ordering of words.

So again, almond flakes, flaked almonds.

When I tokenize those and Iclear out all of the tenses and plurals and that kind of thing, it should match so you get the same.

So I want to tokenize to get rid of any ordering biases going on.

And I need to think about some kind of thresholds todeal with missing words, so for example, our 'blanched almond' and 'whole blanched almonds'.

So how can we skip those through? So just to give you an example, what I'm going to be doing, is I'm going to be adding an extra node to my data modelcalled the ingredient name and that ingredient name is going to be tokenized version of the ingredients.

It's going to have dealt with any plurals, any special symbols andso forth and so forth.

So let's think about whatwe're going to do next.

So I'm going to be doing a bunch of steps where I'm going to do some sortof rudimentary data cleaning and then the next stage, what I want to do, is I want to be a little bit clever now.

So we've done the verylocalized cleaning of the data and now what I want to do next is to use the graph structure to be able to suggestadditional ingredients that might be related to each other.

So if I just go back tothis original slide here, so example where I've got cherry tomato, and this splits out into cherry or tomato, the idea here is thatif I have beef tomato, then the tomato bit of beeftomato will refer to this node here.

And same as if I've got glacee cherries, the cherry bit of thatwill come onto here.

So effectively, we're building in theseconnections between the data and what we can start to think about is well we've added theseextra connections in.

How can we leverage that graph structure to be able to find out more information and more potential entities that are that are the same entity or candidates to beidentified as duplicates.

So first thing we're goingto have a look at is, node similarity.

And what we're going to be doing here, we're going to be using, effectively using ingredientname as our pivot point, we're going to try andcompare ingredients together and what we're trying to say is if there's huge amounts of overlap, so if you've got twoingredients and they've got many ingredient namesthat got mapped together, the more of that that goes on, the more of those structures, the more likely that they are similar.

And we're going to be using the neo4j graph datascience library for this so I'll show you some of the graphs.

And one of the functionswe're going to be using is node similarity.

Something to bear in mind andwhy I've pulled out this graph is, and we'll go throughan example of this, is to use the graph data science library, it's recommended that you load in memory, the graph that you want to process.

And this means you can run thealgorithms very efficiently and it's a great way of beingable to shape your graph based on the algorithm you want to run without having to makephysical changes to your, the graph on your database.

And so one of the structureswe're going to be loading into memory is this bipartite graph and it's bipartite because I'm using two different nodetypes in this situation, could be more.

And this is effectively, like I said, we're using ingredientname as a pivot point to sort of see how otheringredients compare to each other.

So comparing by ingredient name.

The next thing we'regoing to be looking at, again to leverage the graph structures, is we're going to use acommunity detection algorithm to try and group theseingredients together.

And what we're going to be doing, the first step that wedid, the node similarity, we're gonna go throughand when we identify nodes that are similar based to a threshold that we specified, it's going to create a relationship between those two ingredients so for example, almondflakes and flaked almonds, it'll create a relationshipbetween those two and give it a designation or similar to, and then what we're going to do, so we've put in those structures so we're using nodesimilarity to build out some additional helping structures in our graph and then what we're going to do, is we're then going toput another memory graph so again, another graphic memory, so again we are going to reshape the graph according to the nextalgorithm we're going to run so we're going to be runningLouvain community detection.

And the graph that we'rereshaping to bring into memory is this mono-partite graph of ingredient and we're interested in, we're effectively pulling in ingredients similar to ingredient so that's the graph structure that we're interested to bring in.

And what Louvain communitydetection basically does, it looks how nodes areconnected to each other and it tries to pull togetherdensely connected nodes together into communities and the theory is oncethat process is run, it'll say that, a node thatbelongs to a community, it's the best fit forit rather than if it was randomly assigned to another community.

So those are the two algorithms that we're goingto be running on a graph and again, we'releveraging those structures that we have from the initial clean up that the new structures that we put in when we run node similarity and then we're gonna pullsome communities out.

So let's have a look at this in action.

So just to make you aware, so what I've done, so this data is already loaded into neo4j.

I've already done someof the cleaning steps but I'll talk youthrough what's happening.

So just as a rough idea to see how many ingredients we've got.

We've got, I think it's, yeah 3, 077 ingredients.

So the first phase that I went through in going through this data is to just try and do some cleaning up.

So for example, this query here is trying to go through andfix any of the encoding issues that I've got in the data.

There's probably acleaner way of doing this but this is one approach that I used.

So this is going throughand cleaning the data.

And then same idea, I want to try and deal with any errant characters.

So for example, I want toget rid of things like, dealing with any bracketsor dealing with quotes, all that kind of thing.

So just trying to standardize that because there may be different waysthat people are representing or accessing or using, possession so, there might be varying levels of grammar as how people have beensubmitting recipes and that.

So we're just trying to deal with that.

So these are a couple ofcleaning steps that I'm doing.

And then the next step I want to do is I want to start thinking about tokenizing those ingredients.

So, what I've got here, so we're matching all of theingredients in the database.

I'm turning them all to lower, so I want to get rid of anyissues with capitalization, and I'm going to besplitting them by space so now tokenizing each of the items.

So here, this iseffectively looping through all of the items that I've now split and it's going to create new nodes and notice that we're usingmerge here so the idea is, we want to we want to have uniqueingredient name points so if it already exists, we'renot going to create a new one.

So something to bear in mind, so I've done a few splits here.

I've not updated this yet so prior to, I believe 4.

0, actually no I think it may have come in a later version of 3.

X, a change in cipher was made so what you can do now is you can provide an array of characters that you will want to split both, so you don't have torun split multiple times which is what I've done in this example so I will update the blog post which I'll give you a linkto, which this is based on, with the updated splits.

So, I'm gonna take nodes, byspace, I'm going to run this and let that go and because we're going tobe doing a lot of stuff on we're going to be doing a lotof stuff on ingredient name, I will create an index butlet this query run first.

And then we shall do the next part.

Excellent, so I'm gonna create an index which will make the, all the other (mumbling) youcould've done index before the process all ran which probably would've made more sense so the next thing I'm going to do is I'm going to buildall of those hyphenated, any hyphenated words, so again, if there's a newer version of the new sort of array of characterswe can use that to split so anyway I'm going tojust quickly do that and just a quick thing, toflag what I'm doing here.

So I'm doing the same approach here and what I'm what I'm trying to do aswell is just detach delete I.

I'm trying to also yeah so this is to dealwith this situation where this is to deal with this situation where you've created ingredient namethat's got hyphen inside it and we don't want tokeep the ingredient name the hyphen and the two new ingredient name nodes that come off so what's going on here is, we're finding out theoriginal ingredient name so it's component of in, so this is the original ingredient name that's got the hyphenated words.

We're doing the split and then we're remappingthe new split items back onto the ingredient and then we want toget rid of the original ingredient name that has the hyphen so that's what's going on.

We're just cleaning up here.

So we'll run that.

And the next couple ofthings we're going to do is we want to get rid of what we want to do is get rid of some get rid of any stock words, anything like 'and, the, this, with' I also want to get rid ofwords like, 'so' too 'cause I'm again, making the assumption and this is like a domainspecific assumption that, anything that'stwo characters or fewer, is not going to be an ingredient name.

It's probably going to be a stock word or something like that and the thing to bear in mind as well, these are all theingredient name components so the idea is, it doesn't matter, we're using these to create structures in our graph so it may not be a problemif you lose one because we're interested in all ofthe other ingredient names that map and crossover.

So this is a relativelysafe assumption to make for my use case, given thedomain and the business rules that I've specified.

So I'm going to do thisand get rid of those ones.

And then the next, the meaty ones.

So next what we want to try anddo is deal with the plurals.

So, if you think about, there's going to be a numberof ways that you have plurals.

So we're going to have thingslike, things ending with S so those are the straightforward ones.

We also have words, typically if the word ends with a vowelthen the plural might be ES.

So you think about tomatoand tomatoes as an example.

And you also have thingsmodified with an S so, again this is a bit of a messy way of us trying to go through anddeal with those plurals so effectively what we're doing is quite an expensive query here.

We're trying to bring backbasically two pairs comparing every single ingredient name against every other singleingredient name in the database and we're just trying to compare and if it turns out that they match, then we're going to getrid of one of those.

So this is how we'retrying to do with plurals.

So there's an example so let's run that.

And as you can see, we'restarting to use more fuzzier and fuzzier approachesto deal with some of the, to deal with cleaning thespace and getting it ready.

So next thing we're going to do here is we're going to do somefuzzy matching based on the the word lengths in the words.

So Levenshtein distance is avery common one to use in this.

I started to use Sorensen Dice Similarity so again this is going to check the words based on the threshold I've provided here and it's gonna go throughand do the fuzzy matching and again, this is tryingto deal with tenses and any plurals and again, any sort of spares, we'regonna get rid of those ingredient name nodes.

So this will take a little while as it is quite an expensive query so what we're effectively doing is, again, we're doing this sort of, query where we're trying to compare every singledifferent instance away, we've got two ingredient names.

And doing that comparison.

For those of you who are going hey, I want to use Sorensen Dice Similarity, that sounds brilliant, that's available in the APOC library so, if you're using the FPJ desktopyou can add that by plugins.

So that's one of the APOCfunctions that we're using here and another APOC functionwe're going to use is doublemetaphone.

So this one here, we're lookingat the structure with words and we're using a specific one.

And this is basically having looked at, is there a letter or so out and based in ratio, and thensaying well those are similar and it's going to get rid of those.

Another thing we can do, which I use here, is I used doublemetaphoneand what it's going to do, is we're using SorensenDice Similarity to try and reduce the number ofitems we're going to use with doublemetaphone becauseagain, this is quite expensive.

So we're gonna filter out some of those.

And then doublemetaphone isbasically trying to find out if two words sound thesame, so the spelling, the spelling might be different but do these two ingredientname components sound the same? So for example, if you think about like a ton, you may spell ton as T-O-N oryou may have ton as T-O-N-N-E but they both have the same sounds so in this example, I've castthose as both being the same so that's what this is going to do and again, if it finds two components, two ingredient name componentsthat sound the same, it's going to get rid of one of those.

So we're now going to run this one now as the last one has completed.

And okay so we're gonna let this run and then we're gonna have a quick look at just by carrying out thesefirst cleaning steps, we're going to have a look at how many duplicate ingredients have we found, so I'm gonna let this run, and quickly talk you throughwhat's going on here.

So oh this is a verylazy query, I got myself.

So effectively, we're goingto go through the mapping and yeah actually I think I'm going to step past the script now 'cause this is quite an interesting query but what I'm basically trying to do, I'm using lead section andthe buffer to join through and see which ingredients joined together so, what we're trying to.

One moment, I think I'vemissed something off my query.

I shall check up on that.

So basically what's happening, what's happening in this query is it's starting from it's starting from ingredient name but what we're trying to do is we're trying to go from we're trying to findall of the ingredients that have exactly thesame number of ingredient ingredient names between them so this is like 100% intersection so we don't want to have one, well this one's gotthree out the four names and this one's got, we literally just aresaying, exactly the same.

So this is basically trying to say what have we linked together.

So let's just quickly sort that by name.

So you can start to get anidea of what's happened.

So this is just by getting rid of the sort of any characters, any sort of words, (stuttering) two or fewer characters.

We're getting rid of stock words, we're getting rid of plurals, we're dealing with, we're trying to ignore wordordering, that kind of thing.

So you can see just bydoing that initial cleaning, you can see straightaway we're identifying, we're being able to identifyand get rid of a lot of potential duplicates, so you can see here.

So garlic and ginger, you can see we've got sugar.

Great one here, wherewe can see an example of where we've gotten rid of stock words.

So we've got corn cob and corn on the cob.

Again, so just by doingthat data cleaning, we've got really sort of great data set.

We've already, you cansee an example here.

So 348, so you can see how many potential duplicate groupings we'vediscovered by doing this.

So, we've done that great piece of work.

So let's okay.

Right so, what we can do next is, we're now going to use nodesimilarity to try and find try and find both inintersection so bearing in mind, this is often absolute match so both of these ingredientsgot an absolute same ingredient name component.

What we're going to do now is we're going to use node similarity and it's kind of goingto do a similar thing but we're giving it a cut off so it's going to be lookingat the structure and comparing and comparing intersectionbased on the ratio of those ingredients.

And what it's going to dois that all the ingredients that meet those specifiedthresholds or cut off, it's going to create asimilar two relationships between those two ingredients.

So we're using we're going to be using thegraph data science library so what we need to do is, well what I'm going to be doing here is I'm going to becreating an in memory graph and this memory graph is going to be exactly the structure that I want to be able to run this algorithm.

So you call it using this, GDS graph create.

So again, if you're using the FPJ desktop and you want to installgraph data science, again you click on the plugin option and that has the graphic science plugin.

So I'm going to give it a name, so I'm gonna give my graph asimple name of 'similarity'.

And what you do, is the first part you say what nodes you want to use so I'm using the cipher, the cipher crease so I'm using cipher to form what my graphis going to look like.

You can also do a direct load from disk if you know that everythingin the communal space you can pull up all of thelabels or all the relationships.

So you've got a number of options so, have a look at the graphdata science documentation.

There's a link to that atthe end of this presentation.

So what I'm doing here, I'mcreating my bipartite graph so what I want to do, is I want to match all of the nodes that areingredient or ingredient name and I'm going to return the ID's for that and the next part, I'mdeclaring what relationships, so I'm saying how theseconnect, I'm interested in.

So I'm interested in ingredients being connected to an ingredient name.

So I'm gonna return those back.

So I'm going to go offand create that graph.

I've been a bad citizen and I've forgotten to clean.

Clean the graphs.

Okay cool, so I'm gonna create my graph.

And then I'm going to run node similarity.

So here it is again, you call function, GDS name of the algorithm and something to bear in mind is the structure's very ordered when you call the graphdata science algorithm so it's always in this pattern of GDS dot name of the algorithm that's run dot what do you want to do so write is, it'llwrite back to the graph.

You gotta stream, it'llstream the results out.

You've got stats that'll giveyou statistic information about what's gonna happen and so forth.

So it's always in that same format.

So I want to write an effective graph because I want to writethose relationships between similar ingredients and (stammering) my graph is called similarity so that's what I'm gonna call it.

This is my similarity cut off.

So, this is the, I'm saying Iwant a cutoff of 0.

8 so, what I would suggest is go and have a play withthe different cutoffs so if you've got time, I'll have a go and show you what happens when you usedifferent cutoff values.

And you can start to see theimpact on the ingredients.

We want a, I want to write relationship so that the relationship typeis going to be similar to.

I want to write score aswell on the relationship.

So that's something we can do as well.

In fact, what I could probably do is I could drop this and do want to freestyle slightly.

We'll leave it for now.

So basically this is what I'm going to do.

So I've got this cutoff that, so I'm now going to run thisalgorithm and it's going to run that function solet's have a quick look at what it has done.

So I'm just basically saying here, using that similar to, sodone, so this is going to basically record the ingredientsthat met that threshold whilst in that similarity.

I'm gonna collect the names and I'm going to order them by size, basically which oneshave been tagged together as most of them are similar.

Excellent, so you start to get an idea sohere we've got some fun ones.

So yes so here you go, hereyou got a great example one.

You've got the skinlessboneless chicken breast filet and you've got like bonelesschicken breast filet so it skipped the skinless so then you get bonelessskinless breast filet so you kind of get the idea so you can see it startingto do this thing where it's it's dealing with that extra word because there's enough commonalityamong those groups that it's fine and it meets that.

So we've done that, so as a next thing, let's try and join thesetwo communities now.

So I'm going to doanother in memory graph.

I'm gonna call this one community.

And what I'm doing now, so thisis now a mono-partite graph I'm going to now store memory 'cause we're only using one label.

And the one label we'reusing here is ingredients and we're now, what we're going to do is we're going toleverage those structures that we've added to the graph with the previousalgorithms that we've run.

So I want to load up all ingredients and I'm interested in how ingredients are similar to other ingredients, so I'm using that connection there.

So I'm gonna load up on this memory and then I'm going to run the Louvain community detection algorithm.

So what's happening here? So you'll notice here, I'm not using a writer, I'm using a stream and I'm doing this because actually I wantedto create an external node.

And I want to, that node isgoing to have the community ID and I want to link the ingredients off of that community node, that's how I've chosen to do it.

What you can do, you can write, you can write the properties of the nodes what we did Louvain, but Iprefer to have a separate node.

So you have that flexibility as well.

So I'm calling the Louvain algorithm.

I'm yielding the nodeID and the community ID so node ID is going to be ID of the node.

Community ID is the communityidentifier the node belongs to I'm basically turningthe ID into a node now.

It's going to return me atthe node, for community ID and then what I'm going to do is merge current entity whichis gonna be my community so I'm merging that soif it already exists, don't create it again.

And I'm going to merge that that entity contains a member of I.

So, I'm going to run this.

And once that's completed, we're gonna have a look at we're gonna have a lookat some entities in here.

So effectively whatthis is doing is saying match the entity that has ingredients and then we're going toreturn the entity ID.

I'm going to collectall of the ingredients associated to that, to that node.

And I'm just going toorder those based on the communities that have got themost number of ingredients.

So there we go, so you can see now, it's started to join these together so we can see we've got onecommunity with the chicken we've got another communitywith the garlic ginger paste, and so forth so you can start to see what we've been able to do togroup these things together and group the ingredients.

And what we can do as well is, if we wanted to dropthe threshold quickly, so we've got a couple ofminutes so I'll quickly do that so you can get anidea of what's going on.

So I'm going to I'm going to get rid ofthose two memory graphs.

I'm going to delete the entity and thesimilarity relationships.

I'm then going to recreate the the similarity graph.

I can probably, for thesake of completeness, we shall do it like this.

So then going to run this and I'm going to drop thethreshold down to six.

And then I'm going to do these both again.

So that's going to, so we're repeating thatbut what we've done here is we've dropped the similarity threshold so what I could have done, so this is still in the same set up from when I was still using the predecessor to graph data science which is graph algorithms library, and so what I could have done is if I'd written all of theresults from here, is the score then I could have justreloaded the community graph.

So again, that's somethingwe'll update later.

So let's run this, so this will probably take afew seconds, it's pretty quick.

Cool and just as a comparison so we, oh I forgot to drop the thing.

Okay, sorry I forgotto drop the threshold.

So we shall, oh no I did, it's cool.

So you can see an idea, so I dropped the similarity threshold, the thresholds, the cutoffs for the for the node similarity so youcan see what we've got now is we've got much broader sort ofgroupings of things together so you can see all the flours together, all the sugars together, but what you might findis things like sugar syrup and these things joining together.

So you have to bear in mind, when you're using these and you're working withthe thresholds about the dangers of over thethreshold are being set that you over fit yourdata so this is where you picked a threshold thatfits the test data that you used but it doesn't give youthe expected results when you use it later and the challenge aroundunder fitting your data where your threshold is, your threshold is really really high and you can inadvertentlymiss some results.

So that was very quicklygoing through that.

And, so we've got a couple of links as well.

So this is all basically ifyou want to read through this and have a go yourself.

The blog post that I've beenreferring to is this one so it's a Medium article, it's got a very long link so this is why I've givenyou the shortest one here.

If you're not using the FPJ desktop but you're using an FPJ server and you want to download thegraph data science library you can get it in the download center and you've also got the documentation here for the graph data sciencelibrary which gives you an explanation of what eachof the algorithms does, and it gives you examplesabout how to execute them how to load your memorygraphs and that kind of thing.

And one thing I will mention before we go, don't forget to clean upyour graphs after you finish so generally it's recommended that you only have one graph memory.

I was a bit naughty 'causeI know my data's quite small so I used two so I couldquickly switch between them.

But do clean your graphs after use.

https://erickbpod520.hatenablog.com/entry/2020/04/17/193045

https://www.liveinternet.ru/users/i1ouocr879/post469293687//

https://www.evernote.com/shard/s697/sh/a0f587be-f102-a4d1-7b79-e00c63db4d6f/aed968912bc39d1951fc39e241d36ac4

http://damienkalm974.simplesite.com/445750000

https://telegra.ph/nordvpn-review-setup-tutorial-my-unbiased-vpn-review-installation-on-desktop-mobile-04-17

https://andersonoywi613.tumblr.com/post/615639537875550208/nordvpn-review-setup-tutorial-my-unbiased-vpn

http://edgarceve260.bravesites.com/entries/general/nordvpn-review-setup-tutorial---my-unbiased-vpn-review---installation-on-desktop---mobile

https://alexisrgvl943.edublogs.org/2020/04/17/nordvpn-review-setup-tutorial-my-unbiased-vpn-review-installation-on-desktop-mobile/

http://troyzonm337.wpsuo.com/nordvpn-review-setup-tutorial-my-unbiased-vpn-review-installation-on-desktop-mobile

https://penzu.com/p/2eb7a931